Monday, June 26, 2017

DSE Cassandra Node Failed To Start Post OS Upgrade 

One of our Cassandra production cluster node refused to start after OS Patching was done with following errorr.

ERROR [main] 2017-06-25 18:09:40,906 - Exception encountered during startup
        at org.apache.cassandra.hints.HintsDescriptor.readFromFile( ~[cassandra-all-]
        at$3$1.accept( ~[na:1.8.0_66]
        at$2$1.accept( ~[na:1.8.0_66]
        at java.util.Iterator.forEachRemaining( ~[na:1.8.0_66]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining( ~[na:1.8.0_66]
        at ~[na:1.8.0_66]
        at ~[na:1.8.0_66]
        at$ReduceOp.evaluateSequential( ~[na:1.8.0_66]
        at ~[na:1.8.0_66]
        at ~[na:1.8.0_66]
        at org.apache.cassandra.hints.HintsCatalog.load( ~[cassandra-all-]
        at org.apache.cassandra.hints.HintsService.<init>( ~[cassandra-all-]
        at org.apache.cassandra.hints.HintsService.<clinit>( ~[cassandra-all-]
        at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.jav

Upon looking around, we found that there is a reported major bug with this error under this JIRA as following.

CassandraCASSANDRA-12728 Handling partially written hint files

Cause – 
Corruption to the hints tables causing Cassandra to go in failure loop. This could have happen due to following. 

1. Node was rebooted before service was shutdown properly. 
2. service went down abruptly while writing Hints table. 
3. Node rebooted due to power failure. 

Since the cause of the issue was corrupted Hints table, we need to cleanup the hints for the node and then try to restart. 

After that node started fine. Also, since the node was down, it is imperative to run the repair on the node to make sure the data is consistent. 

Hope that helps. 

No comments:

Post a Comment