DSE Cassandra Node Failed To Start Post OS Upgrade
One of our Cassandra production cluster node refused to start after OS Patching was done with following errorr.
ERROR [main] 2017-06-25 18:09:40,906 CassandraDaemon.java:709 - Exception encountered during startup
org.apache.cassandra.io.FSReadError: java.io.EOFException
at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:142) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_66]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_66]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_66]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_66]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_66]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_66]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_66]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_66]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_66]
at org.apache.cassandra.hints.HintsCatalog.load(HintsCatalog.java:65) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293]
at org.apache.cassandra.hints.HintsService.<init>(HintsService.java:88) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293]
at org.apache.cassandra.hints.HintsService.<clinit>(HintsService.java:63) ~[cassandra-all-3.0.8.1293.jar:3.0.8.1293]
at org.apache.cassandra.service.StorageProxy.<clinit>(StorageProxy.jav
Upon looking around, we found that there is a reported major bug with this error under this JIRA as following.
CassandraCASSANDRA-12728 Handling partially written hint files
Cause –
Corruption to the hints tables causing Cassandra to go in failure loop. This could have happen due to following.
1. Node was rebooted before service was shutdown properly.
2. service went down abruptly while writing Hints table.
3. Node rebooted due to power failure.
Since the cause of the issue was corrupted Hints table, we need to cleanup the hints for the node and then try to restart.
After that node started fine. Also, since the node was down, it is imperative to run the repair on the node to make sure the data is consistent.
Hope that helps.
