Many errors in reaper.log after replacing a node

[Jan]

Hi,

I have a problem with reaper after replacing a dead node. This entry shows up in huge amounts in our logs:

INFO   [cluster:00000000-0000-0003-0000-000000000000] i.c.s.RepairRunner - Updating the seed list for cluster cluster as topology changed since the last repair.

ERROR  [cluster:00000000-0000-0003-0000-000000000000] i.c.s.RepairRunner - RepairRun FAILURE, scheduling retry
java.lang.IllegalStateException: Cluster should not be persisted with UNKNOWN state
    at com.google.common.base.Preconditions.checkState(Preconditions.java:459)
    at io.cassandrareaper.storage.PostgresStorage.addAndUpdateClusterAssertions(PostgresStorage.java:238)
    at io.cassandrareaper.storage.PostgresStorage.updateCluster(PostgresStorage.java:224)
    at io.cassandrareaper.service.RepairRunner.updateClusterNodeList(RepairRunner.java:303)
    at io.cassandrareaper.service.RepairRunner.run(RepairRunner.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:117)
    at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:38)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
    at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

What can I do to fix it?

Thanks in advane,

Jan

[Alexander Dejanovski]

Hi Jan,

could you try registering the cluster again with a node that's currently valid in the cluster? No need to remove it before doing this, it should update the whole list of nodes in the cluster table.

Cheers,

-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

http://www.thelastpickle.com

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/e49cf585-0a49-41f5-b236-73e62f42f96d%40googlegroups.com.

[enercast SysOps]

Hi Alexander,

 

thanks for that idea and indeed that worked like a charm. Hopefully someone with the same issue will find this topic :D

 

Cheers,
Jan