Weird Cassandra ConsistencyLevel Error

[Fabian Köhler]

We are getting the following ConsistencyLevel error from Cassandra from one of the KafkaProducer Actors:

- akka.actor.OneForOneStrategy

- akkaSource: akka://contract-9d97dcc6/system/sharding/kafkaProducer-routingEvents/CTR4/CTR4/producer

Exception: java.util.concurrent.ExecutionException:

com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (2 responses were required but only 1 replica responded)

The stack trace does not list any lagom classes because it is all async.

What I do not understand is where the ConsistencyLevel ALL is coming from. Is this from within Lagom somehow? We did not change the default consistency levels that Lagom provides.

Any ideas?

[Joo Lee]

From my experience, this happens when the Lagom keeps Cassandra very busy by persisting a lot into it. 

It keeps Cassandra so busy that Cassandra is unable to respond to the READ requests properly.

One of our services has the end-point that allows the caller to trigger some kind of batch job, that involves calculating some numbers and publish the events.

Whenever we ran this end-point, we used to see quite lots of Consistenty Level warnings like that.

And this is not a Lagom specific error message. 
To my understanding, this is the error message from Cassandra which basically says that "At least two responses from the nodes within the cluster was required according to your setting, but I got the response from only one node!".
We fixed the issue by increasing our Cassandra nodes from 3 to 5, and also by retiring some subscribers who were listening to those events unnecessarily to reduce the WRITE loads whenever we trigger that end-point.