ReadTimeoutException

[Salvador Calogero]

Hi guys, I'm new to this group and also to Cassandra.

We are facing the following exception in our production enviroment:

com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)

We are querying over 8M records and this just happens. We are using the drive version 2.0.1

Can you help? Any ideas?

[Sylvain Lebresne]

You made a query that Cassandra was not able to respond to within it's configured read timeout. Generally this mean one of
1) you've loaded your Cassandra nodes too much, C* is not able to keep up and so some queries timeout
2) you're making queries that either insert or request too much data at once, and C* is just not able to perform those queries within it's alloted timeout

3) you have some nodes that are dead such that there is no replica available for the data inserted/requested.

Check if you have dead nodes to know if it can be 3), and check your monitoring to figure out if 1) is likely. If none of those are likely, then it's probably 2). For 2), you can always increase the timeout Cassandra side (in the cassandra.yaml), but the default timeouts are not particularly low so the proper fix is probably to make sure that your client code don't do queries that insert or read big amounts of data at once.

--

Sylvain

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Salvador Calogero]

Thanks Sylvain for your help

We are running queries with the in statement and it’s limited to 10k

So our query looks like this

SELECT userid FROM userextendtaxonomies2 WHERE age=0 AND gender=3 AND education=0 AND marital=0 AND income=0 AND interest=0 AND cluster=0 AND purchase=0 AND career=0 AND brand=0 AND connection=0 AND country='' AND userid IN ('1eef5b17-6e8d-4759-a4f9-5b83a2a76049')

 

In the in statement we put up to 10k userid. We also tried with 5k and we had the same error message

We run the same query in cqlsh and we did not find any issue.

[Neha Trivedi]

Hi Salvador,

Did you find a fix. I am having similar problem?

regards
Neha

[Rushan Qaisrani]

Use this
cluster.getConfiguration().getSocketOptions().setReadTimeoutMillis(600000);
.......
Also change the following settings in Cassandra.yaml
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
and you will be fine....

[Andrew Tolbert]

Use this
cluster.getConfiguration().getSocketOptions().setReadTimeoutMillis(600000);

I would recommend against having such a high read timeout (600000), since this is when the driver determines that a node has timed out and releases the request from the connection.  Using the default (12000) is sensible as long as the timeouts in cassandra.yaml are 10000 milliseconds or less.

In Salvador's case I am guessing his table is partitioned by user_id, and if that were the case this would likely require reading all those partitions, therefore hitting every node and reading through a lot of sstables, which is prone to taking a long time.

If you are having a similar experience, I would recommend enabling tracing (driver: http://www.datastax.com/documentation/developer/java-driver/2.1/java-driver/tracing_t.html cqlsh: http://www.datastax.com/documentation/cql/3.0/cql/cql_reference/tracing_r.html) to understand why queries could be timing out.

Thanks,

Andy

[Rushan Qaisrani]

I was just pointing out about the options available. Just randomly put that value for timeout (to show he can give a bigger number to make it work).
Yes tracing will allow you to understand what nodes and caches etc your hitting. But from my little experience it's about the schema of the tables. What's your partition and clustering keys. Also do check out different compactions available. They affect the read a lot.
And as the error States your timing out before your query completes. It means your hitting a lot of sstables. So timeout sittings might be a dirty fix.

[Rushan Qaisrani]

Also also considering that driver is token-aware. It would be more efficient to send an individual query for each partition key (SELECT * FROM users WHERE id = ?) which would reach the right replica directly. Then all that’s left is to collate the results client-side.
http://www.datastax.com/dev/blog/java-driver-async-queries