Configuring timeouts

[Pierre]

Hi,

I got exception ReadTimeoutException :

com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)

I did read https://groups.google.com/a/lists.datastax.com/d/msg/java-driver-user/zqHIjLV4kKI/vPeKRw3pf9YJ and I think I'm in case 2) (node not available to process the query in time) because I never had problem with this query but I did a maintenance on the node at the time of the query, moving a 250GB files from hd1 to hd2, both being used by cassandra for commitlog and datadir, making the node very long to respond to this big query.

So it lead me to some questions :

1) Datastax driver (client side) use timeout configured on the node (server side) by cassandra.yaml ? So nothing to configure client side ?
2) If yes, did the client give up after timeout, or did the client receive a "I give up" from the node which detected itself it was to slow to process the query ?
3) What is bothering me is I don't have any error logged in my cassandra node logs, is it ok ? So I assume the server was able to process the query, but too late and the client was disconnected ? Or maybe the server didn't even receive the query ?
4) Did the driver retry the query on the same node ? I'm using the DefaultRetryPolicy.INSTANCE ? (I m in a single node configuration)
5) Is it the good way to enable retry logging : .withRetryPolicy(new LoggingRetryPolicy( DefaultRetryPolicy.INSTANCE )) ?

Full stacktrace if it may help : https://gist.githubusercontent.com/anonymous/1382de1d04823530bdc7/raw/e7ee092638a886089492945dce33322325f168b4/gistfile1.txt

Thank you.

[Andrew Tolbert]

Hi Pierre,

To answer your questions:

1) Datastax driver (client side) use timeout configured on the node (server side) by cassandra.yaml ? So nothing to configure client side ?

Correct, the driver (client) side does not control this kind of read timeout, it is dictated by the configuration on the server side in the cassandra.yaml.  The only reason you would need to configure anything on the client side is if you configured the read timeouts in cassandra.yaml to be above 10 seconds, in which case you should change the read timeout in the driver.

 

2) If yes, did the client give up after timeout, or did the client receive a "I give up" from the node which detected itself it was to slow to process the query ?

The driver receives this timeout from the coordinator node on the Cassandra side.  Depending on your RetryPolicy configuration the driver may retry the request.

3) What is bothering me is I don't have any error logged in my cassandra node logs, is it ok ? So I assume the server was able to process the query, but too late and the client was disconnected ? Or maybe the server didn't even receive the query ?

I don't believe you will see timeout errors on the cassandra end, although you may see messages about 'DROPPED' messages (if the request would have timed out before it got into the executor stage to execute).  You may be able to see timeouts by increasing the log level, but looking at cfstats / cfhistograms might give you a better understanding of how long queries are taking and what the distribution of latency is.

4) Did the driver retry the query on the same node ? I'm using the DefaultRetryPolicy.INSTANCE ? (I m in a single node configuration)

Correct, it retries on the same node on a ReadTimeoutException, the issue typically is not with the coordinator node you are communicating with, it's with the replicas the coordinator is trying to read data from not responding in a timely manner.

5) Is it the good way to enable retry logging : .withRetryPolicy(new LoggingRetryPolicy( DefaultRetryPolicy.INSTANCE )) ?

Yes, that is the right way to enable retry logging.

Thanks,

Andy

[Pierre]

Thanks a lot Andrew, your answer was perfect.

By the way, I reduced the timeout in cassandra.yaml to reproduce the bug and lowered to DEBUG the log level, now I see the timeout error in cassandra too :)

Never used cfstats / cfhistograms, going to rtfm about that.