Speculative executions, retries and read / write timeouts

[Pavel]

Hello, I am observing a behavior which I consider to be strange, probably I misunderstood something around speculative execution. I have speculative execution set up on cluster where I am executing an idempotent read or write queries, but I am getting read or write timeouts sooner than I would expect without retries which I would expect to happen.

Now some more details:
- cluster has 3 nodes, cassandra.yaml configures “read_request_timeout_in_ms: 5000, write_request_timeout_in_ms: 2000”.
- replication factor is 3
- I am setting up cluster with 9 s timeout (cass_cluster_set_request_timeout(m_cluster, 9000) – I assume this is the total timeout for a query to be executed?)
- the cluster has speculative execution on, using 2 retries after 3 s (cass_cluster_set_constant_speculative_execution_policy(m_cluster, 3000, 2))
- statements are set to idempotent (cass_statement_set_is_idempotent(m_statement, cass_true)), they are read or write statement with some bound parameters, the write statements in question are prepared
- the read statements in question are using consistency level 1, the write statements consistency level local quorum

I have 3 different problems:
- Sometimes during the normal system run, I get message like “Read timeout: Operation timed out - received only 0 responses.”. I get this message after the query execution time (cass_session_execute to cass_future_error_code duration) of 5.005 s. Why is that? Shouldn’t the query be re-tried by the speculative execution and time out after 9 s if the other executions fail / do not respond within the time interval as well? Also, this is the read query with consistency level one, it is kind of hard to believe that none of the 3 nodes which should hold the replica were able to respond at all, even without the retry.
- Similar problem with writes, the write queries are using local quorum, errors similar to “Write timeout: Operation timed out - received only 1 responses.” appear after 2.002 s. Again, why is nothing retried automatically?
- When I unplug one of the nodes, some of the clients report “Request timed out: Request timed out” after 9.002 s. This appears like they were not able to contact coordinator node. Again, shouldn’t the speculative execution kick in? Shouldn’t the speculative execution then attempt to contact different coordinator?

Can somebody please explain me what is happening in 3 above described problems? Why am I getting back errors sooner than after 9 seconds / why am I getting them at all when I have speculative execution? What are the relations between the speculative executions and timeouts? Which timeout settings are applied in which situations and when are things retried? (total query timeouts? timeouts for contacting coordinator? timeout of reading from nodes / writing to nodes? when are things retried for idempotent queries?)

Kind regards,
Pavel Cernohorsky

[dakshita chaturvedi]

Hi Pavel, 
Hope you're well.
Were you able to figure out the reason behind this issue and how did you solve it?

Thanks and Regards,
Dakshita

[Bret McGuire]

   Hey Dakshita, thanks for the question!  I thought it might be useful to add a reply to Pavel's original question.  This might at least add some context to your question and also provide some general information.

   The general answer to Pavel's question is that speculative execution is not intended to be a retry mechanism for requests that have timed out (or otherwise failed).  Speculative execution is primarily intended to attempt the same query on other nodes in parallel (or at least generally in parallel, plus or minus a specified delay).  If one of the speculative executions finishes before the original execution then the result of that execution can be returned, otherwise we wait on the original request execution.  The general flow would be something like the following:

1. A request is sent to a node
2. Additional speculative requests are sent to other nodes, possibly after some delay
3. One of these requests completes and sets the value in the corresponding request future
4. If the request fails or otherwise receives an error the retry policy is consulted to see if the request needs to be retried

   You can find more information on retry policies in the documentation.

   In the case cited by Pavel in his original email the "Operation timed out" errors (both the read and write variants) are server-side errors; note that the delay in which they occur corresponds to the configured read and write timeouts in cassandra.yaml (5s and 2s respectively).  The speculative execution configuration in use here (specified via cass_cluster_set_constant_speculative_execution_policy(m_cluster, 3000, 2)) will cause 2 speculative requests to be sent after a 3s delay.  You'll only have time to get one of those off in the read case and you won't get any out before the write timeout kicks in.  Regardless, the decision about whether to then retry either of those requests is made by the configured retry policy; speculative execution isn't in play there at all.

   A similar analysis applies to the third question; just as speculative execution isn't involved in any retry calculation for read and write requests it doesn't change anything when a client loses it's connections to one or more nodes in the cluster.  In this case the configured client-side timeout (cass_cluster_set_request_timeout(m_cluster, 9000) in this case) kicks in.

   Hopefully this information is useful!

   - Bret -