Timeout while executing Async statements

[Padma Asrani]

Hi All,

We are seeing lot of timeouts while executing async statements, I am pasting the code snippet we use to query Cassandra. We see lot of timeouts (C* client timeout while retrieving) for rows which are not in the database. We are waiting for future for 3 seconds which is lot of time, I am not sure why Cassandra doesn't returns within that amount of time, we thought of increasing the timeout on future but it is going to impact the performance of application. we are using Consistency level ONE.

We are using 2.1.7 version of datastax driver and Cassandra one 2.1.9. 

String preparedStatementBody = "SELECT * FROM domain WHERE key = ?";
BoundStatement bs;
try {
    PreparedStatement domainMetadataPS =
            CassandraBase.getInstance().getPreparedStatement(preparedStatementBody)
                    .setConsistencyLevel(ConsistencyLevel.ONE);
    bs = domainMetadataPS.bind(getDomain());
} catch (Exception e) {
    logger.error("Failed to get domain from Cassandra", e);
    throw new BRTSCassandraException("C**: " + getDomain(), e);
}

if (bs == null) {
    throw new BRTSCassandraException("Failed to create BoundStatement");
}

ResultSetFuture future = cassandraSession.executeAsync(bs);
ResultSet resultSet;
try {
    resultSet = future.getUninterruptibly(3, TimeUnit.SECONDS);
} catch (TimeoutException e) {
    boolean futureAlreadyFinished = future.cancel(true);
    if (!futureAlreadyFinished) {
        logger.info("Future already finished/returned while calling cancel");
        throw new BRTSCassandraException("Race in Future during handling of TimeoutException for " + domain, e);
    } else {
        logger.info("C* client timeout while retrieving " + domain, e);
        throw new BRTSCassandraException("Query timeout in client while retrieving metadata for " + domain, e);
    }
} catch (Exception e) {
    // catch all other potential exceptions, as we want to rethrow and
    // NXDomain this request anyways
    throw new BRTSCassandraException("Unusual C* exception while retrieving " + domain, e);
}

if (resultSet == null || resultSet.isExhausted()) {
    throw new DomainNotFoundException(domain + " not found in C*");

} 

Thanks

Padma

[Vishy Kasar]

Is it possible that cassandra co-ordinator is in long GC pause when this happens? 

Did you try using Speculative Query Execution? That may help you get response faster. 

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Kant Kodali]

This code doesn't look right to me. Timeout's can happen because of the async code although that may not be the case here. you have to debug through few steps if you really want to find out where things aren't working but a simpler thing to do is to use ListenableFuture interfaces. In your case why dont you try replace resultSet = future.getUninterruptibly(3, TimeUnit.SECONDS) with resultSet = future.get() and see that works? because that would prove you were able to contact cassandra and retrieve the result. once that works then you can replace resultSet = future.getUninterruptibly(3, TimeUnit.SECONDS) with

ResultSetFuture future = session.executeAsync(bs);

Futures.addCallback(future,

    new FutureCallback<ResultSet>() {

        @Override public void onSuccess(ResultSet result) {

            // do whatever;

        }

 

        @Override public void onFailure(Throwable t) {

            // do whatever;

        }

    },

);




Keep things simple!