com.datastax.driver.core.exceptions.BusyPoolException

[Kant Kodali]

Hi All,

I get the exception below and I believe this is happening because I was trying to make too many async calls even though I am just running async calls on Single Thread. Cassandra is up and running and is in Normal state so "NoHostAvailableException" is a bit misleading however the actual error seems to be 

"com.datastax.driver.core.exceptions.BusyPoolException: [/127.0.0.1] Pool is busy (no available connection and the queue has reached its max size 256)))"

I am running everything on Localhost and and using Cassandra 3.7.

Can someone shine light on whats happening and how to fix it?

Thanks!

Stack Trace

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [/127.0.0.1] Pool is busy (no available connection and the queue has reached its max size 256)))

at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:220)

at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:50)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:291)

at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:358)

at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)

[Alexandre Dutra]

Hi Kant,

Yes, this means that you are submitting too many requests, and not waiting for the futures to complete before submitting more.

The default maximum number of requests per connection is 1024. If this number is exceeded for all connections, the connection pool will enqueue some requests, up to 256. If the queue gets full, a BusyPoolException is thrown.

Of course you can increase the max number of requests per connection, and the number of max connections per host. But the real solution is of course to throttle your thread. You could e.g. submit your requests by batches of 1,000 and then wait on the futures to complete before submitting more, or use a semaphore to regulate the total number of pending requests and make sure they don't exceed a certain number (in theory, this number must stay below num_hosts * max_connections_per_host * max_requests_per_connection – in practice, I don't suggest going above 1,000 as it probably won't bring you more throughput).

Thanks,

Alexandre

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

--

Alexandre Dutra

Driver & Tools Engineer @ DataStax

[Al]

Hi , 

I am new using cassandra. I just wanted to ask regarding the answer below (I have the same problem as above):
"submit your requests by batches of 1,000 " as far as I know batch inserts are not recommended.

what happens in production systems when load of inserts is expected ?

regards

Al

[Alexandre Dutra]

Hi,

First of all, where did you learn that batch inserts are not recommended? They actually are, provided that you do it the right way. Check the following links to understand when to batch and when not to:

https://docs.datastax.com/en/cql/latest/cql/cql_using/useBatchTOC.html

http://batey.info/cassandra-anti-pattern-misuse-of.html

http://batey.info/cassandra-anti-pattern-cassandra-logged.html

That said, what I meant by "submit your requests by batches of 1,000" was not to send one batch request with 1000 child statements, it was rather: send 1000 requests, and then stop sending more and wait until these 1000 pending requests complete before continuing, in order to avoid overwhelming the connection pool. I should have avoided the word "batch" :)

Hope it's more clear now,

Alexandre 

[Al]

thank you very much. that make sense 

[Kant Kodali]

In general this is a Flow control problem if there is reactive version of Cassandra driver that would be great so Cassandra driver can send a feedback to upstream saying at what rate it can consume so an upstream can adjust to that.

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.

[Alexandre Dutra]

Hi,

There is no backpressure mechanism in the native protocol, so what you are asking for is not easily done, but you can always try; the best place to ask is on the Cassandra user mailing list, and you can also vote for CASSANDRA-11380 which targets roughly the same idea.

As for a reactive driver, that's on the roadmap; please vote for JAVA-1505 if you are interested by this feature.

Thanks,

Alexandre

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.