[cluster6-connection-reaper-0] WARN com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.48:9042-13, inFlight=0, closed=true]. This should not happen and is likely a bug, please report. 18:54:00.37

[Per Ahlbom]

We receive  this warnings and error in our service that reads from a Cassandra cluster.

18:54:00.375 [cluster5-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.48:9042-5, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

18:54:00.375 [cluster4-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.49:9042-5, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

18:54:00.384 [cluster5-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.55:9042-17, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

18:54:00.385 [cluster5-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.54:9042-1, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

18:54:00.384 [cluster3-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.55:9042-1, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

18:54:00.384 [cluster1-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.49:9042-5, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

18:54:00.385 [cluster3-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.55:9042-9, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

19:04:57.938 [cluster3-reconnection-0] ERROR com.datastax.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 1000 millisecond

19:15:33.819 [cluster4-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.54:9042-4, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

19:15:33.819 [cluster6-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.48:9042-8, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

19:15:33.819 [cluster6-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.55:9042-19, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

19:15:33.819 [cluster6-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.54:9042-5, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

19:15:33.819 [cluster5-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.55:9042-7, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

19:15:33.819 [cluster3-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.49:9042-10, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

We are using these pooling options:

.setConnectionsPerHost(HostDistance.REMOTE,20,20)
.setConnectionsPerHost(HostDistance.LOCAL,20,20)

the query option: 
.setFetchSize(20000)
The Cassandra driver version:
3.0.0

 

[Olivier Michallat]

Hi Per,

18:54:00.375 [cluster5-connection-reaper-0] WARN  com.datastax.driver.core.Connection - Forcing termination of Connection[/10.48.12.48:9042-5, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

This comes from an internal component that ensures that resources are properly cleaned up when connections get closed. Normally this should happen automatically, therefore we log a warning if the component has to do it (but resources are still cleaned up properly).

This could happen if there is saturation in the driver. Did you observe any unusual CPU or GC activity?

Also, did you customize SocketOptions.readTimeoutMillis?

Not necessarily related, but 20 connections seems a bit high. Which protocol version are you using? If it's v3 (Cassandra 2.1 or higher), one connection should be fine for most usages. See this page for more information about pooling:

http://docs.datastax.com/en/developer/java-driver//3.1/manual/pooling/

I also noticed that you have multiple Cluster instances (at least 6 according to your logs). Not an issue at all but I just wanted to check that it's by design.

19:04:57.938 [cluster3-reconnection-0] ERROR com.datastax.driver.core.ControlConnection - [Control connection] Cannot connect to any host, scheduling retry in 1000 millisecond

This indicates that the control connection is switching to another host because the initial control host went down or became unreachable. As long as the driver reconnects to another host, this is not a major issue. In hindsight, ERROR level is probably too high for this log, I'll create a ticket to fix it.

--

Olivier Michallat

Driver & tools engineer, DataStax

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.

[Olivier Michallat]

This indicates that the control connection is switching to another host because the initial control host went down or became unreachable.

Sorry I misspoke: the control connection tries all hosts every time it reconnects. So this means that no host was reachable at the time the reconnect happened.

My remark about the log level still applies: this is an infrastructure issue, not specific to the driver.

--

Olivier Michallat

Driver & tools engineer, DataStax

[varun....@cengage.com]

I'm ran into same kind of situation like above but only with one server: Forcing termination of Connection[`ip-address`, inFlight=0, closed=true]. This should not happen and is likely a bug, please report.

And we observe that cql responding slowly and then app server went down for couple of mins and get back normally. 

No unusual CPU and GC activity. It worked well all the time from so many months.

Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.