How to close session properly ?

[Vincent CY L]

Hi,

After noticed my program crashed with out-of-file-descriptor exception I made the following test.

and noticed that on driver 3.7.1 Session.shutdown() may not always close and release the connection.

The test program do the following things:

1. creates a Cluster object

2. loop the following for 1000 times:

 2.1. open a session with "system" keyspace

 2.2. make queries to "schema_keyspaces" table

 2.3. shutdown session

 2.4. call "ls /proc/$PID/fd" every 10 loops to print a list of open file descriptors

3. call "ls -l /proc/$PID/fd" to print detailed list of open file descriptors

The testing environment consist 3 boxes:

- ubuntu 12.04 server x3

- apache cassandra 2.2.3

- python 2.7.3 (version shipped with ubuntu)

I ran tests inside virtualenv (1.7.1.2) with two different cassandra-driver version:

2.6.0 and 3.7.1 both driver built with Cython (0.23.4 and 0.24.1 respectively)

I noticed that on driver 3.7.1, after several iterations,

there are few connections left open after Session.shutdown() invoked.

The situation not observed on 2.6.0 driver.

The test program code is pasted at:

https://bitbucket.org/snippets/cheyinl/aaqRM#file-cassandra_session_fd_close_test.py

The results of both test are pasted at:

https://bitbucket.org/snippets/cheyinl/aaqRM#file-output-260.txt

https://bitbucket.org/snippets/cheyinl/aaqRM#file-output-371.txt

In output-371.txt, you can noticed that after 390-th iteration there are more FDs left open:

https://bitbucket.org/snippets/cheyinl/aaqRM#output-371.txt-93

I checked the left open sockets with `netstat -anp`,

they do link to Cassandra port (9042) on one of the server.

Am I close session in the wrong way ?

Thank you for your advice !

[Greg Bestland]

Vincent,

Thanks for the thorough report. With all the context you provided I was easily able to reproduce your problem.
We have concluded that this is likely related to the async connection pool we added recently. You can work around the problem you encountered by adding 
wait_for_all_pools=True to your session connect call. 

dbconn = cluster.connect("system", wait_for_all_pools=True)

This should eliminate the issue with sockets not getting closed.

Also, just for future reference, cycling sessions in the manner you describe is kind of an anti-pattern and should be avoided unless absolutely necessary.

I took the liberty of opening a ticket for your issue here. 
https://datastax-oss.atlassian.net/browse/PYTHON-673

Until we get it resolved I recommend using the wait_for_all_pools=True when connection your sessions.

~Thanks a ton for helping us spot this problem!

Greg Bestland.

--
You received this message because you are subscribed to the Google Groups "DataStax Python Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-driver-user+unsub...@lists.datastax.com.

--