Database connection (to cassandra) times out in a view function, but works in __init__.py
1,462 views
Skip to first unread message
Taylor Gronka
Jun 7, 2014, 1:25:24 AM6/7/14
to pylons-...@googlegroups.com
I can connect to my database in __init__.py, but connecting to the
database in a view function times out. I'm using the DataStax python
driver. I get the same result if I create the connection somewhere else
and import it, so I really doubt it's caused by a timeout due to initialising the connection.
about to create client
about to create client
SimpleClient created
Traceback (most recent call last):
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/pyramid-1.5-py3.4.egg/pyramid/router.py", line 242, in __call__
response = self.invoke_subrequest(request, use_tweens=True)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/pyramid-1.5-py3.4.egg/pyramid/router.py", line 217, in invoke_subrequest
response = handle_request(request)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/pyramid-1.5-py3.4.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/pyramid-1.5-py3.4.egg/pyramid/router.py", line 163, in handle_request
response = view_callable(context, request)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/pyramid-1.5-py3.4.egg/pyramid/config/views.py", line 355, in rendered_view
result = view(context, request)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/pyramid-1.5-py3.4.egg/pyramid/config/views.py", line 501, in _requestonly_view
response = view(request)
File "./tacle/event_views.py", line 63, in event_list
caclient.connect(['127.0.0.1'])
File "./cassaconn/clients.py", line 22, in connect
self.session = cluster.connect()
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 546, in connect
self.control_connection.connect()
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 1498, in connect
self._set_new_connection(self._reconnect_internal())
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 1533, in _reconnect_internal
raise NoHostAvailable("Unable to connect to any servers", errors)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1': OperationTimedOut('errors=Timed out creating connection, last_host=None',)})
[pid: 2130|app: 0|req: 2/2] 127.0.0.1 () {38 vars in 641 bytes}
[Fri Jun 6 21:47:43 2014] GET /event_list => generated 0 bytes in
5007 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
And the view code:
@view_config(route_name="event_list", renderer="templates/events/list_events.jinja2")
def event_list(request):
tacs = {}
print('about to create client')
caclient = SimpleClient()
caclient.connect(['127.0.0.1'])
print('able to connect to cassandra server')
#caclient.catest()
print("---------page reload-----------")
return dict(resource=request.context, tacs=tacs)
SamuraiNinja007
Jun 9, 2014, 4:10:10 AM6/9/14
to pylons-...@googlegroups.com
The database works fine from the view code if I only import SimpleClient within the same view file. This kind of implies that importing SimpleClient (and therefore the python driver) in multiple places is what is breaking my code.
Here is the cassandra stack trace when I set logging.basicConfig(level='DEBUG'). There was a lot of debug code here that I'm not including, which basically showed that multiple connections were created before I ever called my connect function. All of the code I attached is from loading the view - multiple connections are already open, although when my request comes in, the connection is closed immediately.
Here is the cassandra stack trace when I set logging.basicConfig(level='DEBUG'). There was a lot of debug code here that I'm not including, which basically showed that multiple connections were created before I ever called my connect function. All of the code I attached is from loading the view - multiple connections are already open, although when my request comes in, the connection is closed immediately.
I've noticed that simply importing the SimpleClient() without running any other code causes cassandra to generate a new transaction token.
I'm not entirely sure how to solve this, but I'll start by looking at SQLalchemy or other pyramid code; I'm hoping that it's simply an issue of scoping.
I'm not entirely sure how to solve this, but I'll start by looking at SQLalchemy or other pyramid code; I'm hoping that it's simply an issue of scoping.
DEBUG:cassandra.pool:Host 127.0.0.1 is now marked up
DEBUG:cassandra.cluster:[control connection] Opening new connection to 127.0.0.1
DEBUG:cassandra.connection:Sending initial options message for new connection (140149920838936) to 127.0.0.1
DEBUG:cassandra.io.asyncorereactor:Closing connection (140149920838936) to 127.0.0.1
DEBUG:cassandra.io.asyncorereactor:Closed socket to 127.0.0.1
WARNING:cassandra.cluster:[control connection] Error connecting to 127.0.0.1:
Traceback (most recent call last):
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 1524, in _reconnect_internal
return self._try_connect(host)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 1541, in _try_connect
connection = self._cluster.connection_factory(host.address, is_control_connection=True)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 503, in connection_factory
return self.connection_class.factory(address, *args, **kwargs)
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/io/asyncorereactor.py", line 139, in factory
raise OperationTimedOut("Timed out creating connection")
cassandra.OperationTimedOut: errors=Timed out creating connection, last_host=None
ERROR:cassandra.cluster:Control connection failed to connect, shutting down Cluster:
Traceback (most recent call last):
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 546, in connect
self.control_connection.connect()
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 1498, in connect
self._set_new_connection(self._reconnect_internal())
File "/home/taylor/projects/tacle/env34/lib/python3.4/site-packages/cassandra/cluster.py", line 1533, in _reconnect_internal
raise NoHostAvailable("Unable to connect to any servers", errors)
cassandra.cluster.NoHostAvailable: ('Unable to connect to any servers', {'127.0.0.1': OperationTimedOut('errors=Timed out creating connection, last_host=None',)})
DEBUG:cassandra.cluster:Shutting down Cluster Scheduler
DEBUG:cassandra.cluster:Shutting down control connection
DEBUG:cassandra.cluster:Not executing scheduled task due to Scheduler shutdown
SamuraiNinja007
Jun 9, 2014, 4:15:32 AM6/9/14
to pylons-...@googlegroups.com
Forgot to mention - the stack trace is only there to help other people find the topic.
Tres Seaver
Jun 10, 2014, 9:30:25 AM6/10/14
to pylons-...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 06/09/2014 04:10 AM, SamuraiNinja007 wrote:
> The database works fine from the view code if I only import
> SimpleClient within the same view file. This kind of implies that
> importing SimpleClient (and therefore the python driver) in multiple
> places is what is breaking my code.
I would investigate how that module (or maybe others it imports) use
globals: in particular, any globals associated with "THE" connection or
its settings, state, etc.
Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlOXCFMACgkQ+gerLs4ltQ78/QCgrIlh5GBAXjObcxh0eVsxe0xb
ZkUAoKBgjNyMFUefbZmL8BhffYY+seLE
=gtFd
-----END PGP SIGNATURE-----
Hash: SHA1
On 06/09/2014 04:10 AM, SamuraiNinja007 wrote:
> The database works fine from the view code if I only import
> SimpleClient within the same view file. This kind of implies that
> importing SimpleClient (and therefore the python driver) in multiple
> places is what is breaking my code.
globals: in particular, any globals associated with "THE" connection or
its settings, state, etc.
Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlOXCFMACgkQ+gerLs4ltQ78/QCgrIlh5GBAXjObcxh0eVsxe0xb
ZkUAoKBgjNyMFUefbZmL8BhffYY+seLE
=gtFd
-----END PGP SIGNATURE-----
SamuraiNinja007
Jun 12, 2014, 1:26:22 AM6/12/14
to pylons-...@googlegroups.com, tse...@palladion.com
I think I found the cause.
I was looking at using a uwsgi decorator @postfork to make it so that I could declare an import once and only once. In researching it, in the uwsgi docs I came across a section called "2.3.9 The fork() problem when you spawn multiple processes", which mentions that when a uwsgi server is started:
"By default uWSGI loads your application in the first spawned process and then fork() itself multiple times.
It means your app is loaded a single time and then copied.
While this approach speedups the start of the server, some application could have problems with this technique (especially those initializing db connections on startup, as the file descriptor of the connection will be inherited in the subprocesses). If you are unsure about the brutal preforking used by uWSGI, just disable it with the --lazy-apps option. It will force uWSGI to completely load your app one time per each worker."
I verified that my Cassandra session, connection, and query runs properly when enabling --lazy-apps. Using @postfork should be another solution.
I was looking at using a uwsgi decorator @postfork to make it so that I could declare an import once and only once. In researching it, in the uwsgi docs I came across a section called "2.3.9 The fork() problem when you spawn multiple processes", which mentions that when a uwsgi server is started:
"By default uWSGI loads your application in the first spawned process and then fork() itself multiple times.
It means your app is loaded a single time and then copied.
While this approach speedups the start of the server, some application could have problems with this technique (especially those initializing db connections on startup, as the file descriptor of the connection will be inherited in the subprocesses). If you are unsure about the brutal preforking used by uWSGI, just disable it with the --lazy-apps option. It will force uWSGI to completely load your app one time per each worker."
I verified that my Cassandra session, connection, and query runs properly when enabling --lazy-apps. Using @postfork should be another solution.
Jonathan Vanasco
Jun 12, 2014, 11:17:34 AM6/12/14
to pylons-...@googlegroups.com, tse...@palladion.com
fwiw, uwsgi has a `post_fork_hook` that might be useful for you.
in order to use pycrypto on uswgi , you have to do something like this in pyramid:
import uwsgifrom Crypto.Random import atforkdef post_fork_hook():atfork()uwsgi.post_fork_hook = post_fork_hook
atfork() just proxies a call to "_UserFriendlyRNG.reinit()" [ see https://github.com/dlitz/pycrypto/blob/master/lib/Crypto/Random/_UserFriendlyRNG.py ]
SamuraiNinja007
Jun 13, 2014, 6:21:37 PM6/13/14
to pylons-...@googlegroups.com, tse...@palladion.com
I appreciate the leads, Tres and Jonathan - little leads have saved me many times so far on this project haha.
What I'm doing right now is beyond simple. Since I'm having trouble finding postfork samples online, I'll show it:
#models.py
from cassaconn import SimpleClient #cassaconn is a module I created
CaSession = SimpleClient()
#__init__.py
from uwsgidecorators import *
from cassaconn.clients import SimpleClient
from .models import CaSession
@postfork
def connect_cassandra_client():
CaSession.connect(['127.0.0.1'])
When I want to use Cassandra methods, I import CasSession and my SimpleClient class.
What I'm doing right now is beyond simple. Since I'm having trouble finding postfork samples online, I'll show it:
#models.py
from cassaconn import SimpleClient #cassaconn is a module I created
CaSession = SimpleClient()
#__init__.py
from uwsgidecorators import *
from cassaconn.clients import SimpleClient
from .models import CaSession
@postfork
def connect_cassandra_client():
CaSession.connect(['127.0.0.1'])
When I want to use Cassandra methods, I import CasSession and my SimpleClient class.
SamuraiNinja007
Jun 13, 2014, 6:34:19 PM6/13/14
to pylons-...@googlegroups.com, tse...@palladion.com
Omg, right after I figured this out a thread came up on the Cassandra listserv.. ahah
Reply all
Reply to author
Forward
0 new messages