facing connection pooling issue with redis and mysql

[Pramod Bisht]

Dear All,

I am new to gevent, recently we replace our multi-threaded python daemon to gevent based daemon, it work excellently, it helped us in reducing our resources. But now problem which we face that connection to Redis and Mysql have shoot up. We want to avoid that. FYI.. we are using connection pooling for Redis. Could you guys help me out on that. Let me know if any other info is also required.

Thanks

[Grady Player]

You haven’t really given us anything to go on, the packages you use to connect to the db’s, the way you pool connections, when you monkeypatch, the way you handle errors all come into play...

My stab in the dark guess is that something in you pooling code was better behaved because it used blocking io for synchronization. 

Grady

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Madden]

I have another stab in the dark based on an experience we had: python-memcache wants to connect to the memcache server in a thread-local. That becomes a greenlet-local if you monkey-patch threading before you do the imports. If you then have long running greenlets, you can wind up with many more memcache connections than you expected. Since greenlets are "lighter" than threads, it's easy to design a programming scenario where you use one greenlet per client connection. In our case it was websocket connections that were each handled in a dedicated greenlet, so our number of memcache connections was directly tied to the number of websocket connections; gevent and the greenlets and the websocket handling scaled, memcache did not scale as high. :shrug:

tl;dr: Check for anything that's greenlet/threading local, and check the lifetime of your greenlets to see if the excess corresponds. (See also https://github.com/gevent/gevent/issues/981 for a scenario in which locals could appear to leak if they were constructed repeatedly in a greenlet.)

[Stephan Preeker]

Well You can create separate MySQL / Redis gevent worker(s) which are always running and only wait /  send / recieve from a  shared queue and have all gevent proceses write to the queues. There would be a redis queue and a MySQL queue in your case.

Then you have very fine grained control of the amount of connections you use.

Something to put you on the right path:

gist.github.com/mike820324/8683398

Cheers

Stephan.