Designing PyMongo 3.0 for a mix of Gevent and threads

[A. Jesse Jiryu Davis]

Hi Geventers, I have a question about designing a library for people who use Gevent and threads in the same process.

I help maintain PyMongo. In PyMongo 2.x we have special code to support users who want to call patch_socket() but not patch_thread(). Users can set the "use_greenlets" option and PyMongo will explicitly use Gevent's Greenlets in place of threads. It still imports socket from stdlib, so it gets Gevent's patched sockets.

But I'm not sure anyone needs this pattern, since, after all, Gevent sockets don't work properly with real threads. So in PyMongo 3 I'm going to delete the "use_greenlets" option. Users will be required to call monkey.patch_all() to use PyMongo with Gevent.

I wrote about this decision in It Seemed Like A Good Idea At The Time: PyMongo's "use_greenlets":

http://emptysqua.re/blog/it-seemed-like-a-good-idea-at-the-time-pymongo-use-greenlets/

A user commented,

"We have a multithreaded application which cannot monkey patch threading, uses Gevent only in the main thread, and only needs to use PyMongo in that main thread. Our application is structured this way because we need non-patched threading support so we can interface with some extension modules which block. We release the GIL when calling them."

I have two questions:

1. Is this a common pattern? Would you recommend keeping PyMongo's special code to support this pattern?

2. Is monkey.saved a stable API? Can a user rely on monkey.saved['thread']['start_new_thread'] to start a real thread, provided that thread doesn't use any monkey-patched functions or objects from stdlib?

Thanks!

Jesse

[Matt Billenstein]

I've developed a couple complex python+gevent stacks and I think this pattern
is becoming less common - we sometimes had to mix gevent and os threads when
there was socket code buried in C extensions that couldn't be monkey patched
(the old zookeeper lib for example - kazoo solved this).

I think given the evolution of the python ecosystem -- pypy wanting a python
implementation and library authors being more aware of greenlet based
applications (and providing hooks to support them even in C-based libraries
like psycopg2), that your rationale for not having gevent-specific code to
support greenlets + os threads is good.

People who really need it will find a workaround - and it makes the much more
common case of a single-threaded monkey-patched application simpler.

m

> --
> You received this message because you are subscribed to the Google Groups
> "gevent: coroutine-based Python network library" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gevent+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/

[Denis Bilenko]

> 2. Is monkey.saved a stable API? Can a user rely on monkey.saved['thread']['start_new_thread'] to start a real thread, provided that thread doesn't use any monkey-patched functions or objects from stdlib?

There are no plans to remove it, so it can be considered stable.

[A. Jesse Jiryu Davis]

Thanks for your answers, this was really helpful.