implementing gevent-dependant modules

[Ofer Nave]

Is gevent.monkey.patch_all() idempotent?

I'm building a simple service discovery module in Python which assumes/depends on gevent.  Right now I'm calling patch_all() in the main script and not in the module.  However, the module will only work if patch_all() has been called.  Is it safe to call it in both the script and the module so that I, as the module writer, can ensure that it has been done, rather than relying on the developer who will use the module?

More generally, what are some best practices for implementing packages/modules that assume a gevented, monkey-patched environment?  I'd love to learn more about such best practices.  I've already run into one gotcha here:

https://github.com/SiteSupport/gevent/issues/250

-ofer

[vitaly]

It's highly advised that monkey-patching be done at the beginning of the main application script, before other imports. Imagine an app or some other imported module that caches something from a module that is patched by gevent monkey - e.g., g_socketClass = socket.socket. So, by the time that your module is imported, it may be too late to patch.

That said, it is often a bad idea to do system-level patch (especially patch_all) in a "library" module. Although easy and it seems to work at first, this system-level patching may conflict with some applications and cause unpredictable, difficult to debug failures. For example, I have reported and seen reports from others about failures when using the pymysql with dbutils shared connection pool in combination with gevent monkey-patching of socket (which is patched by gevent.monkey.patch_all()). If you have to patch at all, I find it better to perform "directed" patching instead: e.g., to make Thrift client gevent-friendly, instead of patching at system level (possibly causing side-effects), I patch only Thrift like this: import TSocket; TSocket.socket = gevent.socket

Cheers

[Ofer Nave]

To answer my own question:

I don't know if monkey.patch_all() is idempotent, but it seems wise to not depend on it being so, nor to depend on it being called at all -- and certainly not to call it within a module.

Instead, I've avoided the problem entirely by using gevent.socket.socket() in my module rather than the stdlib's socket.socket().  That way I'm using gevented sockets without depending on monkey patching.

This is probably simple and obvious to everyone else, but it took me a bit to "get it".

-ofer

[vitaly]

There are, unfortunately, some 3rd party modules that monkey-patch at system level, thus making those modules impossible to use in apps where the system-level monkey-patching conflicts with the apps' other logic.