Side effects of monkey patching
Pankaj Garg
Jim Fulton
IMO, monkey patching is a great convenience for small scripts.
It's a really bad idea for large applications with many 3rd-party
libraries.
You don't have to do monkey patching to use gevent, although
it sounds like you do need to use monkey patching to use
the async module of the requests library, which is a shame.
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
Denis Bilenko
I disagree. In a large application, possibly serving hundreds of clients, you don't want some greenlet accidentally blocking all others. Without monkey patching, you need to inspect all the 3rdparty code to make sure it does not touch network.
It's relatively easy to miss such code. For instance, lxml is an xml-parsing library. Yet it can fetch schemes from the net using urllib2. If you did monkey patching, you're safe even if you did not know that. If you did not monkey patch, you're blocking the whole process during those fetches.
I recommend to add monkey.patch_all() by default to all gevent-using applications (but not libraries, which preferably should work with and without monkey patching).
Denis.
Jim Fulton
I agree do disagree -- very much. :)
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://www.dublinstore.com/
Matt Billenstein
> It's relatively easy to miss such code. For instance, lxml is an
> xml-parsing library. Yet it can fetch schemes from the net using urllib2.
> If you did monkey patching, you're safe even if you did not know that. If
> you did not monkey patch, you're blocking the whole process during those
> fetches.
without modifying the stdlib?
--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/
Matthias Urlichs
> On Wed, Apr 11, 2012 at 07:08:07PM +0400, Denis Bilenko wrote:
> > It's relatively easy to miss such code. For instance, lxml is an
> > xml-parsing library. Yet it can fetch schemes from the net using urllib2.
> > If you did monkey patching, you're safe even if you did not know that. If
> > you did not monkey patch, you're blocking the whole process during those
> > fetches.
>
> How would you handle this case without monkey patching? Is it even possible
> without modifying the stdlib?
>
blocking functions and replace them with something that switches
greenlets-or-whatever, or you change the calling code not to use the
blocking functions in the first place.
There's a limited number of blocking functions in Python, so monkey-
patching them is easy. In contrast, the callers of those functions are
not under your control, assuming that you even know about all of them
(which you don't).
I'm mildly uncomfortable with monkeypatching myself, but I don't see any
better way, except (as Matt said) modify the stdlib to know about
gevent / greenlets. However, in a way we're already doing exactly
that. ;-)
--
-- Matthias Urlichs
Matt Billenstein
> I'm mildly uncomfortable with monkeypatching myself, but I don't see any
> better way, except (as Matt said) modify the stdlib to know about
> gevent / greenlets. However, in a way we're already doing exactly
> that. ;-)
So, unless you own all the code all the way down to the socket, it seems
impractical to not monkey patch?
Jim, maybe you have other experience here?
thx
m
Jim Fulton
> On Wed, Apr 11, 2012 at 08:14:17PM +0200, Matthias Urlichs wrote:
>> I'm mildly uncomfortable with monkeypatching myself, but I don't see any
>> better way, except (as Matt said) modify the stdlib to know about
>> gevent / greenlets. However, in a way we're already doing exactly
>> that. ;-)
>
> Yeah, +1, I wouldn't want to be carrying around patches to the stdlib...
>
> So, unless you own all the code all the way down to the socket, it seems
> impractical to not monkey patch?
>
> Jim, maybe you have other experience here?
Code that uses the standard library (or other 3rd-party libraries) has
certain expectations about it's semantics. Changing the semantics
behind it's back is a recipe for disaster.
If you have a networking (or any other) library that blocks, and you
don't want to put it in a thread pool, then use some other
library. <shrug>
BTW:
"There's a limited number of blocking functions in Python"
This is incorrect. Reading from a file blocks. Computation
blocks. Lots of things block.
The sweet spot for purely a async architecture is an application (like
half of mine :) that just moves bits between sockets. Anything else,
should be prepared to block (like the other half of mine), which is
why async frameworks are often paired with thread pools.
Matthias Urlichs
> Code that uses the standard library (or other 3rd-party libraries) has
> certain expectations about it's semantics. Changing the semantics
> behind it's back is a recipe for disaster.
*its (both times …)
Most if not all libraries which use blocking are perfectly happy to be
used in a thread, though. Which means that, unless with greenlets, a
different thread can pull the rug from under them at any time it damn
well pleases, not just while they happen to block.
So I fail to see any problem with monkeypatching that the "use a thread"
alternate solution would not also have. In spades.
> "There's a limited number of blocking functions in Python"
>
> This is incorrect. Reading from a file blocks. Computation
> blocks. Lots of things block.
True. If that's a problem, then you need to farm out the work to
Something Else.
As far as I am concerned, however, I'd rather use some sort of
multiprocessing. OS threads don't even let you use a second CPU,
due to the GIL.
--
-- Matthias Urlichs
Matthias Urlichs
> unless with greenlets
*unlike. Sheesh. ;-)
Matt Billenstein
> As far as I am concerned, however, I'd rather use some sort of
> multiprocessing. OS threads don't even let you use a second CPU, due to the
> GIL.
this for awhile and I'm really pleased with it.
