fork safety on unix --- gevent use-case

[Jason Madden]

Hi all,

(There's a TL;DR at the bottom---I'd like to contribute code to help libuv work after a fork() on unix.)

I'm one of the gevent maintainers. gevent is a fairly popular[1] Python library for asynchronous event-driven programming (primarily networking, but other things as well). Currently gevent is based on libev and supports both unix and Windows while implementing an API compatible with the Python standard library.

There's been interest expressed by our community in moving off of libev and onto libuv, primarily for improved Windows support. So recently I've implemented a prototype backend for gevent using libuv [4].

Let me start by saying how much of a joy the libuv API is to work with. It's clean and consistent, and the source code is easy to read. Not all event loops are like that. I really appreciate the work that the maintainers and contributors have put into libuv.

I've been able to get gevent's test suite to pass with libuv on OS X and Linux, with the exception of those tests that involve forking but *not* execing. Being able to continue to run Python code in a forked child process is a crucial part of the way many Python programs on unix are written, including not just network servers like gunicorn but those that use the standard library 'multiprocessing' feature to improve concurrency.

Currently after a fork, libuv either is in an interesting state where things may or may not function (three different results in gevent's test suite on three different linux distros) or it simply aborts the whole process (kqueue, aix and sunos-based systems). I've been unable to find a reliable workaround to this.

Unless gevent can survive a fork, I won't be able to base it on libuv. As a gevent maintainer, that makes be sad because libuv is so nice to work with (and I'm really not interested in maintaining both a libev and libuv backend).


TL;DR: I'm willing to write code to help libuv survive a fork() (I can easily test on OS X and Linux). But before I start, would the maintainers be willing to accept such a patch? I realize that libuv strives to be fully cross-platform, and I also realize that similar issues have been brought up before (e.g., [2], [3]), but I didn't find anywhere that specifically said "no way." There are multiple approaches that could work for gevent, everything from a libev-style 'uv_loop_fork()' call where all the watchers survive (obviously ideal from my point of view, especially if fork watchers got added too :), to simply terminating the loop in the child and letting gevent create a new loop and watchers. If this is something the maintainers are interested in, I look forward to working together to find the best approach. (And if this is best discussed in another venue like a github issue or IRC I'm happy to continue there.)

Thanks,
Jason

[1] Approximately 444,000 downloads last month (https://pypi.python.org/pypi/gevent)
[2] https://github.com/joyent/libuv/pull/1136
[3] https://github.com/joyent/libuv/issues/1405
[4] https://github.com/gevent/gevent/issues/790

[Jason Madden]

[Saúl Ibarra Corretgé]

Hi Jason!

On 04/21/2016 11:40 PM, Jason Madden wrote:
> Hi all,
>
> (There's a TL;DR at the bottom---I'd like to contribute code to help libuv work after a fork() on unix.)
>
> I'm one of the gevent maintainers. gevent is a fairly popular[1] Python library for asynchronous event-driven programming (primarily networking, but other things as well). Currently gevent is based on libev and supports both unix and Windows while implementing an API compatible with the Python standard library.
>
> There's been interest expressed by our community in moving off of libev and onto libuv, primarily for improved Windows support. So recently I've implemented a prototype backend for gevent using libuv [4].
>
> Let me start by saying how much of a joy the libuv API is to work with. It's clean and consistent, and the source code is easy to read. Not all event loops are like that. I really appreciate the work that the maintainers and contributors have put into libuv.
>

Thanks for the kind words! Great work taking over the maintenance of
Gevent, I like gevent myself and attempted to do this in the past [0]
though it was mostly a test, so I'm really happy to see interest in
getting libuv into gevent.

> I've been able to get gevent's test suite to pass with libuv on OS X and Linux, with the exception of those tests that involve forking but *not* execing. Being able to continue to run Python code in a forked child process is a crucial part of the way many Python programs on unix are written, including not just network servers like gunicorn but those that use the standard library 'multiprocessing' feature to improve concurrency.
>
> Currently after a fork, libuv either is in an interesting state where things may or may not function (three different results in gevent's test suite on three different linux distros) or it simply aborts the whole process (kqueue, aix and sunos-based systems). I've been unable to find a reliable workaround to this.
>
> Unless gevent can survive a fork, I won't be able to base it on libuv. As a gevent maintainer, that makes be sad because libuv is so nice to work with (and I'm really not interested in maintaining both a libev and libuv backend).
>
>
> TL;DR: I'm willing to write code to help libuv survive a fork() (I can easily test on OS X and Linux). But before I start, would the maintainers be willing to accept such a patch? I realize that libuv strives to be fully cross-platform, and I also realize that similar issues have been brought up before (e.g., [2], [3]), but I didn't find anywhere that specifically said "no way." There are multiple approaches that could work for gevent, everything from a libev-style 'uv_loop_fork()' call where all the watchers survive (obviously ideal from my point of view, especially if fork watchers got added too :), to simply terminating the loop in the child and letting gevent create a new loop and watchers. If this is something the maintainers are interested in, I look forward to working together to find the best approach. (And if this is best discussed in another venue like a github issue or IRC I'm happy to continue there.)
>

I went through those issues quickly, and from the top of my head, I'd
say this is what you can attempt to do:

- fix the broken global state after forking (IIRC it's just a signal
pipe and the threadpool)
- destroy the loop and recreate it in the child, re-creating all
necessary watchers.

The last part could be tricky because you'd need to dup all fds you want
to reuse (I guess) and close everything nicely in the child.

This *could* work, but I guess the only way to know is to put up the
sleeves and give it a go.

Unless there is strong opposition from other maintainers, I'd say go
sketch up some APIs and open an issue/PR so we start discussing the
code/design.


Cheers,

[0]: https://github.com/saghul/uvent

--
Saúl Ibarra Corretgé
bettercallsaghul.com

[Jason Madden]

Hi all,

On Thursday, 21 April 2016 17:17:12 UTC-5, Saúl Ibarra Corretgé wrote:

Great work taking over the maintenance of
Gevent, I like gevent myself and attempted to do this in the past [0]
though it was mostly a test, so I'm really happy to see interest in
getting libuv into gevent.

Thanks! I did come across that when I was researching libuv and prior art in this area. I noticed that it (of course) ran into the same problems I have.

 

>
>
> TL;DR: I'm willing to write code to help libuv survive a fork() (I can easily test on OS X and Linux). But before I start, would the maintainers be willing to accept such a patch? I realize that libuv strives to be fully cross-platform, and I also realize that similar issues have been brought up before (e.g., [2], [3]), but I didn't find anywhere that specifically said "no way." There are multiple approaches that could work for gevent, everything from a libev-style 'uv_loop_fork()' call where all the watchers survive (obviously ideal from my point of view, especially if fork watchers got added too :), to simply terminating the loop in the child and letting gevent create a new loop and watchers. If this is something the maintainers are interested in, I look forward to working together to find the best approach. (And if this is best discussed in another venue like a github issue or IRC I'm happy to continue there.)
>

I went through those issues quickly, and from the top of my head, I'd
say this is what you can attempt to do:

- fix the broken global state after forking (IIRC it's just a signal
pipe and the threadpool)
- destroy the loop and recreate it in the child, re-creating all
necessary watchers.

The last part could be tricky because you'd need to dup all fds you want
to reuse (I guess) and close everything nicely in the child.

(libev doesn't dup the descriptors at fork, and it's not clear to me why you'd want to...)

 

This *could* work, but I guess the only way to know is to put up the
sleeves and give it a go.

 libev makes it look relatively easy, and I'm sure anything libev can do, libuv can do at least as well :)

Unless there is strong opposition from other maintainers, I'd say go
sketch up some APIs and open an issue/PR so we start discussing the
code/design.

Cool! I've started by putting together some failing test cases (probably initially mostly reflecting the things gevent already does with libev and forking) and next I'll try to play around with some ideas for fixing them so that (a) I'm sure it's at least possible and (b) I have a better idea what I'm talking about when I go to open an issue/PR.

Thanks again,

Jason

[Ben Noordhuis]

On Fri, Apr 22, 2016 at 12:17 AM, Saúl Ibarra Corretgé <sag...@gmail.com> wrote:
> I went through those issues quickly, and from the top of my head, I'd
> say this is what you can attempt to do:
>
> - fix the broken global state after forking (IIRC it's just a signal
> pipe and the threadpool)

Also the fsevents thread on OS X. I don't know how well-behaved
fsevents is in the presence of forking.

> - destroy the loop and recreate it in the child, re-creating all
> necessary watchers.

Jason, how do you plan to deal with file descriptors that are watched
in parent and child both? It's essentially indeterminate which
process is going to receive the events. That's possibly okay (and, I
suspect, in your case intentional) for listen sockets but it breaks
anything that does read/write operations, like regular sockets.

Duplicating file descriptors won't help because epoll, for example,
reports events for file descriptions, not file descriptors. A nasty
consequence is that you can get events for file descriptors that have
been closed in your process but are still alive in other processes;
the stdio file descriptors are prime examples.

Tangential aside: In node.js we found that, at least on Linux and
Solaris, polling a listen socket from multiple processes or threads
results in very unevenly distributed loads; most of the connections
end up in one or two "favored" (by the system scheduler)
processes/threads.

I wound up rewriting the cluster module (node's multiprocessing
equivalent) in round-robin fashion to fix (well, "fix") that. Newer
kernels allegedly fare better but in my limited testing, the
distribution still isn't very fair.

[Jason Madden]

> On Apr 24, 2016, at 03:48, Ben Noordhuis <in...@bnoordhuis.nl> wrote:
> - fix the broken global state after forking (IIRC it's just a signal
>> pipe and the threadpool)
>
> Also the fsevents thread on OS X. I don't know how well-behaved
> fsevents is in the presence of forking.

I hadn't come across that thread yet. I'll make sure to add a test case. Thanks!

>
>> - destroy the loop and recreate it in the child, re-creating all
>> necessary watchers.
>
> Jason, how do you plan to deal with file descriptors that are watched
> in parent and child both? It's essentially indeterminate which
> process is going to receive the events. That's possibly okay (and, I
> suspect, in your case intentional) for listen sockets but it breaks
> anything that does read/write operations, like regular sockets.

I was (tentatively) planning to completely ignore that and leave it up to the application to handle splitting descriptors appropriately.

My reasoning there boils down to "that's how gevent/libev do it now and it works out okay." In the gevent design (presenting synchronous APIs but hiding an event loop under the covers), watchers are started and stopped frequently (every time a blocking API call is entered/exited), so it's highly likely at the time of a fork() that most watchers are stopped and thus not registered with epoll or the like. After the fork, parent and child each continue on with their own work, re-registering watchers that have been partitioned to them or creating new ones. (It's certainly *possible* to have arbitrary other watchers active at a fork() but it's not a recommended pattern and users are cautioned to be very careful.)

Jason

[Jason Madden]

Hi all,

As per this discussion, I've been working on https://github.com/libuv/libuv/pull/846. I'm happy to report that I haven't run into any insurmountable issues. I pushed some more commits today, bringing the tested set of working platforms up to:

- Mac OS X 10.11
- Linux x86
- Linux arm
- FreeBSD 10
- Solaris 11

(I don't have access to AIX so I haven't been able to test there and I didn't want to code blindly, so building on AIX will fail with a linker error. I'll need some help for that platform.)

No doubt the PR needs some fine tuning, but I hope it's in a state that can be usefully discussed. I've left more detailed comments on the PR itself.

Thanks for all the pointers in this thread!

Jason

[Mark Maker]

Hi,

I've read up on the PR and others related  to fork(). I've seen

int uv_loop_fork(uv_loop_t *loop)

is marked experimental. Furthermore, if I understand fork(2) correctly, then a multi-threaded program is never save to fork() without exec.

> After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2).

Is there some guidance as to what functionality is save to use prior to fork with libuv? 

Which parts of libuv use threads and which parts are guaranteed not to?

Use case:

Prior to fork() I need socket/accept and/or pipe operations, and then want to setup SIGCHLD handlers, i.e. the parent  will fork() many times and keep tabs on children, and re-fork() them when they die. In the child I need to get rid of the parent's listener socket and signal handlers, and only keep the socket and/or the right ends of pipes.

_Mark

[Ben Noordhuis]

On Thu, Dec 14, 2023 at 10:51 AM 'Mark Maker' via libuv
<li...@googlegroups.com> wrote:
>
> Hi,
>
> I've read up on the PR and others related to fork(). I've seen
>
> int uv_loop_fork(uv_loop_t *loop)
>
> is marked experimental. Furthermore, if I understand fork(2) correctly, then a multi-threaded program is never save to fork() without exec.
>
> > After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2).
>
> Is there some guidance as to what functionality is save to use prior to fork with libuv?
>
> Which parts of libuv use threads and which parts are guaranteed not to?
>
> Use case:
>
> Prior to fork() I need socket/accept and/or pipe operations, and then want to setup SIGCHLD handlers, i.e. the parent will fork() many times and keep tabs on children, and re-fork() them when they die. In the child I need to get rid of the parent's listener socket and signal handlers, and only keep the socket and/or the right ends of pipes.
>
> _Mark

Libuv makes no guarantees whatsoever. That's also why uv_loop_fork()
will always stay experimental. It's use-at-your-own-risk.