Today I ported my startup's code from eventlet to gevent; I'll share my experience porting it over here.

The easiest code to port was that which uses the core functions: spawn, sleep, etc. Those behave identically in eventlet and gevent, so no surprises there.

--

I had to change the idiom I used for the cases where I wanted to fire a callback using call_after_global (seconds later), but have the ability to *cancel* those calls before they take place. Since I didn't see a way to cancel `spawn_later()` calls, I used this instead:

def callback():
    gevent.sleep(seconds)
    dowork()

cb = gevent.spawn(callback)
# cancel this callback via cb.kill()

Thus I don't actually use "spawn_later".

--

It would be nice if `LinkedExited` and friends stored the greenlet to which they're referring to, so that I can check which greenlet raised that exception via something like `e.source`.

--

Lastly, the one major hurdle was that there isn't currently a baked-in way to call a subprocess (or thread) and wait for its result in a coroutine. For instance, I wanted to use `subprocess.Popen` to run an external program and wait for the result. Denis mentioned that gevent doesn't have this functionality, so I put together the code that you see attached to this e-mail.

It'd be nice to have rudimentary support for this kind of thing, though I don't know how much work it would be to build something like that in. (Also, if you see any flaws in my attached code, by all means let me know.)

Everything seems to work now; I just need to do a little more testing before I put it into production. The gevent code feels clean, easy to follow, and well-maintained; no complaints here. All in all, I like gevent's simplicity a lot.

Marcus

processes.py 1K Download

Hi Marcus,

Thanks for sharing your experience! I have some comments below.

On Tue, Jan 26, 2010 at 5:36 AM, Marcus Cavanaugh

You can use spawn_later and kill together!

def callback():
    dowork()

g = gevent.spawn_later(seconds, callback)
# kill this greenlet with g.kill()

It's less code and it's more efficient (you avoid switching into the
greenlet just to switch out on sleep()).

kill() tries to do the right thing here.

There is one minor bug with this I just noted - it's that g.dead will
remain False in this case.
I have a fix for this, so I will test it and push into the repository.

That sounds reasonable and in fact I considered doing this when I
wrote it but decided against it.
The reason is that LinkedExited and subclasses should not be caught at
all. They are poor means
of communications that are prone to errors. The use case I had mind
for them is this:

1. you have a background job that you want to run concurrently with
the main greenlet
2. the background job is so important that if it dies, the main
greenlet should die as well

so you do:

background_job = gevent.spawn_link_exception(background_main)
main()

If there is an exception in the background greenlet, main greenlet
will be killed with LinkedFailed (subclass of LinkedExited).

Now if I try to extend this pattern, run a bunch of background jobs
and detect their exits
via LinkedExited exceptions the code becomes somewhat fragile:

# BAD EXAMPLE follows

job1 = gevent.spawn_link(main1)
job2 = gevent.spawn_link(main2)

try:
  # sleep forever or do some stuff
except LinkedExited, ex:
  handle_child_death(ex)

When job1 dies, handle_child_death is called and while it runs the
second job could die and thus kill the whole thing.

What should be done, if you need to react to children's is to create a
Queue and link all of the jobs to it:

q = gevent.queue.Queue()
job1 = gevent.spawn(main1)
job1.link(q.put)
job2 = gevent.spawn(main2)
job2.link(q.put)

for x in range(2):
    finished_job = q.get()
    # examine the job's results:
    print finished, finished.value, finished.exception

Maybe your case is different. let me know if you think the above
explanation is missing the point.

This is a cool feature that gevent desperately needs. Ideally, we
would have gevent.os with waitpid/popen* methods
or gevent.subprocess that implements the API of stdlib's subprocess.
Not sure what it takes to write such thing.
I'll check later if the code you attached can be put into examples/

Thanks a lot for taking time to write this,
Denis.

On Tue, Jan 26, 2010 at 5:36 AM, Marcus Cavanaugh

It works great! Why did you put sleep(poll_interval) before calling
wait_read/wait_write? It works without it and I can't figure out the
situation where it won't.

On Jan 26, 2010, at 2:16 AM, Denis Bilenko wrote:

I looked at Eventlet's processes code for inspiration, and it used a poll interval. I was just afraid of burning CPU, I guess.

On Jan 25, 2010, at 11:36 PM, Denis Bilenko wrote:

Cool, I'll do that instead.

That does feel cleaner. I'd suggest putting that explanation somewhere in the docs so that people don't misuse Linked* exceptions like I tried to. I was able to convert to that, but I have a question: You said "LinkedExited and subclasses should not be caught at
all"; I still catch them in a couple places because I don't want to print a traceback. In other words, I'm expecting LinkedExited to occur at some point, so I catch it just to prevent gevent from printing the traceback. Is there a more idiomatic way to do it?

Question about spawning Greenlet subclasses:

If you subclass Greenlet, you can't do this:

MySubclassedGreenlet.spawn_link()

because it expects a function. Nor can you do this:

gevent.spawn(MySubclassedGreenlet)

Thus I've always done:

g = MySubclassedGreenlet.spawn()
g.link()

(I suppose this would also work:)

g = MySubclassedGreenlet()
g.start()
g.link()

Is that also the proper way to do it?

Marcus

On Tue, Jan 26, 2010 at 10:26 PM, Marcus Cavanaugh

For suppressing a traceback, there's util.wrap_errors
http://www.gevent.org/gevent.util.html#gevent.util.wrap_errors

If you write

  g = spawn(wrap_errors(LinkedExited, your_function), arg1, arg2)

then you won't see LinkedExited tracebacks in your_function.
(Also the exception instance will be stored as g.value, not as g.exception).

But your code may be still prone to unpredictable behavior, for
example, if you have "finally" section
that is switching and can be a source of LinkedExited itself.

It seems that if you need to catch LinkedExited then you are using
them more than as a mere asserts that
the other job is running. I'm not sure if it's reasonable in your
case. Maybe you can post
a simplified snippet here and we'll see if it can be improved.

Oops. Actually, spawn_link is a simple shortcut that should not care
about type of its arguments as it just passes them to spawn().
That it requires to pass a function is a bug. Fixed in the repo. Good
catch, thanks!

This is what I always do too.

Any way is fine. But I would review the consequences of linking to greenlet.

Cheers,
Denis.

On Jan 27, 2010, at 12:05 AM, Denis Bilenko wrote:

Could you elaborate on that?

I'd like to add your example processes.py to gevent/examples.

But I'm going to remove the sleep() calls.
I'm pretty sure they are unnecessary. wait_read() already sleeps and it
sleeps exactly for the delay needed to have something to read from the
descriptor.

Could you do the same in your system - run your code with sleeps commented out?
If it was a bad idea, we'll find out sooner or later.

On Wed, Jan 27, 2010 at 12:11 PM, Marcus Cavanaugh

Well, I'll try

try:
  some_operation() # you expect LinkedExited to be raised here
except Exception:
  error_handling() # but it can also be raised here
finally:
  cleanup() # and here

You can of course protect against it, e.g. by doing spawn(cleanup).join(), but
maybe it's simpler not having to worry about LinkedExited in the first place.

On Jan 27, 2010, at 12:19 AM, Denis Bilenko wrote:

I've run it all day today, seems to work fine. I think you're right, that sleep is unnecessary.

Makes sense. "wrap_errors" would eliminate my need to catch LinkedExited.

Marcus

I was wrong about this, g.dead is True in this case, like it's supposed too.
The minor annoyance was that a traceback is printed to stderr,
pointless in this case.
I've fixed it not to do that.