joinall on multiple lists of greenlets

[Aarush Gupta]

So I have a list of greenlets and then I have another greenlet which I don't want to put in the list and I want to use gevent.joinall() on all of them.

What I want to do is something like :

for x in some_list:
    list_of_greenlets.append(gevent.spawn(some_func))
another_greenlet = gevent.spawn(some_other_func)
gevent.joinall(list_of_greenlet, some_other_func)

But apparently joinall(), does not take multiple iterables as input, it only takes one iterable list. It doesn't take list of lists either.

I realize one way to do this is just create a new copy of the initial list and add the new greenlet to it and pass that to joinall but I don't like that way as it would need a lot of memory for very big lists.

Can anybody suggest some cleaner ways of doing this?

[Grady Player]

something like:

for x in some_list:
    list_of_greenlets.append(gevent.spawn(some_func))
another_greenlet = gevent.spawn(some_other_func)


gevent.joinall(list_of_greenlet + [another_greenlet])

?

--
You received this message because you are subscribed to the Google Groups "gevent: coroutine-based Python network library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gevent+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gevent/c2ba50a8-92ec-4622-ba21-6f55f95c4d4d%40googlegroups.com.

[Jason Madden]

greenlets are lightweight, but if there are enough of them running that a second list of references to them is a problem...well, I'd think you'd probably hit other limitations long before that happens.

> Can anybody suggest some cleaner ways of doing this?

The requirements for the argument to joinall are that it be iterable and support __len__. So in this case that's easily accomplished with a little helper class:

class ToJoin:
def __len__(self):
return len(list_of_greenlets) + 1
def __iter__(self):
for g in list_of_greenlet:
yield g
yield another_greenlet

gevent.joinall(ToJoin())

Basically itertools.chain() plus __len__.

Also, know that gevent.joinall() collects and returns the greenlets that finish so their results can be inspected. So you'll wind up with that extra list of greenlets no matter what you do :)

Why does it do that? Because code like this is pretty common:

results = gevent.joinall([gevent.spawn(func, x) for x in data])

(You can use iwait() to avoid the list, but you have to be more careful with it.)

Jason

[Matt Billenstein]

On Thu, Dec 05, 2019 at 05:21:05AM -0800, Aarush Gupta wrote:
> for x in some_list:
> � � list_of_greenlets.append(gevent.spawn(some_func))
> another_greenlet = gevent.spawn(some_other_func)
> gevent.joinall(list_of_greenlet, some_other_func)

Call joinall twice:

gevent.joinall(list_of_greenlets)
gevent.joinall([another_greenlet])

But practically speaking just another list would be fine, you're not copying
the greenlets, a new list has memory overhead proportional to a list of
pointers more or less, so:

gevent.joinall(list_of_greenlets + [another_greenlet])

is just as well.

m

--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/