Gather/Take and threads

[Joe Gottman]

Suppose I have a gather block that spawns several threads, each of which
calls take several times. Obviously, the relative order of items returned
from take in different threads is indeterminate, but is it at least
guaranteed that no object returned from take is lost?

Joe Gottman

[Larry Wall]

On Wed, Dec 06, 2006 at 07:44:39PM -0500, Joe Gottman wrote:
: Suppose I have a gather block that spawns several threads, each of which

: calls take several times. Obviously, the relative order of items returned
: from take in different threads is indeterminate, but is it at least
: guaranteed that no object returned from take is lost?

Currently gather/take is defined over a dynamic scope, and I think
that a different thread is a different dynamic scope (after all,
why does it have its own call stack?), so by default you get nothing
from another thread, and the other thread would get a "take outside
of gather" error. You'd have to set up some kind of queue from the
other thread and take the output of that explicitly.

The microthreading done by hyperops perhaps doesn't count though.
In playing around the other day with a list flattener without explicit
recursion, I found that pugs currently returns hyper-takes in random
order:

pugs> say ~gather { [1,2,[3,4],5]>>.take }
1 2 4 5 3
pugs> say ~gather { [1,2,[3,4],5]>>.take }
2 1 4 5 3

The random order is according to spec, assuming take is allowed at all.
But perhaps it shouldn't be allowed, since the threaded side of a
hyperop could conceivably become as elaborate as a "real" thread, and
maybe we should not make a distinction between micro and macro threads.
And it's not like a take can guarantee the order anyway--the hyperop
is merely required to return a structure of the same shape, but that
does not enforce any order on the actual take calls (as shown above).
Only the return values of take end up in the same structure, which is
then thrown away.

In fact, I'd argue that the value returned by take is exactly the
value passed to it, but I don't think that's specced yet. A take is
just a side effect performed "en passant" under this view. Then we
could write things like:

while take foo() {...}
@thisbatch = take bar() {...}

Larry

[Larry Wall]

Of course, it's also possible that the flipside is true--that
gather/take is just another normal way to set up interthread queueing,
if the thread is spawned in the dynamic scope of the gather.
Under that view all the subthreads share the outer dynamic scope.
Maybe that's saner...

Larry

[Larry Wall]

On Wed, Dec 13, 2006 at 11:01:10AM -0800, Larry Wall wrote:
: Of course, it's also possible that the flipside is true--that

And a subdivision of that view is whether subthreads are naturally
collected at the end of the dynamic scope that spawned them, or
whether they are considered independent unless some dynamic scope
"claims" them all for collection. In that case a gather would one
way (the normal way?) to require termination of all subthreads before
terminating its lazy list.

Larry