Trying to understand how [yield] works for coroutines

[sil...@gmail.com]

Hello all.

In order to try to understand coroutines, I wrote this small example:

proc is_prime {n} {
set max [expr {$n / 2}]
if {$max % 2 == 0} {
incr max
}
for {set i 3} {$i < $max} {incr i 2} {
if {$n % $i == 0} {
return no
}
}
return yes
}

proc prime {} {
set n [yield]
while {1} {
if {[is_prime $n]} {
puts $n
}
set n [yield $n]
}
}

coroutine p1 prime
coroutine p2 prime

set n 3
set m 10000001
while {1} {
p1 $n
incr n 2
p2 $m
incr m 2
}

I was expecting it to output:

3
10000019
5
10000079
7
10000103

But, at least in my computer, it prints:

3
5
7
11
13
17
19
10000019
23
29
31
37
41
43
47
53
59
61
67
71
73
79
10000079
83
89
...

So I started to think that [yield] does not just "return" value to the caller, but can also pass control to another coroutine that is "waiting". Anyway, how would I explain the output above?

[sil...@gmail.com]

On Wednesday, May 23, 2018 at 8:09:32 PM UTC-3, sil...@gmail.com wrote:
> So I started to think that [yield] does not just "return" value to the caller, but can also pass control to another coroutine that is "waiting". Anyway, how would I explain the output above?

Got the answer by just adding [puts [info coroutine]] within the loop at [prime]. It also yields for non-primes, which are many more if n > 100000000, but p1 and p2 are called equally.

[Rich]

sil...@gmail.com wrote:
> Hello all.
>
> In order to try to understand coroutines, I wrote this small example:

You've 'almost' got a question in the title, and you did not actually
ask it in the body, but here goes an answer anyway:

Think of [yield] for coroutines in a way similar to putting a laptop or
smartphone to "sleep".

You pause (put to sleep) the current coroutine, and you can later
unpause (wake it back up) and it comes back exactly like it was when
you paused (put to sleep) the coroutine.

Similar to laptop (or smartphone) sleep where you wake it back up and
everything is as you left it when you put it to sleep (ignore the 'lock
screen' that asks "who are you" for this explanation here).

The difference with yield vs. laptop sleeping is that yield can also
send back a result to whomever last "woke up" the sleeping coroutine
(here consider the first call that starts the coroutine as a first
"wakeup", it is really not, but by the time you get to the yield it is
a close enough approximation to help understand the workings).

Yield is the command that says "put this coroutine to sleep" and "send
a result back to whomever it was that woke this coroutine up".

[sil...@gmail.com]

On Wednesday, May 23, 2018 at 9:58:22 PM UTC-3, Rich wrote:
> You've 'almost' got a question in the title, and you did not actually
> ask it in the body, but here goes an answer anyway:

Thanks! I didn't phrase the question well, but you helped me to understand. My real doubt is how coroutines can be used to build asynchronous communication (that look much natural in a language like Erlang). I thought I discovered it myself with the following example. Is a good example of async communication using coroutines?

# Just a coroutine to generate sequential odd numbers
proc next_number {} {
yield
set n 3
while {1} {
yield $n
incr n 2
}
}
coroutine next next_number
next

# Check if $n is prime

proc is_prime {n} {
set max [expr {$n / 2}]
if {$max % 2 == 0} {
incr max
}
for {set i 3} {$i < $max} {incr i 2} {
if {$n % $i == 0} {
return no
}
}
return yes
}

# Our coroutine. It "delivers" to [recv]: the number; whether or not it is
# prime; and the coroutine
proc prime {} {
set n [next]
while {1} {
set is_prime [is_prime $n]
after idle [list recv $n $is_prime [info coro]]
set n [yield]
}
}

# Receive coroutine result and call it again with the next number
proc recv {n is_prime caller} {
if {$is_prime} {
puts "$caller: $n"
}
set n [next]
$caller $n
}

# Instantiate $maxcoro coroutines
set maxcoro 10
for {set i 0} {$i < $maxcoro} {incr i} {
coroutine p($i) prime
p($i) [next]
}

vwait forever



After that, I wrapped it inside a small Tk application to see if I could fire a command (bound to a button) "at the same time" of the primes calculation:


package require Tk
pack [button .btn -text Test -command {puts Test}]
after 100 {
source coro.tcl
}


It is not very useful, but it works :-) Is this the general idea?

Thanks for kbk, aspect and other guys on IRC for this tip :-)

[briang]

On Friday, May 25, 2018 at 10:15:38 AM UTC-7, sil...@gmail.com wrote:
> On Wednesday, May 23, 2018 at 9:58:22 PM UTC-3, Rich wrote:
> > You've 'almost' got a question in the title, and you did not actually
> > ask it in the body, but here goes an answer anyway:
>
> Thanks! I didn't phrase the question well, but you helped me to understand. My real doubt is how coroutines can be used to build asynchronous communication (that look much natural in a language like Erlang). I thought I discovered it myself with the following example. Is a good example of async communication using coroutines?
>
> # Just a coroutine to generate sequential odd numbers

--deleted--

>
> It is not very useful, but it works :-) Is this the general idea?
>
> Thanks for kbk, aspect and other guys on IRC for this tip :-)

This is my favorite real-life example. This coroutine runs "grep" in background, then spits out the results to stdout. The main program can fire off large numbers of "grep"s "concurrently", and get back a stream of results. Writing the same thing without coroutines takes at least 2 procs plus one or more global variables.

proc GREP {options pattern file} {
set cmd [list grep {*}$options $pattern $file]
set fd [open "| $cmd"]
fconfigure $fd -buffering line -blocking 0
set d ""
fileevent $fd readable [info coroutine]
incr ::fcount
while {1} {
yield
set stat [gets $fd line]
if {$stat >= 0} {
append d $line \n
} elseif {[eof $fd]} {
if {[string length $d]} {
puts -nonewline $d
}
catch {close $fd}
break
}
}
}

-Brian

[Rich]

Well, if is possible to write without coroutines and without globals.
And 'without a second proc' if one does not consider an anonymous proc
a 'proc' per se.:

proc GREP {options pattern file} {
set cmd [list grep {*}$options $pattern $file]
set fd [open "|$cmd"]
fconfigure $fd -buffering line -blocking 0

fileevent $fd readable [list ::apply {{fd d} {
gets $fd line

if {[eof $fd]} {
if {[string length $d]} {
puts -nonewline $d
}
catch {close $fd}

} elseif {![fblocked $fd]} {
append d $line \n
fileevent $fd readable [list {*}[lrange [fileevent $fd readable] 0 end-1] $d]
}
}} $fd ""]
}

[heinrichmartin]

On Saturday, May 26, 2018 at 12:07:46 AM UTC+2, briang wrote:
> This is my favorite real-life example. This coroutine runs "grep" in background, then spits out the results to stdout. The main program can fire off large numbers of "grep"s "concurrently", and get back a stream of results. Writing the same thing without coroutines takes at least 2 procs plus one or more global variables.

Have you considered Expect there? It can spawn many processes and listen for output on all of them simultaneously. (Undoubtedly, this can be done with fileevent, too.)

[sil...@gmail.com]

On Saturday, May 26, 2018 at 2:08:49 AM UTC-3, Rich wrote:
> Well, if is possible to write without coroutines and without globals.
> And 'without a second proc' if one does not consider an anonymous proc
> a 'proc' per se.:

That is interesting and is related to something I was thinking about. I actually cannot see where coroutines help a lot, specially in Tcl, which has other useful features. For example, I can see it have two main uses:

1. Implement the "suspend" feature Rich explained;

2. Implement asynchronous function calls.

1. can be solved by a proc and a global variable. This seems ugly but Tcl has enough features to allow us to implement the same concept in more elegant ways, like namespaces and TclOO (with a object variable, for example).

2. can be done with [after idle {code}]

So that is not very clear for me where coroutines really help.

[sil...@gmail.com]

On Sunday, May 27, 2018 at 11:53:51 PM UTC-3, sil...@gmail.com wrote:
> So that is not very clear for me where coroutines really help.

There is a list here: https://en.wikipedia.org/wiki/Coroutine#Common_uses

I actually see Generators (and Iterators) are good examples. The others can just be implemented with other techniques.

[Rich]

sil...@gmail.com wrote:
> On Saturday, May 26, 2018 at 2:08:49 AM UTC-3, Rich wrote:
>> Well, if is possible to write without coroutines and without globals.
>> And 'without a second proc' if one does not consider an anonymous proc
>> a 'proc' per se.:
>
> That is interesting and is related to something I was thinking about.
> I actually cannot see where coroutines help a lot, specially in Tcl,
> which has other useful features.

Well, keep in mind that while it is /possible/ to implement without
coroutines and without globals, that the actual implementation is very
likely to not be the most efficient one. Redefining the target lambda
of the fileevent for the channel is likely to be a performance killer
overall. So don't confuse /possible/ with /efficient/, the two are not
always the same.

The coroutine version is likely to be much more efficient at the
specific task than my hack.

> For example, I can see it have two main uses:
>
> 1. Implement the "suspend" feature Rich explained;
>
> 2. Implement asynchronous function calls.
>
> 1. can be solved by a proc and a global variable. This seems ugly
> but Tcl has enough features to allow us to implement the same concept
> in more elegant ways, like namespaces and TclOO (with a object
> variable, for example).
>
> 2. can be done with [after idle {code}]
>
> So that is not very clear for me where coroutines really help.

A big reason espoused on many of the wiki pages is that coroutines
allow for writing event driven code in a sequential style (so it looks
and reads reasonably sequentially) while hiding the async. nature of
the event drive aspect.

Typical event driven code is generally a disconnected set of procs that
get called in arbitrary orderings, and so mapping out (and
understanding) the control flow from the source can become quite a
challenge.

Being able to 'unwind' that into a sequential looking layout, while
still having the event driven nature, can be a huge help to making
blocks of code more understandable.

[Donal K. Fellows]

On 28/05/2018 04:32, Rich wrote:
> Being able to 'unwind' that into a sequential looking layout, while
> still having the event driven nature, can be a huge help to making
> blocks of code more understandable.

Especially as you can keep variables over the suspension period,
allowing you to avoid having to figure out how to serialise all that out
as arguments to callbacks or store it all in global variables.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.