Exposing newcoro/coroswitch

[Nuno Cruces]

Hi!

Is there any appetite to expose runtime.newcoro/coroswitch? They seem really useful beyond iter.Pull. They actually have the go:linkname handshake, but are then explicitly banned in the linker.

I wanted to implement this "push like" interface with a "processor" function that accepts an iter.Seq and returns a result:

    func processor(seq iter.Seq[value]) result { ... }

I've been cracking my head, and I'm convinced I can't with just iter.Seq/iter.Pull: I need a co/goroutine (and coroswitch, or channels).

My implementation is here: https://github.com/ncruces/go-sqlite3/pull/229/files

Maybe there's another higher level abstraction like iter.Pull we could build instead, but it's not apparent to me?

Regards,

Nuno Cruces

[Jason E. Aten]

I don't understand the tradeoffs involved, so I can't speak to the 

hesitancy to release coroutines.  They would probably be great

fun to play with. I imagine, however, they would provide ample foot-guns too.

Perhaps that is a part of the reason. I find reading coroutine code in Lua,

for example, a fairly mind-boggling torture exercise.

On Sunday, February 23, 2025 at 12:52:58 AM UTC Nuno Cruces wrote:

I wanted to implement this "push like" interface with a "processor" function that accepts an iter.Seq and returns a result:

    func processor(seq iter.Seq[value]) result { ... }

First using channels:

package main
func consumeAndSum(ch chan int) (sum int) {
    for i := range ch {
        println("consumer sees ", i)
        sum += i
    }
    println("consumer done.")
    return sum
}
func startProducer() chan int {
    ch := make(chan int)
    go func() {
        for i := range 3 {
            ch <- i
        }
        close(ch)
        println("producer exits.")
    }()
    return ch
}
func main() {

    ch := startProducer()

    sum := consumeAndSum(ch)

    println("all done. sum = ", sum)

}

/*   output:

consumer sees  0

consumer sees  1

producer exits.

consumer sees  2

consumer done.

all done. sum =  3 

*/

Second using iter.Seq:

package main

import "iter"

func consumeAndSum(it iter.Seq[int]) (sum int) {
for i := range it {
println("consumer sees ", i)
sum += i
}
println("consumer done.")
return sum
}
func startProducer() iter.Seq[int] {
return func(yield func(int) bool) {
for k := range 3 {
if !yield(k) {
return
}
}
println("producer exits.")
}
}
func main() {
it := startProducer()
sum := consumeAndSum(it)
println("all done. sum = ", sum)
}

/* output:
go run iter.go
consumer sees  0
consumer sees  1
consumer sees  2
producer exits.
consumer done.
all done. sum =  3
*/

[Robert Engels]

You don’t need co routines if you have real concurrency. The generator use case is simply an optimization that wouldn’t be necessary if the concurrency model was more efficient - and there was a more expressive way to use it. I demonstrated on some Java boards that generators are easily implemented without exposing the co routines that make lightweight threading possible. The Go implementation get some compiler support which makes it more efficient. 

I think the Java loom project has proved that the Go concurrency model is ideal. Reading up on “structured concurrency” is a great exercise. 

On Feb 24, 2025, at 8:53 PM, Jason E. Aten <j.e....@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/6f090b16-ac52-4ea0-b5ea-1f7b3b7e6e7cn%40googlegroups.com.

[Nuno Cruces]

I'm… sorry. I don't understand the purpose of this code listing.

In my original post I gave a solution to my own problem implemented in two ways: with goroutines and channels, and with newcoro and coroswitch. If you're trying to show me that I don't need either of those, that iter.Seq is enough, I would kindly ask that you reconsider, and try to implement my interface that way. I don't think it's possible, but I'll be really happy if you can prove me wrong.

If you want I can explain a little further the purpose of this, just so this don't feel like this is an X/Y problem, but the motivation for this is very similar to iter.Pull (which similarly can't be written in Go without goroutines): iter.Seq is not the last word in iteration interfaces; other interfaces may desirably need to be adapted to it.

Regards,

Nuno

[Nuno Cruces]

Ack.

I'll take some time to try to come up with a more general abstraction that might be worthy of being added to package iter.

That's hard just because I only have one use case, but I guess if no one shares their use case (or makes concrete proposals) it's even harder.

Thanks!

Nuno

On Sun 23 Feb 2025, 19:45 Ian Lance Taylor, <ia...@golang.org> wrote:

I don't think we are likely to simply export newcoro and coroswitch,
as they don't provide any functionality that we can't already get from
goroutines, but are harder to use correctly. If someone can define a
clean API that uses them based on channels, we could consider a
proposal for that.

Ian

[Nuno Cruces]

I guess this is my best proposal.

If anyone has comments before I make a formal proposal, please let me know.

I'd love the feedback. I also implemented it on top of coros.

https://github.com/ncruces/go-sqlite3/blob/aad2224/ext/seq/push.go

Regards,

Nuno

[Jason E. Aten]

On Tuesday, February 25, 2025 at 4:29:44 AM UTC Robert Engels wrote:

You don’t need co routines if you have real concurrency. The generator use case is simply an optimization that wouldn’t be necessary if the concurrency model was more efficient - and there was a more expressive way to use it.

Hi Robert,

I always enjoy your thoughts and perspective on these things.

I certainly agree that, strictly speaking, you don't need coroutines when you have goroutines.

But, having implemented a lexer and parser with goroutines instead of coroutines

(e.g. https://github.com/glycerine/zygomys/blob/master/zygo/parser.go and lexer.go)

I strongly suspect that coroutines would make for... a much "nicer" architecture. 

The resulting lexer and parser back-and-forth I suspect, would be easier to read, 

understand, modify, and use. In fact, I hope to re-write that code using coroutines

in the near future because of this. 

Having a ton of different parsing goroutines running in the background, and 

shutting them down when not needed, and starting them up again, when needed

again, simply makes for alot of "goroutine managment" hassle that it appears

could be trivially avoided with coroutines.

Or, perhaps there is a simpler way to use goroutines to do parsing that I

did not discover--and that you have in mind. I'd be curious to see what that

would look like.  Feel free to suggest how to refactor that lexer and parser code

while still using only goroutines, if it helps to make the ideas less abstract 

and more concrete.

Best regards,

Jason

 

I think the Java loom project has proved that the Go concurrency model is ideal. Reading up on “structured concurrency” is a great exercise. 

p.s. if you have resources/links in mind about "structured concurrency", I'd be interested to look into them. 

[Jason E. Aten]

On Tuesday, February 25, 2025 at 7:57:50 AM UTC Nuno Cruces wrote:

I'm… sorry. I don't understand the purpose of this code listing.

In my original post I gave a solution to my own problem implemented in two ways: with goroutines and channels, and with newcoro and coroswitch. If you're trying to show me that I don't need either of those, that iter.Seq is enough, I would kindly ask that you reconsider, and try to implement my interface that way. I don't think it's possible, but I'll be really happy if you can prove me wrong.

Hi Nuno,

It feels like the problem you specified is not the actual problem you have in mind.

An X/Y issue, as you suggest.

It would help us to help you, if that is your wish, if you were to define the 

problem with more rigor--not in code--just in words. Stay high level for a moment,

and try to clearly specify the actual problem you are thinking of. I tried

to read your code but could not discern your intent from it.  It can also

help to start from an example problem (not code) or two, and then to

generalize those to a clear, precise, problem statement; this provides motivation and intent,

which can also be helpful.

The problem specification you originally gave was this:

> "to implement... a "processor" function that accepts an iter.Seq and returns a result:

>    func processor(seq iter.Seq[value]) result { ... }"

This is exactly the consumeAndSum() function example that I provided;

consumeAndSum() is a "processor" by your definition. It accepts an

iter.Seq and returns a result (the sum).

Moreover, you originally said,

"I've been cracking my head, and I'm convinced I can't with just iter.Seq/iter.Pull"

Which says, in contradiction to your later comment/claim, that you did not have a solution

when you asked for one.

The code I provided demonstrates that a solution to the stated problem is indeed possible

with iter.Seq.

Best wishes,

Jason

 

[Robert Engels]

Hi Jason. I don’t think I was clear. I think a parser implementation is a natural fit for a generator - a generator being a specialization of a coroutine. The message I was responding to was the request to open the co routine facility that powers it. That’s what I was going against. I agree that managing the go routines for a generator case would be a pain - which is why I’ve come to agree with its inclusion. For the general concurrency case I think Go routines provide everything needed and co routines are unnecessary. 

As for structured concurrency, I think the wiki gives a decent overview https://en.m.wikipedia.org/wiki/Structured_concurrency

And for more in-depth, this is a great read (in Java but it’s fairly generic) exceptions make it a little more complex than needed in go. https://openjdk.org/jeps/453

On Feb 25, 2025, at 5:57 AM, Jason E. Aten <j.e....@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/8f12ea21-0d26-4864-8a08-a6c558a10ed7n%40googlegroups.com.

[Robert Engels]

Also, there’s this project in Go that looks interesting https://github.com/sourcegraph/conc

On Feb 25, 2025, at 7:17 AM, Robert Engels <ren...@ix.netcom.com> wrote:



To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/F90571DB-277A-443A-BDE4-0D7A614CF2F1%40ix.netcom.com.

[Robert Engels]

That project has a link to a very interesting get in the weeds related blog post https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/

On Feb 25, 2025, at 8:02 AM, Robert Engels <ren...@ix.netcom.com> wrote:



To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/E7368238-3921-4DB7-90C6-085B6E91A67E%40ix.netcom.com.

[Nuno Cruces]

Hi Jason,

First of all, thank you for your interest.

On Tue, 25 Feb 2025 at 12:28, Jason E. Aten <j.e....@gmail.com> wrote:

The problem specification you originally gave was this:

> "to implement... a "processor" function that accepts an iter.Seq and returns a result:

>    func processor(seq iter.Seq[value]) result { ... }" 

 

If you're going to omit important parts of the problem specification (the bit after implement), we'll get nowhere.

I don't want "to implement... a processor", I want to implement this interface with a processor.

To make it easier, here's the interface (which was linked to in the original message):

type AggregateFunction interface {

  // Step is invoked to add a row to the current window.

  // The function arguments, if any, corresponding to the row being added, are passed to Step.

  // Implementations must not retain arg.

  Step(ctx Context, arg ...Value)

  // Value is invoked to return the current (or final) value of the aggregate.

  Value(ctx Context)

}

The first thing you should note is that the consumer of this interface calls me, not the other way around.

It calls me to give me the sequence of values (in Step), and to stop iteration and get a result (in Value).
 

Now assume I have a preexisting function that calculates the sum of an iter.Seq[float64], like so:

func sum(seq iter.Seq[float64]) float64 {

  count := 0

  total := 0.0

  for arg := range seq {

    total += arg

    count++

  }

  return total / float64(count)

}

I want to use this function to "easily" implement an AggregateFunction.

As for why the AggregateFunction is what it is, it's the interface to register an SQLite aggregate function.

And, obviously, sum is not the function I really want to call. But maybe there's a library that provides a nice and complex function that I want to reuse, and which uses what, going forward, is supposed to be the "standard" Go iteration interface.

Moreover, you originally said,

"I've been cracking my head, and I'm convinced I can't with just iter.Seq/iter.Pull"

I wanted a solution that doesn't necessarily involve goroutines and channels, for the same reason that iter.Pull was created: because goroutines and channels add unnecessary parallelism that has the potential to introduce data races, when what is needed is concurrency without parallelism.

 

Which says, in contradiction to your later comment/claim, that you did not have a solution

when you asked for one.

I have a solution, to which I linked in my original email.

The solution I found can be expressed without goroutines and channels, but it can also be better expressed with newcoro / coroswitch.

I can't express it without either.

 

The code I provided demonstrates that a solution to the stated problem is indeed possible

with iter.Seq.

If you can show me how to use your code to implement AggregateFunction, you will have solved my problem as stated.

Otherwise, I'm afraid it falls short.

Kind regards,

Nuno

[Aston Motes]

Nuno, can you say more about the constraints here? It seems you could you accomplish your goal of creating an AggregateFunction out of an iter.Seq processor function by storing all of the results in a slice as Step is called, then running your processor function on the slice of values (slices.Values(storedVals)) at the end when Value is called?

If not, is the idea that you want to avoid intermediate storage and have the processor function aggregating values as you go along? If so, perhaps you could substitute a chan for the slice and send the processor off to do its thing in a goroutine. In that case, you would only need a utility function to adapt a channel to a iter.Seq.

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CAM0U__8k6%2BEiKZO%3DUuxXkVjVh8w6y%2BG9Zc0j2eH3GU02XiJ_iQ%40mail.gmail.com.

[Jason E. Aten]

I agree with Aston's insight.  Step could be just buffering.

Re "the "standard" Go iteration interface." <- this is not.a great API to offer clients, for the reasons you observe. Hence I don't think it should be considered as an API standard, or much of a standard at all, as such. Its better as an implementation detail.

And your larger problem stems from the same source of pain: trying to integrate iter.Seq across API boundaries. Its not really good for that--it will always create an extremely tight coupling between producer and consumer. In essence they are sharing a thread and jumping between each other's stacks.

In fact, if you leave the iter clutter aside, then your problem becomes trivial -- just reuse the guts of the sum/processor function but leave off the problematic seq iterface. 

That said, if you really have to use a sub-routine that insists on an iter.Seq interface, then

I conclude that goroutines are unavoidable, because the iter protocol is a blocking protocol.

Here's the simplest thing otherwise; I did it from spec -- it might be close to your solution-- but it omits the confusing wait channel. If it gives you ideas, great. Perhaps it contributes to your study in that I could not see any way to 

do it without a goroutine. That means channels too.

package main
import "iter"

// the pre-existing "processor"

func consumeAndSum(it iter.Seq[int]) (sum int) {
for i := range it {
println("consumer sees ", i)
sum += i
}
println("consumer done.")
return sum
}

// the shim between clients and processor, that implements Summer.
type UseProcessor struct {
initDone bool
it       iter.Seq[int]
feedme   chan int
result   chan int
done     chan struct{}
}
func (m *UseProcessor) AddObs(obs int) {
if !m.initDone {
m.initDone = true

m.done = make(chan struct{})
m.result = make(chan int)

// To avoid a shutdown race,
// we keep feedme un-buffered; we
// special case yeilding the firstObs
// to acheive this.
m.feedme = make(chan int)
firstObs := obs

// define an iterator to feed the processor.
// capture the channels (feedme,done) in the closure.
m.it = iter.Seq[int](func(yield func(int) bool) {
if !yield(firstObs) {
return
}
for {
select {
case newObs := <-m.feedme:
if !yield(newObs) {
// processor wants to halt early
return
}
case <-m.done:
// client wants to halt.
// by exiting, we tell the
// processor to compute its final
// result sum.
return
}
}
})
// start the processor with the iterator

               // the iterator protocol forces this to be in a goroutine since cosumeAndSum will block.
go func() {
m.result <- consumeAndSum(m.it)
}()
return
}
m.feedme <- obs
}
func (m *UseProcessor) Sum() int {
close(m.done)
return <-m.result
}
type Summer interface {
AddObs(obs int)
Sum() int
}
func main() {
var summer Summer = &UseProcessor{}
for i := range 4 {
summer.AddObs(i)
}
println("all done. sum = ", summer.Sum())
}

[Jason E. Aten]

Thanks Robert!

[Nuno Cruces]

Buffering trivially solves the issue, sure, but the point (of both the interface and Seq) is not to buffer.

The database could spit (many) millions of rows that we want to aggregate into a result.

It's why we have algorithms like HyperLogLog, and T-Digest or KLL.

About the standard interface, which, yes, I can ignore, I'll just quote the proposal:

There is no standard way to iterate over a sequence of values in Go. For lack of any convention, we have ended up with a wide variety of approaches. Each implementation has done what made the most sense in that context, but decisions made in isolation have resulted in confusion for users.

Or the former discussion titled "standard iterator interface":

Most languages provide a standardized way to iterate over values stored in containers using an iterator interface (see the appendix below for a discussion of other languages). Go provides for range for use with maps, slices, strings, arrays, and channels, but it does not provide any general mechanism for user-written containers, and it does not provide an iterator interface.

Maybe it's too early to have this discussion on adapting/repurposing this "newfangled iterator interface" to other tasks, but OTOH, this is what iter.Pull is all about.

BTW, iter.Pull is successful at allowing fs.WalkDir to feed a pull style iterator, which before required goroutines and channels to achieve (and was buggy and racy, though that's my own fault).

PS: the purpose of my "confusing wait channel" is to ensure (as best as I could) that there is no parallelism.

Which, BTW, is one assurance iter.Pull offers (as well as: no races, panics are forwarded, Goexit handled, etc...)

I need to review your solution with more care, thanks!

Kind regards,

Nuno

[Ian Lance Taylor]

On Tue, Feb 25, 2025 at 6:17 AM Nuno Cruces <ncr...@gmail.com> wrote:
>
> I wanted a solution that doesn't necessarily involve goroutines and channels, for the same reason that iter.Pull was created: because goroutines and channels add unnecessary parallelism that has the potential to introduce data races, when what is needed is concurrency without parallelism.

I think what you're presenting is an argument for

package iter

// Push returns an iterator, a yield function, and a stop function.
// The iterator will return all the values passed to the yield function.
// The iterator will stop when the stop function is called.
// This provides a way to flexibly convert a sequence of values into a Seq.
func Push[E any]() (seq iter.Seq[E], yield func(E), stop func())

Ian

[Jason E. Aten]

Hi Ian, I'm not quite understanding -- is Push meant to take an input seq too? like

func Push[E any](inputSeq iter.Seq[E]) (seq iter.Seq[E], yield func(E), stop func())
                 ^^^^^^^^

?

[Brian Candler]

You can curry a function?

[Robert Engels]

I agree. Simplifying the number of ways to perform iteration is a huge win for readability.

I’ve never seen an advanced use of co routines I would consider readable. A higher level construct is immensely more readable.

The blog post I shared goes into the reasons why. A language that doesn’t want exceptions due to flow control concerns should not have coroutines.

> On Feb 25, 2025, at 12:15 PM, Ian Lance Taylor <ia...@golang.org> wrote:

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXamH48NS%2BHeX%2BAjtC4ncoEm49xN66L3b46Wse4qr%2BYsw%40mail.gmail.com.

[Ian Lance Taylor]

On Tue, Feb 25, 2025 at 11:52 AM 'Brian Candler' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> You can curry a function?
>
> On Tuesday, 25 February 2025 at 18:52:09 UTC Jason E. Aten wrote:
>>
>> Hi Ian, I'm not quite understanding -- is Push meant to take an input seq too? like
>>
>> func Push[E any](inputSeq iter.Seq[E]) (seq iter.Seq[E], yield func(E), stop func())
>> ^^^^^^^^
>> ?

I'm suggesting that Push returns, conceptually, two things: an
iter.Seq, and a pair of functions. You hand the iter.Seq off to
something that expects an iter.Seq. You use the two functions to push
values into the iter.Seq. Just as iter.Pull gives you flexibility to
fetch values from the sequence however you like, iter.Push gives you
flexibility to push values into the sequence however you like, without
being tied to the lifespan of a single function. In the original
example, the values to push into the sequence would come from a method
call.

This can all be done already with channels, of course, as shown by the
earlier examples.

Ian

>> On Tuesday, February 25, 2025 at 6:15:40 PM UTC Ian Lance Taylor wrote:
>>>
>>> On Tue, Feb 25, 2025 at 6:17 AM Nuno Cruces <ncr...@gmail.com> wrote:
>>> >
>>> > I wanted a solution that doesn't necessarily involve goroutines and channels, for the same reason that iter.Pull was created: because goroutines and channels add unnecessary parallelism that has the potential to introduce data races, when what is needed is concurrency without parallelism.
>>>
>>> I think what you're presenting is an argument for
>>>
>>> package iter
>>>
>>> // Push returns an iterator, a yield function, and a stop function.
>>> // The iterator will return all the values passed to the yield function.
>>> // The iterator will stop when the stop function is called.
>>> // This provides a way to flexibly convert a sequence of values into a Seq.
>>> func Push[E any]() (seq iter.Seq[E], yield func(E), stop func())
>>>
>>> Ian
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/b46f0864-9dfd-4c1c-b34a-dff83decae7fn%40googlegroups.com.

[Nuno Cruces]

Ian, just to note that your message is not at all lost. I just need to look at it carefully in front of a computer, which I won't be able to do today.

But if it works, I agree that looks like a nicer, more general API than mine proposal, which is exactly what I was hoping for, so many thanks!

Regards,

Nuno

You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/-Kmtx-sr3G8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWmTubCfieowziq9BgyqiJddp6sv2g%3Dm4v44FF5P-5aHg%40mail.gmail.com.

[Andrew Harris]

I just want to highlight a theoretical cliff around  `iter.Push` or `func processor(seq iter.Seq[V]) R` contrasted with solutions involving step and value callbacks (built-in or custom sqlite aggregate functions, also of interest: windowed variants). Aggregate functions expressed like the latter, or some of the pretty amazing aggregate algorithms, really tend towards stepwise, inductive proof over potentially infinite inputs. The stepwise operations have nice algebraic properties, (e.g. they are left- and right-associative, or at least the difference isn't considered essential, or sequence order is assumed a priori, etc. - sometimes the technique is expressing the problem with clever operations). Invariants about the result hold at each step (and for windowed aggregate functions, at inverse steps).

With this kind of setup, `func processor(R, V) R` or the step / value pair `func Step(V)`, `func Value() R` make a ton of sense. However, without this kind of setup, the same callbacks can still be used ... I think what really changes is that `func Value() R` shouldn't be called until the sequence has terminated, and if it's really necessary there's correspondingly less clarity about the time and space complexity needed to support `func Value() R`.

The concurrency of `iter.Push` is definitely interesting, and `func processor(seq iter.Seq[V]) R` is nice, but the combination of the two seems less nice (still interesting) relative to other approaches.

[Jason E. Aten]

I wasn't quite able to see how Push would suspend the processor func 

that takes the seq (without a goroutine).

I'm able to get a variant, here named Push2, to "work" in the sense of 

not needing goroutines at all.  Push2 takes the processor func as a callback

in a yield2() call.  This also addresses Andrew's desire for the possibility

of sampling intermediate results from an infinite stream.

https://go.dev/play/p/EM9mYLnHYM3

(copied here for ease of reference, same as the playground link above:)

package main
import "iter"

// "processor"
// `in` could be an iter.Seq also/in another variant.
func consumeAndSum(in chan int, yield func(int) bool) (sum int) {
i := 0
for v := range in {
sum += v
i++
if i%10 == 0 {
if !yield(sum) {
return sum // return an intermediate result, and keep going.
}
}
}
return sum
}

func main() {
yield2, stop := iter.Push2[int]()
for i := range 5 {
prelimSum, ok, done := yield2(i, consumeAndSum)
if ok {
println("prelimSum = ", prelimSum)
}
if done {
break
}
}
println("all done. final sum = ", stop())
}

[Jason E. Aten]

noted on the playground, here for completeness:

the callback ("processor") would return the result that stop() gives, 

and would never be called again if it yielded after the `in` input stopped.

// return (from the processor callback) is equivalent to a final yeild(sum).
// If yield() is called after `in` is closed,
// it never returns to the caller.

[Jason E. Aten]

Or, without the flexibility (hazard?) of calling back to a different processor each time,

it could be fixed at the initial call to Pull2:

https://go.dev/play/p/dvANfEH2bWn

[Nuno Cruces]

Hi Andrew.

My personal goal here is exclusively to support aggregate functions, which in SQLite terms actually have `Step(V)` and `Final() R` callbacks (rather than `Step(V)` and `Value() R`), because those semantics more closely match iterators and `iter.Seq`.  I have a different interface for window aggregates (that also has `Inverse(V)`). Reusing `Value(R)` for aggregates and moving cleanup to `io.Closer` is my attempt to make the interface more Go-like.

Other databases take an even less "iterative" approach to window functions, instead requiring a `Merge(A, A) A` operation (rather than `Inverse(V)`) which merges two intermediate results, and then they find a tree of the hopefully minimal amount of aggregations and merges to cover a window, reusing aggregations as they go. This is more suitable to be implemented by certain data structures, and also more amenable to computing subtrees in parallel, which is why it's preferred, I guess.

But this is all even more removed from iteration, so I didn't consider it, even though I'm searching for something that's more generally useful than my own narrow needs.

Regards,

Nuno

[Nuno Cruces]

Hi Ian,

On Tuesday 25 February 2025 at 18:15:40 UTC Ian Lance Taylor wrote:

I think what you're presenting is an argument for

package iter

// Push returns an iterator, a yield function, and a stop function.
// The iterator will return all the values passed to the yield function.
// The iterator will stop when the stop function is called.
// This provides a way to flexibly convert a sequence of values into a Seq.
func Push[E any]() (seq iter.Seq[E], yield func(E), stop func())

I haven't been able to make this interface work to hide the goroutine logic from the caller (which IMO is most of the point).

I may be missing something but ISTM that for this to work (and when implemented with channels) the caller of seq(yield) needs to run in its own goroutine.

So, if we return seq(yield), we force the caller of Push to then spin a goroutine to call seq (and it becomes harder to enforce no parallelism).

Am I wrong?

Thanks again!

Nuno Cruces

[Nuno Cruces]

Assuming I'm not wrong, this is my updated proposal:

package iter

// Push takes a consumer function, and returns a yield and a stop function.
// It arranges for the consumer to be called with a Seq iterator.

// The iterator will return all the values passed to the yield function.
// The iterator will stop when the stop function is called.

func Push[V any](consumer func(seq iter.Seq[V])) (yield func(V) bool, stop func())

This is the implementation:
https://github.com/ncruces/go-sqlite3/blob/8adb838/ext/seq/push.go

Regards,

Nuno

[Ian Lance Taylor]

That's a fair point. I think that the communication between the
iterator and the yield/stop function can be done via coroutines, but
that doesn't quite fit the current coroutine implementation in the
runtime.

My concern with your alternative is that it may be fragile. It should
be possible for the function to pass the iterator to another
goroutine, but it's not clear to me what will happen to the coroutines
in that case. With for/range the language inherently constrains what
can happen with the coroutines--and the code is still really
complicated with all kinds of special cases around panics and passing
the yield function around.

Ian

[Nuno Cruces]

I may be missing something, but if your concern about passing the iterator to another goroutine, is with closing the wait channel here (which is indeed problematic), I think this change resolves that issue.

This has the caveat that there's some degree of parallelism introduced by moving the initial wait outside consumer: consumer may now run in parallel with the caller of Push, at least until it blocks ranging next.

The coro version does not have this issue (coroutines start paused), and otherwise it seems the observable behaviour matches this new version better.

Nuno