Thoughts on channel semantics

[Marcelo Cantos]

I've started looking more heavily into Go, and one of the things that surprised me is the semantics of channels. In particular, there seems to be a lot of confusion in the community about how to signal closure of a channel under various usage scenarios.

Several years ago, I created a channel-based micro-threading library in C++. It was just a toy that never went anywhere, but it did have a very simple and elegant model of controlling the channel life-cycle that worked extraordinarily well. All channels were created as a pair of end-points: read and write. Channels themselves were not directly accessible, only their end-points. Each end-point was reference-counted. When an end-point's reference count hit zero, the channel was poisoned and the other end-point permanently set to the signalled state. Thus, everyone waiting at the other end would eventually wake up and throw an exception. If both end-points reached zero, the channel was destroyed.

This model made it incredibly easy to manage the life-cycle, not only of channels, but also of graphs of micro-threads connected by those channels. As a simple example, a micro-thread could be a filter. That is, it consumed the read-end-point of an upstream channel, and transformed received messages for delivery to the write-end-point of a downstream channel. If the producers feeding the upstream channel all went away, the read-end-point would be poisoned and the filter would die the next time it tried to read a value. Likewise, if all downstream consumers went away, the next attempt to write a transformed message would kill the filter. There was no need for a control channel to tell the filtering micro-thread to die. There was also, therefore, no need to explicitly track multiple producers and consumers. Better still, if downstream from the filter was another filter, the second filter would follow suit, so that a whole pipeline of filters could be torn down domino-like simply by releasing the write-end-point at the head of a chain or the read-end-point at the end of the chain. Furthermore, arbitrarily complex topologies could be maintained by writing appropriate exception-handling semantics into micro-threads. For instance, a multiplexer would die when all its inputs were gone, whereas a zipper died when any input ran dry.

The key to the model was that when a reference count reached zero, there was no way to reconstruct that end-point, so the system could conclude, without qualification, that the associated channel would never transmit again. No other signalling was required. Of course, this would have worked just as well in a garbage-collected environment, but another nice thing about the model was that channels, being unbuffered, couldn't participate in reference cycles, so reference-counting would never leak. (I also didn't support SMP, so reference-counting was cheap.)

I'm not a CS major, and I haven't invested my career studying and advancing the state of the art in the design of concurrent systems. So I don't know if my design had some fundamental flaw that would have rendered it useless for large-scale problems. From memory, I was able to send a single message through a chain of one million micro-threads in less than a second on a circa 2005 notebook, so I guess it wasn't obviously and horribly broken. Nonetheless, I won't be so bold as to state that Go should be changed to follow my model. I only thought to describe it in the hope that it suggests some useful ideas for simplifying the idiomatic use of channels in Go.

If I've misunderstood the design of Go's channels or there are good reasons that my design couldn't possibly work (either in Go specifically, or in general) I'd love to get feedback from the folks on this list.

[Kyle Lemons]

Try it :).

On Wed, Jul 25, 2012 at 7:37 PM, Marcelo Cantos <marcelo...@gmail.com> wrote:

I've started looking more heavily into Go, and one of the things that surprised me is the semantics of channels. In particular, there seems to be a lot of confusion in the community about how to signal closure of a channel under various usage scenarios.

I wouldn't say there is a lot of confusion.  There are people who insist on doing it incorrectly for the sake of brevity and there are people who write code that depends on the behavior of the current implementations, but it's exceedingly simple.  A channel close operation is a send operation after which no other messages can be sent.  Receivers discover that the channel is closed the next time they read.

In the same way that a receiver cannot send a message back to the sender on the same channel, so too is it an error for a receiver to close a channel and expect the sender to find out about it.  It is a trivial matter in many cases to arrange for another channel back to the sender for duplex communication, even if it's only to indicate a lack of interest in more values.  The fact that current implementations will panic on the next send if the receiver closes the channel is not guaranteed in the spec the way I read it, and any number of optimizations down the line may behave differently.  The way I read the memory model, the sender may not discover that the channel was closed in a timely fashion (if ever), though (unfortunately, in my opinion) the spec says that it will generate a run-time panic when(/if) it does discover it.

[Øyvind Teig]

kl. 07:57:35 UTC+2 torsdag 26. juli 2012 skrev Kyle Lemons følgende:

Try it :).

On Wed, Jul 25, 2012 at 7:37 PM, Marcelo Cantos wrote:

I've started looking more heavily into Go, and one of the things that surprised me is the semantics of channels. In particular, there seems to be a lot of confusion in the community about how to signal closure of a channel under various usage scenarios.

I wouldn't say there is a lot of confusion.  There are people who insist on doing it incorrectly for the sake of brevity and there are people who write code that depends on the behavior of the current implementations, but it's exceedingly simple.  A channel close operation is a send operation after which no other messages can be sent.

 

If a sender dies without knowing it, and thus unable to send a close, is this also reflected to the receiver? How?

Alternatively: If the sender does not know that it has died, would the run-time system know it - and send the close on behalf of the goroutine (like closing all open files)?

Go is not a multicore non-shared-memory language, so any communication between Go executables on remote cored would be (like) over sockets. And these sockets cannot be part of a select (since a socket is not a channel). This driver type functionality is handled over to C or other languages, I have learnt in this group.

[Steven Blenkinsop]

On Thursday, July 26, 2012, Øyvind Teig wrote:

If a sender dies without knowing it, and thus unable to send a close, is this also reflected to the receiver? How?

Alternatively: If the sender does not know that it has died, would the run-time system know it - and send the close on behalf of the goroutine (like closing all open files)?

What you do is defer closing the channel. That way, if the sender panics, the channel will still be closed. If the defer isn't run, you have bigger things to worry about. It is possible to have a channel closed like a file by having all senders access it via a pointer, and registering a finalizer on that pointer.

[John Beshir]

On Thu, Jul 26, 2012 at 11:18 AM, Øyvind Teig <oyvin...@teigfam.net> wrote:
>
>
> kl. 07:57:35 UTC+2 torsdag 26. juli 2012 skrev Kyle Lemons følgende:
>>
>> Try it :).
>>
>> On Wed, Jul 25, 2012 at 7:37 PM, Marcelo Cantos wrote:
>>
>>> I've started looking more heavily into Go, and one of the things that
>>> surprised me is the semantics of channels. In particular, there seems to be
>>> a lot of confusion in the community about how to signal closure of a channel
>>> under various usage scenarios.
>>
>> I wouldn't say there is a lot of confusion. There are people who insist
>> on doing it incorrectly for the sake of brevity and there are people who
>> write code that depends on the behavior of the current implementations, but
>> it's exceedingly simple. A channel close operation is a send operation
>> after which no other messages can be sent.
>
>
> If a sender dies without knowing it, and thus unable to send a close, is
> this also reflected to the receiver? How?
>
> Alternatively: If the sender does not know that it has died, would the
> run-time system know it - and send the close on behalf of the goroutine
> (like closing all open files)?

Channels aren't owned by goroutines. Assigning responsibility for
reads/writes to sending/receiving goroutines is a design abstraction
created by developers. This means that no, there's no notification to
readers/writers on a channel that a goroutine handling the other end
has gone away.

This isn't a significant issue, though, because channels are
local-only and communicate only between goroutines in the same
process, and goroutines aren't independent processes, and can't be
independently killed by the OS. In general, if something "dies" it is
likely to be your entire program. If a goroutine panics and nothing
catches it, the whole program terminates.

While it is still possible to imagine scenarios where a goroutine
terminates abruptly erroneously and the rest of the program keeps
running, it is typical and much simpler when writing a program to
treat the possibility of another goroutine failing the same way you
treat the possibility of another function you call failing- writing it
assuming everything works as designed and fixing bugs when it doesn't,
rather than trying to model each goroutine as an independent process
which deals with any possible failure of other goroutines.

You can defer the close of the channel, but if a goroutine abruptly
stops handling its channels you probably have bigger problems. If a
panic is not caught somewhere it brings down the whole program, and
even if it is caught, whatever job the goroutine was doing is now not
happening. Trying to deal with all possible cases of this is unlikely
to be worth it, given that they are all part of the same process.

[Øyvind Teig]

kl. 23:50:13 UTC+2 torsdag 26. juli 2012 skrev John Beshir følgende:

These reflections are interesting. I remember I once tried to stop an occam program gracefully, by sending own "poison" messages over the channels. This was in a diesel engine power computation system for ships, and the engines are 4 stories tall. 1992? Occam 2 has no "back door" to the channels, so I had to try it by hand. But when something failed, so did its posibility to send or receive, and the parties that it was connected to were perhaps not there, either. I never succeded, it was too complicated. The standard assert or CAUSERROR that occam 2 had I had to revert to. This was no problem because after all the engine just needed me to restart the whole thing. I needed combustion pressure from all cylinders, not minus one anyhow.

Hoare writes in 1981:

The Emperor's Old Clothes

http://www.cs.fsu.edu/~engelen/courses/COP4610/hoare.pdf

Communications of the ACM, Feb. 1981

Hus Turing Award lecture,

"One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies."

(About ADA at the stage it was:)

"Gradually these objectives have been sacrificed in favor of power, supposedly achieved by a plethora of features and notational conventions, many of them unnecessary and some of them, like exception handling, even dangerous."

I don't know if Hoare still means this, but he certainly helped making occam with David May after this. 

I don't know if he's entirely right either. I assume exception handling has come to stay. Some times I like it. But I must admit seeing a complete trace of an exception from a Java program,.. well - isn't it most often the initial error we want? But then again, I do want to see the caller..

There is no one solution, and the Go designers have given us the possibilities.

The fact "channels aren't owned by goroutines" is also a design choice. But in occam-pi they are (I believe) and one may still send channels over channels. Go even lets me read from a full channel that I attempted to write to (another thread here). Nice, but it also tells that a goroutine neither owns the channel nor its end, exclusively.

So, there is no channel usage verification in go. Therefore "assigning responsibility for reads/writes to sending/receiving goroutines is a design abstraction created by developers." Of course. Even with channel usage verification we are not free of design thinking. And often, this particular tools isn't necessary. Has it been implemented to meet the critics? (Shouldn't everything look like a subset of C++ plusplus?) Not many engineers have the option to say against a language design while it's being created. Most of us has to rely on some clear minds. And try to be one, too.

- Øyvind

[Marcelo Cantos]

Thanks all for the responses. However, I must confess that this all feels somewhat unsatisfying. Firstly, I sense that the simplicity Kyle refers to is primarily for the sake of the implementation, but at the expense of complexity for application authors. Sure, I can call close and the channel neatly wraps itself up, but what if I don't know whether I should call close? What if I have multiple producers that don't know about each other? Does this suggest that, as a best-practice, each channel should be private to a single writer/reader pair? While it is certainly possible to thus compose applications, the degree of constraint it imposes might be quite frustrating at times. For instance, to set up a filtering goroutine with multiple producers and/or consumers, I would have to either code the filter to accept multiple input and output channels, which would substantially complicate the logic (not to mention having a separate control channel to add and remove producers and consumers, as opposed to simply sharing the end-points around), or introduce a multiplexing goroutine before the filter and/or a demultiplexer after it.

So far, nothing I've read suggests that channels couldn't or shouldn't be implemented according to the design I presented, merely that they weren't. I imagine that it could be implemented with almost no change to the language or the runtime: garbage-collect channel end-points independently of each other and call close() in the finalizer (catching and ignoring any panics).

A few asides are in order:

1. I see no fundamental problem with readers signalling channel-death to writers. There is already a form of upstream signalling when a channel is full.
   
2. I think the crashing goroutine scenario, while important to consider, is a red herring in this instance. Problems arise even when goroutines exit normally.
   
3. Channel ownership is also perhaps a non-issue, since, as described above, GC should suffice.
4. Relying on GC would incur some latency in tearing down unreachable goroutines. While this is just par for the course in a GC environment, it might take several GC passes to tear down a deep graph.
   

[Dave Cheney]

Would something like this work for you ?

package main

import "runtime"

func main() {
c := make(chan int)
runtime.SetFinalizer(&c, func(c *chan int) {
close(*c)
})
select {}

[Uriel]

close() becomes very simple if you think of it as simply endrange(),
it is overused because the name is familiar and people expect that if
channels can be close-d they need to be, but most of the time you
don't need close() at all, and you can ignore that it exists.

Uriel

[Marcelo Cantos]

(Prefaced with, "I'm a Go noob, so I could have it all wrong...")

I don't see how that would help. You're closing the channel at the point it's being garbage collection, by which time absolutely no one, reader or writer, has access to it.

[Dave Cheney]

My apologies, I appear to have misinterpreted your previous statement of

"So far, nothing I've read suggests that channels couldn't or
shouldn't be implemented according to the design I presented, merely
that they weren't. I imagine that it could be implemented with almost
no change to the language or the runtime: garbage-collect channel
end-points independently of each other and call close() in the
finalizer (catching and ignoring any panics)."

That said, I wouldn't use the solution outlined below, it was merely a
thought experiment.

[Marcelo Cantos]

Sorry, but I don't quite understand what "endrange()" means or how thinking this way solves the problems that are bothering me. Say I have five producers that don't know about each other, all writing into a single channel read by a filter, which passes transformed/filtered messages through to a downstream consumer. How do the five producers let the filter know when it (and, transitively, the downstream consumer) is not needed any more? What is the idiomatic solution for this very common scenario? Is it a separate control channel? Is it a channel per producer, which dramatically complicates the filter and still requires a control channel? Is it a HELLO-MSG1-MSG2-...-GOODBYE protocol? Just let the filter leak? None of the solutions I can think of are particularly palatable. So, what am I missing?

[Dave Cheney]

How about this ?

http://play.golang.org/p/PSpnGj632G

[si guy]

@Marcello (sorry, can't quote ATM)
You can flip the idiom around, turning the producers into consumers and vice versa. Have the data generators hold the receive end of a channel of requests, where each request contains a chan to respond with data on (commonly referred to as the request-response chan idiom, or something).
When the data receiver closes the request chan the data generator will detect it via the val,ok := <- chan idiom.

There are a bunch of examples of this in go-nuts and other places(talks, std lib, blogs etc.) under request response chan.

This is not too hard to multiplex in a number of ways, for example a load balanced approach where new requests are routed to the first non-busy producer via a select.

close() cleans everything up nicely in a cascaded fashion, where the multiplexer detects the close and closes the request chans to the producers. This has the added benefit of not interrupting the in-flight requests (if you still care about them). I'll post some code examples later today.

[Øyvind Teig]

kl. 06:46:57 UTC+2 mandag 30. juli 2012 skrev si guy følgende:

@Marcello (sorry, can't quote ATM)
You can flip the idiom around, turning the producers into consumers and vice versa. Have the data generators hold the receive end of a channel of requests, where each request contains a chan to respond with data on (commonly referred to as the request-response chan idiom, or something).

That paragraph is difficult to understand for me. A producer produces something and can't just be considered the consumer. If Mercedes produces an A-Class for me and I consume it over the next ten years then I am not turned into a producer just by ordering that car from Mercedes? But I certainly produce an order for Mercedes? But this is an order for the produce, not a produce per se. Or does this thing based metaphor miss the point?

But I'll try see interpret what you say. Clients Client.1-Client.n each has a pair of channels (Chan-request, Chan-respond), which a singe Server listens to Chan-requests from, in a select, and then respond on Chan-respond[iClient]? Server then starts a session, which also includes the requester Client, the Server should certainly not listen in that select again, since Go does not have input guards. I am not certain who's the "data generator" here, and I don't seem to have flipped anything. 

 

When the data receiver closes the request chan the data generator will detect it via the val,ok := <- chan idiom.

There are a bunch of examples of this in go-nuts and other places(talks, std lib, blogs etc.) under request response chan.

This is not too hard to multiplex in a number of ways, for example a load balanced approach where new requests are routed to the first non-busy producer via a select.

What if that non-busy producer has nothing to send? It only produces something when it has been queried for some service by some other process, like me pressing a button in my browser. Has it already "registered" to state that it does have something? Is this "registering" part of the "flip around"? As you see, I am confused.

close() cleans everything up nicely in a cascaded fashion, where the multiplexer detects the close and closes the request chans to the producers. This has the added benefit of not interrupting the in-flight requests (if you still care about them). I'll post some code examples later today.

- Øyvind

[chl]

you mean something like this:

http://play.golang.org/p/d5DNFsTHNI

[Øyvind Teig]

kl. 06:31:57 UTC+2 mandag 30. juli 2012 skrev Dave Cheney følgende:

How about this ?

http://play.golang.org/p/PSpnGj632G

This is a general comment to all who are helpful enough to produce Golang-nuts code. Written by me, who have ever only tried to read Go code. (But I have produced several hundred thousand lines of other concurrent code with channels over some years). To be honest, I am not getting much wiser by some of the code examples I read here, because they mostly are so bewilderingly free of comments. You guys who make code, the couple- to three- or four lines that you think hold the trick, explain them! Code is often so go-ioiomatic-saturated that with no comments it is hard. If this came to your head as an excellent idea, it needs to be understood by readers as what it is. Or is it a go idiom not to explain? Ok, "all producers have exited" is fine, but what actually happens in for _ = range c { } at that position in the code? The answer is probably simple, and it may be so idiomatic that asking for it and explaining it should not be necessary, and I should be ashamed. But then, for myself at least, trying to explain some code often has me understand that it does not work the way I thought. I assume the above code is fine and pinpoints a good point or two, but I at least would not be able to see that.

- Øyvind 

[Uriel]

On Mon, Jul 30, 2012 at 5:51 AM, Marcelo Cantos
<marcelo...@gmail.com> wrote:
> Sorry, but I don't quite understand what "endrange()" means or how thinking
> this way solves the problems that are bothering me.

close() pretty much exists so the 'for range' construct can work on a
channel as an iterator over a limited set of items.

It can be useful in other situations, but often you will do fine
assuming close() doesn't exist, notice that many previous CSP
languages didn't have close() at all.

In other words: CSP programming style doesn't require close().

Uriel

> Say I have five
> producers that don't know about each other, all writing into a single
> channel read by a filter, which passes transformed/filtered messages through
> to a downstream consumer. How do the five producers let the filter know when
> it (and, transitively, the downstream consumer) is not needed any more? What
> is the idiomatic solution for this very common scenario?

I don't think in practice it is a common scenario at all, gorotines
are cheap, specially while blocked on a channel, just let them be.

I think part of the problem is you are thinking in 'abstract' terms,
rather than in terms of a real problem. Tell us what real problem you
are trying to solve, and then we can find an idiomatic answer.

Uriel

[Øyvind Teig]

kl. 09:46:40 UTC+2 mandag 30. juli 2012 skrev Uriel DeLarge følgende:

While this is quite constructive angle for a particular problem, given the language in mind (Go), the 'abstract' thinking may be equally constructive, also for a concrete problem. I smell that the search for "idiomatic answer" in some cases chokes the abstract reasoning here. Marcelo Canto's ideas seem to me to be good ideas, also in Go context, even if they may me "Go non-idiomatic".

- Øyvind

Uriel

[roger peppe]

On 30 July 2012 04:51, Marcelo Cantos <marcelo...@gmail.com> wrote:
> Say I have five producers that don't know about each other, all writing into a single

> channel read by a filter.

If the producers don't know about one another, then they must have
acquired the channel by some means, probably because you've
passed it to them. As a producer, if I have an API that accepts a
channel from outside, it is bad form to close it, because there
may be other writers. In this case, it is conventional to provide
some other means of notification that the writer has exited.

There are several ways of doing this. Probably the simplest
is that a function returns when it has done sending to the channel:

func (*SomeType) Writer(c chan<- int) {
for i := 0; i < 10; i++ {
c <- i
}
}

This makes it straightforward to wrap some synchronisation
around the function call, by sending on a channel or using
a WaitGroup. Here's an example:

http://play.golang.org/p/1hEWeIEgpG

Another way that can work well when the value being sent
on the channel is more complex is to have the sender send
a sentinel value (such as nil) on the channel when it exits.

To be honest, this problem is fairly rare, as it is more
common for producers to return a channel and close
it when they're done. This way you don't have the
above issue, although you may well need to manage
the multiplexing yourself by starting a goroutine for
each producer. In that case, all the producers will know
about each other, so you can use whatever coordination
mechanism you like.

BTW using a channel per producer does not "dramatically
complicate the protocol" - it's really very simple to implement.

[Bob Hutchison]

On 2012-07-29, at 11:51 PM, Marcelo Cantos wrote:

Sorry, but I don't quite understand what "endrange()" means or how thinking this way solves the problems that are bothering me. Say I have five producers that don't know about each other, all writing into a single channel read by a filter, which passes transformed/filtered messages through to a downstream consumer. How do the five producers let the filter know when it (and, transitively, the downstream consumer) is not needed any more? What is the idiomatic solution for this very common scenario? Is it a separate control channel? Is it a channel per producer, which dramatically complicates the filter and still requires a control channel? Is it a HELLO-MSG1-MSG2-...-GOODBYE protocol? Just let the filter leak? None of the solutions I can think of are particularly palatable. So, what am I missing?

Has anybody mentioned sync.WaitGroup yet? Maybe I'm missing the point, or not following along well enough, but that won't stop me :-) If each producer registered with a WaitGroup then called Done when done, then you could write a goroutine to Wait on the WaitGroup and upon waking close the channel.

Cheers,

Bob

[Bob Hutchison]

On 2012-07-30, at 8:12 AM, roger peppe wrote:

> To be honest, this problem is fairly rare

I don't know about that. If I'm reading the question correctly the OP is asking about a common-place concurrency pattern of many-to-one (aka fan-in).

> , as it is more
> common for producers to return a channel and close
> it when they're done.

I'm not sure I buy this either. What would your server look like? It can't know at compile time what the channels are, so you can't use a select. How do you avoid a busy wait loop? Might want to take a look at this thread as an example: https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/O6IVezJZ7lQ

Cheers,
Bob

[Maxim Khitrov]

On Thu, Jul 26, 2012 at 1:57 AM, Kyle Lemons <kev...@google.com> wrote:
> In the same way that a receiver cannot send a message back to the sender on
> the same channel, so too is it an error for a receiver to close a channel
> and expect the sender to find out about it.

Go channels are frequently described as "typed UNIX pipes," yet the
termination of the consumer in a pipe can cause the termination of the
producer via SIGPIPE. I still see no theoretical or practical reasons
why closing the receiving end of a channel as a way of signaling the
sender had to be disallowed. Or, indeed, why this couldn't have been
done implicitly by splitting the two ends and having the GC call close
on the end that is no longer referenced.

I love almost everything about the design of Go, but I agree with
Marcelo that channel semantics feel a bit artificially limited,
resulting in rather inelegant workarounds.

- Max

[Ian Lance Taylor]

On Mon, Jul 30, 2012 at 5:51 AM, Bob Hutchison
<hutch-...@recursive.ca> wrote:
>
> On 2012-07-30, at 8:12 AM, roger peppe wrote:
>
>> To be honest, this problem is fairly rare
>
> I don't know about that. If I'm reading the question correctly the OP is asking about a common-place concurrency pattern of many-to-one (aka fan-in).

Many-to-one is easy to implement in Go. In order to set up the
abstract problem, I think the OP has additional constraints: 1) the
various producers generate an unknown number of values; 2) the single
receiver accepts an unknown number of values, and then stops accepting
values; 3) when the receiver stops accepting values, it's OK for the
producers to have generated extra values; 4) when the receiver stops
accepting values, the producer goroutines should exit; 5) it's OK if
the producer goroutines don't exit immediately. With that set of
constraints, it becomes desirable to have a signalling mechanism from
the receiver to the various producers. The OP is suggesting that a
garbage collector based signalling mechanism would be convenient. One
way to implement a GC based signalling mechanism would be to split
each ordinary Go channel into two separate unidirectional channels.
Then if the receiving channel is garbage collected, the sender
channels receive some sort of notification, details omitted.

So we have a concrete partially specified solution for an abstract
problem. I really don't know how common this abstract problem is; I
have not encountered this scenario myself. As always, it would help
to have a concrete example rather than an abstract one.

It's also worth considering the concrete implementation. What should
happen when sending a value on a channel whose receiver has been
garbage collected? Right now sending a value on a channel is a
statement. Presumably sending to a GC'ed receiver should panic; how
else could it be implemented? We could also introduce a statement "ok
= c <- v" that returns an indication of whether the value was sent,
and similarly a "case ok = c < -v:" in a select statement. Note that
panicing on a send to a GC'ed receiver complicates simple programs;
after all, in a simple program it does not great harm to leave a
goroutine waiting around for a receive that will never happen. So now
simple programs have to become more complicated: they have to check
the result of each send, or recover panics.

Also, since channels are now split into separate senders and
receivers, it becomes harder to use a channel to pass a token back and
forth among different goroutines. You need to have both the sender
and receiver channels. But I guess that is not too difficult.

Are there other ways to solve the OP's abstract problem? Of course,
although they are admittedly more complicated. Each sending goroutine
does need a quit channel, and does need to operate in a select loop
that checks the quit channel. Closing the quit channel then serves as
a useful indicator to tell an arbitrary number of sending goroutines
to quit. So, we can arrange to put the quit channel and the receiving
channel in a struct, and set a finalizer for the struct so that when
it is GC'ed the quit channel is closed. That appears to solve the
OP's original problem, at the cost of complicating all the sending
goroutines to use two channels and a select statement.

So my current answer to this thread would be that yes, the OP's
suggestion is possible, and does simplify a certain class of programs,
but at the cost of complicating a different class of simpler programs.
The language could have been designed as the OP suggests, but it
would not be clearly superior, and the current language does permit
implementing what the OP wants. As we've all said many times,
language design is a matter of trade-offs, and I think Go has made a
reasonable choice here.

Ian

[Ian Lance Taylor]

On Mon, Jul 30, 2012 at 6:23 AM, Maxim Khitrov <m...@mxcrypt.com> wrote:
> On Thu, Jul 26, 2012 at 1:57 AM, Kyle Lemons <kev...@google.com> wrote:
>> In the same way that a receiver cannot send a message back to the sender on
>> the same channel, so too is it an error for a receiver to close a channel
>> and expect the sender to find out about it.
>
> Go channels are frequently described as "typed UNIX pipes," yet the
> termination of the consumer in a pipe can cause the termination of the
> producer via SIGPIPE

You need to unpack that a little bit, though. SIGPIPE terminates the
program. That makes sense in the Unix pipe world, where many programs
exist only to read from stdin and write to stdout. Channels obviously
do not work that way. It would only very occasionally make sense for
a send on a channel whose receiver has been closed to terminate the
program. If you want to extend the analogy to goroutines, you need to
figure out what it means for a goroutine to terminate on a channel
send, and how that should be implemented, and what it means for the
rest of the program.

Ian

[Marcelo Cantos]

[Editorial note: I've spent quite a while on this email, and Ian's post just came through. So, with apologies, I'm going to be lazy and post as-is. I'll digest Ian's comments and perhaps respond in due course.]

It will certainly be easier to talk about concrete examples, but please keep in mind the caveat Øyvind Teig points out — idiomatic usage for specific cases wasn't the key motivation behind my post. I more interesting in exploring a possible improvement to Go that (I hope) would entail negligible change to the language or runtime.

Example 1: Five I/O-heavy producers (five sockets) write into a channel that four CPU-heavy consumers (four cores) read. I wish to interpose a special logging "channel" between the producers and consumers without changing the implementation of either the producers or the consumers.

Example 2: Ten goroutines read URLs from ten files and write them into a single channel. Three consumers read the channel and fetch the documents corresponding to those URLs (three consumers ensures that you don't completely choke your DSL modem), building a word histogram of each document fetched then sending each word/count pair to one of eight channels (one per core), based on hash(word) % 8. Each of the eight channels has a consumer that accumulates incoming word/count pairs into a single histogram. All second stage consumers occasionally report their 10 top hits to a third channel that a single consumer analyses. This final "judging" consumer tracks the most popular terms across all words and stops when it decides that the list has stabilised enough to declare the top 10 words. When the input files run dry and the last document is fetched, I want the judge to immediately declare the winners. Likewise, I want the file readers and document fetchers to stop work as soon as the judge declares the winners early due to stabilisation. Moreover, the network interface might go down, making it impossible to fetch any more documents; at this point I want everyone to stop work and the judge to immediately declare the winners.

In the abstract: I have a multi-stage pipeline with P producers feeding C1 consumers feeding C2 second-stage consumers, etc. The cardinality of each stage is determined by extraneous factors (the number of external input sources, the need for limited parallelism, etc.). Most importantly, any stage of the pipeline might decide to stop work, at which point I want the entire pipeline to collapse and all participating goroutines and channels to go away. I don't want to have to bake in any rules about who is allowed to call a halt to proceedings. I also don't want to impose fixed counts anywhere. For instance, I might decide to increase or decrease the number of document fetchers in response to QoS data.

Note that a pipeline isn't the only topology I have in mind. Cyclic graphs can be constructed to implement feedback loops. The node that combines the input and feedback channels could return either when the input channel runs dry, or when both the input and feedback channels both run dry (depending on the problem), at which point you want the entire feedback loop to tear down. Ditto when all consumers of the loop's output stage go away.

I should reiterate, I'm not that interested in tailored solutions for the particular problems described above. They are merely illustrative. I am sure that it is possible to implement anything I describe using Go as it is today. What I am more concerned about is that all the suggestions I've read seem to amount to either reversing the flow of control (but remaining unidirectional) or implementing by hand the techniques I baked directly into my microthreading library, which could probably be baked into Go with little effort. For instance, the suggestion to use sync.WaitGroup seems like it could solve the problem in a fairly general way, but it means that I have to maintain a separate data structure alongside every channel to track how many producers and consumers are attached to the channel, and I have to remember to call Add and Done in all the right places. And even then, there's no obvious idiomatic way to signal producers when all the consumers are gone (it seems that causing a panic is frowned upon). So, sure, I could probably solve all my problems that way, but it's difficult, noisy and fragile.

I still don't know if I've communicated my intent clearly enough, so here's a self-contained idea that I hope captures the essence of what I'm looking for: I want to be able to treat an arbitrarily complex subsystem with an input and output channel as a drop-in replacement for a regular channel, including having the entire subsystem garbage-collected if the exposed end-points become unreachable from outside the subsystem.

Wouldn't it be nice if Go, which already (almost?) has the necessary information, would do all this bookkeeping for us?

On Monday, July 30, 2012 5:46:40 PM UTC+10, Uriel DeLarge wrote:

[Marcelo Cantos]

I quite agree. In my design, a poisoned channel didn't necessarily imply the death of the microthread. While the default was obviously for the exception to propagate (in turn releasing any other end-points the microthread held), but the microthread could trap the error instead and continue in some altered state (e.g., a multiplexer could happily ignore the exhaustion of a single input channel).

[Ian Lance Taylor]

You appear to be introducing another new concept to the language: the
automatic death of a goroutine in a way that does not kill the rest of
the program. I think a little more unpacking is still required. Is
this a general simplification? Without thinking about it too hard, I
suspect that some things become simpler and some things become harder.

Ian

[Marcelo Cantos]

Sorry, I probably wasn't as clear in my intent as I should have been.

My design used exceptions, but I'm not saying that Go would have to use panics or introduce another concept to achieve the same effect. It could instead convert sending from a statement into a bool expression as you indicated in your other post. Yes, this also changes the way programs must be written, and complicates some simple programs, but it's possibly not as invasive as panicking. I haven't written tons of Go, so I must claim ignorance in this matter. Also, because I'm new to Go, I guess I'm not as concerned about breaking changes as the maintainers are (and should be). I guess I also worry more about how things are vs how they could be, not so much the cost of getting from one to the other.

Your concerns about whether the alternate semantics are actually (or enough of) an improvement are valid. I'll have to give it some more thought.

[roger peppe]

On 30 July 2012 13:51, Bob Hutchison <hutch-...@recursive.ca> wrote:
>
> On 2012-07-30, at 8:12 AM, roger peppe wrote:
>
>> To be honest, this problem is fairly rare
>
> I don't know about that. If I'm reading the question correctly the OP is asking about a common-place concurrency pattern of many-to-one (aka fan-in).

This particular problem is about producers producing values on a shared
channel when they don't know about each other.

In the standard library, I know of only two instances where a channel is passed
for a package to write on - signal.Notify and rpc.Client.Notify. In neither
case is this an issue (the first because the signal handler never quits, and the
second because it sends exactly one value).

>> , as it is more
>> common for producers to return a channel and close
>> it when they're done.
>
> I'm not sure I buy this either. What would your server look like? It can't know at compile time what the channels are, so you can't use a select. How do you avoid a busy wait loop? Might want to take a look at this thread as an example: https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/O6IVezJZ7lQ

It's easy - you funnel changes into a single channel.
Here's an example (imagine that NewGenerator is part of
an external package): http://play.golang.org/p/ZXXl0aYP7q

You might want to watch Rob "Concurrency Patterns" talk:
http://www.youtube.com/watch?v=f6kdp27TYZs

[Maxim Khitrov]

Think of the class of problems solved by generators in Python. When
the iteration ends, the generator object is garbage collected after
raising an exception. If some clean-up is required, catch the
exception, do whatever you need to do, then exit. The same design
pattern and a few others, with some added performance benefits, could
have been supported by Go with no additional complexity. It doesn't
have to be a many-to-one relationship. Having some simple means of
signaling to the sender(s) that no new values are needed is frequently
(in my experience) convenient.

One solution is to quietly garbage collect any goroutine that is
permanently blocked on a send to a channel with no (or closed) receive
ends, following from the idea of splitting the channel parts. The
"comma ok" idiom could be used by producer goroutines that need to
perform some cleanup, so simple code stays simple. This would also
work for cases where a send is performed on the closed send end of a
channel, because I'm not sure that the panic/recover is always
appropriate here. If receives can test for a closed channel, why
require panic for sends?

I don't think you have to remove bidirectional channels to support
this either. A channel returned by make() is bidirectional. The first
cast to a send- or receive-only channel creates a new part of that
type. Thus, the underlying communication framework created by make()
could be referenced by at most three parts of the channel
(bidirectional, send, and receive). The underlying channel stays open
as long as there is at least one bidirectional part, or one pair of
send/receive parts. Closing one receive end closes them all, same goes
for send ends. Closing a bidirectional channel closes everything.

This arrangement gives you a solution to the many-to-one problem for
free - when the receive end is gone (or closed), all senders either
die or return from the send with "ok" set to false. All existing code
using channels is unaffected, provided that it's not your intention to
have deadlocked goroutines stick around indefinitely.

Just some ideas for Go 2 :)

- Max

[Øyvind Teig]

kl. 15:24:56 UTC+2 mandag 30. juli 2012 skrev Ian Lance Taylor følgende:

----

We could also introduce a statement "ok
= c <- v" that returns an indication of whether the value was sent,
and similarly a "case ok = c < -v:" in a select statement.  

I have discussed a primitive channel type that I call XCHAN in an accepted paper for CPA-2012, along the lines of the same ideas as mentioned above. The channel send never blocks but returns a status. A bundled flow control "x-channel" signals when there is space in XCHAN, or the receiver is ready. So, that channel type in a way has three end points.

(The abstract is available at http://wotug.org/cpa2012/programme.shtml (search for XCHAN). Thanks to this group I have also been able to compare some with Go (in the Appendix). The full text will be available on the net after the conference.) 

- Øyvind   

[Marcelo Cantos]

After giving this some more thought, I'm not at all convinced that the proposal to make send a bool-expression would complicate simple programs. Rob Pike's concurrency patterns talk delves into a quit channel about half way in, and it's about the simplest non-trivial program you can have. Here's a paraphrase of his example using current and send-returns-bool semantics (full example at http://play.golang.org/p/IgDmWyhknK)...

// Current

for i := 0; ; i++ {

select {

case c <- fmt.Sprintf("%s %d", msg, i):

// Do nothing

case <-quit:

fmt.Println("Cleaning up...")

quit <- true

}

}

// Send returns bool

for i := 0; c <- fmt.Sprintf("%s %d", msg, i); i++ {}

fmt.Println("Cleaning up...")

done <- true

 

More complex examples may not benefit as much from expression semantics, but, after seeing this example, I would be surprised to see most simple programs become more complicated.

[chl]

if that is true won't people be able to do stuff like this:

if c <- fmt.Sprintln("Hi") || c <- fmt.Sprintln("Hi") || c <- fmt.Sprintln("Hi") {

      fmt.Print("Bye")

}

or even 

if c <- fmt.Sprintln("Hi") {

     fmt.Println("a")

} else if c <- fmt.Sprintln("Hi") {

     fmt.Println("b")

}

and my favourite

switch {

        case c <- fmt.Sprintf("%s %d", msg, i):

// Do nothing

case <-quit:

fmt.Println("Cleaning up...")

quit <- true

}

}

which looks exactly the same as a select statement but do something completely different. I think that having too many ways of doing the same thing complicates the process of understanding the code written by other people, without increasing code complexity.

[chl]

and what would this do ?

c1 <- c2 <- "Hi"

Do we send a value from c2 to c1, or do we send a bool to c1 ?

[Kyle Lemons]

I think this encourages the thinking that a send that doesn't immediately succeed won't succeed, and thus encourage bad programs and an incorrect understanding of channels/concurrency.

[Øyvind Teig]

kl. 16:27:40 UTC+2 tirsdag 31. juli 2012 skrev chl følgende:

if that is true won't people be able to do stuff like this:

if c <- fmt.Sprintln("Hi") || c <- fmt.Sprintln("Hi") || c <- fmt.Sprintln("Hi") {

      fmt.Print("Bye")

}

or even 

if c <- fmt.Sprintln("Hi") {

     fmt.Println("a")

} else if c <- fmt.Sprintln("Hi") {

     fmt.Println("b")

}

and my favourite

switch {

        case c <- fmt.Sprintf("%s %d", msg, i):

// Do nothing

case <-quit:

fmt.Println("Cleaning up...")

quit <- true

}

}

which looks exactly the same as a select statement but do something completely different. I think that having too many ways of doing the same thing complicates the process of understanding the code written by other people, without increasing code complexity.

Merging switch and select seems strange in my eyes, select is an indeterministic operator and switch is simply an ordered if then else. A component in a select may in some languages also be a boolean expression. In Promela this delivers 0 or 1 (average 0.5):

if
:: value = 0;
:: value = 1;
fi

In Go all non-default cases must be communication options, so the example is an aside. But it shows nicely the indeterministic character of ::, select or ALT.

- Øyvind  

On Tuesday, July 31, 2012 8:32:44 PM UTC+8, Marcelo Cantos wrote:

After giving this some more thought, I'm not at all convinced that the proposal to make send a bool-expression would complicate simple programs. Rob Pike's concurrency patterns talk delves into a quit channel about half way in, and it's about the simplest non-trivial program you can have. Here's a paraphrase of his example using current and send-returns-bool semantics (full example at http://play.golang.org/p/IgDmWyhknK)...

// Current

for i := 0; ; i++ {

select {

case c <- fmt.Sprintf("%s %d", msg, i):

// Do nothing

case <-quit:

fmt.Println("Cleaning up...")

quit <- true

}

}

// Send returns bool

for i := 0; c <- fmt.Sprintf("%s %d", msg, i); i++ {}

fmt.Println("Cleaning up...")

done <- true

 

More complex examples may not benefit as much from expression semantics, but, after seeing this example, I would be surprised to see most simple programs become more complicated.

On Monday, July 30, 2012 11:24:56 PM UTC+10, Ian Lance Taylor wrote:

[Rob Pike]

package main

import "fmt"

func main() {
c := make(chan int, 1000)
go func() { for { c <- 1 } } ()
for i := 0; i < 100; i++ {
select {
case <-c: fmt.Println(0)
case <-c: fmt.Println(1)
}
}
}

run it here: http://play.golang.org/p/INacl7a-BU

[Øyvind Teig]

Elegant! I got 45 '1' and 55 '0' on my runs, average 0.45.

I assume I could point to this example in my "A nondeterministic note about nondeterminism" at "http://oyvteig.blogspot.no/2012/04/045-nondeterministic-note-about.html

- Øyvind

[Steven Blenkinsop]

An alternative would be to have a select case that only gets chosen if none of the other cases can possibly proceed (because the channels are nil, or you are sending on a closed channel), as an alternative to default which runs right away. Maybe something like this:

select {

// cases

} else {

    // select would have blocked forever

}

Then people might get confused about when to use default and when to use else, though the distinction should be easily enough made. Of course, the question is whether it would be worth it, and I don't know. It would make it easier to progressively nil out channels to remove cases from a select if you didn't have to manually check all the channels for nil. Also, I remember the main reason for not garbage collecting blocked goroutines was the undesireability of silent failure. This way, if you can define "blocked forever" in a meaningful way, the programmer can choose to handle the case explicitly, and otherwise, programs behave as they do now.

[Rob Pike]

Sure.

-rob

[Kyle Lemons]

That would work with nil'd out channels, but there are also a lot of other ways it can block forever, such as when the other goroutine forgets about the channel, exits, or just simply won't ever send on it again.  Doesn't seem worth its cognitive overhead.

[Steven Blenkinsop]

On Tuesday, July 31, 2012, Kyle Lemons wrote:

That would work with nil'd out channels, but there are also a lot of other ways it can block forever, such as when the other goroutine forgets about the channel, exits, or just simply won't ever send on it again.  Doesn't seem worth its cognitive overhead.

That's why I said "meaningful". That could range from just communicating on nil channels, expand to include sending on closed channels, or go full out and do the whole end analysis thing. However, I don't think that last one is really necessary, since you could just register finalizers on pointers to unidirection channels if you wanted to, and that's where most of the cognitive load arises.

[kortschak]

That's currently illegal, so it's a non-issue it seems (though if something like the proposal were to be implemented the precedence would need to be determined). Having said that, it looks like an ugly construction. Not commenting on the rest of the thread.

Dan

[Marcelo Cantos]

I don't understand what you mean by "doesn't immediately succeed"...

1. If you mean, "returns false", then such an occurrence really does mean that it won't succeed [in future].
   
2. If, OTOH, you mean, "blocks", I don't see how anyone would draw that conclusion. It will eventually return either true, meaning it succeeded, or false, and we're back to point 1.
   

[Marcelo Cantos]

If a select includes a checked receive on a channel and the channel is closed, select can choose to execute the receive, returning false. Thus one can determine that an upstream channel is dead and react accordingly. The proposed semantics provide the same convenience for downstream channels. The alternative suggestion to provide an else clause doesn't provide enough information for the goroutine to react appropriately.

For example, let's say a stats-collection mapper has two downstream channels, an output channel that consumes mapped values and a reports channel that consumes reports containing count, average, min, max, etc. The report is available at all times; sending is triggered by the reporting goroutine performing a receive. If the reports channel dies first, the mapper can probably continue receiving and transforming values until either the input and output channel dies. However, if the output channel dies first, the goroutine probably shouldn't continue delivering reports that will never change. This might seem like a fine point, but it's actually quite an important one. Imagine that the goroutine consuming the reports channel is an isolated reporting function that asks for a report every few seconds in an endless loop. If the mapper doesn't exit when the output channel dies, it will go on feeding frozen reports to the reporting function essentially forever.

[Marcelo Cantos]

On Wednesday, 1 August 2012 00:36:25 UTC+10, chl wrote:

and what would this do ?

c1 <- c2 <- "Hi"

There's a precedence in the chan declaration rules: <- associates with the left-most chan possible. This implies c1 <- (c2 <- "Hi").

Do we send a value from c2 to c1, or do we send a bool to c1 ?

Regarding the examples below, It's already possible to concoct pathological horrors like this using receives on chan bool. On balance, I expect that the benefit of increased symmetry and automatic lifecycle management would more than offset such unlikely hazards.

[Marcelo Cantos]

On closer inspection, there is no way for that expression to parse as a receive, so right-to-left is the only possible legal interpretation. The only other meaningful possibility is left-to-right associativity, which is tantamount to making send chains illegal.

[Marcelo Cantos]

For curiosity's sake, I went and dusted off my old library and came across some interesting elements that I had forgotten about. These might be of interest in this discussion...

1. The library provides a C API on which I built the C++ library as an abstraction layer. It doesn't throw exceptions, but returns bool when a channel is dead.
2. While the C API keeps end-points cleanly separated (the channel creation function takes a pair of handle pointers), the C++ layer has a Channel class for convenience. I vaguely remember this being a questionable decision because you have to make sure Channel instances go out of scope after handing out the end-points.
3. In the core engine, all channels are unbuffered.

That last point is worth dwelling on. I didn't forget to introduce buffered channels. The library simply doesn't need them; I was able to define a simple microthread that behaves for all the world like a buffered channel and cleans up after itself when either end is dead. Even at the API level, it is difficult to distinguish between regular unbuffered channels and buffered pseudo-channels, since their creation functions both take a pair of pointers to end-point handles. In fact, the library also offers a transform pseudo-channel and a filter pseudo-channel.

It's worth noting, too, that my buffered pseudo-channel implementation was unbounded, but because it was implemented as a user-space microthread, I could have made a bounded version, a bounded version that's tuneable via a control channel, an unbounded version with a high-water-mark output channel, and so on. Here's what a regular buffered pseudo-channel might look like (ignoring lifecycle issues due to current semantics, and the lack of generics): http://play.golang.org/p/RLiRfv04Ez.

This was all possible because of one simple difference to what Go has today: upstream dead-channel signalling.

I'm sure Go can get along just fine with the current semantics. I'm not suggesting that the status quo is horribly broken. But, IMO, it's difficult to overstate how much more powerful Go's channels would be if it were possible to create rich pseudo-channels that don't require extra wiring to clean up after themselves. Who knows? It might even be feasible to remove buffered channels from the core.

[Øyvind Teig]

Here is an alternative that seems quite like Marcelo's mechanism (?):

The JCSP library [1] with its Channel class has an interface called Poisonable, which is implemented by all channel ends. See [2] for JCSP documentation.

So dead-channel "upstream" seems to be propagated through PoisonException "beside" the channel. 

This has been so for more than ten years, so the Go designers surely must have studied this and opted not to do it like that(?) Could panic/recover in a usage like this take the same role in Go as exception here does in Java? Have the Go designers left any notes about this?

[1] - http://en.wikipedia.org/wiki/JCSP

[2] - http://www.cs.kent.ac.uk/projects/ofa/jcsp/jcsp-1.1-rc4/jcsp-doc/

- Øyvind

[Jim Whitehead II]

On Mon, Aug 13, 2012 at 9:22 AM, Øyvind Teig <oyvin...@teigfam.net> wrote:
> Here is an alternative that seems quite like Marcelo's mechanism (?):
>
> The JCSP library [1] with its Channel class has an interface called
> Poisonable, which is implemented by all channel ends. See [2] for JCSP
> documentation.
>
> So dead-channel "upstream" seems to be propagated through PoisonException
> "beside" the channel.
>
> This has been so for more than ten years, so the Go designers surely must
> have studied this and opted not to do it like that(?) Could panic/recover in
> a usage like this take the same role in Go as exception here does in Java?
> Have the Go designers left any notes about this?
>
> [1] - http://en.wikipedia.org/wiki/JCSP
> [2] - http://www.cs.kent.ac.uk/projects/ofa/jcsp/jcsp-1.1-rc4/jcsp-doc/
>
> - Øyvind

Well, panic and recovery came into the language much further after
channels, and they are in general designed for truly exceptional
'panic-worthy' circumstances. It's fairly hard to argue that closing
or poisoning a channel falls is such a case.

If you're designing a system that can be gracefully terminated or
reset [1] then that is a part of your system architecture. Handling
those cases in the 'panic/recover' system is then a bit odd, since
you're expecting it to happen. As Peter explains in this paper, the
graceful termination of a channel-based process network is difficult
and prone to race conditions. Poison isn't enough, you also have to
have a bit of code that appropriately sinks each of the input channels
and then also spreads the poison to the other channels in the system.

In fact, creating the equivalent of JCSP's poisonable channels in Go
would not be difficult at all, but you would lose the ability to use
<- to communicate on them (you would use .Send() and .Receive()). The
poisoning/closing mechanism used in Communicating Scala Objects [2]
takes a completely different approach that means closing on a shared
channel end has no impact on the rest of the connected channel ends.
This is very difficult stuff to get right and then to reason about.

I understand the desire to have a slightly different system that makes
it easier to express different kinds of systems architectures. I also
agree that 'close' existing but only really being useful as 1.) a
broadcast channel with no message and 2.) an
end-of-iteration/generation signal can be slightly confusing. I do,
however, feel that the existing channel semantics are incredibly easy
to understand and reason about and I would want to approach any
changes to it with that in the forefront of my mind.

Just some thoughts,

- Jim

[1]: http://kar.kent.ac.uk/20953/1/GracefulWelch.pdf
[2]: http://www.wotug.org/papers/CPA-2008/Sufrin08/Sufrin08.pdf