Proposal for channel sending changes
Gustavo Niemeyer
synchronization around sending on channels. The change in
this CL, which is now merged, causes sends in a channel to
panic if the channel has already been closed.
This means that there's no reliable way anymore to use a channel
with multiple producers without external synchronization, unless
the channel is never closed, or panic-catching logic is used.
People will be tempted to use the incorrect version, which is
mentioned every now and then:
if !closed(c) { c <- v }
The external synchronization isn't entirely straightforward either,
because senders may block, and closing with a previously blocked
sender will also cause a panic.
With that in mind, I propose a minor change in the semantics of
channel sending to simplify the basic workflow around channels,
split in two phases to deal with compatibility issues. The end result
would be as described below.
The following expression would perform a blocking send on the
channel, and would panic if the channel is closed (the behavior
of tip as of right now):
c <- v
The following statement would not panic if v is not delivered because
the channel is closed, and would instead return true. Note that the
'closed' in this statement and closed(c) may differ, since the latter takes
into account if there are still values to be consumed from the channel,
while the former does not:
closed := c <- v
The following statement would also not panic if v is not delivered
because c has been closed, and would also not block, returning
ok = false in case v is not delivered immediately for whatever
reason.
closed, ok := c <- v
These are the suggested changes.
Note that the second statement above would create backwards
compatibility issues, but it would be possible to migrate away from
it by splitting the change in two phases, split across separate
releases (potentially multiple releases).
In the first phase:
1a) Remove old unblocking version: ok := c <- v
1b) Introduce new unblocking version: closed, ok := c <- v
Then, on a second phase:
2) Introduce new blocking version: closed := c <- v
These changes would simplify synchronization on the channel
behavior itself without incurring in any further data stored or
any further synchronization primitives.
--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter
Rob 'Commander' Pike
The right answer will be less intrusive than this.
-rob
Albert Strasheim
On Jan 21, 10:19 pm, Gustavo Niemeyer <gust...@niemeyer.net> wrote:
> People will be tempted to use the incorrect version, which is
> mentioned every now and then:
> if !closed(c) { c <- v }
this anti-pattern before the panic-on-send-to-closed CL was submitted.
This code panics now (as it should), so that was a step in the right
direction.
> The following statement would not panic if v is not delivered because
> the channel is closed, and would instead return true. Note that the
> 'closed' in this statement and closed(c) may differ, since the latter takes
> into account if there are still values to be consumed from the channel,
> while the former does not:
> closed := c <- v
closed(c) { c <- v } hack.
> The following statement would also not panic if v is not delivered
> because c has been closed, and would also not block, returning
> ok = false in case v is not delivered immediately for whatever
> reason.
> closed, ok := c <- v
whether closed or ok comes first. But I guess that's a risk you take
with any function that returns two bools.
Looking forward to a solution for this tricky problem. :-)
Regards
Albert
Albert Strasheim
> with multiple producers without external synchronization, unless
> the channel is never closed, or panic-catching logic is used.
> because senders may block, and closing with a previously blocked
> sender will also cause a panic.
c := make(chan bool, 3)
for i := 0; i < 10; i++ {
go func() {
c <- true
}()
}
// give everyone time to block on send
time.Sleep(2e9)
fmt.Printf("closed(c)=%v\n", closed(c))
for close(c); !closed(c); <-c {}
fmt.Printf("closed(c)=%v\n", closed(c))
and this is indeed what happens. The panic says:
panic: runtime error: send on closed channel
But is this really the right language? Perhaps the panic should say
"send on channel after close"?
Which leads me to wonder whether send after close should still be
allowed, but send after closed is true should panic?
With this behavior, one can almost implement multiple senders with a
receiver that has the ability to shut down:
Before sending, a sender RLocks a RWMutex, checks a flag, and if ok,
sends, then unlocks.
To shut down, the receiver Locks the RWMutex, calls close on the
channel, sets the flag to prevent further sending, unlocks, and
receives until closed is true.
The only problem with this scheme is that a sender that blocks on send
to a full channel is never going to release its RLock, which will
prevent the receiver from being able to take the lock to shut down
cleanly. So to fix that you need an atomic send-that-could-block-and-
unlock-even-if-it-blocks operation.
Regards
Albert
Russ Cox
A channel enables unidirectional communication from sender
to receiver, just like a Unix pipe. Only a sender should close
a channel, and only a receiver should check whether a
channel is closed. Unlike a Unix pipe, a channel does not
have obviously distinguished read and write ends, so it is
easy to forget the unidirectionality, but it's still there.
If you need the receiver to be able to communicate back
to the sender, whether to say 'stop sending!' or something
else entirely, then you need a second channel. Using
close+closed as a back channel is awkward because it
doesn't fit the model.
I have a different, three-step proposal that would address
the problems people have raised and simplify the language
at the same time.
------------------------------------------------------------------
1. Delete non-blocking operations (ok := c <- v; v, ok := <-c)
from the language. Code that wants to avoid blocking can still
use select, which in most cases leads to equally clear (if not clearer)
control flow. The short (not using select) forms of the non-blocking
operations were not in Newsqueak, not in Alef, not in Limbo, and
that seemed fine. They were added to the Plan 9 C thread library,
partly as a thread-safe way to deal with Alef's ?c and c? and partly
(I suspect) because using that library's alt was so awkward
(no language support).
This has the benefit of removing the idea of a context-sensitive
sometimes-expression, sometimes-statement, different-behavior
form from the language (ok := c <- v).
Not much code in the main tree changes.
http://codereview.appspot.com/4046042/ contains required changes
except in the test directory.
Part of the implementation of this change would be to compile
one-case or one-case-plus-default selects into the simple operations
we have now, so that there wouldn't be any overhead to the rewrites.
2. Issue a release without non-blocking channel operations.
This will cause people to update their code to stop using those forms.
3. Delete closed(c) from the language.
Reintroduce v, ok := <-c to mean read v from c, with ok telling
whether v is a sent value (ok=true) or a zero value because the
channel is closed and empty (ok=false). This makes receive and
check for EOF a single, indivisible operation, so that the range loop
for v = range{} would be safely implemented as:
for {
var ok bool
v, ok = <-c
if !ok {
break
}
}
Like v := <-c, this new v, ok := <-c would be a blocking operation.
------------------------------------------------------------------
I like the following consequences of this change sequence:
* Code cannot misuse closed anymore, because it's gone,
and its replacement is inherently tied to receiving.
* Just as there is no operational difference between v = m[x]
and v, _ = m[x] anymore, there would no longer be an operational
difference between v = <-c and v, _ = <-c. That is, in both cases
the ", ok" gives access to additional information about the
operation on the right without changing its meaning.
(Both cases stand in contrast to the Go we released in 2009.)
* Select becomes orthogonal to the other channel operations.
Want a non-blocking v, ok := <-c? Make it a case in a select
with a default, the same way you make any other channel
operation not block.
I am a little sad to see if c <- v {} disappear, but the truth is that it
doesn't come up very often, probably not nearly often enough to
justify a special form in the language.
Russ
Russ Cox
> allowed, but send after closed is true should panic?
You are going down a very strange rabbit hole.
This is no different than closing a file descriptor.
If you have multiple goroutines writing to it, they
need to coordinate to agree about when to close it.
There is no reason that should be part of the close
operation.
Russ
Ian Lance Taylor
> 1. Delete non-blocking operations (ok := c <- v; v, ok := <-c)
> from the language.
> 2. Issue a release without non-blocking channel operations.
> 3. Delete closed(c) from the language.
We need to consider how select should work with a channel that has had
close() called on it.
case c <- v:
If c is closed, presumably this case will never be selected.
case v = <-c:
One possibility is that this case is selected when c is closed, and v is
set to zero. But with no "closed" function, how can the user know that
the channel has been closed? Do we need to add:
case v, ok = <-c:
?
Another possibility is that v = <-c is never selected when c is closed.
But then select has no way to distinguish between a closed channel and a
blocked channel. So again we seem to need v, ok = <-c.
(I prefer that "case v = <-c" does get selected when c is closed.)
Ian
Albert Strasheim
Thanks for this discussion. Very useful.
On Jan 21, 11:59 pm, Russ Cox <r...@golang.org> wrote:
> > Which leads me to wonder whether send after close should still be
> > allowed, but send after closed is true should panic?
> You are going down a very strange rabbit hole.
> This is no different than closing a file descriptor.
> If you have multiple goroutines writing to it, they
> need to coordinate to agree about when to close it.
> There is no reason that should be part of the close
> operation.
Erlang to Go, mainly how to have "processes" that can exchange
messages, fail, inform others of that failure, and be restarted.
In Erlang, any process can send a message to any other process, so
that's where we run into the multiple sender/single receiver problem.
With our current attempt we've modelled the Erlang process as a
goroutine with a Run function that recovers some panics and the
mailbox with a Go channel. But this model is starting to look wrong
given your analogies between channels and pipes/file descriptors.
It seems we should be connecting our "processes" using two channels,
akin to the two file descriptors one has in a socket connection
between two OS processes.
The problem one runs into then is that there is no obvious equivalent
to the select/epoll syscall for Go's channels, which is used in
traditional systems to deal with variable numbers of file descriptors.
Any thoughts on how to address that?
Regards
Albert
Russ Cox
> close() called on it.
Select is defined to choose from the set of cases that could
execute without blocking. I think to answer these questions
we simply look at what happens if you take the case out of the
select and make it a statement by itself:
> case c <- v:
If c is closed, c <- v does not block; it panics.
So in a select, this case is runnable and, if chosen, panics.
> case v = <-c:
If c is closed and empty, v = <-c does not block; it sets v to a zero value.
So in a select, this case is runnable and, if chosen, sets v to a zero value.
> case v, ok = <-c:
Yes, this would be a valid (new) case in a select.
If c is closed and empty, v, ok = <-c does not block; it sets v to a
zero value and ok to false.
So in a select, this case is runnable and, if chosen, sets v to a zero
value and ok to false.
I certainly understand the argument that, for example,
the first case should be not runnable instead of causing
a panic, but I think it much cleaner and more predictable
to keep select orthogonal to whether a channel is closed.
That is, select should only be about blocking.
Russ
Gustavo Niemeyer
> to the sender, whether to say 'stop sending!' or something
> else entirely, then you need a second channel. Using
Receivers closing the channel isn't what I have in mind when I think
of closing with active senders. As an example of what I have in mind,
in gozk the Close() method will inject a termination token in the
session channel to notify receivers, and right now it forcefully
closes the channel immediately to interrupt the communication. This is
no longer safe now, because it is possible that the event dispatching
loop was blocked attempting to send an old token on the session
channel, and rather than interrupting it would panic.
Is purposefully attempting to interrupt a send operation that bad of a
design in all circumstances? I'd honestly would like to learn why, if
so.
> 1. Delete non-blocking operations (ok := c <- v; v, ok := <-c)
> from the language. Code that wants to avoid blocking can still
> use select, which in most cases leads to equally clear (if not clearer)
This sounds good, even more considering that hopefully the select
statement will grow more useful abstractions in the future, such as
timeouts, creating more opportunities than the all or nothing
block/unblock situation we have today.
> 3. Delete closed(c) from the language.
> Reintroduce v, ok := <-c to mean read v from c, with ok telling
Assuming there's no way to correctly check for closed on the sending
side, that sounds good too, since people will be using closed()
incorrectly forever.
Russ Cox
That's probably a mistake. Why does one goroutine think
it should close the channel while another goroutine thinks
the channel is still available for communication?
That is, how does this situation arise? I'd be happy to
suggest other ways to structure your program, but I don't
know what it's trying to do yet.
Russ
Gustavo Niemeyer
> So in a select, this case is runnable and, if chosen, panics.
Picking the choice which knowingly panics will most probably be
surprising. I believe most people will think of the select as picking
the choice which can correctly run given the condition, much like a
type switch or a general switch statement. The fact that nil channels
are ignored and cause the select to block forever if all-nil also
seems to sustain that POV.
> If c is closed and empty, v, ok = <-c does not block; it sets v to a
That sounds good.
> That is, select should only be about blocking.
It'll be pretty inconvenient to select on multiple choices when one of
them may be closed earlier for whatever reason.
Gustavo Niemeyer
>
> That's probably a mistake. Why does one goroutine think
> it should close the channel while another goroutine thinks
> the channel is still available for communication?
G1 is running a callback coming from libzookeeper to notify that some
event is available, blocked on c <- v. G2 asked to Close() the
communication with zookeeper, and doesn't care anymore about that old
event which was available.
How to terminate cleanly?
Rob 'Commander' Pike
Nigel Tao
>>> closing with active senders.
>>
>> That's probably a mistake. Why does one goroutine think
>> it should close the channel while another goroutine thinks
>> the channel is still available for communication?
>
> G1 is running a callback coming from libzookeeper to notify that some
> event is available, blocked on c <- v. G2 asked to Close() the
> communication with zookeeper, and doesn't care anymore about that old
> event which was available.
>
> How to terminate cleanly?
Can you send a sentinal value (e.g. nil if it's a chan *Event, os.EOF
if it's a chan struct { ...; err os.Error }) instead of calling
close(c)?
roger peppe
>>> closing with active senders.
>>
>> That's probably a mistake. Why does one goroutine think
>> it should close the channel while another goroutine thinks
>> the channel is still available for communication?
>
> G1 is running a callback coming from libzookeeper to notify that some
> event is available, blocked on c <- v. G2 asked to Close() the
> communication with zookeeper, and doesn't care anymore about that old
> event which was available.
>
> How to terminate cleanly?
don't close the channel. make it synchronous (unbuffered) if
you care about events being lost. have another channel to signify closed
with a buffer size of 1.
have a convention that when you send on the event channel, you
also try reading from the "closed" channel.
then:
func sendEvent(e eventType) {
select {
case eventChan <- e:
case <-closedc:
closedc <- true
}
}
Russ Cox
> you care about events being lost. have another channel to signify closed
> with a buffer size of 1.
>
> have a convention that when you send on the event channel, you
> also try reading from the "closed" channel.
please continue this on the gozk thread.
just trying to keep separate conversations separate.
Russ Cox
> the choice which can correctly run given the condition, much like a
> type switch or a general switch statement.
>
> The fact that nil channels
> are ignored and cause the select to block forever if all-nil also
> seems to sustain that POV.
That's not the intended point of view.
"May be nil" is a property of what the caller hands to select.
That is, the preparation for a select, before it does anything
having to do with channel communication, is to collect the
arguments, by evaluating the parameters to the communications.
The caller is allowed to set a channel to nil to disable a
case during this particular select. That's a local property
of a value computed by the caller, one that the caller has
complete control over, can inspect without races, and so on.
It's not something the caller must rely on select to do.
It's a mere convenience, so that the caller can write a
single select loop with, say, 5 cases and then turn
individual cases on and off as appropriate while going
around the loop. You can think of passing a nil c to select
as syntactic sugar for passing some channel that no one
else knows about, which would have the same effect.
"Can communicate without blocking" is, in contrast, a global property.
It depends on what other goroutines in the system are doing,
and it might change the moment after you check.
That's what select is for: it is a coherent way to check
this on many channel operations and pick one to execute.
The definition of select is completely orthogonal to channels
being closed. More importantly, it is completely orthogonal
to everything except blocking. For example, given that a select
is defined to wait for one of a collection of cases to be ready
and then execute it, a single-case select:
select {
case xxx:
yyy
}
should, since it is waiting on only a single communication, behave
the same as running the case directly:
{
xxx
yyy
}
Losing that property would, in my opinion, be a serious mistake.
It would make communications behave differently inside select
and outside select, introducing special cases for programmers
to stumble across.
It would be like
switch x {
case y:
...
}
being different from
if x == y {
...
}
>> That is, select should only be about blocking.
>
> It'll be pretty inconvenient to select on multiple choices when one of
> them may be closed earlier for whatever reason.
Yes. Don't do that. Because closing c is a signal only to receivers,
not to other senders, a goroutine should only close c once it has
established that no other goroutine will send on c in the future.
(Happy to keep talking about specific instances in the gozk or
Erlang threads.)
Russ
Gustavo Niemeyer
> case during this particular select. That's a local property
> of a value computed by the caller, one that the caller has
Sounds reasonable, and that actually addresses the use case
I was pondering about on Friday: how to purposefully take a
channel out of a select loop to avoid panicking due to close().
The answer is to nil the channel for the select case once
communication with the specific channel has knowingly been
stopped.
> The definition of select is completely orthogonal to channels
> being closed. More importantly, it is completely orthogonal
> to everything except blocking. For example, given that a select
I see what you mean. Thanks for taking the time to explain.
Albert Strasheim
On Fri, Jan 21, 2011 at 11:53 PM, Russ Cox <r...@golang.org> wrote:
> I think that you should simply not send on closed channels.
> A channel enables unidirectional communication from sender
> to receiver, just like a Unix pipe. Only a sender should close
> a channel, and only a receiver should check whether a
> channel is closed. Unlike a Unix pipe, a channel does not
> have obviously distinguished read and write ends, so it is
> easy to forget the unidirectionality, but it's still there.
It occurred to me that the compiler could actually check this if one
properly uses channel directions in your code:
package main
func main() {
ch := make(chan bool)
var chsend chan<- bool
chsend = ch
var chrecv <-chan bool
chrecv = ch
// ok
close(ch)
// ok
close(chsend)
// should not compile
close(chrecv)
// ok
_ = closed(ch)
// should not compile
_ = closed(chsend)
// ok
_ = closed(chrecv)
}
Thoughts?
Regards
Albert
roger peppe
Corey Thomasson
> i agree that closing a receive-only channel should be illegal.
Why? I've used multiple producers -> single consumer several times,
Wouldn't it still make sense to allow that consumer to close the
channel?
roger peppe
> On 27 January 2011 11:06, roger peppe <rogp...@gmail.com> wrote:
>> i agree that closing a receive-only channel should be illegal.
>
> Why? I've used multiple producers -> single consumer several times,
> Wouldn't it still make sense to allow that consumer to close the
> channel?
nope.
because then the producers will panic.
Russ Cox
> Wouldn't it still make sense to allow that consumer to close the
> channel?
Please read the rest of this thread.
This question has been answered many times over.
Russ