An old problem: lack of priority select cases
1,309 views
Skip to first unread message
T L
Aug 28, 2019, 12:06:33 PM8/28/19
to golang-nuts
The old thread: https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
Go channels are flexible, but in practice, I often encountered some situations in which channel are hard to use.
Given an example:
import "math/rand"
type Producer struct {
data chan int
closed chan struct{}
}
func NewProducer() *Producer {
p := &Producer {
data: make(chan int),
closed: make(chan struct{}),
}
go p.run()
return p
}
func (p *Produce) Stream() chan int {
return p.data
}
func (p *Producer) run() {
for {
// If non-blocking cases are selected by their appearance order,
// then the following slect block is a perfect use.
select {
case(0) <-p.closed: return
case p.data <- rand.Int():
}
}
}
func (p *Produce) Clsoe() {
close(p.closed)
close(p.data)
}
func main() {
p := NewProducer()
for n := p.Stream() {
// use n ...
}
}
type Producer struct {
data chan int
closed chan struct{}
}
func NewProducer() *Producer {
p := &Producer {
data: make(chan int),
closed: make(chan struct{}),
}
go p.run()
return p
}
func (p *Produce) Stream() chan int {
return p.data
}
func (p *Producer) run() {
for {
// If non-blocking cases are selected by their appearance order,
// then the following slect block is a perfect use.
select {
case(0) <-p.closed: return
case p.data <- rand.Int():
}
}
}
func (p *Produce) Clsoe() {
close(p.closed)
close(p.data)
}
func main() {
p := NewProducer()
for n := p.Stream() {
// use n ...
}
}
If the first case in the select block in the above example has a higher priority than the second one,
then coding will be much happier for the use cases like the above one.
In short, the above use case requires:
* for receivers, data streaming end is notified by the close of a channel.
* for senders, data will never be sent to closed channel.
* for senders, data will never be sent to closed channel.
But, as Go 1 doesn't support priority select cases, it is much tedious to implement the code
satisfying the above listed requirements. The final implementation is often very ugly and inefficient.
Does anyone else also experience the pain?
Robert Engels
Aug 28, 2019, 12:36:56 PM8/28/19
to T L, golang-nuts
That's inherently racy - since between when the runtime determines the channel is OK for writing, another routine can close the channel before the write is attempted - so "priorities" won't help.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e3015cd8-c2ec-479e-927d-b9ad762d277e%40googlegroups.com.
T L
Aug 28, 2019, 12:43:33 PM8/28/19
to golang-nuts
On Wednesday, August 28, 2019 at 12:36:56 PM UTC-4, Robert Engels wrote:
That's inherently racy - since between when the runtime determines the channel is OK for writing, another routine can close the channel before the write is attempted - so "priorities" won't help.
I understand this. What I mean is it would be great to support priority select cases.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
Leo Lara
Aug 28, 2019, 1:06:07 PM8/28/19
to golang-nuts
This is connected with my article: https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2
I think there I show it is possible to workaround that limitation using standard Go tools. Of course, the code would be simple with priority select, but also perhaps select would become less efficient.
Robert Engels
Aug 28, 2019, 1:12:10 PM8/28/19
to T L, golang-nuts
And what I was trying to say is that request is inherently racy.
You can already do priority selects. see https://play.golang.org/p/58FfsKIivSr as a way to do it - more realistic though to use buffered channels. Pretty easy to wrap this in a function that takes N channels.
-----Original Message-----
From: T L
Sent: Aug 28, 2019 11:43 AM
To: golang-nuts
Subject: Re: [go-nuts] An old problem: lack of priority select cases
On Wednesday, August 28, 2019 at 12:36:56 PM UTC-4, Robert Engels wrote:That's inherently racy - since between when the runtime determines the channel is OK for writing, another routine can close the channel before the write is attempted - so "priorities" won't help.I understand this. What I mean is it would be great to support priority select cases.
-----Original Message-----
From: T L
Sent: Aug 28, 2019 11:06 AM
To: golang-nuts
Subject: [go-nuts] An old problem: lack of priority select cases
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e3015cd8-c2ec-479e-927d-b9ad762d277e%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0f753567-eaa8-4d1d-9db1-a3f382f59216%40googlegroups.com.
Robert Engels
Aug 28, 2019, 1:14:09 PM8/28/19
to Leo Lara, golang-nuts
Reading the article, why not just wrap the write function in one that uses panic/recover, since the write is expected to panic if the channel is closed.
-----Original Message-----
From: Leo Lara
Sent: Aug 28, 2019 11:24 AM
To: golang-nuts
Subject: [go-nuts] Re: An old problem: lack of priority select cases
This is connected with my article: https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2
I think there I show it is possible to workaround that limitation using standard Go tools. Of course, the code would be simple with priority select, but also perhaps select would become less efficient.
On Wednesday, August 28, 2019 at 6:06:33 PM UTC+2, T L wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b284f880-034a-4721-8686-ef48d3e2c14c%40googlegroups.com.
Robert Engels
Aug 28, 2019, 1:17:24 PM8/28/19
to Leo Lara, golang-nuts
A better solution is to wrap the writes using a RWLock, grab the read lock for writing, and the Write lock for closing. Pretty simple.
Just encapsulate it all in a MultiWriterChannel struct - generics would help here :)
T L
Aug 28, 2019, 1:30:09 PM8/28/19
to golang-nuts
On Wednesday, August 28, 2019 at 1:14:09 PM UTC-4, Robert Engels wrote:
Reading the article, why not just wrap the write function in one that uses panic/recover, since the write is expected to panic if the channel is closed.
Using panic/recover is a way, but it is ugly.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
T L
Aug 28, 2019, 1:31:56 PM8/28/19
to golang-nuts
On Wednesday, August 28, 2019 at 1:06:07 PM UTC-4, Leo Lara wrote:
This is connected with my article: https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2I think there I show it is possible to workaround that limitation using standard Go tools. Of course, the code would be simple with priority select, but also perhaps select would become less efficient.
Aha, multiple writers, which will be much more complex (if priority select cases are not supported).
T L
Aug 28, 2019, 1:37:52 PM8/28/19
to golang-nuts
On Wednesday, August 28, 2019 at 1:12:10 PM UTC-4, Robert Engels wrote:
And what I was trying to say is that request is inherently racy.You can already do priority selects. see https://play.golang.org/p/58FfsKIivSr as a way to do it - more realistic though to use buffered channels. Pretty easy to wrap this in a function that takes N channels.
This is not the priority I expected. Could you make an example based on my code in the first comment? so that I can understand it easily.
In fact, I have verbose version which doesn't need priority select cases.
It is an acceptable solution. But there are many other more complex cases, such as multiple senders.
I haven't found an acceptable solution for such complex cases.
import "math/rand"
type Producer struct {
data chan int
type Producer struct {
data chan int
closing chan struct{}
canSafeClose chan struct{}
canSafeClose chan struct{}
}
func NewProducer() *Producer {
p := &Producer {
data: make(chan int),
closing: make(chan struct{}),
canSafeClose: make(chan struct{}),
canSafeClose: make(chan struct{}),
}
go p.run()
return p
}
func (p *Produce) Stream() chan int {
return p.data
}
func (p *Producer) run() {
for {
select {
case <-p.closing:
close(p.canSafeClose)
return
default:
}
select {
case <-p.closing:
close(p.canSafeClose)
case <-p.closing:
close(p.canSafeClose)
return
default:
}
select {
case <-p.closing:
close(p.canSafeClose)
return
case p.data <- rand.Int():
}
}
}
func (p *Producer) Close() {
close(p.closed)
<-p.canSafeClose
close(p.closed)
<-p.canSafeClose
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0f753567-eaa8-4d1d-9db1-a3f382f59216%40googlegroups.com.
Robert Engels
Aug 28, 2019, 1:49:51 PM8/28/19
to T L, golang-nuts
As I said in another email, using a RWMutex makes the multiple senders problem easily solvable. Grab the Read lock when writing, and the Write lock when closing. Set a 'closed' flag during Close that is checked during the write.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e8e6ceff-1a3e-4c69-93cc-4f5013a3e3c3%40googlegroups.com.
T L
Aug 28, 2019, 2:11:51 PM8/28/19
to golang-nuts
On Wednesday, August 28, 2019 at 1:49:51 PM UTC-4, Robert Engels wrote:
As I said in another email, using a RWMutex makes the multiple senders problem easily solvable. Grab the Read lock when writing, and the Write lock when closing. Set a 'closed' flag during Close that is checked during the write.
Could you show a piece of code for easily understanding?
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e8e6ceff-1a3e-4c69-93cc-4f5013a3e3c3%40googlegroups.com.
Leo Lara
Aug 28, 2019, 2:27:36 PM8/28/19
to golang-nuts
Hi Robert,
From the article: """To bound more the problem, in my case, you control the writers but not the readers"""
So what I was trying to do was to be able to close, with mutiple writers, while being transparent for the readers. The readers only need to read as usual form the channel.
For example, if you want to write a library where the user just reads from a channel, this is an approach I found where the user of the lirbary deos nto have to do anything special. Of course, there might be another solution, but if you need to modify the reader we are talking about a different problem.
Cheers!!
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
Leo Lara
Aug 28, 2019, 2:29:15 PM8/28/19
to golang-nuts
I do not think priority select is *necessary*, it could be a nice addition if the performance does not change.
Robert Engels
Aug 28, 2019, 3:45:17 PM8/28/19
to T L, golang-nuts
I will write and post this evening.
-----Original Message-----
From: T L
Sent: Aug 28, 2019 1:11 PM
To: golang-nuts
Subject: Re: [go-nuts] An old problem: lack of priority select cases
On Wednesday, August 28, 2019 at 1:49:51 PM UTC-4, Robert Engels wrote:As I said in another email, using a RWMutex makes the multiple senders problem easily solvable. Grab the Read lock when writing, and the Write lock when closing. Set a 'closed' flag during Close that is checked during the write.Could you show a piece of code for easily understanding?
-----Original Message-----
From: T L
Sent: Aug 28, 2019 12:37 PM
To: golang-nuts
Subject: Re: [go-nuts] An old problem: lack of priority select cases
On Wednesday, August 28, 2019 at 1:12:10 PM UTC-4, Robert Engels wrote:And what I was trying to say is that request is inherently racy.You can already do priority selects. see https://play.golang.org/p/58FfsKIivSr as a way to do it - more realistic though to use buffered channels. Pretty easy to wrap this in a function that takes N channels.This is not the priority I expected. Could you make an example based on my code in the first comment? so that I can understand it easily.In fact, I have verbose version which doesn't need priority select cases.It is an acceptable solution. But there are many other more complex cases, such as multiple senders.I haven't found an acceptable solution for such complex cases.
import "math/rand"
type Producer struct {
data chan int
closing chan struct{}
canSafeClose chan struct{}
}
func NewProducer() *Producer {
p := &Producer {
data: make(chan int),
closing: make(chan struct{}),
canSafeClose: make(chan struct{}),
}
go p.run()
return p
}
func (p *Produce) Stream() chan int {
return p.data
}
func (p *Producer) run() {
for {
select {
case <-p.closing:
close(p.canSafeClose)
return
default:
}
select {
case <-p.closing:
close(p.canSafeClose)
return
case p.data <- rand.Int():
}
}
}
func (p *Producer) Close() {
close(p.closed)
<-p.canSafeClose
close(p.data)
}
func main() {
p := NewProducer()
for n := p.Stream() {
// use n ...
}
}
-----Original Message-----
From: T L
Sent: Aug 28, 2019 11:43 AM
To: golang-nuts
Subject: Re: [go-nuts] An old problem: lack of priority select cases
On Wednesday, August 28, 2019 at 12:36:56 PM UTC-4, Robert Engels wrote:That's inherently racy - since between when the runtime determines the channel is OK for writing, another routine can close the channel before the write is attempted - so "priorities" won't help.I understand this. What I mean is it would be great to support priority select cases.
-----Original Message-----
From: T L
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e3015cd8-c2ec-479e-927d-b9ad762d277e%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0f753567-eaa8-4d1d-9db1-a3f382f59216%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e8e6ceff-1a3e-4c69-93cc-4f5013a3e3c3%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2f9acd65-682a-45e1-bdd8-ba4036792d68%40googlegroups.com.
Marcin Romaszewicz
Aug 28, 2019, 4:37:52 PM8/28/19
to Leo Lara, golang-nuts
Think of a channel as existing for the lifetime of a particular data stream, and not have it be associated with either producer or consumer. Here's an example:
https://play.golang.org/p/aEAXXtz2X1g
https://play.golang.org/p/aEAXXtz2X1g
The channel here is closed after all producers have exited, and all consumers continue to run until the channel is drained of data.
The producers are managed by something somewhere in your code - and that is the scope at which it makes sense to create channel ownership. I've used a waitgroup to ensure that the channel is closed after all producers exit, but you can use whatever barrier construct you want.
Even if you must have a channel per producer, you can safely close the producer side, without notifying the downstream about this. The example early in the thread uses multiple channels, with one channel being used to signal that the producers should exit. Channels aren't really the right model for this, you want a thread safe flag of some sort. For example:
var exitFlag uint64
func producer(chan data int, wg *sync.WaitGroup) {
defer wg.Done()
for {
shouldExit := atomic.LoadUint64(&exitFlag)
if shouldExit == 1 {
return
}
chan <- rand.Intn(100)
}
}
Here's 10 producers and 3 consumers sharing a channel and closing it safely upon receiving an exit flag:
https://play.golang.org/p/RiKi1PGVSvF
https://play.golang.org/p/RiKi1PGVSvF
-- Marcin
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aeb38a0a-8268-42d7-a8eb-ce5ef01c5380%40googlegroups.com.
Michel Levieux
Aug 28, 2019, 5:10:18 PM8/28/19
to golang-nuts
One should also be careful regarding the conceptual demands he or she is making.
Having a shared resource (that is complex enough that it cannot be atomically accessed or modified) means essentially that "having multiple writers being transparent to the readers", fundamentally, is not possible.
From the moment itself when such a resource is shared, there must be some sort of mecanism (that one using resources atomically usable) that ensures the integrity of it.
From the moment itself when such a resource is shared, there must be some sort of mecanism (that one using resources atomically usable) that ensures the integrity of it.
Maybe what you're talking about is having it transparent in terms of code, in which case we both agree, but if you're looking for something transparent in essence, as in performance, logical construction and all the rest, I think there is a misunderstanding here: even if it was added in the language, there would be many many things going on under the hood, as it is already (and cannot really be otherwise) for channel use alone.
As for the priority using selects, I think it's more of something to be dealt with on the "user-side". There are many kinds of priority in general, and trying to implement something in the language itself would IMO either be too specific compared to the nessecary time to do so or it would probably have a huge overhead on the "classical' use case of the select construct.
+ the fact that it is apparently already possible using RWMutexes.
As for the priority using selects, I think it's more of something to be dealt with on the "user-side". There are many kinds of priority in general, and trying to implement something in the language itself would IMO either be too specific compared to the nessecary time to do so or it would probably have a huge overhead on the "classical' use case of the select construct.
+ the fact that it is apparently already possible using RWMutexes.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvcUhUvrZb_8AKYWj0A%2Bqd5LKBPmbz-RVBb%3DJn_gNZE6w%40mail.gmail.com.
robert engels
Aug 28, 2019, 10:05:06 PM8/28/19
to Michel Levieux, golang-nuts
Here is a version using RWLock https://play.golang.org/p/YOwuYFiqtlf
It won’t run correctly in the playground because it terminates when all routines are asleep - which happens during the test (not sure why it does this, as sleeping is different than a deadlock).
It is probably less efficient, and less orderly than the other example using WaitGroup but you get the idea I hope. It forcibly terminates the writers before they complete by design.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANgi337s1Low95QvqJUAOTsqcVji7uMQ_jr%3DFftpt2uMz5_XSQ%40mail.gmail.com.
Leo Lara
Aug 29, 2019, 1:02:19 AM8/29/19
to golang-nuts
Hi Michael,
The way I always have seen "transparent" used in software engineering is, that the user of something (lirabry, service, framework, etc) can use it without knowing its internal details, just normally, and the magic is done in the thing used.
To in terms of the problem I was trying to solve: being able to close a channel that is written by serveral goroutines, transparent property is that the reader or readers of the channel they just do that, without knowing about something else.
I think this is established in the article beginnings here: https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2
Regarding the priority select, I agree.
I think this is established in the article beginnings here: https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2
Regarding the priority select, I agree.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aeb38a0a-8268-42d7-a8eb-ce5ef01c5380%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvcUhUvrZb_8AKYWj0A%2Bqd5LKBPmbz-RVBb%3DJn_gNZE6w%40mail.gmail.com.
Leo Lara
Aug 29, 2019, 1:13:34 AM8/29/19
to golang-nuts
Hi Robert,
To put you in context, it all started when I read https://go101.org/article/channel-closing.html , that said that it is impossible or at least you shouldn't close a channel that is being written by several goroutines. Then I wrote this article with my solution https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2 also in https://medium.com/@leolara/closing-a-go-channel-written-by-several-goroutines-eba3a6c9404b I then created this issue https://github.com/go101/go101/issues/132 and from there this topic was created by T L.
Your example does not have several goruitnes writing so I think it is a different problem. Perhaps that simple lock would work with several goroutines, but I think there would be more contention with this lock.
Anyway, I think I have already an optimisation to my code, I think using a RW lock, if I put the "Add(1)" in a read lock and the wait in a Write lock it might work better. The race condition that my lock prevents is only related when an "Add" and a "Wait" run concurrently, several "Add" can run concurrently.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aeb38a0a-8268-42d7-a8eb-ce5ef01c5380%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvcUhUvrZb_8AKYWj0A%2Bqd5LKBPmbz-RVBb%3DJn_gNZE6w%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANgi337s1Low95QvqJUAOTsqcVji7uMQ_jr%3DFftpt2uMz5_XSQ%40mail.gmail.com.
Robert Engels
Aug 29, 2019, 7:02:26 AM8/29/19
to Leo Lara, golang-nuts
My example does have several go routines writing simultaneously. You should read up on how a RWLock works. You can change it to a buffered channel for higher concurrency.
The Sleep(1) in the producer is only to add some delay to demonstrate it gets terminated before the desired number of iterations.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e5f37679-bdfb-4da3-854e-fea35cf82cb7%40googlegroups.com.
T L
Aug 29, 2019, 7:25:47 PM8/29/19
to golang-nuts
On Wednesday, August 28, 2019 at 10:05:06 PM UTC-4, robert engels wrote:
Here is a version using RWLock https://play.golang.org/p/YOwuYFiqtlf
Doesn't the Read method need to be guarded by the reader lock?
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aeb38a0a-8268-42d7-a8eb-ce5ef01c5380%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvcUhUvrZb_8AKYWj0A%2Bqd5LKBPmbz-RVBb%3DJn_gNZE6w%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANgi337s1Low95QvqJUAOTsqcVji7uMQ_jr%3DFftpt2uMz5_XSQ%40mail.gmail.com.
Robert Engels
Aug 29, 2019, 8:59:10 PM8/29/19
to T L, golang-nuts
Oops. You are right. The original used two different methods Closed() and Read() and when I refactored I forgot to add the Read lock to the Read(). That's why you always have code reviews...
-----Original Message-----
From: T L
Sent: Aug 29, 2019 6:25 PM
To: golang-nuts
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a5cff3f8-cc1c-4719-9f2f-7b9c31086f6a%40googlegroups.com.
T L
Aug 30, 2019, 9:50:00 AM8/30/19
to golang-nuts
@Robert
I think there is a difference between the code of @Leo and you.
In you code, the Wirte/Read/Close are all possible to block for ever.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a5cff3f8-cc1c-4719-9f2f-7b9c31086f6a%40googlegroups.com.
Robert Engels
Aug 30, 2019, 10:35:29 AM8/30/19
to T L, golang-nuts
I don't think so. Why do you think that is the case? The RWLock is "fair" in the sense that once the 'closer' attempts to get the lock, it is guaranteed to get it (as the code is structured) - the subsequent readers will queue behind the "writer = closer".
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4bf95bb2-33ed-46ed-9436-48df1072914f%40googlegroups.com.
T L
Aug 30, 2019, 12:13:19 PM8/30/19
to golang-nuts
On Friday, August 30, 2019 at 10:35:29 AM UTC-4, Robert Engels wrote:
I don't think so. Why do you think that is the case? The RWLock is "fair" in the sense that once the 'closer' attempts to get the lock, it is guaranteed to get it (as the code is structured) - the subsequent readers will queue behind the "writer = closer".
How about unknown/random number of senders and readers?
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4bf95bb2-33ed-46ed-9436-48df1072914f%40googlegroups.com.
Robert Engels
Aug 30, 2019, 12:39:41 PM8/30/19
to T L, golang-nuts
Makes no difference in the code I posted.... as long as they all use the same MultiWriterChannel. In fact, others can be late started, as they will fail fast if the channel is already closed.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d155dcf6-7c01-4f7e-b408-eef9903cd837%40googlegroups.com.
T L
Aug 30, 2019, 1:05:14 PM8/30/19
to golang-nuts
On Friday, August 30, 2019 at 12:39:41 PM UTC-4, Robert Engels wrote:
Makes no difference in the code I posted.... as long as they all use the same MultiWriterChannel. In fact, others can be late started, as they will fail fast if the channel is already closed.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d155dcf6-7c01-4f7e-b408-eef9903cd837%40googlegroups.com.
Robert Engels
Aug 30, 2019, 1:40:33 PM8/30/19
to T L, golang-nuts
You changed the Read() method incorrectly - it should be using the Read lock, not the Write lock.
Still, as I pointed out when I posted it, Play has a problem where it aborts if all routines are sleeping (not just blocked), so you need to run it locally.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/74653e49-f374-4ac8-998e-fd874cdf6bd4%40googlegroups.com.
roger peppe
Aug 31, 2019, 2:32:26 AM8/31/19
to T L, golang-nuts
The reason you're wanting priority select is because you are shutting down the data channel preemptively, but you can wait for an acknowledgement from the run goroutine instead:
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e3015cd8-c2ec-479e-927d-b9ad762d277e%40googlegroups.com.
T L
Aug 31, 2019, 3:40:20 AM8/31/19
to golang-nuts
On Friday, August 30, 2019 at 1:40:33 PM UTC-4, Robert Engels wrote:
You changed the Read() method incorrectly - it should be using the Read lock, not the Write lock.Still, as I pointed out when I posted it, Play has a problem where it aborts if all routines are sleeping (not just blocked), so you need to run it locally.
My fault. But it doesn't matter, for the Read method is never called (I commented it off).
It also crash locally for all goroutines are blocked.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/74653e49-f374-4ac8-998e-fd874cdf6bd4%40googlegroups.com.
T L
Aug 31, 2019, 4:04:29 AM8/31/19
to golang-nuts
On Saturday, August 31, 2019 at 2:32:26 AM UTC-4, rog wrote:
The reason you're wanting priority select is because you are shutting down the data channel preemptively, but you can wait for an acknowledgement from the run goroutine instead:
Your solution is clever. The Close method can be called multiple time safely.
Is there such a beautiful solution for multiple senders?
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
Jesper Louis Andersen
Aug 31, 2019, 4:53:26 AM8/31/19
to Leo Lara, golang-nuts
On Thu, Aug 29, 2019 at 7:02 AM Leo Lara <l...@leopoldolara.com> wrote:
Hi Michael,The way I always have seen "transparent" used in software engineering is, that the user of something (lirabry, service, framework, etc) can use it without knowing its internal details, just normally, and the magic is done in the thing used.
People use this in the opposite form at times. That is, a transparent data structure is one where you know its internal representation (and can rely on that in your part of the program). In contrast an opaque data structure is abstractly encapsulated: even if you know its internals, you cannot get at it. Thus the latter is the former and the former is the latter compared to your use.
T L
Aug 31, 2019, 5:02:13 AM8/31/19
to golang-nuts
On Saturday, August 31, 2019 at 4:04:29 AM UTC-4, T L wrote:
On Saturday, August 31, 2019 at 2:32:26 AM UTC-4, rog wrote:The reason you're wanting priority select is because you are shutting down the data channel preemptively, but you can wait for an acknowledgement from the run goroutine instead:Your solution is clever. The Close method can be called multiple time safely.Is there such a beautiful solution for multiple senders?
Translating a multi-senders problem to a single sender problem is the only way I can get:
Robert Engels
Aug 31, 2019, 8:24:31 AM8/31/19
to T L, golang-nuts
If you comment out the read method then all threads will block. That is the the behavior of an unbuffered channel - a writer blocks until a reader is ready. Which is why you always need a valid reader running. Unless the channel is closed and then the writer will panic.
The code I provided is valid.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3d75a129-efee-402f-aafa-9fe76af4e789%40googlegroups.com.
T L
Aug 31, 2019, 8:50:36 AM8/31/19
to golang-nuts
On Saturday, August 31, 2019 at 8:24:31 AM UTC-4, Robert Engels wrote:
If you comment out the read method then all threads will block. That is the the behavior of an unbuffered channel - a writer blocks until a reader is ready. Which is why you always need a valid reader running. Unless the channel is closed and then the writer will panic.The code I provided is valid.
In fact, if I comment out the write instead read part, the code will also crash on all goroutines are blocked.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3d75a129-efee-402f-aafa-9fe76af4e789%40googlegroups.com.
Robert Engels
Aug 31, 2019, 9:49:56 PM8/31/19
to T L, golang-nuts
Yes, that is why the original code did not use a lock on the read but the read of the flag was wrong. The version I posted in the other thread works fine locally. time.Sleep() has problems in the playground
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/dfdf8905-f740-434c-a293-d801de4f71dc%40googlegroups.com.
T L
Aug 31, 2019, 11:55:51 PM8/31/19
to golang-nuts
On Saturday, August 31, 2019 at 9:49:56 PM UTC-4, Robert Engels wrote:
Yes, that is why the original code did not use a lock on the read but the read of the flag was wrong. The version I posted in the other thread works fine locally. time.Sleep() has problems in the playground
You mean this one: https://play.golang.org/p/JRSEPU3Uf17 ?
No no, it is not a good ideas to use mutex in write but atomic in read to avoid concurrently accessing the same value.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/dfdf8905-f740-434c-a293-d801de4f71dc%40googlegroups.com.
roger peppe
Sep 1, 2019, 4:35:10 AM9/1/19
to T L, golang-nuts
On Sat, 31 Aug 2019 at 10:02, T L <tapi...@gmail.com> wrote:
On Saturday, August 31, 2019 at 4:04:29 AM UTC-4, T L wrote:
On Saturday, August 31, 2019 at 2:32:26 AM UTC-4, rog wrote:The reason you're wanting priority select is because you are shutting down the data channel preemptively, but you can wait for an acknowledgement from the run goroutine instead:Your solution is clever. The Close method can be called multiple time safely.Is there such a beautiful solution for multiple senders?Translating a multi-senders problem to a single sender problem is the only way I can get:
The answer really depends on what you're actually trying to do. What are the multiple senders? What's the actual problem you're trying to solve?
I'd fairly sure you'll be able do what you want without requiring a prioritised select, which is what this thread was about.
cheers,
rog.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e239c78f-61fc-4238-aa5d-f776cb9aec03%40googlegroups.com.
T L
Sep 1, 2019, 6:02:52 AM9/1/19
to golang-nuts
On Sunday, September 1, 2019 at 4:35:10 AM UTC-4, rog wrote:
On Sat, 31 Aug 2019 at 10:02, T L <tapi...@gmail.com> wrote:
On Saturday, August 31, 2019 at 4:04:29 AM UTC-4, T L wrote:
On Saturday, August 31, 2019 at 2:32:26 AM UTC-4, rog wrote:The reason you're wanting priority select is because you are shutting down the data channel preemptively, but you can wait for an acknowledgement from the run goroutine instead:Your solution is clever. The Close method can be called multiple time safely.Is there such a beautiful solution for multiple senders?Translating a multi-senders problem to a single sender problem is the only way I can get:The answer really depends on what you're actually trying to do. What are the multiple senders? What's the actual problem you're trying to solve?I'd fairly sure you'll be able do what you want without requiring a prioritised select, which is what this thread was about.cheers,rog.
Yes, there are ways to handle the problem of uncertain number of senders, but there are no simple ways.
A mechanism must be designed to avoid any sender writing to a closed channel.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e239c78f-61fc-4238-aa5d-f776cb9aec03%40googlegroups.com.
Robert Engels
Sep 1, 2019, 7:42:38 AM9/1/19
to T L, golang-nuts
That is incorrect. The atomic operations must exhibit the same happens before relationships as the mutex. If the mutex flushes the related cache lines the atomic load will pick it up.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ffd06e6e-fb00-400e-ad5f-5f23d27025f7%40googlegroups.com.
T L
Sep 1, 2019, 10:46:34 AM9/1/19
to golang-nuts
This is not true, at least no Go official documentation admits this.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ffd06e6e-fb00-400e-ad5f-5f23d27025f7%40googlegroups.com.
Robert Engels
Sep 1, 2019, 1:30:30 PM9/1/19
to T L, golang-nuts
The memory model is pretty unspecified but they’re working on it. As a defacto behavior it is pretty hard to have what I stated not be the case.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/dd5a26b4-b080-4dd1-801b-ff59eccf9940%40googlegroups.com.
T L
Sep 1, 2019, 1:55:30 PM9/1/19
to golang-nuts
You can simply validate it by run: go run -race main.go
for you program: https://play.golang.org/p/JRSEPU3Uf17
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/dd5a26b4-b080-4dd1-801b-ff59eccf9940%40googlegroups.com.
Albert Tedja
Sep 1, 2019, 2:05:49 PM9/1/19
to golang-nuts
This is something I ran into a while back, and made a library for it, though, I prefer not to spam the mailing list. Feel free to send me a dm and I'll send you a github link if you are interested.
Robert Engels
Sep 1, 2019, 3:13:18 PM9/1/19
to T L, golang-nuts
Not true. The race detector does not detect certain cases and can overreport. One the memory model becomes fully specified it may not be the case but if would be very hard to implement something that didn’t work as I’ve said
Still you can just as easily change the other two accesses to be atomic but it’s not needed.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/fc0de346-f93e-4458-b301-1c893d24efa3%40googlegroups.com.
T L
Sep 2, 2019, 9:36:29 AM9/2/19
to golang-nuts
On Sunday, September 1, 2019 at 2:05:49 PM UTC-4, Albert Tedja wrote:
This is something I ran into a while back, and made a library for it, though, I prefer not to spam the mailing list. Feel free to send me a dm and I'll send you a github link if you are interested.
It would be great if you can share the library here. I'm surely interested. And I think many other gophers also have interests. :)
Leo Lara
Sep 4, 2019, 5:44:11 AM9/4/19
to golang-nuts
Hi Marcin,
I think https://play.golang.org/p/RiKi1PGVSvF is basically what I do with atomic operations in my blog post https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2 in the section "Some experiments", and then using the wait group as I say later in the section "The solution involves wait groups",
I think it is more constrained than my final solution because:
+ You need to make sure that all producers have started (run Add(1)) before you call Close. You would need to use a mutex around Add and Wait to solve this, I think now that a RWLock have the read arround Add(1) and the write around Wait could be better.
+ All writing goroutines will stay blocked until a reader reads all, using a closing channel is better because you can use select and they could unblock without a reader.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b284f880-034a-4721-8686-ef48d3e2c14c%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aeb38a0a-8268-42d7-a8eb-ce5ef01c5380%40googlegroups.com.
Leo Lara
Sep 4, 2019, 5:55:36 AM9/4/19
to golang-nuts
You should read up on how a RWLock works.
I am not going to answer to that ;-)
About this code:
1. I wouldn't say that there are several goroutines writing, as there are several goroutines block in a mutex. Some might find this not important, although there are some people that have looked into this and think it is less efficient https://opensource.com/article/18/7/locks-versus-channels-concurrent-go ; I think it has to do on how the scheduler changes to another goroutine when the goroutine is lock on a channel.
2. More importantly IMHO, in your example the reader has to be aware that there is a lock. Hence, it is not "transparent" in the sense that the reader has to share something with the writer
The Sleep(1) in the producer is only to add some delay to demonstrate it gets terminated before the desired number of iterations.Hi Robert,To put you in context, it all started when I read https://go101.org/article/channel-closing.html , that said that it is impossible or at least you shouldn't close a channel that is being written by several goroutines. Then I wrote this article with my solution https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2 also in https://medium.com/@leolara/closing-a-go-channel-written-by-several-goroutines-eba3a6c9404b I then created this issue https://github.com/go101/go101/issues/132 and from there this topic was created by T L.Your example does not have several goruitnes writing so I think it is a different problem. Perhaps that simple lock would work with several goroutines, but I think there would be more contention with this lock.Anyway, I think I have already an optimisation to my code, I think using a RW lock, if I put the "Add(1)" in a read lock and the wait in a Write lock it might work better. The race condition that my lock prevents is only related when an "Add" and a "Wait" run concurrently, several "Add" can run concurrently.
On Thursday, August 29, 2019 at 4:05:06 AM UTC+2, robert engels wrote:
Here is a version using RWLock https://play.golang.org/p/YOwuYFiqtlf
It won’t run correctly in the playground because it terminates when all routines are asleep - which happens during the test (not sure why it does this, as sleeping is different than a deadlock).
It is probably less efficient, and less orderly than the other example using WaitGroup but you get the idea I hope. It forcibly terminates the writers before they complete by design.
On Aug 28, 2019, at 4:09 PM, Michel Levieux <m.le...@capitaldata.fr> wrote:
One should also be careful regarding the conceptual demands he or she is making.Having a shared resource (that is complex enough that it cannot be atomically accessed or modified) means essentially that "having multiple writers being transparent to the readers", fundamentally, is not possible.
From the moment itself when such a resource is shared, there must be some sort of mecanism (that one using resources atomically usable) that ensures the integrity of it.Maybe what you're talking about is having it transparent in terms of code, in which case we both agree, but if you're looking for something transparent in essence, as in performance, logical construction and all the rest, I think there is a misunderstanding here: even if it was added in the language, there would be many many things going on under the hood, as it is already (and cannot really be otherwise) for channel use alone.
As for the priority using selects, I think it's more of something to be dealt with on the "user-side". There are many kinds of priority in general, and trying to implement something in the language itself would IMO either be too specific compared to the nessecary time to do so or it would probably have a huge overhead on the "classical' use case of the select construct.
+ the fact that it is apparently already possible using RWMutexes.
Le mer. 28 août 2019 à 22:37, Marcin Romaszewicz <mar...@gmail.com> a écrit :
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvcUhUvrZb_8AKYWj0A%2Bqd5LKBPmbz-RVBb%3DJn_gNZE6w%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANgi337s1Low95QvqJUAOTsqcVji7uMQ_jr%3DFftpt2uMz5_XSQ%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e5f37679-bdfb-4da3-854e-fea35cf82cb7%40googlegroups.com.
Leo Lara
Sep 4, 2019, 6:13:32 AM9/4/19
to golang-nuts
I agree mostly with T L, and I say in the article in the section "Testing for race conditions" https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2 . The race detector will detect race conditions if they happen during the execution of the tests. But, as they are race conditions, some timings and ordering will not trigger them, and hence the race detector.
I run my tests with:
go test -cpu=1,9,55,99 -race -count=1000 -failfast
and run it several times, while running other programs in the background to create more "scheduling noise".
Many times in my original code I had race conditions but it was not detected many of the runs of the tests. So I have validated this empirically.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/fc0de346-f93e-4458-b301-1c893d24efa3%40googlegroups.com.
Leo Lara
Sep 4, 2019, 6:32:51 AM9/4/19
to golang-nuts
Hi Jasper,
Do you have some literature with that use. I honestly have googled: https://www.google.com/search?rlz=1C5CHFA_enES851ES851&biw=1280&bih=698&ei=v4lvXcnpE4_gUbmTiLgK&q=transparency+component+software+engineering&oq=transparency+component+software+engineering&gs_l=psy-ab.3...10733.11100..11571...0.2..0.196.549.0j3......0....1..gws-wiz.......0i71j35i304i39.sGAC4ozKBlY&ved=0ahUKEwiJxe788rbkAhUPcBQKHbkJAqc4ChDh1QMICw&uact=5
and the first result I got https://en.wikipedia.org/wiki/Transparency_(human%E2%80%93computer_interaction)
It is kind of what I said:
"""The purpose is to shield from change all systems (or human users) on the other end of the interface. Confusingly, the term refers to overall invisibility of the component, it does not refer to visibility of component's internals (as in white box or open system)""""
Robert Engels
Sep 4, 2019, 7:20:59 AM9/4/19
to Leo Lara, golang-nuts
I will repeat this - and I meant no offense. You should read up on RWLocks because your criticism is incorrect. Multiple Go readers can hold the Read lock at the same time - so there are multiple writers adding to the channel concurrently (but the channel impl uses a lock, so it’s not strictly true, if they used a lock free queue then it would be)
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8893737c-148e-4d64-9a0c-cd72046cd0d1%40googlegroups.com.
Robert Engels
Sep 4, 2019, 7:22:55 AM9/4/19
to Leo Lara, golang-nuts
As for the readers being aware of a lock, that is not true either, as the lock is hidden by encapsulation. (The last and best version doesn’t even need the lock on the readers)
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8893737c-148e-4d64-9a0c-cd72046cd0d1%40googlegroups.com.
Leo Lara
Sep 4, 2019, 9:24:56 AM9/4/19
to Robert Engels, golang-nuts
By encapsulation? Are you talking about
https://play.golang.org/p/YOwuYFiqtlf or other version of the code?
As I think someone already said, there is a need in the method Read for a lock. And the read from the channel is done from that method. With your pattern, the user of the hypothetical component reading from the channel needs to hold a lock.
Perhaps you do not need to check of c.closed as the closed channel with signal it already with the "ok" result. In that case you wouldn't need a lock in the reader, and it would be transparent for the reader.
Another drawback of this implementations is that the writer does not unlock unless the channel is read. Perhaps in some situations this does not matter, but this also implies that the Close method will not exit until they are read, plus the problem of not being fair, that new writer goroutines could go forward while Close is blocked in the Write mutex, as you block writing on a channel while holding a read lock that increases the chances of unfairness trying to get the write lock. This is a smaller problem in mine as the lock is hold for much shorter time, and I do not block inside the read lock.
I have seen several alternatives here, but still not a critisism of why those implementations would be preferable over mine. The only thing I see in the alternatives is that they combine less different concurrent idioms. Mine needs a lock, a waitgroup and "closing" channel but it seems to have better external qualities: transparency, fairness, performance, writer can spawn any time...
Robert Engels
Sep 4, 2019, 10:06:20 AM9/4/19
to Leo Lara, golang-nuts
No, please review this: https://play.golang.org/p/sPWrNhzRXKz
There is no need to use the lock in Read()
Encapsulation in this case means, the user of MultiWriterIntChan does not know it uses locks behind the scenes, nor does it need to know.
Due to limitations of the playground, the code referenced above runs serially. The sleeps are ONLY in there to demonstrate that early termination works - they are not needed for correctness.
That the writer does not unlock unless the channel is read is a limitation of unbuffered channels in Go - you could get around this by using a select and default. Closing a channel with waiting writers will cause a panic. Your technique is incorrect in my mind because...
The Close() will read from the channel, causing other writes - that the Producer thought were processed to be discarded. This is incorrect behavior in most use cases (think if the Producer was generating log messages).
For the version I posted, every write that completes will be handled by a consumer.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAK9c6k42QraquPapY%2BXR_wH7cCLrE3-DrAEZJsbUa%3DVDa2-sFg%40mail.gmail.com.
T L
Sep 7, 2019, 5:42:48 AM9/7/19
to golang-nuts
@Leo Lara
In face, you code the Mutex+WaitGroup in your code can be replaced with a Cond var plus a counter: https://play.golang.org/p/hVLamTjHj6J
trishc...@gmail.com
Sep 12, 2019, 11:22:35 AM9/12/19
to golang-nuts
Why not just set up priority blockers against the one
T L
Sep 12, 2019, 2:51:25 PM9/12/19
to golang-nuts
On Thursday, September 12, 2019 at 11:22:35 AM UTC-4, trishc...@gmail.com wrote:
Why not just set up priority blockers against the one
What does priority blockers mean?
T L
Oct 4, 2019, 2:46:36 PM10/4/19
to golang-nuts
I just found an example in the "context" package docs:
// // Stream generates values with DoSomething and sends them to out // // until DoSomething returns an error or ctx.Done is closed. // func Stream(ctx context.Context, out chan<- Value) error { // for { // v, err := DoSomething(ctx) // if err != nil { // return err // } // select { // case <-ctx.Done(): // return ctx.Err() // case out <- v: // } // } // }
It looks the send "out <- v" still has a possibility to be executed,
even if "ctx.Done()" is closed.
But if Go supports select case priority, then this will never happen.
T L
Oct 4, 2019, 2:53:15 PM10/4/19
to golang-nuts
On Friday, October 4, 2019 at 2:46:36 PM UTC-4, T L wrote:
I just found an example in the "context" package docs:// // Stream generates values with DoSomething and sends them to out // // until DoSomething returns an error or ctx.Done is closed. // func Stream(ctx context.Context, out chan<- Value) error { // for { // v, err := DoSomething(ctx) // if err != nil { // return err // } // select { // case <-ctx.Done(): // return ctx.Err() // case out <- v: // } // } // }
It looks the send "out <- v" still has a possibility to be executed,
even if "ctx.Done()" is closed.
But if Go supports select case priority, then this will never happen.
This might be some over absolutely.
There is still a samll chance "ctx.Done()" is being closed when "out <- v" is being executed.
Should the docs be changed to:
// func Stream(ctx context.Context, out chan<- Value) error { // for { // v, err := DoSomething(ctx) // if err != nil { // return err // } // select { // case <-ctx.Done(): // return ctx.Err()
// case default:
// }
// select {
// case <-ctx.Done():
// return ctx.Err()
// case out <- v:
// }
// }
// }Robert Engels
Oct 4, 2019, 3:32:31 PM10/4/19
to T L, golang-nuts
You still are not understanding proper concurrent design. Priority select cases do not matter in the case of asynchronous external events.
On Oct 4, 2019, at 1:46 PM, T L <tapi...@gmail.com> wrote:
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com.
T L
Oct 4, 2019, 3:44:23 PM10/4/19
to golang-nuts
On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:
You still are not understanding proper concurrent design. Priority select cases do not matter in the case of asynchronous external events.
It at least avoids code verbosity and improves code quantity..
BTW, I don't understand what you said. Could you elaborate more?
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
Robert Engels
Oct 4, 2019, 4:09:09 PM10/4/19
to T L, golang-nuts
Because ctx.Done() and v being ready for read are independent events. You can not impose ordering on them unless there is an outer mutex that both events are subject to.
As an aside, this is why I think the best 'concurrent software' developers are those that have been exposed to at least some hardware design. Many programmers think in terms of 1 and 0 and everything being ordered. This is certainly not the case in hardware, nor concurrent software. (For example, in computer hardware, the clock/sync line is what is used as the outer controlling event, but still things like propagation times, etc. make even this simple statement not fully correct).
-----Original Message-----
From: T L
Sent: Oct 4, 2019 2:44 PM
To: golang-nuts
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/77fca2e9-120d-45e0-8ae9-5d24b63827bd%40googlegroups.com.
Message has been deleted
T L
Oct 4, 2019, 4:21:45 PM10/4/19
to golang-nuts
On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:
Because ctx.Done() and v being ready for read are independent events. You can not impose ordering on them unless there is an outer mutex that both events are subject to.As an aside, this is why I think the best 'concurrent software' developers are those that have been exposed to at least some hardware design. Many programmers think in terms of 1 and 0 and everything being ordered. This is certainly not the case in hardware, nor concurrent software. (For example, in computer hardware, the clock/sync line is what is used as the outer controlling event, but still things like propagation times, etc. make even this simple statement not fully correct).
Still not understand what you new saying means. ;D
Again, select case priority enables use to deduce two select cases
to one select case in coding in many scenarios.
This often leads to cleaner code, and sometimes can avoid the harm
caused by missing an essential try-receive select case.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/77fca2e9-120d-45e0-8ae9-5d24b63827bd%40googlegroups.com.
Marcin Romaszewicz
Oct 4, 2019, 4:38:36 PM10/4/19
to T L, golang-nuts
What he's trying to say is that it is pointless to order select cases in this example, because it is impossible to guarantee ordering in an asynchronous system.
You have two asynchronous data streams, ctx.Done() and "v", in that example above.
Generally, those select cases will happen one by one, as those asynchronous channels deliver data. If "v" is ready first, shortly before ctx.Done(), then v will be selected first, followed by ctx.Done(). It doesn't matter that ctx.Done() wants to be higher priority, "v" arrived first, before ctx.Done() was posted, so it's handled first. If ctx.Done() happens first, the reverse happens. The only interesting case is when both "v" and ctx.Done() are ready at the same time, which will be unlikely in practice.
If ctx.Done() and "v" happen together, so that both cases of that select statement are available simultaneously, then sure, you can order them, but this will happen very infrequently, and you still MUST handle the case where "v" happens first, very close to ctx.Done(). So, if you MUST handle this case, you don't really need to bother with statement priority, since if your code is written well, this ordering won't matter. ctx.Done() might happen while you're in the middle of handling "v", for example.
The desire to have priority to select statements presumes that you have several cases happen at the same time, which isn't how this will generally work. If you want to order things in a select, you have to change how it behaves, and order events within a particular time window. Say you write a poller which samples the event queue every second. If you do this, then sure, you can order things however you want, but the cost you pay is that second of buffering latency.
On Fri, Oct 4, 2019 at 1:19 PM T L <tapi...@gmail.com> wrote:
On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:Because ctx.Done() and v being ready for read are independent events. You can not impose ordering on them unless there is an outer mutex that both events are subject to.As an aside, this is why I think the best 'concurrent software' developers are those that have been exposed to at least some hardware design. Many programmers think in terms of 1 and 0 and everything being ordered. This is certainly not the case in hardware, nor concurrent software. (For example, in computer hardware, the clock/sync line is what is used as the outer controlling event, but still things like propagation times, etc. make even this simple statement not fully correct).
Still not understanding what you new saying. ;D
Again, select case priority enables use to deduce two select casesto one select case in coding in many scenarios.
This often leads to cleaner code, avoid can avoid the harm causedby missing a try-receive select case.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/77fca2e9-120d-45e0-8ae9-5d24b63827bd%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cf695058-93ce-4b81-99d0-3c622a63202a%40googlegroups.com.
T L
Oct 4, 2019, 4:52:19 PM10/4/19
to golang-nuts
On Friday, October 4, 2019 at 4:38:36 PM UTC-4, Marcin Romaszewicz wrote:
What he's trying to say is that it is pointless to order select cases in this example, because it is impossible to guarantee ordering in an asynchronous system.You have two asynchronous data streams, ctx.Done() and "v", in that example above.Generally, those select cases will happen one by one, as those asynchronous channels deliver data. If "v" is ready first, shortly before ctx.Done(), then v will be selected first, followed by ctx.Done(). It doesn't matter that ctx.Done() wants to be higher priority, "v" arrived first, before ctx.Done() was posted, so it's handled first. If ctx.Done() happens first, the reverse happens. The only interesting case is when both "v" and ctx.Done() are ready at the same time, which will be unlikely in practice.If ctx.Done() and "v" happen together, so that both cases of that select statement are available simultaneously, then sure, you can order them, but this will happen very infrequently, and you still MUST handle the case where "v" happens first, very close to ctx.Done(). So, if you MUST handle this case, you don't really need to bother with statement priority, since if your code is written well, this ordering won't matter. ctx.Done() might happen while you're in the middle of handling "v", for example.
This is not frequently, but not rare.
The desire to have priority to select statements presumes that you have several cases happen at the same time, which isn't how this will generally work. If you want to order things in a select, you have to change how it behaves, and order events within a particular time window. Say you write a poller which samples the event queue every second. If you do this, then sure, you can order things however you want, but the cost you pay is that second of buffering latency.
Yes, select case priority tries to always select one case of the two cases
when both the two cases are available. Without this feature, to achieve
this goal, we often need two select blocks (sorry, I mis-typed select block
as select case in my last comment).
One priority-order select block is not only cleaner than two random-order
select blocks, but also more efficient.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cf695058-93ce-4b81-99d0-3c622a63202a%40googlegroups.com.
Robert Engels
Oct 4, 2019, 5:11:18 PM10/4/19
to T L, golang-nuts
But even the two select blocks does not really make a difference. I have shared code that did this previously.
For another example, think of two channels A and B. A is the high priority channel where all events should be processed before B channel events - this is especially important considering that A events can be processed in 1 ms, and B events take 60 seconds.
So, as soon as you start processing a B event - you have delayed the processing of a "just after" A event by 60 seconds... probably not what you want. So you either need to be able to stop the processing of B, or you need to process them "concurrently" (this is a simplification of how to actually accomplish this).
This is why if closing the channel is independent of the processing of the channel, you need external synchronization and/or ordering.
Are there workloads where the A and B channels are always ready at the same time, probably... but if you ALWAYS processed A first - eventually B would run out of queue/memory space - and either deadlock or crash...
I will reiterate - your thinking on this problem is incorrect, which is why the solution you propose is of limited usefulness. Still, if you really want this, it is simple to write a function 'selectUsingPriority(A,B chan). But if you don't fully understand what I am saying, you are going to have obscure reliability problems in production if you use it.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/857fb242-ebcf-4e31-8f9f-b87738b58c1a%40googlegroups.com.
Wojciech S. Czarnecki
Oct 4, 2019, 5:40:08 PM10/4/19
to golan...@googlegroups.com
On Fri, 4 Oct 2019 13:52:19 -0700 (PDT)
T L <tapi...@gmail.com> wrote:
> One priority-order select block is not only cleaner than two random-order
> select blocks, but also more efficient.
It is neither.
T L <tapi...@gmail.com> wrote:
> One priority-order select block is not only cleaner than two random-order
> select blocks, but also more efficient.
1. It is not cleaner, because you would need to somehow denote priorities.
If you think now about "in written order", you should think of the most frequent
cases that you do not want to have all five, ten, twenty cases be serviced
on order. Not to mention that down the road someone might introduce an
unnoticeable bug for pure aesthetic reasons.
2. It is not more efficient, it is less efficient because every select operation
would need at least a "priority comparison" op, or - in general case - a sort
op, because not only two channels can be ready, it may happen that all of
them will. This would have to run for every select then, unless
introduced by a "select_with_priorities" keyword.
3. Others tried to no avail, I will too in other words:
The select switch is deliberately randomized because language creators
envisioned that otherwise someone would be "smart" and will use
ordering to impose a **dependency** where it should not be made.
Channel operations serviced by select are treated as sourced
from **independent** events. Period.
If your out <-v must be dependent of ctx.Done not being ready you
must resolve this dependency by other means. Eg. by selecting for it
in separate select before.
Hope this helps,
--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE
T L
Oct 4, 2019, 6:15:30 PM10/4/19
to golang-nuts
On Friday, October 4, 2019 at 5:40:08 PM UTC-4, ohir wrote:
On Fri, 4 Oct 2019 13:52:19 -0700 (PDT)
T L <tapi...@gmail.com> wrote:
> One priority-order select block is not only cleaner than two random-order
> select blocks, but also more efficient.
It is neither.
1. It is not cleaner, because you would need to somehow denote priorities.
If you think now about "in written order", you should think of the most frequent
cases that you do not want to have all five, ten, twenty cases be serviced
on order. Not to mention that down the road someone might introduce an
unnoticeable bug for pure aesthetic reasons.
2. It is not more efficient, it is less efficient because every select operation
would need at least a "priority comparison" op, or - in general case - a sort
op, because not only two channels can be ready, it may happen that all of
them will. This would have to run for every select then, unless
introduced by a "select_with_priorities" keyword.
select(ordered) {
case <-ctx.Done():
return ctx.Err()
case out <- v:
}
case <-ctx.Done():
return ctx.Err()
case out <- v:
}
is undoubtedly more efficient than
select(randomized) {
case <-ctx.Done():
return ctx.Err()
default:
}
select(randomized) {
case <-ctx.Done():
return ctx.Err()
case out <- v:
}
case <-ctx.Done():
return ctx.Err()
default:
}
select(randomized) {
case <-ctx.Done():
return ctx.Err()
case out <- v:
}
Not only one less try-receive operation is saved,
but also is the step 2 described in https://go101.org/article/channel.html#select-implementation saved.
3. Others tried to no avail, I will too in other words:
The select switch is deliberately randomized because language creators
envisioned that otherwise someone would be "smart" and will use
ordering to impose a **dependency** where it should not be made.
Channel operations serviced by select are treated as sourced
from **independent** events. Period.
If your out <-v must be dependent of ctx.Done not being ready you
must resolve this dependency by other means. Eg. by selecting for it
in separate select before.
It is not a must. It just tries to select ctx.Done if possible.
In fact, if the two evens are independent with each other,
there is no way to absolutely ensure the must.
Hope this helps,
You can read the first comment in this thread for another use case of priority-order select block.
Kurtis Rader
Oct 4, 2019, 6:43:27 PM10/4/19
to T L, golang-nuts
Please, drop it. You don't know what you're talking about. I've been amazed at how patient everyone has been. I think it's time to be more blunt in pointing out you don't understand concurrency and independent events.
--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0c9aaa9b-ea9d-49f5-a1b4-4f94ee0de424%40googlegroups.com.
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank
T L
Oct 5, 2019, 1:41:43 AM10/5/19
to golang-nuts
> Please, drop it. You don't know what you're talking
about. I've been amazed at how patient everyone has been. I think it's
time to be more blunt in pointing out you don't understand concurrency
and independent events.
Maybe. But you don't know I'm talking about for sure.
Wojciech S. Czarnecki
Oct 5, 2019, 4:46:58 AM10/5/19
to golan...@googlegroups.com
On Fri, 4 Oct 2019 15:42:35 -0700
Kurtis Rader <kra...@skepticism.us> wrote:
> > On Fri, Oct 4, 2019 at 3:15 PM T L <tapi...@gmail.com> wrote:
> > [...]
Kurtis Rader <kra...@skepticism.us> wrote:
> > On Fri, Oct 4, 2019 at 3:15 PM T L <tapi...@gmail.com> wrote:
> Please, drop it. You don't know what you're talking about. I've been amazed
> at how patient everyone has been.
We need this kind of discussions and arguments. It prepares us to deal with
> at how patient everyone has been.
problems of understanding that junior programmers may have,
or even to deal with our own skewed understandings of the foggy corners
of the language.
> > [...]
> I think it's time to be more blunt in pointing out you don't understand
> concurrency and independent events.
Even if mr Tapir really does not, he for sure would not be alone, so we all
> concurrency and independent events.
benefit from his questioning the Emperor's wear. So please don't be rude
to him.
TC,
Wojciech S. Czarnecki
Oct 5, 2019, 6:28:37 AM10/5/19
to golan...@googlegroups.com
On Fri, 4 Oct 2019 15:15:30 -0700 (PDT)
T L <tapi...@gmail.com> wrote:
> > If your out <-v must be dependent of ctx.Done not being ready you
> > must resolve this dependency by other means. Eg. by selecting for it
> > in separate select before.
> >
> It is not a must. It just tries to select ctx.Done if possible.
> In fact, if the two evens are independent with each other,
> there is no way to absolutely ensure the must.
If it is not a must, then introducing an order at best is void (of no use)
T L <tapi...@gmail.com> wrote:
> > If your out <-v must be dependent of ctx.Done not being ready you
> > must resolve this dependency by other means. Eg. by selecting for it
> > in separate select before.
> >
> It is not a must. It just tries to select ctx.Done if possible.
> In fact, if the two evens are independent with each other,
> there is no way to absolutely ensure the must.
at worst it makes sure for a race condition by its very presence.
For coming together events A and B that are independent:
- proceeding them in either order A, B or in order B, A should result
in the same final state as if they came in said order. Hence select's
case ordering is unnecessary.
- otherwise (that final state depends on order) we have a dependency and the
select switch, by language design, is not the right construct to write this
dependency down.
FYI, AFAIR: the "1+default" select case is optimized by the compiler, so now it is more
efficient to have "priority" written explicit than to have more than one case in
a select (what turns on all the heavy machinery of a full select block).
Hope this helps,
T L
Oct 5, 2019, 9:48:58 AM10/5/19
to golang-nuts
I do agree. I really ever benchmarked the difference, the two-select-blocks version
only consumes about 5% more CPU. So, for this use case, the main benefit of
supporting priority-case-order select block is it makes code look less verbose.
As for another use case shown in the first comment,
select {
case <-done: return
case queue <- data:
}
case <-done: return
case queue <- data:
}
In fact, the code shown in that comment is not perfect, because in other goroutines,
the following two lines
close(done)
close(queue)
close(queue)
might present a different order from they look in code. (I didn't find any official Go
documentation makes the guarantee that they will be closed by their appearance order in code.)
But if there is a way which can guarantee that "done" will always be closed before "queue"
(in the views of any goroutines), then
select(ordered) {
case <-done: return
case queue <- data:
}
case <-done: return
case queue <- data:
}
can be executed safely in multiple goroutines concurrently.
This will simplify the implementations for some situations.
Under the current select mechanism, without supporting priority-case-order select block,
programmers need to make more effort to implement the same situations.
(Maybe this is not bad, as programing with channels is really fun. ;D)
Reply all
Reply to author
Forward
0 new messages