channels and the memory model

[Sugu Sougoumarane]

I wanted to use channels to transmit/broadcast signals. Does this code violate go's memory model?

func process() {

  var val interface{}

begin:

  select {

  case val = <-ch:

    if val.IsSignal() {

      // no op: consume the signal.

      // A new signal will be generated if the condition changes.

      // Assumption: This point is reached only if nobody else is waiting.

      goto begin

    }

  default:

    // Wait for message or signal

    val = <-ch

    if val.IsSignal() {

      // Retransmit the signal

      ch <- val

      goto begin

    }

  }

}

The requirement is that all existing waits must receive the signal before the next wait-less receive, which will consume the signal.

The above code is a simplified version of what I'm trying to do. So, it may not make total sense out of context. The question, as mentioned in the assumption is:

Can the select fire while there are other goroutines still waiting on the channel?

[Dmitry Vyukov]

Sure it can. Select is not different from normal chan receives. And
the other goroutine may not even get to "val = <-ch" chan receive.
What are you trying to do?

[Andrew Gerrand]

On 6 May 2013 09:47, Sugu Sougoumarane <sou...@google.com> wrote:

Can the select fire while there are other goroutines still waiting on the channel?

I believe it can, as there's no guarantee that all the goroutines will have been scheduled by the time the signal is sent.

[Sugu Sougoumarane]

Of course. Now that you mention it, it's a dumb idea.

[Sugu Sougoumarane]

Sure it can. Select is not different from normal chan receives. And
the other goroutine may not even get to "val = <-ch" chan receive.
What are you trying to do?

What I was trying to do:

Use the channel as a resource pool (connections).

If the channel is not full, create resources on demand.

If the channel is full, subsequent requests should wait.

If a resource is discarded (channel is not full any more). Existing channels that are waiting should receive a signal.

They verify that size < cap(ch), and create resources until size == cap(ch).

I could keep the signal alive in the channel until size == cap(ch), but its usefulness wears off once there are no more goroutines waiting. So, I was hoping that the select trigger can be used to determine that no one else is waiting and drop the signal.

The next signal won't be generated until size reaches cap and drops back down below cap.

[André Moraes]

>
> What I was trying to do:
> Use the channel as a resource pool (connections).
> If the channel is not full, create resources on demand.
> If the channel is full, subsequent requests should wait.

Already solved at: http://golang.org/doc/effective_go.html#leaky_buffer

> If a resource is discarded (channel is not full any more). Existing channels
> that are waiting should receive a signal.
> They verify that size < cap(ch), and create resources until size == cap(ch).

I think you could adapt the leaky_buffer above to include those restrictions.

--
André Moraes
http://amoraes.info

[Sugu Sougoumarane]

On Mon, May 6, 2013 at 10:22 AM, André Moraes <and...@gmail.com> wrote:

>
> What I was trying to do:
> Use the channel as a resource pool (connections).
> If the channel is not full, create resources on demand.
> If the channel is full, subsequent requests should wait.

Already solved at: http://golang.org/doc/effective_go.html#leaky_buffer

 

Leaky buffer is different. It doesn't limit the number of buffers allocated. More importantly, it never waits on the channel. Waiting on a channel complicates things.

[Péter Szilágyi]

Hi Sugu,

  Are you trying to create a hard limit on the number of resources that may exist in your system at any given time? E.g. in case of network connections to have only N connections up simultaneously?

  If so, what you could do is create a buffered channel with N elements, with your producer inserting some -any- random stuff in there, the point of which is only to block it when N is reached. Whenever a resource is done (i.e. connection is handled and dropped), you remove one element from the channel and the producer(s) can continue.

  Like this: http://play.golang.org/p/qDAd3ERZmY

Cheers,

  Peter

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sugu Sougoumarane]

On Mon, May 6, 2013 at 11:19 AM, Péter Szilágyi <pet...@gmail.com> wrote:

Hi Sugu,

  Are you trying to create a hard limit on the number of resources that may exist in your system at any given time? E.g. in case of network connections to have only N connections up simultaneously?

  If so, what you could do is create a buffered channel with N elements, with your producer inserting some -any- random stuff in there, the point of which is only to block it when N is reached. Whenever a resource is done (i.e. connection is handled and dropped), you remove one element from the channel and the producer(s) can continue.

  Like this: http://play.golang.org/p/qDAd3ERZmY

That was the approach I initially used. See here:

https://code.google.com/p/vitess/source/browse/go/pools/resource_pool.go

And some bug fixes on the way:

https://codereview.appspot.com/9209043/

The downside of pre-filling the channel with empty slots is that it makes you create a new resource until all slots are used. Ideally, the pool should try to reuse a resource that was recently returned, and create a new one only if none are available.

We have an older implementation that uses cond vars and signals, which works as intended:

https://code.google.com/p/vitess/source/browse/go/pools/roundrobin.go

But channels are supposed to outperform condvars.

[Péter Szilágyi]

Ok, now I understood how you imagined the pool to work. But where does the signal broadcasting come into the picture with the resource pool? If all resources are put back in, and Get checks validity creating new ones if needed? Are you aiming at signalling the pool when a resource goes down so the pool can recreate before somebody requests the stale old version?

[Sugu Sougoumarane]

On Mon, May 6, 2013 at 1:46 PM, Péter Szilágyi <pet...@gmail.com> wrote:

Ok, now I understood how you imagined the pool to work. But where does the signal broadcasting come into the picture with the resource pool? If all resources are put back in, and Get checks validity creating new ones if needed? Are you aiming at signalling the pool when a resource goes down so the pool can recreate before somebody requests the stale old version?

Signaling is not needed in the current implementation because the channel is always filled to capacity. I wanted to move to an implementation where the number of items in the channel could shrink below capacity. That's when signaling becomes necessary.

[Dmitry Vyukov]

You can have an elegant and efficient solution with 2 channels:
http://play.golang.org/p/TvtjgNpfff

Fast path for both Put and Get is a single non-blocking chan send/recv.
It creates resources lazily, trying to reuse existing resources first.
It allows resizing within the limit.
And I think it's relatively simple as compared to other solutions.

[Péter Szilágyi]

Hi Dimitry,

  Very nice solution :), albeit, you have a few bugs in it:

• In the NewPool, you forgot to save curcap into p.curcap.
• In the SetCapacity when you scale down, in each iteration you release two resources (break missing from the first select).
• There's a small race condition if you call SetCapacity after close, since it will try and scale up, writing into the closed empty channel.

Cheers,

  Peter

[Jsor]

Close() will also deadlock no matter what, you lock the mutex, and then go into SetCapacity... which also locks the mutex, which will cause the goroutine to wait for itself to unlock the mutex.