slice of channels and select

[Miek Gieben]

Hello,

I'm trying to mimic the time.NewTicker behavior which returns a channel when
called. I want to use this in the DNS package, so that a client can send
multiple queries and then async. wait for a result. However this leads me to the
case where I have to wait for multiple channels. See this example:
http://play.golang.org/p/e7vQISWw-y
(doesn't work on play, does run locally)

However I don't like this code, so I could do two things 1) use a global
channel, which all goroutines use, so that there is only one channel. 2) use
something more elegant than the current code...

Regards,

--
Miek Gieben http://miek.nl

[Rémy Oudompheng]

On 2012/11/18 Miek Gieben <mi...@miek.nl> wrote:
> Hello,
>
> I'm trying to mimic the time.NewTicker behavior which returns a channel when
> called. I want to use this in the DNS package, so that a client can send
> multiple queries and then async. wait for a result. However this leads me to the
> case where I have to wait for multiple channels.

Can you detail the pattern that leads you to wait on an indefinite
number of channels? I'm not sure why that happens.

Rémy.

[Miek Gieben]

[ Quoting <remyoud...@gmail.com> in "Re: [go-nuts] slice of channels and..." ]

> > I'm trying to mimic the time.NewTicker behavior which returns a channel when
> > called. I want to use this in the DNS package, so that a client can send
> > multiple queries and then async. wait for a result. However this leads me to the
> > case where I have to wait for multiple channels.
>
> Can you detail the pattern that leads you to wait on an indefinite
> number of channels? I'm not sure why that happens.

Well, I have a util that acts like 'dig':

./q mx @<nameserver> qname1 qname2 ... qnameN

Each qnameX will result in a query, each query will return a channel. At the
end I loop through all channels showing the replies. That would currently
lead me to use N channels.

I think just returning a global channels is the best way forward.

[mjarco]

Use sync.WaitGroup:

http://play.golang.org/p/1srl9yavab

In above code you don't even need this slice of chans... You may collect results in goroutine spawned inside Add method and return them as a result of Wait

[Jan Mercl]

I use a different approach[1] which allows to use 1..N channels for N
concurrent queries (i.e. also ony 1 channel for all of them):

ch := msg1.GoExchange(conn1, limit, nil)
msg2.GoExchange(conn2, limit, ch)
...
msgN.GoExchange(conn3, limit, ch)

// later or elsewhere
reply <- ch // first remote server to answer wins

-j

[1]: http://go.pkgdoc.org/github.com/cznic/dns/msg#Message.GoExchange

[John Beisley]

Note that by default (as I understand the code from my brief reading
of it), the channel has a buffer size of 100 [2], so if you want to
avoid leaving goroutines hanging indefinitely then you'll either need
to:

1) Consume all replies on the channel.
2) Allocate enough channel buffer size for all but one goroutine.

>
> -j
>
> [1]: http://go.pkgdoc.org/github.com/cznic/dns/msg#Message.GoExchange

[2]: https://github.com/cznic/dns/blob/master/msg/message.go#L537

>
> --
>
>

[Jan Mercl]

On Mon, Nov 19, 2012 at 1:44 PM, John Beisley <grea...@gmail.com> wrote:
>
> Note that by default (as I understand the code from my brief reading
> of it), the channel has a buffer size of 100 [2], so if you want to
> avoid leaving goroutines hanging indefinitely then you'll either need
> to:
>
> 1) Consume all replies on the channel.
> 2) Allocate enough channel buffer size for all but one goroutine.
>

> [2]: https://github.com/cznic/dns/blob/master/msg/message.go#L537

You're right. The default 100 channel items is presumed safe for
typical usage, which is 1-4 concurrent DNS servers being asked the
same question (and thus possibly responding to the same channel). The
average for, for example the .cz DNS zone, is slightly bellow 2, IIRC.

-j

[roger peppe]

I can't see how the buffer size of 100 items helps.
It looks to me as if GoExchangeBuf can send a maximum of
one message.

To be honest, I wouldn't bother defining a function like that - it's
trivial to implement on top of ExchangeBuf and there are at
least two ways to do it, depending on the caller's needs (pass
in a channel or return a channel).

As for Miek's original problem, the easiest way to solve it
would be to pass a channel to the query rather than having
the query return a channel. Then it's possible to pass the
same channel to all the queries and return the first reply
received. It's the same pattern used in various Go talks
for receiving the first reply from a web server.

[Jan Mercl]

On Mon, Nov 19, 2012 at 4:12 PM, roger peppe <rogp...@gmail.com> wrote:
> I can't see how the buffer size of 100 items helps.
> It looks to me as if GoExchangeBuf can send a maximum of
> one message.

Correct, and N `GoExchangeBuf`s can send N messages, possibly to the
same channel - if the client code wants that.

> To be honest, I wouldn't bother defining a function like that - it's
> trivial to implement on top of ExchangeBuf and there are at
> least two ways to do it, depending on the caller's needs (pass
> in a channel or return a channel).

`GoExchangeBuf` supports two ways to do it, depending on the caller's
needs (pass in a channel or return a _new_ channel _on demand_).

> As for Miek's original problem, the easiest way to solve it
> would be to pass a channel to the query rather than having
> the query return a channel.

Yeah, that's what I was trying to communicate (quoting my from earlier
post in this thread and commenting now):

ch := msg1.GoExchange(conn1, limit, nil) // pass nil -> create
a chanel for me please
msg2.GoExchange(conn2, limit, ch) // pass `ch` -> please reply to `ch`
...
msgN.GoExchange(conn3, limit, ch)

> Then it's possible to pass the
> same channel to all the queries and return the first reply
> received.

// later or elsewhere
reply <- ch // first remote server to answer wins

> It's the same pattern used in various Go talks
> for receiving the first reply from a web server.

It's a naturally occurring pattern, agreed. That's why I suggested it
to Miek for his OP problem.

-j

[roger peppe]

[Miek Gieben]

[ Quoting <0xj...@gmail.com> in "Re: [go-nuts] slice of channels and..." ]

> > of it), the channel has a buffer size of 100 [2], so if you want to
> > avoid leaving goroutines hanging indefinitely then you'll either need
> > to:
> >
> > 1) Consume all replies on the channel.
> > 2) Allocate enough channel buffer size for all but one goroutine.
> >
> > [2]: https://github.com/cznic/dns/blob/master/msg/message.go#L537
>
> You're right. The default 100 channel items is presumed safe for
> typical usage, which is 1-4 concurrent DNS servers being asked the
> same question (and thus possibly responding to the same channel). The
> average for, for example the .cz DNS zone, is slightly bellow 2, IIRC.

Interesting approach. Worth copying :-) Why can't you make the channel
blocking?

[Miek Gieben]

[ Quoting <0xj...@gmail.com> in "Re: [go-nuts] slice of channels and..." ]

> `GoExchangeBuf` supports two ways to do it, depending on the caller's
> needs (pass in a channel or return a _new_ channel _on demand_).

I do that now too, but I like Roger's suggestion: why not always
ask for a channel? Makes the whole thing simple and obeys the
principle of least surprise.

[roger peppe]

On 19 November 2012 15:36, Jan Mercl <0xj...@gmail.com> wrote:

> On Mon, Nov 19, 2012 at 4:12 PM, roger peppe <rogp...@gmail.com> wrote:
>> I can't see how the buffer size of 100 items helps.
>> It looks to me as if GoExchangeBuf can send a maximum of
>> one message.
>
> Correct, and N `GoExchangeBuf`s can send N messages, possibly to the
> same channel - if the client code wants that.
>
>> To be honest, I wouldn't bother defining a function like that - it's
>> trivial to implement on top of ExchangeBuf and there are at
>> least two ways to do it, depending on the caller's needs (pass
>> in a channel or return a channel).
>
> `GoExchangeBuf` supports two ways to do it, depending on the caller's
> needs (pass in a channel or return a _new_ channel _on demand_).

The fact that passing a nil channel results in a new channel
with a buffer size of 100 is asking for trouble AFAICS.
Usually the caller will know how many requests they're
issuing and so can size the channel accordingly - 100 is
either way too many or perhaps too few.

The former is just inefficient - the latter is potentially leaky.

>> As for Miek's original problem, the easiest way to solve it
>> would be to pass a channel to the query rather than having
>> the query return a channel.
>
> Yeah, that's what I was trying to communicate (quoting my from earlier
> post in this thread and commenting now):
>
> ch := msg1.GoExchange(conn1, limit, nil) // pass nil -> create
> a chanel for me please
> msg2.GoExchange(conn2, limit, ch) // pass `ch` -> please reply to `ch`
> ...
> msgN.GoExchange(conn3, limit, ch)

I'd suggest that:

ch := make(msg.ExchangeReply, N)
msg1.GoExchange(conn1, limit, ch)
msg1.GoExchange(conn2, limit, ch)
...
msgN.GoExchange(connN, limit, ch)

is more natural and more obvious to the reader.

However, even better, leaving all the automatic concurrency stuff out of the
package would be more in keeping with Go's usual style. We don't have
http.GoGet or net.GoDial.

[Jan Mercl]

On Mon, Nov 19, 2012 at 5:26 PM, roger peppe <rogp...@gmail.com> wrote:
> The fact that passing a nil channel results in a new channel
> with a buffer size of 100 is asking for trouble AFAICS.
> Usually the caller will know how many requests they're
> issuing and so can size the channel accordingly - 100 is
> either way too many or perhaps too few.

Too few it is not, typical zone has two (reasonable number) name
servers. 100 items is an overkill on the safe side, but it costs only
some kilobytes, i.e. it's comparable to the size of the buffers which
have to be used for the communication anyway. And you can alway pass
in a smaller (shorter) channel if you don't want the default size for
any reason. However, 20 instead of 100 would

There is probably some misunderstanding here. The message exchange is
for one specific DNS query. If the _same_ query is to be sent by the
client to more than one server (typically 1 to 4, the average being
less than 2) under certain circumstances then those 2 (sometimes 3 or
4) queries are sent to different servers and the client has the option
to collect all the responses in one channel only. But only if she wish
to do so. It's all configurable in the client's code.

Different queries for different questions are not typically reported
back on the same channel (no good reason for that). Walking the DNS
tree is mostly not an async process at all (at least not as depicted
in the RFCs). The exception to this rule happens when a name server
provides an authority without glue records. Then it is (see the RFC
again), legitimate, or even reasonable, to search for them in
parallel. Again, usually/typically two questions are sent concurrently
in that situation.

> The former is just inefficient - the latter is potentially leaky.

???

> I'd suggest that:
>
> ch := make(msg.ExchangeReply, N)
> msg1.GoExchange(conn1, limit, ch)
> msg1.GoExchange(conn2, limit, ch)
> ...
> msgN.GoExchange(connN, limit, ch)
>
> is more natural and more obvious to the reader.

You can write exactly that, it _is_ supported by `GoExchange`. BTW,
the confusion perhaps steams from that only the mechanism was
suggested (the channel "chaíning" by the functions), not the
implementation. Actually the "msg" package is a very low level one.
Normally DNS clients use the "resolver" package instead. The "msg"
package contains factored out stuff common to a DNS client and a DNS
server, roughly said.

> However, even better, leaving all the automatic concurrency stuff out of the
> package would be more in keeping with Go's usual style. We don't have
> http.GoGet or net.GoDial.

You're right, I would not write that in the standard library either.
In a specific project's support package it is a matter of preferences.
One usually wants to expose the reusable parts for own/other's
(re)use, but also adds on top of that helpers based on patterns
occurring repeatedly in the client code (and yes, your boss wants the
code no later than week ago). And then you use the exposed helpers and
cannot remove them w/o breaking backwards compatibility.

-j

PS: Also it should be probably noted that the code in question, the
"dns" package was started and written mostly in 2010. For sure I would
today, in my fourth year of Go hacking, design (some) things in a
different way. But one usually doesn't simply later change/break
company's production code only b/c it's not nice (but working ;-)

[Miek Gieben]

[ Quoting <0xj...@gmail.com> in "Re: [go-nuts] slice of channels and..." ]

> "dns" package was started and written mostly in 2010. For sure I would
> today, in my fourth year of Go hacking, design (some) things in a
> different way. But one usually doesn't simply later change/break
> company's production code only b/c it's not nice (but working ;-)

looking at the 4-line function I have now, it is indeed obvious how to
do this for experienced Go programmers.

So ... on third thought I might kill the whole function. Less is more.