I'll start assembling this in exp/netchan in the next few days.
-rob
// Package netchan makes Go channels available for communication over
// a network.
package netchan
// Addr represents a network address for a single networked channel.
type Addr struct {
// Net is the network protocol: "tcp"
Net string
// Addr is the network address: "machine:1234"
Addr string
// Name identifies the individual channel at this network address: "name"
Name string // "theChannelName"
}
// Listener provides the functionality to publish and manage networked
// channels.
type Listener struct {
}
// Listen establishes a Listener on this machine with the specified
// network and address.
func Listen(net, addr string) (*Listener, error)
// Publish makes the channel available for communication to other
// machines. Although the type of the argument is interface{}, the
// channel must be a bidirectional channel such as chan string. The
// return value is the Addr that other machines can use to connect
// to the channel and send data to it. After Publish, other machines
// on the network can send to the channel and this machine can receive
// those messages. Also, after Publish only the receive end of the
// channel should be used by the caller.
func (l *Listener) Publish(name string, channel interface{}) (Addr, error)
// Unpublish removes network address for the channel with the specified
// address. After Unpublish returns, no further messages can be
// received on the channel (until it is published again).
func (l *Listener) Unpublish(Addr) error
// Dial connects the channel to the networked channel with the specified
// address. Although the type of the argument is interface{}, the
// channel must be a bidirectional channel, such as chan string, and
// the type of the element must be compatible with the type of the
// channel being connected to. (Details of compatibility TBD.) After
// Dial, this machine can send data on the channel to be received by
// the published network channel on the remote machine. Also, after
// Dial only the send end of the channel should be used by the caller.
func Dial(addr Addr, channel interface{}) error
// NewName returns a name that is guaranteed with extremely high
// probability to be globally unique.
func NewName() string
/*
// Simple example; for brevity errors are listed but not examined.
type msg1 struct {
Addr Addr
Who string
}
type msg2 struct {
Greeting string
Who string
}
func server() {
// Announce a greeting service.
l, err := netchan.Listen(":12345")
c1 := make(chan msg1)
addr, err := l.Publish(c1, "GreetingService")
for {
// Receive a message from someone who wants to be greeted.
m1 := <-c1
// Create a new channel and connect it to the client.
c2 := make(chan msg2)
err = netchan.Dial(m1.Addr, c2)
// Send the greeting.
c2 <- msg2{"Hello", m1.Who}
// Close the channel.
close(c2)
}
}
func client() {
// Announce a service so we can be contacted by the greeting service.
l, err := netchan.Listen(":23456")
// The name of the greeting service must be known separately.
greet := netchan.Addr{"tcp", "server.com:12345", "GreetingService"}
c1 := make(chan msg1)
c2 := make(chan msg2)
// Connect to the greeting service and ask for a message.
err = netchan.Dial(greet, c1)
// Publish a place to receive the greeting before we ask for it.
addr, err := netchan.Publish(c2, netchan.NewName())
c1 <- msg1{addr, "Eduardo"}
// Receive the message and tear down.
reply := <-c2
fmt.Println(reply.Greeting, ", ", reply.Who) // "Hello, Eduardo"
// Tell the library we're done with this channel.
netchan.Unpublish(c2)
// No more data will be sent on c2.
close(c2)
}
*/
> // Publish makes the channel available for communication to other
> // machines. Although the type of the argument is interface{}, the
> // channel must be a bidirectional channel such as chan string.
I think you need to spell out the channel restrictions a bit more. The
examples imply that the channel may not be, e.g., "chan chan int". Is
that right? I assume "chan func()" would also be problematic. What
about "chan *int"? What happens if I use "chan map[int]int" and then
send the same map twice?
Ian
It'll be gob rules, since that's what the transport will be. I hope to make chan chan int work at some point, but no promises. One reason I did this was that when I tried to implement that in the old netchan I got lost in a hall of mirrors. This simplification might make it possible to send a channel (in some restricted form, no doubt) across the network. But again, no promises.
So in short: the channel element type must be gob-encodable.
-rob
I'm not not much of an expert, but 2 remarks
* should netchan.Addr use net.Addr in some way?
* what happens when the sender closes the netchannel? Can we use the
same sender-closes semantics in with netchannels?
--
Han-Wen Nienhuys
Google Engineering Belo Horizonte
han...@google.com
> On Fri, Dec 2, 2011 at 5:02 PM, Rob 'Commander' Pike <r...@google.com> wrote:
>> A proposal follows. The old netchan was too intricate. After talking with Russ and Ken, I've dramatically simplified it, made it unidirectional, and reduced it to a handful of simple functions. Comments welcome.
>>
>> I'll start assembling this in exp/netchan in the next few days.
>
> I'm not not much of an expert, but 2 remarks
>
> * should netchan.Addr use net.Addr in some way?
I don't see why. It would be better not to require true networking, as opposed to mere connectivity, although the current baby-step API doesn't have full generality yet. Also the address must be serializable.
> * what happens when the sender closes the netchannel? Can we use the
> same sender-closes semantics in with netchannels?
What is a netchannel? If you close the channel, the receiver will behave as it would if the channel were local.
-rob
> I can't tell from this whether you plan to hard-code net.Dial & net.Listen, or can this work with any net.Conn & net.Listener I supply?
>
> I'd want this to work over TCP-like connections (e.g. tunneled over ssh, websocket, or the various TCP-in-TCP things)
There should certainly be ways to construct Listeners given existing I/O such as pipes, not just networks, but just getting basics up and running is enough for now. I'm not even sure the name Listener is right.
-rob
Unpublish could perhaps do the closing, since it's the end inside the mux that needs to be closed. That precludes using the channel again, though.
-rob
If Unpublish does the closing, you can defer Unpublish immediately
after calling Publish.
Corrections to example code:
Calls to Listen need to specify the Net (protocol): func Listen(net,
addr string) (*Listener, error)
l, err := netchan.Listen(":12345")
->
l, err := netchan.Listen("tcp", ":12345")
Calls to Publish have args reversed: func (l *Listener) Publish(name
string, channel interface{}) (Addr, error)
addr, err := l.Publish(c1, "GreetingService")
->
addr, err := l.Publish("GreetingService", c1)
Is the Listen-Publish distinction necessary? If not, these could be
collapsed into one Publish(addr, channel) call, and you can eliminate
the Listener type altogether. Could multiple Publish calls to the
same addr.Net, addr.Addr (with different addr.Names) be collapsed
within the library automatically?
One last thought. This line:
greet := netchan.Addr{"tcp", "server.com:12345",
"GreetingService"}
made me think of a URL like:
greet := "tcp://server.com:12345/GreetingService"
I wonder whether using URLs to identify endpoints may make sense
longer term. Not sure, so probably best to leave as-is.
S
>
> -rob
> I'm not even sure the name Listener is right.
What about Receiver, since it's on the receiving end of a channel?
Dave.
On Dec 2, 9:02 pm, Rob 'Commander' Pike <r...@google.com> wrote:
> // Listen establishes a Listener on this machine with the specified
> // network and address.
> func Listen(net, addr string) (*Listener, error)
It would be useful to have another function be able to pass in an
existing net.Listener, like net/http's Serve.
This comes up with systemd on Linux, where one handles a listen socket
passed in by systemd with FileListener. Other people might also have a
reason for custom Listeners.
Cheers
Albert
right now, gob encoder sends data type spec before sending data.
however the type info is hidden from API. could gob expose type info
and allow user code to compare data types?
thanks
yigong
correct me if I'm wrong, but this new setup makes bi-directional comms more complicated as it requires a manual handshake and second connection?
What if the "client" end is behind a NAT or other such state sensitive router/firewall. The ability to ride on established tcp connections is important.
All in all this seems to improve the simplicity of the netchan library, but it doesn't seem to improve the simplicity of using it. I had _hoped_ for single connection bi-directional netchans, you can assemble something like this with the old/netchan using message routing handlers now. But with this new version a many to one or many to many connection may be unmanageably complex and racy.
Honestly, and I hate being critical you guys do great work, this kind of reduction in functionality seems a step backwards.
I will sketch out how I am using old/netchan versus how I would have to use exp/netchan if anyone is interested.
-Simon Watt
I will continue to ponder.
-rob
Don't conflate the API with the implementation. Networking by the package will certainly be handled with much more efficiency than one might assume by assuming every Dial is a new connection. Once a connection is established, it would be used transparently. Far from me to claim that NATs aren't a problem, but I believe the proposal can be implemented in a way that NATs won't break the implementation or render it impractical.
-rob
Channel close behavior was not clear for me too. In the presence of
network communication I expect that it can be more complicated and
subtle. So I think we need to explicitly statement the behavior. What
if remote side closes the chan? What if it
disconnects/terminates/network is down?
Similar concern with sync chans? Is it still the case that send
unblocks only when remote side starts processing the message? If chan
capacity is 2, does that mean that at most 2 messages are in flight?
Greetings,
JS
Don't conflate the API with the implementation. Networking by the package will certainly be handled with much more efficiency than one might assume by assuming every Dial is a new connection. Once a connection is established, it would be used transparently. Far from me to claim that NATs aren't a problem, but I believe the proposal can be implemented in a way that NATs won't break the implementation or render it impractical.
S
> // NewName returns a name that is guaranteed with extremely high
> // probability to be globally unique.
> func NewName() string
Sounds like UUID to me :)
Maybe create a UUID package and let other people use it for other things than netchan?
The use of close() becomes much more clear if you think of it as endrange()
uriel
package iochan
// Connect connects readc and writec to conn such that receiving
values from readc reads from conn, and sending values to writec writes
to conn.
// Although the types of both readc and writec is interface{}, each
must be a bidirectional channel such as chan string.
// Calling Close on the Closer returned by ReadWrite closes readc,
writec, and conn.
// It is an error to send on readc, receive from writec, or directly
close either channel after a successful Connect.
func Connect(conn io.ReadWriteCloser, readc, writec interface{})
io.Closer, error
// Unidirectional variants:
func ConnectRead(conn io.ReadCloser, readc interface{}) io.Closer,
error
func ConnectWrite(conn io.WriteCloser, writec interface{}) io.Closer,
error
// Example use:
conn, err := net.Dial("tcp", ":12345")
if err != nil {...}
readc, writec := make(chan string), make(chan string)
closer, err := iochan.Connect(conn, readc, writec)
if err != nil {...}
defer closer.Close()
writec <- "hello"
fmt.Println(<- readc)
I agree. I would like something like this (I have omitted the
comments, but I hope it is clear enough):
package gobchan
type ChanDecoder interface{
DecodeChan(name string, channel interface{}) error
StopDecodingChan(name string) error
}
func Decoder(dec *gob.Decoder) ChanDecoder
func NewDecoder(r io.Reader) ChanDecoder
type ChanEncoder interface{
EncodeChan(name string, channel interface{}) error
}
func Encoder(enc *gob.Encoder) ChanEncoder
func NewEncoder(w io.Writer) ChanEncoder
type ChanDecoderEncoder interface{
ChanDecoder
ChanEncoder
}
// Example:
in := make(chan byte)
out := make(chan byte)
dec := gobchan.NewDecoder(os.Stdin)
enc := gobchan.NewEncoder(os.Stdout)
err := dec.DecodeChan("", in)
err = enc.EncodeChan("", out)
for b := range in { out <- b }
I'm not sure, but maybe this could even be in the gob package. Add the
Chan.* interfaces there and make *Encoder implement ChanEncoder and
*Decoder implement ChanDecoder.
Then, netchan could be based on a ChanConn type which implemented
ChanDecoderEncoder.
--
- yiyus || JGL .
Also, your iochan design completely sidesteps issues of naming, which are paramount.
Regarding sending channels: Gobs are a low-level transport. It would be a mistake to bring channels into them as a primitive type, because that would mean the gob package would suddenly be managing communication and scheduling and would acquire long-term state. In short, gob is the wrong place for this functionality. The netchan package is intended to lead to the ability to send a channel (or its invisible proxy) over the network. It's not there yet, but I'm working towards it. In turn, this separation of powers requires a delicate design to avoid circular dependencies. Moreover, the previous netchan was unable to bear this load, and a redesign was triggered by that realization.
In other words, all this isn't nearly as simple as turning writes into message sends and putting the magic in gob. There's a lot more to do, and a lot to get right.
-rbo
how does this work when the client is behind a firewall?
> // Send the greeting.
> c2 <- msg2{"Hello", m1.Who}
> // Close the channel.
> close(c2)
> }
> }
>
> func client() {
> // Announce a service so we can be contacted by the greeting service.
> l, err := netchan.Listen(":23456")
> // The name of the greeting service must be known separately.
> greet := netchan.Addr{"tcp", "server.com:12345", "GreetingService"}
> c1 := make(chan msg1)
> c2 := make(chan msg2)
> // Connect to the greeting service and ask for a message.
> err = netchan.Dial(greet, c1)
> // Publish a place to receive the greeting before we ask for it.
> addr, err := netchan.Publish(c2, netchan.NewName())
addr, err := l.Publish(netchan.NewName(), c2)
> c1 <- msg1{addr, "Eduardo"}
> // Receive the message and tear down.
> reply := <-c2
> fmt.Println(reply.Greeting, ", ", reply.Who) // "Hello, Eduardo"
> // Tell the library we're done with this channel.
> netchan.Unpublish(c2)
l.Unpublish(addr)
> // No more data will be sent on c2.
> close(c2)
> }
>
> */
>
One other remark - it would be nice if the channel argument did not
have to be bidirectional.
For instance, it might make sense to hook up an existing unidirectional
channel to a netchan:
func Foo() <-chan string
func client() {
c := Foo()
err := netchan.Dial(addr, c)
}
I won't be drawn in by analogies with sockets, which are a huge symmetry break in the Unix I/O model and which expose too much and do too little for you.
One other remark - it would be nice if the channel argument did not
have to be bidirectional.
that's fine, i think. if c is a send-channel, you can give the receive end
to the library (the client can do the sending on its send end).
i don't think the library needs to be able to both send
*and* receive on the channel that you pass it.
i'm probably missing something though.
Commander,
On Monday, 5 December 2011 00:38:45 UTC+5:30, r wrote:I won't be drawn in by analogies with sockets, which are a huge symmetry break in the Unix I/O model and which expose too much and do too little for you.