using gob's over udp?

[toxicnaan]

Has anyone used gob's over a udp connection, rather than a tcp
connection.

It's seems to be sending okay, i've not tried deencoding at the other
end yet. However when using wireshark, i can see the gob is being
split over 2 udp packets? however it would seem that the struct i'm
sending would fit into 1500 byte udp packet?

* WIll I be able to listen on the other end of the connection and the
gob decoder is clever enough to works it out?
* can i control the chunking of the gob encoder?
* is this something to do with path mtu discovery and or udp
buffering?, is my tcp/udp stack\os breaking it apart?
* is it even a good idea to use gob's over the network, never mind
udp. What's the alternatives?

my application really needs udp, because it need to be quick! :-)
without the overhead of a tcp setup.

I'd really like my gob to be sent as one udp packet, (size
permitting).

Cheers,
Lee

[Ian Lance Taylor]

toxicnaan <toxi...@gmail.com> writes:

> It's seems to be sending okay, i've not tried deencoding at the other
> end yet. However when using wireshark, i can see the gob is being
> split over 2 udp packets? however it would seem that the struct i'm
> sending would fit into 1500 byte udp packet?

It's not an issue of gobs per se, it's an issue of how you send them. I
think that on current tip, if you use the rpc package, a gob that is
small enough will now be sent in a single UDP packet rather than in two.
That was a recent change. However, I don't know if there are any
particular guarantees that rpc will continue to work that way.

> * WIll I be able to listen on the other end of the connection and the
> gob decoder is clever enough to works it out?

The gob decoder just wants bytes, so it will work if you feed it streams
of UDP packets.

> * can i control the chunking of the gob encoder?

Of the encoder, no, but that's not what matters here. You can control
the chunking of how the encoder's bytes are written on the wire.

> * is this something to do with path mtu discovery and or udp
> buffering?, is my tcp/udp stack\os breaking it apart?

I doubt it.

> * is it even a good idea to use gob's over the network, never mind
> udp. What's the alternatives?

Gobs work fine over the network.

> my application really needs udp, because it need to be quick! :-)
> without the overhead of a tcp setup.
>
> I'd really like my gob to be sent as one udp packet, (size
> permitting).

Obviously UDP is unreliable delivery, and the whole system is going to
fall apart if a packet is lost. You have to have a very tolerant
application to use UDP in this way.

Ian

[toxicnaan]

On Mar 19, 1:00 am, Ian Lance Taylor <i...@google.com> wrote:

Thanks for the superquick reply.

Okay, I'm not using it with rpc, I'm just sending the bytes raw over
udp connection.

You right about path mtu discovery to my destination is 1500 bytes, so
it's not that.

If I can keep it one udp packet per transaction, it's just going to
make things
simpler, lost packets , is a lost data structure, i can just time out,
and send
another, how ever if start sending over multiple udp packets, then i'm
going to need
to keep track of what was lost, retransmit's , everything has just got
a lot more complex :-(.

I can use my own code to grab the udp packets of the wire, into my own
buffer and the feed it
to the gob decoder.

or is this a bad idea? Should i let the gob decoder just read directly
from udp connection itself?

Cheers,
Lee

[Russ Cox]

Gob is a stream protocol.
If you send two values of the same type in a row,
the second will be smaller because it can refer to
the type description sent the first time.

To use gob in a context where reordering and loss
can happen you need to create a new gob encoder
for each value, writing to a bytes.Buffer, then encode
the value, then send the buf.Bytes() to the udp connection.
On the other side, create a new gob decoder for each
received packet.

You're giving up some of the benefits of gobs by doing this.
If you can have a long-running tcp connection instead
and the values are small enough that they fit in a single
packet frame anyway, you'll be better off with tcp.

Russ

[lee hughes]

Russ,

I know tcp is a better way of doing this, but for this current instance fire and forget udp with

little overhead is the way. If i was sane I'd be using tcp streams!!! So, what you basically saying

if i understand correctly , I  build the packet myself and then fire it off. My current data structure looks 

like this...

type  connReq struct {

        Cmd int

        Etype int

        Username string

        Spekedata string

        Packet []byte

}

the first four fields are tiny, could it be to do with []byte? , if i send this, It's seems to send the first 4 fields in the first udp packet, and the Packet contents in the second. or am I not understanding something.

thanks for you help. Gobs are cool,..I'm used to asn1 (argggggggghhhhhhhhhhhhh)

Cheers,

Lee

 

[Rob 'Commander' Pike]

If you don't put a buffer into the encoder's output stream, the gob
package will write two packets the first time you encode a value of
that type, and one each subsequent time. The first time it must send,
before the first value, a description of its type. I suspect those two
initial packets are what you're seeing.

Gobs will not work on UDP. They are a stream. If a piece of the
stream is dropped that contains a type description, the subsequent
data will not be decodable.

-rob

[Rob 'Commander' Pike]

By the way, the file src/pkg/gob/doc.go contains a thorough
explanation, with worked examples, of what the gob stream looks like.
The implementation details are in the file but not exposed by godoc.
You might want to look at the source file.

-rob

[toxicnaan]

I think that's exactly what I'm seeing :-), the first packets being
the Type desc,
second being the actual data. I know I'm being bad using udp. Would it
be possible
to create my own buffer in memory, allows the encoder to write to
this, and then
send the whole buffer out via udp (if it's great > udp payload size,
we just throw it away!)
effectively glueing the type description and data together. Doing the
same with decoder
on the 'other side'.I only ever send one struct per encoder instance.
never more that. if the udp
packet get's lost, no sweat, we just time out and send again. no
having to worry about losing half a gob
and confusing the decoder?????? sane? or insane? or am i just barking
up the wrong tree.

Perhaps gobs arn't what I'm looking for at all, in may be a case of
square hole, round peg.

thanks for the advice, 'I'll use the source luke'. Any examples in
code of alternatives methods
of serialisation suited more to udp datagram's would be vey welcome.
maybe google very own protobuff's?

I think gob's really do take the complexity of network struct
serialisation. However i think my approach
to the problem is not correct.

[Sugu Sougoumarane]

Have you looked at bson or protobufs?

[Rob 'Commander' Pike]

Gobs just won't work well on a lossy channel such as UDP.

-rob

[atomly]

If you're going to start writing code for retransmission, then I'm 99% sure that you shouldn't be using UDP.  Once you're reinventing TCP, you might as well be using UDP.

I'm guessing you don't have a ton of different data types, right?  If that's the case, I'd recommend just writing custom serialization code. 

In my current project, we use protobufs for almost all of our decoding, but when I wrote something similar to what you're describing (an event tracking service that powers our logging and monitoring, so billions of events a day), I ended up just building custom serialization stuff because protobufs had a lot of problems simliar to what you're seeing and I really wanted to be able to control precisely how large my packets were.  In my case, though, it was truly fire-and-forget and losing any data structure wasn't a problem-- it was just a missed event which made our graphs very slightly off.

:: atomly ::

[ ato...@atomly.com : www.atomly.com  : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.347.692.8661 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

[ReubenBond]

I would consider using another serialisation format like (as
suggested) BSON or Protocol Buffers.

If you wish to send gobs over UDP, you might want to first create a
stream abstraction over UDP, otherwise you'll be in trouble. Consider
wrapping the UDT library, or perhaps check out this project:
https://github.com/jteeuwen/gnarly (not sure if it supports streaming
over UDP).

I can see why you might prefer UDP, NAT traversal is easier with UDP.

Let us know which way you go.

Reuben Bond

[Dave Cheney]

I can confirm protobufs over UDP work quite well. I did try to convert
my code to use gobs over UDP but failed. Thanks to Robs explanation
that gobs send a prefix packet describing their structure, I
understand why.

Cheers

Dave

[toxicnaan]

Okay. I've got my gob to fill a single udp packet,using bytes.Buffer
and then sending that out over a udp connection.
Checking wireshark everything is good, I've got a single packet, with
the type and data all in one packet :-).

udpsendbuf := new(bytes.Buffer)
enc := gob.NewEncoder(udpsendbuf)
err = enc.Encode(myconnReq)
test(err, "encoding....babel fish")

sendbuf := make([]byte, 1600)
n,err := udpsendbuf.Read(sendbuf)
test(err, "filling slice from buffer")

n,err := remoteconn.Write(sendbuf[:n]) /*fire it out an existing udp
connection*/

this works , great, I'm setting up a buffer for the gob to encode in,
and the draining
that buffer into a normal byte slice, and then firing it out over udp
works great.

However, on the receiving side I'm having trouble getting my head
around the opposite
of this. I need to receive in to a byte slice and use this data to
fill a bytes.Buffer,
and let the gob decoder get to work on this. How does one do that?

atomly mentioned retransmitions, well i don't need retransmitions, I
account for loss, if
the packet goes down a black hole, then no love is lost. The whole
thing will just recover.

Thanks everyone for the help so far. I've had other wade to try
protcol buffers, bson etc,
but gob's i understand, and are simple! So I'd like to continue using
them if possible.

Cheers,
Lee

On Mar 20, 12:56 am, Dave Cheney <d...@cheney.net> wrote:
> I can confirm protobufs over UDP work quite well. I did try to convert
> my code to use gobs over UDP but failed. Thanks to Robs explanation
> that gobs send a prefix packet describing their structure, I
> understand why.
>
> Cheers
>
> Dave
>
>
>
>
>
>
>
> On Sun, Mar 20, 2011 at 11:51 AM, ReubenBond <reuben.b...@gmail.com> wrote:
> > I would consider using another serialisation format like (as
> > suggested) BSON or Protocol Buffers.
>
> > If you wish to send gobs over UDP, you might want to first create a
> > stream abstraction over UDP, otherwise you'll be in trouble. Consider
> > wrapping the UDT library, or perhaps check out this project:

> >https://github.com/jteeuwen/gnarly(not sure if it supports streaming

[Russ Cox]

> However, on the receiving side I'm having trouble getting my head
> around the opposite
> of this. I need to receive in to a byte slice and use this data to
> fill a bytes.Buffer,
> and let the gob decoder get to work on this. How does one do that?

b := bytes.NewBuffer(data)

[lee hughes]

just shortly after i sent that, I got my brain out of freeze mode and 

just wrote it :-)

everything working great. Thanks for you all your help on the matter.

Cheers,

Lee

[Stephen J Day]

I know you have already written your code at this point, but I would
have to suggest JSON here. Its built-in, fast, compressible and will
work well in a message oriented context.