multicasting with encoding/gob

[meera]

I am trying to create a package for game servers using gob. The current approach is an application level multicasting over TCP, having a gob encoder and decoder for each player connection, and set up a goroutine to receive and another to dispatch for each one. The code for the dispatcher is here. But summarized, it simply receives data from a channel and encodes it.

The problem is that if i want to transmit a single piece of data to all players, this piece of data is encoded again and again for each connection, doing duplicate work. With less than 100 players this is not a problem, but with 300+ my machine is at almost 100% usage and the profiler shows that most of it is spent on encoding. Here's the issue on github.

I tryied using a io.MultiWriter but gob complains of duplicate type received, and if i try to write the raw bytes from the gob.Encoder i get corrupted data. An option is using UDP Broadcasting but since gob expects a stream, i'm affraid i will run into unexpected behavior when packets start to be lost and fragmented.

Does gob expect a single encoder and decoder to own the stream? Not allowing two encoders on the server for one decoder on the client?

[meera]

Sorry, accidentally broke the links. 
The code for the dispatcher is here: https://github.com/kazhmir/gna/blob/master/dispatcher.go#L81
And the issue is here: https://github.com/kazhmir/gna/issues/1

[Robert Engels]

You need to encode once to a byte array then send the byte array on each connection. 

On Dec 23, 2020, at 3:45 PM, meera <lordho...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0562184e-bbcc-44c9-adbf-37e8d5411c7cn%40googlegroups.com.

[Artur Vianna]

If i create a bytes.Buffer and a gob.Encoder, each time i write to a group of connections i get "duplicate type received" and if i try and reuse the encoder, i get "corrupted data" and "unknown type".

It seems i can't use both net.Conn.Write and gob.Encoder.Encode in the same connection, i will try always encoding to a buffer in both unicast and multicast like you said and report if it works.

[Matthew Zimmerman]

My understanding is that gob streams are unique. 

From https://golang.org/pkg/encoding/gob/

"A stream of gobs is self-describing. Each data item in the stream is preceded by a specification of its type, expressed in terms of a small set of predefined types."

In my own rudimentary understanding/terms, it sends the struct definition once, then uses shorthand for it afterwards.  E.g, how many bytes and what order.  If you mix and match streams that send definitions in different orders, then chaos ensues.

I think this is why people use other encoders in the scenario you're taking about.  For a one to one stream gob works great, but in this multi scenario I don't think it does.

Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBXN46idvqUbCsGs%2BZbZt%2BCj4MowJ4Ozj3_U9_6-68OWDw%40mail.gmail.com.

[Robert Engels]

Yes, that is why you need to create your own protocol. Use the gob to encode to a buffer then send the buffer on each of the connections using your protocol. 

On Dec 23, 2020, at 4:19 PM, Matthew Zimmerman <mzimm...@gmail.com> wrote:



[Axel Wagner]

The issue with that approach is that gob keeps state about which type-information it still has to send. So if you encode to, say, a bytes.Buffer, it would encode all type-info on every message sent, which is a significant overhead.

TBH, I don't understand why `io.MultiWriter` wouldn't work. It would be helpful to see the code that causes the error message OP is seeing.

However, really, gob just doesn't provide a good API for this sorta thing, as mentioned. The format itself is fine, but the stateful connection means that if you don't want to write *exactly* the same data in exactly the same order to all connections (which can perform poorly and lead to operational problems with timeouts and intermittently lost connections and the like), you are going to have a bad time.

You honestly would fare better with a full-fledged RPC framework such as gRPC. Or, if you don't want the overhead of its IDL, even json. Because at least the "encode once, send to each client" is trivial to solve with that.

But, that's just my 2¢ :)

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/214752B6-2666-4892-A9B8-E4BC4127FD42%40ix.netcom.com.

[Wojciech S. Czarnecki]

Dnia 2020-12-23, o godz. 13:45:10
meera <lordho...@gmail.com> napisał(a):

I'd use flatbuffers. https://google.github.io/flatbuffers/flatbuffers_guide_use_go.html

It is possible to extract gob encoded bytes and copy it over multiple receivers but it
would be an overkill, likely.


--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[Artur Vianna]

I will look into other protocols, although for now the performance is not an issue in servers with less than 100 players. 

The problem with io.MultiWriter is that a player inside the group may disconnect or a new player may come in. This means a new io.MultiWriter must be created each time you dispatch, since the group may have changed in the meantime. This would also need a new encoder and then the "duplicate type received" happens.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGWtULh8Q3Jqu_gq5m5Si4PvJ1oVSZY7DVhu%3D6hGK83bg%40mail.gmail.com.

[Matthew Zimmerman]

If you would "reset" each client with a new decoder each time you make a new encoder, everything should work fine.  Just would take some coordination.  

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBUsmp2sbiEh%3D3z0cC9EhjLig%2B8exXyA05YngBJ-tsC_uA%40mail.gmail.com.

[Robert Engels]

There are ways to control the framing but in my experience a pseudo multicast scenario is best implemented with a proprietary protocol. You can do this easily on top of other transports Luke grpc/protobufs.  

On Dec 23, 2020, at 5:20 PM, Matthew Zimmerman <mzimm...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAD53Lr5yREUMGwgwk%3D5G%2BTDDipRLUKLc1SvPdsWsW_6mjVqvzQ%40mail.gmail.com.

[Axel Wagner]

No, it wouldn't. Because the encoder keeps state about which type-information it already sent and wouldn't sent it again - causing the client to be unable to decode. So you'd also need a new encoder on the server. And at that point, you're back to the status quo, with one encoder per client and the duplication of encoding effort.

A solution would, perhaps, be if the gob API would give you a way to send *only* the type-info (so you could, if the connection breaks, create a new encoder, send all the type info, and *then* multicast the encoded values). But it doesn't.

Really, I think it's far less effort to just use a different format (and I would maintain that even json would probably be fine) than trying to make this work with gob :)

[Artur Vianna]

Before using gob was using encoding.BinaryMarshaler, but that would mean the user of the api would need to implement a MarshalBinary for every type, which is kind of cumbersome.

An option might be to let the user choose gob, BinaryMarshaler or Json etc to best fit the use case, but that takes the simplicity of only gobs away.

I did try your solution to reset the client too but i'm getting inconsistent behaviour, in one server it works and in another it doesn't ("corrupted data or unknown type"). I think synching the server and client will be error prone, while also increasing the use of network.

The easiest solution now is to label the package for ≤32 players and test alternative encodings that keep the API as clean as with gob. I took a look at flatbuffers but it will be cumbersome for the user to create the builders, and i really wanted the simplest possible API.

Maybe i should try UDP Broadcast too and see what happens, probably chaos :D

[Robert Engels]

That is not true. Java serialization works similarly. You can hook it do that you send the metadata once during connect, and then encode the data.so no a new connection only needs the metadata and can decode further stream messages. You may need framing resets to simplify things and reduce the overhead depending on the hierarchy of “objects and references”

On Dec 23, 2020, at 6:09 PM, Artur Vianna <lordho...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBU%3DZBR8O9na44vsXCg9NXrH9tdo5xxuBjYqMOYoGU4N0w%40mail.gmail.com.

[Artur Vianna]

I'm confused to which part of the thread you're referring to

[Robert Engels]

I was referring to the comments about the encoder keeping state. You can reset the encoder. You may need your own framing to do so - I’m not looking at the gob streaming encoder docs - but if it is a decent streaming encoder it should have a reset mechanism. 

That being said it is probably easier to use lightweight encoders like protobufs - still for best efficiency for something like a game that doesn’t need general purpose encoding - a custom encoder is probably most flexible and efficient. 

On Dec 23, 2020, at 6:40 PM, Artur Vianna <lordho...@gmail.com> wrote:



[Axel Wagner]

On Thu, Dec 24, 2020 at 1:46 AM Robert Engels <ren...@ix.netcom.com> wrote:

I was referring to the comments about the encoder keeping state. You can reset the encoder. You may need your own framing to do so - I’m not looking at the gob streaming encoder docs - but if it is a decent streaming encoder it should have a reset mechanism. 

No offense, but saying "that is not true" without looking at the docs is probably not a great idea.

It's possible to implement an API to do this. The API as it exists (at least in the stdlib) doesn't. That's what I was saying.

[Axel Wagner]

On Thu, Dec 24, 2020 at 1:08 AM Artur Vianna <lordho...@gmail.com> wrote:

Before using gob was using encoding.BinaryMarshaler, but that would mean the user of the api would need to implement a MarshalBinary for every type, which is kind of cumbersome. 

An option might be to let the user choose gob, BinaryMarshaler or Json etc to best fit the use case, but that takes the simplicity of only gobs away.

I am all in favor of API simplicity, but gob just isn't super useful for this. Not to repeat myself, but you should really at least try JSON - it provides exactly the same convenience as gob, but doesn't suffer these problems. It might have a bit more overhead and might even be costlier to encode - but the savings from being able to eliminate duplicate effort should offset that (and I'm not even super convinced - I don't think gob is the most well-optimized encoding in the stdlib).

But IMO, if you provide an API, the best solution is to a) just use `[]byte` at the base-layer and then b) provide convenience-wrappers around that for other formats. This lets the users decide what they want (they might want something completely different anyway) while still providing a decently convenient API for simple uses.

[Robert Engels]

That’s good to know. I figured after 20+ years of learnings with Java serialization that a somewhat modern version would have the ability to reset the stream. This is required for large dynamic object models. Very surprising. 

On Dec 23, 2020, at 6:56 PM, 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFcE9fJvLaEB1oSRFaWOWWAOy%2BuZgO6NQs56O_1SxovzQ%40mail.gmail.com.

[Bakul Shah]

I looked at your top level README.md to understand what you are doing.

Do these players join at the same time or different times? If different times, that can explain corruption with MultiWriter.

One suggestion is to use "bufio" to create a writer for the gob encoder out of a []byte buffer. Then after each encoding, grab the buffer and send it out to each receiver on a separate connection. This costs the encoding overhead just once regardless of receivers. You may have to play with this a bit to make sure the receiver side decoder works seamlessly even when you send N such byte strings. [I haven't looked at how gob is implemented so don't know if this is possible]

I'd avoid UDP unless your app can tolerate some loss. If you need reliable delivery, you may as well use TCP. Also, don't expect UDP broadcast to work across the Internet!

Probably a good idea to use JSON as others have suggested, as you can then interoperate with programs written in other languages. [In general external interfaces should be language neutral]

[Artur Vianna]

Exposing the bytes would hurt the abstraction. Game servers usually only process data in certain intervals (say 20 times per second), that means i would either need to expose the net.Conn for reading, or creating a buffer for each connection and acumulate the data on memory, so that each time the server updates, the data is available (without filling up the TCP buffer). 

Also this data is processed into a single place, where you change the state of the server.

Players are expected to connect any time the server is up, or none at all (and the server might still update it's state without the players), that's why i needed to create a new io.MultiWriter every Dispatch.

I will certainly look into Json and maybe MessagePack. But first i may fiddle with the gob source code, if my sanity allows it. If i can control when the type information is sent i may be able to fix this without changing the API. The ideal enconding would be a stateless gob, with the proper decoding/encoding "machines" set on either side when you call gob.Register, but i'm not sure of the feasibility of that.

[Artur Vianna]

In games most code is shared between server and client so normally they're built on the same language, and even in the same framework. Sometimes you need to share load with the client to increase the amount of players. So I'm not too worried about inter-language interoperability.

I'm not sure having a separate connection with the player for multicast would solve the problem, the gob encoder still needs to keep track about the types he sent, and when multicasting the gob doesn't see the connection, either a bytes.Buffer or MultiWriter, and it gets lost because the underlying connections are constantly changing.

Ideally i need a protocol that doesn't need to sent the type information before the data, having the type embedded into the data itself. gob builds a machine of sorts and sends it over the wire if i'm not mistaken, i guess that's where it goes downhill.

[Axel Wagner]

On Thu, Dec 24, 2020 at 6:35 AM Artur Vianna <lordho...@gmail.com> wrote:

Exposing the bytes would hurt the abstraction.

I strongly disagree with this. Exposing the bytes would improve the abstraction, by making the transmission- and encoding method orthogonal. And as I said, you can still provide just as convenient a wrapper around it.

 

I will certainly look into Json and maybe MessagePack. But first i may fiddle with the gob source code, if my sanity allows it. If i can control when the type information is sent i may be able to fix this without changing the API. The ideal enconding would be a stateless gob, with the proper decoding/encoding "machines" set on either side when you call gob.Register, but i'm not sure of the feasibility of that.

Sure, that's definitely possible. I factored that in, when I said "I think it's far less effort to just use a different format than trying to make this work with gob".

But, of course, you do you. :) The decision of what API to expose and what encoding scheme to use and how much work you are putting into what piece is ultimately up to you :)

[Artur Vianna]

I will definitely explore your ideas, thanks for the insights. I'll let you know how it turned out in the end :D

Thanks

[Robert Engels]

Protobufs is what you want. Since you have the schema on server and client the amount of data dictionary overhead per message is minimal - essentially numeric tags and the raw data. 

On Dec 24, 2020, at 8:21 AM, Artur Vianna <lordho...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAE%3DAWBWmiQbVBxQ2FcWguzbd7Y62LUjmQ%3DY0qKe%3D4M0WKgoubg%40mail.gmail.com.