Best binary format

[andrew]

Quick and basic question... When sending data as binary over the socket, what is the best binary format to use, or is it even necessary to wrap custom structs in something? I.e. would using something like Protobuf/Flatbuffers be necessary, or is there another basic way to comprehend the format/layout of the data that is received over the socket (from either server or client)?

The two main scenarios I am interested in are sending custom c++ structs, and also sending images. 

If there are any good examples showing how to do this, would also appreciate pointers. I didn't immediately see one in the examples directory, but perhaps I don't know which of the many there might show this.

[Peter Thorson]

Really depends what is going to consume it. If you are exchanging data via websocket between two native apps (say a mobile java app and a c++ server) then something like protocol buffers is going to be pretty helpful. If the two apps are similar enough (say between two C++ apps) and the data structure is fairly simple you may even be able to just write the bytes and then cast on the other end.

If you are pushing binary data to a browser/javascript client, there are some built in features there for reading semi-formatted binary data. Specifically, take a look at ArrayBuffer and Typed Arrays / DataView.

In most browsers, binary data can be delivered either as a blob or as an ArrayBuffer. ArrayBuffer is used to read a raw byte sequence and is similar to a char array in c++. Typed Arrays let you overlay a fixed length format over the entire array, so if your array is of 32 bit signed ints you can overlay an Int32Array over the raw array buffer and pull out a sequence of signed 32 bit ints. A DataView is also overlaid over an ArrayBuffer but rather than using a uniform type it can be used to extract an arbitrary format (uint8, float32, int64, etc) at an arbitrary byte offset. For many situations this can be used to unpack a custom C++ struct. Typed Arrays and DataView can also be used to pack regular JS variables into a binary array for sending back to a C++ server.

Simple image transfer can be done by just sending uncompressed 8bpp buffers and then copying that into a canvas array. I’ve used this method for drawing realtime simulation visualizations, its fast, simple, maybe not the most bandwidth efficient. If you are sending compressed stuff, say PNG or JPEG, you’d want to use the blob method and Javascript has a createObjectURL that with a bit of additional glue can eat a blob of compressed image data and turn it into something that can be drawn to a canvas context.

If you have multiple different types of clients in different languages, something like protocol buffers gets a lot more useful. I haven’t looked, but there may be Javascript implementations of protocol buffers that would generate the TypedArrays/DataView code for you?

Peter

--
You received this message because you are subscribed to the Google Groups "WebSocket++" group.
To unsubscribe from this group and stop receiving emails from it, send an email to websocketpp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[andrew]

Thanks, good points. I probably should have clarified that I'm using websocketpp on both client and server, all c++ (no browser is involved). Simply casting the binary stream is something I've never done before, but not sure that would work if, say, a struct has an std::vector element that could be of any arbitrary size. I would think that raw casting would require a fixed memory layout/size for the struct being passed.

[Peter Thorson]

Yes, raw casting requires POD types. If you have a lot of STL/non-POD stuff and don’t want to write your own serialization then using protocol buffers & friends is a good strategy.