Delta compression

[Michael Marcin]

I asked this same question of capnproto as the libraries seem to be quite similar.

There was a feature I've been looking for that seems impossible to implement in most of these serialization libraries.

That is delta compression.

Delta compression is a pretty core network optimization in games.

If you're unfamiliar with the concept a good overview is provided here:

http://fabiensanglard.net/quake3/network.php

At least one other person wanted a similar feature from protobuf.

https://groups.google.com/forum/#!topic/protobuf/21jobPMlLVw

With protobuf you could approximate this using all optional fields.

But it was incredibly difficult to construct properly.

[Wouter van Oortmerssen]

Yup, very familiar with delta compression in games. FlatBuffers doesn't support that out of the box, but like ProtoBuf (and unlike Cap'n Proto) it supports all optional fields (that don't take space), so simply not storing a field if it hasn't changed since last time is viable.

The logic to do so is up to you however. What is difficult about it?

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Marcin]

Difficult is maybe not the right word, laborious is perhaps better.

In a traditional (a la doom3) snapshot implementation you serialize each entity.

The user code is simply responsible for reading and writing bits to and from the stream.

The serialization of game state is done only once.

Then the system calculates a binary diff per entity per client.

To accomplish the same thing with optional fields the "delta" calculation has to move to the user code.

The serialization of gamestate has to be done per client

Building a delta compressed protobuf message for a transform (using protobuf.net) looks something like:

http://codepad.org/3vjxogxZ

It's been a few years since I've looked into this and I was reading some documentation to refresh my memory.

Maybe I'm wrong and you can use the protobuf reflection api to move the delta calculation out of user code and into library code.

Either way what you can do with protobuf is irrelevant here.

I want to do the game representation -> FlatBuffers representation conversion only once per snapshot.

Not once per client per snapshot.

I want to do the delta calculation on the FlatBuffers representation.

Benefits from quantization.

Avoids having to convert from common base snapshot (in FlatBuffer representation) to game representation.

I want to do the do the delta calculation in generic library code agnostic of the types being serialized.

This way it only needs to be written once.

Apply snapshots to the common base should be easier.

User code shouldn't have to care about delta compression.

[Wouter van Oortmerssen]

You could do that by generating code. I.e. generate a function that takes two FlatBuffers, visits every field in it (much like the Verifier in the current C++ code generator) and then adds the field in a third buffer when they differ. That be a very fast solution, but non-trivial to write and a lot of work.

We don't have a good reflection mechanism yet (if you parse a schema at runtime you can list fields etc, but it isn't easy to iterate or create FlatBuffers with this knowledge. We hope to provide a better API for this in the future.

Additionally, when applying a delta FlatBuffer, you have the issue that any field you query always appears to be present since you'll get a default back. We plan to expose functionality that lets you check if a field is actually there.