Flatbuffers internals and overhead

[Alberto]

Hello everybody,

I'm working with flatbuffers and really small payloads
 and I am having some issues.<My problem is that I'm using flatbuffers for serialization/deserialization of standard can messages (8 bytes payload) and with the python implementation i have something that looks like an offset:

Timestamp:        0.000000        ID: 0341    S                DLC: 12    06 00 00 00 00 00 11 00 27 00 12 00

DLC is the can's message payload size (can't be greater than 8), after that you can see the actual payload that starts with 06hex.

Timestamp:        0.000000        ID: 0101    S                DLC:  8    05 00 00 00 00 50 00 00
Timestamp:        0.000000        ID: 0321    S                DLC:  8    04 00 00 00 61 00 18 00
Same with those 2 more messages.

I can provide more details if needed, for now my question is simple: does flatbuffers add ANY payload overhead if using something like that for schema.fbs?:

struct HV_VOLTAGE {
   pack_voltage: uint16;
   bus_voltage: uint16;
   max_cell_voltage: uint16;
   min_cell_voltage: uint16;

}

[Alberto]

Here at the end of the format components section you can read:

This may mean two different implementations may produce different binaries given the same input values, and this is perfectly valid.

Does this mean that the python's implementation might have offsets for optimization purposes? Does the python implementation actually have them?

[Wouter van Oortmerssen]

Have a read of the details of the FlatBuffers format here: https://google.github.io/flatbuffers/flatbuffers_internals.html

In short, yes, there is a fair bit of meta information in a FlatBuffers binary, and you can't use it as a naked struct serializer as you appear to be trying.

In fact, what you've serialized there isn't even a valid FlatBuffer, the root must always be a table, not a struct. Presumably the Python API's untyped nature makes this possible.

The 0x00000006 is indeed an offset to the root table, in this case, the root struct.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/flatbuffers/499edf2d-d6f4-42ad-acc4-8a22aef015f3n%40googlegroups.com.

[Alberto]

Thanks for the clarification.

OT

Can I ask you if you have some suggestions of "naked struct serializers" that support multiple languages, specifically python and c beside a custom implementation?

[Wouter van Oortmerssen]

Not aware of a system that solves this for many languages, though I am sure there are some.

Python has this nice library built-in: https://docs.python.org/3/library/struct.html

And C of course can do it natively, assuming you don't work on platforms where alignment/endianness would make things more complicated.

So maybe you don't need a system for this afterall :)

To view this discussion on the web visit https://groups.google.com/d/msgid/flatbuffers/93e17a2a-eec4-4475-b626-c8eee9678979n%40googlegroups.com.

[mikkelfj]

Flatcc (FlatBuffers for C) can export FlatBuffer structs as root objects. This is not supported by other languages. You still get the flatbuffer header but you cheat and just copy the struct part of the buffer either properly by finding the buffer root, but by cheating because it usually is placed at the same offset.

What FlatCC givers you over native C structs is a predictable layout on all platforms and all C compilers, and endian neutral storage if this matters. Of course it will not help with other languages but you might be able to find compatible encoder libraries since the format is predictable.

Mikkel

[Alberto]

That explains why our c implementation worked without problems, thanks for the clarification!