Questions about streaming data

[Piyush Mishra]

I am considering using flatbuffers for messaging. I've 0 experience with it, so some of this might be silly.

Assuming I have complete control of the protocol and how data is passed around,

1. Is it advisable to use small flatbuffers for each "message" or a vector of messages?

2. Given that on a network, individual read operations do not generally end at the end of a logical message/nested-message(depending on #1's answer). Is there a way to calculate remaining length of a message/nested-message?

If the answers are of the "it depends" form, some examples/reading material to get me started would be of great help. \m/

[mikkelfj]

On Thursday, September 22, 2016 at 8:31:07 AM UTC+2, Piyush Mishra wrote:

I am considering using flatbuffers for messaging. I've 0 experience with it, so some of this might be silly.

Assuming I have complete control of the protocol and how data is passed around,

1. Is it advisable to use small flatbuffers for each "message" or a vector of messages?

2. Given that on a network, individual read operations do not generally end at the end of a logical message/nested-message(depending on #1's answer). Is there a way to calculate remaining length of a message/nested-message?

 

Flatbuffers are built back to front so are not easily streamed. I did some work on the C api to support streaming so the last part is sent first (you can control the page size via a custom emitter). However, it is not possible to ship a vector partially. If the vector is only references, such as a vector of tables, then all content can be shipped ad-hoc, and then vector itself can be shipped in chunks of references. Only when the buffer is done do you know the overall size, and can use this at the remote end to fully collect the pieces. Even so, it is technically possible to access received sub-content such as tables referenced by a vector, but this is a bit involved.

The above is limited to the C api, and it is far simpler to just ship a flatbuffer at a time and rely on TCP nagle algorithm to buffer up network packages. There is an overhead in alignment and header data so expect about 8-15 bytes "wasted" on each message.

There is ongoing work on a new schemaless flatbuffer format that is front to back, and this might be easier to stream. I am also personally considering a 64-bit front to back typed flat buffer version for large streaming data, but this would also likely have limited language support if ever done.

[Wouter van Oortmerssen]

Typically FlatBuffers don't know their own size, and partial data is generally not readable until the full buffer has been received. Hence, if you want to process messages as they come in, you'll need to store each message in its own individual FlatBuffer, and send a length-prefixed stream of them.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Piyush Mishra]

Thank you!

len -> fb -> len ->fb

works for me :)

--
You received this message because you are subscribed to a topic in the Google Groups "FlatBuffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flatbuffers/CsjsRw53pvA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flatbuffers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Piyush
https://piyush.cc

[Piyush Mishra]

On Fri, Sep 23, 2016 at 11:05 PM, 'Wouter van Oortmerssen' via FlatBuffers <flatb...@googlegroups.com> wrote:

Typically FlatBuffers don't know their own size, and partial data is generally not readable until the full buffer has been received. Hence, if you want to process messages as they come in, you'll need to store each message in its own individual FlatBuffer, and send a length-prefixed stream of them.

Is there a preferred way to do this in golang?

One way I have found is to store littleEndian builder.Offset() length in builder.Bytes[builder.Head() - 4:Head()] and return builder.Bytes[Head()-4:] to be used directly

Is that a good way to do it? I'm trying to avoid allocations.

I also wonder how one would handle messages longer than unsigned 32bit length?

[mikkelfj]

On Fri, Sep 23, 2016 at 11:05 PM, 'Wouter van Oortmerssen' via FlatBuffers <flatb...@googlegroups.com> wrote:

Typically FlatBuffers don't know their own size, and partial data is generally not readable until the full buffer has been received. Hence, if you want to process messages as they come in, you'll need to store each message in its own individual FlatBuffer, and send a length-prefixed stream of them.

Is there a preferred way to do this in golang?

 

Depends on what you are asking - if you want to intercept partial data, I doubt it but don't know. in C, the flatcc emitter interface guarantees that emit calls happen on an object boundary, and if the buffer building code sets extra information in a shared variable, the emitter can also known what type of object it is, for example setting an is_string flag before calling a method the builds and emits a string.

If you want to send individual buffers, it is just TCP/IP semantics with custom framing, such as a length prefix. I have had success using MQTT to send buffers to and from Go - this handles the framing transparently, only, you should probably need to limit the size of individual buffers quite a bit.

One way I have found is to store littleEndian builder.Offset() length in builder.Bytes[builder.Head() - 4:Head()] and return builder.Bytes[Head()-4:] to be used directly

Is that a good way to do it? I'm trying to avoid allocations.

I haven't looked at C++ code for a while, but I think you could/should hack builder so it writes directly to a file/socket instead of creating a temporary buffer first, at least if you need maximum performance, minimum latency and mimimum allocation - this was the motivation for the emitter object in C.

Regardless, you should probably ensure the bounday is at least 8 byte aligned depending on the data in the flatbuffer and how sensitive the receiving system is towards unaligned data. This is one reason I would like the size stored within the buffer, for example at offset 8 from header. This can guarantee that the buffer is always aligned up to 256 bytes if needed, but not such support in the current standard.

I also wonder how one would handle messages longer than unsigned 32bit length?

You could in principle change the size of certain types such as uoffset_t and see if it compiles, but it is involved. Flatcc is also prepared for this, but it takes some effort to get right, and breaks portability.

I think a better approach is to use smaller buffers, and even within a single buffer break down larger arrays into smaller segments - which uses less RAM during writing and is generally more processing friendly. You can the load pointers to all segments into a single large array across several buffers (read only). The individual buffers could be mmapped into RAM in one operation if separate by a size field. 

[Wouter van Oortmerssen]

To confirm, FlatBuffers does not support buffers >2GB, so using a 32bit size field should always be enough. FlatBuffers does align properly for the largest item used in the buffer, so if you're storing doubles and you pre-fix the buffer with a 32bit size, then doubles may be misaligned when received (given how most memory allocators are 8 or 16 byte aligned).

[Piyush Mishra]

Is the acceptable to overwrite the values on "bb.Head()-4 to bb.Head()" after having called Finish.

Or is there some way in the API/internals to leave a gap before the actual byte buffer starts?

[Wouter van Oortmerssen]

You'll first need to check if there's actually space in the ByteBuffer. And like I said, you're safer writing a 64bit length just to keep the buffer internals aligned, unless you're sure you won't be using any 64bit quantities.

On Mon, Oct 3, 2016 at 9:12 PM, Piyush Mishra <ma...@ofpiyush.in> wrote:

Is the acceptable to overwrite the values on "bb.Head()-4 to bb.Head()" after having called Finish.

Or is there some way in the API/internals to leave a gap before the actual byte buffer starts?

--

[Wouter van Oortmerssen]

btw, added some functionality to make size pre-fixing part of FlatBuffers: https://github.com/google/flatbuffers/commit/486c048a0d5963f83f3a0d6957e4dde41602e2e7

[Piyush Mishra]

On Thu, Oct 13, 2016 at 3:15 AM, 'Wouter van Oortmerssen' via FlatBuffers <flatb...@googlegroups.com> wrote:

btw, added some functionality to make size pre-fixing part of FlatBuffers: https://github.com/google/flatbuffers/commit/486c048a0d5963f83f3a0d6957e4dde41602e2e7

Oh Brilliant! Thank you :D 

[Maxim Zaks]

What about handling it in functional data structures style. (like described in this talk: https://youtu.be/jdn617M3-P4?t=5m28s)

Say we have a fbs like:

table List{
 items : [Item];
}
table Item {
 a:string;
 b:int;
}
root_type List;

streaming List would mean that client sends a list with N items every X seconds to the server. And server appends the received items to it's bigger list.

Now the flatbuffer on the client will be rebuild each time, we need to send data to server. The flatbuffer on server however could implement mechanism for appending the data.

As with functional data structure, what it would have to do is: 

1. add the new items to buffer. 

2. Create a new vector which will contains the references to old items and the new items.

3. Add a new list table with reference to the new vector or just replace the reference in the old one to the new vector.

This solution implies that we have dead weight (the old vector) inside of the buffer. 

But it also means that we can append to the buffer very efficiently. 

At some point we can have a job which will rebuild the buffer throwing away all unreferenced dead weight.

I think in a streaming scenario where efficiency is key, that could be an interesting solution.

Am Mittwoch, 5. Oktober 2016 18:05:55 UTC+2 schrieb Wouter van Oortmerssen:

You'll first need to check if there's actually space in the ByteBuffer. And like I said, you're safer writing a 64bit length just to keep the buffer internals aligned, unless you're sure you won't be using any 64bit quantities.

On Mon, Oct 3, 2016 at 9:12 PM, Piyush Mishra <ma...@ofpiyush.in> wrote:

Is the acceptable to overwrite the values on "bb.Head()-4 to bb.Head()" after having called Finish.

Or is there some way in the API/internals to leave a gap before the actual byte buffer starts?

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.

[Piyush Mishra]

Sorry, hit reply instead of reply all

On Thu, Oct 13, 2016 at 7:44 PM, Piyush Mishra <ma...@ofpiyush.in> wrote:

I am not certain of how the offsets are stored, feasibility might depend on that. In theory it is possible but not desireable, at least in my use case.

In practice, I am doing something similar without the overhead of making another table for it.

The point of making a table is having random access, in streaming all I need is serial access to each buffer as it comes. Prepending just 4 bytes of length is pretty efficient that way.

I inlined the length and actual buffer instead of going through one level of indirection that tables bring with them. 

Sort of like how a struct is described in the docs but with vectors inside.

[Piyush Mishra]

Another way to deal with it can be if I could give the equivalent of getRootAsXYZ to the builder. i.e. I give the builder a byte buffer and the position from where it can start.

Now all the builder needs to do is make all calculations from the start point. This way, I can make a large buffer use it with many builders, keep writing to my large buffer till it is almost full (or a timeout), send all filled bytes over and start building from the beginning.

But I suspect some extra data is stored in the buffer before Finish() is called (which the bb.Head() in golang method keeps track of). So this might not be feasible.

Pardon my ignorance, I am not good at the internals of flatbuffers.

--

Piyush
https://piyush.cc

[Maxim Zaks]

To clarify how FlatBuffers are strucutred internally here is a diagram of my example fbs and a list with two items

The dots represent the offsets they take up 4bytes and encode the distance between current position and the referenced element.

As you can see FlatBuffer is basically a graph where you can reference same nodes. Like for example if you have the same string. No need to put it two times in the buffer. Same works for the vTables. The vTable from Item1 and Item2 are same so they are "reused".

Now here is what could happen if you would like to extend this buffer with another buffer:

You have to "unfinish" the buffer, so removing the reference to the root table.

Add items 3, 4 and 5. In the diagram I assume that you can't reuse vTable from item 2 and 3 and the string.

Than you create a new vector of length 5 where you put ref to item, 5,4,3,1,2

Than replacing the reference inside of the root List table from old vector of two to new vector of five.

And do finish again.

This way we don't have to rebuild the first part of the buffer which in the case of streaming scenario will grow over time. 

We just "unfinish" it, append new data and replace some references. 

[Piyush Mishra]

I am not sure what 6,8,4 and 8,12,4,8 mean
I assume the dots mean pointers(soffset_t?) to the specific points in the buffer (as shown in the diagram).

It looks pretty cool from what little I could understand. I will try to fiddle with it this weekend :D

[Wouter van Oortmerssen]

Maxim: Nice diagrams :)

Something like this is what is possible using the reflection functionality. It is not entirely functional though, i.e. it tries to modify things in place, but thus also generates no or little garbage. The functional way would be easier, but requires a compaction step.

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.

[mikkelfj]

I'm not following this thread very closely.

But, I have been thinking of a modified reader interface that checks each pointer it chases against a hash table. It then either reads from the original buffer or from some other buffer constructed for updates. Such an in progress buffer might be scrapped and everything eventually written to a single buffer. In flatcc (for C), it would require a diffferent kind of emitter that doesn't split the partially built buffer over pages, but over object boundaries. The generated reader would be fairly easy to implement, I suppose. But some extra context argument would be needed, which C++ can hide more easily in pointer wrappers. With such a setup you are essentially getting what Facebook did with android to support updates.

I do not have any immediate plans for this, but it is something that keeps popping up.