Streaming on top of flatbuffers

[Karthik Krishnamurthy]

The use case I have is this: 

- I have a C++ app that buffers some network events (for e.g., DNS name, start time, end time) in memory. 

- Periodically, a timer pops and I want to write the buffered events to disk.

- Rinse and repeat.

and finally, 

- A standalone Java application reads the file on disk to get the network events.

I wanted to keep things really simple (a header-only library, cross platform support, simple API, etc.) and so thought flatbuffers might be an ideal solution. The only hitch is that I want to be able to append these network events to the file on disk. 

So, a flatbuffer schema might look like this:

namespace NetworkEvents.Dns;

struct TimedEvent {

  start: ulong;

  end: ulong;

}

table DNSEvent {

  name:string;

  timestamp: TimedEvent;

}

I want to be able to write a bunch of 'DNSEvent' records every time the timer pops and there is stuff to write out. However, I am stumped as to how do I keep appending these records to this disk while still being able to maintain the overall flatbuffer structure (vtable, and such). Is there a work around for such a use case? I tried to search online, but there aren't enough literature on such things and hence this post.

Thanks,

Karthik

[Wouter van Oortmerssen]

Since FlatBuffers have to a contiguous memory block, it would not be efficient to try and store all records inside a single FlatBuffer.

The best idea is to make each record an individual FlatBuffer, and then to write out length-prefixed buffers to a file. This could be as simple as writing a 32bit length followed by FlatBuffer data.

We have plans to standardize this model with some helper code, but it shouldn't be too hard to do it yourself.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Boutcher]

Any timeframe on the helper code (or pointers to where people can help?)   I need to stream flatbuffers over the network, and have already concluded I need to frame them with a minimum of a length, and likely a message type as well (though I could (mis)use the file_identification

[Karthik Krishnamurthy]

Thanks much for the reply.

The way I am trying to implement this is not to write out individual flatbuffer records; but to collect, say 10 records at a time, construct a single flatbuffer from those records and write out the length of the resulting flatbuffer and the flatbuffer itself. The reader code, can then read the length (4 bytes) and then read the flatbuffer, decode it, rinse and repeat.

While writing out individual DNSEvent records is a possibility, I feel it would be space wise less efficient than collecting as much individual DNSEvent records into a single flatbuffer and the write that out.

Thanks,

Karthik.

[Wouter van Oortmerssen]

I can't make any promises to when I can get to features.. if something is urgent, I suggest a PR.

I haven't given it much thought what the ideal interface would be. For starters, we could have an alternative to FlatBufferBuilder::Finish() that also writes out a length, e.g. FinishLengthPrefixed or something. Such a buffer can then be written out in one go.

Caveat is that such buffers are incompatible with a normal FlatBuffer, so when you call GetRoot on them, it has to be from start + 4, or better yet, we'd use a GetRootLengthPrefixed.

Another issue is that FlatBuffers always generates buffers such that when the buffer contains, say, a double, the whole buffer is a multiple of 8 bytes / 8 byte aligned. FinishLengthPrefixed would need to ensure this is the case as well when the length field is included.