How to save multiple flatbuffer records into one file?

[xi...@tune.com]

Hi,

We are trying to convert some csv formatted files into flatbuffer. Each line of the csv file will be converted on flatbuffer object. But to save multiple objects into one file, we need to save the size of each objects at the head of the file. Is there a standard way to do that?

Thanks,

Xinyu

[Wouter van Oortmerssen]

There's no standard way to do that.. we'd normally recommend to store all data in a single flatbuffer to make this easier.

Ideally, you'd accumulate all data in a single FlatBufferBuilder. 

If that's not possible and you wanted to serialize them individually, you could still combine them into a single buffer later on like this:

table Buffers { buffers:[Buffer]; }

table Buffer { buffer:[ubyte] (nested_flatbuffer: "MyRoot"); }

The nested_flatbuffer annotation creates a convenient accessor to the nested data.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tiago Cogumbreiro]

Hi,

I had a similar requirement and came out with a slightly different approach:

https://github.com/cogumbreiro/fbs-to-jsonl

I prefixed FBS objects iwith an int specifying the size of the data and then concatenated them in a file.

The cool thing about this approach is that it is Unix friendly and you can easily 'cat' multiple streams of FBS objects together.

The library also lets you convert from FBS into JSONL[1], which is  simply a file with a json object per line.

Hope it helps,

Tiago

1. http://jsonlines.org/

[Wouter van Oortmerssen]

Hah, I guess that's a nice and simple approach :)