What do you use FlatBuffers for?

[Wouter van Oortmerssen]

We've seen a lot of people use FlatBuffers, both here and on GitHub, yet we have no idea what products FlatBuffers ends up in. At Google we'd like to know, to see what impact FlatBuffers is having.

So, what do you use FlatBuffers for?

What products will it be used for?

How is it improving on whatever you used before?

I would expect many of you can't say so publicly, in which case I would appreciate a private email: wvo at "thatsearchengine" dot com.

[Andrew Hundt]

I use FlatBuffers to serialize data for sending with ZeroMQ. The purpose is to move data from a C++ application that runs planning for a robot arm to the Java application on the arm that executes the commands. The reduced CPU overhead compared to protobuf makes it ideal for this use case. 

https://github.com/ahundt/grl

Cheers!

Andrew Hundt

Ph.D. Student

Johns Hopkins University

[Wouter van Oortmerssen]

Indeed, we had talked about some of this before I remember. Glad to hear it is so helpful! :)

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Patrick Julien]

https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/

Facebook is using it now to make their Android application

[Wouter van Oortmerssen]

Yeah saw that! Pretty awesome article :)

On Sat, Aug 1, 2015 at 4:06 AM, Patrick Julien <pju...@gmail.com> wrote:

https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/

Facebook is using it now to make their Android application

[Alex Buchanan]

I'm experimenting with using it for a distributed data processing system for biological data. The project is really just a concept currently, but I'm imagining it will involve RPC, shared memory for workers on the same physical machine, storage, and I'm also intrigued by Facebook's use to drive their UI.

So basically everything  :P I like shiny things, what can I say.

If it doesn't pan out with Flatbuffers, I'll probably go for Protobufs, but these are the advantages I think Flatbuffers has:

- zero-copy is sexy, and Facebook's post about their performance savings was pretty convincing

- the IDL is cleaner in my opinion.

- immutability makes certain cases clearer, but might also make people say, "I have to rebuild everything?!" So I'm 50/50 on that.

[Wouter van Oortmerssen]

Sounds like a good fit :)

We actually have minimal support for mutability in C++/Java/C#, and coming to Go. We have more extensive mutability in C++ thru reflection (slower).

[Shahbaz Chaudhary]

I'm using it in a financial trading app.

While my use case isn't for very low latency trading (for this project), latency and performance are still concerns. For example, everything we are converting to FB is written with pre-defined size and tight memory control (for example, using an int or a long to store set of single bit/boolean flags).

Currently we use raw c++, but my hope is that using something like FB will improve how events/data are shared across processes and projects.

I looked at Capn' Proto, which doesn't seem to support visual studio very well (at least not when I started looking at it). Initially my goal was to use SBE but I ran into some issues with that.

[Olivier Binda]

I use flatbuffers to store networks of data on disk, in a srs application for android.
I need to do some computations (stats against different metrics, best knowledge subset to learn) on the whole data, 

I need to access the data very quickly and it's ok for writing to be a little slow

[Manish Rai Jain]

I'm using FlatBuffers for DGraph: https://github.com/dgraph-io/dgraph -- An open source, distributed, low latency graph database.

It's being used for all the internal workings -- storage on disk, in-memory cache, communication / data transfer among machines, and finally the result of the query. Currently, the result is converted to JSON. But soon enough, it would also be the communication protocol for client drivers, and power resulting graph traversal apis. It's a bit hard to work with, but really efficient; so very well suited to my system.

Cheers,

Manish

[Jason Aten]

Hi Wouter,
I'm curious about the mutability support mentioned back in October, especially for Go. Is there someplace/a link where I could read about the work in progress?
Thank you.
Jason

[Wouter van Oortmerssen]

Jason, I'd ask Robert Winslow (cc-ed, also @rw on github) if anyone is currently working on that.

--

[david....@viavisolutions.com]

We are attempting to use FBs to replace using CSV for sending vast amounts of wireless network telemetry data to an analysis and visualization platform.

We picked FB over Protos and other formats because it promises to be both faster in serialization/deserialization and smaller on the wire.

Still working through some feature issues... but we'll get there.

On Wednesday, May 20, 2015 at 1:06:52 PM UTC-5, Wouter van Oortmerssen wrote:

[Maxim Zaks]

I gave a couple of talks about how we used FlatBuffers in the game I worked on.

Here is one recording :

https://www.youtube.com/watch?v=OUNpNo-oyQY

[Maxim Zaks]

Ah and here is a video I made for another conference to showcase the speed of FlatBuffers

https://youtu.be/1uYzMMWvD0Y

[Wouter van Oortmerssen]

Very cool Maxim!

[Martin Ankerl]

We are currently using flatbuffers as storage data structure for a classifier, a Random Forest. We store about an about 80MB binary file after training, and when using the data we just memory map this file and use it for queries. The nice thing is that this has extremely low loading times, fast access, and low memory requirements. It's great :)

Martin

[Wouter van Oortmerssen]

Martin: Awesome, exactly as intended!

[David Budworth]

Not actually in use yet, but I'm likely to use it in a trading app where we have to transmit a high speed data set from our exchange interface, written in go, to a set of back trade idea finding servers, written in java.

Looking at flatbuffers for both the  minimal allocations (read: gc overhead) and fast encode/decode.

[srini...@gmail.com]

I have an idea of storing flatbuffer instead of JSON in a data store (Kind of MongoDB). But it seems FB is intended for immutability (read heavily), not for mutable operation (write/update). For example, update a string field may case performance degradation because the total memory layout has been changed. I Flatbuffer suitable to use as data layout for database?

[Wouter van Oortmerssen]

srinisingha: JSON is even less mutable than FlatBuffers (you can't modify it in place), so if JSON is sufficient for a database, FlatBuffers will be better.

FlatBuffers can be modified in C++ using the reflection API, but that isn't easy. There's also a forthcoming object API that will make mutation a lot easier. If you're using a language other than C++, this may take a while to arrive.

[srini...@gmail.com]

Thank you Wouter, I will take a look of Reflection API.

In any case, update in place with new value that bigger than old one cause the whole buffer to be moved to larger buffer to better suit the new value. What is forthcoming API you mentioned? Can we overcome this re-allocation issue with an affordable price?

I saw in CapnProto, it use multiple arenas (like multiple memory zones  - I guess) which seem reasonable. What do you think about that?

Thank. Srini.

[srini...@gmail.com]

I saw from your old post about mutable FlatBuffers.

https://groups.google.com/forum/#!topic/flatbuffers/_re6XvgnE6M

I am very interested in mutable FlatBuffers, especially in place mutation or whatever mutation, which is my intended use for our database.

May we discuss more in more detail about that? What is the design, pros/cons of it, to make FB in place mutable?

[Wouter van Oortmerssen]

Yes, the reflection API does automatic resizing when mutating strings etc. It is not ideal, but it is useful for patchups.

The new API has an optional way to unpack a FlatBuffer into C++ data structures, which can then be mutated and serialized back with a single call. Of course this not in-line, and thus not as efficient as the base flatbuffer reading/writing code.

[srini...@gmail.com]

Great to hear that. Thank you Wouter.

May I take a preview about new API, hence to evaluate the efficiency to use it as data structure for the database?

Thank you. Srini.

[Wouter van Oortmerssen]

The first version will be out shortly :)

[Wouter van Oortmerssen]

Here you go: https://github.com/google/flatbuffers/commit/3101e327c04f14f9345f60fd184aa3034e2faadf

[srini...@gmail.com]

It looks great, though it is not reflection. I think this is kind of deserialize/serialize think, not a zero copy stuff.

It might not be the ideal solution for database (DB server cannot know the schema at the compile time, so the object API might not be useful), however it provides me a way to construct data structure for object.

I am thinking of Facebook Folly's Dynamic (https://github.com/facebook/folly/blob/master/folly/docs/Dynamic.md), here is the thing: FlatBuffers is used to transfer data over the wire, then the serialized data is unpacked into a dynamic, which allows mutation to be done easily, then the dynamic will be deserialized again to FlatBuffers to transfer back over the wire.

[agallego]

Hi Wouter, 

Is there a way to define the string type used in the schema file? ie.:

typedef string my::String  

This object API works really well for my RPC. You can simply use the type and I'll build it and put it on a flatbuffer and ship it downstream. 

Thanks.

[Wouter van Oortmerssen]

srinisingha: FlatBuffers is really designed for strongly typed (schema) use cases. Though we have some reflection functionality, it is clumsy compared to a system that was designed for it. The object API does not improve this situation.

If you want to store any data (schema-less), you may be better off with a different serialization solution for the moment. I have been working on a schema-less version of FlatBuffers, but that may take a while longer to be ready.

agallego: currently that is not possible, but I agree it should be possible to substitute your own data-types. Whatever string you replace it with would need to correspond to the STL API though.

[Alexander Gallego]

On Mon, Jul 25, 2016 at 2:31 PM, Wouter van Oortmerssen <w...@google.com> wrote:

srinisingha: FlatBuffers is really designed for strongly typed (schema) use cases. Though we have some reflection functionality, it is clumsy compared to a system that was designed for it. The object API does not improve this situation.

If you want to store any data (schema-less), you may be better off with a different serialization solution for the moment. I have been working on a schema-less version of FlatBuffers, but that may take a while longer to be ready.

agallego: currently that is not possible, but I agree it should be possible to substitute your own data-types. Whatever string you replace it with would need to correspond to the STL API though.

Yup my::example_string would be stl-api compatible. 

I saw you guys do that for the GRPC_STRING typedef, though this only works for grpc::string

Thanks wouter! 

[Hải Nguyễn Trung]

Hello Wouter,

I have the similar interest in schemaless solution with zero copy serialization functionality. May I join the early development of this schemaless FlatBuffers, if this version towards to community driven?

I have some prior experience with schema design of Capnproto, FlatBuffers, ProtoBuf, as well as schemaless one like MsgPack, Amazon Ion. I think of getting the best of both worlds: flexibility of schema-less, as well as speed of schema design.

My main goal is a format of object data with reflection forn mutation,  and certainly high speed.

[Wouter van Oortmerssen]

Yes, it would be nice to get a first version of schemaless FlatBuffers out, so y'all can comment on it. Maybe I'll put it in a seperate branch for now. I'll see what I can do.

[Maxim Zaks]

Sorry for the first one the schema was a copy paste from something else and it was not for a DB use case :)

Here is a correct one

You can design a schema something like this:

struct Numeric {

 value : double;

}

struct Boolean {

 value : bool;

}

table Entry {

 columnName : string;

 text : string;

 number : Numeric;

 boolean : Boolean;

}

table Row {

 entry : [Entry];

}

table Column {

 rows : [Row];

}

table DBTable {
 name : string;

 columns : [Row];

}

root_type DBTable;

I specified only one Numeric type but you can do multiple. The scalars has to be wrapped in struct so that you can get NULL value.

Don't worry about Entry having to big. The vTables will be shared and the data part of the entry will only consist of the actual value, so the overhead is minimal if you have a lot of data.

The most incinviniet thing you would have to do is on readign of FtalBuffers you will have to null check every entry and set it on the first not null value. You coud also introduce a type enum and add type property to Entry table. This way you will not lose the type information if the entry is actually NULL.

Hope I could help,

Maxim

[srini...@gmail.com]

Hmm, seems it does not address the issue I tried to address. I want to make a FlatBuffer convenient for mutation. Once a FlatBuffer is finished, it is hard to mutate the value, especially string/vector/sub table, without resizing the original buffer (resizing means copying the untouched data to new location, which is inconvenient).

[Maxim Zaks]

Ach sorry than I didn't understand your question.

The most convenient way for doing this, is to have an object graph which you can change and than encode to FB when you need to persist or send it over the wire.

As you sad when you finish the FB there is no easy way of changing the non scalar values inside of it. With the object approach it is easy, however you are paying for redundant data storing and encoding to FB will create completely new FB even if you changed just one single thing.

I implemented such strategy in FlatBuffersSwift (https://github.com/mzaks/FlatBuffersSwift) I call it eager decoding, an FB will be eagerly decoded into a full object graph. Afterwards you can change change values in the object graph as you wish and call toByteArray to encode again. As the generated classes are plain object you can also create them yourself and call toByteArray on the root type.

I think Wouter wrote somewhere, that it is planed to add such behaviour to flatc as well.

[Hải Nguyễn Trung]

Awesome! Does it support zero serialization by using offset to value (like FlatBuffers)? Does it support schemaless using meta tag preceding the real value, like JSON?

Cannot wait to see it out! 

By the way, I don't like the name schemaless, it sounds like "She-male-les" :)

[Wouter van Oortmerssen]

Maxim: that is trying to emulate a dynamically typed system in FlatBuffers, and will be rather inefficient. I'm talking about something that does that natively.

Hải: yes, it is zero-copy like FlatBuffers, but it knows about types in-line.

[Hải Nguyễn Trung]

Hello Wouter,

How the Schemaless is going? May I have an access to the early design (wire format) for this one? 

Thank,

Hai.

[Wouter van Oortmerssen]

Hải:

Here is a very first attempt at Schemaless FlatBuffers (branch schemaless):

https://github.com/google/flatbuffers/commit/92389cd3765094e55b6a8c3ae9bd710a2f792c5c?diff=unified

This is far from finished. It's something that be can be used to give feedback on its design.

It comes with a basic implementation, some tests, and a rationale doc that should explain more of the direction this is going.

Let me know what you think :)

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.

[Hải Nguyễn Trung]

Great job Wouter! Let me see through, then we will discuss more!

[mikkelfj]

Is there any very significant benefit to this format over msgpack?

Yes msgpack is sort of parsed, and it isn't super fast, but I think tighter msgpack integration should be considered before crearting a new format just becuase. And a schemaless design will make flatbuffers less portable because implemetations will lack for a long time, whereas msgpack will be available. Now, if there really are great benefits to fb schemaless, this may be worthwhile.

That said, for dynamic storage of data in a database, it can be very tricky to manage compiled flatbuffer schema, and msgpack may not be fast enough for all purposes.

The following relates to the layout of the current flatbuffer system - and I don't know the schemaless approach well enough to know if this applies here. But some learnings:

Bottom up:

I did the C implementation which showed that top down can be done fast without preventing bottom up use, but it does add some implementation complexity. I also adds some memory overhead, but likely less than application space memory otherwise needed for bottom up. Having worked with flatbuffers in both C and Go, I must say the bottom up approach has some limitations, although of course I am biased: If you read in a number of external optional string fields, you must first create all the flatbuffer strings, then subsequently add the strings to the buffer. But since the data is optional, you end up with a lot of temporary variables and repeated tests for presence in separate stages of building the buffer. With top down you can just test field presence once, and create the string and add it one step. Top down is also less error prone with respect to violating flatbuffer construction rules.

Back to front:

We have seen issues with streaming data and flushing data to disk. With back to front this requires and extra step to reverse incremental flushes to disk. And in terms of building the buffer or speed of access, these benefits are at most minor or non-existent. Front to back doesn't solve all problems because length fields would also have to be stored after, say, a vector, mening the vector should be referenced by pointing to a tail header. With this, memory constrained devices can more efficiently build buffers in partial steps and flush as data arrives and data can be streamed to disk or network without reordering.

I understand all of this from a historical perspective, but I think the bottom up and and back to front approach should be reconsidered in a new design.

[Hải Nguyễn Trung]

From the document and the source code, I learned that the benefit of schemaless FB over Msgpack is that SL FB has quick access to sub values, saying values inside a vector or nested object. But the performance gain is not so much, IMHO.

I am with you on considering top down approach rather than bottom up.

Another thing about FB in general (including Schemaless FB) is that FB designates for read heavy operation, meanwhile for dynamic storage, an equal read-write ratio should be considered.

We have some existed choices in our shopping list for schemaless design: Amazon Ion, ArangoDB Velocypack (they both claim to be more accessible than Msgpack), and of course Msgpack for this widely adopted.

By the way, we should move the topic of schemaless into another post, leaving this topic for its main purpose: What do people use FlatBuffers for?

[Wouter van Oortmerssen]

Mikkel / Hải: the whole reason to even consider a binary schemaless format is to be able to make significant improvements over the state of the art (e.g. msgpack), not simply to extend interop options.

I feel that in-place reading of data with no parsing / object allocation is a significant enough improvement over msgpack.

You're right that a new format could totally be written front to back. The one disadvantage I can see is that you need to reserve a large (32bit?) root pointer at the start and back-patch it, whereas now it is written at the end and can always be a byte. Unlike typed FB, there's no need for this to be limited to 32bit either, 64bit buffers are possible with the current design.

One way around that would be to say that the root offset is the LAST thing in the buffer, though that may be strange as well? Certainly, I want formats to be optimized for reading at the expense of writing where needed, and reading from the front sounds generally more useful and cache friendly.

As for bottom up, I think here the reasons for doing bottom up are stronger in SFB than TFB (we need a better name! :): The code actually decides on the fly whether to encode offsets and data of its children as 8/16/32/64 quantities, and to do that it needs to have all children already serialized. Being able to do this automatically and optimally without too much overhead is a huge win (many small vectors can use 8bit offsets).

Yes, feel free to move this to a new thread :)

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.

[mikkelfj]

On Monday, August 15, 2016 at 8:00:24 PM UTC+2, Wouter van Oortmerssen wrote:

Yes, feel free to move this to a new thread :)

Please observe that the schemaless discussion has moved to the thread "Schemaless Flatbuffers" .

[Sam Paioletti]

We use fb in our Industrial Control and Automation Products. Our company makes custom Industrial Gateways to our Industrial Cloud and a few other software/hardware/cloud products. fb turned out to be our missing piece, we had a mess of different techs working for us, from JSON to proto and custom formats for serial communications. We have standardized on fb for our protocols where we aren't tying to a third party protocol, and aren't looking back. 

We are even working on an open source release of our internal messaging library https://github.com/t4i/IndisMQ, that is based on fb. Still in its infancy as a pre-alpha WIP, but its our way of contributing back, since we have enjoyed this project so much. Some people will disagree with our take on messaging, but it fits us perfectly, and wouldn't have worked very well w/o fb.

Picking up FB for our Messaging system allowed us to have huge increases in our throughput on minimal hardware (picture sending millisecond sensor data, now picture thousands of sensors). Our server infrastructure is micro-service based almost entirely in GO and the type safety has made a massive improvement in our reliability.

We appreciate the project and the work you (Wouter) and the rest of community put into it.

Looking forward to the 'schemaless' code as well (could I propose Dynamic Schema/Fixed Schema rather than schemaless :) ) It will be a welcome addition to our messaging library.

Sam

[Wouter van Oortmerssen]

Sam,

On Mon, Oct 24, 2016 at 9:34 AM, Sam Paioletti <sampai...@gmail.com> wrote:

We use fb in our Industrial Control and Automation Products. Our company makes custom Industrial Gateways to our Industrial Cloud and a few other software/hardware/cloud products. fb turned out to be our missing piece, we had a mess of different techs working for us, from JSON to proto and custom formats for serial communications. We have standardized on fb for our protocols where we aren't tying to a third party protocol, and aren't looking back. 

That's great to hear!

 

We are even working on an open source release of our internal messaging library https://github.com/t4i/IndisMQ, that is based on fb. Still in its infancy as a pre-alpha WIP, but its our way of contributing back, since we have enjoyed this project so much. Some people will disagree with our take on messaging, but it fits us perfectly, and wouldn't have worked very well w/o fb.

Very cool, a messaging stack that works natively with FlatBuffers will be great to have. Let me know when you think it has "ready" ?, and we'll link it from the FlatBuffers page.

Picking up FB for our Messaging system allowed us to have huge increases in our throughput on minimal hardware (picture sending millisecond sensor data, now picture thousands of sensors). Our server infrastructure is micro-service based almost entirely in GO and the type safety has made a massive improvement in our reliability.

We appreciate the project and the work you (Wouter) and the rest of community put into it.

Looking forward to the 'schemaless' code as well (could I propose Dynamic Schema/Fixed Schema rather than schemaless :) ) It will be a welcome addition to our messaging library.

Yes, I am quite unhappy with the name too, and haven't found anything better yet.

Maybe "Dynamic FlatBuffers" is better? DynamicBuffers? If anyone has opinions they are very welcome, I can still change it at this point :)

Wouter

[Sam Paioletti]

Thinking about it, why does it need another name…why not just call it a flatbuffer non-scaler type “map”. You kind of explain it that way in the docs you made

 

And generate the methods for creating a root map along with the other methods.

 

This would make sense when including it in another non-scaler type

 

table Monster{

meta:map

name:string

}

 

Could also allow for creation of a typed map for some type checking

 

table Monster{

meta:map[string,string]

name:string

}

 

Then I think its easier to integrate into the ecosystem…just a thought

[Wouter van Oortmerssen]

Well, it has its own binary encoding that shares similarities with FlatBuffers but is incompatible with it, it doesn't use schemas.. can be used as a stand-alone serialization mechanism.. so yeah I think it needs a name :)

It's not limited to maps. You can make the root a vector, or a single scalar.

--

[Maxim Zaks]

Using FlatBuffers currently for serialising events produced by Entity Component System Framework.

Events are, willExecuteSystem, didExecuteSystem, createEntity, destroyEntity, addComponent, removeComponent, replaceComponent.

Wrote a small tool in HTML to browser through x megabytes of logged events.

[Maxim Zaks]

RESI News, uses FlatBuffers Swift for persisting application state. 

Application state is a conversation graph (with possible cycles).

https://itunes.apple.com/de/app/resi-news-nachrichten-die/id1123307533?l=en&mt=8

[David Komer]

Coming to the browser from Unity and Flash, trying to figure out best-practices for squeezing as much juice out of it.

FlatBuffers looks like a really cool way to send data via Transferable memory from a worker thread, as well as for network requests.

Nothing commercial or fancy to show - just doing some r&d with it at the moment, but so far so good!

[agallego]

to build the fastest rpc in the west. 

https://github.com/senior7515/smf

[Wouter van Oortmerssen]

Nice!

[filippoco...@gmail.com]

I'm using FlatBuffers to serialize cached ASTs in a compiler project of mine. The project just started and there's not much of it currently, but I've been writing the parser with incremental compilation in mind from the very beginning and FlatBuffers seems like a good fit performance-wise.

[Flaviu]

I investigated flatbuffers if it is suitable for customization of menu configuration, content and transitions, on embedded displays running on ARM architecture.

[Rabble Rabble]

I'm trying to use flatbuffers as a job item in a threaded queue. My idea was to use the multi-threaded version of sqlite and make an in-memory db with flatbuffer blobs per row that the threads pull from, and a seperate table to send/recieve upstream. I'm looking at using some of the flatbuffer to set the state of the FSM in the threads, allowing them to change to the task the job describes. The rest of the payload will contain the job data.If you imagine the ECS concept, the state info would choose and configure which System the thread would become, and the main payload would be the Component data to work on. Something along those lines. I'm hoping all this will make threading a little easier for me, while allowing me to be able to dump the db to file and load it again between gameplays.I'm also hoping I haven't bitten off more than I can code.

[Laurentiu Cristofor]

At GaiaPlatform (gaiaplatform.io) we use flatbuffers as the serialization format for the records of our in-memory database. We map table descriptions to flatbuffers schemas for generating the C++ classes that we use for updating record data and we also use the C++ reflection API for reading record data generically. The reflection API is also relied on by the implementation of a Postgres Foreign Data Wrapper that allows interaction with our database via Postgres.

That is our main use of flatbuffers, but we also use it in a more standard way, for example for serializing the messages exchanged between the database client and server components.

Flatbuffers was selected from the beginning as our serialization format. There were multiple reasons for this: the existing support and wide adoption of the library, as well as specific features such as the ability to read the value of a field without having to parse the rest of the serialization.

We do have some challenges with our principal use of flatbuffers:

• Supporting null semantics is still not easy, even with the new optional features.
• The Reflection API doesn't support updating missing values.
• The Reflection API doesn't support creating new serializations either. We have worked around this by generating JSON data descriptions and calling the flatbuffers parser to generate serializations from them, but it would be nice to have a more straightforward approach.

Thank you
Laurentiu