Generating Julia code from Google's Flatbuffers schema

[Douglas Bates]

I have been looking at http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia

[Jeffrey Sarnoff]

For others: FlatBuffers are Google's developer's revisit of Protocol Buffers; structured information serialization in "performance-critical applications" with many rereads of the schema-guided data serialization.

Why FlatBuffers? for design perspective,  facebook on the phone chooses FlatBuffers  for pragmatic application, FlatBuffers home for guides details and source code.  It is licensed under Apache v2.

[Tom Breloff]

I just want to say that I am interested in this... probably more so for using raw FlatBuffers than the associated dataframes-type libraries.  Is there a FlatBuffers.jl?  I may get involved with development at some point if nothing exists.

[Douglas Bates]

On Wednesday, April 6, 2016 at 7:59:37 AM UTC-5, Tom Breloff wrote:

I just want to say that I am interested in this... probably more so for using raw FlatBuffers than the associated dataframes-type libraries.  Is there a FlatBuffers.jl?  I may get involved with development at some point if nothing exists.

A Flatbuffers.jl package would be ideal but probably beyond my abilities.  The way I understand the process there would need to be extensions to the flatc compiler to be able to produce Julia code from schema.   Currently supported languages are C++, Go, Java, JavaScript, C#, Python and PHP. Each language also has a library written in that language that provides definitions of the top-level classes.

 

Wes McKinney added a C API to libfeather so I used that in Feather.jl package instead of trying to write code directly from the Flatbuffers schema.

[Tom Breloff]

If I understand correctly, a proper FlatBuffers.jl implementation would have 2 stages:

1. Pure-Julia code to define FlatBuffers structs, with a getter/setter API
2. A code-generation tool which would produce boilerplate Julia code (#1) for a given FlatBuffers schema file

So one could use/interact with FlatBuffers data once we have #1.  If anyone starts work on this, please post here!

[Kevin Squire]

Likely not directly useful, but Ragel does something similar, and Daniel Jones spent some time getting it to produce julia from a spec.

https://github.com/dcjones/ragel-julia.  

It won't read FlatBuffers spec files, though.

   Kevin

[Douglas Bates]

On Wednesday, April 6, 2016 at 10:23:18 AM UTC-5, Tom Breloff wrote:

If I understand correctly, a proper FlatBuffers.jl implementation would have 2 stages:

1. Pure-Julia code to define FlatBuffers structs, with a getter/setter API
2. A code-generation tool which would produce boilerplate Julia code (#1) for a given FlatBuffers schema file

So one could use/interact with FlatBuffers data once we have #1.  If anyone starts work on this, please post here!

 

It seems from the code in https://github.com/google/flatbuffers that it would be necessary to create src/idl_gen_julia.cpp

for #2 and a separate julia directory for #1

[Douglas Bates]

I have a preliminary Julia implementation of the reader in https://github.com/dmbates/FlatBuffers.jl