Adding FlatBuffer support for a new language

[Goutam Das]

Greetings everyone!

If I want to add FlatBuffer support for a new language, what are the general steps involved and the procedure for starting out.
If I understand it correctly, supporting a new language means writing the entire code again in the desired target language. Or
did I get this entirely wrong?

Can I get some resources to help me if I am interested in adding support for a new language.

Thanks,
Goutam

[mikkelfj]

There are four steps: the first is to write a new code generator for the target language. This normally involves copying and modifying and existing generator implemented in C++ in the flatc repository, then integrating the generator into the flatc tool. For example javascript: https://github.com/google/flatbuffers/blob/master/src/idl_gen_js.cpp

(Some languages do not use the flatc compiler tool in order to have full control over schema compilation - C has its own schema parser and I think Swift may also be going down that path, but this is much more complicated than it would appear.)

The second step is to provide a language runtime for common routines used to build flatbuffers - this includes alignment and endian handling. This library is typically written in the native target language, here javascript:

https://github.com/google/flatbuffers/tree/master/js

Depending on the language, the runtime may also use the C runtime library for building flatbuffers for increased performance, possibly at the cost of end user build complexity: https://github.com/dvidelabs/flatcc/tree/master/src/runtime (built as libflatccrt.a). Using the C library will also deal with complex alignment issues. Currently only the C binding uses this library. Most language runtimes put some contstraints on the order in which elements can be added (bottom up), this is not necessary with the C runtime as it has a stack.

The third step is to provide thorough test cases:

https://github.com/google/flatbuffers/blob/master/tests/JavaScriptTest.js

The fourth step is to provide documentation and examples for the language on the flatbuffers site:

https://google.github.io/flatbuffers/usergroup0.html

In addition it is neccessary to learn many details about the flatbuffer internals, such as alignment rules, order of placement, reuse of vtables, size of union type fields, which to some extent may be learned by looking at other code generators. Also see this forum where alignment has been discussed in relation to swift language support.

[Goutam Das]

Thanks for the extensive reply. This was exactly what I needed.

I don't exactly understand the function of the second step i.e providing a language runtime for common routines. Can I get a small example of how this works?

[mikkelfj]

On Sunday, March 20, 2016 at 5:23:26 PM UTC+1, Goutam Das wrote:

Thanks for the extensive reply. This was exactly what I needed.

I don't exactly understand the function of the second step i.e providing a language runtime for common routines. Can I get a small example of how this works?

You do not strictly need a runtime library, for example, the C binding only uses the generated code and some support header files, but even this may be seen as a runtime library.

C++ also only uses header files, but fairly complex ones effectively implementing a runtime library.

The library is much more important when building buffers. For example, when starting a new vector, the internal buffer location must be updated and size fields created etc. It makes no sense to generate code for all of this every a table has a vector field, so instead code is generated which calls into the runtime library.

An example in javascript where "builder" is an instance of an object that represents the runtime library (or at least part of it):

https://github.com/google/flatbuffers/blob/master/tests/monster_test_generated.js#L842-L844

Another example in C is a macro that create named strongly typed flatbuffer methods by creating inline functions around a weakly typed, but generic, set of runtime functions.

*NS ## " is just the name prefix of all the runtime library functions, e.g. "flatbuffer_". The macro is then called by generated code to create the relevant schema specific functions:

https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/reflection/flatbuffers_common_builder.h#L142-L148

(I don't expect you to get the details, it is fairly complex metaprogramming to work around C's lack of templates, but the runtime library itself is self-documented here:

https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/flatcc_builder.h

The C binding uses this builder library in 2 different independent contexts: for building flatbuffers, and for building flatbuffers as an effect of parsing JSON - instead of builidng a parse tree or builds a flatbuffer on the fly without using any C flatbuffer support. The runtime library also has other components to support JSON parsing and printing, and for verifying buffers. Not all are needed depending on use case. Reading flatbuffers only require header files.

[mikkelfj]

Correction: the C binding only uses the generated code and some support header files

this was meant for the READ interface. The builder needs a proper runtime library consisting of builder.c and emitter.c files, and headers.

Correction2: JSON parsing comment unclear. What was meant is that the generated JSON parser is independent of generated C flatbuffer builder and reader support code and their implied use of the runtime library - instead the generated JSON parser calls the runtime builder library directly for its own purposes, similar to how any C based language extension could use the runtime builder library.

Here is a small snippet of generated JSON parser code that calls into the runtime library to build a vector as it is being parsed:

... parsers has detected a valid vector begins here, calls into the runtime library to start a vector, then iterates (while(more)) over

flatcc_builder_extend_vector to add elements to the vector as it is being parsed.

Not shown: finally the vector end method is called and the vectors is added as a a table field. One could similarly place calls in a given language extension doing something similar to build an array.

                                        if (!flatcc_builder_start_vector(ctx->ctx, 1, 1, 0, 4294967295)) goto failed;

                                        buf = flatcc_json_parser_array_start(ctx, buf, end, &more);

                                        while (more) {

                                            uint8_t val = 0;

                                            static flatcc_json_parser_integral_symbol_f *symbolic_parsers[] = {

                                                    monster_test_local_MyGame_Example_json_parser_enum,

                                                    monster_test_global_json_parser_enum, 0 };

                                            if (!(pval = flatcc_builder_extend_vector(ctx->ctx, 1))) goto failed;

...

JSON is simpler to show than C code because the C builder code is so heavy on macroes.

[Rajat Kumar]

Hello. 

I will be working on Flatbuffers support in D language. Right now, I am making proposal for the same. I want to know the things which I can add in the proposal for support..

Are the steps same as for C++?

Kindly, reply for the same.