Schema generation at runtime.

[Davide Faconti]

I tried to look at the code of flatbuffers and its tests. and I am pretty sure I can do what I want, but I don't know how.

I want to:

1) generate a schema at runtime, from scratch. I am not modifying an existing schema.

2) Modify the values of the fields.

3) Generate a file that contains the schema (.fbs) and another one that contains the binary data.

To achieve point 1) I need apparently to use the  functions procided by flatbuffers/reflection_generated.h, in particular CreateSchema. The only place where I can see this funtion used is  flatbuffers/idl_parse.cpp -> Parser::Serialize()

To achieve point 2) I can look at test.cpp -> ReflectionTest. Not very clear, but I guess I can figure it out somehow.

But then I don't really know is how to "save" to file my newly created schema. There is GenerateFBS, but it needs a Parser to be passed (I have just a FlatBufferBuilder).

Any clue?

Davide

[Wouter van Oortmerssen]

This is certainly not an easy path.

You'd generate a schema as text (probably the easiest way). Parse that using Parser. Then generate a binary schema using Parser::Serialize(), and use the output of that with the reflection functionality.

The problem is there is no code to create a binary entirely from scratch dynamically. This can probably be done by calling functions in flatbuffers.h, but it is tricky. You might be better off generating a base JSON string with default values for your schema, parse/compile that, then modify fields on that using reflection.

The saving part is easy, the buffer you are modifying with reflection can be written out as-is.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Davide Faconti]

You'd generate a schema as text (probably the easiest way).

This is exactly the point I miss right now.

 

Parse that using Parser. Then generate a binary schema using Parser::Serialize(), and use the output of that with the reflection functionality.

Yes, this was my plan.

 

The problem is there is no code to create a binary entirely from scratch dynamically. This can probably be done by calling functions in flatbuffers.h, but it is tricky. You might be better off generating a base JSON string with default values for your schema, parse/compile that, then modify fields on that using reflection.

Ok, this sound more or less what I was planning, but I miss the generation part of the schema.

I preparade a gist that will show you a simple example of what I need to do.

https://gist.github.com/facontidavide/f08d4c558f788dbcdd1efdd8cb2bcf8a

Thanks in advance

[Wouter van Oortmerssen]

When I said generating a schema as text, I meant creating it as a sting.. using string concatenation or whatever means. Then the parser can process it further.

I don't quite follow you code sample.. there is no API available that does what you want.

Can you explain me what you're trying to do? Why can't you work with a schema the traditional way?

[Davide Faconti]

// When I said generating a schema as text, I meant creating it as a string

I see your point now. Yes, I guess that this would create a more  concise and less error prone code compared to my example, which requires quite a lot of code to build a relatively short schema.

// I don't quite follow you code sample.. there is no API available that does what you want.

It is kind of funny that you said that because I compared the schema obtained by a Parser with the one that I build by "hand" using the API provided by reflection_generated.h, and they look pretty much the same ;)

// Can you explain me what you're trying to do? Why can't you work with a schema the traditional way?

Honestly, I have the feeling that this is not the topic of the thread, but anyway...

Long story short:

Reflection is meant to be used to avoid the "compilation" step on the reader side. What I am trying to do is to avoid that step also on the writer side.

I am trying to build a very basic profiler that stores some time series with as little overhead as possible. The number of fields, their name and the type is specified at runtime, during startup.

Since the number of fields of the root_table is not defined at compilation time,  I can't follow the usual workflow.

Anyway, I understand your point.

I will use the parser to generate the schema instead of the functions provided by reflection_generated.h.

Davide

--

Davide Faconti

[Davide Faconti]

Unfortunately i am still struggling with this problem :(

I am tempted to think that this is a bug, but maybe I am doing something wrong.

OK here there is a new example https://gist.github.com/facontidavide/1bfbc88f0accf4042e648e03e686942b

The problem might be related to the way I generate a builder from a parser... any help is really welcome.

[Wouter van Oortmerssen]

I don't understand what you're trying to do with CopyTable. You're saying you're wanting to copy a table of type Test, but then the table pointer you pass is that of the binary schema of test, which are different things.

What you probably want to do is after you Serialize(), copy the binary schema (as a buffer, not using CopyTable) somewhere for later use. Then invoke the parser with a JSON string that matches your object, e.g. "{ 0, 0, ... }". The new buffer represents a Test object which you can then modify with the reflection functionality.

[Davide Faconti]

ok, I understood what you mean.

it is anyway a bit weird I can't use a Schema to generate automatically a table from it.

to be oblige to use a dummy json looks a bit hackish.

after all, the schema is supposed to be a complete description of the flatbuffer, right?

thank you again

--
Davide Faconti

[Wouter van Oortmerssen]

Yes, we could have a CreateBlankTable function or something similar, we just don't. This is the next best thing :)

[Davide Faconti]

For the records:

Now that I eventually understood that CopyTable won't help, I am implementing myself a simple CreateBlankTable.

I finally realized something that you probably know from the very beginning: usign JUST reflection I can deal well with tables, vectors and basic types, but I can't have structs and enums, since only code generation can truly deal with them.

Silly me.

Anyway, even with these limitations, my trivial implementation will serve my particular use case.

Thanks again for your time.

Davide

ᐧ

--

Davide Faconti