Schema defintion for extra/Optional fields in a JSON Object

[Sameer Segal]

Hi, 

I am evaluating FlatBuffers for my need. I have a well defined data model but it has specific areas that are expected to be completely optional as it is user defined. I treat it like a Map or  a Key/Value pair that I don't deserialize further. How do I do this using FlatBuffers? The following doesn't work:

monsterdata.json

----------------------

{

  pos: {

    x: 1,

    y: 2,

    z: 3

  },

  hp: 80,

  name: "MyMonster",

  version: 2.3,

  extras: {

    moo:"foo"

  }

}

monster.fbs (schema)

----------------------------

table Extra {

}

table Monster {

  pos:Vec3;

  mana:short = 150;

  hp:short = 100;

  name:string;

  friendly:bool = false (deprecated);

  inventory:[ubyte];

  color:Color = Blue;

  version:float;

  extras:Extra;

}

root_type Monster;

$ flatc -b monster_extra.fbs monsterdata.json

Error: monsterdata.json:11:0: error: unknown field: moo

Thanks for the help in advance. 

Best, Sameer

[Wouter van Oortmerssen]

FlatBuffers is a strongly typed system where field names do not take space on the wire (no arbitrary strings). As such it will not accept unknown fields.

If it is completely user defined and you can't define these fields in the schema, the best you can do is to store a vector of these:

table KeyValue { key:string; value:key; }

If this is C++, you can additionally read such a vector as if it was a map, see docs.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sameer Segal]

Hi, thanks for the reply. I just want to confirm that you meant the following:

monster.fbs (schema)

----------------------------

table KeyValue { key:string; value:​string​; }

table Monster {

  pos:Vec3;

  mana:short = 150;

  hp:short = 100;

  name:string;

  friendly:bool = false (deprecated);

  inventory:[ubyte];

  color:Color = Blue;

  version:float;

  extras:[KeyValue];

}

root_type Monster;

[mikkelfj]

Den mandag den 16. november 2015 kl. 23.39.49 UTC+1 skrev Wouter van Oortmerssen:

FlatBuffers is a strongly typed system where field names do not take space on the wire (no arbitrary strings). As such it will not accept unknown fields.

 

I have been pondering this question as well. If unknown fields are not accepted, how does one support schema evolution? On the other hand, if one accepts and ignores arbitrary fields, how does one handle simple mistakes, in say a configuration json file?

For better or worse, some JSON REST api's take the approach to ignore anything unknown: https://docs.docker.com/engine/reference/api/docker_remote_api/

[Wouter van Oortmerssen]

Yes, exactly.

[Wouter van Oortmerssen]

Schema evolution is supported (both forwards and backwards), i.e. an old reader can read data written by a newer schema, and there will be "unknown fields" it ignores.

This however is not the same as what Sameer is talking about. These unknown fields the old reader can't even know what size or type they are. As such, there's no generic way to add unknown data, as they would have to be accounted for in the schema at some point to be used at all.

--

[mikkelfj]

I still don't understand this. It is clear that the unknown fields cannot be stored in binary flatbuffers because the type is unknown. But if Extras.moo is undefined today, someone might add it tomorrow and give you a valid JSON. But since you still have the old binary schema you reject moo. For evolution to work, I would think moo should just be ignored in JSON, at least as an option. You don't need to know the type or size, you just have to parse past the generic json for the unknown member.

[Wouter van Oortmerssen]

We can definitely have a switch that ignores unknown json fields, yes. Feel free to make a PR, should be simple :)