Question and possible issue with --force-defaults option of flatc version 2.0.0

[Laurentiu Cristofor]

Hi,

I was attempting to test the --force-defaults option of the flatc compiler, but I can't seem to make it work as expected.

I am using version 2.0 of the compiler:

flatc --version
flatc version 2.0.0

I reproduced the issue with a simple schema:

namespace default_test;

table test_record
{
  prefix:uint64;
  data:uint64 = 2989;
  suffix:uint64;
}

root_type test_record;

I then created a JSON data file:

{
  prefix: 12302652060373662634,
  suffix: 12302652060373662634,
}

I then used flatc to generate a binary serialization from this JSON file:

flatc --force-defaults -b test_record.fbs  test_record_data.json

I would have expected to see the 'data' field serialized as 2989 (because of the --force-defaults option), but it is missing from the serialization (I verified with a hex viewer).

If I specifically set to a value, I can see the value serialized as expected.

Also, according to https://google.github.io/flatbuffers/md__schemas.html: 

• A field that has the value null (e.g. field: null) is intended to have the default value for that field (thus has the same effect as if that field wasn't specified at all).

But if I try to follow this instruction by setting data to null:

{
  prefix: 12302652060373662634,
  data: null,
  suffix: 12302652060373662634,
}

flatc will complain with:

error: /home/laur/temp/test_record_data.json:6: 0: error: invalid number: "null"

Am I missing something about how --force-defaults is supposed to work or is that setting currently broken? How can I make default values work with a JSON file?

Thank you,

Laurentiu

[mikkelfj]

I don't think you make conclusions based on JSON.

I think null would be more like an absent value in JSON. The `data` field is an integer which doesn't have a null value per se, so the JSON parser can reasonably reject that value as out of scope. Of course it could also choose to treat it as a missing value, but that is an implementation choice of JSON and has nothing to do with force_defaults.

I am more familiar with FlatCC for C and the JSON parser for that system, and here priting a JSON string from a buffer created with force defaults, using C code to build the buffer, would result in written fields being present in JSON, but if you disable force_defaults, then only fields with non-default values would be printed. (From memory, although I wrote the code I don't remember everything).

[Laurentiu Cristofor]

Note that the line that I mentioned (A field that has the value null (e.g. field: null) is intended to have the default value for that field (thus has the same effect as if that field wasn't specified at all).) was specifically from a documentation paragraph about JSON parsing. If null values are not acceptable for scalar values, then that description should be updated in the documentation.

The main concern for me is how to get the fields with default values serialized when they are absent from the JSON. I was using JSON for the convenience of generating different binary values for testing. Until now, I didn't have a need to test default values, but now I do and that's how I came over this issue.

I've just verified that the same behavior can be observed with version 1.11 of the compiler. So this might have never worked as expected.

[mikkelfj]

At any rate, if you need this functionality and you can't get it from flatc, then you should be able to achieve this by generating a FlatCC C JSON parser and specifying runtime flags to the parser so that it will force values - similar to force_defaults, but different.

https://github.com/dvidelabs/flatcc/blob/07ae7dca8118f9ab6d900a7d4797881cab708ba6/include/flatcc/flatcc_json_parser.h#L27

Mind you, you will need to build a small driver app around the parser if you want to use at as a command line tool, but you can use test code as a starting point.

[mikkelfj]

Oh maybe not - you will be able to force adding default values, but I am not sure that that the C parser will currently accept null values for scalar types. It could be supported with a new runtime flag, but that is a non-trivial undertaking. It will also not add absent fields in any case.

[Laurentiu Cristofor]

Should I add an issue on GitHub for tracking this defect?

Looking at the code, it seems like the option should do the right thing:

The compiler sets the option correctly (flatc.cpp):

      } else if (arg == "--force-defaults") {
        opts.force_defaults = true;

And then that option gets transferred to the parser's builder (idl.h):

if (opts.force_defaults) {

builder_.ForceDefaults(true);

}

[Laurentiu Cristofor]

I found the commit that added the --force-defaults setting to the flatc compiler, but it doesn't include any unit test:
https://github.com/google/flatbuffers/commit/741c63052de40c72b21d704bd685dd311ea7e4a5

[Laurentiu Cristofor]

OK, mystery solved.

The issue here is that --force-defaults only applies to forcing the serialization of default values. It does not force the serialization of default values when the field value is not specified at all (when it's missing).

Perhaps this could be clarified in the description of the setting.