Checking if a scalar field is truly missing (not set to default) using reflection

933 views
Skip to first unread message

Zarian Waheed

unread,
Apr 5, 2016, 5:03:47 AM4/5/16
to FlatBuffers
I know there is a checkField function that tells whether a field is missing but it does not distinguish between a missing field or a field set to it default value.

However if we are using reflection then it may be possible to check for nulls. e.g.

1. Call checkField on the scalar field.
2. Call checkField on default_integer or default_real inside reflection::Field table

table Field {
    name:string (required, key);
    type:Type (required);
    id:ushort;
    offset:ushort;  // Offset into the vtable for tables, or into the struct.
    default_integer:long = 0;
    default_real:double = 0.0;
    deprecated:bool = false;
    required:bool = false;
    key:bool = false;
}

However the issue is that default_integer and default_real both have default values :) so you can't tell if they are missing or set to their default. Can we not have default values for these fields. I think it won't be a breaking change because when we call Field->default_integer, we will still get 0 if these fields are missing or 0.0 in case of default_real.

Wouter van Oortmerssen

unread,
Apr 6, 2016, 8:51:47 PM4/6/16
to Zarian Waheed, FlatBuffers
Are you suggesting to have default_integer etc. be not present if no default was specified?

This won't help. If you don't specify a default for a field, the compiler assigns default 0, and it will omit fields based on that. There is no such thing as fields without defaults.

See also: https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html section "Testing whether a field is present in a table"

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mikkelfj

unread,
Apr 7, 2016, 2:18:46 AM4/7/16
to FlatBuffers
I'm also not quite sure what you mean - if it is a problem in how data is stored, or in how field presence is detected.

The best way to differentiate between missing and null values that I have found is to force values into tables when they are not null. In this way a default value will be stored and the is_present call should return true if the field was stored regardless of default value. You can then compare to a known value if present, and decide if it is a missing value.

This isn't ideal because embedded objects such as vectors, strings and tables can only be present or null, so you somehow have to invent missing or default tables etc.

Zarian Waheed

unread,
Apr 14, 2016, 6:51:59 PM4/14/16
to FlatBuffers
force_defaults and wrapping fields in structs are both valid approaches as well. However I was pointing to a 3rd approach. The thing is that a field with a specified default value can never be null logically. So i can check if the field is missing and if the default value for the field is also missing then that field is truly null. The pseudocode for a integer field would be something like:

bool IsFieldNull() {
if ( !flatbufferData->checkField("some_field") ) {
 
if ( !binarySchemaReflectionTable->field("some_field")->checkField("default_integer") ) {
     
// This field is truly null
     
return true;
 
}
 
}

 
return false;
}

Hope this makes sense. The thing is that in my opinion, this is the least intrusive way to determine if the field is null if I compare it to other approaches.

Thanks,
Zarian.

Zarian Waheed

unread,
Apr 18, 2016, 2:58:10 PM4/18/16
to FlatBuffers
Going over the documentation and Wouter's comment, I realized that there is always a implicit default value for scalars. Even if the user does not specify one in schema explicitly. So in flatbuffers scalars cannot be null. I was trying to save some flatbuffers data into postgres and there scalars (int, floats etc.) can have nulls. I guess my best bet is to ask the user to force defaults or create the DB schema with the same defaults as flatbuffer schema default.

Thanks,
Zarian.

On Tuesday, April 5, 2016 at 2:03:47 AM UTC-7, Zarian Waheed wrote:

Wouter van Oortmerssen

unread,
Apr 18, 2016, 3:09:17 PM4/18/16
to Zarian Waheed, FlatBuffers
Note the doc I linked:

"Another option that works in all languages is to wrap a scalar field in a struct. This way it will return null if it is not present. The cool thing is that structs don't take up any more space than the scalar they represent."

struct Integer { i:int }


--
Reply all
Reply to author
Forward
0 new messages