FlatBuffers data fails verification immediately after creation, why?

[Michael McClenney]

I'm setting up a simple test with FlatBuffers (C++) to determine if I'd like to use it to serialize data in my game. I'm running into a problem where the root tables I'm creating are failing verification. I've tried a few different ways:

1. I've output using C++ and builders

2. I've output using Python and builders

3. I've converted JSON to binary with flatc <-- this is how I'd prefer to write out my flatbuffers data

* I have not tried reading JSON text <-- if this is my only option, it makes more sense to write my own tool

They all fail verification. So my question is, can I verify a buffer immediately after finishing? I've tried writing to disk, reading in, then verifying, but I want to eliminate as many variables as possible here just to make sure all of my issues are related to my use of flatbuffers.

Here's the schema I'm using:

namespace zz.fbs;


table Float {
    value:float;
}


table Size {
    width:float = 0;
    height:float = 0;
    scale:bool = false;
}


struct Version {
    major:int;
    minor:int;
    build:int;
}


table GameConfiguration {
  autoSaveSecondsDelta:Float;
  targetResolution:Size;
  version:Version;
}


root_type GameConfiguration;

Here's the code I'm using:

flatbuffers::FlatBufferBuilder builder;


zz::fbs::SizeBuilder sizeBuilder(builder);
sizeBuilder.add_width(1536);
sizeBuilder.add_height(2048);
sizeBuilder.add_scale(true);
auto res = sizeBuilder.Finish();


auto version = zz::fbs::Version(0, 11, 0);
zz::fbs::GameConfigurationBuilder gameBuilder(builder);
gameBuilder.add_autoSaveSecondsDelta(10.0);
gameBuilder.add_targetResolution(res);
gameBuilder.add_version(&version);
auto gameConfig = gameBuilder.Finish();
builder.Finish(gameConfig);


auto bufferVerifier = flatbuffers::Verifier(builder.GetBufferPointer(), builder.GetSize());
auto isBufferOk = zz::fbs::VerifyGameConfigurationBuffer(bufferVerifier);
CCLOG("Verifying GridConfiguration buffer, size: %u, is OK: %d", builder.GetSize(), isBufferOk);
auto gameConfigBuffer = zz::fbs::GetGameConfiguration(builder.GetBufferPointer());
CCLOG("gameConfig: %p", gameConfigBuffer);
auto gameConfig = new GameConfiguration(gameConfigBuffer);
CCLOG("auto save seconds (from buffer) delay: %f", gameConfigBuffer->autoSaveSecondsDelta()->value());
CCLOG("auto save seconds (from custom object) delay: %f", gameConfig->getAutoSaveSecondsDelta().getValue());

CCLOG just logs strings to stdout (yes, I'm obviously making a cocos2d game :D). Here's the output:

"Verifying GridConfiguration buffer, size: 68, is OK: 0"
"gameConfig: 0x7c06f7cc"
"auto save seconds (from buffer) delay: 0.000000"
"auto save seconds (from custom object) delay: 0.000000"

Am I screwing something up here? Not only does the buffer fail verification, but it returns incorrect information (I'm getting 0 for auto delay seconds, where I built it with a value of 10.0). I've even opened the binary files written to disk using iHex and they appear to be valid (can't be sure, but all of the values for my game config are present in the binary file). I have no idea what to try next in order to track down why this fails and my data is corrupt. Does anybody have an idea of what I'm doing wrong, or what to try next? Thanks!

[Wouter van Oortmerssen]

One error I can spot in the above code is: gameBuilder.add_version(&version);

There should be no & there. Not sure why that would even compile.

One thing you can do additionally is put a #define FLATBUFFERS_DEBUG_VERIFICATION_FAILURE before you include flatbuffers.h, and you'll get an assert on the exact field that failed.

If the verifier fails on binaries created thru flatc, there's something more major wrong.. are you maybe loading them in text mode?

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael McClenney]

Yeah, I agree something more serious is wrong here, but for the life of me I can't figure out what. I'm on OSX and created a brand new command line C++ app with nothing else but my flatbuffers code just to isolate everything. So while it's highly unlikely I'm reading that binary data generated by flatc as text (I'm using some file utilities in cocos2d, it's not my own code), at this point I don't even care about that part. I just want to focus as tightly on what's going wrong with the little code snippet above and worry about any other problems once I've gotten it working properly.

However, I can tell you that passing the address of the version struct looks like what my GameConfiguration builder expects (compilation fails without it, or I could just instantiate with new and that would work as well):

void add_version(const zz::fbs::Version *version) { fbb_.AddStruct(GameConfiguration::VT_VERSION, version); }

Here's the code for the GameConfiguration table and its builder (add_version takes a const ptr, not a reference or struct value):

struct GameConfiguration FLATBUFFERS_FINAL_CLASS : private flatbuffers::Table {
  enum {
    VT_AUTOSAVESECONDSDELTA = 4,
    VT_TARGETRESOLUTION = 6,
    VT_VERSION = 8
  };
  const zz::fbs::Float *autoSaveSecondsDelta() const { return GetPointer<const zz::fbs::Float *>(VT_AUTOSAVESECONDSDELTA); }
  const zz::fbs::Size *targetResolution() const { return GetPointer<const zz::fbs::Size *>(VT_TARGETRESOLUTION); }
  const zz::fbs::Version *version() const { return GetStruct<const zz::fbs::Version *>(VT_VERSION); }
  bool Verify(flatbuffers::Verifier &verifier) const {
    return VerifyTableStart(verifier) &&
           VerifyField<flatbuffers::uoffset_t>(verifier, VT_AUTOSAVESECONDSDELTA) &&
           verifier.VerifyTable(autoSaveSecondsDelta()) &&
           VerifyField<flatbuffers::uoffset_t>(verifier, VT_TARGETRESOLUTION) &&
           verifier.VerifyTable(targetResolution()) &&
           VerifyField<zz::fbs::Version>(verifier, VT_VERSION) &&
           verifier.EndTable();
  }
};


struct GameConfigurationBuilder {
  flatbuffers::FlatBufferBuilder &fbb_;
  flatbuffers::uoffset_t start_;
  void add_autoSaveSecondsDelta(flatbuffers::Offset<zz::fbs::Float> autoSaveSecondsDelta) { fbb_.AddOffset(GameConfiguration::VT_AUTOSAVESECONDSDELTA, autoSaveSecondsDelta); }
  void add_targetResolution(flatbuffers::Offset<zz::fbs::Size> targetResolution) { fbb_.AddOffset(GameConfiguration::VT_TARGETRESOLUTION, targetResolution); }
  void add_version(const zz::fbs::Version *version) { fbb_.AddStruct(GameConfiguration::VT_VERSION, version); }
  GameConfigurationBuilder(flatbuffers::FlatBufferBuilder &_fbb) : fbb_(_fbb) { start_ = fbb_.StartTable(); }
  GameConfigurationBuilder &operator=(const GameConfigurationBuilder &);
  flatbuffers::Offset<GameConfiguration> Finish() {
    auto o = flatbuffers::Offset<GameConfiguration>(fbb_.EndTable(start_, 3));
    return o;
  }
};

However, when I run the code with the debug verification failure flag defined, it does give me some insight. Going through the stack trace from the failed assert, it looks like it's choking on the Float table (autoSaveSecondsDelta field) for some reason. This does help give me at least a direction to look into, so thank you very much for such a prompt reply!! Unfortunately, I'll probably be back with more questions, haha.

[Wouter van Oortmerssen]

Sorry, you're right, hadn't spotted that Version is a struct. So that is correct.

Can you post the strack trace of the assert?

For the buffer generated by flatc, try converting it back to json, and see if that gives the right json back (flatc -t myflatbufferbinary.ext)

[Michael McClenney]

Thanks for the advice, here's the stack trace:

#4  0x0000000100006d55 in flatbuffers::Verifier::Check(bool) const at gitrepos_root/flatbuffers/include/flatbuffers/flatbuffers.h:1158
#5  0x0000000100006cf8 in flatbuffers::Verifier::Verify(void const*, unsigned long) const at gitrepos_root/flatbuffers/include/flatbuffers/flatbuffers.h:1165
#6  0x00000001000070d4 in bool flatbuffers::Verifier::Verify<unsigned short>(void const*) const at gitrepos_root/flatbuffers/include/flatbuffers/flatbuffers.h:1170
#7  0x0000000100006dec in flatbuffers::Table::VerifyTableStart(flatbuffers::Verifier&) const at gitrepos_root/flatbuffers/include/flatbuffers/flatbuffers.h:1366
#8  0x0000000100007107 in zz::fbs::Float::Verify(flatbuffers::Verifier&) const at gitrepos_root/flatbuffers_test/FlatBuffersTest/FlatBuffersTest/zanzo_configuration_schema_generated.h:221
#9  0x0000000100006ef8 in bool flatbuffers::Verifier::VerifyTable<zz::fbs::Float>(zz::fbs::Float const*) at gitrepos_root/flatbuffers/include/flatbuffers/flatbuffers.h:1175
#10 0x0000000100006bb4 in zz::fbs::GameConfiguration::Verify(flatbuffers::Verifier&) const at gitrepos_root/flatbuffers_test/FlatBuffersTest/FlatBuffersTest/game_config_generated.h:76
#11 0x0000000100006ae7 in bool flatbuffers::Verifier::VerifyBuffer<zz::fbs::GameConfiguration>() at gitrepos_root/flatbuffers/include/flatbuffers/flatbuffers.h:1237
#12 0x0000000100002535 in zz::fbs::VerifyGameConfigurationBuffer(flatbuffers::Verifier&) at gitrepos_root/flatbuffers_test/FlatBuffersTest/FlatBuffersTest/game_config_generated.h:111
#13 0x0000000100001e7d in main at gitrepos_root/flatbuffers_test/FlatBuffersTest/FlatBuffersTest/main.cpp:35

I get the feeling there's something I'm doing wrong on my end, but I can't continue to troubleshoot at the moment because it's holding up progress in other places. So feel free to ignore this issue. Thanks for all of your help though, when I get some time I'm going to come back to my flatbuffers issue because I'm convinced it's a better solution than rolling my own configuration data serializer.

[Wouter van Oortmerssen]

Actually, I spotted the error I think:

gameBuilder.add_autoSaveSecondsDelta(10.0);

The argument actually needs to be an Offset to a Float table. But apparently C++ is friendly enough to convert your float constant to an Offset (which is an unsigned int underneath).

[Michael McClenney]

I finally found the time to give this another shot (after 5 months!!), and you are correct. I tried your fix and my test code is working great. Now I just have to convert all my game save and config stuff over to flatbuffers. I'm going to hate life for the next couple of days, but it'll be worth it to use this over my crappy JSON solution.

Thanks again for being so responsive and helpful here.

[Wouter van Oortmerssen]

Hah, good luck!

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.