FlatBuffer 2GB limitation

[Vibhor]

After going through the flatbuffers.h I noticed that there is a limit of 2GB for a flatbuffer. Is there a way of overcoming this? We might have a need to define flat buffers for objects bigger than 2GB... 

Thanks

[Wouter van Oortmerssen]

This would be possible by changing the typedefs for uoffset_t and soffset_t in flatbuffers.h to be 64bit types. This would be a format fork however, i.e. code compiled with such setting would be incompatible with any other software / data / implementations of the current FlatBuffers.

It's a bit harder for the other languages, e.g. Java doesn't have typedefs, so it be a fair bit of rewriting of occurrences of int to long.

On Thu, Sep 3, 2015 at 3:39 AM, Vibhor <vibhora...@gmail.com> wrote:

After going through the flatbuffers.h I noticed that there is a limit of 2GB for a flatbuffer. Is there a way of overcoming this? We might have a need to define flat buffers for objects bigger than 2GB... 

Thanks

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vibhor Aggarwal]

Thanks! Yes, I understand that. We would only be using it in our c++ code

[Vibhor Aggarwal]

I tried out by modifying the header and running the tests which come with Flatbuffers. These tests failed on the following assertion:

include/flatbuffers/flatbuffers.h:894: bool flatbuffers::Verifier::Check(bool) const: Assertion `ok' failed.

I am using GCC 5.2 on RHEL 6.6

Do I need another change besides the typedef modification?  I set the following in flatbuffers.h:

typedef uint64_t uoffset_t;

typedef int64_t soffset_t;

typedef uint16_t voffset_t; 

The tests passed without this modification.

[Wouter van Oortmerssen]

If you look at the stack trace, it is failing on the verification of a pre-existing FlatBuffer loaded from disk, one which was written with 32bit offsets.

Like I said, if you make the above changes, you create an entirely incompatible format. If you wanted these tests to succeed, you'd have to regenerate the files it loads using a flatc that has been compiled with your change.

It has passed all tests before that one, where it creates and then read a FlatBuffer from scratch, so it appears to work.