Does flatbuffer have limit on a single buffer field?

[Chunting Jiao]

Hi all,
Now I'm working on flatbuffers, reading from a camera and writing the data to flatbuffer.
My question is when the size of a single buffer reached about 500 MB, the verifier failed; but when the buffer was small, such as 100 MB, the verifier succeeded, and the data could be write into binary file correctly.
From the documents, I learn that the limit of a single buffer field is 2 GB, anyone has tested the limit (2 GB) works well?
Maybe there are some encoding problems in my program, but firstly I'd like to confirm this error is not caused by the large buffer size.

Thanks,
Chunting

[mikkelfj]

Would you be able to run the verifier from the C implementation flatcc. It might produce a different result, or different insights into the problem?

https://github.com/dvidelabs/flatcc#reporting-bugs

https://github.com/dvidelabs/flatcc#debugging-a-buffer

[Wouter van Oortmerssen]

If you #define FLATBUFFERS_DEBUG_VERIFICATION_FAILURE before including any FlatBuffers related headers, you'll get an assert whenever the verifier fails, whose stack-trace should tell you exactly what check failed an on what field etc.

There is nothing about FlatBuffers that does not allow it to reach 500MB. However, the verifier has these max_depth = 64, max_tables = 1000000 default arguments to its constructor that you may be hitting, try increasing those numbers and see if the problem goes away.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chunting Jiao]

Good tips.

Could you please give me more information about max_depth = 64, what does it mean?  max_tables = 1000000 means it allows a maximum of 1000000 tables in one single buffer.

Thanks,

Chunting

--
You received this message because you are subscribed to a topic in the Google Groups "FlatBuffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/flatbuffers/JtDGnBPx9is/unsubscribe.
To unsubscribe from this group and all its topics, send an email to flatbuffers+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

焦春亭
 
 

[Wouter van Oortmerssen]

max_depth means the maximum amount of table nesting. You're only likely to hit this if you store very deep trees, or are emulating linked-lists in FlatBuffers :)

[Chunting Jiao]

Hi, Wouter,

I really hit the limit of max_tables.

But I've another inquery why after adding #define FLATBUFFERS_DEBUG_VERIFICATION_FAILURE, I couldn't get any assert?

Do I need to do some additional work?

Thanks,

Chunting

--

焦春亭
 
 

[Wouter van Oortmerssen]

You simply need to pass your own max_tables to the FlatBufferBuilder constructor.

Pass e.g. INT_MAX and you will guaranteed never hit it :)

[Chunting Jiao]

Thanks for your reply.

I checked out the construction of FlatBufferBuilder. There is no argument given that corresponds to INT_MAX or max_tables_.

I only changed the parameter _max_tables_  in Verifier().

explicit FlatBufferBuilder(size_t initial_size = 1024,

Allocator *allocator = nullptr,

bool own_allocator = false)

: buf_(initial_size, allocator, own_allocator), max_voffset_(0),

nested(false), finished(false), minalign_(1), force_defaults_(false),

dedup_vtables_(true), string_pool(nullptr) {

offsetbuf_.reserve(16); // Avoid first few reallocs.

vtables_.reserve(16);

EndianCheck();

}

/// To expand the capacity of a single buffer, _max_tables is changed to 10000000 from 1000000;

Verifier(const uint8_t *buf, size_t buf_len, uoffset_t _max_depth = 64,

uoffset_t _max_tables = 10000000)

: buf_(buf), end_(buf + buf_len), depth_(0), max_depth_(_max_depth),

num_tables_(0), max_tables_(_max_tables)

#ifdef FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE

, upper_bound_(buf)

#endif

{}

Thanks,

Chunting

--

焦春亭
 
 

[Chunting Jiao]

Hi,

Just pass two parameters to Verifier construction.

/// To expand the capacity of a single buffer, _max_tables is set to 10000000

flatbuffers::uoffset_t _max_depth = 64;

flatbuffers::uoffset_t _max_tables = 10000000;

flatbuffers::Verifier verifier(buf, bufsize, _max_depth, _max_tables);

OK = OK && grl::flatbuffer::VerifyKUKAiiwaStatesBuffer(verifier);

This error is fixed successfully.

Thanks,

Chunting

--

焦春亭