Determine the size of a buffer I decoded?

[Trond Norbye]

Hi,

I've just started looking into Flatbuffers, so please bear with me if I've missed something obvious here. I'm currently looking into using flatbuffers as a serialization format for the payload in a networking protocol. My initial tests so far looks good, but I have some situations where using flatbuffers would result in a significant increase of memory copying (adding a "blob" that may be up to ~20MB). 

In my current protocol we've got our own serialization format, and we're using an IO vector so that I don't copy the blob at all. From what I can see I don't have that possibility in flatbuffers, so I'm currently thinking about just putting the blob behind the flatbuffer-encoded data. I know the entire "frame size", but is there an easy way to determine the size of the flatbuffer buffer when I'm decoding the message?

Let's say I've got something like:

table Foo {
    ...
}

root_type Foo;

And then in my code I've got something like:

auto foo = GetFoo(payload.data());

How do I determine the number of bytes the response occupies? or would I be better off to store the "value size" in the object? (by doing so I could add other "blobs" later on....)

Cheers,

Trond

[Wouter van Oortmerssen]

First, it is possible to implement a no-copy strategy with FlatBuffers: FlatBufferBuilder can take an allocator argument, that allows you to make it point to whatever destination memory you already have.

Second, the serialized data does not store its length internally, it is assumed that whatever code received the buffer also knows its size, thus that can propagate that information if needed.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Trond Norbye]

On Wednesday, May 6, 2015 at 7:07:14 PM UTC+2, Wouter van Oortmerssen wrote:

First, it is possible to implement a no-copy strategy with FlatBuffers: FlatBufferBuilder can take an allocator argument, that allows you to make it point to whatever destination memory you already have.

Ah, thanks. I'll take a deeper look into the API.

 

Second, the serialized data does not store its length internally, it is assumed that whatever code received the buffer also knows its size, thus that can propagate that information if needed.

ok. I thought it could be calculated by looking at the data it decoded (since the accessory methods may grab the data).

Thanks for the response. I'll dig a bit deeper into the API and code.

Cheers,

Trond 

[thomasa...@gmail.com]

It's great to know that there are ways of converting to and from flatbuffers without having to know the size of the buffer and the types contained within however there are other use cases where knowing the size will be useful. I for one would like to know how to get the size of the encoded / decoded buffers.

[Wouter van Oortmerssen]

Since those last message, two kinds of new functionality have arrived for dealing with size:

1) You can compute the size of a message using the verifier. See FLATBUFFERS_TRACK_VERIFIER_BUFFER_SIZE in flatbuffers.h for an example.

2) There's now built-in functionality to prefix a buffer with a size, see e.g. FinishSizePrefixed and GetSizePrefixedRoot

To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.

[German Espinosa]

I can't find GetSizePrefixedRoot in the javascript generated file... is this functionality available for JS?

To unsubscribe from this group and stop receiving emails from it, send an email to flatb...@googlegroups.com.

[mikkelfj]

If there is no function, you can access the first 4 bytes of the buffer directly and read them as buf[0] + (buf[1] << 8) + (buf[2] << 16) + (buf[3] << 24)
where this is pseudo-code. Keep in mind that not all buffers are size prefixed.

[Wouter van Oortmerssen]

Rather than doing that by hand, it be nice to port the function from one of the languages that does support it, and make a PR :)

To view this discussion on the web visit https://groups.google.com/d/msgid/flatbuffers/d6ae2b28-22e8-4dae-bb57-c1f3f750cb23%40googlegroups.com.