semi writeable buffer

[Lee Morgan]

Hi,

Are there plans to have a writeable flatbuffer?

By writeable I mean only writes that wouldn't cause a reallocation. Others would error/assert.

I see theres a table->SetField() but no supporting output in the generated header.

Presumably Vector/Strings would need a second length field for the valid data.

But it would be useful to serialize the data layout but then update the existing layout.

Thanks,

Lee

[Wouter van Oortmerssen]

The plan was originally to allow mutation, but since that got really complicated with resizing, that plan was shelved.

We still have plans to provide a built-in method for mutation, which could go in one of two directions:

1) Allow mutation of scalar fields, and strings/vectors whenever they don't grow. Maybe allow "simple" reallocation by appending new objects at the end, which would generate garbage in the buffer. This is not universally useful, but at least it keeps with the theme of efficiency that FlatBuffers is all about.

2) Allow the code generator to output additional function to turn a read-only FlatBuffer into a mutable data structure, and back into a FlatBuffer. This is more universally useful, but would entail a lot of object allocation, which is very much against the philosophy of FlatBuffers.

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lee Morgan]

My vote would be for definitely be for (1). I'd like to use flatbuffers to send data over the network and skip the serialization in optimized cases where I can be careful to set non default values initially to trigger the allocations.

If (2) was implemented I'd prefer it was explicit (as you describe) that any allocations would occur. 

I guess they're not mutually exclusive features, (1) seems quicker to implement!

Cheers

[Wouter van Oortmerssen]

I think I agree. I should probably prioritize 1) over 2). 

Any other opinions welcome.

[Tim Culver]

Does (2) imply generating a second set of classes? That seems another disadvantage. 

If I had (1), I would presumably be able to manually "collect the garbage" by simply making a fresh copy of the whole message. I think (1) seems like an adequate escape-valve strategy if the data is almost read-only.

[Wouter van Oortmerssen]

Yes it would need completely separate definitions.

I agree. Automatically generating code to generate a "clean copy" from a modified flatbuffer would be a solution that's more in line with the FlatBuffers direction.

[paul Bandler]

Is there any near-term plan to add support to mutate scalar values as described in this post that is now nearly a year old?  

I have a use-case that would benefit from it.  I want to use flat-buffers to encode a complex aggregate that will eventually be written to a relational database in a highly normalised form.  Writing a component to the table results in a unique id and timestamp value, which I need to reflect in the flat-buffer representation that will continue to be held as a cached representation of what is now in the database.

[Wouter van Oortmerssen]

Yes, in fact the first step is already in.

The functionality mentioned in 1) above is in:

https://github.com/google/flatbuffers/commit/3ec5dddb00dac9ae72a2f6808ed28b06cb64cdab

It allows you to set scalar fields in an existing FlatBuffer. Not sure if that's enough for your use case.

That doesn't cover things that can change size, such as strings. That functionality is currently being worked on, and will come as part of the generic reflection functionality. I expect this to land within the next 2 weeks or so.

[paul Bandler]

Great - but from a quick look at the github change, is it only available for C++ presently?

[Wouter van Oortmerssen]

Yes.

I have no ETA for other languages.

[Lee Morgan]

Cool. I'll have to try this. Thanks!