Flatbuffers in Ringbuffer

[T]

Hey all,

is it possible to process a "large" Flatbuffer on-the-fly, being stored only partially in a ring-buffer, that is "smaller" than the Flatbuffer itself?

TL;DR

My use case is the following:
I have a client sending a "large", say 30MB Flatbuffer via network to a server.

The Flatbuffer would mostly contain a few bytes header metadata and a binary payload array.

The server cannot load the entire Flatbuffer into RAM, it would have a sort-of ring-buffer, of say 5MB.
It needs to receive the first part of the Flatbuffer from network, read the headers metadata and according to them forward the binary payload array.

It would then continue reading from the network and forward the binary payload array "chunk by chunk" to another destination (indicated by the already read headers).

Is this possible with Flatbuffers or do I need to first receive the entire buffer?

If possible, would that also be possible with gRPC as communcation protocol over network?

[mikkelfj]

I can only answer for C in with the FlatCC tool. Likely you can do something similar in C++.

FlatCC actually uses a ring buffer when building the buffer initially, but it is not quite the same.
FlatCC has a fairly elaborate macro system to implement its features, but they generally rely on a few low level accessor methods. It is possible to override these methods to for example do a check for buffer presence and then proceed if present. If not, it is a bit harder because you get a lot of error handling to process unless you add some sort of co-routine framework. Also I suggest using a vtable (not a flatbuffer vtable) instead of a ring buffer. You can then check if a segment of the buffer is present using a simple lookup. Another concern is that flatbuffers can move from table to subtable using a single pointer by reading the offset to the subtable and add it to the table pointer (roughly). If you need to do extra lookups you need an extra context pointer for that. You can do this by adding a global variable, or you can align each buffer segment to a memory page and use address magic to find the context pointer.

Altogether it is a fairly big task, but definitely possible.

As to gRPC, FlatCC avoids gRPC because it is a huge installation with dependencies that is not in the spirit of FlatCC. Here I recommend using MQTT instead.

On a related note, there is a longstanding idea of StreamBuffers which write buffers in a different order that would allow partial buffers to be sent and processed remotely although not necessarily read at that point. However, if you add your ring buffer concept, these would be good match. It is not difficult to develop stream buffers, but there are many places that need careful attention.

Mikkel