Flatbuffers with GRPC: Efficient way of sending raw data without making copies

[AS]

Hi,

I have written a small client/server test application in C++ using GRPC and flatbuffers. On the server side I have a pointer to some data uint8_t *data which I want to send to the client. However, I want to send it without copying it unnecessarily. I'm currently creating a new vector<uint8_t> and passing that into  CreateMyDataDirect().Is there a more efficient way i.e can just pass my buffer directly?  

My existing code:

std::vector<uint8_t> dataChunk(data, data+size);

auto offset = CreateMyDataDirect(messageBuilder, &dataChunk);

My fbs file:

table MyData {

  data: [ubyte];

}

table MyRequest {

  id:int;

}

rpc_service StreamMyData {

  // server side streaming

  ServerSendMydata(MyRequest):MyData (streaming: "server");

}

Thanks.

[Wouter van Oortmerssen]

A FlatBuffer must be a contiguous buffer, so if you already have some data you want to store inside the FlatBuffer, it must be copied.

You can do true zero-copy with FlatBuffers and GRPC: you can first allocate a slice from GPRC, then construct the FlatBuffer inside of it (see custom allocator in grpc.h), then send it without copying. You can use CreateUninitializedVector to create space inside a FlatBuffer that can be filled by other code in zero copy style, if you can compute the length ahead of time.

That said, MyData is entirely a wrapper around your data with no additional fields. If you do not intend to store any other FlatBuffer data, it is questionable why this should be wrapped in a FlatBuffer at all?

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.