Can I do better?

[Jitesh Khandelwal]

I am trying to implement a middle layer which serialises some data from a data structure into a buffer, which is owned by and whose contents are written on the wire by an outer layer.

Currently, my code looks as follows
----------------------------------------------------------------------------------------------------------------------------------------
constexpr size_t N = 2048;
char arena_buffer[N]{0};
auto arena = arrayPtr(reinterpret_cast<word*>(arena_buffer), N / sizeof(word));

Encode (some_data_structure, arena, buffer)
{
    auto builder = MallocMessageBuilder(arena);
    ....
    // Step 1. set fields from some_data_structure using builder into arena
    ....

    // Step 2. convert to flat array (this allocates another array)
    auto message = messageToFlatArray(builder);
    auto bytes   = message.asBytes();

    // Step 3. copy to destination
    memcpy(buffer, bytes.begin(), bytes.size());
}
----------------------------------------------------------------------------------------------------------------------------------------

My question is, Can I do better?

My understanding is that the extra allocation in Step 2, is because of the segment framing protocol ?

Can I bypass it somehow, as I am sure that the arena is big enough for my message and there'll be only 1 segment ?

And if yes, then may be I can use the buffer as the arena, as in build directly into the output buffer?

Thanks,
Jitesh

[Kenton Varda]

Hi Jitesh,

Yes, you can avoid the allocation as follows:

1. Construct a kj::ArrayOutputStream that uses your destination buffer as the output.

2. Use capnp::writeMessage() to write the message to that stream.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/a6378e3d-fd2e-4009-8cf4-595011170dc9n%40googlegroups.com.

[Jitesh Khandelwal]

Thanks Kenton, this is working well. Now the allocation in messageToFlatArray is not happening.

Now, is it possible to build the message in the buffer directly ? Currently, I'm building in arena and then calling writeMessage to write into the buffer.
----------------------------------------------------------------------------------------------------------------------------------
auto builder     = MallocMessageBuilder(arena);
...
...
auto message = arrayPtr(reinterpret_cast<byte*>(buffer), N);                                                                                                 
auto aos     = kj::ArrayOutputStream(message);                                                                                                               
capnp::writeMessage(aos, builder);

----------------------------------------------------------------------------------------------------------------------------------

[Kenton Varda]

Hi Jitesh,

Yes, this is possible, but a bit harder.

Instead of using MallocMessageBuilder, you would need to define your own subclass of capnp::MessageBuilder that returns the buffers you want to use.

You will also need to keep track of framing on your own -- that is, keep track of the number of segments and their sizes. Normally Cap'n Proto will write a prefix on the message called the "segment table" which specifies the length of each segment. But, the size of that table depends on the number of segments, which normally isn't known in advance, so you wouldn't know how much space to leave at the start of your message for it. That said, if you are sure your message will always fit in the first segment, then the segment table is not that useful anyway.

Also, if you're manually managing segments, then on the receiving end you'll need to use capnp::SegmentArrayMessageReader, or write a custom MessageReader subclass.

-Kenton

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/b09290d5-e2d3-4711-b309-9bcdaaade8e5n%40googlegroups.com.