PackedMessageReader segfault for larger buffer sizes

[Jonathan Hollocombe]

Hi,

I am using Cap'n Proto to serialise a data tree which is then sent as a blob to a client to be unpacked. I was using something similar to:

kj::VectorOutputStream out;
capnp::writePackedMessage(out, builder);
auto arr = out.getArray();

char* buffer = (char*)malloc(arr.size());

memcpy(buffer, arr.begin(), arr.size());

and

kj::ArrayPtr<kj::byte> buffer(reinterpret_cast<kj::byte*>(const_cast<char*>(bytes)), size);
kj::ArrayInputStream in(buffer);

capnp::PackedMessageReader reader{in};

This is working for small tree sizes but segfaults when I get to larger number of elements.

I've found a solution that works by switching to messageToFlatArray and FlatArrayMessageReader but this dramatically increases the amount of data that needs to be sent, especially for small trees.

I've also found that writing the buffer to a file and then using PackedFdMessageReader also works but i'd rather avoid creating files unnessarily.

Is there a way to use the PackedMessageReader with a byte buffer?

Cheers,

Jonathan.

p.s. my example code can be found at: https://github.com/jholloc/capnp_test

[Kenton Varda]

Hi Jonathan,

I think the problem is with this function:

std::shared_ptr<capnp::MessageReader> uda_capnp_deserialise(const char* bytes, size_t size)
{
    // ArrayPtr requires non-const ptr, but we are only using this to read from the bytes array

    kj::ArrayPtr<kj::byte> buffer(reinterpret_cast<kj::byte*>(const_cast<char*>(bytes)), size);
    kj::ArrayInputStream in(buffer);

    return std::make_shared<capnp::PackedMessageReader>(in);
}

Here, you are returning a PackedMessageReader that is wrapping an ArrayInputStream, but the ArrayInputStream is allocated on the stack, so is no longer valid once the function returns. The stream needs to remain valid until the PackedMessageReader is destroyed. Cap'n Proto can actually lazily read segments of the message from the stream; it doesn't read the entire thing in the constructor.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/ae6cafe1-f6f8-4eca-bab2-9ea3e73c3c8fn%40googlegroups.com.

[Jonathan Hollocombe]

Hi,

Thanks for this! You've saved me much head-scratching -- it didn't occur to me to check if the PackedMessageReader was borrowing or taking ownership of the input stream.

I've create a custom reader class that holds both an ArrayInputStream and a PackedMessageReader and this seems to be working perfectly.

Cheers,

Jonathan.