Deserializing Capnp Messages - Getting an exception

[noah...@gmail.com]

HI,

Using capnproto-c++-0.6.1 I am trying to  serialize a Capnp message to bytes to send over a multicast socket. I am getting an error when trying to deserialize. 

Here is what I am trying to do. 

auto flatArray = capnp::messageToFlatArray(message);
std::cout << "flatArray size : " << flatArray.size() << std::endl;  // flatArray size : 17


auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
std::cout << "bytesToSend size : " << bytesToSend.size() << std::endl;  // bytesToSend size : 136

// The following works fine
capnp::FlatArrayMessageReader msg(flatArray);

// But the following throws an exception: capnp/serialize.c++:43: failed: expected array.size() >= offset; Message ends prematurely in segment table.

char result[bytesToSend.size()]; // what i would like to write to my UDP socket
auto words = kj::heapArray<capnp::word>(bytesToSend.size()/ sizeof(capnp::word));
std::cout << "words size : " << words.size() << std::endl; // words size : 17 (as expected)

memcpy(words.begin(), &result, bytesToSend.size());
capnp::FlatArrayMessageReader msg(words); // Exception!!

 Note - this is similar to this previous topic : https://groups.google.com/forum/#!topic/capnproto/OcYhwDfB4vE but the comments there didn't get me going. 

Thank you very much for your time.

[Kenton Varda]

Hi Noah,

I think you're missing a line like:

    memcpy(result, bytesToSend.begin(), bytesToSend.size());

So `result` never gets initialized, so at the end you're trying to parse uninitialized data. :)

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

[Noah Keen]

Hi Kenton. 

You are of course absolutely correct. If I memcpy bytesToSend into result the example works without exception. Much thanks for catching that one. I have a followup question though. 

If I use my code above to serialize my message, my client (in pycapn) fails to deserialize my message. It throws an error - No Root Node Found.

        // My client fails to deserialize this packed on receipt

        auto bytesToSend = capnp::messageToFlatArray(message).asBytes();

        sendto(sock, bytesToSend.begin(), bytesToSend.size(), 0,
                   (struct sockaddr *)&destinationAddress, sizeof(struct sockaddr_in));

 This could be an issue with my client, but with some trial and error I found if I use the following code to serialize my message my client works great:

        kj::byte result[capnp::messageToFlatArray(message).asBytes().size()];
        kj::ArrayPtr<kj::byte> bufferPtr = kj::arrayPtr(result, sizeof(result));
        kj::ArrayOutputStream arrayOutputStream(bufferPtr);
        capnp::writeMessage(arrayOutputStream, message);

        sendto(sock, arrayOutputStream.getArray().begin(), arrayOutputStream.getArray().size(), 0,

                   (struct sockaddr *)&destinationAddress, sizeof(struct sockaddr_in));

Looking at raw bytes that get written I have confirmed the two approaches differ at the packet level. The second, working approach, has 0x10 in the 4th byte, but that's the only difference I see.

My question is, what is capnp::writeMessage(arrayOutputStream, message) doing that my first approach is not? Why can't I just get the bytes and send?

Is my second approach the correct way? Is there a better technique? It seems a little clunky to me - especially the first line where i use

 

        kj::byte result[capnp::messageToFlatArray(message).asBytes().size()];

Thanks again - I appreciate your time.

Noah

[Kenton Varda]

Hi Noah,

On Sat, Jun 16, 2018 at 4:17 PM, Noah Keen <noah...@gmail.com> wrote:

Hi Kenton. 

You are of course absolutely correct. If I memcpy bytesToSend into result the example works without exception. Much thanks for catching that one. I have a followup question though. 

If I use my code above to serialize my message, my client (in pycapn) fails to deserialize my message. It throws an error - No Root Node Found.

        // My client fails to deserialize this packed on receipt
        auto bytesToSend = capnp::messageToFlatArray(message).asBytes();
        sendto(sock, bytesToSend.begin(), bytesToSend.size(), 0,
                   (struct sockaddr *)&destinationAddress, sizeof(struct sockaddr_in));

messageToFlatArray() returns an Array<word>, but calling .asBytes() on it returns ArrayPtr<byte>. Array<> owns its contents while ArrayPtr<> points to an array owned by someone else. Since you don't assign the Array anywhere, it is destroyed at the end of the line. So, `bytesToSend` ends up being a dangling pointer, and the bytes it points to could be overwritten with garbage at any time.

To fix this, you will need to assign the word array to a local variable, and then call `.asBytes()` on a separate line.

On another note, keep in mind that messageToFlatArray() makes a copy. You could avoid this copy by using sendmsg() instead of sendto(), which allows you to do a gather-write with msg_iov. You would call message.getSegmentsForOutput() to get an array of arrays of words to write. You also need to add the segment table to the front of this; see how writeMessage() is implemented in the capnp repo in c++/src/capnp/serialize.c++. Note that if you always have exactly one segment (e.g. because you made sure to configure your MessageBuilder to always allocate enough space), then the segment table is always 8 bytes: four bytes of zeros, followed by a four-byte little-endian integer indicating the size of the first segment in words.

-Kenton

 

[Noah Keen]

Nice! Thank you. For you your assistance and the lib! 

I'll try to remember to post my finished working solution here once I know it works.

 

On Thursday, June 14, 2018 at 7:22:21 PM UTC-5, Noah Keen wrote: