how to get the bytes for a message (to send over network)

[Jason E. Aten]

I'm trying to get the byte array from a builder, but failing to do so (the diff at line 124 fails). I suspect lines 106-109 in getbytes.cpp (attached) are not correct:

    // now get bytes - somehow this is failing...  line 105                            
    ::capnp::MallocMessageBuilder message;
    Z::Builder z = message.initRoot<Z>();
    z.setZdate(zd); // correct or on? can we reuse the builder zd like this?      
    kj::ArrayPtr<word> words = messageToFlatArray(message); // correct?   

Full code attached. 

What is the proper approach to retrieving the bytes from a builder?

Intent: I want to get a contiguous buffer (e.g. pointer and size), so I can send the struct buffer through a network channel.

Thanks.

Jason

[Kenton Varda]

createAndWriteSampleZdate() allocates a MallocMessageBuilder on the stack.  The message is therefore destroyed when the function exits.  However, you are returning a Zdate::Builder from that function.  That builder is thus no longer valid -- it's effectively a dangling pointer.  A debug build of your code should crash.

readZdate() has a similar problem.

More generally, it is not guaranteed that two messages with the "same" data will actually have identical bytes.  Objects within the message can be allocated in arbitrary order.

If your goal is to write to a network channel, and you're using anything other than Windows, you can do so using writeMessageToFd().  On Windows, you should really create a subclass of kj::OutputStream which wraps the socket interface, then write to that.  If you use messageToFlatArray(), you are forcing an extra unnecessary copy in order to put the message together into one contiguous array in memory.

Note that Cap'n Proto messages are self-delimiting, so you can use writeMessageToFd() (or writeMessage() on an arbitrary OutputStream) multiple times to send a stream of messages, without needing to write the size or any other delimiter in between.

>  // is there a faster way to write the Array<word> to file?

Yes.

  write(fd, array.begin(), array.size() * sizeof(word));

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.

[Jason E. Aten]

On Fri, Oct 11, 2013 at 2:13 AM, Kenton Varda <temp...@gmail.com> wrote:

createAndWriteSampleZdate() allocates a MallocMessageBuilder on the stack.  The message is therefore destroyed when the function exits.  However, you are returning a Zdate::Builder from that function.  That builder is thus no longer valid -- it's effectively a dangling pointer.  A debug build of your code should crash.

readZdate() has a similar problem.

Fixed. Thanks for pointing these out.  I changed to passing around a MallocMessageBuilder*.

 

More generally, it is not guaranteed that two messages with the "same" data will actually have identical bytes.  Objects within the message can be allocated in arbitrary order.

Good to know. I'll decode before diffing.

 

If your goal is to write to a network channel, and you're using anything other than Windows, you can do so using writeMessageToFd().  On Windows, you should really create a subclass of kj::OutputStream which wraps the socket interface, then write to that.  If you use messageToFlatArray(), you are forcing an extra unnecessary copy in order to put the message together into one contiguous array in memory.

The network library I'm using demands a single contiguous array, so I'm afraid there's no avoiding a (potential) copy here. Here I'm not writing to a file descriptor directly; instead I need to obtain a pointer to a buffer (char*) and pass that and its length to the networking library.

Conclusion: I tried changing to passing the MallocMessageBuilder* around, but still no luck when I do messageToFlatArray on it.

  $ cat fromflatwords.ztb | capnp decode date.capnp Z                          
  ()                                                                           
  *** ERROR DECODING PREVIOUS MESSAGE ***
  The following error occurred while decoding the message above.
  This probably means the input data is invalid/corrupted.
  Exception description: expected segment != nullptr && segment->containsInterval(segment->getStartPtr(), segment->getStartPtr() + 1); Message did not contain a root pointer.
  Code location: src/capnp/message.c++:57
  *** END ERROR ***
  *** Uncaught exception ***
  src/kj/io.c++:40: requirement not met: expected n >= minBytes; Premature EOF

 full code attached in getbytes2.cpp

[Kenton Varda]

messageToFlatArray() returns Array<word>, but you're assigning it to ArrayPtr<word>.  This creates a pointer to the array, and then the array itself is destroyed because you didn't capture it.  So you have another dangling pointer.  You need to change the type of "words" to kj::Array<word>.

Also, while it usually isn't an issue for disk files, you really ought to check that write() actually wrote all the bytes you gave to it.  Technically write() is allowed to stop early, as long as it has written at least 1 byte.

-Kenton

[Jason E. Aten]

On Fri, Oct 11, 2013 at 3:10 PM, Kenton Varda <temp...@gmail.com> wrote:

messageToFlatArray() returns Array<word>, but you're assigning it to ArrayPtr<word>.  This creates a pointer to the array, and then the array itself is destroyed because you didn't capture it.  So you have another dangling pointer.  You need to change the type of "words" to kj::Array<word>.

That did it. Works now. Yay!  Thanks Kenton.

 

Also, while it usually isn't an issue for disk files, you really ought to check that write() actually wrote all the bytes you gave to it.  Technically write() is allowed to stop early, as long as it has written at least 1 byte.

Thanks for reminding me about that. Was just being lazy.

Working file attached.  If you want to include it or any part in the examples with the distribution, feel free.