Example/tutorial for reading from sockets

[amg]

Hi everyone,

being rather new to capnp/kj, I'm currently trying to figure out some good/best practices. Unfortunately it seems to be quite difficult to find some elaborate tutorials and use-case code examples other than those on https://capnproto.org/ and the git repo itself. I'd very much appreciate if anyone could provide some good general references.

Apart from a general interest, I also have a very specific issue that I need to resolve: The software I need to implement (c++) is supposed to receive multiple (unpacked) capnp message streams over network sockets and process their (deserialized) contents.

I started with a single threaded version receiving data on a single socket in blocking mode using a kj::FdInputStream that I passed on to a MessageReader. So far, this worked quite well.

The next step was to set the socket to non-blocking mode and that is where I got stuck. The MessageReader, I guess, expects (at least) the entire next message to be available but I don't know how to decide if enough bytes have been read from the socket. Obviously I need some clever buffering mechanism in between --- only I can't figure out how.

I'd really appreciate any help on this. Some code examples would be more than welcome :)

Thank you very much in advance!

[Kenton Varda]

Hi,

There are two options for reading from non-blocking sockets:

1) You can read data into your own buffer, and call capnp::expectedSizeInWordsFromPrefix() to determine when the full message has been received. Check out the declaration and doc comment for more details:

https://github.com/capnproto/capnproto/blob/93e5be76ab22f93ef9ef75bf618af794abe59802/c++/src/capnp/serialize.h#L126-L138

2) Go all-in with KJ and use the KJ async event loop / Promise framework. For this, you'd need to call kj::setupAsyncIo() to arrange to use async I/O in the thread. You can then use the interfaces that provides to wrap your socket in a kj::AsyncIoStream, which you can then pass to the interfaces in capnp/serialize-async.h. The problem with this approach is that if you already have some other event loop you were trying to use, then you can't actually use setupAsyncIo() and instead have to adapt all the KJ interfaces to work on top of your own event loop -- probably more work than it's worth just for reading a stream of messages, though could be worth it if you wanted to use Cap'n Proto RPC.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/1c53b738-8158-4df4-aeb9-f0f8c9a1fbaa%40googlegroups.com.

[amg]

Hi Kenton,

thank you very much for your detailed reply!

As the overall processing speed will be crucial (in my aforementioned naive single-threaded approach the tcp recv buffer turned out to be a bottleneck), I'd like to borrow you expertise once more.

In your opinion, which of the two options you indicated (expectedSizeInWordsFromPrefix vs. AsyncIoStream) will be the more promising when it comes to processing speed?

Given the first option: How many bytes do I have to read before expectedSizeInWordsFromPrefix() can determine the definitive amount of bytes of the current message? Based on previous discussions on this board, I'd say at least 8 bytes (4 bytes = count of segments + N x 4 bytes = sizes of the segments)... is that correct?

Thank you very much in advance,

   amg

[Kenton Varda]

On Fri, Nov 1, 2019 at 3:22 AM amg <drrer...@gmail.com> wrote:

Hi Kenton,

thank you very much for your detailed reply!

As the overall processing speed will be crucial (in my aforementioned naive single-threaded approach the tcp recv buffer turned out to be a bottleneck), I'd like to borrow you expertise once more.

In your opinion, which of the two options you indicated (expectedSizeInWordsFromPrefix vs. AsyncIoStream) will be the more promising when it comes to processing speed?

I am not sure, but using expectedSizeInWordsFromPrefix() will give you much more control which will allow you to optimize for your use case.

 

Given the first option: How many bytes do I have to read before expectedSizeInWordsFromPrefix() can determine the definitive amount of bytes of the current message? Based on previous discussions on this board, I'd say at least 8 bytes (4 bytes = count of segments + N x 4 bytes = sizes of the segments)... is that correct?

That is correct. Once the full segment table is available, then expectedSizeInWordsFromPrefix() will return the final size of the message.

You can look at the code for expectedSizeInWordsFromPrefix() to see how it works -- it's not that long. :)

-Kenton