How to organize multi-threading communication to server using C++ RPC?

[leko...@gmail.com]

Question related only Linux for my tasks.

I want to use EzRpcClient from several threads to send single server in another process.

I've tried to create EzRpcClient in one thread which I want to use only as messages sender thread. And then in each other threads I've created WaitScope and Capability (client->getMain<ClassName>). Then I create Request. And at send step I receive exception saying that there no Event Loop in current thread.

How to correcly do communications from threads?

From my userstanding there several options but I don't know how to implement them.

1. Using one thread only as message sender thread.

So Sender thread have some kind of infinite Event Loop. 

I read somewhere here at Google Group old post about there() method of Promise. If it is exists in current version, I think it send event to Sender thread and wait until it finish processing so Worker thread is in blocking for time period.

If there() is not blocking how to make clone of all message data since I read in KJ style guide that Cap'n Proto does not own data so when Sender thread try to send data it no longer there.

Pros. One EzRpcClient instance and so one TCP connection to server.

Cons. Since only one message  send in time each Worker thread must wait until it's message will be sent.

2. Using one thread only as one-way message sender thread

In this case when I preparse Requests I will make clone of them and data it points to (say how to do this correctly in KJ style) and put them in the queue. Sender thread poll that queue from time to time and sends prepared Requests.

So questions in this case. How to correctly to clone request and its data and pass it ownship to Sender thread? In that case Worker can safely delete all data once it put it's clone to queue.

Pros. Worker does not wait answer so performance impact is only related to cloning data.

Cons. Workers can not receive any anwsers from server.

3. Using one EzRpcClient per thread.

In this case each EzRpcClient instance stored in thread local storage (e.g. using std::thread_local). Using each client instance is like in single threaded app. Easy.

Pros. Easier implementation.

Cons. Worker threads must wait for answer even I don't needed it (fire-and-forget). This is because messages send in Event Loop and it created in current thread so it will not work until I execute it using Request::wait(WaitScope).

How to implement each type in KJ style and what is considered best practice?

I have two types of communications. One-way is for CPU intensive tasks in Worker threads. And two-way in other Manager threads. I don't need to mix two communication types in single thread.

[Kenton Varda]

Hi there,

Sorry, but the KJ event loop is singe-threaded. If you have multiple threads, you have to create a separate event loop (or separate EzRpc{Client,Server}) in each one. Cap'n Proto RPC objects are intrinsically tied to the event loop in which they were created, so you cannot share these objects between threads. To communicate between threads, you'll have to use a pipe or eventfd with the low-level UnixEventPort, since the promise framework is not thread-aware.

In the future I'd like to have builtin support for threads in such a way that you can use Cap'n Proto RPC to send messages between threads without sending them through a pipe, but that's not there yet, and it still wouldn't do what I think you're asking for here.

Also note that there's no way to have multiple threads servicing the same connection, but you could potentially have multiple threads accept()ing from the same listen socket. This, however, will require you to write separate code to do the accept()ing, and then import the connections into the event loop using LowLevelAsyncIoProvider.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.