Accept connection in new process

[pepij...@gmail.com]

Hey,

I'm writing an API for some shared libraries that are written with a "binary" use in mind.

They use a ton of global state and are not thread safe.

Of course as a server, multiple users could try to use the server at the same time, these libraries can't deal with that. What's more they also like to segfault sometimes.

So what I'd like to do is have these shared libraries completely in their own address space.

What would be the best way to do that?

One thing I thought could work is have a blocking accept loop, and then fork the handling of the connection. Not sure how this would work with kj.

The other solution would be that you launch a process in the method handler that runs its own capnp server, and then proxy calls to it. Seems like it adds more overhead.

Regards,

Pepijn

[Kenton Varda]

Hi Pepijn,

Yeah, forking won't work well. I don't recommend that. One problem is that if you fork() with an event loop running, both the parent and child processes will end up with the same epoll FD and so will receive each other's events, which will likely lead to chaos.

So I recommend spawning a subprocess and talking to it using RPC over a unix socket. Yes, it adds some overhead, but really not that much.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/99fcf2cf-e6e4-4918-8734-22b7f85cfb85n%40googlegroups.com.

[pepijn de vos]

Hi Kenton,

I got back to this problem and tried implementing the fork idea but ran into another silly problem where the C++ templates are smarter than me it seems.

You recommended to not use the EzRPC server, but there is a great lack of examples of other ways to go about it.

I found a Rust example that uses the low-level API but of course not exactly the C++ one: https://github.com/capnproto/capnproto-rust/blob/master/capnp-rpc/examples/calculator/server.rs

I found some code that's copy-pasted all over using github search that seems to use the lower level API: https://github.com/Nickan/nickan.github.io/blob/b0eb31c33a3b6720606974f9576e663f3b7852ca/drawio/etc/sandstorm/server.c%2B%2B#L420-L422

However, I cannot quite get it to work.

kj::AsyncIoContext ctx = kj::setupAsyncIo();

auto stream = ctx.lowLevelProvider->wrapSocketFd(clientFd);

auto network = capnp::TwoPartyVatNetwork(*stream, capnp::rpc::twoparty::Side::CLIENT);

kj::Own<kj::Filesystem> fs = kj::newDiskFilesystem();

const kj::Directory &dir = fs->getCurrent();

auto sim = kj::heap<SimulatorImpl>(dir);

auto rpc = capnp::makeRpcServer(network, sim);

So I set up asyncio, wrap my unix socket FD into a stream, use that to create a network (client or server??)

The problem is the call to makeRpcServer. SimulatorImpl is a ::Server subclass.

This seems to match the random copy-pasted example, but gives compile errors that it can't convert sim to whatever is expected.

I tried reading the source code, which tells me it expects a restorer or a bootstrap, but the restorer is deprecated, but also what EzRPC seems to be using.

I tried looking at the tests but that appears to do the same thing, just pass it an owned instance of the implementation??

https://github.com/capnproto/capnproto/blob/e5af75ff0513e6d971fa7b48e8108427766e51f9/c%2B%2B/src/capnp/rpc-test.c%2B%2B#L1273-L1279

I am at a loss.

Pepijn

On Thu, May 6, 2021 at 8:22 PM Kenton Varda <ken...@cloudflare.com> wrote:

I don't recommend using EzRPC. That interface turns out to be too restrictive.

To set up a KJ event loop, use kj::setupAsyncIo().

Then to initiate or accept Cap'n Proto RPC connections, use capnp::TwoPartyClient / capnp::TwoPartyServer.

For FD passing, create a capnp server object that overrides the virtual method `kj::Maybe<int> Capability::Server::getFd()` to return an FD number. Then on the Client objects pointing to that server, you can call `getFd()` to get that file descriptor -- even from another process, as long as the connection between the two is a unix domain socket.

-Kenton

On Thu, May 6, 2021 at 1:07 PM pepijn de vos <pepij...@gmail.com> wrote:

Thanks for the suggestions.

This is still for the simulation server, so each connection is pretty expensive anyway, and it's unlikely there will be more in flight than the machine has cores available. It might even make sense to have a pool as you suggest, and just reject connections if there are no cores available. Concern is that you need to supervise these pools in case they crash. With fork you can just launch a new process. Any pointers to useful functions for this kinds of hacks would be appreciated. I don't have a clue how I'd go about obtaining and sending file descriptors over RPC and making new servers out of them. I should probably start by studying EzRPC and kj::LowLevelAsyncIoProvider.

Pepijn

On Thu, May 6, 2021 at 7:48 PM Kenton Varda <ken...@cloudflare.com> wrote:

Ohhh, sorry, I misread. If you haven't started the KJ event loop until after the fork, you are probably OK.

You can definitely construct a KJ socket from a raw socket using kj::LowLevelAsyncIoProvider.

But if you're OK with the entire connection being handled in the isolated process, there are some other options too:

- Have a pool of processes that are all waiting to accept from the same file descriptor. Each connection will only be accepted by one of the processes. Make sure that process doesn't accept a new connection until the old connection is done.

- Cap'n Proto supports sending file descriptors between processes in RPC messages. So you could have one process that is accepting connections, but each time it does, it sends the file descriptor to some other processes via an RPC. Then that process actually handles the connection.

These options are both more complicated than the simple fork() approach you mentioned, so aren't necessarily better. But fork() is pretty expensive so it might turn out reusing processes has some performance benefit.

-Kenton

On Thu, May 6, 2021 at 12:39 PM pepijn de vos <pepij...@gmail.com> wrote:

Hi Kenton,

Agree forking an eventloop will lead to disaster, which is why I suggested having a blocking accept loop, and then fork and start the event loop in the new process. But I'm not sure if it's at all possible to convert a raw unix socket to a jk socket and then give it to an eventloop to handle the connection.

RPC to a process certainly requires less hacks.

Pepijn

[pepijn de vos]

Ah I solved it... by not storing sim in a variable. Or probably kj::mv would have done the job too.

It's kinda unfortunate the error you get just tells you it's the wrong type, rather than.. hey you should move the thing.

Onwards!!

Pepijn

[Kenton Varda]

Yeah, C++ error messages are the worst. :(

You kind of have to get used to knowing that "can't convert" errors are usually because you forgot to kj::mv...

-Kenton