Server process stops accepting connections

[Jens Alfke]

I’ve got a CapnP-based server program running on a Raspberry Pi 4. It’s written in C++, using CapnP 0.9.1. It’s single-threaded; the main() function just sets up a listener, calls kj::NEVER_DONE.wait, and stays there.

It works fine, except that every few weeks it stops accepting connections; it either doesn’t accept the incoming socket at all or never reads from it, I can’t tell which; all I see is that the client times out and gives up after 15 seconds. Then I have to kill and relaunch the server, after which it works fine again for a while.

The server process itself isn’t hung; it’s in epoll waiting for events. I attached gdb and got this backtrace:

#0  0xb6b8563c in epoll_wait (epfd=7, events=0xbecb4210, maxevents=16,  timeout=-1)

#1  0x0052edcc in kj::UnixEventPort::doEpollWait (this=0x1fed420, timeout=-1)

#2  0x0052e58c in kj::UnixEventPort::wait (this=0x1fed420)

#3  0x00487744 in kj::EventLoop::wait (this=0x1fed4e0)

#4  0x00488460 in kj::_::waitImpl (node=..., result=..., waitScope=…)

#5  0x00488bd8 in kj::_::NeverDone::wait (this=0x72d3c0 <kj::NEVER_DONE>,  waitScope=…)

…

The last line in the log shows that it received a connection from some unknown IPv4 address which geolocates to Russia (this happens a few times a day; the server’s on my home LAN but exposes a public port and I assume these are random hackers looking for vulns.) This connection never did anything, not surprisingly since I doubt hackers are expecting anything other than HTTP, but the logs show the socket never closed. And sure enough, if I run `lsof` I see an open TCP port from that address. (Which has been open for at least 24 hours, strangely; doesn’t the disconnect idle TCP connections after 90 minutes?)

I’m not sure what to do about this. I assume CapnP is capable of handling multiple incoming connections, so this idle socket won’t block others from connecting, right? But if not, then why isn’t it accepting any connections?

—Jens

[Kenton Varda]

Hi Jens,

Have you tested whether your server is able to accept concurrent connections normally? E.g. if you open a connection with telnet or something without sending any bytes, leave that open, and then try to use your server, does it work?

What does your listen loop look like?

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/008DEC56-3B4D-4CF7-BFC3-5BECC13BE502%40mooseyard.com.

[Jens Alfke]

On Nov 9, 2022, at 6:47 AM, 'Kenton Varda' via Cap'n Proto <capn...@googlegroups.com> wrote:

Have you tested whether your server is able to accept concurrent connections normally? E.g. if you open a connection with telnet or something without sending any bytes, leave that open, and then try to use your server, does it work?

Thanks! That was indeed the problem. The telnet test showed my server would accept multiple connections at once, but would only authenticate one of them at a time, so a client that didn’t send the right handshake would block others. 

(I’m using custom code, largely copied from rpc.c++ and tls.c++, but substituting the SecretHandshake protocol (from Scuttlebutt) for TLS. When I adapted TlsConnectionReceiver::acceptLoop, I got things in the wrong order, so the tail-recursion back to acceptLoop didn’t occur until the secure handshake finished.)

I’ve fixed it now. Next question: if I want to run client connections on multiple threads, I presume that after the accept-and-handshake finishes, I would create a new thread with an event loop, then use an Executor to pass the AsyncIoStream to it, and then do the magic RPC setup on that thread. On disconnect I’d either exit the thread or return it to a pool for reuse. Most of this seems like boilerplate … are there any working examples I could steal from?

(Why would I want multiple threads when CapnP is async? Because my actual RPC methods do SQLite queries, which are blocking and sometimes slow, and I’d rather not create async wrappers for my entire complicated database API.)

—Jens

[Kenton Varda]

Hi Jens,

Sorry for the slow reply, I've been on leave.

In general, KJ async I/O objects are tied to the thread / event loop where they were created. This means you cannot pass an AsyncIoStream between threads. You can, however, pass a file descriptor. If you are careful, you could tear down the AsyncIoStream wrapping your socket, pass the socket file descriptor to another thread, and construct a new AsyncIoStream around it there.

-Kenton