Xcode hits SIGUSR1 signal when I use an Executor

[Jens Alfke]

I have Cap’nProto RPC (in C++, on macOS) working well on the main thread, but I want to move it to a background thread for various reasons. However, my process is now raising SIGUSR1 signals, which the debugger breaks at, preventing the code from working.

I’m following the directions in the docs: in my background thread’s main function I call kj::setupAsyncIo(), then I call kj::getCurrentThreadExecutor() to get the Executor object. My main thread uses that Executor reference to pass lambdas that will run on the background thread and do the RPC stuff.

My immediate problem is that as soon as the main thread schedules a call with the Executor, LLDB (in Xcode) takes control, letting me know that the background thread has raised a SIGUSR1 signal. The backtrace is:

* thread #5, stop reason = signal SIGUSR1
* frame #0: 0x00000001ab672eb0 libsystem_kernel.dylib`poll + 8
frame #1: 0x0000000100bb0c78 `kj::UnixEventPort::PollContext::run(int) + 40
frame #2: 0x0000000100bb09ec `kj::UnixEventPort::wait() + 352
frame #3: 0x0000000100ba4710 `kj::EventLoop::wait() + 40
frame #4: 0x0000000100ba4c90 `kj::_::waitImpl(kj::Own<kj::_::PromiseNode>&&, kj::_::ExceptionOrValue&, kj::WaitScope&) + 436
frame #5: 0x0000000100ba52fc `kj::_::NeverDone::wait(kj::WaitScope&) const + 76
…

If I continue from this, the code seems to be deadlocked: nothing happens until I kill the process.

If I run without LLDB, the process gets past this point; the Executor seems to behave correctly and my lambda starts running on the background thread. However, working without a debugger is unacceptable; there’s no way I’m going to be able to troubleshoot the rest of my code without LLDB.

Is there a workaround to this? Like, a way to get LLDB to ignore the signal, or to get Cap’nP to use something other than a signal? I’m guessing I haven’t run into this before because I’ve only been using a single thread and for whatever reason Cap’nProto only uses this signal for inter-thread messaging … ?

—Jens

[Jens Alfke]

TL;DR: Can the two streams created by kj::newTwoWayPipe() be used on different threads? It kind of appears not.

I’ve found a workaround, the LLDB command
process handle --stop false SIGUSR1
Unfortunately adding it to my .lldbrc file does nothing; I have to enter it by hand every time I start the process.

The next roadblock is that my unit tests create a client and a server object, then connect them together by calling kj::newTwoWayPipe() and giving one end of the pipe to each. This worked fine in a single thread. However, now one end (an AsyncIoStream) gets passed into the new background thread where the client object lives. I get an exception
"expected threadLocalEventLoop == &loop || threadLocalEventLoop == nullptr; Event armed from different thread than it was created in. You must use Executor to queue events cross-thread.”

From this and the backtrace it looks as though when I write to this end of the pipe, it wants to directly notify the other end, which won’t work because it’s the wrong thread for that. I was hoping that the streams would use normal Unix I/O, since the comment about the TwoWayPipe class says "Typically backed by socketpair() system call”.

So how do I set up streams to do I/O between threads?

—Jens

[Kenton Varda]

Hi Jens,

Indeed, KJ uses SIGUSR1 to communicate between threads. It sounds very obnoxious that your debugger insists on breaking on this signal. The signal is not a problem, it's a normal part of KJ's operation. With that said, you could try compiling with `-DKJ_USE_PIPE_FOR_WAKEUP`, which causes it to avoid using signals for this. You need to compile both KJ itself and your own code that depends on it with this define; if they don't match you may get undefined behavior.

`kj::newTwoWayPipe()` (the global function) is a completely in-memory implementation of TwoWayPipe which is tied to a single thread. If you use `kj::AsyncIoProvider::newTwoWayPipe()` instead, it will create an implementation backed by a socketpair(). However, unfortunately, the KJ AsyncIoStream wrapper objects are still tied to the particular thread that created them. In order to communicate between threads, you will need to manually create a socketpair(), and then use kj::LowLevelAsyncIoProvider::wrapSocketFd() to create the AsyncIoStream wrappers. Each thread would need to use its own LowLevelAsyncIoProvider object to create the AsyncIoStream for its end of the pipe. Sorry, there isn't currently a more-convenient way to do this.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/EA5DCF15-8128-4F65-AE98-708AE1B75D1E%40mooseyard.com.

[Jens Alfke]

> On Aug 30, 2022, at 7:37 PM, 'Kenton Varda' via Cap'n Proto <capn...@googlegroups.com> wrote:
>
> Hi Jens,

>
> It sounds very obnoxious that your debugger insists on breaking on this signal.

I’ve filed a bug report with Apple.

> With that said, you could try compiling with `-DKJ_USE_PIPE_FOR_WAKEUP`, which causes it to avoid using signals for this. You need to compile both KJ itself and your own code that depends on it with this define; if they don't match you may get undefined behavior.

For posterity: since I’m building with CMake, I accomplished this by adding `add_compile_definitions(KJ_USE_PIPE_FOR_WAKEUP)` to my top-level CMakeLists.txt, above the line `add_subdirectory(vendor/capnproto)`.

Now my code can talk inter-thread RPC over the fake socket! Yay!

But of course there’s always another roadblock. The next one I hit is a fatal exception "expected !loop.running; wait() is not allowed from within event callbacks.” Apparently code running in an Executor block on the target thread is not allowed to call Promise.wait()? This is a bummer, as so far I’ve been lazy and written my RPC client code in blocking style. Looks like it’s time to fully async-ify it.

—Jens

[Kenton Varda]

Hi Jens,

Two options for writing sync-like code that is actually async:

- KJ supports "Fibers", which allocates a separate stack that runs in the same thread. When running on that stack, you get a `WaitScope` that you can use to wait on promises. When you wait, the thread switches back to the main stack.

- We have some support for C++20 coroutines using co_await/co_return (aka async/await). However, it's currently based on `-fcoroutines-ts` in Clang; it may not work with any other compiler, and this flag is going to go away soon, though we'll likely upgrade to the final version of coroutines before that happens.

-Kenton