Exceptions on shutdown

[Joe Ludwig]

Hi,

I'm hitting a variety of exceptions on shutdown. Exactly what I hit seems to shift as I move things around to try to solve the problem, but here's the current one. This doesn't happen 100% of the time, so there must be a race involved:

> avrenderer.exe!aardvark::CServerThread::Run::__l4::<lambda>(kj::Exception && exception) Line 683 C++
  [External Code]
  avrenderer.exe!aardvark::CRecoverableExceptionHandler::onRecoverableException(kj::Exception && exception) Line 656 C++
  avrenderer.exe!kj::throwRecoverableException(kj::Exception && exception, unsigned int ignoreCount) Line 878 C++
  avrenderer.exe!kj::_::Debug::Fault::~Fault() Line 332 C++
  avrenderer.exe!kj::`anonymous-namespace'::AsyncStreamFd::read::__l2::<lambda>(unsigned __int64 result) Line 242 C++
  avrenderer.exe!kj::_::MaybeVoidCaller<unsigned __int64,unsigned __int64>::apply<unsigned __int64 <lambda>(unsigned __int64) >(kj::`anonymous-namespace'::AsyncStreamFd::read::__l2::unsigned __int64 <lambda>(unsigned __int64) & func, unsigned __int64 && in) Line 135 C++
  avrenderer.exe!kj::_::TransformPromiseNode<unsigned __int64,unsigned __int64,unsigned __int64 <lambda>(unsigned __int64),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 401 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue & output) Line 721 C++
  avrenderer.exe!kj::_::TransformPromiseNode<kj::_::Void,unsigned __int64,void <lambda>(unsigned __int64),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 396 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue & output) Line 721 C++
  avrenderer.exe!kj::_::TransformPromiseNode<bool,kj::_::Void,bool <lambda>(void),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 396 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue & output) Line 721 C++
  avrenderer.exe!kj::_::TransformPromiseNode<kj::Maybe<kj::Own<capnp::MessageReader> >,bool,kj::CaptureByMove<kj::Maybe<kj::Own<capnp::MessageReader> > <lambda>(kj::Own<capnp::MessageReader> &&, bool),kj::Own<capnp::`anonymous namespace'::AsyncMessageReader> >,kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 396 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue & output) Line 721 C++
  avrenderer.exe!kj::_::TransformPromiseNode<kj::Maybe<kj::Own<capnp::IncomingRpcMessage> >,kj::Maybe<kj::Own<capnp::MessageReader> >,kj::Maybe<kj::Own<capnp::IncomingRpcMessage> > <lambda>(kj::Maybe<kj::Own<capnp::MessageReader> > &&),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 396 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue & output) Line 721 C++
  avrenderer.exe!kj::_::TransformPromiseNode<bool,kj::Maybe<kj::Own<capnp::IncomingRpcMessage> >,bool <lambda>(kj::Maybe<kj::Own<capnp::IncomingRpcMessage> > &&),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 396 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::getDepResult(kj::_::ExceptionOrValue & output) Line 721 C++
  avrenderer.exe!kj::_::TransformPromiseNode<kj::_::Void,bool,void <lambda>(bool),kj::_::PropagateException>::getImpl(kj::_::ExceptionOrValue & output) Line 396 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get::__l3::<lambda>() Line 703 C++
  avrenderer.exe!kj::_::RunnableImpl<void <lambda>(void) >::run() Line 303 C++
  avrenderer.exe!kj::_::runCatchingExceptions(kj::_::Runnable & runnable) Line 1023 C++
  avrenderer.exe!kj::runCatchingExceptions<void <lambda>(void) >(kj::_::TransformPromiseNodeBase::get::__l3::void <lambda>(void) && func) Line 315 C++
  avrenderer.exe!kj::_::TransformPromiseNodeBase::get(kj::_::ExceptionOrValue & output) Line 703 C++
  avrenderer.exe!kj::TaskSet::Task::fire() Line 180 C++
  avrenderer.exe!kj::EventLoop::turn() Line 373 C++
  avrenderer.exe!kj::WaitScope::poll() Line 413 C++
  avrenderer.exe!aardvark::CServerThread::Run() Line 689 C++

The specific exception in this case is that AsyncStreamFd wants to read some bytes but none are available, presumably because the client has disconnected.

The overall flow of the system is something like this:

1. server starts up
2. several clients start up. Some are in-process. Some are other processes on the same system.
3. Run for a while
4. Clean up all the clients
5. sleep(1000) <-- this is where the exceptions happen
6. clean up the server 

The sleep is only there to help me sort out this problem. Once I make shutdown deterministic it'll go away. But it appears that clients disconnecting and being freed is actually causing an exception on the server.

Is there some way for a server to handle client disconnections at arbitrary times without throwing? I could just swallow all exceptions as "must be a disconnect", but that doesn't seem quite right either since there could also be bugs in my own server code causing exceptions. I could make changes client-side too, but since clients can always just crash and not tell me about it that doesn't fully solve things.

I *am* able to disconnect and reconnect clients while running in general. This seems to be unique to shutdown in some way I don't understand. (The server doesn't know it's shutting down yet when the exception occurs.)

Any ideas?

Joe

[Kenton Varda]

Hi Joe,

That stack trace isn't very helpful because:

1. The exception description is missing.

2. The source file names are missing (despite line numbers being present).

3. It's a synchronous trace from the point where the exception was thrown, so it mostly just shows the event loop machinery rather than tracing through application code.

If you log the exception at the point where it is caught, you should get a more useful stack trace that traces through the asynchronous call stack, provided by KJ. What I'm trying to figure out is what high-level API you called that threw the exception.

FWIW you can distinguish whether an exception is caused by a network disconnect by examining its type, i.e.:

    if (exception.getType() == kj::Exception::DISCONNECTED)

That said, normally in a shutdown scenario I would delete all the promises in order to proactively cancel the tasks they represent, rather than wait for them to fail with DISCONNECTED. The KJ promise framework is designed to allow for cancellation in this way.

(Also if the process is exiting, then I'd just call exit() and be done with it. "Clean teardown" before process exit is usually a waste of time.)

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/b816998f-a402-4f7f-a7ff-c2ace394f3b7%40googlegroups.com.