Correct shutdown Yesod

[Dmitriy Nikitinskiy]

Hello All.

How to correct shutdown Yesod application?

Calling in route handler liftIO (exitSuccess) just raises Internal Server erorr and server continue working.

I found liftIO (raiseSignal sigTERM) that terminates process itself, but this method is not crossplatform and I not sure about correct releasing used resources by such method.

Is any correct method gracefully shutdown Yesod application?

Best regards,
Dmitriy.

[Michael Snoyman]

It's funny you should ask this, Greg and I were just discussing signal
handling ourselves. The answer is that shutdown will depend on which
WAI handler you're using. Let's assume you're using Warp (since I
think everyone is at this point). Warp runs in an infinite loop
accepting connections and handing them off to new threads. Warp also
ignores all exceptions thrown from the application, to make sure that
one bad request won't take down the whole server. That seems to be why
exitSuccess causes an internal server error.

So the question is how do we signal Warp to shutdown. One possibility
would be to kill its containing thread, though that will interrupt any
active requests. I think a better idea is to have an (optional) IORef
passed to Warp that will indicate whether it should accept new
connections. Once the IORef gets switched to False, Warp will exit its
loop, returning control to your program. You'll then be responsible
for any cleanup you want to do, and should probably also threadDelay
for about 5 seconds to give active connections a chance to finish.

We could then use this IORef to handle signals (as is necessary on Heroku).

Any thoughts on this?

Michael

2011/7/21 Dmitriy Nikitinskiy <nikit...@gmail.com>:

[Greg Weber]

If someone that knew more about signal handling in GHC could make suggestions, that would be helpful :).

I do see there is support for installing signal handlers in GHC: http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/Signals

Why do we need an IORef *passed* to Warp? It seems like Warp should be able to handle this on its own.

I am wondering what Dmitry's use case is, because it doesn't seem like a frequent case that one would want to shutdown the application from inside it! It seems more appropriate to send a signal from the outside, at which point Warp will stop accepting new connections and will return from its loop. Could it return thread ids to wait on or is there too much overhead associated with that?

One ideal usage case for Warp and Signal handlers would be to send a signal that would allow one to switch to a new version of the application with zero downtime. But there might be other approaches to achieving this that work better.

[Dmitriy Nikitinskiy]

Hello,

It's ok for me, using IORef running flag.
Maybe add such IORef into Wai.Handler.Warp.Settings ?

Or as variant maybe add ResponseControl to Response data to controlling Wai handler (not only running-flag)?
For example:
data Response = ... | ResponseControl WaiControlCommand (Maybe ResponseType {- for contnent with control -})
data WaiControlCommand = ...

Best regards,
Dmitriy

[Michael Snoyman]

On Thu, Jul 21, 2011 at 4:46 PM, Greg Weber <gr...@gregweber.info> wrote:
> If someone that knew more about signal handling in GHC could make
> suggestions, that would be helpful :).
> I do see there is support for installing signal handlers in
> GHC: http://hackage.haskell.org/trac/ghc/wiki/Commentary/Rts/Signals
> Why do we need an IORef *passed* to Warp? It seems like Warp should be able
> to handle this on its own.

Warp *could* handle this on its own, but:

1) Signals only exist on POSIX systems.
2) It wouldn't be appropriate for Warp to install a process-wide
signal handler on its own by default. I have no problem providing
helper functions to make this easier on people, but the default should
be to not affect the rest of the process.

> I am wondering what Dmitry's use case is, because it doesn't seem like a
> frequent case that one would want to shutdown the application from inside
> it! It seems more appropriate to send a signal from the outside, at which
> point Warp will stop accepting new connections and will return from its
> loop. Could it return thread ids to wait on or is there too much overhead
> associated with that?

I can imagine having a "shutdown" button on an admin panel. Especially
if you have your application process being monitored, it could be
convenient for some use cases.

What do you mean by return thread ids?

> One ideal usage case for Warp and Signal handlers would be to send a signal
> that would allow one to switch to a new version of the application with zero
> downtime. But there might be other approaches to achieving this that work
> better.

I think the approach just mentioned would achieve this: sending the
signal will shut down the main thread and release the main listening
thread, still letting active connections finish. So starting up a new
process right after sending the signal should work.

[Michael Snoyman]

On Thu, Jul 21, 2011 at 4:51 PM, Dmitriy Nikitinskiy
<nikit...@gmail.com> wrote:
> Hello,
>
> It's ok for me, using IORef running flag.
> Maybe add such IORef into Wai.Handler.Warp.Settings ?
>
> Or as variant maybe add ResponseControl to Response data to controlling Wai
> handler (not only running-flag)?
> For example:
>  data Response = ... | ResponseControl WaiControlCommand (Maybe ResponseType
> {- for contnent with control -})
>  data WaiControlCommand = ...
>

It's possible to do this, but I'd rather not. I don't want to start
confusing the request/response setup of WAI with control messages.

Michael

[Greg Weber]

By thread ids I mean the thread id for each request that is still on-going.

This shutdown scenario still wouldn't be true zero downtime- the new application need to be loaded into memory before the old begins to shutdown. There is a server for Ruby apps called Unicorn with true zero downtime- it accomplishes this by having several worker forks that get replaced by new worker forks of the new application. Compiling Warp directly into the application doesn't allow for this approach. It should be possible to use a proxy that would start sending requests to the new application while the old is shutting down. Of course there is an issue of memory consumption under such an approach.

I am still confused about the recommended approach- someone has to catch signals- is Yesod going to do that by default now?

Greg

[Michael Snoyman]

On Thu, Jul 21, 2011 at 5:46 PM, Greg Weber <gr...@gregweber.info> wrote:
> By thread ids I mean the thread id for each request that is still on-going.
> This shutdown scenario still wouldn't be true zero downtime- the new
> application need to be loaded into memory before the old begins to shutdown.
> There is a server for Ruby apps called Unicorn with true zero downtime- it
> accomplishes this by having several worker forks that get replaced by new
> worker forks of the new application. Compiling Warp directly into the
> application doesn't allow for this approach. It should be possible to use a
> proxy that would start sending requests to the new application while the old
> is shutting down. Of course there is an issue of memory consumption under
> such an approach.

I think we should defer true zero downtime to a load balancer outside
of our process. As you're pointing out, there's no way to have true
zero downtime with a single process like this.

> I am still confused about the recommended approach- someone has to catch
> signals- is Yesod going to do that by default now?

The idea is that we'd set up a signal handler before calling Warp. In
pseudo-code:

flag <- newIORef True
installHandler sigINT (writeIORef flag False)
runWarpWithFlag flag
cleanup

Warp will continue running until that flag is set to False, which will
happen when the signal gets sent.

Michael

[Greg Weber]

ok, that looks good. What I was getting at with thread ids is that if they are returned from Warp then you could actually wait for them rather than doing an opaque hack like sleeping. The other possibility would be for Warp to write back to the IORef when all of its threads are completed (this in itself would require creating a new thread for the current Warp call to return, or at least to return an action that waits on existing threads). But I know little about the Warp implementation to know if that is a minor or major change.

[Felipe Almeida Lessa]

On Thu, Jul 21, 2011 at 11:53 AM, Michael Snoyman <mic...@snoyman.com> wrote:
> The idea is that we'd set up a signal handler before calling Warp. In
> pseudo-code:
>
>    flag <- newIORef True
>    installHandler sigINT (writeIORef flag False)
>    runWarpWithFlag flag
>    cleanup
>
> Warp will continue running until that flag is set to False, which will
> happen when the signal gets sent.

There could also be an exception

data StopWarp = StopWarp deriving (Show, Typeable)
instance Exception StopWarp where

that could be thrown by the application. This would be only exception
to the exception handler (pun intended) and would be equivalent to
setting the flag to False, just more convenient.

However, I don't know if it would be possible to do away with the flag
and use throwTo. It may be simpler to keep it.

Cheers,

--
Felipe.

[Michael Snoyman]

Interesting idea, it could be a very good approach. The only possible
problem is that Yesod intercepts all exceptions and returns 500 pages,
so Yesod would need to have special handling[1] for this as well. In
any event, we'd need to have the flag at the Warp level, if only
internally, so that a child thread could notify the main thread, so I
don't see any performance reason for this change.

Michael

[1] Exceptional exceptions.

[Michael Snoyman]

OK, scratch everything I said before, it was a bad approach. An IORef
flag would indeed work, but only *after* one more connection was made,
since the main Warp loop is still stuck in an accept call. Instead, it
seems that we have a much simpler approach available that works with
Warp as it is right now. Here's an example of interrupt signal
handling[1], though it should work well for Dmitriy's case as well I
think.

Michael

[1] https://gist.github.com/1098954