finatra with request draining and graceful shutdown

[Dave Richardson]

Hi folks,

I've inherited an api server app based on on finatra. It has to deal with some long lived uploads (several minutes). It doesn't handle graceful shutdowns - on receiving SIGTERM it just stops, breaking the uploads in progress. I'd like to let any in progress uploads complete after shutdown is initiated.

I think the functionality I want is implemented in finagle's Request Draining module (https://twitter.github.io/finagle/guide/Clients.html#request-draining), but I haven't figured out how to add this to a finatra application. Can anyone give me some pointers?

Many thanks,

Dave  

[Dave Richardson]

I decided to work on something simpler than our real application, so I used this template app:

https://github.com/forthy/finatra.g8

I made a couple of minor changes: 

 - Make one endpoint return slowly by adding a call to Future.delayed(Duration.fromSeconds(30)) in the controller

 - Set a default close grace period of 60 seconds in the Server implementation

If I use the admin shutdown endpoint, the server shuts down gracefully after all request are complete.

If I send SIGTERM to the server process, the server shuts down even if requests are in progress.

Is this the expected behaviour?

Is there a way to persuade finatra to shutdown more gracefully on SIGTERM? 

Thanks,

Dave

[Dave Richardson]

Adding this to Server appears to solve my problem: 

sys.addShutdownHook({
  Await.ready(this.close(defaultCloseGracePeriod))
})

Sending SIGTERM now results in the server closing after the requests have completed.

 

[Carlos Saltos]

Maybe adding a timeout to that hook can be a good idea, something like:

sys.addShutdownHook({

Await.ready(this.close(defaultCloseGracePeriod), Duration.fromMinutes(5))

})

Thanks for the tip !!