Passenger processes getting stuck in shutting down state.

[Himanshu Sachdeva]

Hello,

We've been facing this issue where for one of our applications all of the passenger processes eventually get stuck in the shutting down state. Passenger processes are assigned max memory allocation limit of 330M. As processes exceed this limit they are shut down by passenger but they are not getting shut down, instead they get stuck in that state. As a result, they don't free up the memory they are holding. But with the load on the application passenger keeps spawning new processes and eventually the web box runs out of memory. This whole process takes around 3 days and we have to restart the application to free up memory. 

Upon seeing the requests these processes are serving or were serving before getting stuck, we get the following : 

 Client 46:

    host                        = www.limeroad.com

    uri                         = <request_uri>

    connected at                = 2017-07-07 13:20:45 (158252 sec ago)

    state                       = FORWARDING_BODY_TO_APP

    session pid                 = 29009 (<repo_home>#default)

    session gupid               = 17d535f-v013pYnEShB

    session initiated           = true

    requestBodyIsBuffered       = false

    requestIsChunked            = false

    requestBodyLength           = 431

    requestBodyAlreadyRead      = 431

    responseContentLength       = -1

    responseBodyAlreadyRead     = 0

    clientInput                 = 0x7f05e402bc00 fd=46, state=LIVE, buffer(0)="", paused=1, socketPaused=1, nextTickInstalled=0, generation=3, error=0

    clientInput started         = false

    clientBodyBuffer started    = false

    clientBodyBuffer reachedEnd = false

    clientOutputPipe started    = true

    clientOutputPipe reachedEnd = false

    clientOutputWatcher active  = false

    appInput                    = 0x7f05e400f770 fd=58, state=LIVE, buffer(0)="", paused=0, socketPaused=0, nextTickInstalled=0, generation=2, error=0

    appInput started            = true

    appInput reachedEnd         = false

    responseHeaderSeen          = false

    useUnionStation             = false

I also tried connecting to one of these hung processes using gdb/crash-watch to see the backtrace but couldn't figure out much from it. If it can be useful to debug the issue, please let me know so I'll paste that here as well. 

Please help us solving this problem. We can't have our server become unresponsive every other day. Any pointers will be much appreciated. Thanks.

[Daniel Knoppel]

The memory limiter attempts graceful shutdown, which waits for the process to finish handling its requests. It doesn't kill the process to avoid confronting users with an error, so that explains what you're seeing.

The question is: does the shutting down process get stuck while handling a request (use `passenger-status` to see if a stuck process still has Sessions > 0)?

* If it still has sessions, then you can use the max request time option: https://www.phusionpassenger.com/library/config/reference/?a=passenger_max_request_time. This will kill processes that get stuck on handling requests.

* If it doesn't have any sessions, then it's getting stuck on shutdown and you should figure out why using the backtraces (send SIGQUIT to the PID of the stuck process). You can try posting the trace if you need help with interpretation.

- Daniel