nginx: worker process is shutting down ( version-1.9*)

[Abhishek Garg]

After updating openresty from version 1.7.4.1 to latest  (openresty/1.9.7.4) for http2 support my openresty workers thread are not closing properly when performing reload.

Let's say we have running openresty  with 4 thread worker process every time I'm reloading 2 workers in unclean state. I try to look around for issue with different versions and load. Issue is replicating above version 1.7.10.2.

Any idea or workaround for same ?

[Yichun Zhang (agentzh)]

Hello!

On Wed, May 25, 2016 at 6:21 AM, Abhishek Garg wrote:
> After updating openresty from version 1.7.4.1 to latest (openresty/1.9.7.4)
> for http2 support my openresty workers thread are not closing properly when
> performing reload.
>

OpenResty 1.7.4.1 uses NGINX 1.7.4, which does not have HTTP/2 support
yet (NGINX first added HTTP/2 support in its 1.9.5 version).

> Let's say we have running openresty with 4 thread worker process every time
> I'm reloading 2 workers in unclean state. I try to look around for issue
> with different versions and load. Issue is replicating above version
> 1.7.10.2.
>
> Any idea or workaround for same ?

There has quite some bugs fixed in the official NGINX core since 1.9.7
(used by OpenResty 1.9.7.4, obviously). Please try the latest
OpenResty 1.9.15.1 RC2 version instead.

Regards,
-agentzh

[Suchit Puri]

Hi AgentZH,

We really need some help here, we have been trying to upgrade openresty to version ~ 1.9 as it has http2 support and makes our CDN faster.

The issue is whenever we reload the openresty we are left with 2 dangling worker processes stuck in "worker process is shutting down" state, They never die down.

Currently we have written scripts which kill them after reload, but its a big problem for us. Can you guide us how to debug this ? 

[Lord Nynex]

Hello,

Please be aware that you have no provided enough data to make any sort of determination. Please provide 

* The output of nginx -V

* A *minimal* nginx.conf that demonstrates the issue and allows us to reproduce/test. 

Without this information, we are not able to speculate on why you're experiencing this behavior. 

-Brandon

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Suchit Puri]

Hi Brandon,

Please find the below details.

Here is the configuration file http block http://pastie.org/10860074

Here you can find the openresty build arguments we are using http://pastie.org/10860079

Here are the nginx version and other details.

nginx version: openresty/1.9.15.1rc2
built by gcc 4.7.2 (Debian 4.7.2-5)
built with OpenSSL 1.0.1s  1 Mar 2016
TLS SNI support enabled

The problem occurs when we upgrade the openresty from 1.7.4 to 1.9.X using

/etc/init.d/openresty quietupgrade

 we have tried both 1.9 RC as well as the stable 1.9 version.

Do let me know if you need any more info.

Thanks

Suchit

[Yichun Zhang (agentzh)]

Hello!

On Tue, May 31, 2016 at 11:28 AM, Suchit Puri wrote:
> Hi AgentZH,
>

Please do not use upper case letters in my nick. Thank you.

> We really need some help here, we have been trying to upgrade openresty to
> version ~ 1.9 as it has http2 support and makes our CDN faster.
>

Can you reproduce it with HTTP2 disabled? Wondering if it's related to
the HTTP2 implementation in the nginx core.

> The issue is whenever we reload the openresty we are left with 2 dangling
> worker processes stuck in "worker process is shutting down" state, They
> never die down.
>

One way to directly inspect the shutting-down worker processes is to
use GDB to check what timers are pending. You can check the timer
handler pointer to find out what parts of the nginx/openresty are
responsible for such dangling timers.

FYI, the nginx core uses the following code for a worker process to
test whether it can shutdown safely:

if (ngx_exiting) {
ngx_event_cancel_timers();

if (ngx_event_timer_rbtree.root == ngx_event_timer_rbtree.sentinel)
{
ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");

ngx_worker_process_exit(cycle);
}
}

It's in the ngx_worker_process_cycle C function defined in the file
src/os/unix/ngx_process_cycle.c.

You can also check out the ngx_event_cancel_timers C function for an
example of iterating through the redblack tree for pending timers.

Best regards,
-agentzh