Doing timers right

[Chris Tanner]

Couple of questions about timer best practices. 

I want to run a single timer in the background which dumps some stats from a shared dict into redis. Currently the code just calls ScheduleTimer() which contains the ngx.timer.at() call and error handling, the callback then dumps to redis, and reschedules itself by calling ScheduleTimer() again.

1. Whats the best way to run a single timer process in the background? If I use init_worker_by_lua then I end up with 1 per worker, which is more than I need.

2. The first time I run the code it works fine, with a timer repeating every 5 seconds, however when I do 'nginx -s reload' and it re runs my lua code, the timer fails with the error 'process exiting, context: ngx.timer'

Seems like the timer is being told that the process is exiting even though it should be just starting up? Or is it trying to access processes still active from before the reload?

Cheers,

Chris

[Yichun Zhang (agentzh)]

Hello!

On Wed, Jan 14, 2015 at 9:10 AM, Chris Tanner wrote:
> 1. Whats the best way to run a single timer process in the background? If I
> use init_worker_by_lua then I end up with 1 per worker, which is more than I
> need.
>

You need to use the shared dict (or just use lua-resty-lock which uses
the same trick) to ensure only one worker is actually doing the job in
its timer handler though other workers still run timers that do
nothing. See how the lua-resty-upstream-healthcheck library handles
this:

https://github.com/openresty/lua-resty-upstream-healthcheck

The reason that all the workers should still run the timers anyway is
for stability (in extreme conditions some workers may crash abnormally
though such cases should be really rare and deserve fixing). Running
your re-occurring timers in every worker should not really hurt
performance in a measurable way. You just need to ensure only one
worker's timer handler is actually doing any useful job at a time.

> 2. The first time I run the code it works fine, with a timer repeating every
> 5 seconds, however when I do 'nginx -s reload' and it re runs my lua code,
> the timer fails with the error 'process exiting, context: ngx.timer'

This "process exiting" error is officially documented in
ngx.timer.at's documentation:

https://github.com/openresty/lua-nginx-module#ngxtimerat

Basically it's not allowed to create new timers while the worker
processes already receive the signal (from the master process) to shut
down (which also happens during HUP reload), otherwise your old
workers will stuck there forever unless you kill them with brute force
(like "kill -9"). So you should quietly handle this case and error
string without logging to the error log file.

> Seems like the timer is being told that the process is exiting even though
> it should be just starting up?

Upon HUP reload, all the nginx workers are replaced by the new workers
though the master process remains the same. This is how nginx works.

BTW, on *NIX systems, it's highly recommended to send HUP signal to
the nginx master process directly (via the kill command, for example)
instead of using the "nginx -sreload" hack mainly for Windows (which
has bad side-effects).

> Or is it trying to access processes still
> active from before the reload?
>

It happens in the old nginx workers which are supposed to shutdown
during the HUP reload.

Regards,
-agentzh

[Hamish Forbes]

I asked (and agentzh answered) the same thing a while back, so i created this self-contained nginx.conf that demonstrates the correct way to get 1 timer with multiple workers:

https://gist.github.com/hamishforbes/1bdc679893eabbbab3bd

Slghtly easier to understand than the real code in lua-resty-upstream-healthcheck as this code doesn't *do* anything except run a worker.

[Chris Tanner]

Perfect, this is exactly what I needed, thanks guys!