Running a single recurring background thread

[Hamish Forbes]

Hi All,

I've got a situation where I need a single background thread to be running via ngx.timer.at, initialised at startup.

With multiple nginx workers init_worker_by_lua, obviously, runs in each worker and you end up with multiple recurring background threads.

At the moment the best solution I've come up with is something like this: https://gist.github.com/hamishforbes/34a4b85a4a29ad46b9fc

So on startup every worker will write it's PID into a shared dictionary key, overwriting the previous worker.

Then the background function checks the PID for the worker it's running in matches the entry in the shared dict.

This works but the initial timer delay (the one in init_worker_by_lua) needs to be long enough for all the workers to have fully started up.

If you've got a lot of workers, a lot of config or a slow machine that time can vary dramatically so its not a great solution from that point of view.

Its also a little messy with the extra dict get and check every time the background function runs.

I was just wondering if anyone else had come up with an alternate / better solution for doing stuff like this?

Thanks!

Hamish

[Yichun Zhang (agentzh)]

Hello!

On Thu, May 1, 2014 at 11:28 AM, Hamish Forbes wrote:
> I've got a situation where I need a single background thread to be running
> via ngx.timer.at, initialised at startup.
> With multiple nginx workers init_worker_by_lua, obviously, runs in each
> worker and you end up with multiple recurring background threads.
>

Well, this looks very much like the way how
lua-resty-upstream-healthcheck library works:

https://github.com/openresty/lua-resty-upstream-healthcheck

Basically I use shdict:add() to acquire and test a lock there. Quite simple.

Do not keep worker pids in shdict, because workers get quit upon HUP
reload (or abnormal crashes). They are not reliable.

Regards,
-agentzh

[Hamish Forbes]

Ahh, OK. So the main difference is you let the function continue to be called in each worker but only do some work if you grab the lock.

I think at some point I may have tried something like this but overlooked it because I was running into a race condition when I didn't actually have my background thread doing anything.

The function would get and release the lock so fast that the next worker could also grab the lock and run in the same cycle.

Heres a gist with a minimal config, have I missed anything? 

https://gist.github.com/hamishforbes/1bdc679893eabbbab3bd

I also noticed that if you call ngx.timer.at at the *end* of your background function your workers can get out of sync and you end up not having regular intervals.

With the above config but the timer call at the end of the function I was rapidly ending up with something like this:

2014/05/01 22:29:54 [debug] 25979#0: [lua] init_by_lua:9: 25979 locked
2014/05/01 22:29:56 [debug] 25981#0: [lua] init_by_lua:9: 25981 locked
2014/05/01 22:30:04 [debug] 25980#0: [lua] init_by_lua:9: 25980 locked
2014/05/01 22:30:06 [debug] 25979#0: [lua] init_by_lua:9: 25979 locked
2014/05/01 22:30:08 [debug] 25981#0: [lua] init_by_lua:9: 25981 locked

I guess it depends exactly what you're trying to do with your background thread but for me the fixed regular interval is what I wanted.

Thanks Yichun!

Hamish