Timers "leaking"

[Bartosz Bubak]

Hi,
On several of our servers we use the "healthcheck" module, unfortunately something is wrong because after a while, the messages start to appear:

"failed to create timer: too many pending timers".

I know it's a standard error message, but it seemed strange to me so I decided to check what the situation was like. 

It looks like a few timers appear very quickly (although I have an error here that the number is negative), but the number of pending timers is constantly increasing up to the configured maximum. Increasing the maximum amount dosen't  helps, just saturation takes a little more time.

After ca. hour in logs got values like this:

Current runing timers -2129 (??!!!)
Current penging timers 4096
failed to spawn health checker: failed to create timer: too many pending timers,

I do not think it possible for healthcheck to take so long time to block timers, unfortunately my skills in the lua are a bit too low to debug it properly. 

Is this a known error? Can it be avoided somehow?

P.S. "Our" healthcheck - https://github.com/openresty/lua-resty-upstream-healthcheck
P.S.S. Our config looks like this:

       local hc = require "resty.upstream.healthcheck"
 
        local ok, err = hc.spawn_checker{
            shm = "healthcheck1",
            upstream = "application_karaf",
            type = "http",
            http_req = "GET /tenant/health HTTP/1.0\r\nHost: localhost\r\n\r\n",
            interval = 3000, timeout = 1500, fall = 3, rise = 2,
            valid_statuses = {200, 302},
            concurrency = 10,
        }
        if not ok then
            ngx.log(ngx.ERR, "failed to spawn health checker: ", err)
            ngx.log(ngx.ERR, "Current penging timers ", ngx.timer.pending_count())
            ngx.log(ngx.ERR, "Current runing timers ", ngx.timer.running_count())
            return
        end

BR

Bart