why ngx.balancer.get_last_failure always return nil?

[xfwang...@gmail.com]

Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout  http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code.  please suggest.

Thanks

Xiaofeng

[Robert Paprocki]

Hi,

On Mon, Mar 6, 2017 at 7:01 AM, <xfwang...@gmail.com> wrote:

Hello

I have successfully use the balancer to dynamically change the upstream servers with a simple round-robin strategy. I also want to implement some failover policy like the nginx builtin upstream's max_fails and fail_timeout  http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server, and I find the ngx.balancer.get_last_failure function. But it always returns nil even the last attempt failed, both of state_name and state_code.  please suggest.

Can you please post a _full_, minimal, reproducible example of your config (emphasis on full ;) ) so someone can try to examine the behavior you're describing. Thanks :)

[xfwang...@gmail.com]

here are the codes and conf, you see, I specify the port of the second node as 18867 but it should be 8867, when the request hits the second node, the page reports 502 bad gateway but get_last_failure return nil

init_by_lua_block {

    local servers = { { "127.0.0.1", 8866 }, { "127.0.0.1", 18867 }, { "127.0.0.1", 8868 }}

    local rr = require("my.rr") -- the round-robin module

    local rrc = rr:new(servers)

    package.loaded.my_servers = servers

    package.loaded.my_rr = rrc

}

upstream dyn_ups {

    server 0.0.0.1:8866;   # just an invalid address as a place holder

    balancer_by_lua_block {

        local b = require("ngx.balancer")

        local rr = package.loaded.my_rr

        local srvId = rr:next()

        local srv = package.loaded.my_servers[srvId]

        ngx.log(ngx.WARN, "srv: " .. table.concat(srv, ":"))

        local state_name, state_code = b.get_last_failure()

        if state_name then 

            ngx.log(ngx.ERR, "last failure state " .. state_name .. "code " .. state_code)

        end       

        local ok, err = b.set_current_peer(srv[1], srv[2])

        if not ok then

            ngx.log(ngx.ERR, "failed to set the current peer: ", err)

            return ngx.exit(500)

        end

    }

    keepalive 10;

}

server {

    listen 8765;

    location /dyn_ups{

        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;

        proxy_pass http://dyn_ups;

    }

}

server {

    listen 8866;

    location /dyn_ups{

        content_by_lua 'ngx.say(ngx.var.server_port)';

    }

}

server {

    listen 8867;

    location /dyn_ups{

        content_by_lua 'ngx.say(ngx.var.server_port)';

    }

}

server {

    listen 8868;

    location /dyn_ups{

        content_by_lua 'ngx.say(ngx.var.server_port)';

    }

}

[Rob Kay]

From the documentation:

Retrieves the failure details about the previous failed attempt (if any) when the next_upstream retrying mechanism is in action.

I believe the "next_upstream" mechanism is only active when you have multiple server directives within an upstream block and since you're using ngx.balancer to override a single server directive there is never a previous attempt. 

[xfwang...@gmail.com]

I dont think so. The document says it should work. I guess I'm missing something like configuration or initialization.

[xfwang...@gmail.com]

OK. I figured out this. I need invoke b.set_more_tries(10) before b.set_current_peer to explicitly tell the balancer block to retry the fail upstream request. And I also need to set the nginx proxy_next_upstream_tries directive to some value greater than zero. 

For the step 2, I think it's a bug or at least it's incompatible with the nginx original upstream failover policy. Because the default 0 value indicates that nginx won't limit the retry times other than disable the retry. check more at the official document http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries

[Robert Paprocki]

 Hi,

On Tue, Mar 7, 2017 at 6:28 AM, <xfwang...@gmail.com> wrote:

For the step 2, I think it's a bug or at least it's incompatible with the nginx original upstream failover policy. Because the default 0 value indicates that nginx won't limit the retry times other than disable the retry. check more at the official document http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_tries

I don't think this assessment is quite accurate. The docs specifically say "The 0 value turns off this limitation.".This should mean that, by default (and depending on the configuration of proxy_next_upstream) ngx.balancer should continue to retry the request based on the value of set_more_tries (https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md#set_more_tries). In your case, examining the debug logs would be very useful (if you have them captured and can reproduce behavior you think is buggy/undocumented, please post the debug logs so we can have a look [and use a pastebin for debug logs, it'll be pretty verbose ;)]).

Additionally, have a look at https://github.com/openresty/lua-nginx-module/blob/master/src/ngx_http_lua_balancer.c#L686. The next_upstream mechanism is only shorted by balancer by lua when the value is not 0 (which is the default, see https://github.com/nginx/nginx/blob/master/src/http/modules/ngx_http_proxy_module.c#L2946).

[xfwang...@gmail.com]

Confirm with agentzh that it really is a known issue and fixed for a long while. It has been patched in the latest master branch. Unfortunately it's not fixed in the latest release v1.11.2.2. You need to build from the latest master branch source code.

see also :

enable retry in balancer_by_lua_block: need to set proxy_next_upstream_tries · Issue #90 · openresty/lua-resty-core

bypass next_upstream_tries check when its value is 0 by kingluo · Pull Request #853 · openresty/lua-nginx-module

Hope helpful.