understanding balancer_by_lua

[RJoshi]

In my current openresty deployment,  I have an upstream_server.conf which contains 20+ upstream blocks.  As we add a new endpoint(ELB), I have to manually update this configuration file and restart nginx.  Upstream backend information is stored in the MySQL database along with service endpoint and other routing information.

1. Can I achieve above without restarting nginx using balancer_by_lua? I can read the DB and get all domain/port/conn-pool size etc.

2.   Do I need multiple upstream backend { } to match my current config? If yes, h How can I register multiple upstream backend { } sections dynamically? 

3. Does balancer.set_current_peer(host, port) add the host/port to current upstream connection pool or just used for routing current request?

Thanks,

[RJoshi]

NOTE: In my current configuration,  I get the upstream backend section name dynamically.

proxy_pass https://$upstream_domain$upstream_uri;

[Guanglin Lv]

在 2016年3月2日星期三 UTC+8上午8:51:09，RJoshi写道：

In my current openresty deployment,  I have an upstream_server.conf which contains 20+ upstream blocks.  As we add a new endpoint(ELB), I have to manually update this configuration file and restart nginx.  Upstream backend information is stored in the MySQL database along with service endpoint and other routing information.

1. Can I achieve above without restarting nginx using balancer_by_lua? I can read the DB and get all domain/port/conn-pool size etc.

 of course, you can  achieve that without reload nginx.

 

2.   Do I need multiple upstream backend { } to match my current config? If yes, h How can I register multiple upstream backend { } sections dynamically? 

 i'm afraid that you have to need multiple upstream backend with a server defined  to placeholder.

  

3. Does balancer.set_current_peer(host, port) add the host/port to current upstream connection pool or just used for routing current request?

  upstream foo.com {  

   server 0.0.0.1; # just an invalid address as a place holder 

   balancer_by_lua_file "/path/to/your/lua/file"; 

} 

  

 you use lua-resty-mysql library to get your upstream server list and take your load balance algorithm.  set your target server use set_current_peer.

 notice: 

  1. you need to resolve the hostname of server firstly. lua-resty-dns library is there.

  

 more details and examples see https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md#synopsis

[RJoshi]

Thanks.
Our upstream services are categories and are always behind the load balancer(ELB/F5) So I really do not need to add/remove servers at runtime.

When new services are registers, I am storing the service Uris and ELB address in the database. Each ELB is configured using it's own upstream backend_elb_xxx {} block which is dynamically used in the proxy_pass section.
E.g. proxy_pass https://$upstream_backend_elb$upstream_uri
Both of these variables are populated based on the uri matching to upstream section from the database.

I want to add these upstream backend_elb_xxx {} sections without restart.

[Yichun Zhang (agentzh)]

Hello!

On Tue, Mar 1, 2016 at 4:51 PM, RJoshi <rohit....@gmail.com> wrote:
> In my current openresty deployment, I have an upstream_server.conf which
> contains 20+ upstream blocks. As we add a new endpoint(ELB), I have to
> manually update this configuration file and restart nginx. Upstream backend
> information is stored in the MySQL database along with service endpoint and
> other routing information.
>
>
> 1. Can I achieve above without restarting nginx using balancer_by_lua? I can
> read the DB and get all domain/port/conn-pool size etc.
>

This looks like a perfect use case of balancer_by_lua*. But you'll need to

1. load the peers metadata from MySQL nonblockingly via
lua-resty-mysql in the context of earlier phase handlers like
access_by_lua*, and
2. pass the metadata obtained in access_by_lua* or alike to
balancer_by_lua* via the ngx.ctx magic table.

It's better to

1. cache the metadata by means of lua_shared_dict and/or
lua-resty-lrucache to reduce expensive mysql queries to a neglectable
level, and
2. enable backend connection pool via the standard keepalive directive
in your upstream {} block:

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

> 2. Do I need multiple upstream backend { } to match my current config? If
> yes, h How can I register multiple upstream backend { } sections
> dynamically?

Nope. You only need a single upstream block. You do the host dispatch
in balancer_by_lua* itself. So that you can run thousands or even
millions of sites with a single upstream {} block and a single
connection pool.

> 3. Does balancer.set_current_peer(host, port) add the host/port to current
> upstream connection pool or just used for routing current request?
>

The sock address you set in Lua is used for both the current request
and any subsequent requests that can reuse this connection (if a later
request sets a different peer address, then it cannot reuse this very
connection in the pool).

Regards,
-agentzh

[RJoshi]

Thanks @agentzh.

1. When I use http port, the example given in https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md is works  fine but if I use https, nginx process exits with following information.

2016/03/02 21:37:26 [notice] 85327#0: signal 20 (SIGCHLD) received

2016/03/02 21:37:26 [info] 85334#0: *5 client closed connection while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:8080

2016/03/02 21:37:26 [alert] 85327#0: worker process 85329 exited on signal 11

I am using openresty version 1.9.7.3 with below modules.

built by clang 7.0.2 (clang-700.1.81)

built with OpenSSL 1.0.1g 7 Apr 2014 (running with OpenSSL 1.0.2d 9 Jul 2015)

TLS SNI support enabled

configure arguments: --prefix=/opt/openresty/nginx --with-cc-opt='-O2 -I/usr/local/include -I/usr/local/Cellar/openssl/1.0.1g/include/' --add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.58 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.29 --add-module=../form-input-nginx-module-0.11 --add-module=../encrypted-session-nginx-module-0.04 --add-module=../srcache-nginx-module-0.30 --add-module=../ngx_lua-0.10.0 --add-module=../ngx_lua_upstream-0.04 --add-module=../headers-more-nginx-module-0.29 --add-module=../array-var-nginx-module-0.04 --add-module=../memc-nginx-module-0.16 --add-module=../redis2-nginx-module-0.12 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.14 --add-module=../rds-csv-nginx-module-0.07 --with-ld-opt='-Wl,-rpath,/opt/openresty/luajit/lib -L/usr/local/lib -L/usr/local/Cellar/openssl/1.0.1g/lib' --with-pcre-jit --with-http_ssl_module

2. How do I enable connection pool for different upstream hosts?  E.g I reuse the upstream section "backend123 " as below and set the different host for routing different requests, how does connection pool work? Let's say I have 20 different host/port for upstream servers and requested routed though balancer.set_current_peer(), would it create a new connection everytime?

upstream backend123 {

      server 0.0.0.1;   # just an invalid address as a place holder

      balancer_by_lua_block {

            local balancer = require "ngx.balancer"

            local ok, err = balancer.set_current_peer(ngx.var.host, tonumber(ngx.var.port))

            if not ok then

                ngx.log(ngx.ERR, "failed to set the current peer: ", err)

                return ngx.exit(500)

            end

        }

        keepalive 10;  # connection pool

}

[Guanglin Lv]

在 2016年3月2日星期三 UTC+8下午7:46:46，RJoshi写道：

  if you want add/update some upstream without restart, this module can help you.

   https://github.com/yzprofile/ngx_http_dyups_module

  anyway, balancer_by_lua is still the best choice

  thanks.

.

[RJoshi]

Thanks Guanglin.  When i tried using balancer_by_lua for SSL connection, it kills the nginx process.

Copying the details from earlier post

: When I use http port, the example given in https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md is works  fine but if I use https, nginx process exits with following information.

2016/03/02 21:37:26 [notice] 85327#0: signal 20 (SIGCHLD) received

2016/03/02 21:37:26 [info] 85334#0: *5 client closed connection while SSL handshaking, client: 127.0.0.1, server: 0.0.0.0:8080

2016/03/02 21:37:26 [alert] 85327#0: worker process 85329 exited on signal 11

I am using openresty version 1.9.7.3 with below modules.

built by clang 7.0.2 (clang-700.1.81)

built with OpenSSL 1.0.1g 7 Apr 2014 (running with OpenSSL 1.0.2d 9 Jul 2015)

TLS SNI support enabled

configure arguments: --prefix=/opt/openresty/nginx --with-cc-opt='-O2 -I/usr/local/include -I/usr/local/Cellar/openssl/1.0.1g/include/' --add-module=../ngx_devel_kit-0.2.19 --add-module=../echo-nginx-module-0.58 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.29 --add-module=../form-input-nginx-module-0.11 --add-module=../encrypted-session-nginx-module-0.04 --add-module=../srcache-nginx-module-0.30 --add-module=../ngx_lua-0.10.0 --add-module=../ngx_lua_upstream-0.04 --add-module=../headers-more-nginx-module-0.29 --add-module=../array-var-nginx-module-0.04 --add-module=../memc-nginx-module-0.16 --add-module=../redis2-nginx-module-0.12 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.14 --add-module=../rds-csv-nginx-module-0.07 --with-ld-opt='-Wl,-rpath,/opt/openresty/luajit/lib -L/usr/local/lib -L/usr/local/Cellar/openssl/1.0.1g/lib' --with-pcre-jit --with-http_ssl_module

[Yichun Zhang (agentzh)]

Hello!

On Thu, Mar 3, 2016 at 6:54 AM, RJoshi wrote:
> Thanks Guanglin. When i tried using balancer_by_lua for SSL connection, it
> kills the nginx process.
>

This issue was reported in the following ticket:

https://github.com/openresty/lua-nginx-module/issues/696

And it has just been fixed in the git master branch of
lua-nginx-module as the following commit:

https://github.com/openresty/lua-nginx-module/commit/c226a00

Please try it out on your side.

Thanks!

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Wed, Mar 2, 2016 at 6:49 PM, RJoshi wrote:
> 1. When I use http port, the example given in
> https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/balancer.md
> is works fine but if I use https, nginx process exits with following
> information.
>

It's already fixed. Please see
https://github.com/openresty/lua-nginx-module/issues/696 for more
details.

BTW, please do not post duplicate mails. Thank you.

Regards,
-agentzh

[RJoshi]

Thanks.  I validated this patch and seem to be working.

If requests are routed to 10 different servers (ip) using balacer_by_lua upstream section, how can I set connection pool for each IP?

[Yichun Zhang (agentzh)]

Hello!

On Sun, Mar 6, 2016 at 3:10 PM, RJoshi wrote:
> Thanks. I validated this patch and seem to be working.
>

Cool.

> If requests are routed to 10 different servers (ip) using balacer_by_lua
> upstream section, how can I set connection pool for each IP?
>

They share the same pool. Only the in-pool connection matching the
peer address requested will be reused for a particular session. So
don't worry about conflicts.

Regards,
-agentzh

[RJoshi]

Thanks. What is the significance of 

keepalive directive when balancer_by_lua is used.  If I set it to keepalive 10;, would it manage 10 connections per upstream ip?

[Yichun Zhang (agentzh)]

Hello!

On Mon, Mar 7, 2016 at 9:02 AM, RJoshi wrote:
> Thanks. What is the significance of
>
> keepalive directive when balancer_by_lua is used. If I set it to keepalive
> 10;, would it manage 10 connections per upstream ip?
>

One keepalive directive, one connection pool (per worker). Different
peers share the same pool without conflicts (since only matching
connections in the pool can be reused, others are skipped).

Regards,
-agentzh