Number of active connections

[Simon Eskildsen]

Hello!

We're starting to make more and more use of ngx-lua, but one of my worries is exceeding `worker_connections` under load. I've been trying to monitor the number of active connections (client, upstream, ngx lua tcp connections) that, as far as I understand, all count towards this limit. I've been trying to grab `cycle->free_connection_n` with stap, but it just gives me a static number even under load. (See https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp for the script). The `ngx-active-req` from the resty stap tools only seems to count client connections, and not all connections—I haven't been able to successfully tweak it to count all connections.

Would love some help!

[Brian Akins]

Are you using http://nginx.org/en/docs/http/ngx_http_stub_status_module.html ?  You can look through it's code to see how it counts connections.

FWIW, I've ran instances that had 100k+ connections per nginx worker with no issues in production doing "real work" in Lua.  Depends on your workload, etc.

--
You received this message because you are subscribed to the Google Groups "openresty-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Hørup Eskildsen]

I did a test with the status module where I opened 100 connections from Lua and slept for 10s. I saw the connections with `lsof`, but no number on the status page climbed above 100. It seems to not count the Lua connections?

[Simon Hørup Eskildsen]

Hm, with the test described in the previous email, monitoring `free_connection_n` (which I was monitoring in the first place) is actually what I want; it's climbing above 100. It looks like the test I used the first time wasn't right. So I've got the numbers I needed now!

The `stub_status_module` page only shows 1 additional connections for such a request. I checked the source, and it indeed only shows a count of HTTP connections. I'm assuming other people are monitoring `free_connection_n` to make sure `ngx.timer` and TCP connections from Nginx lua are counted as well?

Cheers

[Yichun Zhang (agentzh)]

Hello!

On Thu, Jan 22, 2015 at 7:57 AM, Simon Eskildsen wrote:
> I've been trying
> to grab `cycle->free_connection_n` with stap, but it just gives me a static
> number even under load. (See
> https://raw.githubusercontent.com/Sirupsen/system-tools/nginx-cycle/nginx/cycle.stp
> for the script).

The direction is correct but your systemtap script has bugs itself.
For example, ngx_cycle itself is a pointer, addressing it with "&"
again turn it into something else. Also, ngx_cycle->files is an
instance of ngx_list_t, whose ->size field is not what you want (it's
just the size of each element in the queue, not the size of the
queue).

I've just tweaked your script and come up with this:

https://github.com/openresty/stapxx/blob/master/samples/ngx-count-conns.sxx

I've added it as a sample tool for my stap++ wrapper for systemtap.
Hopefully you don't mind :)

Also added some docs for it:

https://github.com/openresty/stapxx#ngx-count-conns

One sample output in a production box:

$ ngx-count-conns.sxx -x 32743
Start tracing 32743 (/opt/nginx/sbin/nginx)...

====== CONNECTIONS ======
Max connections: 32768
Free connections: 27674
Used connections: 5094

====== FILES ======
Max files: 131072
Open files: 2
Free files: 131070

Regarding your questions, yes, ngx_lua's cosocket connections and
timers are also counted in the "Used connections" numbers shown above.

Regards,
-agentzh

[Simon Hørup Eskildsen]

You rock, thanks Yichun!

I've written another script which may be useful to other people in the same situation as me (want to monitor connection/timer ceilings, critical to not loose data) to measure # of timers in ngx-lua: https://github.com/openresty/stapxx/pull/10

[Simon Hørup Eskildsen]

Another question for this thread. I just ran ngx-count-conns (https://github.com/openresty/stapxx/blob/master/samples/ngx-count-conns.sxx) in production and get:

====== CONNECTIONS ======

Max connections: 20480

Free connections: 20419

Used connections: 61

====== FILES ======

Max files: 200000

Open normal files: 30

Then I `curl` to the `ngx_http_stub_status_module` (http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) and get something like:

Active connections: 8036 

server accepts handled requests

 351374622 351374622 553813639 

Reading: 1 Writing: 89 Waiting: 7930

Why are they so different? I confirmed that Lua TCP sockets indeed make it to "Used connections" for the stap script, but I have a hard time figuring out what each number here means now.

[Yichun Zhang (agentzh)]

Hello!

On Thu, Jan 22, 2015 at 3:16 PM, Simon Hørup Eskildsen wrote:
> Then I `curl` to the `ngx_http_stub_status_module`
> (http://nginx.org/en/docs/http/ngx_http_stub_status_module.html) and get
> something like:
>
> Active connections: 8036
> server accepts handled requests
> 351374622 351374622 553813639
> Reading: 1 Writing: 89 Waiting: 7930
>
> Why are they so different?

Maybe some of your worker processes crashed abnormally since the
startup? Better enable core dump for your nginx worker processes.

Also please keep in mind that the stub_status metrics are per-server
(including all the workers) while the stap script is per-worker.

Regards,
-agentzh

[Abdul Hakeem]

Hello,
Can anyone share the connections per second and the maximum concurrent connections
ever tested on Openresty, with details of the reference hardware ?
Cheers,
Abdul Hakeem