Ip rate limiting zone size question

[Mindaugas Bernatavičius]

Greetings openresty-en group, 

I wanted to ask a question related more to nginx, than openresty per se.

One of the modules that is often employed ngx_http_limit_req_module

has the following precaution in the documentation:

 

If the zone storage is exhausted, the server will return the 503 

(Service Temporarily Unavailable) error to all further requests.

It is interesting for me how is the zone defined? 

All the information needed for the rate limit?

If so, what does it contain - the data structure. 

I have multiple users on the website served by openresty.

Would the zone size be exhausted if the unique IP count

would reach 8K unique IP's in some time frame?

How do I determine the lower bound of the zone? 

For example do I need 5m or 20m? 

Trying to look at ngx_http_limit_req_module.c I saw only configure

time error being thrown when the zone size is specified incorrectly:

if (size < (ssize_t) (8 * ngx_pagesize)) { 

ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "zone \"%V\" is too small", &value[i]); 

return NGX_CONF_ERROR;

 }

(8 * ngx_pagesize), if I'm not mistaken is 8 * 4096 = 32768

I confirmed experimentally that the smallest size is indeed 32768 bytes = 32KB.

The function contains some interesting data:

static ngx_int_t ngx_http_limit_req_lookup(ngx_http_limit_req_limit_t *limit, 

ngx_uint_t hash, 

ngx_str_t *key, 

ngx_uint_t *ep, 

ngx_uint_t account)

         node = ngx_slab_alloc_locked(ctx->shpool, size);

         if (node == NULL) {

             ngx_log_error(NGX_LOG_ALERT, ngx_cycle->log, 0,

                           "could not allocate node%s", ctx->shpool->log_ctx);

             return NGX_ERROR;

         } 

I suppose this is the error thrown when zone size limit is reached?

Would really appreciate your help on this issue.

[rpaprocki]

Hi,

On Friday, October 7, 2016 at 4:15:33 AM UTC-7, Mindaugas Bernatavičius wrote:

Greetings openresty-en group, 

I wanted to ask a question related more to nginx, than openresty per se.

One of the modules that is often employed ngx_http_limit_req_module

has the following precaution in the documentation:

 

If the zone storage is exhausted, the server will return the 503 

(Service Temporarily Unavailable) error to all further requests.

It is interesting for me how is the zone defined? 

All the information needed for the rate limit?

If so, what does it contain - the data structure. 

The underlying data structure is a red-black tree. The limt_req_zone directive (http://nginx.org/en/docs/http/ngx_http_limit_req_module.html#limit_req_zone) allows the user to define the key for a given element inside the zone. The larger the key, the more memory this will take. If you're using the remote IP as the key, the variable $binary_remote_addr is the most efficient way to store this. The value stored is type ngx_uint_t, whichis typedef'd to uintptr_t.

 

I have multiple users on the website served by openresty.

Would the zone size be exhausted if the unique IP count

would reach 8K unique IP's in some time frame?

How do I determine the lower bound of the zone? 

For example do I need 5m or 20m? 

From the docs (assuming you're using $binary_remote_addr as the key):

> A client IP address serves as a key. Note that instead of $remote_addr, the $binary_remote_addr variable is used here. The $binary_remote_addr variable’s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses. The stored state always occupies 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms. One megabyte zone can keep about 16 thousand 64-byte states or about 8 thousand 128-byte states. If the zone storage is exhausted, the server will return the 503 (Service Temporarily Unavailable) error to all further requests.

[Mindaugas Bernatavičius]

Hi, thanks a lot for responding.

I would like to recompile nginx for testing purposes

with the limit decreased so that I would not have to

use hundreds of IPs to see what happens. 

The main question for me to answer would be: 

If my size of the zone 1M  how many IP addresses 

can I have at most and for how long w/o getting 

the 503 due to zone memory exhaustion?

2 related question groups:

1. How small do I have to make the allowed memory 

so I would exhaust the space with 2/3 ips (I could use 

larger data structures like ip + url + some headers of course).

2. To test the expiration behavior to answer the questions:

how does the state expire? Is it saved till a reload or restart?

Does a reload pass the state (copies it) to the new worker?

If my size of the zone 1M how many IP addresses can I have 

at most and for how long?

So the node is of type ngx_rbtree_node_t . 

The definition of this type is this:

 struct ngx_rbtree_node_s {
     ngx_rbtree_key_t       key;   ===> 4 bytes
     ngx_rbtree_node_t     *left;  ===> 8 bytes (pointer size on 64bit)
     ngx_rbtree_node_t     *right; ===> 8
     ngx_rbtree_node_t     *parent;===> 8
     u_char                 color; ===> 8
     u_char                 data;  ===> 8
 };

So the size of one node 44 bytes apparently (please correct me if I'm wrong).

But the state is said to occupy 128 bytes, so there is something 

more to the state than only the RBtree, is the state fully defined 

with the struct below or multiple scructs? Could you help me find it?

 typedef struct { 

   ngx_rbtree_t                  rbtree; 

   ngx_rbtree_node_t             sentinel; 

   ngx_queue_t                   queue; 

 } ngx_http_limit_req_shctx_t;

Thank you very much

[Thijs Terlouw]

On Saturday, October 8, 2016 at 7:17:32 AM UTC+2, Mindaugas Bernatavičius wrote:

The main question for me to answer would be: 

If my size of the zone 1M  how many IP addresses 

can I have at most and for how long w/o getting 

the 503 due to zone memory exhaustion?

I had the same question. By looking at the modules source code ( https://github.com/git-mirror/nginx/blob/master/src/http/modules/ngx_http_limit_req_module.c ) we can learn the following:

1. the module first attempts to use all zone memory
2. if the memory in a zone is full, it attempts to expire data ( ngx_http_limit_req_expire(ctx, 0) )
3. nginx will then check these things:
- check if the oldest node is unused
- check if the oldest node is older than 60 seconds
- check if the oldest node is not in burst/excess range
if all is ok, then the node is removed and a new node can be stored

So for your question about how long: you would be rejecting new requests for all new clients for 60 seconds.