Could not connect to Redis - Cannot assign requested address

[Apoorv Verma]

Hello Experts, 

We were running certain performance load tests on Redis 7.0.6 version using redis-benchmark utility. As a part of this test, 

• We first populate Redis memory with certain expiry and non-expiry data. 
• Then we run the following redis-benchmark command 

redis-benchmark -h <hostname> -p 6379 -c 1000 -n 1700000 -t get -d 18000 -P 300 -l

• We keep repeating the above command with different pipeline numbers.
• And this test keeps running for 30 mins for each pipeline.

 However, we are noticing that after few minutes of the test the redis-benchmak command stops giving the following error

99.047% <= 66.111 milliseconds (cumulativeCould not connect to Redis at <hostname>:6379: Cannot assign requested address

This keeps happening in every test.

Can you please let us know -

• The reason behind this error? Is this an error behavior? 
• What can be done to eradicate this error.

Best Regards,

Apoorv

Best Regards,

Apoorv

[Ankit Gupta]

Observed similar behavior with additional observation that test stops
after getting this error.
This seems harmful !

Experts please share your opinion on this.


Thanks & Regards,
Ankit Gupta

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/abcdbcbd-69d3-4ab7-8896-43cc0e89869en%40googlegroups.com.

[David Maier (Redis)]

Hi Apoorv,

This could be because Redis rejects a new connection that is attempted by redis-benchmark. Please take a look at https://redis.io/docs/reference/clients/ for further details.

Here are a few questions to analyze your issue a bit further:

• Is Redis still up and running after the error message occurs?
• Is there any error message in the Redis log file? (or standard/error output)
• What is the maximum number of open file descriptors that you configured on the Redis host and the benchmark host?
• Are you using any kind of network abstraction (e.g., a Docker network)?
• What is the maximum number of connections that you configured for your Redis instance?
• To which value is the configuration actually set? The command `CONFIG GET` might be helpful for you.
• How many connections are open on the side of Redis when the error occurs? The command `INFO CLIENTS` will provide details about the number of connected clients.

BTW: The solution for issue https://github.com/redis/redis/issues/340 might also be useful.

Regards,

David

 

> To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/abcdbcbd-69d3-4ab7...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CAJ_z65o5ACkKMjciu=RZOnjJyuHy391QeG...@mail.gmail.com.

Disclaimer

The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

[Greg Andrews]

Have you done a search of the error message "Cannot assign requested address" and found potential causes such as the one described in the answers to this StackOverflow question?

How many file descriptors is the redis-benchmark command being allowed by your distro's configuration, and how many simultaneous connections is your redis-benchmark trying to open?

[Apoorv Verma]

Hi David 

Please find below the answers - 

Is Redis still up and running after the error message occurs?

  - Yes the Redis remains up and running , though the redis-benchmark command stops after this error.

Is there any error message in the Redis log file? (or standard/error output)

  - There is no unusual error in Redis Logs.

What is the maximum number of open file descriptors that you configured on the Redis host and the benchmark host??
   - Requested maxclients of 10000 requiring at least 10032 max file descriptors

Are you using any kind of network abstraction (e.g., a Docker network)?

  - Yes we are using Docker container for both Redis and Sentinel containers.

What is the maximum number of connections that you configured for your Redis instance?

   [root@b4ff69352e57 /]# redis-cli 2d1e1630f50e11eb9a030242ac130003 GET maxclients
    1) "maxclients"
    2) "10000"

To which value is the configuration actually set? The command `CONFIG GET` might be helpful for you.

   [root@b4ff69352e57 /]# redis-cli 2d1e1630f50e11eb9a030242ac130003 GET maxclients
    1) "maxclients"
    2) "10000"

How many connections are open on the side of Redis when the error occurs? The command `INFO CLIENTS` will provide details about the number of connected clients.

This is the current value .. I will update the value when I will repeat the test again 

    # Clients
    connected_clients:7
    cluster_connections:0
    maxclients:10000
    client_recent_max_input_buffer:20480
    client_recent_max_output_buffer:0
    blocked_clients:0
    tracking_clients:0
    clients_in_timeout_table:0

Also I tried the solution mentioned in   https://github.com/redis/redis/issues/340 but it did not work for me.

Ran the following command from outside the docker container 

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

But the problem still persists . Awaiting for your further suggestions ,

Best Regards,

Apoorv

[Ankit Gupta]

We increased the maxclients to 20k , and error stopped coming. any thoughts ?

[David Maier (Redis)]

Hi Ankit,

it seems that you opened more than 'maxclients' connections to Redis. By configuring your Redis instance accordingly you were able to prevent the error.

Here an excerpt of the documentation article that I referenced earlier:

"After the client is initialized, Redis checks if it is already at the limit configured for the number of simultaneous clients ... . When Redis can't accept a new client connection because the maximum number of clients has been reached, it tries to send an error to the client in order to make it aware of this condition, closing the connection immediately ..."

Hope this helps.

Regards.

David

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/a6193680-34a0-4b18-aadc-22b2aee5801en%40googlegroups.com.

[Ankit Gupta]

Thanks for a prompt reply David, but during the test we are using constant #clients (1000) . Will this still be valid ?

[Xamidullo Xudoyberdiyev]

My name is xamidullo.

You wrote the wrong name. Got it?

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/CAMzF2K39NXPrMJ1XRfeatK-XR05k4BRHEdRVGMnH7-XVbqL-eA%40mail.gmail.com.