Could not get resource exception while inserting large data to redis

[Sonali Palkar]

Hi All:

I am using redis 2.2.12 server and using jedis 2.0 as a redis client. I have a single redis server instance on which I perform insert/update and read operations. I have 3 - 4 java applications which are inserting and updating around 23M keys in redis in a day. The data is not coming at a fixed interval of time with fixed size. Some times there could be a large push. My apps use jedis pool to get redis connection and insert data into redis. I have configured jedis pool to use around 1000 threads.

My application throws this exception at times when there is a large push. I do not see this exception the entire day. It happens only when there is large data getting inserted.

"jedis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool".

Any idea why this exception comes at times and not the entire day even though I am using fixed number of threads via a thread pool to control the number of threads and hence the number of connections to redis?

Best regards.
Sonali

[Josiah Carlson]

I can see two potential causes:
1. If you have a 1k threadpool on each of 3-4 machines, you are
potentially creating and using 3-4k connections to Redis. If Redis is
configured with a maximum connection limit under 3-4k, you may get
exceptions like this.
2. If Jedis isn't re-using connections, but is instead creating and
destroying connections as they are needed, you can run out of outgoing
ports for connections, as most OSes recycle ports in a fairly lazy
manner (1 minute or more to reuse a port isn't uncommon).

Check your configuration. If your configuration is fine, check to make
sure Jedis is re-using connections. If that is fine, then it's
possible that you are moving enough data into Redis that you have run
out of threads. You could then catch the exception, wait, and retry.

Regards,
- Josiah

[Sonali Palkar]

Thanks for the reply.
1. Regarding redis server max client setting, I have set max client to 0 meaning no limit in redis.conf.
2. Regarding jedis pool, I have configured jedis pool to block when exhausted so jedis pool should just wait if all connections are blocked right? And My app has a java thread pool with fixed no. of threads e.g 1500 threads, each thread basically gets the resource from jedis pool, performs lpush operations for around 1M data in a for loop and closes the connection and exits. So at a time there could be 1500 threads each getting resource from jedis pool and putting data to redis queue and returning resource to jedis pool.

Please find below the exact exception. Here from exception it does not look like it is a jedispool configuration issue. Some socket level issue. Any idea?

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:22)
at helper.TestTask$1.run(TestTask.java:67)
at java.lang.Thread.run(Thread.java:662)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused
at redis.clients.jedis.Connection.connect(Connection.java:120)
at redis.clients.jedis.Jedis.connect(Jedis.java:1638)
at redis.clients.jedis.JedisPool$JedisFactory.makeObject(JedisPool.java:63)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
at redis.clients.util.Pool.getResource(Pool.java:20)
... 2 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at redis.clients.jedis.Connection.connect(Connection.java:115)
... 6 more

Best regards.
Sonali

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Josiah Carlson]

Check both your client and server ulimit settings for open file handles.

As per a recently mentioned bug:
https://github.com/antirez/redis/issues/162 , older versions of Redis
on certain platforms were limited by default system ulimit (1024
handles limit). You can force it to be larger, which is discussed in
that bug.

Regards,
- Josiah

[Sonali Palkar]

Thanks for the links.

ulimit -n on client and server has value 1024

I have a test program to figure out if it fails at fixed no. of connections. This program consistently throws similar exception (instead of connection refuse it says 'Cannot assign requested address') exactly at 28232 count.

redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
        at redis.clients.util.Pool.getResource(Pool.java:22)

        at redislatencychecker.Main.main(Main.java:91)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Cannot assign requested address

        at redis.clients.jedis.Connection.connect(Connection.java:120)
        at redis.clients.jedis.Jedis.connect(Jedis.java:1638)
        at redis.clients.jedis.JedisPool$JedisFactory.makeObject(JedisPool.java:63)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
        at redis.clients.util.Pool.getResource(Pool.java:20)

        ... 1 more
Caused by: java.net.ConnectException: Cannot assign requested address
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
        at java.net.Socket.connect(Socket.java:546)
        at redis.clients.jedis.Connection.connect(Connection.java:115)
        ... 5 more

Here is the snippet of test program.

  JedisPoolConfig jConfig = new JedisPoolConfig();
  jConfig = new JedisPoolConfig();
  jConfig.setMaxActive(30000);
  jConfig.testOnBorrow = true;
  jConfig.setWhenExhaustedAction(GenericObjectPool.WHEN_EXHAUSTED_GROW);
  jPool = new JedisPool(jConfig, host, port, Integer.MAX_VALUE, "foobared");


   for (int i = 0; i < 50000; i++) {
   try {
             Jedis j = jPool.getResource();
              if (j != null) {
                   String info = j.info();
                    log.info("got jedis conn in = " + i);
                    Long clientCount = parseConnectedClients(info);
                    log.info("Client count = " + clientCount);
                    //    jPool.returnResource(j);
              }
   } catch (JedisConnectionException jex) {
          jex.printStackTrace();
          log.info("Failed at count = " + i);
          break;
   } catch (Exception ex) {
          ex.printStackTrace();
          break;
   }

If I get the connections from jedis pool without returning them it fails at 28232 always. If I return the connection every time then it works fine.
I checked the open file descriptors on server machine which were around 4558 or more. And total open FDs for redis server were around 30-40. Not much.

Any clue?

Best regards.
Sonali

[Jonathan Leibiusky]

This is because of pool exhaustion. Are you returning jedis instances
to the pool after using them?

On 11/11/11, Sonali Palkar <sona...@gmail.com> wrote:
> Hi All:
>
> I am using redis 2.2.12 server and using jedis 2.0 as a redis client. I
> have a single redis server instance on which I perform insert/update and
> read operations. I have 3 - 4 java applications which are inserting and
> updating around 23M keys in redis in a day. The data is not coming at a
> fixed interval of time with fixed size. Some times there could be a large
> push. My apps use jedis pool to get redis connection and insert data into
> redis. I have configured jedis pool to use around 1000 threads.
>
> My application throws this exception at times when there is a large push. I
> do not see this exception the entire day. It happens only when there is
> large data getting inserted.
>

> *"jedis.clients.jedis.exceptions.JedisConnectionException: Could not get a
> resource from the pool"*.

>
> Any idea why this exception comes at times and not the entire day even
> though I am using fixed number of threads via a thread pool to control the
> number of threads and hence the number of connections to redis?
>
> Best regards.
> Sonali
>

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to
> redis-db+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/redis-db?hl=en.
>
>

--
Sent from my mobile device

[Sonali Palkar]

In the test program I am purposefully not returning the connection to see what is the max connection limit it supports. There I have setWhenExhaustedAction to WHEN_EXHAUSTED_GROW so it should not exhaust.

In real program I do return the connection after using them but still I get connection refuse exception. So to trouble shoot when exactly it fails I have written a test program and set WHEN_EXHAUSTED_GROW to jedis pool config to ensure that jedis pool should behave properly and still I get the exception.

Best regards.
Sonali

[Jonathan Leibiusky]

Would you mind sharing the code, or a version of it, to see what you
are doing and how?

Thanks!

Jonathan

[Josiah Carlson]

Redis is likely disconnecting the clients due to inactivity, but the
client isn't recognizing that (they are not being used, so Java
doesn't know they are closed), so only some are recycled by the
server... until you run into the actual open files limit. You can use
"sudo lsof -i -p <pid>" to get the actual open sockets used by your
client, and a "sudo lsof -i -p <pid> | wc -l" will give you a total
open file handle count.

Regards,
- Josiah

[Sonali Palkar]

Okay. Thanks for the inputs.

So basically I need to retry in case of such exceptions to see if the server can again accept the connection. I will handle it in code.

Best regards.
Sonali

[Josiah Carlson]

Alternatively, you can reduce your total number of threads to 1000 per
client process and increase your total number of allowed connections
on the Redis side of things to something like 10k (in configuration,
and by "ulimit -n" as mentioned here and elsewhere). If you upgraded
to a more recent Redis, you would get the Redis-side configuration for
free.

- Josiah

[Sonali Palkar]

Okay. Yes I am planning to get redis 2.4.2 version. As you said in that version I can specify max clients to something like 10k and it would internally set rlimit right?
This should hopefully solve the problem.

I will upgrade and then get back here again in case the exception still comes.

Best regards.
Sonali

[Josiah Carlson]

It should try to automatically set the limit. You may have to manually
set the ulimit in your init.d script.

Regards,
- Josiah