Transient write failures with JedisPool and default JedisPoolConfig.

[Vinay Ganesh]

I am using Jedis-2.1.0 with JedisPool and the default JedisPoolConfig to configure the JedisPool. Every now and then a Write to Redis will fail with this error:

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
        at redis.clients.jedis.Protocol.process(Protocol.java:79) ~[stormjar.jar:na]
        at redis.clients.jedis.Protocol.read(Protocol.java:131) ~[stormjar.jar:na]
        at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:162) ~[stormjar.jar:na]
        at redis.clients.jedis.BinaryJedis.setex(BinaryJedis.java:447) ~[stormjar.jar:na]
        at com.yahoo.boson.core.MemoryStoreWriter.persist(MemoryStoreWriter.java:114) [stormjar.jar:na]
        at com.yahoo.boson.core.MemoryStoreWriter.process(MemoryStoreWriter.java:70) [stormjar.jar:na]
        at com.yahoo.boson.base.Processor.execute(Processor.java:182) [stormjar.jar:na]
        at backtype.storm.daemon.executor$eval4022$fn__4023$tuple_action_fn__4025.invoke(executor.clj:610) [na:na]
        at backtype.storm.daemon.executor$mk_task_receiver$fn__3943.invoke(executor.clj:381) [na:na]
        at backtype.storm.disruptor$clojure_handler$reify__1642.onEvent(disruptor.clj:43) [na:na]
        at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84) [storm-0.9.0-wip16.jar:na]
        at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58) [storm-0.9.0-wip16.jar:na]
        at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62) [na:na]
        at backtype.storm.daemon.executor$eval4022$fn__4023$fn__4035$fn__4082.invoke(executor.clj:715) [na:na]
        at backtype.storm.util$async_loop$fn__363.invoke(util.clj:385) [na:na]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_13]
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.7.0_13]
        at java.net.SocketInputStream.read(SocketInputStream.java:150) ~[na:1.7.0_13]
        at java.net.SocketInputStream.read(SocketInputStream.java:121) ~[na:1.7.0_13]
        at java.net.SocketInputStream.read(SocketInputStream.java:107) ~[na:1.7.0_13]
        at redis.clients.util.RedisInputStream.fill(RedisInputStream.java:109) ~[stormjar.jar:na]
        at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:45) ~[stormjar.jar:na]
        at redis.clients.jedis.Protocol.process(Protocol.java:64) ~[stormjar.jar:na]
        ... 16 common frames omitted

Just so that I am defensive I am catching all RuntimeExceptions and do a pool.returnBrokenResource(jedis). Do about 1000 writes per second with writes happening to localhost redis instance. Size of the object that I am writing is less than 1kb. Is this kind of timeouts expected?

How and when does the socket timeout? I let this application run for a few hours and I see 1 such failure every 1 hour or so.

Thanks.

[har...@snapchat.com]

On a related note, do we need to return a broken resource whenever we see timeouts? 

It is possible that some requests might have taken long. So for SocketTimeoutException, we can catch it and reuse the resource? Is this a terrible idea?

Thanks,
Harish

[Jonathan Leibiusky]

Hi. It is hard to say what could be generating this timeout exception.

Is your redis persisting the information on disk using AOF or BGSAVE? How often?

--
You received this message because you are subscribed to the Google Groups "Jedis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Vinay Ganesh]

Hi Jonathan,

Yes, I am using redis with aof enabled. and its doing an fsync every second. Also have BGREWRITEAOF once the file grows to 1gb.
Still having this issue and have not figured out the reason for it.

Thanks.
-vinay

[Vinay Ganesh]

are you hinting that the aof fsync or the re-write of the aof when it kicks in, the writes may be timing out? I ll try to corelate the timeouts with redis logs to see if its the problem. Otherwise, could there be other reasons for this to happen too?

Also, would be interested to know if its ok to re-use the same jedis instance that had a socket timeout ex for future writes/reads or am I better of returning the instance back to the pool as broken jedis instance?

Thanks Jonathan.

-vinay