connection abort: socket write error after a while

[Joe Fawzy]

Hi
i am using Jedis 1.3 with Redis 2.0.2
i got the exception java.net.SocketException: Software caused
connection abort: socket write error
on all jedis commands if the connection was idle for sometime (about 5
min)
Note: it is not a timeout error, and i also got this for any timeout
value , even 0(never timeout)

if i put continuous load on the connection it never get this error

i tried to call jedis.connect() before every call but this does not
change anything, by looking at the source, i discovered that jedis do
just that anyway

i first noticed it on my dev env on windows and i thought it is the
server's fault as the cygwin impl is not that stable
BUT when i tested the server on ubuntu linux with just the same
results

thanks in advance



Exception
java.net.SocketException: Software caused connection abort: socket
write error

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
92)

at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

at
redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:
31)

at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:
225)

at redis.clients.jedis.Protocol.sendCommand(Protocol.java:40)

at redis.clients.jedis.Connection.sendCommand(Connection.java:60)

at redis.clients.jedis.Client.smembers(Client.java:257)

at redis.clients.jedis.Jedis.smembers(Jedis.java:1181)

[Jonathan Leibiusky]

Hi!
Some questions to have a better insight on you bug.
Are you using the same Jedis instance in different threads/requests?
Are you connecting Jedis and leaving it open for a while before sending another command?

Thanks,

Jonathan

[Joe Fawzy]

Hi dear
thanks for your prompt reply

> Are you using the same Jedis instance in different threads/requests?

actually NO,but is that allowed,i think there is no documentation on
this
how this is best done

> Are you connecting Jedis and leaving it open for a while before sending
> another command?

what do you mean by "leave it open"?
am i supposed to call disconnect after every call?is this how it
should be used? is this perform well?
this is not documented also

please , tell me the steps for making the best of the library

thanks

Joe

On Oct 13, 12:11 am, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> Hi!
> Some questions to have a better insight on you bug.
> Are you using the same Jedis instance in different threads/requests?
> Are you connecting Jedis and leaving it open for a while before sending
> another command?
>
> Thanks,
>
> Jonathan
>

[Jonathan Leibiusky]

Sorry about the lack of documentation, I'm actually working on this, should have something out there this week.

On Tue, Oct 12, 2010 at 8:13 PM, Joe Fawzy <joew...@gmail.com> wrote:

Hi dear
thanks for your prompt reply

> Are you using the same Jedis instance in different threads/requests?

actually NO,but is that allowed,i think there is no documentation on
this
how this is best done

Jedis is not thread safe and you shouldn't use the same instance in different threads.
In this scenarios you should use JedisPool, which is a thredsafe pool of Jedis, that will reuse Jedis instances which is a performance boost.
 

> Are you connecting Jedis and leaving it open for a while before sending
> another command?

what do you mean by "leave it open"?
am i supposed to call disconnect after every call?is this how it
should be used? is this perform well?
this is not documented also

I think I didn't explain myself clear enough here. The normal usage of Jedis, when you are not using JedisPool, is creating a Jedis instance, connecting it, performing bunch of commands and disconnecting Jedis when done.
Leaving a Jedis instance open for a while could lead to timeout or read/write exceptions as you experience, since Redis server can disconnect the sockets on his end, leaving the socket open in the application end and when you want to execute a command you will have an error.
Some libraries try to reconnect when this happens. Probably Jedis should support this, but I would like to know if this is your case. By that I mean:
1. You create a Jedis and connect it
2. Execute some commands
3. Leave it open and unused for a while
4. Try to execute another command
 

[Joe Fawzy]

Hi dear

> Jedis is not thread safe and you shouldn't use the same instance in
> different threads.

i am using grails right now, and jedis instance is in a service, which
provide methods using this instance and these methods accessed by
Controller, that means for each client there is a thread accessing the
instance
i think making these methods synchronized will work BUT with horrible
performance degradation as syncing is not working good for large no.
of threads

> 1. You create a Jedis and connect it
> 2. Execute some commands
> 3. Leave it open and unused for a while
> 4. Try to execute another command

yes this is how i did things

i think that creating a jedis instance every time or calling
disconnect and re- connect() for the next command is expensive as
connect:
if (!isConnected()) {
socket = new Socket(host, port);
socket.setSoTimeout(timeout);
outputStream = new RedisOutputStream(socket.getOutputStream());
inputStream = new RedisInputStream(socket.getInputStream());
}

it create a new socket, and inputStream and outputStream which is
expensive in terms of JVM resource utilization

i am using redis to develop an web analytics solution which will have
may be hundreds or thaosands of hits per second (may be more,and this
is why i choose redis)
so creating a socket each time is not a wise solution

as a temporary solution , i made a schedualer that create a new Jedis
instance every minute , that means a new Socket every minute
But if you think that there is a better solution , please let me know

i do not used JedisPool b4 as i thought it is used for sharding only
and i have a one Redis server for now
if this is the way to go, please write a sample code, that show best
practice for using the library and does not exhaust the JVM resources

On Oct 13, 1:20 am, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> Sorry about the lack of documentation, I'm actually working on this, should
> have something out there this week.
>

[Jonathan Leibiusky]

You should definitely use JedisPool, as it was made for this kind of scenarios.
Just create JedisPool and leave it available statically. Then from every thread do a:

Jedis jedis = jedisPool.getResource();

use it and after that do a:

jedisPool.returnResource(jedis);

This way you won't be creating new sockets for every connection. You'll be reusing Jedis instances and by that achieve better performance and no timeout problems.

You can see more examples here: http://github.com/xetorthio/jedis/blob/master/src/test/java/redis/clients/jedis/tests/JedisPoolTest.java

Let me know if you need more help on this.

Jonathan

On Tue, Oct 12, 2010 at 8:51 PM, Joe Fawzy <joew...@gmail.com> wrote:

Hi dear

> Jedis is not thread safe and you shouldn't use the same instance in
> different threads.

i am using grails right now, and jedis instance is in a service, which
provide methods using this instance and these methods accessed by
Controller, that means for each client there is a thread accessing the
instance
i think making these methods synchronized will work BUT with horrible
performance degradation as syncing is not working good for large no.
of threads

> 1. You create a Jedis and connect it
> 2. Execute some commands
> 3. Leave it open and unused for a while
> 4. Try to execute another command

yes this is how i did things

i think that creating a jedis instance every time or calling
disconnect and re- connect() for the next command is expensive as
connect:
 if (!isConnected()) {
socket = new Socket(host, port);
socket.setSoTimeout(timeout);
outputStream = new RedisOutputStream(socket.getOutputStream());
inputStream = new RedisInputStream(socket.getInputStream());
}

it create a new socket, and inputStream and outputStream which is
expensive in terms of JVM resource utilization

i am using redis to develop an web analytics solution which will have
may be hundreds or thaosands of hits per second (may be more,and this
is why i choose redis)
so creating a socket each time is not a wise solution

as a temporary solution , i made a schedualer that create a new Jedis
instance every minute , that means a new Socket every minute
But if you think that there is a better solution , please let me know

i do not used JedisPool b4 as i thought it is used for sharding only
and i have a one Redis server for now
if this is the way to go, please write a sample code, that show best
practice for using the library and does not exhaust the JVM resources

On Oct 13, 1:20 am, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> Sorry about the lack of documentation, I'm actually working on this, should
> have something out there this week.
>

[Joe Fawzy]

Hi dear

this means
//this is one time init (may be in service or a singleton bean)
JedisPool p=new JedisPool()
p.init()

//for every command
Jedis j=p.getResource()
j.ping() // or any other command
p.returnResource (j)

am i get it right

thanks
Joe

On Oct 13, 1:58 am, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> You should definitely use JedisPool, as it was made for this kind of
> scenarios.
> Just create JedisPool and leave it available statically. Then from every
> thread do a:
>
> Jedis jedis = jedisPool.getResource();
>
> use it and after that do a:
>
> jedisPool.returnResource(jedis);
>
> This way you won't be creating new sockets for every connection. You'll be
> reusing Jedis instances and by that achieve better performance and no
> timeout problems.
>

> You can see more examples here:http://github.com/xetorthio/jedis/blob/master/src/test/java/redis/cli...

>
> Let me know if you need more help on this.
>
> Jonathan
>

[Joe Fawzy]

Hi
should i call init() or it is impplicitly called by the framework, i
can see its not called by the JedisPool constructor?
thanks
Joe

[Jonathan Leibiusky]

you should call init() explicitly.
and you don't need to get a resource for every command, it is better if you use one jedis per request.

//at the beginning of the app
JedisPool jedisPool = new JedisPool()
jedisPool.init()

//at the beginning of the request
Jedis jedis = jedisPool.getResource()

...
do lots of stuff with jedis instance
...

//at the end of the request
jedisPool.returnResource(jedis)

//at the end of the app
jedisPool.destroy()


Since you are using this as an analytics solution, probably you would have something like this:

class AnalyticsService {
  //singleton instance injected for you
  def jedisPool;

  def countSomeInfo() {
    def jedis = jedisPool.getResource()
    jedis.incr("foo")
    ... do more increments, etc ...
    jedisPool.returnResource(jedis)

[Joe Fawzy]

Thanks a lot
Joe