Jedis in a multithreaded environment

[Bernard D]

Hi,

Here is my java code that returns a sorted list of records from the Redis DB. It uses the sort method of the Jedis client.

        Jedis jedis = jedisPool.getResource();
        SortingParams sortingParameters = new SortingParams();
        String sortBy = key+"*->status";//assume key is 1:2:
        String filterSetName = key+"jobIds";
        sortingParameters.get(col1, col2, col3);//assume that col1, col2, col3 are defined
        sortingParameters.by(sortBy);
        List<String> filteredAndsortedList = null;
        try {
            filteredAndsortedList = jedis.sort(filterSetName, sortingParameters);
            System.out.println("Sorted List size "+ filteredAndsortedList.size());
            for(String str : filteredAndsortedList){
                //System.out.println(str);
            }
        }
        catch (Exception e) {
            System.out.println("-----Exception thrown-----");
            System.out.println(e);
            System.out.println(" returned value is "+ filteredAndsortedList );
            e.printStackTrace();
        } finally {
            jedisPool.returnResource(jedis);
        }

Works like a charm when called by one thread, no matter how many times.
Now, I test this via JMeter. Number of threads(users) = 10, with a ramp up time of 5 seconds.

StackTrace1:
redis.clients.jedis.exceptions.JedisConnectionException: Unknown reply: 1
redis.clients.jedis.exceptions.JedisConnectionException: Unknown reply: 1
        at redis.clients.jedis.Protocol.process(Protocol.java:76)
        at redis.clients.jedis.Protocol.read(Protocol.java:131)
        at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:199)
        at redis.clients.jedis.Connection.getMultiBulkReply(Connection.java:192)
        at redis.clients.jedis.Jedis.sort(Jedis.java:1725)
        at com.redis.client.RedisClientShard.sortRedis(RedisClientShard.java:380)
        at com.redis.client.JavaQueryRequest.runTest(JavaQueryRequest.java:43)
        at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:191)
        at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:429)
        at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)

StackTrace2:
redis.clients.jedis.exceptions.JedisConnectionException: It seems like server has closed the connection.
redis.clients.jedis.exceptions.JedisConnectionException: It seems like server has closed the connection.
        at redis.clients.util.RedisInputStream.readLine(RedisInputStream.java:90)
        at redis.clients.jedis.Protocol.processMultiBulkReply(Protocol.java:115)
        at redis.clients.jedis.Protocol.process(Protocol.java:68)
        at redis.clients.jedis.Protocol.read(Protocol.java:131)

StackTrace3:     
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: socket write error
redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Software caused connection abort: socket write error
        at redis.clients.jedis.Connection.flush(Connection.java:66)
        at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:197)
       
.

Seems like Jedis doesnt scale well in a mutithreaded environment. What am I doing wrong? What can I check/tweak to fix this?

Thanks!
Ranjit D'Souza

[Colin Vipurs]

I believe what's happening here is the Redis server is closing a client connection due to timeout and you're trying to reuse that connection (StackTrace2).  When this happens you have to let the JedisPool know by calling returnBrokenResource (or something like that), otherwise any other thread that reuses that connection will fail (StackTrace3).

What is your idle connection timeout and JedisPool connection limit set to?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

--
Maybe she awoke to see the roommate's boyfriend swinging from the chandelier wearing a boar's head.

Something which you, I, and everyone else would call "Tuesday", of course.

[Ranjit D]

"What is your idle connection timeout and JedisPool connection limit set to?"

 I haven't set any timeouts yet.  So its just the default values
For example, in redis.conf:
# Close the connection after a client is idle for N seconds (0 to disable)
timeout 0


Regarding JedisPool connection limit, I have not passed a new config object while creating a pool . So it is still using the defaults. Do you think its a good idea to do so?
If I do pass one, which do I need to set:

setMaxActive, setMaxIdle etc

[Jonathan Leibiusky]

Hi! Just to make all this a bit less confusing and ordered, I've extracted your test to a totally independent class with a main method. You can find it in this gist: https://gist.github.com/xetorthio/8807079

I think I have reproduced what you are doing with JMeter, but without it (just to not add noise).

As you can see there is a JedisPool, with default configuration.

I have created a ThreadPool of size 10, and submitted to it 10 tasks (each will run in a different thread as you can see in the output). Each task is the sorting you are running in a while(ended) {} clause, so basically during the entire test, each task will run as many times as it can.

After submitting all the tasks, the test wait 10 seconds, and then sets ended to true, which makes every thread to finish (exiting the while loop).

So what this basically is doing is to run for 10 seconds, in 10 threads your sorting command.

It doesn't complain about anything and ending nice and happy.

Please lets iterate over this gist, so we all use the same test case. 

Hopefully you can either find an error in your test or find an error in Jedis and we can fix it ASAP :)

Thanks a lot for your time!

Jonathan

--

[Ranjit D]

Jonathan, Colin: Thanks for the quick turnaround, you guys rock!

I did try out the gist. Worked without any issue when I pointed it to a redisDB that was empty. Now I added about 100,000 records and ran the test. Now, each task takes approximately 1.2 seconds to complete, and i saw these exceptions appear in the console:

redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
    at redis.clients.jedis.Protocol.process(Protocol.java:79)

    at redis.clients.jedis.Protocol.read(Protocol.java:131)
    at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:199)
    at redis.clients.jedis.Connection.getMultiBulkReply(Connection.java:192)
    at redis.clients.jedis.Jedis.sort(Jedis.java:1725)

    at lalala.RedisTestJonathan$1.run(RedisTestJonathan.java:41)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketTimeoutException: Read timed out


Which brings me to the question, is this an issue with Jedis or Redis? Is there anyway to increase the read/write timeout?
I am going to try passing a config object while creating a JedisPool and revert back with my findings

[Colin Vipurs]

It looks like Jedis sets a default timeout of 2 seconds on the socket level, so when your operations are taking 1.2 seconds, you should only get a response from the first operation before the rest timeout on the socket.

Looking at the Jedis source, it looks like you can set this when creating the pool:

https://github.com/xetorthio/jedis/blob/master/src/main/java/redis/clients/jedis/JedisFactory.java

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Ranjit D]

Thanks Colin, here are the results

With counter n set to 100, and number of threads in the pool also set to 100:

1. On an empty RedisDB, almost 9,500 tasks got completed.

2. With the sort command taking roughly 1.2 secs (while sorting through 100,000 records), 104 tasks got completed

Who was that guy who cast doubts on Jedis being multithreaded? i wanna have a word with you!

[Neelesh Korade]

Hi All

I am in the middle of debugging similar SocketTimeoutExceptions (Read timed out) and bumped into this thread. Note that our setup uses a JedisPool based client and we have TwemProxy servers in front of Redis instances. 

One thing I didn't understand in the case being discussed is that if the sort command takes roughly 1.2s and if the default Jedis timeout is 2 seconds, why would any of the sort command timeout? As I understand it, every time a jedis.sort() call is made, it should return within 2 seconds, no matter how many times it is being called. Or am I amiss somewhere? Will appreciate your help in understanding this.

Thanks

Neelesh

--

[Josiah Carlson]

Each command takes 1.2 seconds to run. But in Redis, they are running serially. So when two commands execute, it takes 2.4 seconds, causing some clients to time out on the Jedis side. Others squeeze in after commands complete, etc.

So really if all 10 clients were connected initially, and all clients executed the same sort command at more or less the same time (within a few tens of milliseconds), only the first to be read and executed by Redis would be executed. All others would time out, as all would have to wait a minimum of 2.4 seconds (which is greater than the 2 second default timeout).

- Josiah

(This was written on my phone. Please forgive my errors and/or brevity)

[Neelesh Korade]

Makes perfect sense. Thanks a ton! Will work on this to see if this is what's happening in our case too, as soon as I reach home. Incidentally, Salvatore responded to a similar thread today on redis-db detailing how to debug/diagnose it on redis side. That will be helpful too.

Thank you very much.

Neelesh

Sent from my iPhone