Mysterious JedisDataException

[UHassan]

Hello moderators,

We are trying to use Redis 2.0.1 (git-sha1: 068eb3bf) in our
production environment (2.6.28-11-generic #42-Ubuntu SMP x86_64),
where multiple multi-threaded applications try to write to a single
Redis instance (on a different machine within the same data center)
under master-slave replication using Jedis 1.5.2.

While in our test environment (low data volume, with a single, multi-
threaded application), this seems to have worked correctly; in
production, we consistently receive a large number of the following
exceptions:

2011-03-03 17:31:49,603|ERROR| ERR value is not an integer
redis.clients.jedis.exceptions.JedisDataException: ERR value is not an
integer
at redis.clients.jedis.Protocol.processError(Protocol.java:55)
at redis.clients.jedis.Protocol.process(Protocol.java:62)
at redis.clients.jedis.Protocol.read(Protocol.java:123)
at
redis.clients.jedis.Connection.getIntegerReply(Connection.java:161)
at redis.clients.jedis.Jedis.incrBy(Jedis.java:579)
...
at java.lang.Thread.run(Thread.java:619)

It seems that this exception should be thrown when Jedis#incrBy(...)
does not receive or properly encode (Using Jedis' SafeEncoder) a value
as type long, but any problem in type casting should appear as a
NumberFormatException elsewhere instead. So far, we have not been able
to replicate this error in our test environment using the same dataset
as production.

Any ideas as to why this might be happening?

[Jonathan Leibiusky]

How are you using Jedis?
Are you using JedisPool? If not, you should since it is the way to go with multiple threads and concurrency.
Check in the wiki under "Using Jedis in a multithreaded environment" (http://wiki.github.com/xetorthio/jedis/)

[UHassan]

Thank you for the quick reply, Jonathan.

We are using Jedis Pool with the following properties:

redis.timeout = 300000
jedis.pool.config.test.while.idle = true
jedis.pool.config.min.eviction.idle.time.millis = 600000
jedis.pool.config.time.between.eviction.runs.millis = 300000

Here's an example of how we use it:

public Long incrBy(Object key, Long count) {
if (key == null)
throw new IllegalArgumentException("Key must be non-null");
boolean broken = false;
Jedis jedis = null;
try {
jedis = jedisPool.getResource();
return jedis.incrBy(key.toString(), count);
} catch (JedisException jedisException) {
if (jedisException instanceof JedisConnectionException) broken =
true;
throw jedisException;
} finally {
if (jedis != null) {
if (broken) {
jedisPool.returnBrokenResource(jedis);
} else {
jedisPool.returnResource(jedis);
}
}
}
}

Thanks!

[Jonathan Leibiusky]

I can only think of 2 ways to generate this error.
The first would be doing an incrBy on a key that is not a number.
For example:
jedis.incr("foo);
jedis.set("foo", "bar");
jedis.incr("foo"); -----> EXCEPTION!

The second one involves threading, but from your code it seems like it is not possible. Although I will explain it. Every command in Jedis is doing essentially 2 things: send the command through the socket, read response from the socket.

Thread1 executes jedis.incr("foo");
while it is sending the command through the socket, Thread2 is sending another command, let say jedis.get("foo");
since you are under load, Thread2 gets through faster than Thread1, so it will read the response of Thread1 (the incr) while Thread1 will read the response of Thread2. Obviously this will lead to an exception.

So lets discard scenario 2 for now. Are you sure you are not changing the value of your incrBy key to some string somewhere in your code?

[Umayr Hassan]

Thank you for these suggestions. The problem we found was similar to scenario 1 in that some keys storing fractional values were being incremented, which resulted in this exception.