errors when using select

[SBoyle]

I just updated to jedis 2.0. My project compiles fine but at run time
I am getting the following error:
[java.lang.ClassCastException: java.lang.Long cannot be cast to [B
at redis.clients.jedis.Connection.getStatusCodeReply(Connection.java:
152)
at redis.clients.jedis.Jedis.select(Jedis.java:339)
at CCMRedisActor.pushSiteData(CCMActorRedis.scala:273)

This worked perfectly in jedis 1.5.2. I am using the jedis pool, I
get a connection and then issue the select command. The error seems
to happen in different places at different times but always seems to
fail on the select command.

Any ideas?

[f00biebletch]

We've seen similar behavior as well under 2.0, although not in select,
e.g.

java.lang.ClassCastException: [B cannot be cast to java.lang.Long
at
redis.clients.jedis.Connection.getIntegerReply(Connection.java:178)
at redis.clients.jedis.BinaryJedis.hset(BinaryJedis.java:677)

We have only observed this at relatively high volumes. We are using
the sharded pool.

[Jonathan Leibiusky]

usually a ClassCastException occurs when using a Jedis instance from different threads at the same time. It is a race condition problem.
Please check if that is the case. Remember that Jedis instances are NOT threadsafe.

[Dean]

Are instances issued from the pool threadsafe, provided that they're
not shared cross-threads?

Cheers.

[Jonathan Leibiusky]

the pool itself is threadsafe but not instances from the pool.
so if you get an instance from the pool and then share it among several threads you'll get a ClassCastException.

[SBoyle]

Okay, assuming that instances from the pool are only used by a single
thread, what else can cause the ClassCastException?

Thanks.

[f00biebletch]

We are using ShardedPool only between threads and relying on the
synchronization inside that class, and not sharing jedis instances
between threads, although the symptom does smell like a race
condition.

In addition to reviewing our code for possible bugs, we are working on
reproducing the behavior in a smaller stand-alone test.

[Jonathan Leibiusky]

Nothing else that I am aware of beside a bug. Based on past experience almost always that exception is because of problems with threads.
But I am willing to debug and help you with your problem. If you have some piece of code to share that reproduces this, I will help you :) you can pastebin it or send it here or to me directly.

[Jonathan Leibiusky]

ok, that would be very very helpful. once you have that piece of code I can help you find the problem, and if it is a bug we can fix it very quick.

[ib84]

ah, good to know that, if was doing exactly that, handle around Jedis instances. 

by the way, isn't there an error / ambigous formulation in the wiki: first, it says "regular Jedis is not threadsafe", from what follows, that "regular" implies to me, that when using with JedisPool, a Jedis instance is then threadsafe. It should simply say "Jedis is not threadsafe".

Furthermore, is a JedisPool a pooled connection rather than a "pool of reusable Jedis instances"? 

[SBoyle]

You are correct, the was a bug were I did not get the response after
using a pipeline. The next thing to get that same connection out of
the pool would get the response and thus the error.

Thanks for pointing me in the right direction.

[Triet]

Jonathan,

I have created a sample load-test app that reproduces this problem.
My observation is that it seems to be related to memory usage + GC.

Here is the setup:
- 4 redis servers runs on a Mac Pro.
- load-test app runs on a Mac Book Pro.
- load-test app has default heap size and default GC
- monitor GC of the app with 'jstat -gc'

Observation: the ClassCastException tended to occur when OU (Old Space
Utilization) approaching OC (Old Space Capacity), which triggered FGC
(Full GC).

I will send the app directly to you @iona...@gmail.com.

Thanks,

Triet

On Jun 29, 4:01 pm, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> ok, that would be very very helpful. once you have that piece of code I can
> help you find the problem, and if it is a bug we can fix it very quick.
>

[Jonathan Leibiusky]

I'm now confused. Is this related to the pipeline issue that SBoyle was experiencing or you have some other issue that is similar because the exception is the same?

[Jonathan Leibiusky]

Sure! You are totally right. Please change it if you can. Thanks a lot!

[Triet Kha]

We are not using pipeline in this app.

Triet

[Triet]

Jonathan,

Did you get the code i sent you? Is that helpful at all? Please let
me know if i can assist in any way.

Thanks,

Triet

On Jul 7, 7:03 pm, Triet Kha <triet....@gmail.com> wrote:
> We are not using pipeline in this app.
>
> Triet
>

> On Jul 7, 2011, at 5:43 PM, Jonathan Leibiusky <ionat...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I'm now confused. Is this related to the pipeline issue that SBoyle was experiencing or you have some other issue that is similar because the exception is the same?
>

> > On Thu, Jul 7, 2011 at 8:25 PM, Triet <triet....@gmail.com> wrote:
> > Jonathan,
>
> > I have created a sample load-test app that reproduces this problem.
> > My observation is that it seems to be related to memory usage + GC.
>
> > Here is the setup:
> > - 4 redis servers runs on a Mac Pro.
> > - load-test app runs on a Mac Book Pro.
> > - load-test app has default heap size and default GC
> > - monitor GC of the app with 'jstat -gc'
>
> > Observation: the ClassCastException tended to occur when OU (Old Space
> > Utilization) approaching OC (Old Space Capacity), which triggered FGC
> > (Full GC).
>

> > I will send the app directly to you @ionat...@gmail.com.

[Jonathan Leibiusky]

sure! got the code, hopefully will have time to spend on this over the weekend!

[Triet]

Jonathan,

Just want to follow up to see if you had a chance to investigate the
reported problem.

Thanks,

Triet

[Jonathan Leibiusky]

Would you mind sending your version of the code that reproduces this error?

I spent some time on this but couldn't reproduce the error.

On Sun, Jul 15, 2012 at 3:46 PM, jedix <winson...@gmail.com> wrote:

Hi,

We got the same problem too. Just wonder if the issue reported here was fixed or not.

Thanks

[jedix]

What we see is that this error happen in production in bursty manner -- like it occurs for a period of 2 minute every half to one hour, during that time all Jedis calls failed. So these are random errors. I doubt it has anything to do with GC since I observed that it could occur while there is additional FGC, nor FGC necessary cause it.

I have checked all our Jedis objects are obtained from the JedisPool and then return to the pool within the same method and so I would assume they won't share across threads.

Could it be that the underlying apache object pool is faulty?

The log looks like:

2012-07-16 00:33:50,236 [pool-2-thread-4] ERROR - java.lang.Long cannot be cast to [B
2012-07-16 00:33:50,237 [pool-2-thread-4] ERROR - [redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:172), redis.clients.jedis.Connection.getBulkReply(Connection.java:161), redis.clients.jedis.Jedis.get(Jedis.java:65),

[jedix]

sorry, typo. I meant "it could occur while there is no additional FGC"

[jedix]

OK, I worked around the problem by calling returnBrokenResource() on exception, setTestWhileIdle to true, and setMinEvictableIdleTimeMillis to 5 minutes. The first action may do the most help. I suspect the protocol FSM is stucking in the wrong state.

I think the Jedis connection should be able to tell if it encountered an exception and mark it as such and dispose even on returnResource() call? (it may also just replace the underlying client / protocol FSM if not in a transaction.)

And setTestWhileIde should be default to true.

Thanks

On Monday, July 16, 2012 2:20:05 PM UTC-7, jedix wrote:

[Jonathan Leibiusky]

Oh ok. I see.

So you had connection errors and was returning jedis objects to the pool, but they were broken, so next time, the pool would pick them up and you'd get these strange errors.

Not sure how the defaults you suggest can make jedis users happier :)

Can you explain better why those defaults?