Jedis causing Redis 100% CPU

[SBoyle]

I'm having an issue with Jedis and Redis where when I push over 4K
operations/second redis goes to 100% CPU. I am using Jedis Pooling.
I see on the redis side that redis's CPU usage is about 3% User space
CPU and about 97% Sytem space CPU. Has anyone else seen this kind of
behavior? What could I be doing wrong to cause this type of problem?

Using redis benchmark against the same redis instance with a similar
command set I can get 80K operations/second. Not sure how to dig into
to Jedis to see what is going on. Would appreciate some feedback.

There are some more details available on our redis instance in the
Redis DB google group:
http://groups.google.com/group/redis-db/browse_thread/thread/38276975e8537a2e


Thanks,
Steve Boyle

[Eric Hauser]

I've been load testing an application using Jedis for the past week,
and I am able to get ~40K operations per second on a laptop (20 SETBIT
operations in a pipeline). However, I dumped commons pool awhile ago
after I was able to regularly get it to deadlock under heavy load and
saw lots of people reporting similar findings. If you are interested
in the pool I am using, it is available here:

https://gist.github.com/1162766

I was going to look into the commons pool issue, but it does not
provide any useful metrics about the state of the pool, so I just went
another direction that gave me more insight.

[Jonathan Leibiusky]

Interesting. It really seems like an issue in the pool. Maybe a
configuration thing.
Can you share your conf so I can try to reproduce this?

[SBoyle]

I had seen a connection thrashing issue with the pool when I had
maxIdle set to less than maxActive in the pool and the actual number
of connections in use was less than maxIdle - I sorted that out. I
put the pool config in a file that looks like this:
# jedis pool settings, documented here
# http://commons.apache.org/pool/apidocs/org/apache/commons/pool/impl/GenericObjectPool.html
minIdle = 1
maxIdle = 15
maxActive = 15
connTimeout = 299000
# see http://commons.apache.org/pool/apidocs/constant-values.html
whenExhaustedAction = 1
testWhileIdle = false
whenExhaustedMaxWait = -1
minEvictableIdleTimeMillis = 125000
timeBetweenEvictionRunsMillis = 120000
numTestsPerEvictionRun = 3


My (scala) code that starts the JedisPool looks like this:
def startCorePool(): JedisPool = {
try {
// create a new jedis configuration and set attributes from the
configuration file
val dbConf = new JedisPoolConfig()
dbConf.setMinIdle(config.getInt("ccm.redis.minIdle", 5))
dbConf.setMaxIdle(config.getInt("ccm.redis.maxIdle", 25))
dbConf.setMaxActive(config.getInt("ccm.redis.maxActive", 25))

dbConf.setWhenExhaustedAction(config.getInt("ccm.redis.whenExhaustedAction",
1).toByte)

dbConf.setTestWhileIdle(config.getBool("ccm.redis.testWhileIdle",
false))

dbConf.setMaxWait(config.getInt("ccm.redis.whenExhaustedMaxWait",
60000))

dbConf.setMinEvictableIdleTimeMillis(config.getInt("ccm.redis.minEvictableIdleTimeMillis",
-1))

dbConf.setTimeBetweenEvictionRunsMillis(config.getInt("ccm.redis.timeBetweenEvictionRunsMillis",
30000))

dbConf.setNumTestsPerEvictionRun(config.getInt("ccm.redis.numTestsPerEvictionRun",
-1))

// get redis access info from config file
val redisHost = config.getString("ccm.redis.coreHostname",
"corehost")
val redisPort = config.getInt("ccm.redis.corePort", 6379)
val redisAuth = config.getString("ccm.redis.coreAuth", "")
val redisTimeout = config.getInt("ccm.redis.connTimeout", 2000)
// create the jedis pool
val pool = new JedisPool(dbConf, redisHost, redisPort,
redisTimeout, redisAuth)
logger.info("CORE Redis connection pool instantiated.")
//return the pool
pool
}
catch {
case e: InstantiationException => {
logger.info("Could not instantiate CORE Redis connection
pool", e)
throw new RuntimeException(e)
}
case unknown => {
logger.info("CORE Redis unhandled exception")
throw new RuntimeException(unknown)
}
}
}

Thanks for taking a look.

Regards,
Steve Boyle

On Aug 22, 9:12 am, Jonathan Leibiusky <ionat...@gmail.com> wrote:
> Interesting. It really seems like an issue in the pool. Maybe a
> configuration thing.
> Can you share your conf so I can try to reproduce this?
>

> On 8/22/11, Eric Hauser <ewhau...@gmail.com> wrote:
>
>
>
> > I've been load testing an application using Jedis for the past week,
> > and I am able to get ~40K operations per second on a laptop (20 SETBIT
> > operations in a pipeline).  However, I dumped commons pool awhile ago
> > after I was able to regularly get it to deadlock under heavy load and
> > saw lots of people reporting similar findings.  If you are interested
> > in the pool I am using, it is available here:
>
> >https://gist.github.com/1162766
>
> > I was going to look into the commons pool issue, but it does not
> > provide any useful metrics about the state of the pool, so I just went
> > another direction that gave me more insight.
>

> > On Mon, Aug 22, 2011 at 11:48 AM, SBoyle <sboyle...@gmail.com> wrote:
> >> I'm having an issue with Jedis and Redis where when I push over 4K
> >> operations/second redis goes to 100% CPU.  I am using Jedis Pooling.
> >> I see on the redis side that redis's CPU usage is about 3% User space
> >> CPU and about 97% Sytem space CPU.  Has anyone else seen this kind of
> >> behavior?  What could I be doing wrong to cause this type of problem?
>
> >> Using redis benchmark against the same redis instance with a similar
> >> command set I can get 80K operations/second.  Not sure how to dig into
> >> to Jedis to see what is going on.  Would appreciate some feedback.
>
> >> There are some more details available on our redis instance in the
> >> Redis DB google group:

> >>http://groups.google.com/group/redis-db/browse_thread/thread/38276975...
>
> >> Thanks,
> >> Steve Boyle

[Bobby]

I am having a similar problem but without issuing any redis commands.
I am using spring and have posted over there but without issuing any
actual redis commands, just creating an instance of a Jedis Connection
object my cpu hits 100%. I noticed here you guys mentioned pooling so
I both set that to true and false but no luck. However sending
commands works just fine in a timely manner. I wonder are you sure
that the cpu is taxed because of commands or if it is there before but
you just send the commands immediately? Just a thought.

> Redis DB google group:http://groups.google.com/group/redis-db/browse_thread/thread/38276975...
>
> Thanks,
> Steve Boyle

[SBoyle]

To be clear, I'm talking about the redis instance on the redis server
going to 100% cpu, not my client side using 100% cpu. Also, the
problem only occurs for me when I get over about 4k commands/per
second to redis. When my client is idle and not issuing commands to
redis then there is no cpu problem for me. I'm still investigating
the issue, have not received much help from the community.
Personally, I'm reday to drop Jedis for something else (just a matter
of finding the time) - we have other software/clients that connect to
redis through other libs (like one of the C libs) that do not have
this issue.

-Steve