Command time outs (10 secs) with Java Lettuce client. Lua based durable message queue and expiring keys.

[Jeff Bourcy]

We're using Redis in our system for interprocess communications.  We have implemented durable message queues with lists, and use Lua scripts for all access: push, pop, commit, and rollback.  The scripts are primarily used to handle functionality for a message ID (add it during push, parse it during pop, commit up until a certain ID).

We are also using key/value storage with expiration, and pub/sub on one channel.

The client we use is the Lettuce Java client based on Netty.  We do not do asychronous calls.

The problem we are seeing, is the Lettuce client gets command time outs.  The timeout has been set as high as 10secs, and we still get them.  

The slowlog shows the Lua scripts taking anywhere from 10ms to 25ms.  There are up to 2000 messages being pushed or popped at a time, by a single push client, and a single pop client per list.

I would appreciate any theories on what would cause the client to time out frequently at 10 seconds.

[Josiah Carlson]

If each command execution takes 10-25ms, and you have 2000 Lua calls executing at any given time, then those 2000 requests would take 20-50 seconds to complete. That means that anywhere from 50%-80% of your calls should be timing out.

I don't know what you are doing in your Lua scripts, but if you intend for them to actually complete, you're going to need to have them execute faster by reducing what they do, profiling your code to determine what is slowing it down, or if you are running on a slow VM, getting a faster VM.

You should also check the processor, memory, and/or network utilization of your machine to see how Redis is doing. If Redis is not using 100% CPU, yet you are having these issues, then you've got a network or memory issue. If Redis is using 100% CPU, then you need to optimize your code or upgrade your Redis machine (note that Redis is effectively single-threaded, so you nominally want a machine with 2 fast cores for Redis - one for primary execution, one for network IO and for dump/AOF rewriting). If Redis is using too much memory, you could be having swapping issues, so check the IO wait information in top.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Jeff Bourcy]

Thank you.  I just wanted to clarify, that the Lua scripts that take 10-25ms,  accept or return up to several thousand messages, not one per call.  

We have many cores on the box, but we are not dedicating any to a particular process.  Do you suggest we dedicate 2 cores fully to Redis?

- Jeff

[Josiah Carlson]

If each one runs 10-25ms, and you are returning several thousand results per call, then I believe we've just found your bottleneck. You are probably running into CPU and/or network IO issues. Give 'ifstat' a run to watch your network IO and top to watch your CPU. You may also want to watch the results of INFO; specifically the longest client buffer.

Generally speaking, assigning Redis to cores can help when you've got multiple Redis servers running on your machine. But if Redis is the only thing running on your machine, then you shouldn't need to assign it to specific cores.

 - Josiah