Measuring utilization of the Redis worker thread

[George Necula]

Hello,

   I have a production Redis app, and I want to know how much room to grow it has from the Redis point of view. For this purpose I would like to know in a certain time interval (e.g., last 1 or 5 minutes) what percentage of the time the worker thread was idle. I want to monitor this number and use it to know how far from hitting the limits of Redis I am. Is this likely to be an effective predictor ? 

  I wonder what is the easier way to compute this number. I can try to monitor the redis-server process CPU usage (perhaps with "info cpu", but I am not sure if this will accurately reflect the worker thread's idleness, e.g., if there are other threads doing other stuff. I have thought also of using  "info commandstats" to add up the time used by all commands. 

Thanks,

George.

[Stefan Parvu]

> I have a production Redis app, and I want to know how much room to grow
> it has from the Redis point of view. For this purpose I would like to know
> in a certain time interval (e.g., last 1 or 5 minutes) what percentage of
> the time the worker thread was idle.

If I understood right, Redis runs as a single process, not using any worker threads
nor having the work split across many workers. So there cant be any other worker
threads.

So Redis shows up as a single process which multiplexes the client's requests using
for example on Linux, epoll interface.

Monitoring the redis-server process should be just enough ... But probable
you would be interested as well in:

* storage usage: latency issues, highly important when using persistence [1]

* memory usage, your redis instance will make use of the available RAM
present on the system


[1] - http://redis.io/topics/latency

--
Stefan Parvu <spa...@systemdatarecorder.org>

[Josiah Carlson]

While it is definitely the case that "info cpu" won't give you a breakdown of command execution, background IO, etc., those pieces are also overhead that Redis must pay to execute.

Aside from situations where a machine may have CPU stolen (reducing machine capacity without being wholly observable, generally only counted in VM setups), checking used_cpu_* periodically is not unreasonable for Redis CPU overhead. But in addition to monitoring just Redis CPU, you should also be monitoring other resources. Stefan mentioned storage and memory, but it also makes sense to monitor outstanding clients, new connections/second, and really just about any system or Redis-level metric that you can reasonably monitor.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[George Necula]

Thank you both for your help. 

I do plan to monitor other resources of course, but since there are many factors that can lead to overload in a Redis instance (too many clients, too many requests, requests too slow) I also wanted to have the "ultimate" CPU-capacity monitor that tells me if the database is close to overload. 

I will proceed with the assumption that on a non-VM the increase in used_cpu in a given time window tells me how close to CPU-capacity is the DB. 

-- George.

[George Necula]

I have done some experiments by calling periodically "info" and "info commandstats" and it seems that "used_cpu_user" + "used_cpu_sys" increases by more than the sum of the times in the commandstats, by a factor of 3-5. For example, in a time interval the used_cpu increases by 650ms and the sum of the command stats only by 150ms. 

 I do assume that there is more work to be done than executing commands, but I did not expect so big of a discrepancy. I know that there is a single worker thread for executing commands, but are there other threads in Redis for doing bookkeeping or other stuff? Is the accounting for the command execution time in commandstats ignoring some work that must be done for each command (e.g., to send the results out, perhaps by a separate thread)? 

If there is a single thread, then the used_cpu time tells me when I am close to saturating the CPU. If there are multiple threads, then the sum of the commandstats tells me when I am close to saturating the CPU. 

Thanks,

George.

[Josiah Carlson]

On Fri, Oct 31, 2014 at 2:37 PM, George Necula <gcne...@gmail.com> wrote:

I have done some experiments by calling periodically "info" and "info commandstats" and it seems that "used_cpu_user" + "used_cpu_sys" increases by more than the sum of the times in the commandstats, by a factor of 3-5. For example, in a time interval the used_cpu increases by 650ms and the sum of the command stats only by 150ms. 

The 'INFO COMMANDSTATS' output is the result of only counting the actual command execution times (check src/redis.c, lines 2009-2011 in current unstable, starts with "start = ustime();"), and ignores everything else that Redis does that isn't executing a command (socket IO, disk IO, ...). On the other hand, the used_cpu_* values are retrieved from the kernel via the getrusage() call, and accounts for *all* CPU used by the process - not just directly in the execution of commands.

 I do assume that there is more work to be done than executing commands, but I did not expect so big of a discrepancy. I know that there is a single worker thread for executing commands, but are there other threads in Redis for doing bookkeeping or other stuff? Is the accounting for the command execution time in commandstats ignoring some work that must be done for each command (e.g., to send the results out, perhaps by a separate thread)? 

It ignores IO, background key expiration, loading of a snapshot, and anything else that isn't the actual implementation of the function call itself. The actual code that accounts for this is:

    start = ustime();

    c->cmd->proc(c);

    duration = ustime()-start;

And a little later...

    if (flags & REDIS_CALL_STATS) {

        c->cmd->microseconds += duration;

        c->cmd->calls++;

    }

If there is a single thread, then the used_cpu time tells me when I am close to saturating the CPU. If there are multiple threads, then the sum of the commandstats tells me when I am close to saturating the CPU.

Your analysis is incorrect. Single-threaded or multi-threaded, the time accounted for in command stats is not even close to all the things that Redis does. The best indicator in Redis is the used_cpu_* results, which accounts for the CPU-related work that involves Redis in all threads.

 - Josiah

[George Necula]

Thank you, I will use the used_cpu measures. 

George.