Gearman throughput
Nick Andrew
on an OpenVZ virtual machine. I'm using trivial transactions to try
to understand Gearman's raw throughput characteristics. I've tried
various combinations of synchronous and asynchronous clients, number
of worker processes, and number of virtual machines (up to 3).
I'm seeing an exponential slowdown in Gearman as the number of worker
processes rises. Specifically:
1 x virtual machine
1 x Gearman server process
1 x Asynchronous client process
N x worker processes
As N increases from 1 to 49 the total throughput drops exponentially,
from a peak of 1600/sec with 1 worker down to 200/sec with 49 workers.
I feel like I must be doing something wrong. Here's the worker process
code:
#!/usr/bin/perl
use strict;
use Gearman::Worker qw();
my $worker = Gearman::Worker->new();
$worker->job_servers('127.0.0.1');
$worker->register_function('hello' => \&handle_hello);
my $count = 0;
while (1) {
$worker->work();
}
exit(0);
sub handle_hello {
$count ++;
return $count;
}
Not much there, right?
And the asynchronous client code:
#!/usr/bin/perl -w
use strict;
use Gearman::Client qw();
my $workers = 1;
while ($workers < 50) {
time_workers();
system("./worker.pl &");
$workers ++;
}
exit(0);
sub time_workers {
my $client = Gearman::Client->new();
if (! $client) {
die "Unable to make new Gearman::Client - $@";
}
$client->job_servers('127.0.0.1');
my $start = time();
my $runs = 0;
my $taskset = $client->new_task_set();
while (time() < $start + 30) {
foreach my $iteration (1..100) {
$taskset->add_task("hello", 44);
$runs ++;
}
}
$taskset->wait();
my $end = time();
printf("%d workers, Ran %d times in %d seconds (%d per second)\n",
$workers,
$runs,
($end - $start),
$runs / ($end - $start),
);
my $speed = $runs / ($end - $start);
printf("%d %d\n", $workers, $speed);
}
The client code runs in a loop to test the throughout as the number of
worker processes increases from the start of 1 to the limit of 49.
I'll attach a graph to this message.
I'm seeing about 20,000 context switches/second in this scenario. The
machine is a quad-core AMD x86_64 processor, 8 Gigs RAM, otherwise
lightly loaded.
If I use a synchronous client which calls do_task(...) in a loop,
the slowdown is even more dramatic, from approx 900/sec with a
single worker down to 75/second with 49 workers.
I notice in http://lists.danga.com/pipermail/gearman/2007-December/000053.html
Joe Stump (at Digg) posted about the same issue.
Tracing the worker processes I notice that they all communicate with
the server every 10 seconds when idle.
Tracing the gearmand server, it seems that the server communicates
with almost all of the worker processes before choosing one and
forwarding the request. This is why it is drowning my box with
context switches.
Can anybody advise on speeding this up and improving its scalability
with increased numbers of worker processes?
Nick.
--
PGP Key ID = 0x418487E7 http://www.nick-andrew.net/
PGP Key fingerprint = B3ED 6894 8E49 1770 C24A 67E3 6266 6EB9 4184 87E7
Joe Stump
On Sep 3, 2008, at 10:56 PM, Nick Andrew wrote:
> I'm seeing an exponential slowdown in Gearman as the number of worker
> processes rises.
This is a known (and previously discussed) issue with Gearman. Chris
saw this at Yahoo! and I saw it at Digg. Most people on the list will
tell you to run CPU * Cores * 2 workers. At Digg we have a gearmand
running on each web server so it just submits jobs to itself (along
with the formula I just mentioned). I think we're doing around 300,000
jobs a day without any issues.
--Joe
Nick Andrew
> This is a known (and previously discussed) issue with Gearman.
It seems that the server talks to (nearly) every worker on every request.
Is this necessary? Why can't the server talk to only the worker which
it's forwarding the request to?
> Chris saw
> this at Yahoo! and I saw it at Digg. Most people on the list will tell
> you to run CPU * Cores * 2 workers. At Digg we have a gearmand running on
> each web server so it just submits jobs to itself (along with the formula
> I just mentioned). I think we're doing around 300,000 jobs a day without
> any issues.
Ok, your use case is different to mine. I need much more parallelism:
40 or more gearman workers running in parallel, each one might take
0.1 seconds to run but the jobs can't be allowed to queue up behind
only 2 workers, and I can't dedicate 20+ processors for this.
Nick.
dormando
> It seems that the server talks to (nearly) every worker on every request.
> Is this necessary? Why can't the server talk to only the worker which
> it's forwarding the request to?
I think your benchmark is finding a bias in the design.
The flow is:
- worker connects to all gearmand servers.
- worker registers what functions it supports.
- worker asks for jobs.
- if no jobs, sends command 'pre_sleep' to all gearmand's and sleeps.
Client does:
- Connect to gearmand.
- submit's a job for a particular func.
Gearmand does:
- Acks the job, finds all *sleeping workers* related to the function.
- Sends them all a 'noop' command to wake them up.
Worker does:
- Urk, I'm awake now.
- Worker asks for jobs.
- If jobs, do work.
- If no jobs, sends command 'pre_sleep' to all gearmand's, etc.
---
What your benchmark is doing:
- From a *single* client, serially:
- submit a job.
- that completes almost instantly.
- submits another job.
... so as you scale workers, the number of bored workers will increase and
the number of wakeup calls will increase, which lowers throughput.
I'm not actually going to write this test, but you should be able to
confirm the other behavioral extreme by doing the following:
- Write a worker that for the *first* request does a `sleep 120`
- For all further jobs, process as fast as possible.
- Fire up parallel job submissions maybe 5 or so. Fewer than the count of
workers for purposes of the test.
- Submit at least 20,000 jobs.
- Watch the dequeue rate once the workers come back to life.
- Run the benchmark across a scale of workers 1..100
In this case (since all workers are busy) adding a new job should be
practically a noop, and the rate the jobs are removed should be crazy
fast.
In reality you would expect at least a certain percentage of your workers
to not be bored. If they are you're probably running too many of them...
---
A possible adjustment:
- random sort the list of sleepers.
- wake up 5 (arbitrary number).
- If more than 5 sleepers exist, add a Danga::Socket timer to check back
in 0.1 seconds or so.
- On firing of timer, and if there are still unclaimed jobs, wake up X
more then reschedule timer.
... then up for debate on whether or not to wake up more from that list
for every individual job submitted. If you wake up at least one worker per
job submission, you can have 40 jobs arrive in 0.1 seconds and have all 40
possible workers wake up within that 0.1 seconds.
So each job submission would attempt to wake up 2-5 processes until
there're no sleepers, with sleepers automatically woken up every 0.2
seconds if a queue exists, until they're all awake.
---
Obviously the other entirely different method is to have gearmand push
jobs to sleeping workers, instead of sending the noop and having them run
a grab call.
That would require more reworking in gearmand to add reliability and help
guarantee job latency in the face of dead/dying workers.
We know all of who is still sleeping, but not of which are still
alive. Which is why the algorithm works the way it does now. We can tell
all workers 'noop, wake the fuck up', but only the ones which aren't in
the process of timing out their connection will come back and grab their
job instantly.
Or just single-fire noops, I guess. Which is the above suggestion.
Workers run exactly as they do now. wait for a noop to wake up, and grab
jobs when they do. That gives the confirmation cycle a chance to work.
This modification does increase the odds of minor latency blips happening
a little, but removes that "walk all idle servers on job submission"
thing.
If you folks think it'd really make a difference, although I suspect in
production you won't see such a dramatic amount of time spent waking up
workers, I could prototype up this change pretty easily and submit a patch
this week.
Bribes warmly welcomed, but not required.
-Dormando
Ron Gorodetzky
Hash: SHA1
The benchmark you setup isn't really the sort of thing one would use
gearman for. In fact, I'm not sure normal threads or multiprocessing
would do spectacularly well with this sort of benchmark either. The
operation you have the worker doing is way outweighed by the overhead in
just setting up and managing the job. You should test with a job that
is closer to what you actually want your jobs to do.
One thing to remember is that for cpu bound operations you're not going
to magically get more throughput by running many more processes. Each
cpu core can only do one thing at a time (this is a reasonable
simplification). So, with 50 workers that are trying to do cpu bound
operations (say rendering a movie or calculating primes) they will all
be fighting for cpu resources. That's why a good rule of thumb in that
situation is to keep the number of threads/processes/workers close to
the number of cores. If their functions will try to use 100% cpu they
won't be contending for the cpu resource cause they'll each have a whole
core to themselves.
Now, you _can_ gain some nice performance improvements with many workers
per core if the kind of work your workers are doing includes some
network I/O. So while one worker is waiting on the network another
worker can make use of the cpu. Or if there is a timing component to
your workers where they may decide to sleep for some reason, then yeah,
run a bunch of them.
The limitation you're running into, is not in gearman really. It's just
a truth about multi/parallel processing in general. It just so happens
that the gearman multiprocessing has some more overhead as a trade off
for its flexibility.
At digg many of our workers' jobs spend much of their time waiting on
the network, so we've found that 20 workers provides a nice balance of
resource usage. This is on the same servers that run apache/php on dual
dual-core machines (4 cores total). All told there are around 100-150
processes available to do work on those servers (gearman, apache etc.),
but since they're mostly network i/o bound the cpu tends to stay in the
40-60% utilization range. If a handful of those processes decide to eat
up cpu by spinning in some tight loop or something, then not much else
would be able to run. This would only happen if there was some bug in
our code, which, of course, never ever happens.
- -Ron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFIv7awD70UdgAYraURAndwAJ9y84T8BbBfA0/rLEzDXn+gKMg4tACeLAlX
i9E/upGPwQD2/VsXyQGokg4=
=RqG2
-----END PGP SIGNATURE-----
Nick Andrew
On Thu, Sep 04, 2008 at 01:01:58AM -0700, dormando wrote:
> I think your benchmark is finding a bias in the design.
>
> The flow is:
>
> - worker connects to all gearmand servers.
> - worker registers what functions it supports.
> - worker asks for jobs.
> - if no jobs, sends command 'pre_sleep' to all gearmand's and sleeps.
>
> Client does:
>
> - Connect to gearmand.
> - submit's a job for a particular func.
>
> Gearmand does:
>
> - Acks the job, finds all *sleeping workers* related to the function.
> - Sends them all a 'noop' command to wake them up.
>
> Worker does:
>
> - Urk, I'm awake now.
> - Worker asks for jobs.
> - If jobs, do work.
> - If no jobs, sends command 'pre_sleep' to all gearmand's, etc.
Thanks for the protocol explanation. I'm still not sure why Gearmand
needs to wake up all sleeping workers; if it has 1 job in its queue
then it needs to only wake up one of them. Retries (in the case of
dead workers) can be sent to other workers after a delay.
> What your benchmark is doing:
>
> - From a *single* client, serially:
> - submit a job.
> - that completes almost instantly.
> - submits another job.
I've tried submitting concurrently too (I posted the code for the
concurrent script) and ran multiple clients too. I got some rate
improvement with 3 clients but it wasn't spectacular.
> ... so as you scale workers, the number of bored workers will increase
> and the number of wakeup calls will increase, which lowers throughput.
Yep. And that's really counter-intuitive. If there's a worker with
nothing to do, I expect it to not slow down the rest of the system
(except inasmuch as it's using memory, maybe a small amount of server
comms traffic).
A typical graph of total system throughput versus number of workers
should show throughput increasing rapidly at first and then leveling
off, like the pink line in this Wikipedia image:
http://en.wikipedia.org/wiki/Image:Parallelization_graph.jpg
The line in that particular image levels out but often once the
sweet spot is exceeded, total throughput decreases. Unfortunately
for Gearman, every idle worker process decreases total throughput
and my benchmark showed an exponential decay.
But that's with a worker process which does nothing. So the 20k context
switches per second and the 100% CPU utilisation was pure gearman
overhead, no _actual_ work was being done.
You suggested the following test:
> I'm not actually going to write this test, but you should be able to
> confirm the other behavioral extreme by doing the following:
>
> - Write a worker that for the *first* request does a `sleep 120`
> - For all further jobs, process as fast as possible.
> - Fire up parallel job submissions maybe 5 or so. Fewer than the count of
> workers for purposes of the test.
> - Submit at least 20,000 jobs.
> - Watch the dequeue rate once the workers come back to life.
> - Run the benchmark across a scale of workers 1..100
>
> In this case (since all workers are busy) adding a new job should be
> practically a noop, and the rate the jobs are removed should be crazy
> fast.
I didn't have time to do that one, but I did have time to add a fixed
delay of 0.1 seconds into _every_ worker function. The theory was, based
on your protocol description above, if I keep the workers busy then they
won't slow down the server and I should see a linear throughput increase
up to 49 workers and 490 transactions/second.
Also the 0.1 second delay doesn't use any CPU time so it's inherently
massively parallelisable.
When I ran it I got the throughput graph attached - it peaks at about
26 worker processes and 200/sec, stays flat for a while then goes
downhill rapidly beyond 41 workers. Somehow the client can't keep all
those workers busy.
I tried it again, tweaking the client to send more jobs before waiting
for the first ones to finish and was able to increase the total
throughput to 200/sec at 49 workers. That's higher throughput than
when the workers were doing nothing (again counter-intuitive).
I just tried running a few clients in parallel: 5 concurrent clients
and 49 workers achieved 540/sec. 15 concurrent clients and 49 workers
achieved 617/sec. I'm not sure why it's higher than 500 which should
be the maximum possible throughput, but it certainly proves the point
that the throughput only scales up for busy workers (and idle hands
are doing the devil's work).
> A possible adjustment:
>
> - random sort the list of sleepers.
> - wake up 5 (arbitrary number).
> - If more than 5 sleepers exist, add a Danga::Socket timer to check back
> in 0.1 seconds or so.
> - On firing of timer, and if there are still unclaimed jobs, wake up X
> more then reschedule timer.
It sounds reasonable. But why 5? Reduction of latency for further jobs?
> Obviously the other entirely different method is to have gearmand push
> jobs to sleeping workers, instead of sending the noop and having them run
> a grab call.
Yes, that's how I expected it would work.
> If you folks think it'd really make a difference, although I suspect in
> production you won't see such a dramatic amount of time spent waking up
> workers, I could prototype up this change pretty easily and submit a
> patch this week.
There are a few variables. If your workers take time to run, but are not
CPU-bound, then increased parallelism will increase total throughput
without increasing latency. If you want to be able to handle bursty
traffic without latency issues, that's why you'd want to run more
workers.
My expected need for gearman covers a few different use cases. One is
to fire off infrequent long running jobs on whichever server is
available and gearman admirably handles that already. Another is to
split a job which is currently large and sequential into many smaller
pieces which can process independently. Gearman is fine for that too.
But the final requirement is to replace an existing server process
which is a bit inefficient. This server frequently has tens or
hundreds of requests outstanding; load is bursty. It reads a big iron
database. To use Gearman in this scenario Gearman will need to have
tens of worker processes which should not be putting load on the system
when request load is light.
> Bribes warmly welcomed, but not required.
If you submit a patch then I'll owe you one for later.
Nick.
Nick Andrew
Thanks for your reply.
On Thu, Sep 04, 2008 at 03:21:37AM -0700, Ron Gorodetzky wrote:
> The benchmark you setup isn't really the sort of thing one would use
> gearman for.
Gearman's great for distributing work. I need to know how much work it
can distribute and how well it scales to larger workloads. There are
two main metrics required: transactions per second and latency (the
latency will depend on how much work is put through it). And after
that, how to grow it to even more transactions per second.
I thought that using null worker processes would reveal the maximum
transactions/second, and apparently it did, at around 800-1500/sec
on my box with a single client and a single worker. 800-1500 isn't
all that hot, and I want Gearman to be really, really fast. There's
a quote on the net here:
http://grokbase.com/topic/2007/08/23/catalyst-a-perl-message-queue/VGq_dYOdDincw7KLsQYLAGlF2vo
"Good message queue systems can exceed 100,000 messages per second in
throughput. They say OpenAMQ does, for example."
I don't need 100k/sec, but 10k/sec is certainly desirable.
So the next question is how to scale Gearman up. Multiple virtual
servers and multiple clients helped increase the total throughput
but not greatly, however multiple workers sent throughput down
because the server spent time talking to idle workers rather than
sending out work.
> In fact, I'm not sure normal threads or multiprocessing
> would do spectacularly well with this sort of benchmark either. The
> operation you have the worker doing is way outweighed by the overhead in
> just setting up and managing the job.
Well, in the use case I mentioned, 0.1 second runtime is about 100 times
the gearman overhead, so I think that's an acceptable amount of
overhead. Where I've got a problem with the current design is that
if I have too many workers then there will be idle workers which
suck system resources through CPU time and context switches. And if
I have too few workers then jobs will queue up waiting for free
workers which increases the latency for the jobs. And when traffic
bursts I'll be seeing really long queues for too few workers.
> One thing to remember is that for cpu bound operations you're not going
> to magically get more throughput by running many more processes. Each
> cpu core can only do one thing at a time (this is a reasonable
> simplification).
Certainly, but my workload isn't CPU bound. It's not like, say,
converting images to thumbnails which requires primarily number
crunching.
So the test I did just then which added an 0.1 second delay in each
worker function is actually a better approximation to my expected
workload and it does show Gearman scaling better, but only to the
extent that the 49 worker processes are kept busy.
> The limitation you're running into, is not in gearman really. It's just
> a truth about multi/parallel processing in general.
For CPU-bound jobs yes, sure, the maximum throughput is limited by the
quantity of CPU provided. But what I found is an actual design problem
in Gearman (or Gearman's protocol) whereby idle workers increase the
total amount of effort required to process every job.
Nick.
dormando
> needs to wake up all sleeping workers; if it has 1 job in its queue
> then it needs to only wake up one of them. Retries (in the case of
> dead workers) can be sent to other workers after a delay.
The idea is to guarantee low latency by waking up all workers. Again, we
don't know which ones are dead or dying, so the alternative could add 0.1+
seconds of lag on picking up the job. Waking up all workers lowers the
chance of you telling a dead guy to do work.
> I've tried submitting concurrently too (I posted the code for the
> concurrent script) and ran multiple clients too. I got some rate
> improvement with 3 clients but it wasn't spectacular.
At no delay in work I wouldn't expect that to get (much) better.
> Yep. And that's really counter-intuitive. If there's a worker with
> nothing to do, I expect it to not slow down the rest of the system
> (except inasmuch as it's using memory, maybe a small amount of server
> comms traffic).
I don't believe we've ever had an issue with gearmand's performance in
production. It's actually *very* intuitive if you want to guarantee
latency like I mention above. Even if I wrote this code it's unlikely we'd
enable the option :( A lot of gearman jobs we run are syncronous with a
web page load, so we'd prefer to guarantee latency and eat a little extra
CPU.
> I didn't have time to do that one, but I did have time to add a fixed
> delay of 0.1 seconds into _every_ worker function. The theory was, based
> on your protocol description above, if I keep the workers busy then they
> won't slow down the server and I should see a linear throughput increase
> up to 49 workers and 490 transactions/second.
>
> Also the 0.1 second delay doesn't use any CPU time so it's inherently
> massively parallelisable.
>
> When I ran it I got the throughput graph attached - it peaks at about
> 26 worker processes and 200/sec, stays flat for a while then goes
> downhill rapidly beyond 41 workers. Somehow the client can't keep all
> those workers busy.
>
> I tried it again, tweaking the client to send more jobs before waiting
> for the first ones to finish and was able to increase the total
> throughput to 200/sec at 49 workers. That's higher throughput than
> when the workers were doing nothing (again counter-intuitive).
>
> I just tried running a few clients in parallel: 5 concurrent clients
> and 49 workers achieved 540/sec. 15 concurrent clients and 49 workers
> achieved 617/sec. I'm not sure why it's higher than 500 which should
> be the maximum possible throughput, but it certainly proves the point
> that the throughput only scales up for busy workers (and idle hands
> are doing the devil's work).
Sure.
> It sounds reasonable. But why 5? Reduction of latency for further jobs?
Increasing the chance of hitting an alive worker. Could be as low as 2,
but I'd want to avoid 1...
> Yes, that's how I expected it would work.
Requires entirely different internals... Not for really much benefit
either. In order to avoid having a long timeout on jobs gone out to lunch,
the worker would have to ping back to confirm it got the job. Which is the
same back<->forth handshake as `noop -> fetch job` anyway.
> There are a few variables. If your workers take time to run, but are not
> CPU-bound, then increased parallelism will increase total throughput
> without increasing latency. If you want to be able to handle bursty
> traffic without latency issues, that's why you'd want to run more
> workers.
Sure, I'll give you that. For a long time we just haven't cared about
gearmand's performance given how fast it works for us. But the use case of
keeping a lot of shit idle to handle bursty traffic is worthy enough of a
patch.
I'll likely make it an option, so you can twiddle between allowing lots of
workers to idle, and having a better latency guarantee. Gimme a week-ish,
or if anyone else understands my proposal and wants to give it a shot, go
ahead.
> My expected need for gearman covers a few different use cases. One is
> to fire off infrequent long running jobs on whichever server is
> available and gearman admirably handles that already. Another is to
> split a job which is currently large and sequential into many smaller
> pieces which can process independently. Gearman is fine for that too.
> But the final requirement is to replace an existing server process
> which is a bit inefficient. This server frequently has tens or
> hundreds of requests outstanding; load is bursty. It reads a big iron
> database. To use Gearman in this scenario Gearman will need to have
> tens of worker processes which should not be putting load on the system
> when request load is light.
Okey dokey. I hope you re-read this and understand why the latency
guarantee is designed as such though.
-Dormando