Why is Gearman slow?

[pcdinh]

Hi all,

I have made a simple script to test how fast Gearman's asynchronous
and persistent job is. However, I am a bit disappointed. It turned out
that Gearman is much slower than I think. I am using TokyoCabinet as
storage backend. TokyoCabinet is advertised to be very fast.

It takes me 30 - 31s to send 1000 background jobs to Gearmand. It
means that 1 asynchronous job takes 0.3s

- PHP 5.2.13
- Gearmand 0.13 + tokyocabinet 1.4.44 (for persistent job)
- Gearman PHP Extension (0.7.0)
- CentOS 5.4
- Start command: gearmand -d -uroot -q libtokyocabinet --
libtokyocabinet-file=/tmp/tokyo.tch --port=4730
- PHP Code: http://gist.github.com/403660
- My disk speed test

/dev/cciss/c0d0p1:
Timing cached reads: 2908 MB in 2.00 seconds = 1454.40 MB/sec
Timing buffered disk reads: 76 MB in 3.04 seconds = 25.03 MB/sec

Test results
==========

[root@clipserver clients]# ./loop_client_reverse.php
Sending 1000 job
Elapsed time: 30.6671800613
[root@clipserver clients]# ./loop_client_reverse.php
Sending 1000 job
Elapsed time: 31.2572209835
[root@clipserver clients]# ./loop_client_reverse.php
Sending 1000 job
Elapsed time: 30.9182720184

Has someone experienced such as slowness?

Regards,

Dinh

[Jay Paroline]

Hi there,

The first thing I'd do is try it without persistent storage and see
what that does. That will help to narrow down the source of the
slowness for you.

Jay

[James M. Luedke]

Any chance you could link your client code?

-James

[Clint Byrum]

On May 17, 2010, at 4:32 AM, pcdinh wrote:

> Hi all,
>
> I have made a simple script to test how fast Gearman's asynchronous
> and persistent job is. However, I am a bit disappointed. It turned out
> that Gearman is much slower than I think. I am using TokyoCabinet as
> storage backend. TokyoCabinet is advertised to be very fast.
>
> It takes me 30 - 31s to send 1000 background jobs to Gearmand. It
> means that 1 asynchronous job takes 0.3s
>

While refactoring the tokyocabinet driver I was only able to achieve about 2000 job changes per second on very nice hardware (DL380 G5 w/ 4 disk RAID5 and 512MB battery backed write cache). Without that, it would not be very fast. The issue is that currently we must call the sync/flush every time a job is added/deleted. Even with hardwhere where that is only writing into RAM, its still slows everything down as we have to block on it in the main processing thread (I think, Eric or Brian may have to confirm or deny that).

I believe there is an open blueprint to add a "flush thread" where you can have gearmand call the persistence layer flush every X seconds, or X jobs, allowing for some loss but retaining 99.9% of the data. If there isn't also a call for a redis-style "redo log + forked snapshot" style of persistence, there should be, as that would at least allow for append-only writing.

[Brian Moon]

In my recent testing, I found that the way you are looping there and
sending jobs one at at time is the slowest possible way to run gearman
jobs. Just saying that you may be benchmarking something other than
gearmand there.

<?php

$jobs = array();

for($x=0;$x<1000;$x++){
$jobs[] = array(
"function" => "reverse",
"args" => "foo"
);
}

Gearman::run_multi($jobs);

?>

# time php test.php

real 0m0.134s
user 0m0.117s
sys 0m0.016s


Gearman::run_multi is our internal wrapper for Net_Gearman. This is
with an sqlite queue.

Brian.
--------
http://brian.moonspot.net/