Load testing with mc-crusher

[Martin Grigorov]

Hi Dormando,

This is a continuation of the mail thread with subject "Is ARM64 officially supported ?" [1]

I've refreshed my Perl coding skills and came up with this wrapper around mc-crusher:

=================================================

#!/usr/bin/env perl

# sudo cpan Cache::Memcached

use strict;
use warnings;

use Cache::Memcached;
use Data::Dumper;
use Time::Piece;

my $TESTBED_HOME = $ENV{'TESTBED_HOME'};
die "Env variable 'TESTBED_HOME' is not set!" if ! defined($TESTBED_HOME);
my $MC_CRUSHER_HOME = $ENV{'MC_CRUSHER_HOME'};
die "Env variable 'MC_CRUSHER_HOME' is not set!" if ! defined($MC_CRUSHER_HOME);

if (scalar(@ARGV) < 2) {
        die "Usage: mc-crusher.pl <conf> <host> <port> <timeoutInSecs>. For example: mc-crusher.pl cmd_get a.b.c.d 11211 60"
}

my $config = shift @ARGV;
my $host = shift @ARGV;
my $port = shift @ARGV || 8080;
my $duration = shift @ARGV || 5 * 60; # 5 mins

system("$MC_CRUSHER_HOME/mc-crusher --conf $TESTBED_HOME/etc/memcached/mc-crusher/conf/$config.conf --ip $host --port $port --timeout $duration");

print "MC Crusher status: $!\n" if $!;

my $serverAddr = "$host:$port";
print "Server: $serverAddr\n";
my @servers = ($serverAddr);

my $memcached = new Cache::Memcached;
$memcached->set_servers(\@servers);

my $stats = $memcached->stats(['misc'])->{'hosts'}->{$serverAddr}->{'misc'};
warn Dumper($stats);

my $timestamp = localtime->datetime();
my $cmd_per_sec = int($stats->{$config}) / $duration;
my $bytes_written = $stats->{'bytes_written'};
my $bytes_read = $stats->{'bytes_read'};
my $time_system = $stats->{'rusage_system'};
my $time_user = $stats->{'rusage_user'};

my $today = localtime->ymd();
my $folder = "$TESTBED_HOME/etc/memcached/$today";
system("mkdir -p $folder");
print "Cannot create '$folder': $!\n" if $!;
my $filename = "$folder/memcached-mc-crusher-report-$host-$config.csv";

my $headerPrefix = "${host}_${config}";
open(my $fh, '>', $filename) or die "Could not open file '$filename': $!";
print $fh "timeStamp,${headerPrefix}_per_sec,${headerPrefix}_bytes_written,${headerPrefix}_bytes_read," .
    "${headerPrefix}_time_system,${headerPrefix}_time_user,config,host\n";
print $fh "$timestamp,$cmd_per_sec,$bytes_written,$bytes_read,$time_system,$time_user,$config,$host\n";
close $fh;
print "done\n";

==========================================

Basically it executes mc-crusher with different configurations against different servers (x64 vs arm64) and stores the stats in CSV files, which later I visualize in Kibana.

So far I've ran with these configs:

cmd_get.conf:

send=ascii_get,recv=blind_read,conns=50,key_prefix=foobar,key_prealloc=0,pipelines=8
send=ascii_set,recv=blind_read,conns=10,key_prefix=foobar,key_prealloc=0,pipelines=4,stop_after=200000,usleep=1000,value_size=10

cmd_set.conf:

send=ascii_set,recv=blind_read,conns=5,key_prefix=foobar,value_size=2,value=hi,key_prealloc=1

I always run those against freshly started Memcached server (started with: ./memcached -p 8080).

After mc-crusher finishes I get the value of the current command, e.g. cmd_get, and divide it by

the duration, to get the cmds per second.

And here are the numbers I have so far:

gets/s:

- x64: ~450 000

- arm64: ~310 000

sets/s:

- x64: ~25 000

- arm64: ~63 000

I.e. x64 is faster for GET and arm64 is faster for SET.

Those numbers are averages from multiple runs. 

Do you see anything wrong in my Perl script ? Or any ideas how to improve it.

Any hints how to profile the server to see why/where it is slower on different platforms ?

1. https://groups.google.com/forum/#!topic/memcached/8hT2BT9cgEc

Regards,

Martin

[Martin Grigorov]

Hi,

Does it make sense to have a command 'flush_stats' ?

'flush_all' resets the caches but the stats stay, e.g. cmd_get value is not reset.

I'd find it useful for performance measurements:

1) run random warm-up

2) reset_stats

3) run perf test

4) collect stats

Right now I do:

1) run random warm-up

2) collect stats

3) run perf test

4) collect stats

5) substract 2) from 4) and store the result

Regards,

Martin

[Emilio Fernandes]

Hola Martin,

I was able to use your Perl script to run some tests here. It took me a while to figure out how to provide the configurations but I managed :-)

I am experiencing a similar behavior on my machines - set command is faster on ARM64 and get command on x86_64.

With a colleague of mine we are trying to find out the reason for these differences with the help of 'perf' tool. http://www.brendangregg.com/perf.html is a good resource how to use it.

Gracias!

Emilio

 

1. https://groups.google.com/forum/#!topic/memcached/8hT2BT9cgEc

Regards,

Martin

--

---
You received this message because you are subscribed to the Google Groups "memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/CAMomwMrES-%2BsTxYQrK9rQNaNCPRjA2FZUOYKR_rSgRUCiz5TiA%40mail.gmail.com.

[dormando]

Hey,

I didn't try your script but it's not quite clear how it's gathering
statistics. just once at each end of the test?

I'd left some test suites from my blog posts in the test-suites/ directory
of mc-crusher, but they weren't very clean or easy to modify...

Found another one I'd done later with a refactoring that's a lot better:
https://github.com/memcached/mc-crusher/blob/master/test-suites/test-suite-example

just cleaned up some bits and wrote a few notes into it. I don't have the
if tooling I was using to process the data into json blobs (maybe I'll add
it you want it :P)

There are a bunch of parameters to tune, or tests you can outright delete.
Any mc-crusher configuration file can be represented in there (see the
"get_*_conf" subs).

This will grab a bunch of data _while_ the perf test runs, which you can
then plot or examine for consistency. It also runs a scaled set of tests,
so you can find the performance cliffs. I don't believe any benchmark is
valid unless you find the test to failure point and show what latency
looks like on your way there.

So this will run a bunch of tests and scale up the mc-crusher config. You
get:

- stats sample output (see sample_args()). You get the rate every N
seconds for X periods, and at the end it'll have an average from the full
run. You can look into this for spikes/dips/consistency.

- the stats sampling starts _after_ the benchmark starts, so you don't end
up weighting ramp up time into the average.

- latency sampled output (along with some summary information). You can
examine these files to see when the benchmark throughput has exceeded your
latency budget.

- Some other crap I probably forget.

Anyway this is how I test stuff for those blog posts with all the graphs.
You mostly tweak it a bit and ignore it for a while, then tweak and try
again.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/CAMomwMqQwxDYchMFde6vDDpB6hAqRsnznYcUc5HP2RkKUW-v_Q%40mail.gmail.com.
>
>