Re: Periodical stalls in write intensive benchmark in Percona Server + TokuDB

[MARK CALLAGHAN]

If the workload is linkbench, then it is public as of April 1

https://www.facebook.com/notes/facebook-engineering/linkbench-a-database-benchmark-for-the-social-graph/10151391496443920

On Sun, Apr 28, 2013 at 10:01 AM, Vadim Tkachenko <va...@percona.com> wrote:

Hi, 

I am testing Percona Server + TokuDB in write intensive benchmark.

The workload is not public yet, but only because I want to polish all corners—and I can publish it if that will help to solve issue.

Basically workload is INSERT INTO .. ON DUPLICATE KEY UPDATE statements running in concurrent.

In this case I use 64 threads, and sometimes I see

[6030s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 49968.58, response time: 6.75ms (95%)

[6040s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51348.88, response time: 6.68ms (95%)

[6050s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 47735.92, response time: 6.76ms (95%)

[6060s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 48724.48, response time: 7.16ms (95%)

[6070s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 44953.92, response time: 6.73ms (95%)

[6080s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 44418.50, response time: 6.84ms (95%)

[6090s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51354.90, response time: 6.80ms (95%)

[6100s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51312.20, response time: 6.78ms (95%)

[6110s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 49050.80, response time: 6.77ms (95%)

[6120s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 36474.26, response time: 6.66ms (95%)

[6130s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6140s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6150s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6160s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6170s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6180s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6190s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6200s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6210s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6220s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6230s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6240s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6250s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 0.00, response time: 0.00ms (95%)

[6260s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 9317.91, response time: 10.51ms (95%)

[6270s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51758.30, response time: 6.71ms (95%)

[6280s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 52051.02, response time: 6.67ms (95%)

[6290s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51696.27, response time: 6.73ms (95%)

[6300s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51877.41, response time: 6.70ms (95%)

[6310s] threads: 64, tps: 0.00, reads/s: 0.00, writes/s: 51570.61, response time: 6.77ms (95%)

That is the system  stops processing the queries.

Interesting is that it seems the system recovers after I run gdb to get all stack traces.

gdb stack traces are here:

http://gist.github.com/0e1d08445b504b0c6376

It seems like threads stuck in

#3  0x00007fe73dcb9dca in toku_mutex_lock (this=0x7fe579f7b030) at /opt/vadim/ft-engine/scripts/build-tokudb-7.0.1/ft-index/portability/toku_pthread.h:204
#4  toku::lock_request::start (this=0x7fe579f7b030) at /opt/vadim/ft-engine/scripts/build-tokudb-7.0.1/ft-index/locktree/lock_request.cc:218
#5  0x00007fe73dcb331e in toku_db_start_range_lock (db=0x7fe6f0851e80, txn=) at /opt/vadim/ft-engine/scripts/build-tokudb-7.0.1/ft-index/src/ydb_row_lock.cc:253

--
You received this message because you are subscribed to the Google Groups "tokudb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tokudb-user...@googlegroups.com.
To post to this group, send email to tokud...@googlegroups.com.
Visit this group at http://groups.google.com/group/tokudb-user?hl=en.
 
 

--
Mark Callaghan
mdca...@gmail.com

[Vadim Tkachenko]

No, it is not LinkBench.

It is just INSERT .. ON DUPLICATE KEY UPDATES statements running in parallel.

--
Vadim Tkachenko, CTO, Percona
Phone +1-925-400-7377, Skype: vadimtk153
Schedule meeting: http://meetme.so/VadimTkachenko

Looking for Replication with Data Consistency?
Try Percona XtraDB Cluster!

[Tim Callaghan]

Vadim,

I've been unable to reproduce the stall on our servers.  As we discussed, please let me known if you find a machine where you can reproduce.

-Tim