Performance issues on Docker and Gentoo

[michael.weinig@gmail.com]

Hi,

I'm trying to migrate from cassandra to scyllaDB on:

2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz with 12 logical cores each

RAM: 256G

Disk: 2x 4TB RAID-1, no SSD unfortunatly

FS: XFS

OS: 4.1.15-gentoo-r1

I pulled the scylladb/scylla docker and ran it according to http://www.scylladb.com/doc/getting-started/ for production mode.

The DB works so far, but is significantly slower than the cassandra 2.1.8, I used to run on the Host system (about 5 times on avg.)

Here is how I started ScyllaDb in the Docker (at first in developer-mode):

/usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --developer-mode true --default-log-level info --options-file $HOME/scylla.yaml --listen-address $IP --rpc-address $IP --network-stack posix > ~/scylla.log &

And the log:

EAL: Detected lcore 0 as core 0 on socket 0

EAL: Detected lcore 1 as core 1 on socket 0

EAL: Detected lcore 2 as core 2 on socket 0

EAL: Detected lcore 3 as core 3 on socket 0

EAL: Detected lcore 4 as core 4 on socket 0

EAL: Detected lcore 5 as core 5 on socket 0

EAL: Detected lcore 6 as core 0 on socket 0

EAL: Detected lcore 7 as core 1 on socket 0

EAL: Detected lcore 8 as core 2 on socket 0

EAL: Detected lcore 9 as core 3 on socket 0

EAL: Detected lcore 10 as core 4 on socket 0

EAL: Detected lcore 11 as core 5 on socket 0

EAL: Detected lcore 12 as core 0 on socket 0

EAL: Detected lcore 13 as core 1 on socket 0

EAL: Detected lcore 14 as core 2 on socket 0

EAL: Detected lcore 15 as core 3 on socket 0

EAL: Detected lcore 16 as core 4 on socket 0

EAL: Detected lcore 17 as core 5 on socket 0

EAL: Detected lcore 18 as core 0 on socket 0

EAL: Detected lcore 19 as core 1 on socket 0

EAL: Detected lcore 20 as core 2 on socket 0

EAL: Detected lcore 21 as core 3 on socket 0

EAL: Detected lcore 22 as core 4 on socket 0

EAL: Detected lcore 23 as core 5 on socket 0

EAL: Support maximum 128 logical core(s) by configuration.

EAL: Detected 24 lcore(s)

EAL: VFIO modules not all loaded, skip VFIO support...

EAL: Setting up physically contiguous memory...

EAL: Cannot open /proc/self/pagemap: Operation not permitted. virt2phys address translation will not work

EAL: TSC frequency is ~2000001 KHz

EAL: Master lcore 0 is ready (tid=930d2880;cpuset=[0])

EAL: lcore 17 is ready (tid=7e6cf700;cpuset=[17])

EAL: lcore 23 is ready (tid=7b6c9700;cpuset=[23])

EAL: lcore 7 is ready (tid=836d9700;cpuset=[7])

EAL: lcore 13 is ready (tid=806d3700;cpuset=[13])

EAL: lcore 15 is ready (tid=7f6d1700;cpuset=[15])

EAL: lcore 2 is ready (tid=85ede700;cpuset=[2])

EAL: lcore 5 is ready (tid=846db700;cpuset=[5])

EAL: lcore 8 is ready (tid=82ed8700;cpuset=[8])

EAL: lcore 9 is ready (tid=826d7700;cpuset=[9])

EAL: lcore 14 is ready (tid=7fed2700;cpuset=[14])

EAL: lcore 20 is ready (tid=7cecc700;cpuset=[20])

EAL: lcore 10 is ready (tid=81ed6700;cpuset=[10])

EAL: lcore 16 is ready (tid=7eed0700;cpuset=[16])

EAL: lcore 18 is ready (tid=7dece700;cpuset=[18])

EAL: lcore 12 is ready (tid=80ed4700;cpuset=[12])

EAL: lcore 6 is ready (tid=83eda700;cpuset=[6])

EAL: lcore 1 is ready (tid=866df700;cpuset=[1])

EAL: lcore 11 is ready (tid=816d5700;cpuset=[11])

EAL: lcore 4 is ready (tid=84edc700;cpuset=[4])

EAL: lcore 22 is ready (tid=7beca700;cpuset=[22])

EAL: lcore 21 is ready (tid=7c6cb700;cpuset=[21])

EAL: lcore 3 is ready (tid=856dd700;cpuset=[3])

EAL: lcore 19 is ready (tid=7d6cd700;cpuset=[19])

EAL: PCI device 0000:03:00.0 on NUMA socket 0

EAL:   probe driver: 8086:1528 rte_ixgbe_pmd

EAL:   Not managed by a supported kernel driver, skipped

EAL: PCI device 0000:03:00.1 on NUMA socket 0

EAL:   probe driver: 8086:1528 rte_ixgbe_pmd

EAL:   Not managed by a supported kernel driver, skipped

Scylla version 1.2.0-20160614.c384b23 starting ...

Scylla API server listening on 127.0.0.1:10000 ...

Write stress test:

bash-4.2$ cassandra-stress write -node $IP -mode cql3 native
INFO  15:51:14 Found Netty's native epoll transport in the classpath, using it
INFO  15:51:15 Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
INFO  15:51:15 New Cassandra host /172.17.0.2:9042 added
Connected to cluster: Test Cluster
Datatacenter: datacenter1; Host: /172.17.0.2; Rack: rack1
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Warming up WRITE with 50000 iterations...
Failed to connect over JMX; not collecting these stats
WARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level analysis only
Running with 4 threadCount
Running WRITE with 4 threads until stderr of mean < 0.02
Failed to connect over JMX; not collecting these stats

...

Results:

op rate                   : 3266 [WRITE:3266]

partition rate            : 3266 [WRITE:3266]

row rate                  : 3266 [WRITE:3266]

...
8 threadCount

Results:

op rate                   : 6851 [WRITE:6851]

partition rate            : 6851 [WRITE:6851]

row rate                  : 6851 [WRITE:6851]

...
16 threadCount

Results:

op rate                   : 13114 [WRITE:13114]

partition rate            : 13114 [WRITE:13114]

row rate                  : 13114 [WRITE:13114]

I'm sure not expecting > 1M ops on this machine, but this is significantly slower than cassandra was...

Then I tried to run scylla_io_setup:

bash-4.2$ scylla_io_setup

Generating evaluation file sized 10GB...10GB written in 82 seconds

Refining search for maximum. So far, 1132 IOPS

Maximum throughput: 1132 IOPS

Recommended --max-io-requests: 237

basic_ios::clear when writing configuration file. Please add them to your seastar command line

/var/lib/scylla did not pass validation tests, it may not be on XFS and/or has limited disk space.

This is a non-supported setup, and performance is expected to be very bad.

For better performance, placing your data on XFS-formatted directories is required.

 To override this error, see the developer_mode configuration option.

Although, the FS is XFS: 

bash-4.2$ df -T ~

Filesystem     Type  1K-blocks       Used Available Use% Mounted on

/dev/sdb3      xfs  3840342380 3295945380 544397000  86% /var/lib/scylla

starting scylla with --max-io-requests 237 instead of --developer-mode true did not improve the performance...

Any help with this issue is highly appreciated!

Thanks,

Michael

[Glauber Costa]

Is this an SSD? 

1132 4k IOPS = ~ 4 MB/s. My guess is that this is running in docker's unionFS filesystem 

 

Any help with this issue is highly appreciated!

Thanks,

Michael

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/62f1570d-b235-44a7-806d-f8a608116f98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Avi Kivity]

On 07/26/2016 07:53 PM, michael...@gmail.com wrote:

Hi,

I'm trying to migrate from cassandra to scyllaDB on:

2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz with 12 logical cores each

RAM: 256G

Disk: 2x 4TB RAID-1, no SSD unfortunatly

That's one issue.  We don't really test on HDDs (though they should give you a lot more than you got).

Are you running stress from a different host, or the same host?

Then I tried to run scylla_io_setup:

bash-4.2$ scylla_io_setup

Generating evaluation file sized 10GB...10GB written in 82 seconds

Refining search for maximum. So far, 1132 IOPS

Maximum throughput: 1132 IOPS

Recommended --max-io-requests: 237

basic_ios::clear when writing configuration file. Please add them to your seastar command line

/var/lib/scylla did not pass validation tests, it may not be on XFS and/or has limited disk space.

This is a non-supported setup, and performance is expected to be very bad.

For better performance, placing your data on XFS-formatted directories is required.

 To override this error, see the developer_mode configuration option.

Although, the FS is XFS: 

bash-4.2$ df -T ~

Filesystem     Type  1K-blocks       Used Available Use% Mounted on

/dev/sdb3      xfs  3840342380 3295945380 544397000  86% /var/lib/scylla

starting scylla with --max-io-requests 237 instead of --developer-mode true did not improve the performance...

scylla_io_setup is broken on spinning disks.  Parameters suitable for your configuration are --max-io-requests 8 --num-io-queues 1.

I hope you used -V in the docker command line like the documentation recommends, or docker would kill your performance.

Any help with this issue is highly appreciated!

Thanks,

Michael

[Glauber Costa]

Avi,

There is a sysfs property that tell us if a particular disk is a spinning media.

Shouldn't we try to detect that and just spit the right configuration in that case?

 

I hope you used -V in the docker command line like the documentation recommends, or docker would kill your performance.

Any help with this issue is highly appreciated!

Thanks,

Michael

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/62f1570d-b235-44a7-806d-f8a608116f98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/8a5447ee-405e-b7cd-3489-973745e6c6fa%40scylladb.com.

[Avi Kivity]

We could (4 * spindles).  There is a problem with hardware RAID or dm-cache backed devices, though, where that heuristic doesn't work.

I guess we could use libblkid to walk down the topology tree, if we end up with an SSD, bench it, if we end up with rotational disks or software raid of rotational disks, apply the heuristic, and if we end up point to a web page explaining what to do (run the tool manually and examine the results carefully, comparing them with what's in the box).

[Avi Kivity]

On 07/26/2016 07:53 PM, michael...@gmail.com wrote:

Hi,

I'm trying to migrate from cassandra to scyllaDB on:

2x Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz with 12 logical cores each

RAM: 256G

Disk: 2x 4TB RAID-1, no SSD unfortunatly

btw, I remember severe write performance issues on RAID-1 due to interaction between AIO and RAID.  I recommend switching to RAID-0.

[michael.weinig@gmail.com]

I used --max-io-requests 8 --num-io-queues 1 and tried it again, ops round about doubled, but it's still slower than cassandra.

 I'm running cassandra-stress on the same machine, and used a HDD without RAID now.

Do you think the problem is related to using docker and Gentoo instead of one of the recommended OS? Or is it not going to work on spinning disks?

We use cassandra for KairosDB time series DB. It's about 3TB data. The write requests all fall into the latest rows, as do the majority of the read requests, I would estimate the hot dataset to be around 100GB. Since it's not recommended to run cassandra with more than 8GB heap, I hoped to be able to keep the whole hot dataset in memory... 

[Avi Kivity]

I think spinning disks can work (nowhere near as fast as SSDs, but at least as fast as Cassandra), but the RAID-1 may interact badly with AIO.

You can test it by building fsqual [1] and running it with your RAID as the home directory.  The first line of the output is the only one that matters.

[1] https://github.com/avikivity/fsqual

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/831deec4-6af8-4017-a19f-4325f2feb452%40googlegroups.com.