scylladb performance tuning

[xiagujinqiao@gmail.com]

hi，I'm testing the performance of scylladb，but the result is slow.

3 node cluster,ip list: 172.17.0.3, 172.17.0.11, 172.17.0.12

#scylladb.yaml 

cluster_name: 'Test Cluster'

num_tokens: 256

data_file_directories:

    - /var/lib/scylla/data

commitlog_directory: /var/lib/scylla/commitlog

commitlog_sync: periodic

commitlog_sync_period_in_ms: 20000

commitlog_segment_size_in_mb: 64

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

      parameters:

          - seeds: "172.17.0.3,172.17.0.12"

listen_address: "172.17.0.11"

native_transport_port: 9042

read_request_timeout_in_ms: 5000

write_request_timeout_in_ms: 2000

endpoint_snitch: GossipingPropertyFileSnitch

rpc_address: "172.17.0.11"

rpc_port: 9160

api_port: 10000

api_address: 172.17.0.11 

batch_size_warn_threshold_in_kb: 5

batch_size_fail_threshold_in_kb: 50

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

commitlog_total_space_in_mb: -1

internode_compression: none

murmur3_partitioner_ignore_msb_bits: 12

api_ui_dir: /usr/lib/scylla/swagger-ui/dist/

api_doc_dir: /usr/lib/scylla/api/api-doc/

# cat cpuset.conf

CPUSET="--cpuset 2-31 "

# cat io.conf

SEASTAR_IO="--max-io-requests=612"

hardware:

32core Intel(R) Xeon(R) Gold 61xx CPU  2500Mhz, 128G mem,  2 X 3TB nvme ssd(raid0), 10Gbps network

2 node for test,ip list: 172.17.0.10, 172.17.30.252, 32core,96G mem

#mytest conf:

#egrep -v '^$|^#' wrtest1.yaml 

keyspace: testname

keyspace_definition: |

  CREATE KEYSPACE testname WITH replication = {'class': 'SimpleStrategy','replication_factor': '2'}  AND durable_writes = true;

table: device

table_definition: |

  CREATE TABLE device (

        tdid text,

        datatime int,

        features text,

        PRIMARY KEY (tdid, datatime)

  ) WITH CLUSTERING ORDER BY (datatime DESC)

  AND bloom_filter_fp_chance = 0.01

  AND caching = {'keys': 'ALL','rows_per_partition': 'ALL'}

  AND comment = ''

  AND compaction = {'class': 'SizeTieredCompactionStrategy'}

  AND compression = {'sstable_compression': ''}

  AND dclocal_read_repair_chance = 0.1

  AND default_time_to_live = 604800

  AND gc_grace_seconds = 0

  AND max_index_interval = 2048

  AND memtable_flush_period_in_ms = 0

  AND min_index_interval = 128

  AND speculative_retry = '99.0PERCENTILE';

columnspec:

  - name: tdid

    size: fixed(120)

    population: gaussian(1..30000000000,15000000000,6000000000)

  - name: datatime

    size: fixed(120)

    cluster: fixed(1)

  - name: features

    size: fixed(120)

insert:

  partitions: fixed(1)

  batchtype: UNLOGGED

  select: fixed(1)/1

queries:

  read1:

    cql: select tdid,datatime,features from device where tdid = ? limit 1

    fields: samerow

#cassandra-stress user profile=/home/testname/wrtest1.yaml 'ops(insert=1,read1=4)' duration=100h no-warmup -mode native cql3 -rate threads=256 -node 172.17.0.11,172.17.0.3,172.17.0.12 -pop dist=gaussian\(1..30000000000,15000000000,6000000000\) 

# cat /etc/redhat-release 

CentOS Linux release 7.5.1804 (Core) 

# uname  -a

Linux VM_0_11_centos 4.18.5-1.el7.elrepo.x86_64 #1 SMP Fri Aug 24 11:35:05 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

but result is too slow

how performance tuning?

[Dor Laor]

The CPU is at ~50%-60%, how many clients do you run? Try to increase their amount

--
You received this message because you are subscribed to the Google Groups "ScyllaDB development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-dev...@googlegroups.com.
To post to this group, send email to scylla...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-dev/939991a8-3821-436c-b223-16791f98b7ef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[xiagujinqiao@gmail.com]

2 nodes, 256 threads per node

cassandra-stress user profile=/home/testname/wrtest1.yaml 'ops(insert=10,read1=40)' duration=100h no-warmup -mode native cql3 -rate threads=256 -node 172.17.0.11,172.17.0.3,172.17.0.12 -pop dist=gaussian\(1..30000000000,15000000000,6000000000\)

If I increase the thread, the cpu load will rise to 100%, and the scylladb read/write delay will suddenly increase.

在 2018年9月17日星期一 UTC+8下午1:10:00，Dor Laor写道：

[Dor Laor]

On Sun, Sep 16, 2018 at 10:33 PM <xiaguj...@gmail.com> wrote:

2 nodes, 256 threads per node

It's not a lot at all

 

cassandra-stress user profile=/home/testname/wrtest1.yaml 'ops(insert=10,read1=40)' duration=100h no-warmup -mode native cql3 -rate threads=256 -node 172.17.0.11,172.17.0.3,172.17.0.12 -pop dist=gaussian\(1..30000000000,15000000000,6000000000\)

If I increase the thread, the cpu load will rise to 100%, and the scylladb read/write delay will suddenly increase.

cpu load on the server? What do you mean by read/write delay?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-dev/83533026-f23e-44c9-9517-6673c1a6c9dc%40googlegroups.com.

[xiagujinqiao@gmail.com]

exampel ： I increase the thread ，512 threads per node，write/read latency increase.
thanks @Dor Laor

在 2018年9月17日星期一 UTC+8下午1:42:17，Dor Laor写道：

[Dor Laor]

One of the servers (the one in green color) suffers from an extremely bad read latencies. 

Is it absolutely the same as the others? Worth to see how all of the cores function, whether

there is one busy core on it (there is a core view and a 'top' view too)

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-dev/178a5707-d64a-4ac4-b6e8-296c3e83f1a2%40googlegroups.com.

[xiagujinqiao@gmail.com]

the green color server extremely bad read latencies；

the 3 node is running on kvm；

but scylladb running 32core 128G mem，2 X 3T nvme SSD，best practices , What is the approximate read and write performance? or does my conf need to be tuned?

在 2018年9月17日星期一 UTC+8下午2:26:45，Dor Laor写道：

[Dor Laor]

On Sun, Sep 16, 2018 at 11:54 PM <xiaguj...@gmail.com> wrote:

the green color server extremely bad read latencies；

the 3 node is running on kvm；

but scylladb running 32core 128G mem，2 X 3T nvme SSD，best practices , What is the approximate read and write performance? or does my conf need to be tuned?

About the approximated perf, it's hard to say, there are so many potential things that affect performance (like the performance of your drive).

It's not 32 core machine but 32 hyper threads. Is each VM is the single workload on the host? Need to make sure

that the VM are pinned to the same physical cores on the host for optimal performance and no hyper thread is dominated by

interrupts (could be the case of your green server)

 

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-dev/755be313-0624-4ab8-9383-fc34424303bf%40googlegroups.com.