scylladb insert 性能不稳定

[郑长发]

打算使用scylladb，现在在测试稳定性和性能阶段，昨晚一晚的测试结果显示，insert很不稳定，效率也上不去

自己没有cassandra的经验，所以调起来比较吃力，还望各位给予指导，不胜感激！！

4台服务器，一样的硬件配置，都是：

       3.5TB pcie ssd

       250G  内存

       32 核

每一台部署一个scylladb，总共四个节点

scylladb配置一样，如下：

cluster_name: 'Test Cluster'

num_tokens: 256

data_file_directories:

    - /data/scylla/data

commitlog_directory: /data/scylla/commitlog

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

      parameters:

          - seeds: "10.0.145.49,10.0.135.108,10.0.145.50,10.0.135.109"

listen_address: 10.0.145.49

native_transport_port: 9042

read_request_timeout_in_ms: 5000

write_request_timeout_in_ms: 2000

endpoint_snitch: SimpleSnitch

rpc_address: 10.0.145.49

rpc_port: 9160

api_port: 10000

api_address: 10.0.145.49

batch_size_warn_threshold_in_kb: 5

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

commitlog_total_space_in_mb: -1

developer_mode: true

murmur3_partitioner_ignore_msb_bits: 12

api_ui_dir: /usr/lib/scylla/swagger-ui/dist/

api_doc_dir: /usr/lib/scylla/api/api-doc/

测试用的是cassandra-stress，如下：

#配置文件，scylla_test.yaml

### DDL ###

###########

# Keyspace Name

keyspace: stresscql

# The CQL for creating a keyspace (optional if it already exists)

keyspace_definition: |

  CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3};

# Table name

table: blogposts

# The CQL for creating a table you wish to stress (optional if it already exists)

table_definition: |

  CREATE TABLE blogposts (

        domain text,

        body text,

        PRIMARY KEY(domain)

  )

### Column Distribution Specifications ###

##########################################

columnspec:

  - name: domain

    size: gaussian(5..20)       #domain names are relatively short

  - name: body

    size: gaussian(100..1000)    #the body of the blog post can be long

   

### Batch Ratio Distribution Specifications ###

###############################################

insert:                           #每个batch只插入一次（？）， 比如可以用insert=1，或者insert=2，...

  partitions: fixed(1)            #Our partition key is the domain so only insert one per batch

  select:    fixed(1)/1000        #We have 1000 posts per domain so 1/1000 will allow 1 post per batch

  batchtype: UNLOGGED             #Unlogged batches

### A list of queries you wish to run against the schema ###

############################################################

queries:

   singlepost:

      cql: select * from blogposts where domain = ? LIMIT 1 

      fields: samerow

#测试脚本 test_scylla.sh

#!/bin/bash

PROFILE="./scylla_test.yaml"

SEEDS="10.0.145.49,10.0.135.108,10.0.145.50,10.0.135.109"

TIME=`date +"%Y%m%d-%H%M%S"`

LOGSUFFIX="$TIME.log"

LINENUM=100000000

for ((var=300; var <= 1000; var=var+100))

do

    cassandra-stress user profile=$PROFILE n=$LINENUM ops\(insert=1\) cl=QUORUM -rate threads=$var -node $SEEDS -log file="w-thread"$var"_"$LOGSUFFIX

    cassandra-stress user profile=$PROFILE n=$LINENUM ops\(singlepost=1\) cl=QUORUM -rate threads=$var -node $SEEDS -log file="r-thread"$var"_"$LOGSUFFIX

done

[Amos Kong]

Hi Changfa,

Which version of scylla did you used?

Can you share the content of /etc/scylla.d/cpuset.conf  /etc/scylla.d/io.conf?

使用英文能得到更多的关注，当然如果你使用中文更加遍历，可以继续使用中文 :-)

On Tue, Jul 25, 2017 at 3:54 PM, 郑长发 <zhengc...@gmail.com> wrote:

打算使用scylladb，现在在测试稳定性和性能阶段，昨晚一晚的测试结果显示，insert很不稳定，效率也上不去

自己没有cassandra的经验，所以调起来比较吃力，还望各位给予指导，不胜感激！！

4台服务器，一样的硬件配置，都是：

       3.5TB pcie ssd

       250G  内存

       32 核

每一台部署一个scylladb，总共四个节点

scylladb配置一样，如下：

cluster_name: 'Test Cluster'

num_tokens: 256

data_file_directories:

    - /data/scylla/data

commitlog_directory: /data/scylla/commitlog

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

      parameters:

          - seeds: "10.0.145.49,10.0.135.108,10.0.145.50,10.0.135.109"

四台机器不在同一个子网？相互之间访问速度怎么样？

--

                        Amos.

[郑长发]

 /etc/scylladb/cpuset.conf:

     CPUSET="--cpuset 0-31 "

/etc/scylladb/io.comf:

    SEASTAR_IO="--max-io-requests=264"

四台机器间的网络都在同一机房，访问速度不是瓶颈

ping 10.0.135.108

PING 10.0.135.108 (10.0.135.108) 56(84) bytes of data.

64 bytes from 10.0.135.108: icmp_seq=1 ttl=62 time=0.060 ms

64 bytes from 10.0.135.108: icmp_seq=2 ttl=62 time=0.059 ms

64 bytes from 10.0.135.108: icmp_seq=3 ttl=62 time=0.054 ms

64 bytes from 10.0.135.108: icmp_seq=4 ttl=62 time=0.058 ms

64 bytes from 10.0.135.108: icmp_seq=5 ttl=62 time=0.053 ms

[Dor Laor]

Glauber, can you see the pattern inside of the excel graph? 

It looks similar to the ones you fixed with the 50% cap on compactions.

There may be an exception storm too.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/4394a270-7778-40a5-921f-58f435f77016%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Glauber Costa]

On Tue, Jul 25, 2017 at 4:55 PM, Dor Laor <d...@scylladb.com> wrote:
> Glauber, can you see the pattern inside of the excel graph?
> It looks similar to the ones you fixed with the 50% cap on compactions.
> There may be an exception storm too.
>

Why do you suspect an exception storm?

The patterns indeed look like compaction issues - and the fact that
the average latencies are another indication
that this is playing a role.

The rest of the issues look a lot more like bad interrupt routing than
anything else.

One way or another, checking the CPU usage will tell.
If high sysirq time in the Scylla CPUs: likely bad IRQ routing
If high system time in the Scylla CPUs: likely Linux scalability issues.

> On Tue, Jul 25, 2017 at 12:12 PM, 郑长发 <zhengc...@gmail.com> wrote:
>>
>> /etc/scylladb/cpuset.conf:
>> CPUSET="--cpuset 0-31 "
>>
>> /etc/scylladb/io.comf:
>> SEASTAR_IO="--max-io-requests=264"
>>
>> 四台机器间的网络都在同一机房，访问速度不是瓶颈
>> ping 10.0.135.108
>> PING 10.0.135.108 (10.0.135.108) 56(84) bytes of data.
>> 64 bytes from 10.0.135.108: icmp_seq=1 ttl=62 time=0.060 ms
>> 64 bytes from 10.0.135.108: icmp_seq=2 ttl=62 time=0.059 ms
>> 64 bytes from 10.0.135.108: icmp_seq=3 ttl=62 time=0.054 ms
>> 64 bytes from 10.0.135.108: icmp_seq=4 ttl=62 time=0.058 ms
>> 64 bytes from 10.0.135.108: icmp_seq=5 ttl=62 time=0.053 ms
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "ScyllaDB users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to scylladb-user...@googlegroups.com.
>> To post to this group, send email to scyllad...@googlegroups.com.

[Asias He]

Hello,

In order to understand what is going now.  Can you please:

1) Run /usr/sbin/scylla_setup if it was not run on all nodes

2) Paste the output of sudo /usr/lib/scylla/posix_net_conf.sh <iface_name>

3) Collect the `vmstat 1` and `iostat -x 1` results for a few minutes during the period the performance gets bad

On Tue, Jul 25, 2017 at 3:54 PM, 郑长发 <zhengc...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/c833e70e-51bd-4349-af7b-329a8619526e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Asias

[Asias He]

Ok，So, they found the issue. After changing the disk from HDD to SSD, they saw stable insert performance with small fluctuation, with ~240K TPS for a very long period of test. 

On Tue, Jul 25, 2017 at 3:54 PM, 郑长发 <zhengc...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/c833e70e-51bd-4349-af7b-329a8619526e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Asias

[郑长发]

OK，Thanks，I will use scylladb instead of cassandra, and maybe support redis agreement，and i want to know Specific performance about map ,but cassandra-stress don't support map type; I don't know how to do,  and I have doubts about the range query， how  scylladb support that?

To post to this group, send email to scyllad...@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/c833e70e-51bd-4349-af7b-329a8619526e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Asias

--
You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/gGDtBLKvdeU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-users+unsubscribe@googlegroups.com.

To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAO1GqFbtrpHpL3sGzds%3DpK6-Zq3e9LbPW6Gm2r6Um3c_g2N-aw%40mail.gmail.com.

[郑长发]

OK，Thanks，I will use scylladb instead of cassandra, and maybe support redis agreement，and i want to know Specific performance about map ,but cassandra-stress don't support map type; I don't know how to do,  and I have doubts about the range query， how  scylladb support that?

2017-07-31 15:32 GMT+08:00 Asias He <as...@scylladb.com>:

To post to this group, send email to scyllad...@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/c833e70e-51bd-4349-af7b-329a8619526e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Asias

--
You received this message because you are subscribed to a topic in the Google Groups "ScyllaDB users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scylladb-users/gGDtBLKvdeU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scylladb-users+unsubscribe@googlegroups.com.

To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAO1GqFbtrpHpL3sGzds%3DpK6-Zq3e9LbPW6Gm2r6Um3c_g2N-aw%40mail.gmail.com.

[Tzach Livyatan]

Indeed Map type does not support range query, but clustering key does support it in an efficient way.

If you can share your schema, or pseudo schema, with the map, we can advise about converting it to clustering key component, is possible

Regards

Tzach

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAO1GqFbtrpHpL3sGzds%3DpK6-Zq3e9LbPW6Gm2r6Um3c_g2N-aw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.

To post to this group, send email to scylladb-users@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/CAOck73pC2KHnZAO7f5zP5kR%3DP6CcGZPatO8F_J5Yy5vOtEa31w%40mail.gmail.com.