Re: Re: Questions about the lack of memory in scylla

[Glauber Costa]

On Wed, Jun 20, 2018, 9:18 PM zh...@49you.com, <zh...@49you.com> wrote:

Dear scylla:

    Scylla's method of setting memory limits: Modify /etc/sysconfig/scylla， add --memory “scylla memory ”--reserve-memory “system memory”，isn't it?

For example, my system has 32g of memory. I need to allocate 28g to Scylla and the remaining 4g to the operating system. add “--memory 28G --reserve-memory 4G”，isn't it?

You don't have to specify both. If you specify --memory 28G automatically the remainder will be reserved. (And vice versa)

I am moving your question to the users list.

---

朱锡江    运维工程师

 

公  司：广州四九游网络科技有限公司 

网  址：www.49you.cn

地  址：广州市天河区黄埔大道西平云街163号广电平云广场B塔8楼

手  机：13268008258   固话：020-87265661

企业Q：2355907775     邮箱：zh...@49you.com

                   页 游                                     手 游

 

From: Tzach Livyatan

Date: 2018-06-19 16:08

To: zh...@49you.com

CC: info

Subject: Re: Questions about the lack of memory in scylla

Hi zhuxj

Thanks for letting us know.

This is a question for scylla user list https://groups.google.com/d/forum/scylladb-users

or, as it looks like a bug, https://github.com/scylladb/scylla/issues

In either case, please include more info, in particular, the scylla version, deployment, loaders, schema etc[1]

Regards

Tzach

[1] http://docs.scylladb.com/operating-scylla/troubleshooting/report_scylla_problem/

On Tue, Jun 19, 2018 at 10:14 AM, zh...@49you.com <zh...@49you.com> wrote:

Dear SCYLLA:

        Hello, I have a problem that I can't handle. Can you help me?

        8-core 32-memory server, installed scylla, about 23G data volume, suddenly a certain time the memory usage is close to 100%, all clients can not even connect to the scylla to query modify delete, but the 9042 port is still listening, When the nodetool status is executed, the UP status is displayed. The logs of /var/log/messeges are:

        Jun 15 17:53:41 iZbp1c8xeqe9vojfvzynj6Z scylla: [shard 6] storage_proxy - exception during mutation write to 10.47.96.108: std::bad_alloc (std::bad_alloc)

        Jun 15 17:53:41 iZbp1c8xeqe9vojfvzynj6Z scylla: [shard 6] storage_proxy - Failed to apply mutation from 10.25.0.168#3: std::bad_alloc (std::bad_alloc)

        Jun 15 17:53:41 iZbp1c8xeqe9vojfvzynj6Z scylla: [shard 1] storage_proxy - Exception when communicating with 10.47.96.108: std::bad_alloc

        Jun 15 17:53:41 iZbp1c8xeqe9vojfvzynj6Z scylla: [shard 6] storage_proxy - Exception when communicating with 10.47.96.108: std::bad_alloc

        Jun 15 17:53:41 iZbp1c8xeqe9vojfvzynj6Z scylla: [shard 2] storage_proxy - Exception when communicating with 10.47.96.108: std::bad_alloc

        Jun 15 17:53:41 iZbp1c8xeqe9vojfvzynj6Z scylla: [shard 6] storage_proxy - Exception when communicating with 10.47.96.108: std::bad_alloc

The /etc/scylla/scylla.yaml configuration file is as follows:

num_tokens: 256

data_file_directories:

    - /var/lib/scylla/data

commitlog_directory: /var/lib/scylla/commitlog

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

      parameters:

          - seeds: "10.47.96.108"

listen_address: 10.47.96.108

native_transport_port: 9042

read_request_timeout_in_ms: 5000

write_request_timeout_in_ms: 2000

endpoint_snitch: GossipingPropertyFileSnitch

rpc_address: 10.47.96.108

rpc_port: 9160

api_port: 10000

api_address: 10.47.96.108

batch_size_warn_threshold_in_kb: 5

batch_size_fail_threshold_in_kb: 50

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

commitlog_total_space_in_mb: -1

murmur3_partitioner_ignore_msb_bits: 12

api_ui_dir: /usr/lib/scylla/swagger-ui/dist/

api_doc_dir: /usr/lib/scylla/api/api-doc/

experimental: true

[zhuxj@49you.com]

For example：

SCYLLA_ARGS="--log-to-syslog 1 --log-to-stdout 0 --default-log-level info --collectd-address=127.0.0.1:25826 --collectd=1 --collectd-poll-period 3000 --network-stack posix --memory 28G"

isn't it？

[Glauber Costa]

On Wed, Jun 20, 2018, 9:27 PM zh...@49you.com, <zh...@49you.com> wrote:

For example：

SCYLLA_ARGS="--log-to-syslog 1 --log-to-stdout 0 --default-log-level info --collectd-address=127.0.0.1:25826 --collectd=1 --collectd-poll-period 3000 --network-stack posix --memory 28G"

isn't it？

Yes.

From: Glauber Costa

Date: 2018-06-21 09:20

To: zhuxj

CC: Tzach Livyatan; ScyllaDB users

[zhuxj@49you.com]

Okay, thank you for your help.

[zhuxj@49you.com]

I heard that Scylla has 1 million concurrent reads and writes, but our company has only 100,000 concurrent tests. is it necessary to debug into dpdk mode?

 

From: zh...@49you.com

Date: 2018-06-21 09:33

To: Glauber Costa

CC: Tzach Livyatan; ScyllaDB users

Subject: Re: Re: Questions about the lack of memory in scylla

Okay, thank you for your help.

[Glauber Costa]

These numbers are heavily dependent on the workload, on the hardware, and other factors.

Also, what happens very often is that if you have a single client writing to it, your client hits a bottleneck before your server does. In that case if you add more clients you will see more throughput. 

Dpdk is usually not needed to get good throughput numbers.

On Wed, Jun 20, 2018, 11:07 PM zh...@49you.com, <zh...@49you.com> wrote:

I heard that Scylla has 1 million concurrent reads and writes, but our company has only 100,000 concurrent tests. is it necessary to debug into dpdk mode?

From: zh...@49you.com

Date: 2018-06-21 09:33

To: Glauber Costa

CC: Tzach Livyatan; ScyllaDB users

Subject: Re: Re: Questions about the lack of memory in scylla

Okay, thank you for your help.

[zhuxj@49you.com]

"nodetool info" shows "off memory" (3.7Gb), and "free -m" shows that the available memory is about 20GB, and the server memory is 32GB in total, so what is the use of the other 8GB memory?

[Avi Kivity]

They're in use by Scylla. Try running top and pressing M.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/5b2b470f.1c69fb81.8f520.1cd7SMTPIN_ADDED_BROKEN%40gmr-mx.google.com.
For more options, visit https://groups.google.com/d/optout.

[zhuxj@49you.com]

[zhuxj@49you.com]

The "nodetool repair" command fixes all the keyspaces, but it takes a long time. What parameter Settings allow it to fix only the last few days of keyspaces

[Asias He]

On Wed, Jun 27, 2018 at 4:26 PM, zh...@49you.com <zh...@49you.com> wrote:

The "nodetool repair" command fixes all the keyspaces, but it takes a long time. What parameter Settings allow it to fix only the last few days of keyspaces

$ nodetool repair <keyspace_name>

--

Asias

[Tzach Livyatan]

http://docs.scylladb.com/nodetool-commands/repair/

 

--

Asias

[zhuxj@49you.com]

Can you help me ?

发件人： zh...@49you.com

发送时间： 2018-06-27 16:26

收件人： Avi Kivity; ScyllaDB users; Glauber Costa

抄送： Tzach Livyatan

主题： Questions about the “nodetool repair” in scylla

[Glauber Costa]

Hi,

Both Tzach and Asias have responded to your request.

Is there anything still not clear ?

--

You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/5b343b33.1c69fb81.9903b.b5ffSMTPIN_ADDED_BROKEN%40gmr-mx.google.com.

[zhuxj@49you.com]

Dear All：

        My scylla cluster has two data centers, DC1 and DC2, and the external client will connect with all the scylla servers (10.26.251.27, 10.47.96.108, 10.47.106.136, 10.25.1068), which can only be related to the data center

DC1 (10.26.251.27, 10.47.96.108, 10.47.106.136) establishes a connection, does not establish a connection with the data center DC2 (10.25.0.168), and the data center DC2 is only used for backup data usage.

Can a scylla cluster allow only one data center to provide external services?

[zhuxj@49you.com]

can you help me ?

From: zh...@49you.com

Date: 2018-08-03 10:52

To: Avi Kivity; ScyllaDB users; Glauber Costa

CC: Tzach Livyatan

[Botond Dénes]

To my knowledge there isn't a way to limit the usage of a DC in such
way. If you want to make backups (and you should) there are dedicated
tools for that.

A side note: even if you manage to only connect to nodes in DC1 and
"hide" DC2 this would probably not result in what you wanted. If you
accidentally deleted data from DC1 it would be only a matter of time
until the deletion would be replicated to DC2 as well. Similarly for
accidental overwrites. That is not the definition of backup.

On 2018-08-03 09:59, zh...@49you.com wrote:
>
> can you help me ?
>

> *From:* zh...@49you.com <mailto:zh...@49you.com>
> *Date:* 2018-08-03 10:52
> *To:* Avi Kivity <mailto:a...@scylladb.com>; ScyllaDB users
> <mailto:scyllad...@googlegroups.com>; Glauber Costa
> <mailto:gla...@scylladb.com>
> *CC:* Tzach Livyatan <mailto:tz...@scylladb.com>
> *Subject:* Scylla cluster prohibits reading and writing across data

> centers
>
> Dear All：
> My scylla cluster has two data centers, DC1 and DC2, and
> the external client will connect with all the scylla
> servers (10.26.251.27, 10.47.96.108, 10.47.106.136,
> 10.25.1068), which can only be related to the data center
> DC1 (10.26.251.27, 10.47.96.108, 10.47.106.136)
> establishes a connection, does not establish a
> connection with the data center DC2 (10.25.0.168), and
> the data center DC2 is only used for backup data usage.
>
>
>

> Can a scylla cluster allow only one data center to
> provide external services?
>

> --
> You received this message because you are subscribed to the Google
> Groups "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to scylladb-user...@googlegroups.com

> <mailto:scylladb-user...@googlegroups.com>.

> To post to this group, send email to scyllad...@googlegroups.com

> <mailto:scyllad...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/scylladb-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/scylladb-users/5b63fd59.1c69fb81.cfec3.22ddSMTPIN_ADDED_BROKEN%40gmr-mx.google.com
> <https://groups.google.com/d/msgid/scylladb-users/5b63fd59.1c69fb81.cfec3.22ddSMTPIN_ADDED_BROKEN%40gmr-mx.google.com?utm_medium=email&utm_source=footer>.

[Tzach Livyatan]

Hello

You can limit your application(driver) to connect to only one DC. Once you do that, you can even block the external CQL connection of the second DC.

The DC will continue to sync with each other over the internal connection.

Does this answer your question?

Regards

Tzach

On Fri, Aug 3, 2018 at 10:02 AM, Botond Dénes <bde...@scylladb.com> wrote:

To my knowledge there isn't a way to limit the usage of a DC in such way. If you want to make backups (and you should) there are dedicated tools for that.

A side note: even if you manage to only connect to nodes in DC1 and "hide" DC2 this would probably not result in what you wanted. If you accidentally deleted data from DC1 it would be only a matter of time until the deletion would be replicated to DC2 as well. Similarly for accidental overwrites. That is not the definition of backup.

On 2018-08-03 09:59, zh...@49you.com wrote:

can you help me ?

    *From:* zh...@49you.com <mailto:zh...@49you.com>
    *Date:* 2018-08-03 10:52
    *To:* Avi Kivity <mailto:a...@scylladb.com>; ScyllaDB users

    <mailto:scylladb-users@googlegroups.com>; Glauber Costa

    <mailto:gla...@scylladb.com>
    *CC:* Tzach Livyatan <mailto:tz...@scylladb.com>
    *Subject:* Scylla cluster prohibits reading and writing across data
    centers

                Dear All：
                My scylla cluster has two data centers, DC1 and DC2, and
                the external client will connect with all the scylla
                servers (10.26.251.27, 10.47.96.108, 10.47.106.136,
                10.25.1068), which can only be related to the data center
                DC1 (10.26.251.27, 10.47.96.108, 10.47.106.136)
                establishes a connection, does not establish a
                connection with the data center DC2 (10.25.0.168), and
                the data center DC2 is only used for backup data usage.



                Can a scylla cluster allow only one data center to
                provide external services?

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com <mailto:scylladb-users+unsubscri...@googlegroups.com>.
To post to this group, send email to scyllad...@googlegroups.com <mailto:scylladb-users@googlegroups.com>.

[zhuxj@49you.com]

How can I restrict my php application to only one DC and not another DC? Do you have related help documents? I hope you can help me. It's very important to me. 

---Original---

From: "Tzach Livyatan"<tz...@scylladb.com>

Date: Sun, Aug 5, 2018 14:37 PM

To: "zh...@49you.com"<zh...@49you.com>;

Cc: "Avi Kivity"<a...@scylladb.com>;"Botond Dénes"<bde...@scylladb.com>;"ScyllaDB users"<scyllad...@googlegroups.com>;"Glauber Costa"<gla...@scylladb.com>;

Subject: Re: [scylladb-users] Re: Scylla cluster prohibits reading and writingacross data centers

[Tzach Livyatan]

I'm not a PHP driver expert, but until one join the discussion, here are a few examples of using the PHP driver with DC aware policy

Regards

Tzach

On Sun, Aug 5, 2018 at 10:15 AM, zh...@49you.com <zh...@49you.com> wrote:

How can I restrict my php application to only one DC and not another DC? Do you have related help documents? I hope you can help me. It's very important to me. 

---Original---

From: "Tzach Livyatan"<tz...@scylladb.com>

Date: Sun, Aug 5, 2018 14:37 PM

To: "zh...@49you.com"<zhuxj@49you.com>;

Cc: "Avi Kivity"<a...@scylladb.com>;"Botond Dénes"<bde...@scylladb.com>;"ScyllaDB users"<scylladb-users@googlegroups.com>;"Glauber Costa"<gla...@scylladb.com>;

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/tencent_3A0CC7910B450226154159D5%40qq.com.

[zhuxj@49you.com]

can you help me ?

 

发件人： zh...@49you.com

发送时间： 2018-08-05 15:15

收件人： Tzach Livyatan

抄送： Avi Kivity; Botond Dénes; ScyllaDB users; Glauber Costa

主题： Re: [scylladb-users] Re: Scylla cluster prohibits reading and writingacross data centers

[Tzach Livyatan]

Hello

Here are examples for using the PHP driver with DC aware policy 

The example with withDatacenterAwareRoundRobinLoadBalancingPolicy is what you are looking for.

Regards

Tzach

On Tue, Aug 7, 2018 at 8:15 AM, zh...@49you.com <zh...@49you.com> wrote:

can you help me ?

发件人： zh...@49you.com

发送时间： 2018-08-05 15:15

收件人： Tzach Livyatan

抄送： Avi Kivity; Botond Dénes; ScyllaDB users; Glauber Costa

主题： Re: [scylladb-users] Re: Scylla cluster prohibits reading and writingacross data centers

How can I restrict my php application to only one DC and not another DC? Do you have related help documents? I hope you can help me. It's very important to me. 

---Original---

From: "Tzach Livyatan"<tz...@scylladb.com>

Date: Sun, Aug 5, 2018 14:37 PM

To: "zh...@49you.com"<zhuxj@49you.com>;

Cc: "Avi Kivity"<a...@scylladb.com>;"Botond Dénes"<bde...@scylladb.com>;"ScyllaDB users"<scylladb-users@googlegroups.com>;"Glauber Costa"<gla...@scylladb.com>;

To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-users+unsubscribe@googlegroups.com.
To post to this group, send email to scylladb-users@googlegroups.com.

Visit this group at https://groups.google.com/group/scylladb-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/5b692af9.1c69fb81.673de.0ca9SMTPIN_ADDED_BROKEN%40gmr-mx.google.com.