std::bad_alloc

[jkne81@gmail.com]

scylladb version 1.7

scylladb.yaml

cluster_name: 'Mapping Cluster'

num_tokens: 256

data_file_directories:

    - /data1/scylla/data

commitlog_directory: /data1/scylla/commitlog

commitlog_sync: periodic

commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 16

seed_provider:

    - class_name: org.apache.cassandra.locator.SimpleSeedProvider

      parameters:

          - seeds: "ip bla bla bla"

listen_address: this node ip

native_transport_port: 9042

read_request_timeout_in_ms: 20000

write_request_timeout_in_ms: 20000

endpoint_snitch: SimpleSnitch

rpc_address:  this node ip

rpc_port: 9160

api_port: 10000

api_address: 127.0.0.1

batch_size_warn_threshold_in_kb: 5

batch_size_fail_threshold_in_kb: 50

partitioner: org.apache.cassandra.dht.Murmur3Partitioner

concurrent_reads: 64

concurrent_writes: 64

concurrent_counter_writes: 64

commitlog_total_space_in_mb: -1

prometheus_address:  this node ip

murmur3_partitioner_ignore_msb_bits: 12

api_ui_dir: /usr/lib/scylla/swagger-ui/dist/

api_doc_dir: /usr/lib/scylla/api/api-doc/

RF=1 CL=ONE

/var/log/syslog:

May 25 13:44:16 scylladb16 scylla[24543]: ERROR 2018-05-25 13:44:16,045 [shard 4] storage_proxy - exception during mutation write to 10.144.115.36: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: ERROR 2018-05-25 13:44:16,059 [shard 11] storage_proxy - Exception when communicating with 10.144.115.36: std::bad_alloc

May 25 13:44:16 scylladb16 scylla[24543]: ERROR 2018-05-25 13:44:16,123 [shard 7] storage_proxy - Exception when communicating with 10.144.115.36: std::bad_alloc

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,127 [shard 0] storage_proxy - Failed to apply mutation from 10.30.137.254#5: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,181 [shard 5] storage_proxy - Failed to apply mutation from 10.30.137.254#8: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,211 [shard 0] storage_proxy - Failed to apply mutation from 10.28.205.85#6: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,286 [shard 3] storage_proxy - Failed to apply mutation from 10.30.249.92#10: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,292 [shard 11] storage_proxy - Failed to apply mutation from 10.144.115.144#7: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: ERROR 2018-05-25 13:44:16,295 [shard 6] storage_proxy - Exception when communicating with 10.144.115.36: std::bad_alloc

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,301 [shard 8] storage_proxy - Failed to apply mutation from 10.144.115.188#4: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: WARN  2018-05-25 13:44:16,304 [shard 7] storage_proxy - Failed to apply mutation from 10.144.113.242#6: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: ERROR 2018-05-25 13:44:16,361 [shard 1] compaction_manager - compaction failed: std::bad_alloc (std::bad_alloc)

May 25 13:44:16 scylladb16 scylla[24543]: INFO  2018-05-25 13:44:16,361 [shard 1] compaction_manager - compaction task handler sleeping for 20 seconds

May 25 13:44:36 scylladb16 scylla[24543]: INFO  2018-05-25 13:44:36,362 [shard 1] compaction - Compacting [/data1/scylla/data/mapping/miaozhen-de8c97c0356e11e78fa400000000000d/mapping-miaozhen-ka-48665-Data.db:level=0, /data1/scylla/data/mapping/miaozhen-de8c97c0356e11e78fa400000000000d/mapping-miaozhen-ka-60509-Data.db:level=0, /data1/scylla/data/mapping/miaozhen-de8c97c0356e11e78fa400000000000d/mapping-miaozhen-ka-85569-Data.db:level=0, /data1/scylla/data/mapping/miaozhen-de8c97c0356e11e78fa400000000000d/mapping-miaozhen-ka-121283-Data.db:level=0, ]

May 25 13:45:01 scylladb16 CRON[25927]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)

May 25 13:45:01 scylladb16 CRON[25928]: (root) CMD (/root/cron/scylladb_monitor.sh > /dev/null 2>&1)

HELP!!! SOS

[Glauber Costa]

Those errors indicate that you have memory exhaustion. This could either be because you have no more memory or because memory is severely fragmented, so that allocations can't be served.

Also please note: scylladb 1.7 is no longer supported. 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To post to this group, send email to scyllad...@googlegroups.com.
Visit this group at https://groups.google.com/group/scylladb-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/3a802ac3-9e32-41e1-af7e-24e1ed0c45f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jkne81@gmail.com]

how to fix memory fragmented?

[Glauber Costa]

Depends on the root cause.

Usually a node restart can help.

To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/1da9fce9-8a87-48c7-ab7e-81e9baa325e1%40googlegroups.com.

[searchpcc@gmail.com]

Hello，

Now I set memory 29G，But the memory is shown used 15G. The error log show：ERROR 2018-05-30 08:48:06,449 [shard 2] storage_proxy - Exception when communicating with xx.xx.xx.xx: std::bad_alloc

Q: How can I reduce memory use?
I try to mv db file, but It's not useful. 

在 2018年5月25日星期五 UTC+8下午9:41:18，Glauber Costa写道：