reader_concurrency_semaphore messages in log

[Michael Wohlwend]

Hi,

I get those messages in the log (scylla 4.6.3):


[shard 1] reader_concurrency_semaphore - (rate limiting dropped 8 similar
messages) Semaphore _read_concurrency_sem with 100/100 count and
2830368/153385697 memory resources: timed out, dumping permit diagnostics:
permits count memory table/description/state
100 100 2764K table-1/data-query/active/blocked
8 0 0B table-2/data-query/waiting

Total: 701 permits with 100 count and 2764K memory resources


There are more of them. Is this ok?

thanks
Michael

[bdenes@scylladb.com]

Looks like your node is overloaded and can't keep up with the reads,
which seem to all go to disk.

[Michael Wohlwend]

Am Donnerstag, 5. Mai 2022, 10:34:42 CEST schrieb bde...@scylladb.com:
> Looks like your node is overloaded and can't keep up with the reads,
> which seem to all go to disk.
>

load is currently near zero.
But the cluster was completly upgraded, nodes were decommissioned, nodes were
added and 4 of 8 have replaced old nodes. This was the last node which
replaced an old one. It all worked quite well, but I thought, maybe some
things are not in a "good" state, so a restarted scylla on this node. It did a
long reshape (why? it got the data from the other nodes...) and then worked
normal.
Maybe that replacing overloaded something...
After that I restarted the other nodes, some of them did some reshaping other
not.
Now all nodes work as expected.

Michael

[bdenes@scylladb.com]

Reshape means the sstables are not laid out in a way the compaction
strategy expects it. We now detect this and trigger a reshape which
recompacts the data into a more optimal form. This doesn't mean there
is anything wrong with the data, it just means the files are not
organized in an optimal manner. This can cause reads for example to not
be as fast as they would otherwise be.

>
>  Michael
>
>
>
>

[Michael Wohlwend]

Am Donnerstag, 5. Mai 2022, 13:22:39 CEST schrieb bde...@scylladb.com:

> Reshape means the sstables are not laid out in a way the compaction
> strategy expects it. We now detect this and trigger a reshape which
> recompacts the data into a more optimal form. This doesn't mean there
> is anything wrong with the data, it just means the files are not
> organized in an optimal manner. This can cause reads for example to not
> be as fast as they would otherwise be.
>

ah, ok, thanks for the explanation

Michael

[Andrey V. Bondarenko]

And when the CPU is overloaded, can this be?

I have the same records in the log but the disk not busy, all the data in cache. But several cores is 100% busy.

Can you tell how to understand this:

scylla[12345]:  [shard 1] reader_concurrency_semaphore - (rate limiting dropped 3277 similar messages) Semaphore _read_concurrency_sem with 1/100 count and 27879/148520304 memory resources: timed out, dumping permit diagnostics:
                                                           permits        count        memory        table/description/state
                                                           1        1        27K        keyspace7.values/data-query/active/used
                                                           77        0        0B        keyspace1.values/data-query/waiting
                                                           13        0        0B        keyspace2.values/data-query/waiting
                                                           48        0        0B        keyspace3.table1/mutation-query/waiting
                                                           1        0        0B        keyspace3.table2/data-query/waiting
                                                           30        0        0B        keyspace3.table3/data-query/waiting
                                                           147        0        0B        keyspace3.table4/data-query/waiting
                                                           34        0        0B        keyspace4.values/data-query/waiting
                                                           7        0        0B        keyspace7.table1/data-query/waiting
                                                           59        0        0B        keyspace7.values/mutation-query/waiting
                                                           19        0        0B        keyspace3.table3/mutation-query/waiting
                                                           100        0        0B        keyspace3.table1/data-query/waiting
                                                           12        0        0B        keyspace5.table1/data-query/waiting
                                                           213        0        0B        keyspace6.values/data-query/waiting
                                                           101        0        0B        keyspace8.values/data-query/waiting
                                                           452        0        0B        keyspace5.values/data-query/waiting
                                                           283        0        0B        keyspace7.values/data-query/waiting
                                                           6        0        0B        keyspace1.table1/data-query/waiting
                                                           79        0        0B        keyspace3.table4/mutation-query/waiting
                                                           61        0        0B        keyspace5.values/mutation-query/waiting
 
                                                           1743        1        27K        total
 
                                                           Total: 1743 permits with 1 count and 27K memory resources

четверг, 5 мая 2022 г. в 11:34:46 UTC+3, Botond Dénes:

[bdenes@scylladb.com]

This report is caused by CPU bottleneck. The semaphore tries to keep just one read active at a time, so long as they only consume CPU. In practical terms this means until they read only from cache. If your CPU becomes a bottleneck, you will see reports like this, with just one active read (1 count resource) and lots of reads waiting.

Note that you have many read repairs (mutation-query), consider running a repair to get rid of these.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/9dcae340-169c-421c-89db-e14f22e74f5cn%40googlegroups.com.