reader_concurrency_semaphore timeout

[Maxime FRYSOU]

Hi guys!

I'm performing a stress test on a chat application that uses Scylla.

After a few minutes, Scylla breaks and I can see this log message in the Scylla instance:

reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 1/100 count and 21778/30408704 memory resources: timed out, dumping permit diagnostics:
2023-11-14 14:54:59 permits   count  memory table/description/state
2023-11-14 14:54:59 1  1    17K   chat.room_member/data-query/active/used
2023-11-14 14:54:59 16 0    5K   chat.room_member/push-view-updates-2/active/unused
2023-11-14 14:54:59 47 0    0B   chat.room_member_uid_idx_index/data-query/waiting
2023-11-14 14:54:59 1  0    0B   chat.messages_by_room/data-query/waiting
2023-11-14 14:54:59 101 0    0B   chat.user/data-query/waiting
2023-11-14 14:54:59 247 0    0B   chat.room_member/data-query/waiting
2023-11-14 14:54:59
2023-11-14 14:54:59 413 1    21K   total
2023-11-14 14:54:59
2023-11-14 14:54:59 Total: 413 permits with 1 count and 21K memory resources

It looks like Scylla is running out of memory but in docker desktop I see 980MB of memory usage and it doesn't look like it's the case, plus I assigned 1.5G to the instance ...
I'm running Scylla as a Docker service like this:
scylla-node:
  image: scylladb/scylla:latest
  restart: always
  command: --seeds=scylla-node --smp 1 --memory 1500M --overprovisioned 1 --api-address 0.0.0.0
  volumes:
   - ./docker/scylla:/etc/scylla # provides cassandra-rackdc.properties and scylla.yaml through this binding
   - ./docker/scylla/chat_schema.cql:/chat_schema.cql # used to create the keyset
   - scylla-vol:/var/lib/scylla
  networks:
   - chat-net
  ports:
   - 9042:9042
   - 19042:19042

Thank you so much.

[Avi Kivity]

You gave no information about your hardware setup, configuration, what the load generator does and how the load generator is set up, so it's hard to say anything.

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/1bea800a-052e-4dab-aa80-1b1448204bdan%40googlegroups.com.

[Maxime FRYSOU]

Hi Avi,

I got a response in Parallel thanks to Botond Dénes and Felipe Cardeneti Mendes on the Slack channel:
From the following log:

reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 1/100 count and 21778/30408704 memory resources: timed out, dumping permit diagnostics:

We see:
- 21778/30408704 memory resources > so the memory is not saturated
- 1/100 count > so the disk is not the bottleneck

From there, we can deduce that that issue is the CPU.

So, I just set the -smp flag to 2 instead of 1 and it's working perfectly now.

Thank you again.