Non-deterministic number of Memcached children processes other than worker threads

[Alireza Sanaee]

Hello,

I'm running Memcached on two different machines with different specifications. And I specify the number of worker threads = 4 for both machines. However, the number of child processes of the Memcached server is different on two machines. On one of them, I have 6 Memcached child processes, and on the other server, I have 10 Memcached child processes. I'm curious to understand how many children processes Memcached is basically spawning other than the worker threads, and for what tasks?

I expect the Memcached to spawn only 4 children processes or a certain number of children processes on two machines, however, it seems not true.

Thanks,

Alireza

[dormando]

What version of memcached is on each machine?

memcached doesn't use processes, it's multi-threaded. Different versions
may have a different number of background threads. In the latest version
there should be at least:

- listener thread (main "process")
- N worker threads
- hash table maintenance thread
- async log thread (for `watch` commands)
- LRU maintainer thread
- LRU crawler thread
- slab rebalancer thread

they're all idle unless they need to do work. LRU maintenance thread is
probably the most active, since it executes LRU maintenance work deferred
from the worker threads. Older versions have some of these threads, but
they were not enabled by default until 1.5.0.

-Dormando

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/da7f492f-c12d-4763-86fc-d07311d21c5a%40googlegroups.com.
>
>

[Alireza Sanaee]

Hi,

Thank you for the information,

Sorry for miss using the word there, yes that's all threads. I'm using the Memcached 1.5.20. I build it myself and then run my experiments($MEMCACHED -u root -p 11211 -m $MAXMEM -c 1024 -t $MEMCACHED_THREADS). And I'm checking the number of Memcached threads in htop output. It showed me 10 threads(workers included) in one machine and 6 threads(workers included) on the other one.

To share some more information, I have 200GB of memory for the bigger machine that creates only 6 threads, and we have only 16GB of memory for the machine that creates 10 threads. I'm just thinking maybe because the smaller machine has less amount of space, and I'm actually filling in up to 15GB then I might have more work to do and creates more threads.

According to your information, I should expect at least 5 threads other than the main workers. So 10 threads look OK, but how about the bigger machine which spawns only 6 threads?  

I also had difficulties in detecting the worker threads that respond to GET/SET requests on my results, I have attached two pictures, one of them shows the actual location of each worker on various cores, and the second one is showing userspace time spent for each worker. Apparently worker thread number 1,2,4 and 5 have spent more time in userspace, so I'm concluding here that 1,2,4 and 5 are my actual worker threads, and worker 3 and 6 are just internal worker threads of Memcached. Does that make sense to you?

Thanks,

Alireza

You received this message because you are subscribed to a topic in the Google Groups "memcached" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/memcached/L05nSQruHRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to memcached+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912141516160.3156%40dskull.

[dormando]

What're you trying to accomplish?

Can you include the output of "stats" and "stats settings" on both
machines?

Dumb question but you've looked at the output of `ps auxH`? If just using
htop you may not see the threads that're idle.

TCP connections are pinned to a specific worker thread on connection.
Trivial benchmarks may not load the worker threads evenly, as the
connections are handed to threads evenly via round robin.

> https://groups.google.com/d/msgid/memcached/CAL%3D0poZ-A0yCr%3DLAAoqcC041bEfC9rthdVZRHCVV-f7xUJ-e4g%40mail.gmail.com.
>
>

[Alireza Sanaee]

Hi,

I'm investigating the Linux load balancer, meanwhile, I'm trying to understand what is happening in Memcached and just noticed the different number of Memcached threads on my machines. Linux doesn't provide immediate access to generally latency-sensitive threads/processes like Memcached worker threads, causing HoL and eventually long-tail latency, not a new thing though. 

But before that, I should know what worker threads I need to consider. You gave me some pointers to different internal threads of Memcached though. Even intermittent occurrence of some internal worker threads of Memcached(rebalancer, crawler or ...) might block some requests (I'm not sure if that's the case or not). ULE scheduler sounds suffering from the same flaw.

The connection dispatcher is there, but I don't have so many connections creation, I have a good number of clients, and send requests at some rates until the end of the experiment. I'm using mutilate as my workload generator.

Sure, I can check whether those are idle or not, I'm actually recording everything from `/proc/<memcached-pid>/..../stat` so the status is also available there. It is a true fact that internal threads are IDLE most of the time and it is some sort of visible in the plot, but I just want to make sure that all mostly idle ones are just internal threads and not workers. As you said some times workloads are not loaded evenly making the worker threads more difficult to distinguish. I think this doesn't matter now. 

The main issue here is that I have 6 threads on my big machine and 10 threads on my small machine, while I have the same Memcached configuration for both machines. I have attached the numbers for the two machines.

Thanks,

Alireza

To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.1912142337470.3156%40dskull.

[Alireza Sanaee]

Hi,

I did a stupid mistake, the big machine Memcached version seems different. I guess that is the problem.

Thanks,

Alireza

[dormando]

Yup :) looks like you left a system installed version of memcached running
on the other one.

> https://groups.google.com/d/msgid/memcached/CAL%3D0pobwb6VSGLcPONdvrk4nQJAXzk_mypxiZCYmSSTtheyeVA%40mail.gmail.com.
>
>