Channels - unbalanced load for threaded workers

[biwa...@gmail.com]

I am trying to deploy Django project with Channels. After some trial and error I settled on 4 workers with 16 threads each. However as attached screen shows, the processes that are spawned by supervisor have CPU usage at ~15% while threads are around ~1-2%. Number of connections at the moment is less than 100. I tried load testing this setup, and as more messages are coming in the CPU usage is increasing both on processes that were spawned by supervisor and threads of these processes - so there is still disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure it some other way to better balance the load? 

[Andrew Godwin]

Hi - I am not sure what you are saying exactly. That the load only goes to some processes? Python threads are strange, the OS may not show stats correctly and I'd generally advise multiple processes instead.

Andrew

On Wed, May 17, 2017 at 1:53 AM, <biwa...@gmail.com> wrote:

I am trying to deploy Django project with Channels. After some trial and error I settled on 4 workers with 16 threads each. However as attached screen shows, the processes that are spawned by supervisor have CPU usage at ~15% while threads are around ~1-2%. Number of connections at the moment is less than 100. I tried load testing this setup, and as more messages are coming in the CPU usage is increasing both on processes that were spawned by supervisor and threads of these processes - so there is still disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure it some other way to better balance the load? 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b566a9bc-5313-4573-9351-d6dd81915251%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen J. Butler]

CPython uses a global interpreter lock, which means that only one thread per process can ever be running Python code. Where threading really helps is if your python project is mostly I/O bound (waiting on database work, network connections, etc). If it's CPU bound then you probably won't see much concurrency with multiple threads.

I'm guessing this is why you're seeing all your work being accounted to a couple processes. You'd get better concurrency moving to a multiprocess/fork model.

https://www.ploggingdev.com/2017/01/multiprocessing-and-multithreading-in-python-3/

On Wed, May 17, 2017 at 3:53 AM, <biwa...@gmail.com> wrote:

I am trying to deploy Django project with Channels. After some trial and error I settled on 4 workers with 16 threads each. However as attached screen shows, the processes that are spawned by supervisor have CPU usage at ~15% while threads are around ~1-2%. Number of connections at the moment is less than 100. I tried load testing this setup, and as more messages are coming in the CPU usage is increasing both on processes that were spawned by supervisor and threads of these processes - so there is still disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure it some other way to better balance the load? 

--

[biwa...@gmail.com]

Yes, that was my concern that only some processes are loaded. So running more 'runworker' commands with less '--threads' should give better results?

W dniu czwartek, 18 maja 2017 02:18:56 UTC+2 użytkownik Andrew Godwin napisał:

Hi - I am not sure what you are saying exactly. That the load only goes to some processes? Python threads are strange, the OS may not show stats correctly and I'd generally advise multiple processes instead.

Andrew

On Wed, May 17, 2017 at 1:53 AM, <biwa...@gmail.com> wrote:

I am trying to deploy Django project with Channels. After some trial and error I settled on 4 workers with 16 threads each. However as attached screen shows, the processes that are spawned by supervisor have CPU usage at ~15% while threads are around ~1-2%. Number of connections at the moment is less than 100. I tried load testing this setup, and as more messages are coming in the CPU usage is increasing both on processes that were spawned by supervisor and threads of these processes - so there is still disproportion (~50% and ~10%). Is it normal behavior? Any tips to configure it some other way to better balance the load? 

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Andrew Godwin]

Yes. Running with no --threads option will give you the best spread, but result in more RAM usage. Rather than relying on process statistics via htop or similar, I'd suggest benchmarking actual throughput, as that's the thing you want to optimise for at the end of the day; it might be that one way uses less CPU but has higher throughput, for example.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b31841c7-ea17-4189-a9b8-f6f0e0d7dc79%40googlegroups.com.