How to debug Daphne CPU 100% issue

[Fogmoon]

Hi, 

Could you please give some suggestions to inspect the high CPU (most time above 90% and some time above 100%) consumed by Daphne. 

Environment:

• OS: Ubuntu 18.04.1 LTS x86_64 GNU/Linux
• Django: 2.1.1
• Daphne: 2.3.0
• Channels: 2.2.0
• channels-redis: 2.4.0

Just use the Daphne as the websocket server with redis as the Channel layer. 

Now I do the following steps:

1. top command show the daphne process consume high cpu
2. check the thread numbers with ps -o thcount 22549, which shows 40. 
3. check the websocket connected numbers with redis cli: ZCOUNT asgi::group:USER_MESSAGE -inf +inf, which shows 190
4. check the system call with sudo strace -c -p 22549 for a minute, the result as the following:

Best Wishes.

[Integr@te System]

Hi Fogmoon,

Plz show again your image file's result and addition check 

iostat -ctz

netstat -anob

to see more root cause.

Dont for get update python version.

Nice.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/913143a8-c8c8-47ce-a96d-6c9d349eac6a%40googlegroups.com.

[sagar ninave]

can you solve my this issue

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAP5HUWobxGBszupnc_3iGNMi1j1mF18VJ%3DkjkbGXxvYN28KmNg%40mail.gmail.com.

--

sagar ninave about.me/sagarninave

[Fogmoon]

Thanks very much for your reply.

Which make me confused now the CPU of Daphne is normal, keep 8% ~20%. I update the new info and new check here. I think it will return to 100% because I have not found the reason and do nothing.

Environment:

• OS: Ubuntu 18.04.1 LTS x86_64 GNU/Linux

• Python: 3.6.5

• Django: 2.1.1
• Daphne: 2.3.0
• Channels: 2.2.0
• channels-redis: 2.4.0

Check Info:

1. check the thread numbers with ps -o thcount 22549, which shows 45 ( > 40)
2. check the websocket connected numbers with redis cli: ZCOUNT asgi::group:USER_MESSAGE -inf +inf, which shows 160  ( < 190)

1. check the system call with sudo strace -c -p 22549 for a minute, the result as the following:

New Check Info:

1. check the system files open limits with : sudo sysctl -p | grep file, which shows: 65535
2. check process files open limits with: cat /proc/22549/limits | grep -E 'files | Soft', which shows: 

           In fact, I set the system and process open files(with supervisord conf) limit long long ago.

       3. check the process open files number with ls -l /proc/22549/fd |  wc -l: 145

       4. check the cpu and io stat with iostat -ctz:

       5. check the net stat with netstat -ano | grep daphne.sock (because ubuntu do not support the b parameter): which show 133, 1 is listening status, 132 is connected status.

在 2020年3月2日星期一 UTC+8下午7:40:11，Integr@te System写道：

To unsubscribe from this group and stop receiving emails from it, send an email to django...@googlegroups.com.

[Fogmoon]

Yes, the CPU return %80 ~ 100%.

I still check all the items, it just seems OK,  which have the similar value like that when CPU is normal.

One obvious difference is the strace command (sudo strace -c -p 22549) output:

When CPU is high, the output always show the right 64bit mode after the wrong x32 mode:

strace: Process 22549 attached

strace: [ Process PID=22549 runs in x32 mode. ]

strace: [ Process PID=22549 runs in 64 bit mode. ]

^Cstrace: Process 22549 detached

While CPU is low, the output just has the attach info:

strace: Process 22549 attached

^Cstrace: Process 22549 detached

Do not mind to misleading the inspection (I know strace may show two modes, check here.), but until now, I just find the one different thing.

Thanks.

在 2020年3月2日星期一 UTC+8下午9:49:04，Fogmoon写道：