uwsgi with web2py hard crash/locking with nginx

[lucas]

hello one and all,

i'm using web2py 2.25.1 with uwsgi 2.0.22 which was downloaded and compiled using pip3 under python 3.10.12 under ubuntu 22.04.  the nginx is apt installed version 1.18.

with that being said, if i install and run my high calculation / high plotly plot pages about 6 to 12 times, the whole damn system comes to a crash.  hard crash.  this also seems to be the case under redhat / amazon linux 2023 similar setup and config.

so i tried crashing the system under just "python3 web2py.py -i 0.0.0.0 -p 80 --no_gui" and also under "uwsgi --http :80 --chdir /opt/web2py_apps/web2py -w wsgihandler:application" but i could NOT get either to crash.  it is only when i nginx daemon it with uwsgi and web2py layered/socketed under nginx that it DOES crash hard.  i think i found under the systemctl logs for uwsgi a logging of the crash, or:

Oct 16 13:48:30 iquanta.info systemd[1]: uwsgi.service: A process of this unit has been killed by the OOM killer.
Oct 16 13:48:30 iquanta.info uwsgi[1264]: [emperor] *** RAGNAROK EVOKED ***
Oct 16 13:48:30 iquanta.info uwsgi[1265]: SIGINT/SIGTERM received...killing workers...
Oct 16 13:48:30 iquanta.info uwsgi[1264]: Mon Oct 16 13:48:30 2023 - [emperor] stop the uwsgi instance web2py.ini
Oct 16 13:48:30 iquanta.info uwsgi[1265]: Mon Oct 16 13:48:30 2023 - received message 0 from emperor
Oct 16 13:48:31 iquanta.info uwsgi[1265]: worker 5 buried after 1 seconds
Oct 16 13:48:32 iquanta.info uwsgi[1265]: worker 1 buried after 2 seconds
Oct 16 13:48:32 iquanta.info uwsgi[1265]: worker 2 buried after 2 seconds
Oct 16 13:48:32 iquanta.info uwsgi[1265]: worker 3 buried after 2 seconds
Oct 16 13:48:32 iquanta.info uwsgi[1265]: worker 4 buried after 2 seconds
Oct 16 13:48:32 iquanta.info uwsgi[1265]: goodbye to uWSGI.
Oct 16 13:48:32 iquanta.info uwsgi[1265]: VACUUM: unix socket /run/uwsgi/web2py.sock removed.
Oct 16 13:48:33 iquanta.info uwsgi[1264]: Mon Oct 16 13:48:33 2023 - [emperor] removed uwsgi instance web2py.ini
Oct 16 13:48:34 iquanta.info uwsgi[1264]: Mon Oct 16 13:48:34 2023 - The Emperor is buried.
Oct 16 13:48:34 iquanta.info systemd[1]: uwsgi.service: Failed with result 'oom-kill'.
Oct 16 13:48:34 iquanta.info systemd[1]: uwsgi.service: Consumed 10.241s CPU time.
Oct 16 13:48:34 iquanta.info systemd[1]: uwsgi.service: Scheduled restart job, restart counter is at 1.
Oct 16 13:48:34 iquanta.info systemd[1]: Stopped uWSGI Emperor service.
Oct 16 13:48:34 iquanta.info systemd[1]: uwsgi.service: Consumed 10.241s CPU time.

with not much to say under the nginx logs.  so, does anyone know what i can be dealing with?  thank you in advance, lucas

[Tom Clerckx]

Maybe you should review the 'high calculation/high plotly' pages for memory usage?

As the 'OOM killer' kicked in, it looks like the process is using too much memory.

One thing to verify this is to monitor the VmRSS of the processes involved (VmRSS is one of the fields in the output when you do 'cat /proc/<pid>/status') 

VmRSS gives you an idea about the amount of RAM memory your process is using.

If it's growing out of proportion, you're claiming too much memory and the process will be killed by the OOM-killer.

[Christian Varas]

While I can't provide a specific solution, it might be a better approach to keep resource-intensive tasks separate from web2py, using a tool like Celery. This way, these tasks can run in the background without impacting the web server.

This is a small docx about how to put it together: https://docs.google.com/document/d/1JXToOdSSeDlxV_ev2NYYPeVKtY8eJS2v/edit?usp=sharing&ouid=107281473670459955205&rtpof=true&sd=true

Greetings

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/af50af3f-2764-4cbf-8bfd-06c802d0ff93n%40googlegroups.com.

[lucas]

yes, i think once the RAM was consumed the server ground to a halt where it appeared to crash.  so, i believe it was a soft crash instead of a hard crash.  i used "top" to monitor the memory and processes and the soft crash occurred when the RAM was exhausted.  so i created a swap file to extend the RAM using this series of CLI/sh commands:

df -h
free -m
swapon --show

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

#under /etc/fstab, add the line:
/swapfile swap swap defaults 0 0

swapon --show
free -m
df -h

which did the trick as i still watched under top.  therefore, creating the swap file did the trick.  i also updated my uwsgi initialization file to allow for more processes, or:

[uwsgi]
chdir = /opt/web2py_apps/web2py
module = wsgihandler:application

master = true
processes = 5
threads = 2
thread-stacksize = 512

uid = web2py
gid = web2py
socket = /run/uwsgi/web2py.sock
chown-socket = web2py:nginx
chmod-socket = 660
vacuum = true

thank you for your help.  lucas