OpenWisp Device limit? NginX throwing errors with 700 devices.

[Dieter Lubbe]

I've installed Openwisp as a POC to monitor our OpenWRT devices on our network. 

When the device count was less than 100 it ran flawlessly, but ever since more devices started auto registering, Redis and UWSGI has started to consume all available CPU resources. 

Currently I have 708 devices with more being added daily. These devices are all on the same L2 network. 

Nginx will generate "502 Bad Gateway" errors unless I clear the Redis DB and restart the services.

Where can I change the interval that OpenWisp checks in with devices? I think this might be the cause of the issue.

Thanks

Dieter

[Manu Kumar]

Interval settings can be changed in the vi /opt/openwisp2/openwisp2/settings.py file by default it is set every 5min (300 seconds). How did you delete the Redis db? 

Thanks & Regards

Manukumar N M

--
You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/d2ffb115-052b-4307-af2e-340a6afd37fdn%40googlegroups.com.

[Manu Kumar]

I don't think changing the interval of configuration will reduce this problem but there is a way to reduce it in the configuration of the redis. Just check it out by increasing the configuration checks time. If it didn't work we can discuss the same here. Because you have to increase the Celery workers also to reduce load of the server if you keep on increasing the devices in your server. 

Thanks & Regards

Manukumar N M

On Thu, Jan 11, 2024 at 11:26 PM Dieter Lubbe <diete...@gmail.com> wrote:

--

[Federico Capoano]

You can increase the intervals in the openwisp-config and openwisp-monitoring configuration files, eg:

/etc/config/openwisp (for openwisp-config)

/etc/config/openwisp-monitoring (for openwisp-monitoring)

See config options:

https://github.com/openwisp/openwisp-config?tab=readme-ov-file#configuration-options

https://github.com/openwisp/openwrt-openwisp-monitoring?tab=readme-ov-file#configuration-options

This will reduce the load on the system.

In order to increase capacity, there's also other server side tuning you can do on the linux system or distribute the load on multiple servers, but these steps are complex and the exact way to implement them depends on your infrastructure and deployment specific so I cannot explain it here in detail.

Best regards

Federico Capoano

--

[Mohammed Ismail]

i had similar thing with uwsgi 

i had to manually change config 

[uwsgi]
chdir=/opt/openwisp2
uid=www-data
module=openwisp2.wsgi:application
master=True
pidfile=/opt/openwisp2/uwsgi.pid
socket=/opt/openwisp2/uwsgi.sock
processes=12
threads=2
listen=600
harakiri=20
max-requests=5000
vacuum=True
enable-threads=True
env=HTTPS=on
buffer-size=8192
ignore-sigpipe
ignore-write-errors
disable-write-exception

this is my running config for 500+ devices VPS 8 CPUs 8 GB RAM 

device monitoring is resource extensive. also take in mind the number of interfaces being monitored. 

for my case i chose only wan interface to monitor. 

To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/CAAGgX6%2BxWvZ9FooKHcQ%3DAWJ4oPyT6ZFmbzqUN%2BSMmP%3DZhbJJBg%40mail.gmail.com.

[Manukumar NM]

Hii @Federico Capoano 

As you communicated earlier, If I increase the intervals in the openwisp-config and openwisp-monitoring configuration files, Do we need to urgest the PING check intervals on server side to reduce the load. 

Thanks & Regards,

Manukumar N M

To view this discussion on the web, visit https://groups.google.com/d/msgid/openwisp/CAAGgX6%2BxWvZ9FooKHcQ%3DAWJ4oPyT6ZFmbzqUN%2BSMmP%3DZhbJJBg%40mail.gmail.com.

[Federico Capoano]

If you want you can, you can even reduce the ping size or count if you want, it depends on the actual needs of your use case and available budget.

The upcoming version should bring some improvements to efficiency once released (or you can use the development version).

Best regards

Federico Capoano