Scaling out Netbox over multiple machines

[Yusuf Tran]

Hi,

Just wondering if put netbox on a couple VMs under a load balancer if it's possible to scale out.

Things I can break out:

- db

- redis

Do I need to be concerned with rq worker? do I need to ensure it only runs on one instance?

Thanks

Yusuf

[Brian Candler]

Related question: https://github.com/netbox-community/netbox/discussions/7065

Good question about workers.  As far as I know, it's perfectly fine to have multiple workers consuming the same queue - the rq documentation here says it's fine (and this is how you perform work concurrently).

[Yusuf Tran]

thanks Brian

[Yusuf Tran]

I seem to be having an issue with breaking out redis (to google cloud memorystore - redis), when I startup rq worker tries to use the CLIENT command which google block on managed redis.

Just wondering what the ideal workaround is?

Is it safe to run redis locally on multiple instances?

[Yusuf Tran]

output from docker-compose:

Starting netbox_netbox_1 ... done
Starting netbox_netbox-worker_1 ... done
Attaching to netbox_netbox_1, netbox_netbox-worker_1
netbox-worker_1  | 🧬 loaded config '/etc/netbox/config/configuration.py'
netbox-worker_1  | 🧬 loaded config '/etc/netbox/config/extra.py'
netbox-worker_1  | 🧬 loaded config '/etc/netbox/config/logging.py'
netbox-worker_1  | 🧬 loaded config '/etc/netbox/config/plugins.py'
netbox-worker_1  | Traceback (most recent call last):
netbox-worker_1  |   File "/opt/netbox/netbox/manage.py", line 10, in <module>
netbox-worker_1  |     execute_from_command_line(sys.argv)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
netbox-worker_1  |     utility.execute()
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
netbox-worker_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
netbox-worker_1  |     self.execute(*args, **cmd_options)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
netbox-worker_1  |     output = self.handle(*args, **options)
netbox-worker_1  |   File "/opt/netbox/netbox/extras/management/commands/rqworker.py", line 26, in handle
netbox-worker_1  |     super().handle(*args, **options)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/django_rq/management/commands/rqworker.py", line 122, in handle
netbox-worker_1  |     w = get_worker(*args, **worker_kwargs)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/django_rq/workers.py", line 51, in get_worker
netbox-worker_1  |     return worker_class(queues,
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/rq/worker.py", line 224, in __init__
netbox-worker_1  |     connection.client_setname(self.name)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/redis/client.py", line 1206, in client_setname
netbox-worker_1  |     return self.execute_command('CLIENT SETNAME', name)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
netbox-worker_1  |     return self.parse_response(conn, command_name, **options)
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
netbox-worker_1  |     response = connection.read_response()
netbox-worker_1  |   File "/opt/netbox/venv/lib/python3.9/site-packages/redis/connection.py", line 756, in read_response
netbox-worker_1  |     raise response
netbox-worker_1  | redis.exceptions.ResponseError: unknown command `CLIENT`, with args beginning with: `SETNAME`, `41f6c8eef71443e8a7715e1906e96d99`,
netbox_1         | ⏳ Waiting on DB... (0s / 30s)
netbox_netbox-worker_1 exited with code 1
netbox_1         | ⏳ Waiting on DB... (3s / 30s)
netbox_1         | ⏳ Waiting on DB... (6s / 30s)
netbox_1         | ⏳ Waiting on DB... (9s / 30s)
netbox_1         | ⏳ Waiting on DB... (12s / 30s)
netbox_1         | ⏳ Waiting on DB... (15s / 30s)
netbox_1         | ⏳ Waiting on DB... (18s / 30s)
netbox_1         | ⏳ Waiting on DB... (21s / 30s)
netbox_1         | ⏳ Waiting on DB... (24s / 30s)
netbox_1         | ⏳ Waiting on DB... (27s / 30s)
netbox_1         | ❌ Waited 30s or more for the DB to become ready.
netbox_netbox_1 exited with code 1

[Brian Candler]

That's a perfectly standard redis command:

https://redis.io/commands/client-setname

Is it possible that the Google service requires you to authenticate before using it?  Try using redis-cli against the Google service to understand what's going on.

To rule in or out a problem with the Google redis service, you could run your own instance of redis in a container and configure Netbox to talk to it.