Reviewboard 6.0.2 in Docker connects to the wrong memcached server 50% of the time

[Szymon Niemiec]

Hello everyone,

We are using latest reviewboard and we've recently migrated to Docker setup. Problem is, reviewboard randomly spits error 500 while browsing the site. Looking into `docker compose logs -f`, we can see that sometimes (not always) it tries to connect to memcached server on localhost:11211:

 

reviewboard-1  | 2024-03-14 10:36:23,523 - ERROR - None - AnonymousUser - /health/ - djblets.util.views - Health check: unable to connect to cache "default": All servers seem to be down right now
reviewboard-1  | 2024-03-14 10:36:23,524 - ERROR - None - AnonymousUser - /health/ - django.request - Service Unavailable: /health/
reviewboard-1  | 2024-03-14 10:36:23,525 - DEBUG -  - pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1  | [2024-03-14 10:36:23 +0000] [17] [DEBUG] Ignoring connection reset
reviewboard-1  | [2024-03-14 10:36:27 +0000] [20] [DEBUG] GET /
reviewboard-1  | 2024-03-14 10:36:27,126 - DEBUG -  - pymemcache.client.hash - retrying failed server: ('127.0.0.1', 11211)
reviewboard-1  | 2024-03-14 10:36:27,127 - ERROR -  - djblets.cache.synchronizer - Unexpected error checking for expiration in cached synchronization state key "{OUR_DOMAIN}:/:{OUR_DOMAIN}:siteconfig:1:generation". Is the cache server down? Error = [Errno 111] Connection refused
reviewboard-1  | Traceback (most recent call last):
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line 82, in is_expired
reviewboard-1  |     sync_gen = self._get_latest_sync_gen()
reviewboard-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line 162, in _get_latest_sync_gen
reviewboard-1  |     return cache.get(self.cache_key)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py", line 77, in get
reviewboard-1  |     return self._cache.get(key, default)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347, in get
reviewboard-1  |     return self._run_cmd("get", key, default, default=default, **kwargs)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322, in _run_cmd
reviewboard-1  |     return self._safely_run_func(client, func, default_val, *args, **kwargs)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 199, in _safely_run_func
reviewboard-1  |     result = func(*args, **kwargs)
reviewboard-1  |              ^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687, in get
reviewboard-1  |     return self._fetch_cmd(b"get", [key], False, key_prefix=self.key_prefix).get(
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133, in _fetch_cmd
reviewboard-1  |     self._connect()
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424, in _connect
reviewboard-1  |     sock.connect(sockaddr)
reviewboard-1  | ConnectionRefusedError: [Errno 111] Connection refused
reviewboard-1  | 2024-03-14 10:36:27,128 - DEBUG -  - pymemcache.client.hash - marking server as dead: ('127.0.0.1', 11211)
reviewboard-1  | 2024-03-14 10:36:27,129 - ERROR -  - djblets.cache.synchronizer - Unexpected error checking for expiration in cached synchronization state key "{OUR_DOMAIN}:/:extensionmgr:reviewboard.extensions:gen". Is the cache server down? Error = [Errno 111] Connection refused
reviewboard-1  | Traceback (most recent call last):
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line 82, in is_expired
reviewboard-1  |     sync_gen = self._get_latest_sync_gen()
reviewboard-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line 162, in _get_latest_sync_gen
reviewboard-1  |     return cache.get(self.cache_key)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py", line 77, in get
reviewboard-1  |     return self._cache.get(key, default)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347, in get
reviewboard-1  |     return self._run_cmd("get", key, default, default=default, **kwargs)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 322, in _run_cmd
reviewboard-1  |     return self._safely_run_func(client, func, default_val, *args, **kwargs)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 211, in _safely_run_func
reviewboard-1  |     result = func(*args, **kwargs)
reviewboard-1  |              ^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 687, in get
reviewboard-1  |     return self._fetch_cmd(b"get", [key], False, key_prefix=self.key_prefix).get(
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 1133, in _fetch_cmd
reviewboard-1  |     self._connect()
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/base.py", line 424, in _connect
reviewboard-1  |     sock.connect(sockaddr)
reviewboard-1  | ConnectionRefusedError: [Errno 111] Connection refused
reviewboard-1  | 2024-03-14 10:36:27,131 - ERROR -  - djblets.cache.synchronizer - Unexpected error checking for expiration in cached synchronization state key "{OUR_DOMAIN}:/:integrationmgr:djblets.integrations.manager.IntegrationManager:gen". Is the cache server down? Error = All servers seem to be down right now
reviewboard-1  | Traceback (most recent call last):
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line 82, in is_expired
reviewboard-1  |     sync_gen = self._get_latest_sync_gen()
reviewboard-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/djblets/cache/synchronizer.py", line 162, in _get_latest_sync_gen
reviewboard-1  |     return cache.get(self.cache_key)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/django/core/cache/backends/memcached.py", line 77, in get
reviewboard-1  |     return self._cache.get(key, default)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 347, in get
reviewboard-1  |     return self._run_cmd("get", key, default, default=default, **kwargs)
reviewboard-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 314, in _run_cmd
reviewboard-1  |     client = self._get_client(key)
reviewboard-1  |              ^^^^^^^^^^^^^^^^^^^^^
reviewboard-1  |   File "/venv/lib/python3.11/site-packages/pymemcache/client/hash.py", line 182, in _get_client
reviewboard-1  |     raise MemcacheError("All servers seem to be down right now")'

This won't work, because memcached is running in separate container in compose setup. But in settings-local.py the address is correct:

# cat sitedir/conf/settings_local.py
...

# Cache backend settings.
CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.memcached.PyMemcacheCache",
        "LOCATION": "memcached:11211"
    }
}
...

And there is no 127.0.0.1:11211 anywhere in settings, so I assume this is default value that somehow ended up in the cache server pool.

How to fix this?
The problem is frequent, basically 50% page loads fail due to this.

Thanks in advance,
Best regards,
Szymon

[David Trowbridge]

Can you check what the cache configuration is at /admin/settings/general/ ?

David

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/f083ab4c-cea6-4eda-aa17-0aa9e28b6239n%40googlegroups.com.

[Szymon Niemiec]

Hello, you are right, the setting was wrong, as seen from the screenshot below.
Putting proper URL fixes the problem.

Sorry for bothering, then. I was handling the whole migration process from the command line and didn't occur to me to check the admin area, I assume what's in the DB overrides the setting in settings-local.py

Thank you for help!
BR
Szymon