Intermittent Connection reset by peer in us-east4
Daniel DeSousa
BACKGROUND
I have a long running discord bot (3+ years) written in discord.py which has always run on GCP, zone us-east4-a. The bot runs in k8s using discord.py 1.7.2 and python 3.9.
PROBLEM
In the past month or two, I have started to see an increasing number of connection interruptions, [Error 104] Connection reset by peer. The resets are not tied directly with the amount of activity on the bot. They happen intermittent throughout the day in production (every few minutes on average).
These resets cause random failures to the discord HTTP API and result in a high level of disconnects on the WebSocket. Many of these shard disconnects are able to RESUME but many (~200 per day) end up resulting in an IDENTIFY call like a new connection and sometimes trigger extended backoff waits and partial outages.
EXAMPLE
Here is an example of a disconnect:
Traceback (most recent call last): File "/opt/venv/lib/python3.9/site-packages/discord/shard.py", line 187, in reconnect self.ws = await asyncio.wait_for(coro, timeout=60.0) File "/usr/local/lib/python3.9/asyncio/tasks.py", line 481, in wait_for return fut.result() File "/opt/venv/lib/python3.9/site-packages/discord/gateway.py", line 305, in from_client gateway = gateway or await client.http.get_gateway() File "/opt/venv/lib/python3.9/site-packages/discord/http.py", line 967, in get_gateway data = await self.request(Route('GET', '/gateway')) File "/opt/venv/lib/python3.9/site-packages/discord/http.py", line 192, in request async with self.__session.request(method, url, **kwargs) as r: File "/opt/venv/lib/python3.9/site-packages/aiohttp/client.py", line 1117, in __aenter__ self._resp = await self._coro File "/opt/venv/lib/python3.9/site-packages/aiohttp/client.py", line 544, in _request await resp.start(conn) File "/opt/venv/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 890, in start message, payload = await self._protocol.read() # type: ignore File "/opt/venv/lib/python3.9/site-packages/aiohttp/streams.py", line 604, in read await self._waiter aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
EXPERIMENT TO ISOLATE THE PROBLEM
I performed an experiment to isolate what is causing the issue. I deployed a container with my bot to a VM (not k8s) and isolated it such that it only communicates with discord (no outside database) and automatically send it commands to simulate user behavior and load (I send about 60 commands per minute in the same server -- well under my production load). I run this for 20 minutes or until I observe if connection resets happen, and I see the following:
- In us-east4-a, I am able to reproduce intermittent connection resets.
- In us-east4-b, I am able to reproduce intermittent connection resets.
- In us-east4-c, I am able to reproduce intermittent connection resets.
- In us-central1-a, I am not able to reproduce any connection resets (even after 3 hours -- no shard disconnects at all).
- In us-east1-b, I am not able to reproduce any connection resets.
- On my laptop (residential internet on the east coast), I am not able to reproduce any connection resets.
All experiments use the same container, same machine-type and same test procedure.
I repeated the experiment in us-east4-a with multiple machine types up to 8 vCPU and with both the premium and standard network tiers and I still see resets. I also tried another VM in a different project, but the connection issues always persist in us-east4.
I have a support case open with GCP as it appears to be a region specific issue.
Are there any additional experiments I could provide to attempt to narrow down the cause of this? Are there any common GCP configuration problems that could result in this problem?
Short of moving to another region, I feel as though I am out of options.
