Connection reset by peer error problems

[Santiago Del Valle]

We are using a managed memorystore connected to out application using serverles VPC access, our application is python based.

It was working properly until last week, it started failing in one of our environments. We suspect that the this type of connection we are using could cause the pool of connetions to sleep, but it was working properly and then not.

We get a lot of

ConnectionResetError: [Errno 104] Connection reset by peer

errors

and the redis is totally unresponsive, the problem solved itself but know we have the same problem in another one of our environments, again, with no code changes.

Is there a known root cause for these kind of errors?, here is the full stack trace

"Traceback (most recent call last):

  File "/env/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner

    response = get_response(request)

  File "/env/lib/python3.7/site-packages/django/core/handlers/base.py", line 156, in _get_response

    response = self.process_exception_by_middleware(e, request)

  File "/env/lib/python3.7/site-packages/django/core/handlers/base.py", line 154, in _get_response

    response = response.render()

  File "/env/lib/python3.7/site-packages/django/template/response.py", line 106, in render

    self.content = self.rendered_content

  File "/env/lib/python3.7/site-packages/django/template/response.py", line 83, in rendered_content

    content = template.render(context, self._request)

  File "/env/lib/python3.7/site-packages/django/template/backends/django.py", line 61, in render

    return self.template.render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 171, in render

    return self._render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 163, in _render

    return self.nodelist.render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 937, in render

    bit = node.render_annotated(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 904, in render_annotated

    return self.render(context)

  File "/env/lib/python3.7/site-packages/django/template/loader_tags.py", line 150, in render

    return compiled_parent._render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 163, in _render

    return self.nodelist.render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 937, in render

    bit = node.render_annotated(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 904, in render_annotated

    return self.render(context)

  File "/env/lib/python3.7/site-packages/django/template/loader_tags.py", line 188, in render

    return template.render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 173, in render

    return self._render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 163, in _render

    return self.nodelist.render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 937, in render

    bit = node.render_annotated(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 904, in render_annotated

    return self.render(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 987, in render

    output = self.filter_expression.resolve(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 671, in resolve

    obj = self.var.resolve(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 796, in resolve

    value = self._resolve_lookup(context)

  File "/env/lib/python3.7/site-packages/django/template/base.py", line 837, in _resolve_lookup

    current = getattr(current, bit)

  File "/env/lib/python3.7/site-packages/wagtail/core/models.py", line 794, in get_full_url

    url_parts = self.get_url_parts(request=request)

  File "/env/lib/python3.7/site-packages/wagtail/core/models.py", line 765, in get_url_parts

    for pk, path, url in self._get_site_root_paths(request)

  File "/env/lib/python3.7/site-packages/wagtail/core/models.py", line 742, in _get_site_root_paths

    cache_object._wagtail_cached_site_root_paths = Site.get_site_root_paths()

  File "/env/lib/python3.7/site-packages/wagtail/core/models.py", line 151, in get_site_root_paths

    result = cache.get('wagtail_site_root_paths')

  File "/env/lib/python3.7/site-packages/django_redis/cache.py", line 32, in _decorator

    return method(self, *args, **kwargs)

  File "/env/lib/python3.7/site-packages/django_redis/cache.py", line 81, in get

    client=client)

  File "/env/lib/python3.7/site-packages/django_redis/client/default.py", line 203, in get

    value = client.get(key)

  File "/env/lib/python3.7/site-packages/redis/client.py", line 1332, in get

    return self.execute_command('GET', name)

  File "/env/lib/python3.7/site-packages/redis/client.py", line 836, in execute_command

    conn = self.connection or pool.get_connection(command_name, **options)

  File "/env/lib/python3.7/site-packages/redis/connection.py", line 1065, in get_connection

    if connection.can_read():

  File "/env/lib/python3.7/site-packages/redis/connection.py", line 682, in can_read

    return self._parser.can_read(timeout)

  File "/env/lib/python3.7/site-packages/redis/connection.py", line 295, in can_read

    return self._buffer and self._buffer.can_read(timeout)

  File "/env/lib/python3.7/site-packages/redis/connection.py", line 205, in can_read

    raise_on_timeout=False)

  File "/env/lib/python3.7/site-packages/redis/connection.py", line 173, in _read_from_socket

    data = recv(self._sock, socket_read_size)

  File "/env/lib/python3.7/site-packages/redis/_compat.py", line 58, in recv

    return sock.recv(*args, **kwargs)

ConnectionResetError: [Errno 104] Connection reset by peer"  

[eric.d...@gmail.com]

I am suddenly (first became aware Aug 15th) of a similar issue on my Flask-based application. In my case I am using redis/memorystore to handle sessions, so the application fails entirely in this case. I have only been on GAE for a few weeks but never had a problem like this when I was running it on a traditional VM. I'm just not sure how to deal with it.. 

ConnectionResetError: [Errno 104] Connection reset by peer

[Josh L.]

We're seeing intermittent Memorystore command/connection timeouts in our Python (Flask) GAE Standard app that's accessing Memorystore over a VPC connection as well. We're mostly using Redis as a caching layer, so we try to fail fast and just fall back to querying directly from the database. Still, it adds unnecessary delay when this is happening and I'd love to better understand why we're seeing this and what we can do to mitigate it. 

[joeost...@gmail.com]

I'm experiencing the same issue with a NodeJS application. Getting ECONNRESET when making a cold request. This happens multiple times per day and has forced us to rewrite portions of the application so we can query the DB as a fallback. I hope somebody figures this out soon...

On Monday, August 12, 2019 at 3:10:33 PM UTC-4, Santiago Del Valle wrote:

[Josh L.]

We experienced sustained connectivity issues during Google's widespread Auth issues yesterday (GCloud, GSuite) and again this morning. I saw you posted in the Google Cloud Slack in the #memorystore channel, so I cross posted in the #networking channel to see if anyone there has any ideas.

[ar...@gaincompliance.com]

Did you find a solution to this issue?

[snar...@google.com]

We on Appengine are investigating this issue. There seem to be two separate discussions here:

Connection Resets: This is under investigation, in the meantime for python apps using health_check_interval on redis connections might be helpful. https://github.com/andymccurdy/redis-py/issues/1186

Connection Timeouts: We are rolling out a fix for this and should be available end of next week.

[Andy Summers]

Apologies for bumping an old thread but just wanted to check whether there was any progress made on the reset investigation?

Facing a similar issue with a NodeJS Cloud Run application & MemoryStore Redis where the connection is reset every 30 minutes.