App Engine doesn't send response leads to timeout at client

[Nikhil Patil]

I have Django application deployed on App Engine standard environment. Not all the time but sometimes App engine doesn't send response back which leads to timeout at client side. I have logged the response contents just before sending response in django code and the contents are correct. All the source code execution is as expected just no response . Please suggest me if anyone had same kind of issue.

[Andres Fiesco Casasola]

Do you get any errors when you get the timeout? Could please post your code and your logs? I found these stackoverflow cases that can help you (most common errors): 

[1]:https://stackoverflow.com/questions/68420538/google-cloud-app-engine-deploy-fail-django

[2]:https://stackoverflow.com/questions/70128374/google-app-engine-django-project-infinite-loading-and-error-500-does-not-raise-a 

Here is some Google documentation that can help you running Django in App Engine: 

[3]:https://cloud.google.com/python/django/appengine

[Nikhil Patil]

I think no issue on application side. I have logged upto last line of my post method and after that is return response. 

Code:

class myView(generics.ListCreateAPIView):
permission_classes = (IsAuthenticated,)
http_method_names = ['get', 'post']
search_fields = ['some', 'thing', 'here'']
filter_backends = (DynamicSearchFilter,)
pagination_class = PageNumberPagination
serializer_class = mySerializer
queryset = mymodel.objects.all().order_by("-update_date")

def __init__(self, **kwargs):

.

.

.

def create(self, request, *args, **kwargs):

.

.

.

logger.info(f"Returning response of size {len(response_data)}")

return Response(response_data, content_type='application/json')

here are my logs.
[
  {
    "textPayload": "2022-04-14 03:05:21,344 INFO app.myview  create  Returning response of size 11",
    "insertId": "62578f71000541533a99d6a0",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "version_id": "20220406t140722"
      }
    },
    "timestamp": "2022-04-14T03:05:21.344403Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:05:21.478498355Z"
  },
  {
    "textPayload": "[start] 2022/04/14 03:20:24.968447 Quitting on terminated signal",
    "insertId": "625792f8000ec856db894b8e",
    "resource": {
      "type": "gae_app",
      "labels": {
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "version_id": "20220406t140722",
        "project_id": "my-prj"
      }
    },
    "timestamp": "2022-04-14T03:20:24.968790Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/%2Fvar%2Flog%2Fgoogle_init.log",
    "receiveTimestamp": "2022-04-14T03:20:24.971407404Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:24 +0000] [10] [INFO] Handling signal: term",
    "insertId": "625792f8000ed2e870511881",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:24.971496Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:24.975480517Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:24 +0000] [24] [INFO] Worker exiting (pid: 24)",
    "insertId": "625792f8000f38c249d53b78",
    "resource": {
      "type": "gae_app",
      "labels": {
        "module_id": "myapp",
        "project_id": "my-prj",
        "zone": "asia-northeast1-1",
        "version_id": "20220406t140722"
      }
    },
    "timestamp": "2022-04-14T03:20:24.997570Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [27] [INFO] Worker exiting (pid: 27)",
    "insertId": "625792f90000530acac710eb",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj"
      }
    },
    "timestamp": "2022-04-14T03:20:25.021258Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [25] [INFO] Worker exiting (pid: 25)",
    "insertId": "625792f9000053b58a77b5a5",
    "resource": {
      "type": "gae_app",
      "labels": {
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj",
        "version_id": "20220406t140722"
      }
    },
    "timestamp": "2022-04-14T03:20:25.021429Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [26] [INFO] Worker exiting (pid: 26)",
    "insertId": "625792f9000053c2b6f2baa3",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "project_id": "my-prj",
        "zone": "asia-northeast1-1",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:25.021442Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [21] [INFO] Worker exiting (pid: 21)",
    "insertId": "625792f9000091ae98501e35",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:25.037294Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [28] [INFO] Worker exiting (pid: 28)",
    "insertId": "625792f90000d65126c969c9",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:25.054865Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [29] [INFO] Worker exiting (pid: 29)",
    "insertId": "625792f900085afde9ceaafa",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:25.547581Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.641464142Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [30] [INFO] Worker exiting (pid: 30)",
    "insertId": "625792f90009cadbec1f368e",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:25.641755Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.972881563Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:27 +0000] [10] [INFO] Shutting down: Master",
    "insertId": "625792fb000a0140cf4bc85e",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "project_id": "my-prj",
        "zone": "asia-northeast1-1",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:27.655680Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:27.972510397Z"
  },
  {
    "textPayload": "[start] 2022/04/14 03:20:27.698037 Start program failed: termination triggered by nginx exit",
    "insertId": "625792fb000aa82c3c8ba36a",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:27.698412Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/%2Fvar%2Flog%2Fgoogle_init.log",
    "receiveTimestamp": "2022-04-14T03:20:27.968578942Z"
  },
  

[Horace (Cloud Platform Support)]

Hello

Since we don't have visibility on your project to check your App Engine settings including your yaml file and further use our internal tools to find out what is happening, kindly take a look at your configuration such as your max instances, idle time out, and scaling. From your log, the ‘Quitting on terminated signal’ indicates there is a shutdown of an instance [1], it could be that there are no requests hence the instance was shutdown and there could be a spike in subsequent requests that the instance was not ready to respond since there are no instances available.

Lastly, kindly take a look at your scaling, do note that there is no guaranteed uptime for basic and manual scaling [3]. Due to this behavior, if you want a higher Monthly Uptime Percentage, try to configure your instance with automatic scaling. You can set the max_instances and max_idle_instances to values that better achieve your architecture in the app.yaml file [4]

[1] https://cloud.google.com/appengine/docs/standard/go/how-instances-are-managed#shutdown

[2] https://cloud.google.com/appengine/docs/standard/python3/configuring-warmup-requests

[3] https://cloud.google.com/appengine/docs/standard/go/how-instances-are-managed#instance_uptime

[4] https://cloud.google.com/appengine/docs/standard/go/config/appref#scaling_elements

[Nikhil Patil]

To Explain my overall situation, My app response is lost though the API program executes as expected. This happens randomly.

here is my app.yaml

#
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# [START django_app]
# [START gaestd_py_django_app_yaml]

## std + dev-apc
#runtime: python39
#service: dev-apc
#instance_class: F1


handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
static_dir: staticfiles/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
script: auto
# [END gaestd_py_django_app_yaml]
# [END django_app]

[Rogelio Monter Rodriguez]

Have you tried debugging your app locally using the local development server or using the local development server?

[Nikhil Patil]

I tried but it seams 

• The dev_appserver tool does not support development of Python 3 apps on Windows.

https://cloud.google.com/appengine/docs/standard/python3/tools/local-devserver-command

[Osvaldo Lopez Acuña]

It would be useful if you describe what kind of call your frontend is doing. Quota limits[1] could be reached sometimes and then you get this behavior. Appengine location, network configuration, Python, and Django versions would also be helpful to narrow your issue.

In many cases, elevated latency in your application will eventually result in 5xx server errors. Therefore, it makes sense to follow a similar set of troubleshooting steps to narrow down the root cause of both error and latency spikes, given that the causes of each might be the same.

Check Troubleshoot elevated latency in your app [2].

This document [3] answers some App Engine connectivity questions. 

[1]:https://cloud.google.com/appengine/quotas#Requests 

[2]:https://cloud.google.com/appengine/docs/troubleshooter/latency

[3]:https://cloud.google.com/appengine/docs/troubleshooter/connectivity-questions