App Engine doesn't send response leads to timeout at client

92 views
Skip to first unread message

Nikhil Patil

unread,
Apr 17, 2022, 11:37:22 AMApr 17
to Google App Engine
I have Django application deployed on App Engine standard environment. Not all the time but sometimes App engine doesn't send response back which leads to timeout at client side. I have logged the response contents just before sending response in django code and the contents are correct. All the source code execution is as expected just no response . Please suggest me if anyone had same kind of issue.

Andres Fiesco Casasola

unread,
Apr 19, 2022, 11:27:58 AMApr 19
to Google App Engine

Do you get any errors when you get the timeout? Could please post your code and your logs? I found these stackoverflow cases that can help you (most common errors): 


[1]:https://stackoverflow.com/questions/68420538/google-cloud-app-engine-deploy-fail-django

[2]:https://stackoverflow.com/questions/70128374/google-app-engine-django-project-infinite-loading-and-error-500-does-not-raise-a 


Here is some Google documentation that can help you running Django in App Engine: 


[3]:https://cloud.google.com/python/django/appengine

Nikhil Patil

unread,
Apr 21, 2022, 10:30:00 AMApr 21
to Google App Engine
I think no issue on application side. I have logged upto last line of my post method and after that is return response. 
Code:

class myView(generics.ListCreateAPIView):
permission_classes = (IsAuthenticated,)
http_method_names = ['get', 'post']
search_fields = ['some', 'thing', 'here'']
filter_backends = (DynamicSearchFilter,)
pagination_class = PageNumberPagination
serializer_class = mySerializer
queryset = mymodel.objects.all().order_by("-update_date")

def __init__(self, **kwargs):
.
.
.


def create(self, request, *args, **kwargs):
.
.
.
logger.info(f"Returning response of size {len(response_data)}")

return Response(response_data, content_type='application/json')




here are my logs.
[
  {
    "textPayload": "2022-04-14 03:05:21,344 INFO app.myview  create  Returning response of size 11",
    "insertId": "62578f71000541533a99d6a0",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "version_id": "20220406t140722"
      }
    },
    "timestamp": "2022-04-14T03:05:21.344403Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:05:21.478498355Z"
  },
  {
    "textPayload": "[start] 2022/04/14 03:20:24.968447 Quitting on terminated signal",
    "insertId": "625792f8000ec856db894b8e",
    "resource": {
      "type": "gae_app",
      "labels": {
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "version_id": "20220406t140722",
        "project_id": "my-prj"
      }
    },
    "timestamp": "2022-04-14T03:20:24.968790Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/%2Fvar%2Flog%2Fgoogle_init.log",
    "receiveTimestamp": "2022-04-14T03:20:24.971407404Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:24 +0000] [10] [INFO] Handling signal: term",
    "insertId": "625792f8000ed2e870511881",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:24.971496Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:24.975480517Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:24 +0000] [24] [INFO] Worker exiting (pid: 24)",
    "insertId": "625792f8000f38c249d53b78",
    "resource": {
      "type": "gae_app",
      "labels": {
        "module_id": "myapp",
        "project_id": "my-prj",
        "zone": "asia-northeast1-1",
        "version_id": "20220406t140722"
      }
    },
    "timestamp": "2022-04-14T03:20:24.997570Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [27] [INFO] Worker exiting (pid: 27)",
    "insertId": "625792f90000530acac710eb",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj"
      }
    },
    "timestamp": "2022-04-14T03:20:25.021258Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [25] [INFO] Worker exiting (pid: 25)",
    "insertId": "625792f9000053b58a77b5a5",
    "resource": {
      "type": "gae_app",
      "labels": {
        "module_id": "myapp",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj",
        "version_id": "20220406t140722"
      }
    },
    "timestamp": "2022-04-14T03:20:25.021429Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [26] [INFO] Worker exiting (pid: 26)",
    "insertId": "625792f9000053c2b6f2baa3",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "project_id": "my-prj",
        "zone": "asia-northeast1-1",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:25.021442Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [21] [INFO] Worker exiting (pid: 21)",
    "insertId": "625792f9000091ae98501e35",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:25.037294Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [28] [INFO] Worker exiting (pid: 28)",
    "insertId": "625792f90000d65126c969c9",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:25.054865Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.306874254Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [29] [INFO] Worker exiting (pid: 29)",
    "insertId": "625792f900085afde9ceaafa",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:25.547581Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.641464142Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:25 +0000] [30] [INFO] Worker exiting (pid: 30)",
    "insertId": "625792f90009cadbec1f368e",
    "resource": {
      "type": "gae_app",
      "labels": {
        "project_id": "my-prj",
        "version_id": "20220406t140722",
        "module_id": "myapp",
        "zone": "asia-northeast1-1"
      }
    },
    "timestamp": "2022-04-14T03:20:25.641755Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:25.972881563Z"
  },
  {
    "textPayload": "[2022-04-14 03:20:27 +0000] [10] [INFO] Shutting down: Master",
    "insertId": "625792fb000a0140cf4bc85e",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "project_id": "my-prj",
        "zone": "asia-northeast1-1",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:27.655680Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/stderr",
    "receiveTimestamp": "2022-04-14T03:20:27.972510397Z"
  },
  {
    "textPayload": "[start] 2022/04/14 03:20:27.698037 Start program failed: termination triggered by nginx exit",
    "insertId": "625792fb000aa82c3c8ba36a",
    "resource": {
      "type": "gae_app",
      "labels": {
        "version_id": "20220406t140722",
        "zone": "asia-northeast1-1",
        "project_id": "my-prj",
        "module_id": "myapp"
      }
    },
    "timestamp": "2022-04-14T03:20:27.698412Z",
    "labels": {
      "clone_id": "00c61b117cd9f4551488432b6f7441914e77d3c1dae0ed84f783550c94129238df03e3311f8837126c38d73af6577b91baa8a9c0ed6b68487259cdaad779413ebb5f1bf9d9ef33fa5de9b7b2f6d7a5"
    },
    "logName": "projects/my-prj/logs/%2Fvar%2Flog%2Fgoogle_init.log",
    "receiveTimestamp": "2022-04-14T03:20:27.968578942Z"
  },
  

Horace (Cloud Platform Support)

unread,
Apr 21, 2022, 7:11:55 PMApr 21
to Google App Engine

Hello

Since we don't have visibility on your project to check your App Engine settings including your yaml file and further use our internal tools to find out what is happening, kindly take a look at your configuration such as your max instances, idle time out, and scaling. From your log, the ‘Quitting on terminated signal’ indicates there is a shutdown of an instance [1], it could be that there are no requests hence the instance was shutdown and there could be a spike in subsequent requests that the instance was not ready to respond since there are no instances available.

Lastly, kindly take a look at your scaling, do note that there is no guaranteed uptime for basic and manual scaling [3]. Due to this behavior, if you want a higher Monthly Uptime Percentage, try to configure your instance with automatic scaling. You can set the max_instances and max_idle_instances to values that better achieve your architecture in the app.yaml file [4]


[1] https://cloud.google.com/appengine/docs/standard/go/how-instances-are-managed#shutdown

[2] https://cloud.google.com/appengine/docs/standard/python3/configuring-warmup-requests

[3] https://cloud.google.com/appengine/docs/standard/go/how-instances-are-managed#instance_uptime

[4] https://cloud.google.com/appengine/docs/standard/go/config/appref#scaling_elements

Nikhil Patil

unread,
May 6, 2022, 3:06:58 AMMay 6
to Google App Engine
To Explain my overall situation, My app response is lost though the API program executes as expected. This happens randomly.

here is my app.yaml

#
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# [START django_app]
# [START gaestd_py_django_app_yaml]

## std + dev-apc
#runtime: python39
#service: dev-apc
#instance_class: F1


handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
static_dir: staticfiles/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
script: auto
# [END gaestd_py_django_app_yaml]
# [END django_app]

Rogelio Monter Rodriguez

unread,
May 6, 2022, 6:22:15 PMMay 6
to Google App Engine

Have you tried debugging your app locally using the local development server or using the local development server?

Nikhil Patil

unread,
May 12, 2022, 11:46:04 PMMay 12
to Google App Engine
I tried but it seams 
  • The dev_appserver tool does not support development of Python 3 apps on Windows.

https://cloud.google.com/appengine/docs/standard/python3/tools/local-devserver-command

Osvaldo Lopez Acuña

unread,
May 13, 2022, 2:59:01 PMMay 13
to Google App Engine
It would be useful if you describe what kind of call your frontend is doing. Quota limits[1] could be reached sometimes and then you get this behavior. Appengine location, network configuration, Python, and Django versions would also be helpful to narrow your issue.

In many cases, elevated latency in your application will eventually result in 5xx server errors. Therefore, it makes sense to follow a similar set of troubleshooting steps to narrow down the root cause of both error and latency spikes, given that the causes of each might be the same.

Check Troubleshoot elevated latency in your app [2].


This document [3] answers some App Engine connectivity questions. 


[1]:https://cloud.google.com/appengine/quotas#Requests 

[2]:https://cloud.google.com/appengine/docs/troubleshooter/latency

[3]:https://cloud.google.com/appengine/docs/troubleshooter/connectivity-questions
Reply all
Reply to author
Forward
0 new messages