GAE-Flex environment - 502 error when latency grows too high?

[jy...@blitzoo.com]

I've noticed that I am getting a lot of 502 error as the latency of my service starts to grow. The number of 502 error basically grows with the number of requests sent, and it seems like I can only process very small number of events on the GAE.

The service is written in Python, and is performing data ETL of my app events that is then loaded into Google BigQuery. 

I've tried changing the instance class or increasing the maximum number of instances but I thing there is some other problem with my setup as we are still in development and shouldn't really need that much computing power.

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT standard-event-service:app
threadsafe: true

runtime_config:
  python_version: 3

service: standard-event-service

instance_class: F1
automatic_scaling:
  min_num_instances: 1
  max_num_instances: 10
  max_idle_instances: automatic  # default value
  max_concurrent_requests: 20

[Kenworth (Google Cloud Platform)]

Latency can be caused by a number of things - geography, startup time, size of payload, etc. You can visit this blog on How to Troubleshoot Latency in your App Engine Application and hopefully might help you isolate the latency problem.

It is also helpful to observe how long request takes to respond through Stackdriver Trace. If this is not possible because most of the requests timeout, one should try to make split up or defer the work being done to respond to the request. 

If you are still affected after trying the above, I recommend you post on StackOverflow which we also monitor. You'll be in touch with a much larger user-base of people who are going to help you, in a format which is designed for that purpose. Check out our community support page for the list of tags we monitor.

[jy...@blitzoo.com]

Thanks for the reply.

It seems like the VM health check is contribution to the high ratio of 502 server errors. I've change the liveness_check and readiness_check in the yaml file and it appears to have improved the GAE stability. I will run a few stress tests and see the 502 error comes back.