Push Task Queues in Python Flexible instances are randomly failing

[Eyal Yavor]

During our upgrade/migration off of vm: true environment, we've deployed a service with definitions

runtime: python-compat
env: flexible
threadsafe: true
beta_settings:
  enable_app_engine_apis: true

for 4 instances (manual scaling).

They are all accepting data from network, enqueue tasks to Task Queue, and process tasks from the queue at a rapid pace.

Since we changed to this environment, every few minutes, there's a batch of failed tasks, and after a retry, they all succeed.

Can't see any clear failure in the logs, nor can find logs for specific tasks that failed (log filters that used to work no longer work).

In the logs we see random Nginx warnings: "*452968 a client request body is buffered to a temporary file {path}" - Is it related?

Memory usage is stable, CPU is stable at 30-40%.

It started a few hours after the version was up, while the load hasn't increased or changed dramatically.

We'll migrate to Cloud Tasks once it's more stable - but at the moment we're stuck between Google turning off the ManagedVM and an environment in alpha.

[Kenworth (Google Cloud Platform)]

I noticed that you also posted this in Public Issue Tracker. As it may potentially qualify as a defect report, we will be helping you on the issue tracker created. I recommend community users that experiences the same issue to post their feedback (star it) on the issue tracker so proper weight and attention be given to it. Please feel free to update this thread once a resolution has been reached.