App Engine old instances still running

[Lukáš Caldr]

Hi all!

We've build few services on Google App Engine, we've ship them into production and everything worked as expected. But now I have little issue with one of my integration service which runs on flexible environment.

I've deployed serveral times new version and maybe some of deploys wasn't successfully finshed or something like that. Now I see in Stack Driver logging some error logs from running instances of some old version (by version id in "resource" property). It made me few problems with with some version "missmatch" in integration tasks. I don't see this old versions in GC CLI SDK or in web dashboard. I've tried to delete whole service and redeploy it, but after first deploy, there are still new log entries in Stack Driver. I've changed DB credentials to prevent older versions to make mess in DB ¯_(ツ)_/¯. Production environment now runs OK, but there are still some old services spamming logging with errors about DB auth. I cant find this old versioned instances anywhere except this new log entries.

Is it normal behaviour? Do you have some tips?

Thank you in advance :)

[Tiago (Google Cloud Platform Support)]

Hi Lukas,

Thank you for your message.

The issue you described is not the expected behavior for App Engine Flex. As such, I've created a private issue in the Issue Tracker [1] to properly follow up as we'll need your project ID in order to further investigate.

Looking forward to hear from you.

[1] https://issuetracker.google.com/131927808

[Dmitry Mazuro]

Hi,
We are also experiencing this issue with App Engine Flex for ruby.
Zombie instances are exhausting our 3rd party api quotas and this also causes serious issues with background job processing.
This is blocking our app release, which is pretty urgent.
Is there something we can do to help with the investigation?

thanks,

- Dmitry

[Sam (Google Cloud Support)]

Hi Dimitry,

A workaround was found for this issue.

The problem was that the "old versioned" instances were still running, so upon deploying a new version of your app, specifying the version ID as the "old version" like so [1]:

gcloud app deploy --project PROJECT_ID --version VERSION_ID

the old version will be moved to "stopping" state as the new version takes on "starting" state with the previously used version ID. Try the following to resolve the issue:

1. Filter version IDs of the "zombie" instances from the logs

2. Then deploy the new version with the version ID of "zombie" instance. 

3. If you have a script for automating deployments, reconfigure it to use fixed "version ID" to prevent same issue in future.

Once deploy, you should notice in the dashboard's "instances" view the result of the recent changes. If this doesn't resolve your issue, then provide more details on your method and description of the behavior in our Public Issues Tracker tool and we will investigate the issue further there [2].

I hope this helps.

[1] specifying version: https://cloud.google.com/sdk/gcloud/reference/app/deploy#--version

[2] public issue tracker tool: https://issuetracker.google.com/