Tasks duplicated when deploying new version of Python 3 service

[Ludovic Vaugeois Pepin]

We noticed that when a new version of a "basic" (B4_1G) Python 3.9 service is deployed, the running cloud tasks executing on the hitherto latest version are sometimes (or perhaps always?) started again on the newly deployed version. We noticed that the same task then runs on both the older and the new version until their respective completions.

This may be expected, yet the documentation seems to imply this should now occur in general: https://cloud.google.com/tasks/docs/common-pitfalls#duplicate_execution

The new versions are deployed with "gcloud app deploy --no-stop-previous-version ..."

So my question is: is this normal behavior?

[Alexis (Cloud Platform Support)]

Hi,

You asked if it's expected behavior and the article you linked says that it could happen and it's expected behavior, so "Developers should take steps to ensure that duplicate execution is not a catastrophic event"

I think your question is more about frequency? Please tell us how often duplicates happen.

On a side-note. Tasks are architecturally decoupled from the versions semantics since it's a separate API. It's technically part of the Cloud Tasks API as mentioned here[1]. So this means "gcloud app deploy --no-stop-previous-version" would probably entice more duplication as the state of the code is being scaled-up, the previous configs remain active because they are decoupled. In this case you'd have to ask yourself, what is the state or the condition of the code doing? And how often is it happening. This would happen in any microservice environment that scales and I think it's no2 "Asynchronicity" here[2]. Depending on your needs, you may require to code a completely separate service that is more elaborate than our tasks API. It's one of the pitfalls of having things decoupled.

[1] https://cloud.google.com/tasks/docs/queue-yaml#introduction

[2] https://blog.bernd-ruecker.com/3-common-pitfalls-in-microservice-integration-and-how-to-avoid-them-3f27a442cd07

[Ludovic Vaugeois Pepin]

Hi, thanks for your reply.

This seems to happen every time we deploy a new version while tasks are executing on the previously serving version. We use "gcloud app deploy --no-stop-previous-version" to let these tasks complete on the previous version. Doing this seems to systematically cause Cloud Tasks to start a new "execution" of some (or all) of these tasks on the new version. We clearly see this in the logs, we see two requests with the same task name/id running concurrently on the older and the new version.

Perhaps this is normal, perhaps it is not. Is it?

Many thanks

[Alexis (Cloud Platform Support)]

Hi,

When a new version is spun-up, it should not carry state. However, as mentioned previously, because the Cloud Tasks API is decoupled and you are keeping an older version alive, the new version will grab the current state of the API's queue that is decoupled and a duplicate situation will happen because the API hasn't received acknowledgment that this task in the queue is done. I think this happens particularly because it's just versioning and the way our load balancing works. This is why I don't think that keeping an older version is the best choice and it is expected behavior. However, you could capitalize on forcing them to not complete on the previous version by not using "-no-stop-previous-version" and somehow restart the task when that happens. There is a section here[1] for retrying upon failure.

[1] https://cloud.google.com/tasks/docs/manage-cloud-task-scaling#retries