Data migrations ran multiple times during deploy

[Harris Lapiroff]

Hi, all! Curious about how folks do data migrations as we ran into an issue recently:

Our production deployment consists of three containers running the application. When each container comes up it runs a series of startup tasks include `./manage.py migrate`. Usually it's fine to run this whenever because it's idempotent and Postgres should reject simultaneous schema changes.

So I was surprised recently when we performed a deploy that included a data migration that created some objects and it created the objects multiple times. I'm assuming what happened is that the containers each kicked off a migration command simultaneously (thus all three containers saw that the migration hadn't been run yet) and Postgres didn't block any of the transactions as conflicting.

How do you all avoid this?

Thanks!

Harris

[Ryan Nowakowski]

tl;dr it's not safe to run multiple manage.py migrate simultaneously

https://stackoverflow.com/a/69608486

[quentinsf]

Most of our stuff is deployed under Docker Swarm - though it would work with compose or other systems -  and typically we have a separate service, using our standard container image for the app but with the command overridden to run 'manage.py migrate'. The swarm is told to run just one replica of this, and not to restart it.

Whenever we deploy a new version of the image, therefore, this gets run just once.

It does mean that the app can start up and run before the migration, but if it fails as a result it will be restarted after a short delay. In practice this has never been an issue for us.

Quentin

[Madhava Raju Sripathi]

thank you subscribe my channel @VictoriaPalanki9573974636

[Aviv Avitan]

We experience the same problems and those races are probably inevitable

1. django-syzygy (quroum) may solves this issue as it will use a semaphore to prevent any migrations before seeing "X migrates". this will prevent any races. On the same note, it will introduce migrate --pre which should avoid incompatibilities between new application and old database schema

2. two phase deployment: one app container will deploy the new code (but having no traffic going into it), migrate and redirects all new traffic the new app. secondly, upgrade all other app to new code and reinserting to the deployment load balancer.

[Harris Lapiroff]

Thanks, all for the responses! They're helpful, particularly:

• Using a separate single service to run migrations
• django-syzygy looks interesting and might also solve another issue we've been having lately trying to achieve zero-downtime deploys

Best,

Harris