multiple database migrations

[Jeroen Bakker]

Hello all,

first of all, let me say that django-south is a great product!

We use a setup where we have per customer a database using the same database model. Via the django database router we select per request the correct database and set the right username/password. This is working ok, but the simpliest database migration takes at least 10 seconds. This is partly as the database migration is triggered via a bash-script.

for each customer
    set environment variables
    trigger django migrate
        <django loads>
        <django south performs migration>

Currently we have 100k lines of python code, 400 databases, 130 tables per database and around 300 database migrations in the code base. and per customer we see a timespan of at least 10 seconds, meaning our total database migrationtime for simple data migrations is over one hour. Complex migrations can take several hours. Currently we managed this by carefully planning.

The goal for our business is to grow and I would like to reduce the data migration times. And see two options:
1. let a single migration perform several database migrations sequencial. this could be an solution when the loading of django and parsing of the migrations takes the longest time.
2. let the bash script run two migrations in parallel. (ideally would reduce the migration time by 50%)

I do not know the south codebase and would like to have some opinions about the validity of both approaches. Perhaps the solution would be to implement both at the end.

Does anyone have experience with this kind of setup?

Best regards,
Jeroen Bakker
l1nda.nl

[Andrew Godwin]

Hi Jeroen,

Running the migrations in parallel should be the best solution; there's no state storage outside of the database South is connected to, so you can just run all of them in parallel at once if you want to.

Andrew

--
You received this message because you are subscribed to the Google Groups "South Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to south-users...@googlegroups.com.
To post to this group, send email to south...@googlegroups.com.
Visit this group at http://groups.google.com/group/south-users.
For more options, visit https://groups.google.com/d/optout.

[Jeroen Bakker]

Hi all,

just to inform you all that we have adapted the djadmin migrate function to migrate multiple databases in the same run. Eventually this was decided as the risks were smaller and the gains more clear.

Our migration times went back from 2.5 hours to 40 minutes. The code we used is fairly straight forward.

First put all databases in the DATABASES setting of django. South makes a copy of this setting and it needed to be filled at forhand. Reload south.db so the south copy of the setting is reread

Read all migrations (south.migration.all_migrations())

for each database:
    for each application
        check if there are new migrations
        south.migration.migrate_app()

We also added options to select which databases will be migrate --less-than=g --greather-than=d so in the future we can do parrallel processing of the database migrations.

At the end all errors are collected and send to the error emailaddress.

It currently only runs on our specific django application, so I cannot share the code (yet).

Thanks!

Jeroen

--

L1NDA B.V.

www.l1nda.nl

www.facebook.com/L1NDA.NL
Nieuwezijds Voorburgwal 101-III

1012 RG, Amsterdam

020-7526690

KvK: 57861765