Use migrations api to create a command for data migration between different databases

[Bruno Ribeiro da Silva]

Hey guys, I'm not sure if I should be asking this here, but it's related to django internals.

I have this idea to create a command that uses the migrations api to migrate all the data from one database (eg. MySQL) to another database (PostgreSQL).

So I would run the migrations in the new target database, then flush all the tables there, and model by model do a bunch of bulk_create into the new database using the result queryset from the source database.

The questions are: do you think it is feasible to make all of this inside a command? If I have to change the migrations api to make it happen, do you think this could be a nice feature to have in Django?

Thanks in advance!

[Stephen J. Butler]

I usually just use dumpdata/loaddata to do these kinds of things.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1f588512-dc82-4182-a87b-cbdabea71a11%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bruno Ribeiro da Silva]

The problem here is that this works well for small databases, but for bigger ones it uses too much memory and is a slow process.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAD4ANxWFjR0FJimBrk-1WqEHYzw3hc%2Bf949u3DGhKexrmkM%3Deg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Bruno Ribeiro da Silva

Python Dev and Homebrewer!

[Marc Tamlyn]

I'd suggest your approach is also quite slow. Andrew Godwin did something similar once at a lower level - see https://github.com/lanyrd/mysql-postgresql-converter (use at your own risk)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAE-2%3DJxMSXtLsF03OFBrF6CChTJL3AQDnJK_Q_u2AuktjSk5NQ%40mail.gmail.com.

[Shai Berger]

Hi Bruno,

I think that putting such an operation in a migration doesn't make much sense. If it's part of the project migrations, it means that the canonical way to set up the database is to start up on MySQL and move to PG later. This is almost surely not what you intend.

Copying data across is usually a one-time thing, not something you want as part of your living history. A management command fits much better than a migration.

My 2 cents,
Shai

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Bruno Ribeiro da Silva]

I agree with you Shai, that's why I proposed a command that makes use of migrations internals (not to use a migration per se) to do the job.

Thanks for the feedback Marc, I know having the migration to use django orm is a much slower process than importing it from a textual source, but the main objective is the correctness of the data. Using django orm would guarantee that all the tables are created the way django expects it and the data is inserted into the database as well as the way django does it.

My concern is migrate without having to change anything on the models or get an unexpected behavior.

Thanks for all your feedback, I'm gonna try to have some PoC and see if it's doable.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/9B1A79E7-E458-4DA5-BC94-C1838DA4908E%40platonix.com.

For more options, visit https://groups.google.com/d/optout.