how to do a django data migration on a massive Postgres data table (solved)

[Vernon D. Cole]

Django migrations are run in a single transaction (if your database is capable of doing a rollback of DDL, as PostgreSQL is) so data migrations of large tables become impossible.

I was able to solve this problem by having the migration run a no-wait subprocess which performs the data migration using raw SQL without the django ORM.  In order to keep things together, the subprocess is contained in the same source file as the migration... when run as a module it runs itself as a main program.  In my case, it ran for about two days, converting my 10 million row table 1000 rows at a time. [Actually, I had to restart it three times by pretending to remove it and then re-applying it.]

In case I need to do it some other time, I am putting the code into a GIST.  I am posting here so that others may find it when searching for the right keywords...
https://gist.github.com/9adedbab1899224a4eaf.git

[Larry Martell]

Thanks very much Vern for doing this. I'm sure it will be very useful
to a lot of people.

[aRkadeFR]

Thanks for the email on the django-user ML.

I don't see the gist though, is the problem from me?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/86fda457-03a2-4ef7-9d3f-e540fa0e8c1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
aRkadeFR

[Vernon D. Cole]

Hmm.  Sorry, I think that I sent the download link when I intended the web link.  The trailing ".git" made things bad.
Try https://gist.github.com/vernondcole/9adedbab1899224a4eaf