Corrupted Migration?

[Daniel Chen]

Hi all,

A migration I was running (manage.py migrate) died in the middle. I deleted the migration file, fixed the problem (wrong default value), recreated the migration, and re-ran it. But now, I'm getting an inconsistent state (error message: [Model] has no field named [field]). When I check the database, all the old fields are there. However, when I hop into a Django shell (manage.py shell) and check the model fields ([Model]._meta.fields), they reflect the new, post-migration state. What is going on, and how do I fix this?

Thanks!

[Mike Dewhirst]

Do it again. Delete the migration. Take note of the last valid
migration. Open the django_migrations table and delete the migration
record(s) after the last valid migration.

makemigrations again and then migrate again to bring the database into
line with your models.

BTW, the shell is looking at the model not the database.

Mike

>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com
> <mailto:django-users...@googlegroups.com>.
> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Daniel Chen]

Thanks for the reply. I should have mentioned that I tried all of that: the problem is the model state is ahead of the database state, not behind it. It seems that the database migration was atomic, so everything was rolled back as that transaction failed, but the same thing didn't happen at the model level. So, now I'm stuck with a model state that the previous migrations don't capture.

On Wednesday, September 30, 2015 at 5:08:22 AM UTC-7, Mike Dewhirst wrote:

On 30/09/2015 4:16 PM, Daniel Chen wrote:
> Hi all,
>
> A migration I was running (manage.py migrate) died in the middle. I
> deleted the migration file, fixed the problem (wrong default value),
> recreated the migration, and re-ran it. But now, I'm getting an
> inconsistent state (error message: [Model] has no field named [field]).
> When I check the database, all the old fields are there. However, when I
> hop into a Django shell (manage.py shell) and check the model fields
> ([Model]._meta.fields), they reflect the new, post-migration state. What
> is going on, and how do I fix this?

Do it again. Delete the migration. Take note of the last valid
migration. Open the django_migrations table and delete the migration
record(s) after the last valid migration.

makemigrations again and then migrate again to bring the database into
line with your models.

BTW, the shell is looking at the model not the database.

Mike

>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users+unsub...@googlegroups.com>.

[Mike Dewhirst]

On 1/10/2015 6:51 AM, Daniel Chen wrote:
> Thanks for the reply. I should have mentioned that I tried all of that:
> the problem is the model state is ahead of the database state, not
> behind it. It seems that the database migration was atomic, so
> everything was rolled back as that transaction failed, but the same
> thing didn't happen at the model level. So, now I'm stuck with a model
> state that the previous migrations don't capture.

The model is the master. It is outside the transaction and will never be
touched by migrations.

Your task is to identify which previous migration matches the current
database structure and delete all migrations after that point. That
means deleting the migration files after the one you identified AND
deleting the migration records in django_migrations in the database.

Thereafter, the next migration will pick up all the changes in the model
and bring the database back into alignment.

This is all much the same as I said last time so if this doesn't work it
means I haven't understood your problem.

> > an email to django-users...@googlegroups.com <javascript:>
> > <mailto:django-users...@googlegroups.com <javascript:>>.

> > To post to this group, send email to django...@googlegroups.com

> <javascript:>
> > <mailto:django...@googlegroups.com <javascript:>>.

> > Visit this group at http://groups.google.com/group/django-users

> <http://groups.google.com/group/django-users>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users...@googlegroups.com>.

> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Daniel Chen]

I have tried that (by the way, the offending migration didn't show up in the django_migrations table because the migration failed).

I think we might be talking past each other. Sorry if I'm being unclear. Here's an example of what's going on:

1. Before migration, my database is at state X, and my migrations files reflect that.
   
2. I change my model file, updating the state (let's call it state Y). Then, I run make migrations, generating a new migration file that corresponds to state Y.
   
3. I run the migrate command to get my database to state Y. It fails, so now my database is back at state X. There is nothing in the database to delete (not even from the django_migrations table) because the transaction was rolled back.
   
4. I delete the migration file that corresponds to state Y. My database is back in state X, and my migration files again reflect that.
   
5. I run the migrate command again, and it fails (same problem). The error is django.db.models.fields.FieldDoesNotExist: [Model] has no field named [field].
   
6. [field] corresponds to a new field in state Y, not in an existing field in state X. I'm confused - why is it looking for a field that doesn't exist in the pre-migrated state (state X)?
   
7. So, I assume that the model state somehow is de-synced from the database/migration state. I hop into the Django shell and check the model metadata. The fields exactly match state Y. My database is still in state X.

Does that make sense?

>      > <mailto:django-users+unsub...@googlegroups.com <javascript:>>.

>      > To post to this group, send email to django...@googlegroups.com
>     <javascript:>
>      > <mailto:django...@googlegroups.com <javascript:>>.
>      > Visit this group at http://groups.google.com/group/django-users
>     <http://groups.google.com/group/django-users>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com
>     <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users+unsub...@googlegroups.com>.

[Mike Dewhirst]

On 1/10/2015 1:36 PM, Daniel Chen wrote:
> I have tried that (by the way, the offending migration didn't show up in
> the django_migrations table because the migration failed).
>
> I think we might be talking past each other. Sorry if I'm being unclear.
> Here's an example of what's going on:

You haven't mentioned the Python/Django version. Does ticket
https://code.djangoproject.com/ticket/23226 apply?

>
> 1. Before migration, my database is at state X, and my migrations files
> reflect that.
> 2. I change my model file, updating the state (let's call it state Y).

> Then, I run make migrations, generating a new migration file that
> corresponds to state Y.

> 3. I run the migrate command to get my database to state Y. It fails,

> so now my database is back at state X. There is nothing in the
> database to delete (not even from the django_migrations table)
> because the transaction was rolled back.

> 4. I delete the migration file that corresponds to state Y. My database

> is back in state X, and my migration files again reflect that.

> 5. I run the migrate command again, and it fails (same problem). The

> error is django.db.models.fields.FieldDoesNotExist: [Model] has no
> field named [field].

> 6. [field] corresponds to a new field in state Y, not in an existing

> field in state X. I'm confused - why is it looking for a field that
> doesn't exist in the pre-migrated state (state X)?

> 7. So, I assume that the model state somehow is de-synced from the

> > Â Â On 30/09/2015 4:16 PM, Daniel Chen wrote:
> > Â Â Â > Hi all,
> > Â Â Â >
> > Â Â Â > A migration I was running (manage.py migrate) died in
> the middle. I
> > Â Â Â > deleted the migration file, fixed the problem (wrong
> default value),
> > Â Â Â > recreated the migration, and re-ran it. But now, I'm
> getting an
> > Â Â Â > inconsistent state (error message: [Model] has no field
> named
> > Â Â [field]).
> > Â Â Â > When I check the database, all the old fields are
> there. However,
> > Â Â when I
> > Â Â Â > hop into a Django shell (manage.py shell) and check the
> model fields
> > Â Â Â > ([Model]._meta.fields), they reflect the new,
> post-migration
> > Â Â state. What
> > Â Â Â > is going on, and how do I fix this?
> >
> > Â Â Do it again. Delete the migration. Take note of the last valid
> > Â Â migration. Open the django_migrations table and delete the
> migration
> > Â Â record(s) after the last valid migration.
> >
> > Â Â makemigrations again and then migrate again to bring the
> database into
> > Â Â line with your models.
> >
> > Â Â BTW, the shell is looking at the model not the database.
> >
> > Â Â Mike
> >
> > Â Â Â >
> > Â Â Â > Thanks!
> > Â Â Â >
> > Â Â Â > --
> > Â Â Â > You received this message because you are subscribed to
> the Google
> > Â Â Â > Groups "Django users" group.
> > Â Â Â > To unsubscribe from this group and stop receiving
> emails from it,
> > Â Â send
> > Â Â Â > an email to django-users...@googlegroups.com <javascript:>
> > Â Â Â > <mailto:django-users...@googlegroups.com
> <javascript:> <javascript:>>.
> > Â Â Â > To post to this group, send email to
> django...@googlegroups.com
> > Â Â <javascript:>
> > Â Â Â > <mailto:django...@googlegroups.com <javascript:>>.
> > Â Â Â > Visit this group at
> http://groups.google.com/group/django-users
> <http://groups.google.com/group/django-users>
> > Â Â <http://groups.google.com/group/django-users
> <http://groups.google.com/group/django-users>>.
> > Â Â Â > To view this discussion on the web visit
> > Â Â Â >
> > Â Â
> https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com>
>
> > Â Â
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com>>
>
> >
> > Â Â Â >
> > Â Â
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer>
>
> > Â Â

> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/django-users/95f33338-c53b-497d-8dcd-a4c1e444b5c4%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> >

> > Â Â Â > For more options, visit
> https://groups.google.com/d/optout <https://groups.google.com/d/optout>
> > Â Â <https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to django-users...@googlegroups.com <javascript:>
> > <mailto:django-users...@googlegroups.com <javascript:>>.

> > To post to this group, send email to django...@googlegroups.com
> <javascript:>
> > <mailto:django...@googlegroups.com <javascript:>>.
> > Visit this group at http://groups.google.com/group/django-users
> <http://groups.google.com/group/django-users>.
> > To view this discussion on the web visit
> >

> https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/django-users/559fd3eb-36e4-4ace-a270-3caf1e7fd874%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>

> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users...@googlegroups.com>.

> To post to this group, send email to django...@googlegroups.com
> <mailto:django...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/0ac300f7-0354-4716-bd9b-5552940ae26a%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/0ac300f7-0354-4716-bd9b-5552940ae26a%40googlegroups.com?utm_medium=email&utm_source=footer>.

[James Schneider]

> I have tried that (by the way, the offending migration didn't show up in the django_migrations table because the migration failed).
>
> I think we might be talking past each other. Sorry if I'm being unclear. Here's an example of what's going on:
> Before migration, my database is at state X, and my migrations files reflect that.
> I change my model file, updating the state (let's call it state Y). Then, I run make migrations, generating a new migration file that corresponds to state Y.
> I run the migrate command to get my database to state Y. It fails, so now my database is back at state X. There is nothing in the database to delete (not even from the django_migrations table) because the transaction was rolled back.
> I delete the migration file that corresponds to state Y. My database is back in state X, and my migration files again reflect that.
> I run the migrate command again, and it fails (same problem). The error is django.db.models.fields.FieldDoesNotExist: [Model] has no field named [field].

After you delete Y migration, have you searched your migration files to verify that there are no references to that field?

> [field] corresponds to a new field in state Y, not in an existing field in state X. I'm confused - why is it looking for a field that doesn't exist in the pre-migrated state (state X)?

Do you have any RunPython or other custom modifications in your migrations that would cause an object based on your modified model to be created (and saved), possibly relying on a default value for the new 'field' specified in your Y-version model, maybe as part of a data migration?

> So, I assume that the model state somehow is de-synced from the database/migration state. I hop into the Django shell and check the model metadata. The fields exactly match state Y. My database is still in state X.
> Does that make sense?

Makes perfect sense when you think about it. Being out of sync in this manner happens constantly, and is exactly why migrations exist.

Django makes the assumption that the current values in your models.py files exactly match the layout of the DB, since you should be syncing everything using migrations and validating via tests.

Any time you modify a model, the DB and you model definitions become out of sync, with your models being 'ahead' of your DB since you haven't informed the DB of your model changes via migrations yet.

Think of it like someone writing you a check. Your relative personal wealth increases as soon as you receive the check, but your bank statement won't agree with you until you submit the check to them so that they can credit your account accordingly.

Feel free to write me a check (for a large amount, bigger the better) so I can demonstrate the effect it has on my bank account. ;-)

-James

[Daniel Chen]

@Mike: I'm using Python 2.7.10 and Django 1.7. I'm not sure the ticket applies, but I appreciate the link!

@James: Sorry, I misspoke! [field] actually corresponds to a old field in state X. Before adding the new fields in state Y, the migration is trying to remove some old fields in state X but isn't finding them. I think this means that I'm unable to re-run that migration since it's looking for state X fields to remove, but can't find them (the model is already in state Y). My confusion is in the fact that the model state wouldn't roll back to state X after the first failed transaction (like the DB did).

This would be analogous to my writing you a check that will bounce. Once the check bounces, both your personal wealth and your bank statement should be back to what they were originally. But in this particular case, it looks like your personal wealth is stuck in a state that assumes the check has been cashed and your account has been credited, but your bank account disagrees :)

By the way, I talk about model state under the assumption that the Django ORM somehow internally keeps track of the models - I'm not sure exactly how it does this or if it does this at all. If you could tell me more about this, that'd be much appreciated!

[Mike Dewhirst]

On 3/10/2015 8:35 AM, Daniel Chen wrote:
> My confusion is in the fact that the model state wouldn't roll back to
> state X after the first failed transaction (like the DB did).

No. The model is all your own work. Anything you write stays written.
The migration system tries to implement your work. If it fails, it will
roll the database back to its previous state. It does nothing to your
work. You need to take note of the failure, decide how to fix your work
and try again.

If you eliminate the error in your work (assuming no bugs in the
framework) the migration will work and advance the database state to
match the model state.

Mike

[James Schneider]

On Fri, Oct 2, 2015 at 3:37 PM, Daniel Chen <dan...@a16z.com> wrote:

@Mike: I'm using Python 2.7.10 and Django 1.7. I'm not sure the ticket applies, but I appreciate the link!

@James: Sorry, I misspoke! [field] actually corresponds to a old field in state X. Before adding the new fields in state Y, the migration is trying to remove some old fields in state X but isn't finding them. I think this means that I'm unable to re-run that migration since it's looking for state X fields to remove, but can't find them (the model is already in state Y). My confusion is in the fact that the model state wouldn't roll back to state X after the first failed transaction (like the DB did).

That makes even less sense than your original issue. Are you sure that migration X was applied properly? What versions of migrations are stored in your django_migrations table? Is the latest migration listed in there for your app? You should be able to find it with a raw SQL statement, something like "select * from django_migrations where app = '<app_name>';".

Does [field] currently exist in your database so that it can be removed? If not, you've had issues with migrations beyond this particular i

No. The model is all your own work. Anything you write stays written. The migration system tries to implement your work. If it fails, it will roll the database back to its previous state. It does nothing to your work. You need to take note of the failure, decide how to fix your work and try again.

ssue. Were any migration files removed? Does the database match migration X?

Even better, can you also post the full traceback that you receive during the failed migrations? That might lead to more clues.

 

This would be analogous to my writing you a check that will bounce. Once the check bounces, both your personal wealth and your bank statement should be back to what they were originally. But in this particular case, it looks like your personal wealth is stuck in a state that assumes the check has been cashed and your account has been credited, but your bank account disagrees :)

Oh well, worth a shot. :-D

 

By the way, I talk about model state under the assumption that the Django ORM somehow internally keeps track of the models - I'm not sure exactly how it does this or if it does this at all. If you could tell me more about this, that'd be much appreciated!

The ORM will use whatever model definitions contained within the models.py, regardless of database state. It has no idea whether or not the equivalent fields exist in the database (until an exception is thrown when the DB returns an error accessing a missing field, etc.). Migrations are a process completely outside and separate of Django running as an application via runserver or a WSGI process. Migration files are the only files that would keep versions of your models, and are not consulted at any time except during migrations. So if you change anything in models.py, you should create a new migration and apply it before doing any further development.

-James

[James Schneider]

@James: Sorry, I misspoke! [field] actually corresponds to a old field in state X. Before adding the new fields in state Y, the migration is trying to remove some old fields in state X but isn't finding them. I think this means that I'm unable to re-run that migration since it's looking for state X fields to remove, but can't find them (the model is already in state Y). My confusion is in the fact that the model state wouldn't roll back to state X after the first failed transaction (like the DB did).

That makes even less sense than your original issue. Are you sure that migration X was applied properly? What versions of migrations are stored in your django_migrations table? Is the latest migration listed in there for your app? You should be able to find it with a raw SQL statement, something like "select * from django_migrations where app = '<app_name>';".

Does [field] currently exist in your database so that it can be removed? If not, you've had issues with migrations beyond this particular i

No. The model is all your own work. Anything you write stays written. The migration system tries to implement your work. If it fails, it will roll the database back to its previous state. It does nothing to your work. You need to take note of the failure, decide how to fix your work and try again.

ssue. Were any migration files removed? Does the database match migration X?

Even better, can you also post the full traceback that you receive during the failed migrations? That might lead to more clues.

Whoops, looks like part of my response got wrapped around Mike's by accident. Here's what it was supposed to say:

"That makes even less sense than your original issue. Are you sure that migration X was applied properly? What versions of migrations are stored in your django_migrations table? Is the latest migration listed in there for your app? You should be able to find it with a raw SQL statement, something like "select * from django_migrations where app = '<app_name>';".

Does [field] currently exist in your database so that it can be removed? If not, you've had issues with migrations beyond this particular issue. Were any migration files removed? Does the database match migration X?

Even better, can you also post the full traceback that you receive during the failed migrations? That might lead to more clues."

-James

[Daniel Chen]

Hi all,

Thanks for all the help! Sorry for the incredibly slow response, but I just wanted to give an update:

The problem was that I was trying to remove a foreign key (let's call that foreign key "book", referencing a "book" table). I had to manually go into the migration and add '_id' (e.g., "book_id") in every reference to that model (odd that you need to manually update the migrations, but whatever). Afterwards, I was able to successfully dump the DML - which looked more or less correct.

However, the next problem was that I had unique_together constraints on that foreign key and another existing key (e.g., unique_together("book_id", "author_id")). I got the corresponding error message: "ValueError: Found wrong number (0) of constraints for main_tablename(book_id, author_id)", so I still wasn't able to complete the migration.

I worked around this by not dropping that column ("book_id" is still in the table), and the migration finally worked. However, I'd love to get rid of that column once and for all. So, if anyone knows a solution, please let me know!

Dan

[Daniel Chen]

Another update:

The actual problem was the automatically generated migrations from the makemigrations command removed the database fields before removing the constraints on those fields (leading to a FieldDoesNotExist error when trying to drop the constraint). The solution was just to move the RemoveField operation below the AlterUniqueTogether operation in the migration file. (Also, it turns out that removing the "_id" suffix didn't matter after all).

Thanks again for the help, everyone!