application of migration fails

Francis Fisher

unread,

Dec 12, 2016, 4:45:55 PM12/12/16

to Django users

Any idea if I'm doing something wrong or if I'm running into a bug?

I added some fields to a model, and generated a migration.

When I apply the migration, the migrate command crashes, with both django 1.10.3 and 1.10.4.

I had a similar migration issue with another app in the same project, that occured in django 1.10 and 1.10.3.

Is this crash a known issue/common pitfall?

Cheers,

Frankie

---------------------------------------------

Result of the "makemigrations":

Migrations for 'eveshoppinglist':

evetools/eveshoppinglist/migrations/0009_auto_20161212_2140.py:

- Rename field build_location_type on buildline to build_facility_type

- Rename field from_location_id on moveline to from_facility_id

- Rename field from_type on moveline to from_facility_type

- Rename field to_location_id on moveline to to_facility_id

- Rename field to_type on moveline to to_facility_type

- Add field build_facility_id to buildline

- Add field build_system to buildline

- Add field build_facility_id to buyline

- Add field build_facility_type to buyline

- Add field build_system to buyline

- Add field from_system to moveline

- Add field to_system to moveline

-----------------------------------------------------------------

Result of "migrate"

File "evetools/manage.py", line 10, in <module>

execute_from_command_line(sys.argv)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line

utility.execute()

File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute

self.fetch_command(subcommand).run_from_argv(self.argv)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv

self.execute(*args, **cmd_options)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute

output = self.handle(*args, **options)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 204, in handle

fake_initial=fake_initial,

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 115, in migrate

state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards

state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 244, in apply_migration

state = migration.apply(state, schema_editor)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/migration.py", line 129, in apply

operation.database_forwards(self.app_label, schema_editor, old_state, project_state)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/operations/fields.py", line 282, in database_forwards

to_model._meta.get_field(self.new_name),

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 495, in alter_field

old_db_params, new_db_params, strict)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql/schema.py", line 117, in _alter_field

new_db_params, strict,

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 504, in _alter_field

fk_names = self._constraint_names(model, [old_field.column], foreign_key=True)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 931, in _constraint_names

constraints = self.connection.introspection.get_constraints(cursor, model._meta.db_table)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql/introspection.py", line 178, in get_constraints

"foreign_key": tuple(used_cols[0].split(".", 1)) if kind.lower() == "foreign key" else None,

Tim Graham

unread,

Dec 12, 2016, 5:01:12 PM12/12/16

to Django users

It could be a bug or your database might be in an inconsistent state somehow. The next step would be to debug it a bit more and put together a minimal project that reproduces the issue.

Francis Fisher

unread,

Dec 12, 2016, 6:40:56 PM12/12/16

to Django users

The above migration crash was when applying to a postgres database, I've tried applying this migration to a mysql database and it still crashes, albeit differently.

--------------

File "evetools/manage.py", line 10, in <module>

execute_from_command_line(sys.argv)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line

utility.execute()

File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 359, in execute

self.fetch_command(subcommand).run_from_argv(self.argv)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 294, in run_from_argv

self.execute(*args, **cmd_options)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 345, in execute

output = self.handle(*args, **options)

File "/usr/local/lib/python2.7/dist-packages/django/core/management/commands/migrate.py", line 204, in handle

fake_initial=fake_initial,

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 115, in migrate

state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards

state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/executor.py", line 244, in apply_migration

state = migration.apply(state, schema_editor)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/migration.py", line 129, in apply

operation.database_forwards(self.app_label, schema_editor, old_state, project_state)

File "/usr/local/lib/python2.7/dist-packages/django/db/migrations/operations/fields.py", line 282, in database_forwards

to_model._meta.get_field(self.new_name),

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 495, in alter_field

old_db_params, new_db_params, strict)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 558, in _alter_field

self.execute(self._rename_field_sql(model._meta.db_table, old_field, new_field, new_type))

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/schema.py", line 112, in execute

cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute

return self.cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 94, in __exit__

six.reraise(dj_exc_type, dj_exc_value, traceback)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute

return self.cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 110, in execute

return self.cursor.execute(query, args)

File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in execute

self.errorhandler(self, exc, value)

File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler

raise errorvalue

django.db.utils.OperationalError: (1138, 'Invalid use of NULL value')

Francis Fisher

unread,

Dec 12, 2016, 8:06:41 PM12/12/16

to Django users

I can see why django is crashing at File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql/introspection.py", line 178, in get_constraints

If I run the query at that line in postgres, I get the following response:

constraint_name | column_name | constraint_type | array

-------------------------------------------------------+------------------------+-----------------+-----------------------------------

eveshoppinglist_buildline_shopping_list_id_fkey | shopping_list_id | FOREIGN KEY | {eveshoppinglist_shoppinglist.id}

eveshoppinglist_buildline_item_type_id_fkey | item_type_id | FOREIGN KEY | {invTypes.typeID}

eveshoppinglist_buildline_build_location_type_id_fkey | build_location_type_id | FOREIGN KEY | {invTypes.typeID}

eveshoppinglist_buildline_pkey | id | PRIMARY KEY | {eveshoppinglist_buildline.id}

However, when I fetch the results of that query from python via psycopg2 I get:

['eveshoppinglist_buildline_shopping_list_id_fkey', 'shopping_list_id', 'FOREIGN KEY', ['eveshoppinglist_shoppinglist.id']]

['eveshoppinglist_buildline_item_type_id_fkey', 'item_type_id', 'FOREIGN KEY', []]

['eveshoppinglist_buildline_build_location_type_id_fkey', 'build_location_type_id', 'FOREIGN KEY', []]

['eveshoppinglist_buildline_pkey', 'id', 'PRIMARY KEY', ['eveshoppinglist_buildline.id']]

In the 2nd and 3rd rows, the final value is an empty list, but in both lines should contain 'invTypes.typeID'. Because this result is empty, used_cols[0] raises an exception.

So what I need to figure out is, why this query response fails to transfer to python, and is the problem within psycopg2, or is it a bad query from django. My initial hypothesis is that its because the unmanaged table+column names are not entirely lower case.

For reference, this occurs with psycopg2 versions 2.6.1 and 2.6.2 and django 1.10.3 and 1.10.4.

More anon...

Francis Fisher

unread,

Dec 12, 2016, 8:57:09 PM12/12/16

to django...@googlegroups.com

So it turns out that a different user owned the non-managed tables, which seemed to prevent the django user from accessing certain information about those tables.

Tim Graham

unread,

Dec 12, 2016, 10:19:46 PM12/12/16

to Django users, fra...@terrorise.me.uk

I guess there's nothing Django could do except perhaps throw a more helpful error message. Feel free to submit a ticket and/or patch if you see any value in that and if it seems feasible. I'm not sure about that offhand.

Reply all

Reply to author

Forward