Error converting an app to South

[Jose]

hey guys, got a question, not sure what I'm doing wrong. Dug through
the documentation about converting an app to south and followed the
following steps:

1. put south in installed apps
2. ran syncdb
3. ran ./manage.py convert_to_south
4. made a change in my models.py
5. ran ./manage.py startmigration myapp name_of_migration --auto
6. checked the migration script to make sure it's ok (it is)
7. ran ./manage.py migrate myapp

At this point it fails with error: OperationalError: (1060, "Duplicate
column name 'reference_key'"), where reference_key is the column name
in the changed models.py

In another terminal window I'm confirming that there is no column
named 'reference_key' in my database, so not sure where that error
would come from.

anything in those steps in error? If not, any ideas why I would get
this error?

[Andrew Godwin]

I've already replied to something very similar to this on IRC, so it was
probably you. For list posterity, it's logged at
http://www.aeracode.org/logs/django-south/%23django-south_20091130.log, line
38.

Andrew

> --
>
> You received this message because you are subscribed to the Google Groups
> "South Users" group. To post to this group, send email to
> south...@googlegroups.com. To unsubscribe from this group, send email
> to south-users...@googlegroups.com. For more options, visit this
> group at http://groups.google.com/group/south-users?hl=en.
>

[Jose]

Yep that was me. Thought I'd also post on the mailing list so that a
resolution would be documented here. Also wanted to see if anyone else
ran into this issue.

There seems to be an inconsistency on what South thinks the database
looks like versus the actual database schema. I'm not sure what
mechanism South is using to make decisions as to the database so not
sure where to start trailing the bug.

The error in essence is: South thinks changes I want it to migrate
have already been made. They are made in my models.py, a migration is
generated in South via --auto, ./manage.py migrate <<app>> is run,
then the errors tell me either the column already exists (for add
column operations) or that it can't find the column (for delete column
operations).

I've also tried to go step-by-step instead of using convert_to_south,
but I get the same errors.

Any help is appreciated. The annotated command line stdout of the
procedure and error is below

J

###### our database has data, but no South. We add 'south' to
INSTALLED_APPS then run:
ubuntu$ ./manage.py syncdb
Syncing...
Creating table south_migrationhistory

Synced:
> django.contrib.auth
> django.contrib.contenttypes
> django.contrib.sessions
> django.contrib.sites
> beans.login
> django.contrib.admin
> beans.markers
> south

Not synced (use migrations):
-
(use ./manage.py migrate to migrate these)



###### Now we run the convenience function 'convert_to_south' to get
us up to speed
ubuntu$ ./manage.py convert_to_south markers
Creating migrations directory at '/var/www/beans/../beans/markers/
migrations'...
Creating __init__.py in '/var/www/beans/../beans/markers/
migrations'...
+ Added model 'markers.Marker'
+ Added model 'markers.LinkageGroup'
+ Added model 'markers.URLReference'
+ Added model 'markers.GBrowse'
Created 0001_initial.py.
- Soft matched migration 0001 to 0001_initial.
Running migrations for markers:
- Migrating forwards to 0001_initial.
> markers: 0001_initial
(faked)

App 'markers' converted. Note that South assumed the application's
models matched the database
(i.e. you haven't changed it since last syncdb); if you have, you
should delete the markers/migrations
directory, revert models.py so it matches the database, and try again.



###### the model and the DB do match, so that's ok so far. Here we see
that the migration is in the db as normal
mysql> select * from south_migrationhistory;
+----+----------+--------------+---------------------+
| id | app_name | migration | applied |
+----+----------+--------------+---------------------+
| 1 | markers | 0001_initial | 2009-12-01 20:53:57 |
+----+----------+--------------+---------------------+
1 row in set (0.00 sec)



###### I now make a change to my markers/models.py, adding "ref_id =
m.IntegerField()", and save it. There are no other changes.
ubuntu$ ./manage.py startmigration markers add_ref_id --auto
+ Added field 'markers.marker.ref_id'
Created 0002_add_ref_id.py.



###### Just to be safe, let's look at the describe for our table in
question
mysql> describe markers_marker;
+------------------+--------------+------+-----+---------
+----------------+
| Field | Type | Null | Key | Default |
Extra |
+------------------+--------------+------+-----+---------
+----------------+
... bunch of columns, none of which is 'ref_id' ...
+------------------+--------------+------+-----+---------
+----------------+



###### No ref_id column, great! Now we try to migrate our markers app,
hitting this error
ubuntu$ ./manage.py migrate markers
Running migrations for markers:
- Migrating forwards to 0002_add_ref_id.
> markers: 0002_add_ref_id
Traceback (most recent call last):
File "/usr/local/lib/python2.6/site-packages/south/migration.py",
line 329, in run_migrations
runfunc(orm)
File "/var/www/beans/../beans/markers/migrations/
0002_add_ref_id.py", line 11, in forwards
db.add_column('markers_marker', 'ref_id', orm
['markers.marker:ref_id'])
File "/usr/local/lib/python2.6/site-packages/south/db/generic.py",
line 227, in add_column
self.execute(sql)
File "/usr/local/lib/python2.6/site-packages/south/db/generic.py",
line 86, in execute
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/
util.py", line 19, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/
mysql/base.py", line 84, in execute
return self.cursor.execute(query, args)
File "/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.2-
py2.6-linux-x86_64.egg/MySQLdb/cursors.py", line 166, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.2-
py2.6-linux-x86_64.egg/MySQLdb/connections.py", line 35, in
defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1060, "Duplicate column name 'ref_id'")
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had to
! leave it in an interim state between migrations.

! You *might* be able to recover with:

! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS.

On Dec 1, 5:37 am, Andrew Godwin <and...@aeracode.org> wrote:
> I've already replied to something very similar to this on IRC, so it was

> probably you. For list posterity, it's logged athttp://www.aeracode.org/logs/django-south/%23django-south_20091130.log, line

[Andrew Godwin]

Jose wrote:
> Yep that was me. Thought I'd also post on the mailing list so that a
> resolution would be documented here. Also wanted to see if anyone else
> ran into this issue.
>
> There seems to be an inconsistency on what South thinks the database
> looks like versus the actual database schema. I'm not sure what
> mechanism South is using to make decisions as to the database so not
> sure where to start trailing the bug.
>

South never looks at the database itself; all decisions about what to do
are based on the frozen models (and thus your Django models). If you
follow the South contract - to do all modifications using South, or tell
it about them - this works well. Otherwise, you need to check your
models match by hand, and then issue a new non-auto migration.

> The error in essence is: South thinks changes I want it to migrate
> have already been made. They are made in my models.py, a migration is
> generated in South via --auto, ./manage.py migrate <<app>> is run,
> then the errors tell me either the column already exists (for add
> column operations) or that it can't find the column (for delete column
> operations).
>
> I've also tried to go step-by-step instead of using convert_to_south,
> but I get the same errors.
>
> Any help is appreciated. The annotated command line stdout of the
> procedure and error is below

Looking through this, it almost looks like dry_run isn't working right
and the migration's being run twice. Did the new column actually get
created after all this? Also, it would be good to know the exact South
release/commit ID (if you run off trunk) you're running off of, and the
same for Django.

Andrew

[Jose]

Hey Andrew,

On Dec 2, 6:00 am, Andrew Godwin <and...@aeracode.org> wrote:
> South never looks at the database itself; all decisions about what to do
> are based on the frozen models (and thus your Django models). If you
> follow the South contract - to do all modifications using South, or tell
> it about them - this works well. Otherwise, you need to check your
> models match by hand, and then issue a new non-auto migration.
>

This is what I thought was the case. I'm using South for the
migrations i.e. making a change in my models.py, then having South
create a migration, then changing the database via South.

> Looking through this, it almost looks like dry_run isn't working right
> and the migration's being run twice. Did the new column actually get
> created after all this? Also, it would be good to know the exact South
> release/commit ID (if you run off trunk) you're running off of, and the
> same for Django.

I'm using the South version downloaded from tarball (0.6.2), Django
1.1.0 final, Python 2.6.1, and MySQL 5.0.5 via ubuntu as the DB

It looks like the operations were run then rolled back. You can't tell
when you add a column, but when I went to remove two columns via
South, I saw the description of the table change, leading me to
believe that the action was run successfully, some error happened,
then it ran the backwards() method. Here's the procedure for this:

###### I follow the steps outlined above, but instead of adding a
column, I went on to create a migration which removed two columns:
ubuntu$ ./manage.py startmigration markers remove_features --auto
- Deleted field 'markers.marker.feature_stop'
- Deleted field 'markers.marker.feature_start'
Created 0002_remove_features.py.



###### Here's what the table looks like before I run the migration

mysql> describe markers_marker;
+------------------+--------------+------+-----+---------
+----------------+
| Field | Type | Null | Key | Default |
Extra |
+------------------+--------------+------+-----+---------
+----------------+

..... rows above
| feature_start | int(11) | YES | | NULL
| |
| feature_stop | int(11) | YES | | NULL
| |
..... rows below
+------------------+--------------+------+-----+---------
+----------------+
19 rows in set (0.00 sec)



###### I run the migration and receive the expected error

ubuntu$ ./manage.py migrate markers
Running migrations for markers:

- Migrating forwards to 0002_remove_features.
> markers: 0002_remove_features

Traceback (most recent call last):
File "/usr/local/lib/python2.6/site-packages/south/migration.py",
line 329, in run_migrations
runfunc(orm)
File "/var/www/beans/../beans/markers/migrations/

0002_remove_features.py", line 11, in forwards
db.delete_column('markers_marker', 'feature_stop')
File "/usr/local/lib/python2.6/site-packages/south/db/mysql.py",
line 91, in delete_column
super(DatabaseOperations, self).delete_column(table_name, name)
File "/usr/local/lib/python2.6/site-packages/south/db/generic.py",
line 578, in delete_column
self.execute(self.delete_column_string % params, [])
File "/usr/local/lib/python2.6/site-packages/south/db/generic.py",

line 86, in execute
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/
util.py", line 19, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.6/site-packages/django/db/backends/
mysql/base.py", line 84, in execute
return self.cursor.execute(query, args)

File "/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.3c1-
py2.6-linux-x86_64.egg/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.3c1-
py2.6-linux-x86_64.egg/MySQLdb/connections.py", line 36, in
defaulterrorhandler
raise errorclass, errorvalue
OperationalError: (1091, "Can't DROP 'feature_stop'; check that column/
key exists")

! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had to
! leave it in an interim state between migrations.

! You *might* be able to recover with:

! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS.

###### But when I describe the table, I note the column definition has
changed. It's also at the end of the table.

mysql> describe markers_marker;
+------------------+--------------+------+-----+---------
+----------------+
| Field | Type | Null | Key | Default |
Extra |
+------------------+--------------+------+-----+---------
+----------------+

.... rows above
| feature_stop | int(11) | NO | | NULL
| |
| feature_start | int(11) | NO | | NULL
| |
+------------------+--------------+------+-----+---------
+----------------+
19 rows in set (0.00 sec)

[Andrew Godwin]

That's _very_ odd. Have you tried running with --verbosity=2 to see the
actual SQL it's running?

Andrew

[Jose]

Hey Andrew,

Sorry for the late reply, I was pretty sick over a week and have just
now returned to work.

I ran manage.py migrate -v 2 and got no output. I also ran it with --
db-dry-run and it didn't do a dry run, it actually changed the
database! Very odd.

Does this info help any?

J

[Andrew Godwin]

It's certainly not what I would expect - it seems like your commandline
arguments are getting ignored, or mixed up, or something.

I'm not sure I can help; your configuration is pretty similar to my
MySQL testing install. The only things I can think of are to check that
"./manage.py shell; import south; south.__file__" points to the right
place, and try clearing the .pyc files.

Andrew