Data Migration: orm.Model.objects.all() not working

Eduardo Klein

unread,

Feb 2, 2012, 7:44:18 AM2/2/12

to South Users

Hi,

I started using South and my first migration (not lucky) was to
transform an abstract model into an concrete model.

Here are my initial Models (before schemamigration)

class UserProfile(models.Model):
user = models.OneToOneField(User, primary_key=True, \
related_name='profile')
class Meta:
abstract=True
class SpecificProfile(UserProfile):
url = models.URLField()

After migration, the models look like this:

class UserProfile(models.Model):
user = models.OneToOneField(User, primary_key=True, \
related_name='profile')
class SpecificProfile(UserProfile):
user_profile = models.OneToOneField(UserProfile, parent_link=True)
url = models.URLField()

To migrate, I used the following:

class Migration(SchemaMigration):

def forwards(self, orm):

# Renaming field 'BloggerProfile.user'
db.rename_column('specificprofile', 'user_id',
'user_profile_id') #@UndefinedVariable
db.alter_column('specificprofile', 'user_profile_id',
models.ForeignKey(orm['myapp.UserProfile']), explicit_name=True)

# Adding model 'UserProfile'
db.create_table('myapp_userprofile', ( #@UndefinedVariable
('user',
self.gf('django.db.models.fields.related.OneToOneField')
(related_name='profile', unique=True, primary_key=True,
to=orm['auth.User'])),
))
db.send_create_signal('account', ['UserProfile'])

The schema migration works as expected.

Now, in the data migration process, I would like to create one
UserProfile entry per SpecificProfile and assign UserProfile.user_id
to SpecificProfile.user_profile_id.

So, my data migration forwards is:

class Migration(DataMigration):

def forwards(self, orm):
for spec in orm.SpecificProfile.objects.all():
user_profile = orm.UserProfile()
user_profile.user_id = spec.user_profile_id
user_profile.save()

This seems correct for me, but it does not create any new entry for
the UserProfile model, which is strange.
I used pdb to debug the issue and I verified that
orm.SpecificProfile.objects.all() is returning an empty list, BUT
orm.SpecificProfile.objects.count() returns 8 and
orm.SpecificProfile.objects.values_list('user_profile_id') returns the
8 different values I have in the db.

How is this possible?

Shai Berger

unread,

Feb 26, 2012, 9:51:47 AM2/26/12

to south...@googlegroups.com

Hi Eduardo,

I'm sorry that it took so long for you to get a reply... I hope you have
solved the issue already. If so, it would be nice if you could comment. If
not, I hope this is still relevant:

This is quite surprising: The code seems to create a foreign key from
SpecificProfile to UserProfile, before creating the UserProfile table. The fact
that it seemed to work is probably a bug; it seems to be triggered by creating
a FK to an abstract class ("orm['myapp.UserProfile']" is the UserProfile class
before the migration -- the abstract version).

> Now, in the data migration process, I would like to create one
> UserProfile entry per SpecificProfile and assign UserProfile.user_id
> to SpecificProfile.user_profile_id.
>
> So, my data migration forwards is:
>
> class Migration(DataMigration):
>
> def forwards(self, orm):
> for spec in orm.SpecificProfile.objects.all():
> user_profile = orm.UserProfile()
> user_profile.user_id = spec.user_profile_id
> user_profile.save()
>
> This seems correct for me, but it does not create any new entry for
> the UserProfile model, which is strange.
> I used pdb to debug the issue and I verified that
> orm.SpecificProfile.objects.all() is returning an empty list, BUT
> orm.SpecificProfile.objects.count() returns 8 and
> orm.SpecificProfile.objects.values_list('user_profile_id') returns the
> 8 different values I have in the db.
>
> How is this possible?

The Data Migration uses the definitions after the Schema Migration; according
to these, the database records for SpecificProfile objects are made by joining
the SpecificProfile table with the UserProfile table -- and this, indeed, should
return no records; the result from count() and values_list(), apparently, use
only the SpecificProfile table. This, in itself, is *not* a bug -- these would
be correct optimizations, if the FK constraint would be enforced. In the
situation you describe, it is clearly violated -- the FK probably doesn't
exist in the database.

The right way to do this is:

1) A Schema migration which only adds the UserProfile table, without modifying
the SpecificProfile table
2) A Data migration which creates the records in UserProfile
3) A Schema migration to fix the SpecificProfile table and its FK.

There is a little problem with this: As far as I can see, there's no way to
generate the Schema migrations automatically; you have to edit the migrations
manually. It's unfortunate that you ran into this limitation of the automatic
process on your very first migration, but that's life.

Hope this helps (anybody),

Shai.

Eduardo S. Klein

unread,

Feb 27, 2012, 7:14:56 AM2/27/12

to south...@googlegroups.com

Hi Shai,

Thanks for your reply and explanations.
To work around this problem, I iterated over the values returned by
"values_list" like this:

for (_id,) in orm.SpecificProfile.objects.values_list('user_profile_id'):
spec_profile = orm.SpecificProfile.objects.get(pk=_id)
#manipulations as described before

This worked fine.
As you said, it's unfortunate that it happened in the 1st time I was
using south, but that's ok. I've been using south since then without any
problems.