Data migration fails because contenttypes not yet ready

[Torsten Bronger]

Hallöchen!

I want to implement an initial data migration (basically, the same
thing initial_data.json used to do). The problem is that my initial
data must be connected with ContentType instances. However,
update_all_contenttypes() hasn't been called at this stage, so my
migration fails.

As far as I can see, update_all_contenttypes is idempotent. Can I
safely call update_all_contenttypes during the data migration? I
wonder whether the models may be available in an older version, and
whether this may confuse update_all_contenttypes.

That said, it *seems* to work ...

Tschö,
Torsten.

--
Torsten Bronger Jabber ID: torsten...@jabber.rwth-aachen.de
or http://bronger-jmp.appspot.com

[Markus Holtermann]

Hey Torsten,

Are you talking about Django 1.7 migrations or South? In the former case
you need to make sure that your datamigration depends on the
contenttypes application.

If you use South: yes, calling update_all_contenttypes seems to be a
valid solution.

Best,

Markus

>--
>You received this message because you are subscribed to the Google Groups "Django users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
>To post to this group, send email to django...@googlegroups.com.
>Visit this group at http://groups.google.com/group/django-users.
>To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/87tx2x9y0k.fsf%40physik.rwth-aachen.de.
>For more options, visit https://groups.google.com/d/optout.

--

[Torsten Bronger]

Hallöchen!

Markus Holtermann writes:

> Are you talking about Django 1.7 migrations or South? In the
> former case you need to make sure that your datamigration depends
> on the contenttypes application.

I'm talking about Django 1.7 migrations. Unfortunately,
contenttypes is already migrated before my app. But this doesn't
help because update_all_contenttypes is a post_migrate signal.

[Daniel Hahler]

Hello,

I was having the same issue as Torsten: it does not appear to be possible to load initial data related to contenttypes or auth during migrations.

As for the problem with contenttypes reported by Torsten, a workaround appears to be calling `update_all_contenttypes` manually from your migration (from django.contrib.contenttypes.management).

But then there's the problem with creating auth.groups.

I've tried the following via `RunPython`:

    from django.contrib.auth.management import create_permissions
    create_permissions(apps.get_app_config('auth'), verbosity=0)

This aborts because `app_config.models_module` is None:

    app_config.__dict__
    {'models_module': None, 'name': 'auth', 'models': OrderedDict([('group_permissions', <class 'Group_permissions'>), ...]), 'module': None, 'label': 'auth', 'verbose_name': 'Auth'}

Is this related to https://code.djangoproject.com/ticket/23822 ("Serialize model managers in migrations"), and would be possible in Django 1.8?

I am trying to use https://github.com/alexhayes/django-migration-fixture, which is meant to provide a convenient wrapper around the initial_data fixtures (files). My PR with the changes mentioned above is viewable at: https://github.com/alexhayes/django-migration-fixture/pull/2/files.

What is the suggested way to have initial data with Django 1.7, given that initial_data is not used for apps with migrations, and RunPython has the above shortcomings?


Thanks,
Daniel.

[Markus Holtermann]

Hi,

first of all, both, update_all_contenttypes and create_permissions, expect their respective app to be migrated completely in order to work as of 1.7.4 (https://github.com/django/django/commit/478546fcef38d95866a92bc44d10e15b26c7254c).

The "serialize model manager in migrations" issue, which is part of 1.8, will allow you to use custom methods on your model manager inside `RunPython` (e.g `MyUser.objects.create_a_fancy_user_account()`) and isn't related to the problem you are running into.

The sanest way to generate all ContentTypes and Permissions is to migrate those two apps explicitly before migrating the remaining apps:

$ python manage.py migrate contenttypes

$ python manage.py migrate auth

$ python manage.py migrate

That way you can make sure the contenttypes app is completely migrated and all contenttypes are created, the auth app is completely migrated and all permission are created, and last but not least the remaining apps should smoothly migrate forwards. There are thoughts about putting the migration graph and the respective model states before/after all migrations into the pre/post_migrate signals (#24100) as well as initial ideas emitting a signal after each migration (no ticket yet afaik). The latter could be used to tackle the above problem.

If you want to create a specific permission during a migration, do the following in `RunPython`:

def forwards(apps, schema_editor): 

    Permission = apps.get_model('auth', 'Permission')

    Permission.objects.create(...)

This is also the way to load initial data starting Django 1.7: using `RunPython` or `RunSQL`.

The django-migration-fixture app seems to do some crazy stuff in the management command that I'm not going to speculate about how long it's going to work. That is completely undocumented and internal API. Apart from that, the (de)serializers in django.core.serializers (specifically .python.Deserializer) use the global apps for deserialization. Thus, a fixture referring to a model that is later removed/renamed, or refers to a field that doesn't exist later or must not be null, cannot be applied when you run the migrations on an empty database.

/Markus

[Daniel Hahler]

Hi Markus,

thanks for your answer and explanations!

I am now using a management hook to call the `loaddata` command manually
via the post_migrate signal:

% cat project/appname/management/__init__.py
# Handle initial data in Django 1.7+.

from django.db.models import signals
from django.core.management import call_command

def load_initial_data(app_config, sender, **kwargs):
if sender.label == __package__.split(".")[0]:
call_command('loaddata', 'initial_data.yaml', app_label='appname')

signals.post_migrate.connect(load_initial_data, weak=False)

Using a normal migration for this does not work when `flushdb` is called
during tests, and using pytest/pytest-django with the `--reuse-db` option.

Is there another option to fix this in Django 1.7+?

The TransactionTestCase is often only required because of the LiveServerTestCase.

Using serialized_rollback might be an option (https://code.djangoproject.com/ticket/22487),
but then it tends to be slower than not using "--reuse-db".

(With pytest-django the tests are not being ordered like with Django
currently, but that could be done - although I don't think it would help
much by itself.)


Thanks,
Daniel.

On 22.01.2015 14:23, Markus Holtermann wrote:

> The sanest way to generate all ContentTypes and Permissions is to
> migrate those two apps explicitly before migrating the remaining apps:
>
> $ python manage.py migrate contenttypes
> $ python manage.py migrate auth
> $ python manage.py migrate

That's not really an option for tests, is it?

> initial ideas emitting a signal after each migration (no ticket yet
> afaik). The latter could be used to tackle the above problem.

Yes, that seems sensible.

> If you want to create a specific permission during a migration, do the following in `RunPython`:
>
> def forwards(apps, schema_editor):
> Permission = apps.get_model('auth', 'Permission')
> Permission.objects.create(...)

If I understand it correctly this would cause the same problem, wouldn't
it - if you run all migrations in a single step (during tests)?

It would also not be there after `flushdb`.

> The django-migration-fixture app seems to do some crazy stuff in the
> management command that I'm not going to speculate about how long it's
> going to work. That is completely undocumented and internal API.

Yes, but that part is only used to auto-generate the migration file for
you. In the end is generated a file with a `RunPython` operation.

> Apart from that, the (de)serializers in django.core.serializers
> (specifically .python.Deserializer) use the global apps for
> deserialization. Thus, a fixture referring to a model that is later
> removed/renamed, or refers to a field that doesn't exist later or must
> not be null, cannot be applied when you run the migrations on an empty
> database.

Good to know. It

>
> /Markus
>
> On Thursday, January 22, 2015 at 12:43:33 PM UTC+1, Daniel Hahler wrote:
>
> Hello,
>
> I was having the same issue as Torsten: it does not appear to be possible to load initial data related to contenttypes or auth during migrations.
>
> As for the problem with contenttypes reported by Torsten, a workaround appears to be calling `update_all_contenttypes` manually from your migration (from django.contrib.contenttypes.management).
>
> But then there's the problem with creating auth.groups.
>
> I've tried the following via `RunPython`:
>
> from django.contrib.auth.management import create_permissions
> create_permissions(apps.get_app_config('auth'), verbosity=0)
>
> This aborts because `app_config.models_module` is None:
>
> app_config.__dict__
> {'models_module': None, 'name': 'auth', 'models': OrderedDict([('group_permissions', <class 'Group_permissions'>), ...]), 'module': None, 'label': 'auth', 'verbose_name': 'Auth'}
>

> Is this related to https://code.djangoproject.com/ticket/23822 <https://code.djangoproject.com/ticket/23822> ("Serialize model managers in migrations"), and would be possible in Django 1.8?
>
> I am trying to use https://github.com/alexhayes/django-migration-fixture <https://github.com/alexhayes/django-migration-fixture>, which is meant to provide a convenient wrapper around the initial_data fixtures (files). My PR with the changes mentioned above is viewable at: https://github.com/alexhayes/django-migration-fixture/pull/2/files <https://github.com/alexhayes/django-migration-fixture/pull/2/files>.