Migrations and Reusable Apps

[Mark Lavin]

Hello,

I noticed some changes over the past few days to the migrations and I was concerned about how this could impact reusable applications planning to support 1.7.

In particular there is a note in https://docs.djangoproject.com/en/1.7/topics/migrations/#dependencies

Be aware, however, that unmigrated apps cannot depend on migrated apps, by the very nature of not having migrations. This means that it is not generally possible to have an unmigrated app have a ForeignKey or ManyToManyField to a migrated app; some cases may work, but it will eventually fail.

Given that contrib.auth now has migrations and many reusable applications have FKs to the User model, doesn't this force most reusable applications to ship migrations

in order to be compatible with 1.7? Is this an unintended consequence or an expected requirement? I know the plan is to eventually require migrations but this

seems like a hard requirement which could make upgrading to 1.7 more difficult due to lagging dependencies.

Is there an acceptable approach for a reusable application to support 1.6 + South and 1.7 + db.migrations? This http://treyhunner.com/2014/03/migrating-to-django-1-dot-7/ seemed

like a good approach though it does highlight the backwards compatibility issues.

Best,

Mark

[Tim Graham]

I'm sure Andrew will have more info, but you could use MIGRATION_MODULES and generate migrations for any 3rd party apps that don't have them. I'm not sure how the transition when the app starts shipping its own migrations will go.

[Mark Lavin]

This might be a solution for projects which wish to upgrade in advance of their dependencies but as you note its implications for future app upgrades are unclear.

This still doesn't provide an answer for app developers. The deprecation timeline says migrations aren't required until 2.0. But if your app depends on contrib.auth it will be required in 1.7 based on this previous note.

As best I can tell, supporting Django < 1.7 + South and Django 1.7+ requires making a backwards incompatible release of the app. That's fine if that is the only way forward but I think the community could use more

guidance on this.

[Trey Hunner]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Mark,

I've left replies inline below.

Note that I am not heavily involved with South development nor the
Django 1.7 release so my perspective may be skewed from reality.

On 06/17/2014 06:18 AM, Mark Lavin wrote:
> Given that contrib.auth now has migrations and many reusable
> applications have FKs to the User model, doesn't this force most
> reusable applications to ship migrations
> in order to be compatible with 1.7? Is this an unintended consequence or
> an expected requirement? I know the plan is to eventually require
> migrations but this
> seems like a hard requirement which could make upgrading to 1.7 more
> difficult due to lagging dependencies.

Reusable apps should add Django 1.7 migrations if they have none. Going
from no migrations to Django 1.7 migrations is much easier than going
from South migrations to 1.7 migrations (assuming South compatibility
should be maintained).

> Is there an acceptable approach for a reusable application to support
> 1.6 + South and 1.7 + db.migrations?
> This http://treyhunner.com/2014/03/migrating-to-django-1-dot-7/ seemed
> like a good approach though it does highlight the backwards
> compatibility issues.

There have been a few more developments since I wrote that blog post.

Andrew and I discussed the idea of supporting a south_migrations module
in the next release of South. This would allow that special
south/__init__.py code to be removed so long as users were asked to
upgrade to the latest version of South.

Relevant south-users mailing list thread:
https://groups.google.com/forum/#!msg/south-users/YW7zQ1KeuvM/o-EzlEQX4jsJ
Relevant discussion on the django-email-log pull request:
https://github.com/treyhunner/django-email-log/pull/8

>
> Best,
>
> Mark
>
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.
> To post to this group, send email to django-d...@googlegroups.com
> <mailto:django-d...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/afd7da9e-02e3-4c80-bef9-71dac036c415%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/afd7da9e-02e3-4c80-bef9-71dac036c415%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

- --
Trey Hunner
http://treyhunner.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJToG6eAAoJEOpnfp/Nreon0gEH/2cwlY4xvoPKK5IjQA/opb75
umcsQDwc479iHXMUIHt91IirvMRt7Mt4AyO8RxyHMymF3Rfn3kHvHpdjIvwuz4b3
smr8ddIbtK33JfvO2HFESSnIuV+o2znjduaZRbAXMzAyPKb0w1IrcHyxl+HIwOHB
FkkQs2zWdexmg3RhnhBq20cy7qqolsKGPBkQzSyQjkiWyaoDfh6gKsOWO8wRurZa
NR0G3L/buOCDUAdrmCee8JWbESPqWU3PqPoSCv9y0A2Zy9g1v7rrbj9Y/aczx3ME
LYacXeYLdOcXgMAvWOGw4grzqSFsUclMfM+pzPBu6DDY1TGwgtFIx+FcJDvwkRo=
=Vdz5
-----END PGP SIGNATURE-----

[Andrew Godwin]

Trey is correct about south supporting south_migrations. I'm going to try and get that release out this week so people can start preparing.

As for the dire warning about unmigrated apps not relying on migrated ones, it's true that converting auth does this, although the very existence of AUTH_USER_MODEL has unfortunately implied this from the beginning; unless you're not willing to support swappable users, you have to be migrated so that if someone swaps in a user model from a migrated app you're willing to support it. 

You can easily disable the built-in contrib migrations by using MIGRATION_MODULES if you wish, too. We want to ship them this release so we can finally ship changes to the core models (auth.User's length limits in particular) next release.

1.7 is not going to be a pretty release for third-party apps for a number of reasons: this, the app-loading refactor, the changes to custom fields, and a few other things. I might leave an open offer of help upgrading from myself after the release to any app maintainers to try and smooth that process along.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/53A06E9E.1060705%40treyhunner.com.

[Trey Hunner]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/17/2014 10:05 AM, Andrew Godwin wrote:
> 1.7 is not going to be a pretty release for third-party apps for a
> number of reasons: this, the app-loading refactor, the changes to custom
> fields, and a few other things. I might leave an open offer of help
> upgrading from myself after the release to any app maintainers to try
> and smooth that process along.

+1 that. This has been the hardest recent release for my own apps
(mostly because of the app-loading refactor).

I'm grateful for the help I received while migrating
django-simple-history to 1.7.

Sign me up for that offer also. I can help out with 1.7 support advice
as my abilities allow.

- --
Trey Hunner
http://treyhunner.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJToHv2AAoJEOpnfp/NreonGzoIAMOpD4xBRiZDDfw1WwuHqVBd
mrq/VbyA7iTGMzQZ0ZSXhUT4Qd1dR0T1k0Qcyaeg837GdufkL4ikj8mFXHIRVAes
pXHYkV2W3NI8J6VqPDw1fB41nNcFxa7q/1GoABT0yLsAAfCu6v2WNpx+qipFmI7g
6ugpL7fa+9lqd2K2bDsgW9s0UR/Z67DenpCdmYN9kFk8h49c+B/v/EbDV89qT3tg
pogtGKE/UOsvQG76v4BEcQmnQkAKNPzY9jFFb8wkhgzwkBj6tTXhbuqRw2TDhkKH
ORJj5xEKrIeSflFHqb4wT2zjlOMpyW3dwDs+X0PUe8mU0ZhPhpvtVFgWjA1aDOw=
=Marr
-----END PGP SIGNATURE-----

[Mark Lavin]

Support of south_migrations directory in South goes a long way to addressing this problem from my perspective so that's very good news to me.

[Aymeric Augustin]

2014-06-17 19:33 GMT+02:00 Trey Hunner <tr...@treyhunner.com>:

+1 that.  This has been the hardest recent release for my own apps
(mostly because of the app-loading refactor).

I'm grateful for the help I received while migrating
django-simple-history to 1.7.

At this point, I think you're the only person who reached out for help :-)

So, if anyone else is maintaining a third-party app and could use tips

about how to deal with app-loading, let me know and I'll do my best!

--
Aymeric.

[Sebastian Vetter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey Aymeric,

> At this point, I think you're the only person who reached out for
> help :-)

well, not anymore ;)

> So, if anyone else is maintaining a third-party app and could use
> tips about how to deal with app-loading, let me know and I'll do my
> best!

I appreciate the offer and will gladly take it :) I'm currently in the
middle of a migration to Django 1.7 incl. backwards-compatibility. I'm
collecting questions about best practises right now and would love to
get your input. Would this list be the right place to post them?

Cheers,
Seb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTqPjDAAoJEEdPmY9Czr0eMdwH/j3uMthWPC12OCSnpF5WfUHS
V0oukKB/OGbUdNoojOZlbhpa2xI6bjqGF6LiU5sZVMZkvH1CcmJP+HhwsCCBFUxb
zfJ9OOr0twUF9S4jUIMEo0Rl4FjlEzAqUE5sIhlqDiR3n2kKVphCNwSGXxqSDq/L
4FMfXygcWVQlaYjbGZlGCRRZeqwD2qsD04V210ehRBTWpUiz/67uOkihCB4xgFS2
/NbS8vFAntFs+FYLvJ5ng63XVqESwPQh5FeQFFloDY3ssZu/Nqo3oNFF2+COav+n
yuEighwbMI5kUePOu80SRIrlN742vXyLqPVYxY82JiybpXmHH4/zg9uOCUCYS/c=
=o8TL
-----END PGP SIGNATURE-----

[Florian Apolloner]

On Tuesday, June 24, 2014 6:07:05 AM UTC+2, Sebastian Vetter wrote:

I appreciate the offer and will gladly take it :) I'm currently in the
middle of a migration to Django 1.7 incl. backwards-compatibility. I'm
collecting questions about best practises right now and would love to
get your input. Would this list be the right place to post them?

Since it is a new feature under development, I'd say it's fine to post here. This way others can learn from the answers too.

Cheers,
Florian

P.S.: Btw you might wanna switch from inline pgp to pgp/mime.

[Sebastian Vetter]

Hey Aymeric, hey all,

it took a little longer than I hoped but I eventually managed to put
together a few questions that came up while porting a pluggable app to
Django 1.7 with backward-compatibility through to 1.4. I hope all of
these make sense and I am looking forward to the feedback:

1) Creating a subclass of AppConfig for a Django app is (according to
the docs) not required. If none is defined, Django falls back to AppConfig.

For a pluggable app, is it recommended to always subclass AppConfig and
provide default_app_config?

2) The recommended way to register signals with receivers now (assuming
they should be registered automatically) is the ready() method.
Previously, the only "safe" way to ensure they would be connected was
doing that in the 'models.py'.

For a pluggable app this can be tricky if you are trying to maintain
backward-compatability. The only way I can think of is checking in the
'models.py' for a Django version < 1.7 and only run the code then. Is
there another/better way to do that?

3) Across several pluggable apps that I've worked on, there have been
several occasions where a model from one app is imported into the
'models.py' of another app. Now this causes issues with the new
AppConfig if I call 'get_model' at the module level. The only way I see
around it is to pull 'get_model' into that function(s) where this model
is used...which seems bad. An example would be here [1]. It might not be
the best example but one that I've come across just a few days ago.

Is there a better way to deal with this when using the AppConfig? Or
should that be avoided in general to import model in another apps'
'models.py'.

4) A project that I am working with is django-oscar [2] which defines an
'app.py' which contains a subclass of 'Application' (e.g. [3]). The main
purpose is to provide an overridable way to define URL pattners by
providing them through a 'get_urls' method.

It seems intuitive to me to consolidate functionality like that into the
AppConfig subclass for an app. Are there any reasons for or against
doing this? Are there any implications this might have on the loading of
the app registry?

Thanks for any thoughts and advice on this. And thanks for the great new
appregistry, it's a pleasure to work with.

Cheers,
Sebastian

PS: @Florian, thanks for the PGP/Mime hint. I hope it looks better this
time :)

[1]
https://github.com/tangentlabs/django-fancypages/blob/issue/33/support_for_django17/fancypages/models/blocks/content.py#L127
[2] https://github.com/tangentlabs/django-oscar
[3]
https://github.com/tangentlabs/django-oscar/blob/master/oscar/apps/basket/app.py

[Sebastian Vetter]

Hi Andrew, hi Trey,

as part of the migration to Django 1.7 I just mentioned in my post about
the AppConfig, I've obviously also looked into migrations and I am a big
fan :) For reference, the project in question is django-fancypages [1].

The route the I have taken for now is the same one described in Trey's
blog post. Specifically variant 3) which uses 'migrations' for Django
migrations and 'south_migrations' for South. For now, I've documented
the setting 'SOUTH_MIGRATION_MODULES' as a requirement for pre-1.7
projects. This seems a good enough solution for now and I'm happy to
encourage people to move to Django 1.7 by making it a bit easier.

The one thing that I am currently struggling with a bit is how to ensure
consistency of both migrations whenever a model changes happens. Do you
have any suggestions on how to deal with that?

Is there a more sensible way than putting a couple of scripts together
that, e.g. run migrations in different Django version and then run tests
for migrations checking the schema of both. Any suggestions on that? I
might be overthinking this but I don't like it when I have to remember
these things...I'm bad at that.

Thanks for you input on this and thanks to Andrew and anyone who helped
getting this amazing migrations framework into Django 1.7!

--
Cheers,
Seb

[1] https://github.com/tangentlabs/django-fancypages

[Aymeric Augustin]

2014-06-30 14:32 GMT+02:00 Sebastian Vetter <seba...@roadside-developer.com>:

1) Creating a subclass of AppConfig for a Django app is (according to
the docs) not required. If none is defined, Django falls back to AppConfig.

For a pluggable app, is it recommended to always subclass AppConfig and
provide default_app_config?

There are two reasons for doing that:

1) Provide a translatable verbose_name for the admin. If your application

doesn't have models, or if it isn't translated and the capitalised app_label

is what you need, this reason doesn't apply.

2) Run code at start-up. If your application does this, I would encourage

you to move it inside AppConfig.ready(), even if it works just fine at

module-level in models.py.

If neither matters for your application, don't bother with default_app_config,

just use Django's default AppConfig.

2) The recommended way to register signals with receivers now (assuming
they should be registered automatically) is the ready() method.
Previously, the only "safe" way to ensure they would be connected was
doing that in the 'models.py'.

For a pluggable app this can be tricky if you are trying to maintain
backward-compatability. The only way I can think of is checking in the
'models.py' for a Django version < 1.7 and only run the code then. Is
there another/better way to do that?

That's the pattern I'm advocating. One may find explicit version checks

inelegant, but in practice they just work. For an example, see:

https://github.com/django-debug-toolbar/django-debug-toolbar/blob/master/debug_toolbar/models.py

 

3) Across several pluggable apps that I've worked on, there have been
several occasions where a model from one app is imported into the
'models.py' of another app. Now this causes issues with the new
AppConfig if I call 'get_model' at the module level. The only way I see
around it is to pull 'get_model' into that function(s) where this model
is used...which seems bad. An example would be here [1]. It might not be
the best example but one that I've come across just a few days ago.

Is there a better way to deal with this when using the AppConfig? Or
should that be avoided in general to import model in another apps'
'models.py'.

You can import models from other apps explicitly:

from otherapp.models import SomeModel

 

What doesn't work is to obtain them through get_model().

This is often an issue with get_user_model(), which is just a shortcut for

get_model(settings.AUTH_USER_MODEL).

At this point, my best advice is to use string references eg. 'app.Model'

at import time and resolve them at runtime. That's how Django's custom

user models are implemented.

If you're asking with django-oscar in mind, I'm following closely the story

there. My team is currently building an Oscar shop. We're going live in

September and we'd like to do so on Django 1.7.

4) A project that I am working with is django-oscar [2] which defines an
'app.py' which contains a subclass of 'Application' (e.g. [3]). The main
purpose is to provide an overridable way to define URL pattners by
providing them through a 'get_urls' method.

It seems intuitive to me to consolidate functionality like that into the
AppConfig subclass for an app. Are there any reasons for or against
doing this? Are there any implications this might have on the loading of
the app registry?

I see this as a variant of the common suggestion of managing app-specific

settings in AppConfig classes. We might want to implement that at some

point. But I'd like to see how the dust settles on my refactoring first.

In the mean time, nothing prevents third-party apps from providing mixins

implementing that feature.

Regarding django-oscar, I have a hunch that its Application class should

subclass AppConfig on Django 1.7. But that would make it difficult to

support both 1.6 and 1.7 in the same release.

-- 

Aymeric.

[Sebastian Vetter]

Hey Aymeric,

thanks a lot for your immediate feedback. I appreciate that a lot. I've
added a few comments inline.

On 30/06/14 23:55, Aymeric Augustin wrote:
> 2014-06-30 14:32 GMT+02:00 Sebastian Vetter
> <seba...@roadside-developer.com

> <mailto:seba...@roadside-developer.com>>:

>
> 1) Creating a subclass of AppConfig for a Django app is (according to
> the docs) not required. If none is defined, Django falls back to
> AppConfig.
>
> For a pluggable app, is it recommended to always subclass AppConfig and
> provide default_app_config?
>
>
> There are two reasons for doing that:
>
> 1) Provide a translatable verbose_name for the admin. If your application
> doesn't have models, or if it isn't translated and the capitalised app_label
> is what you need, this reason doesn't apply.
>
> 2) Run code at start-up. If your application does this, I would encourage
> you to move it inside AppConfig.ready(), even if it works just fine at
> module-level in models.py.
>
> If neither matters for your application, don't bother with
> default_app_config,
> just use Django's default AppConfig.

That makes sense. I gues in practise this means defining a AppConfig
most of the time if just for reason 1).

> 2) The recommended way to register signals with receivers now (assuming
> they should be registered automatically) is the ready() method.
> Previously, the only "safe" way to ensure they would be connected was
> doing that in the 'models.py'.
>
> For a pluggable app this can be tricky if you are trying to maintain
> backward-compatability. The only way I can think of is checking in the
> 'models.py' for a Django version < 1.7 and only run the code then. Is
> there another/better way to do that?
>
>
> That's the pattern I'm advocating. One may find explicit version checks
> inelegant, but in practice they just work. For an example, see:
> https://github.com/django-debug-toolbar/django-debug-toolbar/blob/master/debug_toolbar/models.py

Inelegant is the right word :) But I agree that it works and with a
deprecation on top of it they might not have to stick around for too long.

> 3) Across several pluggable apps that I've worked on, there have been
> several occasions where a model from one app is imported into the
> 'models.py' of another app. Now this causes issues with the new
> AppConfig if I call 'get_model' at the module level. The only way I see
> around it is to pull 'get_model' into that function(s) where this model
> is used...which seems bad. An example would be here [1]. It might not be
> the best example but one that I've come across just a few days ago.
>
> Is there a better way to deal with this when using the AppConfig? Or
> should that be avoided in general to import model in another apps'
> 'models.py'.
>
>
> You can import models from other apps explicitly:
> from otherapp.models import SomeModel
>
> What doesn't work is to obtain them through get_model().

As far as I remember, the explicit import of models was discourage in
favour of get_model(). Is that no longer the case with the new app
registry? Do you see portential problems with explicit import and the
app registry initialisation?

> This is often an issue with get_user_model(), which is just a shortcut for
> get_model(settings.AUTH_USER_MODEL).
>
> At this point, my best advice is to use string references eg. 'app.Model'
> at import time and resolve them at runtime. That's how Django's custom
> user models are implemented.
>
> If you're asking with django-oscar in mind, I'm following closely the story
> there. My team is currently building an Oscar shop. We're going live in
> September and we'd like to do so on Django 1.7.

I'll be following that conversation closely. A few of the apps that I'm
maintaining/working on around Oscar are using similar patterns for
obvious reasons and the challenges/solutions are most likely to be the
same or similar.

> 4) A project that I am working with is django-oscar [2] which defines an
> 'app.py' which contains a subclass of 'Application' (e.g. [3]). The main
> purpose is to provide an overridable way to define URL pattners by
> providing them through a 'get_urls' method.
>
> It seems intuitive to me to consolidate functionality like that into the
> AppConfig subclass for an app. Are there any reasons for or against
> doing this? Are there any implications this might have on the loading of
> the app registry?
>
>
> I see this as a variant of the common suggestion of managing app-specific
> settings in AppConfig classes. We might want to implement that at some
> point. But I'd like to see how the dust settles on my refactoring first.
>
> In the mean time, nothing prevents third-party apps from providing mixins
> implementing that feature.

That makes absolute sense. I mainly wanted to check that there isn't
anything in there or in the works that might make this difficult.

> Regarding django-oscar, I have a hunch that its Application class should
> subclass AppConfig on Django 1.7. But that would make it difficult to
> support both 1.6 and 1.7 in the same release.

I think your mixin suggestion from above would still be possible. It's
something that I was planning to suggest on the mailinglist over there.

Thanks again,
Sebastian

[Aymeric Augustin]

> Le 1 juil. 2014 à 04:46, Sebastian Vetter <seba...@roadside-developer.com> a écrit :
>
> As far as I remember, the explicit import of models was discourage in
> favour of get_model(). Is that no longer the case with the new app
> registry? Do you see portential problems with explicit import and the
> app registry initialisation?

That only applies when you implement an AppConfig.ready() method,
in order to avoid a chicken'n'egg problem between apps and models.

--
Aymeric.