Some thoughts on upgrade pain & deprecations.

[Tom Christie]

I've just finished upgrading Django REST framework to support 1.10, and it's given me some food for thought on things that can make Django upgrades painful.

Firstly, here's a couple of things that I needed to track down in order to upgrade that weren't obvious at first...

* A bunch of test views started returning 404s.

* None of the app templates were being found.

* `user` was no longer being passed into template contexts.

All of these were due to changes that have been part of the standard deprecation cycle, which is fair enough, however they weren't necessarily all that easy to debug. Figuring them out involves digging through the set of removed features, and trying to figure out which of those would result in the behavior change that's been seen.

Aside: I think the `.urls = ...` -> `@override_settings(ROOT_URLCONF=...)` change to test cases is missing from those notes?

I expected to have seen a deprecation warning when running the tests under 1.9, although wasn't exactly sure if and when those warnings would be displayed.

Seeing the deprecation warnings under 1.9 would have made the upgrade far simpler, as I'd be warned about the changes I needed to make, rather than seeing a behavioral change and having to debug what might have caused it.

Figuring out how to ensure that the tests properly displayed deprecation warnings (under py.test) wasn't all that easy either, eventually I got there with...

   python -Wd [...]/py.test tests/ -s

(The behavior still isn't ideal then as each instance of warning is output once every time for each occurrence, rather than once and once only)

The upshot of all this is that I believe many of our users are likely to be hitting problems with upgrades because they're not seeing deprecation warnings, and then get hit by a behavioral change that they weren't expecting to have to deal with, and end up having to debug the cause of.

I'm wondering if we could do more to help our users here?

For instance, a user is running under 1.9 and wants to upgrade to 1.10. Could we have a reliable and documented mechanism by which they'd first run the tests under 1.9, and have any Deprecation warnings treated as errors?

If they're running 1.8, could we document how to run the tests so that any PendingDeprecation warnings are treated as errors?

Could we collate instances of these warnings, rather than displaying them in an overly verbose fashion?

python -Werror sort of achieves this, but doesn't differentiate between deprecation and pending deprecation.

An example of the sort of thing we could do might be to have an environment variable `DJANGO_UPGRADE`, that if set to a version number, forces the relevant class of deprecation / pending deprecation warnings to be written to a log file or similar. I don't think that interface is quite right, but suggesting it as an example of how we might make our user's lives easier.

Any other suggestions or thoughts in this area?

  Tom

[Tom Christie]

The low tech solution to this, may just be to have the release notes recommend running your test suite using

`PYTHONWARNINGS=once`, and making sure not to swallow any output, eg. with py.test that'd be...

PYTHONWARNINGS=once py.test tests -s

I'm fairly sure that making that point explicitly would save an awful lot of folks a more painful upgrade, by making it more clear how to see upcoming issues without getting bitten by them and having to debug without the help of an associated warning.

[Josh Smeaton]

I like the general idea. Going with your low tech solution we could have snippets for the X most popular test runners to treat warnings as errors. Perhaps ./manage test.py --warnings-as-errors or similar for the interface django provides. Every time django makes a release there are some that express frustration at deprecations, so making the fixing process easier can only help there. 

[Tim Graham]

 

Aside: I think the `.urls = ...` -> `@override_settings(ROOT_URLCONF=...)` change to test cases is missing from those notes?

The 1.10 release notes say, "SimpleTestCase.urls is removed."
 

I expected to have seen a deprecation warning when running the tests under 1.9, although wasn't exactly sure if and when those warnings would be displayed.

Deprecation warnings are loud by default when using "manage.py test", however, I've removed that behavior starting with Django 1.11 as per https://docs.djangoproject.com/en/dev/releases/1.11/#deprecating-warnings-are-no-longer-loud-by-default. I'll probably duplicate those tips in the "how to upgrade Django" doc (https://docs.djangoproject.com/en/dev/howto/upgrade-version/).

(The behavior still isn't ideal then as each instance of warning is output once every time for each occurrence, rather than once and once only)

I don't think it's a sensible default to see each warning once rather than each time the deprecated behavior is invoked. For some warnings it might make sense, but for others, then you'd have to rerun the test suite after fixing each warning to see if the warning pops up elsewhere (unless I'm missing your rationale?).
 

The upshot of all this is that I believe many of our users are likely to be hitting problems with upgrades because they're not seeing deprecation warnings, and then get hit by a behavioral change that they weren't expecting to have to deal with, and end up having to debug the cause of.

I guess this is currently limited to anyone not using "manage.py test", but will affect all users starting with Django 1.11 per the previously mentioned changed.
 

For instance, a user is running under 1.9 and wants to upgrade to 1.10. Could we have a reliable and documented mechanism by which they'd first run the tests under 1.9, and have any Deprecation warnings treated as errors?

If they're running 1.8, could we document how to run the tests so that any PendingDeprecation warnings are treated as errors?

warnings.simplefilter("error", RemovedInDjango20Warning)

With the latest deprecation/release schedule, I think technique described in the 1.11 release notes should work for the majority of cases. If projects want to support consecutive LTS releases, they shouldn't worry about deprecation warnings until a "dot zero" release, at which point they should follow the tip and fix all warnings.
 

Could we collate instances of these warnings, rather than displaying them in an overly verbose fashion?

I'm against adding custom behavior in Django (like what's been removed in https://github.com/django/django/commit/5b94b17feff15a9f0345f92fc0568bfe7038e3a3) that makes warnings work differently from in Python.
 

An example of the sort of thing we could do might be to have an environment variable `DJANGO_UPGRADE`, that if set to a version number, forces the relevant class of deprecation / pending deprecation warnings to be written to a log file or similar. I don't think that interface is quite right, but suggesting it as an example of how we might make our user's lives easier.

I've never had trouble with the console output format myself, but I suppose you could experiment with some third-party package that does what you want and we'll see if people find it useful.

[Tom Christie]

> I'm against adding custom behavior in Django

That's entirely reasonable, yes.

In which case, what do folks think of the following suggestions:

* Linking prominently to the 'upgrading django' section from at the begining of every new version's release notes.

* Tweaking the upgrading django section so that it recommends *first* running your existing test case with warnings enabled and resolving those at that point, *before* upgrading django, and *then* upgrading and running the tests again, at which point you don't necessarily need warnings enabled. (As it currently stands it's rather the wrong way around)

* Tweaking the upgrading django section so that it also mentions making sure that your test runner isn't silencing the warning output. (eg "Using the py.test runner, you probably want to use something like `PYTHONWARNINGS=default py.test tests -s`")

Cheers,

  Tom

[Tom Christie]

Picking this up in trac: https://code.djangoproject.com/ticket/26713

And github: https://github.com/django/django/pull/6732