Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pkl]

Hello,

I've *at last* had some time to experiment with the idea I had thrown years again, of a system to workaround the recurring breakages introduced by new django releases, breakages which sometimes lead to unreconcilable requirements between django apps.

After a sprint at PyconFR, with a colleague (R. Gomès), and some cleanups and improvements, here it is:

https://github.com/pakal/django-compat-patcher  (also known as DCP)
https://pypi.python.org/pypi/django-compat-patcher

The philosophy of the module is that, with extremly agile languages such as python, one doesn't have to choose between a fast-moving project, and a compatibility-friendly project.
So, in an approach reminding "aspect oriented programming", DCP separates the Django codebase, which may move forward and be left mostly free of compatibility shims, from retrocompatibility shims ; and changing the "level of compatibility" of a project is just the matter of tweaking some Django settings.
Since, as usual, "explicit is better than implicit", logging and code warnings are used to ensure that project users are informed about what compatibility fixers are applied, and how project dependencies are rated deprecation-wise.

I've noticed there had been debates and evolutions for the past years, until the switch to a "lax semantic versioning" for Django.
Surely it's an effort in the right direction, but I guess I'm not the only one worried by the "lax" adjective ; especially after the long history of breakages occurring for quite cosmetic reasons (eg. the removal of urlpatterns(), which broke most django apps available, whereas dotted-path URLs were rather harmless).
That's why I think DCP is still relevant, as it gives back some control over compatibility to end users of Django ; and it should remove part of the apprehension some may have known too, when installing a new Django version.

DCP has been successfully used to upgrade wide projects from django 1.7 to django 1.10, with no hassle. But more feedback would be nice, before it can be marked "production ready".
The project welcomes patches, especially new backward/forward compatibility fixers one might need.


Apart from that : I think many people could benefit from more instructions about how to handle (backwards and forwards) compatibility, so here are some ideas I had :


1) Advertising compatibility systems for Django, in the official docs.

That is to say, packages like "django-compat" (which I only discovered lately), dedicated to library developers who need a wide compatibility.
And packages like "django-compat-patcher" for project maintainers who have compatibility issues and deadlines (it's not *always* the right moment to fork and patch a bunch of big dependencies).
All that would come in addition to the docs now mentioning "how to support 2 LTS versions at a time".
These measures might diminish the number of django app using handmade compatibility shims, or completely ignoring previous versions of django.


2) Pushing a compatibility system like DCP into Django itself

What would be the gain ? It would pleasantly reduce the work needed for Django evolutions.

At the moment, first compatibility shims must be introduced, and specifically tested in code ; then these compatibility shims must be removed, and their tests with them.

With a separate compatibility system, only half the work would be needed : once the code has been modified, the old version is moved to a fixer, and its associated unit-test is moved to the compatibility system test suite.
The activation/deactivation of this particular fixer is then only a matter of django settings, not of new code commits.
Most importantly, this way, django main tests suites  would stop making asserts on the presence or absence of deprecation warnings (which is a way of mixing unrelated aspects of the framework). Django tests suites could thus be leveraged by compatibility systems to ensure that the whole system is well sane and backwards/forward compatible, whatever the series of django shims applied.


Any remarks about DCP concept itself, or about suggestions for doc/core inclusions ?

thanks,
regards,
Pascal

[Tim Graham]

Our new deprecation policy was designed to eliminate the need for this type of compatibility library. If you want to support more Django versions than what our guidelines recommend (which may involve supporting versions that no longer receive security updates), I guess this library could be useful, but I wouldn't give it an endorsement in the Django docs.

Perhaps you could elaborate on "Most importantly, this way, django main tests suites  would stop making asserts on the presence or absence of deprecation warnings (which is a way of mixing unrelated aspects of the framework)." I don't understand your complaint about the current way we're testing deprecation warnings.

[Pkl]

Hello,

thanks for your feedback,

Actually this DCP layer is mainly aimed at backwards compatibility of dependencies : running, on the latest version of django (with maximal security bugfixes), older reusable apps which are incompatible with each-other, or with said latest version of Django.

When I hear "lax semantic versioning", it makes me think about an "almost alive parrot" or a "mostly non-lethal drink", i.e not something especially reassuring ; especially when I remember how the sentences "Django promises API stability and forwards-compatibility since version 1.0. In a nutshell, this means that code you develop against a version of Django will continue to work with future releases." were imho untrue.

If I understand correctly, with the new release process, reusable apps and libraries would still be broken every two LTS releases, i.e every 16 months. For a big software ecosystem, this is still a really short lifespan. And since not every developers cares about deprecation warnings (when they are visible), it means any pypi package for django which hasn't been updated for 6 or 12 months would quite likely be broken. This means lots of forks without any added value, without evolutions nor bugfixes, just "so that it keeps working" .

Many major languages or frameworks have strong commitments on retrocompatibility, and have succeeded on this commitment for years (python2 being one of them) ; I remember a blogpost where a Qt developer explained at great lengths how they carefully crafted every function signature, so that it could evolve without breaking older code. If they can achieve this C++, how comes major python frameworks would do less ?

My opinion is that the "walk or die" approach of Django is harmful to its ecosystem, and furthermore unnecessary, since a layer like DCP can ensure BOTH a wart-free & fast-moving codebase, and a strong retrocompatibility.

About the current tests of deprecation warnings, to me this is a wrong "separation of concerns". Checking that a web framework feature works, and checking that it properly triggers specific warnings, are aspects which should not be mixed with each other.
Why ? Because it prevents advanced uses of test suites. For example, how do you check that Django version X.Y hasn't broken code written for previous releases ? Just because it passes its own test suite ? Just because "committers have been careful" ? I don't think it says much. What I'd do, is rather run the unit-tests of the previous django version (excluding some specifically marked "short-lived" tests), against the newest code. This would mean so much more.

Here is my view on why using a compatibility layer in Django (or any framework, actually) would be a good idea :

PROs:

- cleaner django codebase

- less work to deprecate features

- automated checks of compatibility between django version

- easily achieve long-term compatibility

- more control given to django users regarding compatibility choices

CONs:

- small additional delay at django startup ?

Am I missing something there ?

regards,

Pascal Chambon

[Tim Graham]

> "Django promises API stability and forwards-compatibility since version 1.0. In a nutshell, this means that code you develop against a version of Django will continue to work with future releases." were imho untrue.

Perhaps the API stability statements are too strong and need some clarification, but please don't quote without reading the rest of the document, e.g. "If, for some reason, an API declared stable must be removed or replaced, it will be declared deprecated but will remain in the API for at least two feature releases. Warnings will be issued when the deprecated method is called."

> If I understand correctly, with the new release process, reusable apps and libraries would still be broken every two LTS releases, i.e every 16 months.

Yes, they may break if they aren't updated, although LTS releases are every 24 months.

My understanding of your proposal is that Django never removes deprecated features but rather moves them to a "compatibility" library. Removing deprecated features helps simplify things. I believe that prohibiting code to be removed from Django would be a significant overhead in complexity and be a drag on the ability to evolve the framework.

About testing, can you give an example of a problem that wasn't detected by our current approach? I'm unsure what problem your suggestions are trying to solve.

[Pascal Chambon]

I agree that reading the whole document gave some hints about the incoming troubles, but I guess many people (like me), on first pass, just thought "OK that's all I wanted to hear" and went by. Plus, it's a little like saying "this dogs doesn't bite", and then later "if the dogs wants to bite you, it barks first" - disturbing at least.   ^^

My bad, if people are guaranteed 2 x 24-month cycles before a feature gets removed, it's already much better. However, the same pattern as previously appears in docs : "each feature release will continue to have a few documented backwards incompatibilities where a deprecation path isn’t possible or not worth the cost.". I might be paranoid, but I foresee lots of dependency breakages in the future, if incompatibilities continue to be introduced at developer's will.... History proved even seemingly harmless django modifications (ex. import aliases removed) broke external code, sometimes forcing "commit reverts" in django code.

The idea of DCP is not to prevent ANY code removal (in particular when it deals with django settings, that any project can always update), but rather to allow very, very long-term compatibility, for trivial modifications like renames and signature changes ; that being said, even import proxies (white magic with meta importers, to keep alive modules like django.contrib.comments) seem to work quite well, hassle-free.

Concerning django unit-tests, here is the question : the version XXXX of django, how can one prove that it really is backwards compatible with the branch/version YYYY (except for a few documented breakages) ? Correct me if I'm wrong, but currently I see no automated way to check that, so an unwanted regression is always possible ; if "warnings" aspects are removed from tests, this becomes possible.
I dunno which such regression happened by the past, if they did, (this would require deep-browsing the Git repo), but it's a point of weakness of the backwards compatibility commitment imho, as *any* untested commitment is. On a side note, this mixing of concerns prevents me, for example, from testing that LatestDjango+DCP is well compatible with previous versions of django (since warnings, at least, differ).

regards,

Pascal

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/dKeLB-qR7lY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b5b58695-2e5c-49f2-86b9-63c64f44bc99%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[James Pic]

If you've been maintaining several django apps for several versions of Django (ie. stable, oldstable, lts) then it's pretty easy to imagine how useful this can be. The deprecation policy removes the need of a compatibility layer for code that should support only one version of Django, but does not help maintaining code that should support several versions of Django, which is a pretty cool challenge if you ask me :P

Thanks for doing this, looking forward to use it to make new django versions easier to support in apps, keep up the great work B)

[James Bennett]

On Sun, Jan 15, 2017 at 1:09 PM, Pkl <chambon...@gmail.com> wrote:

My bad, if people are guaranteed 2 x 24-month cycles before a feature gets removed, it's already much better. However, the same pattern as previously appears in docs : "each feature release will continue to have a few documented backwards incompatibilities where a deprecation path isn’t possible or not worth the cost.". I might be paranoid, but I foresee lots of dependency breakages in the future, if incompatibilities continue to be introduced at developer's will.... History proved even seemingly harmless django modifications (ex. import aliases removed) broke external code, sometimes forcing "commit reverts" in django code.

Just to clarify, the future plan -- beginning with Django 2.0, the next release after 1.11, which is about to feature-freeze -- is:

1. Releases go in a cycle of X.0, X.1, X.2, (X+1).0. So, for example, 2.0, then 2.1, then 2.2, then 3.0.

2. Each X.2 is an LTS.

3. Code which runs with no deprecation warnings on an LTS will also run (though may raise deprecation warnings) on the next LTS.

4. Thus, any time you run on an LTS with no deprecation warnings, you know you're safe to upgrade to the next LTS. And once you clear any new deprecation warnings on the new LTS, you know you're clear to upgrade to the one after that.

Regarding backwards-incompatible changes in general: they do happen, but they also follow the guidelines in the API stability document. When they occur, it's because there's a security issue or larger bug being solved. Additionally, many of the seemingly-large list of such changes each release are, if you dig into them, changes to private/internal or at least undocumented APIs not covered by the stability policy, but we've learned people still rely on those APIs and so we document changes to them even if we don't guarantee stability in them.

[Pascal Chambon]

Hi,

@James Pc - thanks for the support, if you happen to miss some fixers in DCP and don't have the opportunity to contribute them, please open an issue so that I have a look

@Tim Graham & James James Bennett - from what I sum up, the new policy simply extends the delay between breaking changes, and so the overall life expectancy of django reusable apps, but other limitations remain as is. I've detailed the misc benefits that a compatibility layer would bring, I just hope not too many people needing big compatibility will fail to learn about django-compat or DCP, and complicate their life with handmade shims or useless forks.

regards,

Pascal

--

You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/dKeLB-qR7lY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAL13Cg8_0BRpu1b_zz2Dx_YXcLaXGXw8Qw0jfA_uDhaxt%3DD_%3Dw%40mail.gmail.com.

[Dmitry Gladkov]

Hi,

Making Django upgrades less painful is a great goal, but I believe the patching Django and restoring removed functionality is not the right solution. JavaScript world has the same problem with far more frequent major compatibility breakages and they solve it with automatic codebase upgrade tools like jscodeshift [1]. This approach uses AST transformation similar to what 2to3 does, but with configurable transformation rules. There's also another project written in Python [2] that implements simpler and more general, but less reliable regex-based approach.

I believe adding codebase upgrade rules for each major Django release and giving users ability to apply them by running something like  ./manage.py upgrade_from 1.7 after Django upgrade will greatly simplify upgrading of large Django projects.

[1] https://github.com/facebook/jscodeshift

[2] https://github.com/facebook/codemod

--
Best wishes,
Dmitry Gladkov

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.

To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CA%2BXNUS35Au2um1an3bV-P8q%3DivfOM067r3Qay4jntd%2BCHADkPg%40mail.gmail.com.

[Pascal Chambon]

Hi Dmitry,

thanks for the alternative way, however it seems more a complement than a replacement of DCP.

Indeed, your "rewriting" approach updates a codebase to support the latest django API, but it raises a number of issues :

- how could it be applied to django reusable apps, installed via pip ? Does it mean a package hook should be used to automatically transform code on "pip install" for django-related modules, as setuptools allows it for the 2to3 case ? Or a special import hook for django ?

- it doesn't ensure backwards compatibility of the updated module, so unless every django-app maintainers use that system to stick to the latest django version, conflicts would remain when handling big projects.

- imo it can only handle trivial changes (renames mostly) ; when features disappear, or are replaced by much different approaches, unless this rewritter itself injects big shims, it wouldn't work. For example, the SortedDict, in which keys could be inserted at arbitrary positions, has been replaced by stdlib OrderedDict which is "readonly", so the code around needs to be rethought accordingly.

That being said, I guess everyone would love it, if they could upgrade THEIR codebase semi-automatically, instead of doing mass regex search/replaces.

There are plenty of AST modifiers for python (https://github.com/berkerpeksag/astor, or others, pytest does some nifty ast hacking too...), else a regex based django-command would already be nice too.

regards,

Pascal Chambon

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CA%2BkbqrW0F8YZSn8v5wkApAx9Z7a7Q20drM7z%3Dxn26qR4DsU7ng%40mail.gmail.com.

[Raphaël Barrois]

Hi Pascal,

I'm unsure as to what problem you're trying to solve; if I understand correctly, this project aims at making a "new"
Django behave like an "old" one.


I see a few possible use cases in the conversation:

* Running a legacy application (without new development) beyond the upstream support schedule (namely Django)
* Developing libraries (or reusable apps) that need to support several Django versions at once
* Using libraries that don't support (yet/anymore) your chosen Django version

In my opinion, the Django project has no reason to try to support the first one: those projects are basically
requesting longer security support from both Django and a set of third-party libraries; this requires time from the
maintainers of those projects.
If the project has decided to *not* provided longer-term-support, it is often for good reason, mostly "nobody is paying
developers enough for that additional support to be viable — if developers are paid at all".


For the second part, as a library developer, I'm indeed interested in a compatibility layer that would allow me to use
the *new* Django APIs in older versions; not the other way around.
This way, I can benefit immediately from improvements in the Django codebase and prepare my code for future upgrades —
akin to Python's ``from __future__ import ...```.
However, such a feature would need to provide a *wrapper* around Django's APIs and not modify it: other code in
the project is likely to rely on the documented behaviour.


For the third part, I'm unsure how you'd handle various libraries having different Django target versions?

On Mon, 23 Jan 2017 13:09:35 +0100
Pascal Chambon <chambon...@gmail.com> wrote:

> Hi Dmitry,
>

> thanks for the alternative way, however it seems more a *complement *than a

> >>> django-develop...@googlegroups.com.
> >>> To post to this group, send email to django-d...@googlegroups.com.

> >>> Visit this group at https://groups.google.com/group/django-developers.
> >>> To view this discussion on the web visit https://groups.google.com/d/ms
> >>> gid/django-developers/CAL13Cg8_0BRpu1b_zz2Dx_YXcLaXGXw8Qw0jf
> >>> A_uDhaxt%3DD_%3Dw%40mail.gmail.com

> >>> <https://groups.google.com/d/msgid/django-developers/CAL13Cg8_0BRpu1b_zz2Dx_YXcLaXGXw8Qw0jfA_uDhaxt%3DD_%3Dw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>> .

> >>>
> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Django developers (Contributions to Django itself)" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to django-develop...@googlegroups.com.
> >> To post to this group, send email to django-d...@googlegroups.com.

> >> Visit this group at https://groups.google.com/group/django-developers.
> >> To view this discussion on the web visit https://groups.google.com/d/ms

> >> gid/django-developers/CA%2BXNUS35Au2um1an3bV-P8q%3DivfOM067r
> >> 3Qay4jntd%2BCHADkPg%40mail.gmail.com
> >> <https://groups.google.com/d/msgid/django-developers/CA%2BXNUS35Au2um1an3bV-P8q%3DivfOM067r3Qay4jntd%2BCHADkPg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Django developers (Contributions to Django itself)" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/django-developers/dKeLB-qR7lY/unsubscribe.

> > To unsubscribe from this group and all its topics, send an email to

> > django-develop...@googlegroups.com.
> > To post to this group, send email to django-d...@googlegroups.com.

> > Visit this group at https://groups.google.com/group/django-developers.
> > To view this discussion on the web visit https://groups.google.com/d/

> > msgid/django-developers/CA%2BkbqrW0F8YZSn8v5wkApAx9Z7a7Q2
> > 0drM7z%3Dxn26qR4DsU7ng%40mail.gmail.com
> > <https://groups.google.com/d/msgid/django-developers/CA%2BkbqrW0F8YZSn8v5wkApAx9Z7a7Q20drM7z%3Dxn26qR4DsU7ng%40mail.gmail.com?utm_medium=email&utm_source=footer>
> > .

[Pascal Chambon]

Hi Raphaël ,

the goal of DCP is to "unite" the behaviour of different django versions, so that all the code needed by a project (misc. pypi dependencies, as well as custom project modules) may work together, whatever the exact django version each of these module actually targets. That's closer to your 3).

At the moment, available fixers are all for backwards compatibility, so the idea is to install the latest django version (with latest security features), and be able to run all kinds of dependencies with it, even if they haven't yet been updated. If fixers for forward compatibility got contributed, people could at the contrary remain on an old django version, yet begin to use new APIs and features. As far as I've seen, most changes can be patched in a way which allows both the old and new behaviour to work at the same time (that's what happens during the PendingDeprecation period of a change, by the way).

However, note that DCP is mainly aimed at *project maintainers*, eg. a specific website or saas. When developing a pluggable apps or library, you can't expect all your users to install and activate DCP in their own project, so a compatibility toolkit like "django-compat" (https://github.com/arteria/django-compat) is more relevant - at least as long as no DCP-like system is built into django.

regards,

Pascal

> >>> django-developers+unsubscribe@googlegroups.com.
> >>> To post to this group, send email to django-developers@googlegroups.com.

> >>> Visit this group at https://groups.google.com/group/django-developers.
> >>> To view this discussion on the web visit https://groups.google.com/d/ms
> >>> gid/django-developers/CAL13Cg8_0BRpu1b_zz2Dx_YXcLaXGXw8Qw0jf
> >>> A_uDhaxt%3DD_%3Dw%40mail.gmail.com

> >>> <https://groups.google.com/d/msgid/django-developers/CAL13Cg8_0BRpu1b_zz2Dx_YXcLaXGXw8Qw0jfA_uDhaxt%3DD_%3Dw%40mail.gmail.com?utm_medium=email&utm_source=footer>

> >>> .
> >>>
> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Django developers (Contributions to Django itself)" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to django-developers+unsubscribe@googlegroups.com.
> >> To post to this group, send email to django-developers@googlegroups.com.

> >> Visit this group at https://groups.google.com/group/django-developers.
> >> To view this discussion on the web visit https://groups.google.com/d/ms
> >> gid/django-developers/CA%2BXNUS35Au2um1an3bV-P8q%3DivfOM067r
> >> 3Qay4jntd%2BCHADkPg%40mail.gmail.com

> >> <https://groups.google.com/d/msgid/django-developers/CA%2BXNUS35Au2um1an3bV-P8q%3DivfOM067r3Qay4jntd%2BCHADkPg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Django developers (Contributions to Django itself)" group.
> > To unsubscribe from this topic, visit https://groups.google.com/d/
> > topic/django-developers/dKeLB-qR7lY/unsubscribe.

> > To unsubscribe from this group and all its topics, send an email to

> > django-developers+unsubscribe@googlegroups.com.
> > To post to this group, send email to django-developers@googlegroups.com.

> > Visit this group at https://groups.google.com/group/django-developers.
> > To view this discussion on the web visit https://groups.google.com/d/