Introducing django-compat - arteria's solution for for- and backwards compatibility from Django 1.4.x to 1.8.x/1.9.x

[Philippe O. Wagner]

TLDR; Introducing django-compat - arteria's solution for for- and backwards compatibility from Django 1.4.x to 1.8.x./1.9.x

SITUATION

We really love how Django evolves and how the core gets better and better. New major versions of the framework that comes with changes, bugfixes and new features are released quickly. This is great and nothing is wrong with that!

But there are issues from the business/agency/our point of view:

* We are not as fast as Django is

* We have reusable apps that must work with multiple Django versions and

* We have a lot of these apps, open and closed source

A lot of (3rd party/open source) apps

 

* ignore older Django version due to the additional effort or

* have this try/except pattern everywhere in the code or

* encapsulate them in a per app compat.py file, see a some example in the projects README

All our "reusable apps" for client project and products where built on and for the Django LTS 1.4 version. With the release of the new LTS version we started every new project on 1.8, but still have all other older projects that runs on 1.4 and depends on these apps and it's update that must be compatible with both versions in our case.

SOLUTION

To handle this problem we created django-compat [1], which is something similar to six. The goals of django-compat are:

* Eliminate code duplication from app to app and handle them in one central place.

* Make apps working with multiple Django version and provide a backward compatibility

* Bringing things that are availbale in newer releases (sth. like importing from future) into older one

* Have a stable and testet single library that handles these compatible objects ...

CURRENT STATE / WHATS NEXT

We are using this library/approach successful in production on our clients project, in our products and its in open source apps. Eg django-hijack, django-background-tasks, ... to provide best possible stability It's tested using the test cases that are shipped with Django itself.

So it provides for- and backward compatibility between Django versions (we basically cover what is supported by Django itself and (will, WIP) align to the official releases.

We already started to integrate 1.9 support. Next will be to add more of 1.9 and more tests.

We are aware that there are some issues with the approach in some cases. Eg the get query set renaming. [2]

I'm curious what you think about django-compat and if it would also simplify your other djangonauts' life.

Regards,

Philippe

 

PS I: I'm introducing this on the developer mailing list due to a chat on the DjangoCon Europe with Loic Bistuer.

PS II: This thing was discoussed already on reddit. [3]

[1] https://github.com/arteria/django-compat

[2] http://lukeplant.me.uk/blog/posts/handling-django%27s-get_query_set-rename-is-hard/

[3] http://www.reddit.com/r/django/comments/2jrr4l/whats_the_best_practice_to_provide/

[Tim Graham]

When do you drop support for old versions of Django? The main concern I have is that it somewhat encourages running on unsupported and insecure versions of Django (currently 1.5, 1.6; and 1.4 will be end of life in October). Therefore I don't thinking giving it an official blessing is a good idea.

In the "1.9 release planning" thread I proposed a new deprecation schedule to make it easier for third-party apps to support the currently supported Django versions now that we have LTS releases. Here's that proposal:

https://docs.google.com/document/d/1bC6A8qc4skCmlagOnp8U7ddgyC-1XxXCBTlgrW690i0/edit?usp=sharing

Feedback on that will would be welcome.

[Philippe O. Wagner]

Thanks Tim for your response and sharing the report of the survey.

As mentioned in the introduction, we will (give our best to) keep aligned to the official releases and do not intend to bypass the security concept of Django.

We will update the project README to communicate the concept correctly. https://github.com/arteria/django-compat/issues/28

Compatibility seems to be common issue. For internal use, we are interested in work from LTS to LTS. For our open source apps we want to support what's supported officially to not exclude others. That's why we started this thing.

I'd really welcome the bi-yearly LTS release cycle with one year of LTS support overlap - the longer the better.

Philippe

[Loïc Bistuer]

Thanks Philippe for bringing this up.

I'm currently upgrading a large Django app with dozens of dependencies from 1.4 to 1.8, here are some observations:
- Most popular and/or maintained libraries actually supports every Django version between 1.4 to 1.8. (Many thanks to their maintainers!)
- Libraries that support 1.4 and 1.8 but not with a single version add a lot more overhead to the upgrade process.
- Libraries have their own backwards incompatibilities and deprecations. By itself it's easily manageable when the new version still support your current Django version, but it gets messy when you need to upgrade both Django and 3rd-party apps at the same time.

Obviously, no production projects should depend on an unsupported version of Django, but you still need to upgrade to intermediary versions as you work your way to the latest LTS. 3rd-party apps that support both LTS have been a huge help to us: you can upgrade them making changes to your project as required, then they stay out of the way when you upgrade Django itself. I wouldn't consider that libraries still supporting Django 1.5 are encouraging running unsupported versions of Django, they are just providing an upgrade path. Without such upgrade paths in the Django ecosystem, I think LTS while good on paper, are a bad idea in practice.

The new proposal to have an LTS every third release is an improvement over the current situation since 3rd-party apps need to support one less version to bridge between two LTS. But I'm not convinced with "deprecated features won’t be dropped until the version those features were deprecated in is no longer supported"; it adds overhead to Django's maintenance while still requiring 3rd-party apps to create shims if they want to support both LTS to ease with the upgrade process.

How about dropping all the shims in the release immediately following an LTS? For example (slightly rewriting the past):

1.8 (LTS): No features dropped.
1.9: Dropped features deprecated in 1.4, 1.5, 1.6, 1.7
2.0: No features dropped.
2.1 (LTS): No features dropped.
2.2: Dropped features deprecated in 1.8, 1.9, 2.0

Cheers

--
Loïc

> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/631f206c-5b06-4137-8c03-f24c2c728306%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Tim Graham]

Carl proposed a similar thing: https://groups.google.com/d/msg/django-developers/qCjfOu-FPxQ/hccAcVChHMkJ
It would be a bit better than he outlined there with an LTS every 2 years instead of 3 as I proposed.

It seems difficult to estimate how much such a policy would increase maintenance costs in Django itself. The current proposal already keeps features deprecated in 1.8 around an extra 8 months compared to the old scheme. Your suggestion extends this to 16 months (and for features deprecated in 1.9, an extra 8 months too). An interesting exercise would be to look through the features deprecated in 1.8 and try to guesstimate the maintenance costs of keeping them around longer. I guess such estimation will be basically impossible though -- only when working on new features will we realize if supporting old APIs will be annoying or a problem. The only thing that comes to my mind is dotted paths in reverse() and url() -- once we can remove that stuff the URL resolver is quick a bit simpler and I know Marten was planning for that to be gone with his GSoC URL proposal (although I guess the new API can probably be implemented without support for that anyway).

For what it's worth, about 25% of survey respondents indicated when starting a new project they use LTS as opposed to the latest stable version. Are the benefits to that fraction of the community large enough to outweigh the costs?

[Philippe O. Wagner]

Hi all, 

    

I'd like to highlight these two points from the OP:

    

    * Eliminate code duplication from app to app and handle them in one central place.

    * Have a stable and testet single library that handles these compatible objects ...

    

It's not only about the version to version support, but also to follow the DRY principle and support other 3rd party apps. It follows the same idea like six does for Python 2.x to Python 3.x compatibility. 

    

BTW, good point Loïc about "providing an upgrade path" and keep the intermediate versions even not for production. We will keep this in mind for django-compat.

    

Tim, from my point of view it's not the number, 25% in this case, that matters. It depends who starts with the latest greatest version and who use the LTS.  I think agencies with a lot of customer and big, complex projects will/tend to build on a LTS version. These numbers depends totally who answers the survey.

[Tim Graham]

Just in case followers of this thread didn't see it, Collin proposed a new schedule in the "1.9 release planning thread" that I believe solves these concerns and doesn't delay deprecation removals as much as the schedule proposed by Loic. Please take a look and continue the discussion there. Thanks!