Ideas for a new DEP on long-term Django API compatibility

[Pkl]

Hello everyone,

I'm planning on writing a PEP on long-term API stability for Django.
Most of the rationale behind this kind of commitment is described here : https://www.freecodecamp.org/news/api-stability-is-cheap-and-easy/

The idea is that the django ecosystem has suffered a lot, over the past decade, from the numerous breakages happening at each (minor) release. I could write pages just about what "outsiders" like me endured regarding neverending URL patterns changes.

Lots of pluggable apps, and their own plugins and bridges, have become incompatible with each other due to sticking to different django versions, or to lack of Tox-like cross-version testing. Some undocumented breaking changes (like classes replaced by other ones but missing functionality, or laziness introduced without autoloading measures for backwards compatibility (RequestContext?)) also hindered misc projects, as seen in their bugtrackers.

With *less* work than currently, except small changes in procedures, we can revive the majority of existing packages (except python2vs3 troubles), currently maintained or not, and make upgrades much smoother, which will also improve the overall security of django-backed sites (since users won't fear upgrades anymore). We can thus reach the same regard for compatibility as most industrial-size ecosystems, without sacrificing the cleanness of the codebase, nor its maintainability.

For this, imho the best way is just have to include in Django official support for a companion library like https://github.com/pakal/django-compat-patcher , with which I've had immense success

There are several ways of doing this, and for this more thoughts are necessary:

- putting compatibility sets in the repository, as a django.compt submodule for example?
- putting them in an external repository, but strongly advising them in tutorials and otehr docs, like a prime "official project" of django?

There are probably aspects of the issue that I'm missing, as I'm not into django "core" development.
So your feedbacks are more than welcome!

Thanks,
regards,
Pakal

[Aymeric Augustin]

Hello Pakal,

If I understand correctly, you're proposing that:

1. The Django project should maintain a higher level of backwards compatibility.

2. This would be easier to achieve outside the main codebase, in a submodule or in a separate repository.

3. This would reduce the amount of work required for maintaining Django (triaging tickets, fixing bugs, adding features).

Backwards-compatibility is a major concern for the development of Django, as you can see from in contribution documentation, in the archives of this mailing list, in DEPs, etc. Maintaining more compatibility with the same effort or less should be uncontroversial. Now the question is to get consensus that your proposal will indeed achieve the stated goals.

I'm quite skeptical of the "same effort or less". The more behaviors Django maintains — and you're proposing essentially to maintain all behaviors that existed at all previous releases — the more complexity accrues. You can't solve that just by shuffling deprecation warnings or compatibility imports in another module (and it seems to me that it would be harder to maintain, for the reason the Python ecosystem usually eschews monkey-patching: non local effects). Keeping old ways around forever increases the barrier to mastering Django for people who haven't been writing Django code for ten years like you and me. It also increases the scope on which compatibility must be maintained, and that can get in the way of making Django better.

One way to frame the debate is: does Django aim at being a fossilizing ecosystem or a living one? Up to this point, we've chosen to maintain a living ecosystem. At a given point in time, all maintained libraries support 2 or 3 versions of Django (latest, previous, LTS). Things change a bit; unmaintained libraries that don't adapt fall out of favor; new libraries build upon the experience of previous ones and try to do better. One can see this as progress and a way to keep up with the moving web ecosystem. One can also see this as useless churn. As you call it "walk or die", it's pretty clear which camp you're in :-) I'd say there's truth in both!

As an experiment, let's just consider the first backwards-incompatible change from the latest release: https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes (I'm skipping the database backends changes because they're explicitly out of the backwards-compatibility policy, but we started documenting them for the maintainers of third-party database backends.) This is a good example. Django behaved in a non-Pythonic way; it didn't respect the Principle of Least Astonishment. Since we believe Django is alive and will have infinitely more future users than past users, we make the change. How can we maintain backwards-compatibility for this? We can try to detect if subclasses have all the actions of their parent classes and, if not, raise a warning. But then we raise inappropriate deprecations warnings in legitimate use cases where a subclass mustn't have some actions of its parent class, which is probably the use case for which this change was requested.

Experimenting with a third party module like you did is absolutely the way to go. You're already ahead of the usual advice :-) Keep in mind that the barrier for making a third-party project an official Django project or to merging it in Django is very high. With 1 fork and 2 stars, django-compat-patcher doesn't pass it yet.

Writing a DEP to formalize arguments on both sides is a valid idea. Don't forget the other side. In your freecodecamp article, you're spending two lines on the downsides of Compat Patchers, and this is not enough. How would Django communicate to users which backwards-incompatible changes are covered by Compat Patchers and which aren't? How would we respond to users asking for a Compat Patcher for a backwards-incompatible change for which it's impossible? Eventually, will every library document the list of Compat Patchers it requires or it's incompatible with?

As you can see, there's more than "blinding self-absorption" and "harmful psychological bias" here. Your freecodecamp article makes valid points. It also misses completely what's hard in maintaining backwards compatibility. I wish you hadn't found it useful to insult the work put into managing backwards-compatibility over the years and the people who did it.

Best regards,

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1d2f1399-3310-4b36-ae74-9581a0695002%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Florian Apolloner]

Hello World,

On Sunday, June 30, 2019 at 9:24:05 AM UTC+2, Aymeric Augustin wrote:

As an experiment, let's just consider the first backwards-incompatible change from the latest release: https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes ....

I think Aymeric is making a good point here on why any project like django-compat-patcher is ultimately doomed to be a failure. You simply cannot magically "fix" changes like this without knowing user intent. I can also remember the change from get_query_set to get_queryset (1.6 IIRC), you simply cannot magically patch that to work with multiple sublcasses and dynamic attributes. There will __always__ be situations where this will break in horrible and subtle ways. I'd argue that this would be worse then not being able to use a project that is not updated for a while. 

Pakal wrote in his initial mail:

With *less* work than currently, except small changes in procedures, we can revive the majority of existing packages (except python2vs3 troubles), ...

This is a bold argument for which I'd like to see some proof, also *less* work for whom?

And last but not least, if one assumes all your arguments hold true, then why isn't django-compat-patcher used by existing 3rd party libraries (At least according to public usage https://github.com/pakal/django-compat-patcher/network/dependents)? Either the usecase you are suggesting isn't as strong as you make it to be or 3rd party packages are maintained well enough for this to be not a problem? Personally I also think that a package that wasn't updated since 2015 should probably not be magically patched to theoretically work on a current Django.

Cheers,

Florian

[Pkl]

Hello everyone,


(apologies in advance for the long post, but there are lots of aspects to dig here, it's not for the pleasure of writing but a necessary to progress on the relevance of a DEP)

"As you can see, there's more than "blinding self-absorption" and "harmful psychological bias" here. Your freecodecamp article makes valid points. It also misses completely what's hard in maintaining backwards compatibility. I wish you hadn't found it useful to insult the work put into managing backwards-compatibility over the years and the people who did it."

I'd rather address that immediately: as you read in my article, "blinding self-absorption", "harmful psychological bias" and other strong words were all aimed at something very specific, the "feeling of return to purity" that a person could have when destroying compatibility shims. I persist and sign regarding this one: ruining the day of thousands of anonymous plugin/website maintainers all over the world should *always* be made under compulsion of heavy constraints, and with regrets; never lightheartedly.

Now, maybe my attempt at explaining the roots of nowadays' mass breakages by a cultural problem are all wrong (my bad), and there are other reasons behind the *apparently* reckless compatibility destructions that I witness left and right. But what are they? Since it's not technical inability, since it's not laziness, since it's not ideology, what makes that (as far as I'm concerned) the Django ecosystem fails on that aspect, contrary to loads of others "big" frameworks, runtimes, libraries etc?

If it's "we don't have enough resources", it's all understandable, but at least let's make it clear. The weirdest, in the whole story, is the global idea that everything is fine, that compatibility is already a "major concern" and treated so, that maybe it can be improved a little bit but no need to push it. It's not that way I see it. And a problem can't be addressed if it's not first recognized as so.

Now, I know that it requires a good amount of diplomacy to criticize a behaviour, or here a whole policy, without concerned people feeling attacked. I don't have this kind of talent (and probably won't ever), so all I can do is be honest with my view on the situation, and bring attempts at solutions. It's precisely because Django is a great piece of software (with its omnipresent lazy-objects, it's automatic admin and model forms, its migration system, its focus on security headers...) that having it scarifying itself is unbearable. So I apologize in advance for all the criticisms that expand below, but without them there'd be no need for any DEP or change in the first place.

I also want to make this clear: I didn't mean that breaking changes were never justified, just that they had to be the last option, and thoroughly explained. When the AngularJS team, after the years of hype, admitted that the whole designed was flawed, reset the whole framework as Angular version 2, and made great efforts to ease part-by-part migration towards the new behaviour, I felt unlucky but not angry. When trivial changes ruined tons of pluggable Django apps that did one thing and did it well, it was a different matter.

Before answering remarks, I think it's worth summarizing what the "problem" is. This is not lightly that I used the words "dependency hell". My experience with Django, purely regarding compatibility, has been abysmal compared to lots of other ecosystems (in C/C++, PHP, Jquery...) that I have crossed, and which didn't have the malleability of Python; I see several cumulated reasons explaining that :

- The pace of breakages: an arm's long list of new breaking changes introduced every 8 months, dropping at the same time all shims older than 2-3 years, is imo a too fast breakage pace, even by the Web's standard. When changes are mainly aesthetical, compatibility shims had better remain for very long periods of time, else the benefit/harm balance is just indefensible.
- More importantly, the *scope* of breakages: as long as it's only about fixing corner cases of the framework, and only a tiny portion of users is impacted (those who abused counter-intuitive and unpythonic behaviours), one can understand, and just hope not to be shot by the next upgrade; but here, we're dealing with changes that deliberately break about every existing application, even those which scrupulously followed official tutorials: ForeignKey's "on_delete" argument killed most migration files when becoming mandatory, moving url-related primitives left and right was as destructive as expected...
- The growing "less batteries included" philosophy. Valueable contrib modules were outsourced from the codebase, and other pluggable applications (likes CMS apps) follow this trend, putting all valuable features into external plugins. This "minimal core" philosophy would work fine if main applications counter-balanced it with a more delicate approach to breaking changes; but it is the contrary that happens, surprisingly. So in such fast-breaking environment, only monolithic applications can avoid turning into nightmares.
- The apparent needlessness of lots of changes, aggravated by the severe lack of explaining in release notes. As Linus said "And those reasons really need to be very good, and spelled out and explained". For sure I'm thankful that these release notes exist at first. I really am. Especially because without them, Django would be doomer than doomed. But even after digging the commits regarding almost 30 changes, and tracking some of them in tickets, most of them continue to look injustifiable to me. I have probably missed some incredibly deep pondering and reasonings backing them, but this miss is a problem in itself. When a perfectly suitable utility gets dropped in favor of a castrated stdlib one without a word of justification, when URL patterns (the very starting points of most pluggable apps) get broken 10 times in 5 years, it feels just like being spat on, "deal with it" style.

To illustrate, I'd like to redraw the history of said URL resolving and reversal system, as I've lived it (temporal order of changes not guaranteed).

- A few years years ago, you had to use the patterns() function, with tuple arguments including dotted strings, and an optional string prefix.
- Then you had to provide individual url() objects instead of tuples.
- Then the "url defaults", that everyone "star-imported" from as per official tutorials, changed their location.
- Then patterns() were replaced by a list of urls, or by a 2-tuple (!!) when including the app_name with it.
- Then url reversal by dotted strings was made impossible, one had to use named urls
- Then providing urls as dotted strings became impossible, views had to be imported and inserted directly.
- Then include() didn't accept a 3-tuple anymore, for some reason.
- As a side effect, include(admin.site.urls) was dropped, one had to directly use admin.site.urls in patterns
- Then url() got replaced by re_path(), when introducing simpler path matchers, and basic classes like RegexUrlPattern got removed.
- And recently lots of utilities have been continuously transferred from django.core.urls and django.core.urlresolvers to django.urls, breaking imports on their way, for a reason that most end users ignore.

My honest opinion is that, for a framework which is so reknowned and massively used, and for a set of features that are so central to every pluggable app, this avalanches of breaking changes is rather hard to sustain; especially considered the short lifespan granted to compatibility shims. Django is not just a website engine, but also a toolkit for building bigger blocks, more specialized frameworks, which themselves ought to get their own plugins, themes, bridges to interesting utilities... but it's hard to build big castles on too shaky grounds.

The worse is, the forward-evolutivity of the whole system has LOWERED in the process. URL Patterns() was a powerful factory design-pattern, which could return any object it wanted; this object could fake being a list, could customize its __add__() behaviour, could accept more arguments... And what do we get in the end ? A *tuple* ([patterns], app_name), maybe the most inevolutive, semantic-less container type in Python. How can I explain all this to a Django newbie who would want to upgrade his site after 5 years of stagnation? I fear that as breaking changes become more and more normal, less and less care will be taken to pick the most evolutive data types and designs along the road. Whereas long term compatibility also requires a habit of enforcing powerful scaffolding (like forcing user code to inherit from some base clases and their metaclasses), so that these can step in at any moment and fix future changes transparently.

The Way of the Cross that has been the python2to3 migration taught us that the pace of core developers is far from the pace of the silent mass of users (python3.0 was released more than 10 years ago, and I still get panicked requests of enterprises trying to migrate their huge codebase of python and C extensions); that there is always big optimism bias when introducing breaking changes (remember the u"" notation, that had to be reintroduced back because its removal was too much burden for project maintainers?); that preferring purity to practicality is always a bad idea (it seemed ugly to leave import aliases in the stdlib, as a result all big projects import from *six*, so it's the same ugliness and twice the porting efforts). I feel that the whole Django ecosystem needs more awareness on this point.

Another example is the Turbogears framework. It was like a pythonist's dream, mixing "best of breeds" libraries; but whenI tried to install it, years ago, after several hours of efforts I couldn't find a "known working set". Lots of small libraries broke compatibility as if they were alone in the world, ruining (for me, YMMV) a perfectly valid concept.

"If I understand correctly, you're proposing that:
1. The Django project should maintain a higher level of backwards compatibility.
2. This would be easier to achieve outside the main codebase, in a submodule or in a separate repository.
3. This would reduce the amount of work required for maintaining Django (triaging tickets, fixing bugs, adding features)."

Indeed. Note that just letting comptibility shims 2 or 3 times longer would also solve the problem, without resorting to new concepts.

"I'm quite skeptical of the "same effort or less". The more behaviors Django maintains — and you're proposing essentially to maintain all behaviors that existed at all previous releases — the more complexity accrues. You can't solve that just by shuffling deprecation warnings or compatibility imports in another module (and it seems to me that it would be harder to maintain, for the reason the Python ecosystem usually eschews monkey-patching: non local effects). Keeping old ways around forever increases the barrier to mastering Django for people who haven't been writing Django code for ten years like you and me. It also increases the scope on which compatibility must be maintained, and that can get in the way of making Django better."

My own experience, although much smaller than that of Django maintainers, differs quite much from these assertions. When I mentored a dozen students in internships, from a very basic knowledge of python and the web to an intermediate level in Django, I can't think of a single time where having some deprecated aliases and behaviours around hindered them. Beginners who delve into official (and up-to-date) tutorials are not supposed to meet legacy examples. If they integrate/audit third-party packages, and encounter legacy code, they might have some questions. With long-term compatibility, these questions will be answered on launch by explicit logging and warnings; with current deprecation policy, the scaffolding being forcibly destroyed, beginners will face completely abstruse crashes, or worse, will wonder for hours why their own code is never called. That's what I consider a steep learning curve (and a big waste of time).

Regarding the "scope" of maintenance, of course keeping more compatibility is harder than a "fire and forget" approach. But the benefit is huge for the community, my experiments show that fixers are quite easy to maintain (only once did I have to refactor an older fixer to account for a new breaking change on the same feature), and they don't seem to block the evolution of Django at all, since they "time travel". If a blocking case occurs one day, THIS might be a valid reason to drop compatibility on a particular feature, but at leats it'll have been justified (and will have lasted as long as possible). No one demands eternal compatibility, just a breakage pace compatible with little-maintained projects and big software. As of today I can't envision how we could have Wordpress-size ecosystems, with themes shops and auto-install plugins, without it falling immediately into dependency hell. A decade seems a good start to me.

"One way to frame the debate is: does Django aim at being a fossilizing ecosystem or a living one? Up to this point, we've chosen to maintain a living ecosystem. At a given point in time, all maintained libraries support 2 or 3 versions of Django (latest, previous, LTS). Things change a bit; unmaintained libraries that don't adapt fall out of favor; new libraries build upon the experience of previous ones and try to do better. One can see this as progress and a way to keep up with the moving web ecosystem. One can also see this as useless churn. As you call it "walk or die", it's pretty clear which camp you're in :-) I'd say there's truth in both!"

I don't really buy the dilemma between "living" and "fossilizing" here. Long term compatibility doesn't encourage project maintainers to be lazy, unless new versions of the framework bring nothing interesting (which would be a problem in itself). Better compatibility just allows low-community plugins - or those who block on other little-maintained dependencies - to keep working until a next surge of activity, or until the approach of the plugin is really deemed obsolete. It encourages people to upgrade their Django instead of dreading and delaying the next upgrade. It almost makes supporting multiple versions (and LTS...) useless, since people just have to "pip -U django" if security issues arise. It removes a LOT of burden from pluggable app maintainers, time that they can allocate to handling normal tickets or - who knows - pushing their best ideas towards Django itself. What I'm sure of, is that a "living" ecosystem doesn't have to stab half his packages (and definitely murder a good part of them) at each minor version release. The awesome https://djangopackages.org/grids/ sometimes looks like a half-cemetery, in which one doesn't compare packages by features, but by "what are the odd that it still works with my particular Django version"? This hasn't to be so. "Perfectionnists with deadlines" should avoid wasting time needlessly.

On the precise subject of compatibility, we should take inspiration from in-browser ecosystems like jQuery. One can take datepickers, grid displayers, notification plugins, sometimes 5 or 10 years old, they just works No need to read release nodes, to do dependency conflict resolution, to hack JS sources and rebuild, to fallback on less relevant but alive plugins. Except some rare cases (like the .live() deprecation in favor of .on()), one can just download the latest versions of big frameworks, stick them in its statifiles, and all works; in a web-browser ecosystem which is yet known for rushing like mad. Why couldn't Django offer the same thing, with all the magic Python has built-in (magic which keeps expanding, with keyword-only and soon position-only arguments, helping future compatibility shims)?
In the releases notes of the last 7 versions of Django, how many compatibility shims were dropped because they really hindered Django evolution? How many trivial shims were dropped, at the contrary, "because that's what we do"?

"As an experiment, let's just consider the first backwards-incompatible change from the latest release: https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes (I'm skipping the database backends changes because they're explicitly out of the backwards-compatibility policy, but we started documenting them for the maintainers of third-party database backends.) This is a good example. Django behaved in a non-Pythonic way; it didn't respect the Principle of Least Astonishment. Since we believe Django is alive and will have infinitely more future users than past users, we make the change. How can we maintain backwards-compatibility for this? We can try to detect if subclasses have all the actions of their parent classes and, if not, raise a warning. But then we raise inappropriate deprecations warnings in legitimate use cases where a subclass mustn't have some actions of its parent class, which is probably the use case for which this change was requested."

Thanks for showcasing this one change, it illustrates perfectly lots of ideas I'm trying to push forward.
This is here an example of non-trivial change; contrary to many fixers I had to code, which were just aliases and tiny wrappers - fixers asking for about zero thinking time and zero maintenance. This kind of change requires more brainstorming, but you are right on this aspect: if it's changed in-place, without any deprecation path, django-compat-patcher is not able to automatically fix it (though a fixer could expose a setting so that project maintainers indicate the ModelAdmins they want fixed). This is imho one particularly shocking and inimical change: a perfectly documented and widespread feature removed suddenly, without deprecation path if I understand correctly (nor the usual advice for lib maintainers on how to handle both behaviours), letting who knows how many Django users wondering why some controls disappeared from their website. That's a way of making people paranoid, or ensuring that the least experienced spend half their time digging on StackOverflow.

Some changes are just not fixable in-place. But I know about none which cant be fixed out-of-place. In this case you mention, we should have found another suitable English word (there are 170.000+ available I heard), or combination of words, to express a similar idea. For example "admin_actions". These admin_actions attributes would have been the new documented feature, with a properly expected and pythonic behaviour. The old "actions" attribute would have silently and slowly died of old age, keeping until the end their weird but getting-stuff-done behaviour. A compatibility shim in admin submodule (or outsourced to a Compat patcher) would have handled the legacy behavior for years and decades, without harming anyone.

What I'm describing here is similar to what happened to MIDDLEWARE_CLASSE=>MIDDLEWARES migration, I guess. I have no idea why these admin actions were changed in the most brutal way possible, but you are right on this point: like any compatibility shims, compatibility fixers need a tiny bit of collaboration from project maintainers, else sometimes the harm is irreparable.
These "unfixable cases" are maybe the main reason I'm trying to raise awareness about compatibility (that, and the fact that I'm worried to see people waste their time in bugtrackers discussing how to work around breakages). It's one thing to be confronted to breaking changed, it's another thing to be prevented from applying one's own compatibility shims.

I'd like to point that this kind of breaking chance is especially surprising as it goes against the own policy of Django : "All the public APIs (everything in this documentation) will not be moved or renamed without providing backwards-compatible aliases. [...] If, for some reason, an API declared stable must be removed or replaced, it will be declared deprecated but will remain in the API for at least two feature releases. Warnings will be issued when the deprecated method is called. [...] We’ll only break backwards compatibility of these APIs if a bug or security hole makes it completely unavoidable."
https://docs.djangoproject.com/en/2.2/misc/api-stability/
https://docs.djangoproject.com/en/2.2/internals/release-process/#official-releases

"Experimenting with a third party module like you did is absolutely the way to go. You're already ahead of the usual advice :-) Keep in mind that the barrier for making a third-party project an official Django project or to merging it in Django is very high. With 1 fork and 2 stars, django-compat-patcher doesn't pass it yet.

Writing a DEP to formalize arguments on both sides is a valid idea. Don't forget the other side. In your freecodecamp article, you're spending two lines on the downsides of Compat Patchers, and this is not enough. How would Django communicate to users which backwards-incompatible changes are covered by Compat Patchers and which aren't? How would we respond to users asking for a Compat Patcher for a backwards-incompatible change for which it's impossible? Eventually, will every library document the list of Compat Patchers it requires or it's incompatible with?"

Sure I would list these remarks, but haven't most be long answered by lots of other projects? Compatibility IS maintained by default, unless its is not possible for "spelled out and explained reasons" (security, obvious contradiction of behaviour, really too heavy development effort for too few users impacted...). The good new is that people worried about compatibility could submit patches to help this effort, instead of breaking their neck against official policies. And libraries don't have to know anything about compat patchers, they are just meant to aim for the latest version of the framework, and themselves seek wide support (eg. with the django-compat lib, not to be confused with DCP), letting project maintainers tweak their patcher config as needed (by looking at Django version requirements of dependencies, or better disabling fixers' families one by one until unit-tests break, or better again checking the warnings and the usage report of the patcher - the latter not implemented yet - to know what fixers must be left enabled).

I'm worried about the reversal of the burden of proof here, though. Historically, it has always been the role of people wanting to break compatibility, to justify their goal; to demonstrate that few people will be impacted; that proposed compatibility shims are wrong or unmaintainable; that the old behaviour is too harmful or blocks evolutions.
Now it's as if *I* had to demonstrate (in 2019) that a strong commitment to backwards compatibility is important; that Django users and projects all around are negatively impacted by these changes; that long-term compatibility is *not* hard, nor big waste of time, nor doomed.
What am I to do, pay for a world survey of Django developers? Wait for 10 years so I can tall "See? My websites are still compatible with Django 1.7, told yah"? Do big data or prophecies to compute how much time exactly maintaining compatibility shims will take to core devs? If it's a cultural axiom of the whole team that the current situation is just fine, it's not a technical issue anymore, it becomes like trying to convince a torero that corrida is bad.

"I think Aymeric is making a good point here on why any project like django-compat-patcher is ultimately doomed to be a failure. You simply cannot magically "fix" changes like this without knowing user intent. I can also remember the change from get_query_set to get_queryset (1.6 IIRC), you simply cannot magically patch that to work with multiple sublcasses and dynamic attributes. There will __always__ be situations where this will break in horrible and subtle ways. I'd argue that this would be worse then not being able to use a project that is not updated for a while."

I agree with you, in some cases, the dynamic nature of the language can play against compatibility. But once again, no one demands perfection, just a *sufficient* level of compatibility. I hadn't followed the get_query_set() removal effort (https://code.djangoproject.com/ticket/15363), I don't know what happend to the nice proposals of the ticket ("renamed_method_factory" and the likes), but all I know is that with rather limited efforts I've revived the vast majority of the modules I needed ; and upgraded to Django 2.2 my site, when major dependencies explicitely prohibited anything else than Django 1.11. That's not what I call a doomed failure ^^

" > With *less* work than currently, except small changes in procedures, we can revive the majority of existing packages (except python2vs3 troubles), ...
   
This is a bold argument for which I'd like to see some proof, also *less* work for whom?"

It's only a rough estimate, but I think that the time spent removing shims from the codebase is equal-or-above the rare bugfixing (or removal) of oldest compatibility fixers, for the corde dev team. Of course, I assumed the presence of deprecation paths. Labor-wise, even Compat Patchers can't beat the "Instant Death" policy that lead to the unfixable situation you both mentioned. Concerning third-party plugin maintainers, long term compatibility wouldmean a huge relief, of course, and that could "backfire" positively for everyone.

"And last but not least, if one assumes all your arguments hold true, then why isn't django-compat-patcher used by existing 3rd party libraries (At least according to public usage https://github.com/pakal/django-compat-patcher/network/dependents)? Either the usecase you are suggesting isn't as strong as you make it to be or 3rd party packages are maintained well enough for this to be not a problem? Personally I also think that a package that wasn't updated since 2015 should probably not be magically patched to theoretically work on a current Django."

Let's note that DCP is aimed at project maintainers, not library developers, who rather rely on django-compat and the likes. Without official support from Django, it'd indeed be bold of a library to impose DCP as dependency (instead of just targeting the latest Django version as it should). There is some download activity https://pypistats.org/packages/django-compat-patcher ; no idea if its bots, curious people or real users though.

Concerning projects, I see lots of possible answers though:

- I'm bad at (and hate doing) communication/marketing on my projects
- it's a quite new concept of "companion application for long-term compatibility", so people are naturally reluctant
- people naturally doubt the relevance and longevity of new packages, that are not born of figures of authority (look how asyncio overcame trio).
- tons of F.U.D regarding monkey-patching and even the possibility of having long term compatibility
- there is a cultural acceptation of the situation, of the resigned "necessity" to do useless forks, or waste time repairing regression with patch-ups jobs, as if it made software better
- indeed people staying close of "mainstream" packages can indeed do without DCP (I just did on my latest 30-days project); but what a have fun trying to build a CMS site with usual niceties on a non-monolithic framework...

I don't know if DCP is the best way to achieve long-term compatibility, I just know that it was easy to code (except the import alias mechanism), and worked like a charm for my dependency-full projects. And that when wandering on repositories I cross too many regression tickets, or even people complaining about the time it takes to upgrade (e.g. https://www.reddit.com/r/django/comments/7u84gj/django_release_schedule_and_python_3/ when I was seeking a release schedule image).

I know that on this issue the interests of Django core devs are not exactly the same as those of end users, but I hope some progress can be made somehow, at least to ensure that those who need compatibility can achieve it on their own.

Sorry for the long post,

regards,
Pascal Chambon

[Luke Plant]

Hi Pascal,

I know this is a very late reply, but there have been some things going round my head that I thought I should get down.

First, some positive changes I think we can make from your suggestions:

1. I think in our release notes, for breaking changes that might need some justification, we should link to the tickets and/or django-devs discussions behind them. Adding full reasoning in the notes themselves would likely bloat them far too much, but a link could be helpful for anything which is not obviously necessary.

2. We should change the wording in our API stability docs, which does indeed initially sound like a promise never to break working code that follows docs:

Django promises API stability and forwards-compatibility since version 1.0.

We do have this line later on:

If, for some reason, an API declared stable must be removed or replaced, it will be declared deprecated but will remain in the API for at least two feature releases. Warnings will be issued when the deprecated method is called.

But we do make changes at a rate faster than that document would make you think.

I suggest that as a contrast to the existing text on that page, we also point out that we are continually improving Django, and have a "(eventually) one way to do it" policy regarding how we plan APIs, which means we will remove old things, while always providing better alternatives. I'm happy to write a patch for this.

Second, there were things I wanted to say that might be useful to you (Pascal) in understanding why you are not likely to get very far with your current approach.

Your exaggerated tone is not likely to help you. You talk about Django's compatibility breakages as "ruining the day of thousands of anonymous plugin/website maintainers all over the world". When we make these kind of changes, we always do it with very clear and detailed upgrade notes, and almost always with several releases emitting deprecation warnings, as per our policy. And no-one is forced to upgrade immediately. Those who value stability above other things can use LTS, which gives them a 3 year release cadence, not 8 months, which is far from excessively fast compared to other projects.

My involvement in Django these days is primarily not as a core contributor, but as a user and library maintainer, and I've also contributed many patches to different 3rd party Django projects to fix them for Django upgrades. The impression that you give of Django being a continuously shifting sand is very far from my experience. The core features and structure of Django have remained remarkably stable over its history. A few import changes here and there are very easy to fix (and, more importantly, noisy and easy to diagnose), and other changes are usually straightforward. Normally upgrades to libraries I maintain to support the latest Django version require very little work, sometimes zero, and often I'm able to support 4 or 5 Django versions without a problem. Using something like django-compat-patcher, or some of the other similar libraries, would add more complexity than the few lines of code I need otherwise.

Even larger, more complex Django projects like Mezzanine, which use quite a few hacks and internals, are often able to span across two LTS versions.

With all this in mind, it's difficult to see how we are making so many people's lives miserable.

Instead of debating in abstract, or answering your points one by one, I thought a few illustrations might help you see the benefits of Django's approach. The first is the longest, it might be of historical interest to others.

1. A 14 year old Django project

I started work on my first Django project in September 2005, and it went live in March 2006. It has been running continuously since then - perhaps by now it can claim to be one of the oldest continuously running Django sites? I'd be interested to know if anyone can beat it...

Here's a snippet from one of my models.py nearly 14 years ago:

class Message(meta.Model):
    fromUser = meta.ForeignKey(User,
        verbose_name="from user",
        related_name="messageSent"
    )
    toUser = meta.ForeignKey(User, 
        verbose_name="to user",
        related_name="messageReceived")
    time = meta.DateTimeField("At")
    text = meta.TextField("Message")
    box = meta.PositiveSmallIntegerField("Message box",
        choices = MESSAGE_BOXES)
    
    def __repr__(self):
        #return str(self.id)
        return "[" + str(self.id) + "] to " + repr(self.get_toUser())  + " from " + repr(self.get_fromUser())
    
    class META:
        admin = meta.Admin(
            list_display = ('toUser', 'fromUser', 'time')
        )
        ordering = ('-time',)

Wow, just look at that! ORM stuff under a `meta` namespace. Admin options stored under the model META inner class. These funky `get_XXX()` accessors for related fields. (Not to mention my naming conventions and code formatting back then...) While I have memories of some of the big changes in syntax that happened pre 1.0, I had no memory that we use to do most of these things. Why? Because this project no longer does it this way. My brain thankfully reclaimed the space taken up by these old ways of doing things. If we developed Django according to the backwards compatibility philosophy you are suggesting, that would not have happened. Instead, my code would be just as bad/quirky today as was did back then, for two reasons:

1. Without being forced to upgrade, I probably wouldn't have done.

2. If Django devs had needed to keep supporting old ways of doing things for long periods of time, two things could have happened:

• they would have been very conservative about adding new things, because it adds complexity to Django itself, and because of the principle of "One Way To Do It". So the new improved ways often wouldn't even exist.
  
  
• or, they would have just piled on the new things while leaving the old. Quite quickly you get to a point where the quantity of cruft, and the need to support all the different legacy ways, means that adding new things becomes very difficult, and development would have stagnated and quality dropped. To use your illustration, you can't build a castle on a weak foundation.

When I work on this old project I find:

1. It is virtually indistinguishable from brand new Django code.

2. The code quality has improved over time, and the project has never been in better shape than now.

I have none of that "Oh no I've got to fix that old project, how did we use to do this stuff again?" feeling. This project is a breeze to work with, and feels completely modern. If I needed to pass it on to someone else, I would be less worried now about doing so than I ever have been. How many other 14 year old projects can you say such things about?

I could multiply examples. Take URLs. Here's the old code:

from django.conf.urls.defaults import *

urlpatterns = patterns('cciw.apps.cciw.views',
    (r'^thisyear/$', 'camps.thisyear'),
    (r'^camps/$', 'camps.index'),
    (r'^camps/(?P<year>\d{4})/?$', 'camps.index'),
    (r'^camps/(?P<year>\d{4})/(?P<number>\d+)/?$', 'camps.detail'),
    (r'^sites/$', 'sites.index'),
    (r'^sites/(?P<name>.*)/$', 'sites.detail'),
    (r'', 'htmlchunk.find')
)

Versus the current code:

from django.urls import path

from cciw.cciwmain.views import camps as camp_views
from cciw.cciwmain.views import sites as sites_views

urlpatterns = [
    # Camps
    path('thisyear/', camp_views.thisyear, name="cciw-cciwmain-thisyear"),
    path('camps/', camp_views.index, name="cciw-cciwmain-camps_index"),
    path('camps/<yyyy:year>/', camp_views.index, name="cciw-cciwmain-camps_year_index"),
    path('camps/<yyyy:year>/<slug:slug>/', camp_views.detail, name="cciw-cciwmain-camps_detail"),

    # Sites
    path('sites/', sites_views.index, name="cciw-cciwmain-sites_index"),
    path('sites/<slug:slug>/', sites_views.detail, name="cciw-cciwmain-sites_detail"),   

    # ...
]

There are many improvements, for example:

• Star imports have been removed. There are lots of good reasons why every Python code linter complains about these. Or would you rather still have them?
  
• In my editor I can put my cursor on a view function like `index`, do "jump to definition" and it does the right thing, because it is normal Python. Do you think dotted strings were better?
  
• My list of URLs is just a list, and can be manipulated with normal list methods and functions (which I often need to do). Do you think wrapping everything in `patterns()` would be an improvement? For the sake of an extensibility we might just use someday (though we didn't in well over a decade)?
• The addition of `path` is a great help, and has really cleaned up URL handling a ton, and even more in other areas of my project. I wasn't forced to make this change, but it's a great improvement. However, this is the kind of big addition that the Django developers only considered and were able to do because the Django codebase was not a pile of backwards compatibility hacks and cruft already. And yes, doing so meant we lost some undocumented classes which were implementation details, which you also seem to be complaining about.
  

Everything is simpler and more Pythonic - while at the same time the fundamental structure and abstractions are extremely similar, and every migration has been straightforward. It is unfortunate that we didn't have a crystal ball in order to optimize these migrations a bit more, or indeed get it right from the start. But that's life, we don't have crystal balls.

All of the changes with URLs have had thought through reasons. Often the changes fix issues that newbies consistently trip over (based on feedback from people who do a lot training of new Django users) and security issues we've had.

2. Wordpress

You mentioned PHP and Wordpress, and I would agree that their approach is quite different. But it also comes with huge costs. Thankfully I already wrote a blog post that covers pretty much everything I want to say, with quite a few comparisons to Django:

https://lukeplant.me.uk/blog/posts/wordpress-4.7.2-post-mortem/

A zero day vulnerability in which your web site gets defaced and possibly fully hacked is what I think of when I hear the phrase "my day was ruined", and that is the kind of vulnerability we've so far been able to avoid in Django.

Critical security vulnerabilities like the above are also just the tip of the iceberg - the same issues that cause them also cause all kind of bugginess which hurt you in much more every-day ways, but these costs add up.

3. AngularJS

I'm surprised you let AngularJS off the hook so lightly. One of my clients has a project with 25,000 lines of AngularJS code. Having looked at the detailed upgrade instructions, in the context of our application the first step essentially amounts to "first re-write your entire app" (to use components everywhere, which we are not). There are entire books devoted to this upgrade, it is a massive undertaking that requires a deep understanding of both AngularJS and Angular. The team is considering ditching everything and starting with a new framework. This is the kind of situation that leaves businesses in very bad situations, having to choose between either being able to deliver new features for the next six months to a year or rewriting and paying off the technical debt.

For my past and current clients, and all my side projects, I have never faced anything remotely closely to that choice with Django. Sure, we've had to schedule in some time for upgrades, especially if we've had a lot of dependencies, but doing the upgrades, and fixing most of the deprecation warnings too, for small to medium sized projects, usually has been work on the scale of hours, sometimes days — maybe a week of development work if we were ahead of the curve in terms of upgrading and had to submit patches to lots of 3rd party dependencies. But never months — never anything close to "senior management needs to get involved with this decision".

To sum up, Django has so far avoided the huge upgrade costs of something like AngularJS. We've never reached the stage where we've thought "It's too awful, we'll have to start again", and never had to write that blog post - exactly the kind of news which might indeed ruin your day as a web developer or project manager. This might have been luck at choosing the right abstractions while AngularJS had some bad luck, but whatever the explanation, it's a great thing. At the same time we've avoided the horrors of a codebase like Wordpress which has just kept going with layer upon layer of cruft. Old Django projects (that have been maintained), and the Django code base itself, remain a pleasure to work with. That's also why we're able to attract excellent developers, like our current Django Fellows, so that Django continues to improve.

And we're not done yet - Django appears to have quite a bit of life left it in still! I'm convinced that all of this has been due to it's approach to improvement, gradual backwards compatibility breakages, and not tolerating previous bad design, rather than despite it.

Hope that helps,

Luke

[Pkl]

Hello Luke,

thanks for your comments, mine are below

On Thursday, August 1, 2019 at 10:34:22 AM UTC+2, Luke Plant wrote:

Hi Pascal,

I know this is a very late reply, but there have been some things going round my head that I thought I should get down.

First, some positive changes I think we can make from your suggestions:

1. I think in our release notes, for breaking changes that might need some justification, we should link to the tickets and/or django-devs discussions behind them. Adding full reasoning in the notes themselves would likely bloat them far too much, but a link could be helpful for anything which is not obviously necessary.

Yes documenting the rationale behind breaking changes, via a ticket link or another medium, would be indeed a nice improvement.

 

2. We should change the wording in our API stability docs, which does indeed initially sound like a promise never to break working code that follows docs:

Django promises API stability and forwards-compatibility since version 1.0.

We do have this line later on:

If, for some reason, an API declared stable must be removed or replaced, it will be declared deprecated but will remain in the API for at least two feature releases. Warnings will be issued when the deprecated method is called.

But we do make changes at a rate faster than that document would make you think.

I suggest that as a contrast to the existing text on that page, we also point out that we are continually improving Django, and have a "(eventually) one way to do it" policy regarding how we plan APIs, which means we will remove old things, while always providing better alternatives. I'm happy to write a patch for this.

As you imagine, I'd rather have had a change in actual django policy, rather than a doc update to reflect the current status; but it'd be better than nothing. However, the second sentence "If, for some reason, an API declared stable must be removed or replaced, it will be declared deprecated but will remain in the API for at least two feature releases" is itself still misleading, since at least one recent major breakage violates it ; A pronouncement of the core dev team would be needed here: is this behaviour change a procedure error? Or is it really the new policy, to allow changes which create immediate silent errors, and prevent third-party compatibility shims from getting implemented?

Having "only one obvious way to day it" is a useful and common commitment in Python, but please note the "obvious". You won't have "one way to do it", ever. If people want to use sys.stdout.write(), or do a libc-cffi call to output text to console, instead of just calling print(), they will do so. And compatibility shims don't change anything on this matter: the "obvious", "standard" way of doing is just the one mentioned in official docs and which doesn't trigger DeprecationWarnings.

 

Second, there were things I wanted to say that might be useful to you (Pascal) in understanding why you are not likely to get very far with your current approach.

Your exaggerated tone is not likely to help you. You talk about Django's compatibility breakages as "ruining the day of thousands of anonymous plugin/website maintainers all over the world". When we make these kind of changes, we always do it with very clear and detailed upgrade notes, and almost always with several releases emitting deprecation warnings, as per our policy. And no-one is forced to upgrade immediately. Those who value stability above other things can use LTS, which gives them a 3 year release cadence, not 8 months, which is far from excessively fast compared to other projects.

Imho dependency hell is absolutely unrelated to detailed release notes or LTS versions. Dependency hell is just the fact that, when each minor version breaks compatibility, the app ecosystem inevitably explodes, especially in its outskirts (i.e the least popular packages), into a myriad fragments, each targeting specific django versions.
Your project remains on 1.11 LTS ? Nice, but one of your dependency (and its latest security bugfixes) require a new feature of Django 2.0; while another dependency is stuck on Django 1.10, either because the maintainer vanished, or because he is himself trapped by the requirements of a subdependency.

That's why Semantic Versioning was invented: so that developers can gradually update their own codebase (since each minor version brings improvements and deprecation warnings), while still ensuring all dependencies keep working under the same major versions (the project just has to install the latest minor version). The concept of "lax semantic versioning" sounds like a joke, it destroys the very purpose of this versioning scheme; would you put a "laxly waterproof watch" under water?

You mention Mezzanine, and indeed this CMS, being quite monolithic, should not have big troubles with the Django compatibility policy. More generally, as long as one sticks to a bunch of big packages (Mezzanine, Django-rest-framework, Django-extensions...), things work fine. Dependency hell occurs when you use modular architectures, and want to cherry pick the most relevant modules for your needs, even those who haven not invested into custom shims and multi-version testing, or which have not received updates for a few years.

Such modular architectures are precious, they ensure that users don't get trapped by feature choices of underlying frameworks; but they need this extra bit of carefulness about softare compatibility.

Take for example Django-CMS, a dozen external content type plugins (ckeditor, restructuredtext, videos, musics, image galleries...), a blog engine and its bridge (Zinnia), some utilities (easy-thumbnails, django-filer...); and there you are, struggling to make it all work despite large-impact breakages of the Django API. All this, not to do rocket science, just to build what we would call a simple website.

Let's note that Mezzanine had to fork its own dependencies, by the way, and in big part for what reason?
"At the time of grappelli_safe's creation, Grappelli was incorrectly packaged on PyPI, and had also dropped compatibility with Django 1.1 -- grappelli_safe was therefore created to address these specific issues
At the time of filebrowser_safe's creation, FileBrowser was incorrectly packaged on PyPI, and had also dropped compatibility with Django 1.1 -- filebrowser_safe was therefore created to address these specific issues."
Nice, more fragmentation.

Now, I can't talk about percentages. I just estimate that thousands of projects (a part of the total) are relying on modular frameworks like django-cms, or must make lots of small django apps play together. That's for these people that days are wasted, even if they don't realize how unneeded all this work duplication is. For sure, dependency hell sounds like a made-up or exaggerated concept, until you have to deal with it; to take a harsh metaphor, people of Europe see hippos as peaceful giants, but that's not the opinion of the 3000+ people they kill each year in Africa.

If there is an exageration somewhere, I bet it's rather on all that "backwards compatibility is too hard/costly/dangerous" speech I'm being served. My experiments at the contrary showed that backwards compatibility was easy to setup and maintain, provided you have some tooling under the hand (tooling that I did implement, like import aliasers). Why be worried about "Django codebase (becoming) a pile of backwards compatibility hacks and cruft", when compat-patchers precisely separate the compatibility concern from the rest?

As for the fear of "non local reasoning" expressed in other posts, I'm sorry, this doesn't apply here; compatibility fixers don't do evil things, like renaming some objects and removing others. They only ADD shims, they are activated on demand, they output logs and warnings to keep users informed, and if a shim breaks, you have maybe 99% chances the faulty wrapper/utility will appear in the traceback. So there is no backstabbing here.

By the way, some points I should have added to my initial article:

• The most widely breaking changes are, generally, the easiest to fix. Moving a core function/module will break every app, and can be fixed in <10 lines of code (tests included). Changing the corner-case behaviour of an ORM query method is hard to maintain, but very few users should be impacted by this change. That's why deprecation periods should be set on a case-by-case basis, and maybe 90% of breakages can be avoided just with some dozens of tiny long-term shims.
• If the framework code evolves too fast for its own maintainers to cope with compatibility shims, then it probably also evolves too fast for plugin and project maintainers to keep track, too. In this case, it should be called alpha-stage software, and people should wait for it to stabilize.

Thats why I don't buy your "Quite quickly you get to a point where the quantity of cruft, and the need to support all the different legacy ways, means that adding new things becomes very difficult, and development would have stagnated and quality dropped". When implementing DCP, I didn't experience this nightmare you describe. Shims can be dropped in rare cases, when they conflict with new features or introduce security holes, and deprecated features don't have to be supported on bugtrackers, so no hands are tied.

 

 Instead, my code would be just as bad/quirky today as was did back then, for two reasons:

1. Without being forced to upgrade, I probably wouldn't have done.

I have trouble following you here. Why do so many developers format their code, lint it (with pylint, xenon, doc8, mypy...), test it, package it, tox-integrate it, document it, internationalize it? Are they compelled to do so? Will python refuse to run if one of these best practices is not respected? No.
Developers just keep a codebase clean because they know it's the best way to keep it maintainable, and to attract users and contributors (+ the happiness feeling of properly done work).

Fixing (deprecation) warnings is only one of these best practices. Why would people skip this particular cleanup task, and ignore deprecation warnings but not other alerts or test failures? Especially if, as you mention, upgrading a single codebase to comply with new django standards is quite straightforward.

Note that if the new way of doing things seems less intelligent to your users than the old way (no additional clarity, no new features, no new bugfixes), there is probably a problem with the development process, and twisting user's arms by quickly removing old ways of doing is not a solution.

Backwards compatibility shims are not meant as a laziness appeal for active projects; but as a solution to dependency hell for modulare projects, and as a last recourse for low-maintainance plugins; it avoids the explosion of unmergeable forks while a project waits for a new maintainer.

By the way, your old code is not pretty, but it's understandable; and if your project is unmaintained, I'd rather have it not pretty BUT working, than not pretty and plain BROKEN.

There are many improvements, for example:

• Star imports have been removed. There are lots of good reasons why every Python code linter complains about these. Or would you rather still have them?
  
• In my editor I can put my cursor on a view function like `index`, do "jump to definition" and it does the right thing, because it is normal Python. Do you think dotted strings were better?
  
• My list of URLs is just a list, and can be manipulated with normal list methods and functions (which I often need to do). Do you think wrapping everything in `patterns()` would be an improvement? For the sake of an extensibility we might just use someday (though we didn't in well over a decade)?
• The addition of `path` is a great help, and has really cleaned up URL handling a ton, and even more in other areas of my project. I wasn't forced to make this change, but it's a great improvement. However, this is the kind of big addition that the Django developers only considered and were able to do because the Django codebase was not a pile of backwards compatibility hacks and cruft already. And yes, doing so meant we lost some undocumented classes which were implementation details, which you also seem to be complaining about.
  

To clarify, I'm not questioning the interest of individual django changes, just asserting that leaving the "old ways" around for a much longer period would have harmed no one and helped a lot. (btw, string references are still everywhere in Django, and a decent IDE like Pycharm will easily bring you from INSTALLED_APPS, MIDDLEWARES or template_name to the proper python/html file).

Regarding your last points, I think we all agree that Wordpress is a dumpster fire of security holes (with a jaw-dropping database schema furthermore), and that AngularJS was an epic failure: an experimental toy which was advertised as a next-gen production-ready framework, whereas it just didn't scale - and I fell for it too (but at that time, alternatives like ReactJS didn't feel so future-proof to me either).

Django avoided these big traps, and it's very nice. This doesn't impact the separate topic of backwards compatibility, in which Django could shine by just applying minor cultural changes imho.

To sum up, I know that I'm not likely to change the local mindset, on this "phobia of shims" as I dare calling it. Personally, I already solved my recurring problems, thanks to django-compat-patchers. I'd just like not to get spokes in the wheels, with non-shimmable changes, hence the pronuncement asked above. Can we all agree at least on this (and correcting the documentation as you nicely proposed)?

It'll still pain me to see so many old but nice plugins mass-forked for nothing, when 2 lines of project-level code would have fixed everything; but in software world, choosing the longest work path is a recurring problem (remember all these people who reimplement half their project in broken asyncio, when greenthreads would have solved their performance troubles for free until further changes are needed, or Trio/Curio would have at least paved a robust future?)

regards,

Pascal

[Tim Graham]

Pascal, you can read about the reason for the action actions change on the ticket https://code.djangoproject.com/ticket/29917 ...and on this mailing list... https://groups.google.com/d/topic/django-developers/-OWoYL_zryM/discussion. I hesitate to link to things like that in the release notes because the discussions can be long and somewhat difficult to follow. Rather, as appropriate, I tried to include a short summary of the rationale for backwards incompatible changes in the release notes.

In my years working on Django, I found it reasonable to occasionally make backwards incompatible changes if there's a consensus that a deprecation isn't feasible or worthwhile. Sometimes practicality beats purity, in my experience.

Luke's experience and views parallel mine. I think the current deprecation cycle works well.

[Adam Johnson]

Pascal,

I have had a short reply sitting in draft status for a while but not found the energy to read all of your posts.

I'd just like to chime in and echo the sentiment from Aymeric, Florian, Luke, and Tim. I worked with Django for nearly 5 years before being accepted onto the core team.

One of the main tasks I've done in my career has been upgrading Django - I did a recent count and I've done it at least 10 times on major applications (100k+ lines). Upgrading Django has always been easy in comparison with other changes. The Django upgrade process is miles above other Python packages, which often miss changelogs let alone use standardized versioning. I've always found the backwards incompatible changes in Django improve my project's code standards incrementally (as Luke said).

When reviewing features in Django I've normally erred towards putting everything through the deprecation cycle only to discuss it with other developers and decide it's not worth it. Practicality beats purity as Tim says.

The relative stability of Django is one of the things that attracted me to it and continues to keep me here, working on it and recommending it to others.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3adb7b80-c750-4cf9-95df-128f9f6fe7cc%40googlegroups.com.

--

Adam

[Pkl]

Pascal, you can read about the reason for the action actions change on the ticket https://code.djangoproject.com/ticket/29917 ...and on this mailing list... https://groups.google.com/d/topic/django-developers/-OWoYL_zryM/discussion. I hesitate to link to things like that in the release notes because the discussions can be long and somewhat difficult to follow. Rather, as appropriate, I tried to include a short summary of the rationale for backwards incompatible changes in the release notes.

In my years working on Django, I found it reasonable to occasionally make backwards incompatible changes if there's a consensus that a deprecation isn't feasible or worthwhile. Sometimes practicality beats purity, in my experience.

Luke's experience and views parallel mine. I think the current deprecation cycle works well.

The threads you pointed to, regarding "Admin Actions" semantic changes, are disturbing for the least. In just 5 lines of discussion and some wild guess / wishful thinking ("nobody must be using that"), a feature gets removed in a way which directly violates Django's official policy.

I guess what is practical for core developers is NOT the same as what is practical for users, and since there are many more users than developers of Django, it would be a big optimization to provide a larger compatibility window; even if it means creating a task force just for keeping a slight eye on compatibility shims. Anyway.

I have had a short reply sitting in draft status for a while but not found the energy to read all of your posts.

I'd just like to chime in and echo the sentiment from Aymeric, Florian, Luke, and Tim. I worked with Django for nearly 5 years before being accepted onto the core team.

One of the main tasks I've done in my career has been upgrading Django - I did a recent count and I've done it at least 10 times on major applications (100k+ lines). Upgrading Django has always been easy in comparison with other changes. The Django upgrade process is miles above other Python packages, which often miss changelogs let alone use standardized versioning. I've always found the backwards incompatible changes in Django improve my project's code standards incrementally (as Luke said).

When reviewing features in Django I've normally erred towards putting everything through the deprecation cycle only to discuss it with other developers and decide it's not worth it. Practicality beats purity as Tim says.

The relative stability of Django is one of the things that attracted me to it and continues to keep me here, working on it and recommending it to others.

--

This is somehow frustrating. I did my best to build a constructed argumentation, so as to demonstrate how compatibility is cheaper than expected, and how the current policy creates dependency hells for modular projects (not ALL projects), without bringing advantages compared to proper semantic versioning. All that to face red herrings ("but this new way of doing XYZ is better than the old one"), and some kind of "it works for me, so the problem doesn't exist in the first place". If I'm doing it all wrong, then I'd at least be pleased to know how you properly deal with these upgrades without diving into fork-fests.

With a long term policy, you could pick any package in https://djangopackages.org/grids/g/sms/ or https://djangopackages.org/grids/g/caching/ or wherever else; if it does one thing and does it properly it, would keep working, instead of breaking as soon as it isn't actively updated for *your* little version of Django. This ecosystem compatibility is what I call a killing feature for a framework. I might be wrong, but I feel this instability of the "engine" is a big factor why there is no famous Liferay-like or Plone-like app based on Django: sewing best-of-breed pluggable apps together to make an advanced multi-tenant portal would lead to an upgrade nightmare. Punishing users of small-community packages is just a way of encourage still more bandwagon jumping, at the expense of the ecosystem's diversity.

Anyway, please someone clear https://docs.djangoproject.com/en/2.2/misc/api-stability/ ; it is *very* far from the work process you all just described, so boasting about a strong commitment to API stability ends up hard to swallow.

regards,

Pascal

PS: this related thread popped recently: Django LTS support time ( https://groups.google.com/forum/#!topic/django-developers/Res0E-8DhtE )

[Carlton Gibson]

Hi Pascal. 

On 17 Aug 2019, at 18:21, Pkl <chambon...@gmail.com> wrote:

In just 5 lines of discussion...

Just for the record, we spent a good time discussion this change (to bring inheritance of Admin Actions in line with Python’s expected inheritance rules). We reviewed the entire history of the feature. We explored an alternative approach, which would have maintained BC. We put the discussion to the mailing list, where further people reviewed it, and we came to a decision, as a group, without any objections being raised. 

We didn’t make this decision lightly. 

On balance, we felt the breaking change was justified by the ease of adaption, the fact nobody could identify a single use-case of the existing behaviour in the wild — and to be fair, there have not been issues raised because of it, which we might have expected — and finally, that Django would be better for the change.  

I’m sorry that we have a different approach than the one you’ve asked for, but hopefully you can accept the reasons that people have given, and, more importantly, that decisions are made in good faith, for the best outcome that we, as a group, can find.

Kind Regards,

Carlton

[Jason Johns]

Pascal, I don't think anyone here disagrees with your overall goal to reduce the breaking changes.  As someone that got handed a 1.2 project with zero tests and updated it to 1.8, good test coverage to lay the foundation for reviving an ambitious project that went comatose before being prioritized again, the amount of breakage and updating required was surprisingly small.  It took about a month or so to do, and a quarter of that was setting up staging and prod environments on AWS.  The core devs here have done a great job, in my opinion, of keeping the breaking changes to a minimum, and making them in a thoughtful manner with an eye to the future.

Tech debt is a very real thing.  Keeping it under control is vital to the pace of improvement and updates for any project, large, middling or small.  Any project requires maintanence at minimum, just like owning a car or house.  If you buy a car and run it every day for five years, only filling up the gas tank and adding a splash of oil now and then,  you don't have any excuse to be upset at the mechanic or manufacturer for the size of the repair bill when things start breaking.  At least with software, its alot easier to get fixes out than with hardware.  You make a persuasive argument and have great passion, but the overall goal of enabling out of date and unsupported releases to remain around longer than they should be makes me feel really uncomfortable.  A spftware project is not buy once and forget, neither is a car.  And your package seems to encourage that lax practice to a level that I would be highly resistant at working at a company if I couldn't persuade management to prioritize keeping up to date and tech debt at a manageable level.

Open source has a pretty low barrier to entry, and django projects like the ones you listed on django packages are reflective of that.  Its easy to make a package and put it out, but maintaining and iterating and releasing take alot more in terms or time, effort and emotional state.  Fortunately, Python as a whole seems to be much easier on package devs than the JS community, but as I'm sure Carlton, Adam, Tim, and anyone that's published a package with requests for support can attest, it can get exhausting.  That's when packages just die on the vine.  Some are lucky to be resuscitated by another dev taking over, or forking the original, but that doesn't happen for the majority.  Deliberately going out of your way to ensure that old code which hasn't been looked at in however long and was last working on two LTS releases ago gives me a very squicky feeling in my stomach.  

[Aymeric Augustin]

Hello Pascal,

As we're getting to the end of useful arguments here, I'd like to ask you to step back for a minute and take a calm look at what you're writing. Imagine you were receiving such messages rather than sending them. Would you want to spend more time collaborating with the person addressing you that way and to receive more such messages? Perhaps this is another reason why you aren't getting as much support as you'd hope.

You did your best to provide nuanced feedback on a debatable backwards-compatibility issue — current behavior is obviously wrong but can't be fixed without a hard compatibility break:

some wild guess / wishful thinking

You spent thousands of hours thinking about what would be best for users of the framework — getting occasional thanks and only occasional abuse, as we're lucky to have a positive community:

what is practical for core developers is NOT the same as what is practical for users

You don't need backwards compatibility because you're comfortable living on the edge but you always take the time to create deprecation paths according to the compatibility policy:

even if it means creating a task force just for keeping a slight eye on compatibility shims

You do your best to discuss positively with an agressive contributor who started with a strongly-worded blog post and doesn't seem to process the responses he's getting:

All that to face red herrings

Here's literal belittling:

*your* little version of Django

And the cherry of the cake, stating core devs are secret sadists:

Punishing users

This isn't how we hold discussions here. You need to take a clue from how other people interact on this mailing list. Then you can put your energy and ideas to good use for improving Django.

From a pragmatic perspective, putting people off isn't a good way to get them to cooperate with you.

For example, Luke was open to amending the deprecation policy to better reflect reality. You should have suggested some concrete changes, erring on the side of caution. Instead you chose to repeat the same argument, except more agressively. This doesn't get us any closer to a documentation patch. In fact this reduces the likelihood that someone will choose to spend time writing that patch.

You last email sounds like you're throwing your toys and walking away because you didn't get full agreement with your position. However, that outcome was largely predictable when you were bringing up a position that diverges so much from the current practice. To create progress, it's up to you to create consensus around small steps and to create the corresponding pull requests, in that order.

Best regards,

-- 

Aymeric.

[Patryk Zawadzki]

As someone who has spent over 12 years working with Django, I'd like to offer an opposite point of view:

I love where Django is headed. I love all of those breaking changes that have to happen so we're not perpetually stuck with decisions from 2005.

What I truly miss is strong static typing support that would tell that when my code is not going to work before I even begin to test it. With static types and tools like mypy and pyright upgrading dependencies is usually a matter of a single afternoon.

[Dave Burkholder]

> I love where Django is headed. I love all of those breaking changes that have to happen so we're not perpetually stuck with decisions from 2005.

+100 to this!!

Django user since 1.6 running on python2.6. Upgraded to 1.7 and python3.4 at the same time, and _that_ migration was hard. Everything since then has been a cakewalk.

Mega-props to the caretakers of Django. Carry on...!

[Pascal Chambon]

Hello,

Just for the record, we spent a good time discussion this change (to bring inheritance of Admin Actions in line with Python’s expected inheritance rules). We reviewed the entire history of the feature. We explored an alternative approach, which would have maintained BC. We put the discussion to the mailing list, where further people reviewed it, and we came to a decision, as a group, without any objections being raised. 

That's good and reassuring news; if so, that's precisely another occurrence of "those reasons really need to be very good, and spelled out and explained". Most Django users will never see farther than the release notes, so if abrupt in-places changes occur without a rationale behind, it *appears* reckless (whetever really happened in the background). If release notes are considered too long already, wouldn't html tooltips/collapsibles be useful, so that one may know more about the rational behind changes?

Tech debt is a very real thing.  Keeping it under control is vital to the pace of improvement and updates for any project, large, middling or small.  Any project requires maintanence at minimum, just like owning a car or house.  If you buy a car and run it every day for five years, only filling up the gas tank and adding a splash of oil now and then,  you don't have any excuse to be upset at the mechanic or manufacturer for the size of the repair bill when things start breaking.  At least with software, its alot easier to get fixes out than with hardware.  You make a persuasive argument and have great passion, but the overall goal of enabling out of date and unsupported releases to remain around longer than they should be makes me feel really uncomfortable.  A spftware project is not buy once and forget, neither is a car.  And your package seems to encourage that lax practice to a level that I would be highly resistant at working at a company if I couldn't persuade management to prioritize keeping up to date and tech debt at a manageable level.

We agree on the necessity of updating code, and indeed DCP was never meant to encourage slacking but to solve external dependency conflicts. The latter - and the additional costs involved in solving them - are precisely used by management to refrain from upgrading codebases.

As we're getting to the end of useful arguments here, I'd like to ask you to step back for a minute and take a calm look at what you're writing. Imagine you were receiving such messages rather than sending them. Would you want to spend more time collaborating with the person addressing you that way and to receive more such messages? Perhaps this is another reason why you aren't getting as much support as you'd hope.

My initial article, as well as most of this ML discussion, is not small feature/bugfix request, but a rant on the whole deprecation process. The aim is to raise awarenesss, to shake minds, to push points as deep as possible, and to not let any falsehood or diversion slip by. As thus, it uses all kinds of strong words, of emphases, of rhetorical figures, of denounciations, to deliver the argumentation. I guess a rant will always look aggressive, even if it only states the most scientific facts.

So I'm sorry if it sounded insulting, but personally I don't mind facing adamant insistence, blunt statements, criticisms of any current state of affairs (whatever the amount of effort one might have put into getting there), and heated discussions, as long as they don't turn into ad hominem attacks; in the excepts that you pointed out, I was mainly stating my frank opinion on the (visibly incomplete) evidence that I was shown, and I would only have sounded hypocritical by masking the seriousness these issues have for me. Maybe some feel that Django as already having admirable commitment regarding API stability, alas some other languages and frameworks have set the bar much higher (farther than semantic versioning), and *they* are to be used as references (although too extended compatibility is considered harmful by some here).

Surely some of my words could have been better chosen ("punishing" shall be understood as "putting at a disadvantage", and "little version" as "minor version"), my bad, like some of yours ("throwing your toys" sounds condescending, though I couldn't care less); but I feel that politeness euphemisms and tone policing destroy the productiveness of crucial controversies like this one, instead of helping them. I'd rather be called names, but not see already-debunked untruths reappear in the conversation ("current behavior [...] can't be fixed without a hard compatibility break", "always take the time to create deprecation paths"...); this intellectual scrambling is what makes *me* stomach-sick, much more than old unmaintained packages.

For example, Luke was open to amending the deprecation policy to better reflect reality. You should have suggested some concrete changes, erring on the side of caution. Instead you chose to repeat the same argument, except more agressively. This doesn't get us any closer to a documentation patch. In fact this reduces the likelihood that someone will choose to spend time writing that patch.

Imho I was not rambling, I was reformulating and pushing the issue farther to try to understand what was really going on; and it proved useful because Carlton Gibson, Tim Graham and Adam Johnson eventually answered my questions on this famous "Admin Actions" change process .  
Thus my final analysis on it is that none of the promises made in "API Stability" doc are true anymore, that's why I propose its clearing (the other "Django’s release process" doc is here to describe what's actually occuring). Or one could reformulate this  "API Stability" doc with lots of "often", "unless it's deemed not worthy" and so on, but only the core dev group can express its official stance on this, not me. Is my analysis correct?

You feel that sugar-coating my assertions might have changed something to the adoption of my proposals, but considered that the current release process is completely praised by other contributors to the talk, and that dependency conflicts / fork-fests are considered a minor issue, I don't have this impression. If my convictions on all these subjects (the difficulty of setting up compatibility, the hazard of using old packages, the necessity of forcing users to upgrade all at once, the interest of highly modular architectures...)  are the polar opposite of people here, there is at the contrary nothing to regret - a change of release policy is a big and brutal thing anyway, not something to be done by small steps and small PRs.

So documentation changes only, it will blatantly be.

thanks,

regards,

Pascal

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/GgqjkYSqfR0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6b6f6b14-c3cc-4ba9-9d77-92ecaab23e11%40googlegroups.com.