Over the past few weeks, the core development team have been discussing how we can streamline our process to get more work done quicker. It's pretty clear that the bulk of the community wishes Django could move forward a bit faster [1], and we'd like to be able to deliver more awesomeness in less time. Ultimately, the goal will be to put out releases quicker, hit our deadlines more accurately, and be able to respond to community suggestions and patches in a more timely manner.
[1] This isn't unique to Django, of course; almost every open source project wishes they could release more features more quickly.
One way that we'd like to improve our speed is by modifying our existing backport policy. To refresh your memories, right now we backport any non-feature-containing patches to the previous release. So if we fix a bug on trunk, we backport the bug fix to the 1.3.X branch, where it would become part of the 1.3.1 release. This is a fair bit of work: it basically means we have to fix each bug twice.
The core team has come to a rough consensus and we're planning to drop this backport-everything policy. Instead, we'll only backport critical patches. That is, we'd only backport patches for:
* Security issues. * Data-loss bugs. * Crashing bugs. * Major functionality bugs in newly-introduced features.
In other words, we'd basically only backport bugs that would have prevented a release in the first place.
Practically, this means that if there's a minor bug in 1.3 -- for example, #15795, a bug with the representation of RegexURLPattern -- the fix *will not* be applied to the 1.3.X branch, and thus no 1.3.* release will contain the fix, even if it's fixed in trunk. To get the fix for these bugs, users will have to upgrade to 1.4.
The upside, of course, is that bug fixes will be quicker to apply, so we can fix more bugs, so we can get that 1.4 release out sooner [2]. Additionally, this'll give users more of an incentive to upgrade to the latest-and-greatest. This means that they get new features, and we (the development community) get to focus more clearly on moving forward, not maintaining older releases.
[2] Look for a proposed release date soon. Spoiler alert: it's sooner than you think.
We'd like to make this change effective immediately, but we also don't want to just issue this change by fiat. So we're requesting comments and feedback on this change. Do you like the idea? Why or why not? Will this policy make it more likely you'll contribute? Less likely?
> Over the past few weeks, the core development team have been > discussing how we can streamline our process to get more work done > quicker. It's pretty clear that the bulk of the community wishes > Django could move forward a bit faster [1], and we'd like to be able > to deliver more awesomeness in less time. Ultimately, the goal will be > to put out releases quicker, hit our deadlines more accurately, and be > able to respond to community suggestions and patches in a more timely > manner.
> [1] This isn't unique to Django, of course; almost every open > source project wishes they could release more features more quickly.
> One way that we'd like to improve our speed is by modifying our > existing backport policy. To refresh your memories, right now we > backport any non-feature-containing patches to the previous release. > So if we fix a bug on trunk, we backport the bug fix to the 1.3.X > branch, where it would become part of the 1.3.1 release. This is a > fair bit of work: it basically means we have to fix each bug twice.
> The core team has come to a rough consensus and we're planning to drop > this backport-everything policy. Instead, we'll only backport critical > patches. That is, we'd only backport patches for:
> * Security issues. > * Data-loss bugs. > * Crashing bugs. > * Major functionality bugs in newly-introduced features.
> In other words, we'd basically only backport bugs that would have > prevented a release in the first place.
> Practically, this means that if there's a minor bug in 1.3 -- for > example, #15795, a bug with the representation of RegexURLPattern -- > the fix *will not* be applied to the 1.3.X branch, and thus no 1.3.* > release will contain the fix, even if it's fixed in trunk. To get the > fix for these bugs, users will have to upgrade to 1.4.
> The upside, of course, is that bug fixes will be quicker to apply, so > we can fix more bugs, so we can get that 1.4 release out sooner [2]. > Additionally, this'll give users more of an incentive to upgrade to > the latest-and-greatest. This means that they get new features, and we > (the development community) get to focus more clearly on moving > forward, not maintaining older releases.
> [2] Look for a proposed release date soon. Spoiler alert: it's > sooner than you think.
> We'd like to make this change effective immediately, but we also don't > want to just issue this change by fiat. So we're requesting comments > and feedback on this change. Do you like the idea? Why or why not? > Will this policy make it more likely you'll contribute? Less likely?
> Thanks!
> Jacob
Hi All -
In general I prefer to think about this the other way around: If we have a most current stable release of product X, it makes sense to fix any bugs that come up in that product. For the sake of the next release, assuming the bug was not fixed or made obsolete by some other change, it then also makes sense to /forward port/ that bug fix to the current (unstable) development trunk. Calling it "forward porting" rather than "back porting" makes it sound better, too. :-)
This workflow fits quite well if you're using one of the DVCS mirrors of the Django repository, as it can be as easy as merging the latest release branch into default/master on a regular basis. There is obviously more to the story (running tests, dealing with the occasional merge conflict), but to me, it seems like the majority of the work is in fixing the original bug in the first place, not in applying that same fix to another branch of the code. If my assumption is true, we might as well apply the fix in both places once it has been made.
Being a non-core-committer perhaps I don't fully understand the additional work involved, but I do appreciate the focus on stability in Django's releases and I'd rather be forced to upgrade due to new features that I want, rather than some silly bug that'll never be fixed -- especially if a stable release that includes the fix I need is not even available yet. In other words, if a bug in a product has been fixed, it seems odd to me that no stable release of the product will include that fix.
What if the window for bug fixes was time boxed, so that bug fixes would be back/forward ported as usual up until 1-2 months prior to a new stable release (perhaps coinciding with the first release candidate)? This way at least we have some semblance of a stable release destined to include all bug fixes at all points in time, and the actual window in which a bug fix might not make it into the current official stable release is reduced.
On 19 avr. 2011, at 21:22, Jacob Kaplan-Moss wrote:
> We'd like to make this change effective immediately, but we also don't > want to just issue this change by fiat. So we're requesting comments > and feedback on this change. Do you like the idea? Why or why not? > Will this policy make it more likely you'll contribute? Less likely?
Hello Jacob,
I thought that the work required to backport a patch was relatively low compared to the work required to understand a ticket and review proposed changes. Not being a core developer, I'll take your word that it is "a fair bit of work".
As someone who manages development projects on a 1-2 year scale, I am +1 on this new policy. At some point, these projects stabilize and development turns into maintenance. We want to use the most recent release of Django at that point, and then stay on that branch: - more frequent releases means we could use releases during the "build" (rather than trunk) and still get fixes within a decent timeframe; - less changes in maintenance branches means it would be safer to update Django during the "run".
As someone who hacks on Django, the main factor preventing me from contributing more is the relatively low chance that someone will review or commit my patches. Reviewing patches written by others hits the same problem: more often than not, I upload an improved version, thus I can not mark the ticket as RFC. The backlog of unreviewing and uncommitted patches is quite discouraging. If this new policy helps core developers to review and commit patches faster, I'm also +1.
Perhaps augment the new policy by stating that contributed backports for bugs that are left out by this change will be favorably looked upon, and committed to the branch unless there is a good reason no to commit them. This still achieves the goal of making the routine bugfix commit cycle faster, but allows people that depend on stable releases to "scratch their itch", backport the important bugfixes, and get them as part of a "blessed" stable branch release.
On Tue, 19 Apr 2011 22:22:26 +0300, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
> Over the past few weeks, the core development team have been > discussing how we can streamline our process to get more work done > quicker. It's pretty clear that the bulk of the community wishes > Django could move forward a bit faster [1], and we'd like to be able > to deliver more awesomeness in less time. Ultimately, the goal will be > to put out releases quicker, hit our deadlines more accurately, and be > able to respond to community suggestions and patches in a more timely > manner.
> [1] This isn't unique to Django, of course; almost every open > source project wishes they could release more features more quickly.
> One way that we'd like to improve our speed is by modifying our > existing backport policy. To refresh your memories, right now we > backport any non-feature-containing patches to the previous release. > So if we fix a bug on trunk, we backport the bug fix to the 1.3.X > branch, where it would become part of the 1.3.1 release. This is a > fair bit of work: it basically means we have to fix each bug twice.
> The core team has come to a rough consensus and we're planning to drop > this backport-everything policy. Instead, we'll only backport critical > patches. That is, we'd only backport patches for:
> * Security issues. > * Data-loss bugs. > * Crashing bugs. > * Major functionality bugs in newly-introduced features.
> In other words, we'd basically only backport bugs that would have > prevented a release in the first place.
> Practically, this means that if there's a minor bug in 1.3 -- for > example, #15795, a bug with the representation of RegexURLPattern -- > the fix *will not* be applied to the 1.3.X branch, and thus no 1.3.* > release will contain the fix, even if it's fixed in trunk. To get the > fix for these bugs, users will have to upgrade to 1.4.
> The upside, of course, is that bug fixes will be quicker to apply, so > we can fix more bugs, so we can get that 1.4 release out sooner [2]. > Additionally, this'll give users more of an incentive to upgrade to > the latest-and-greatest. This means that they get new features, and we > (the development community) get to focus more clearly on moving > forward, not maintaining older releases.
> [2] Look for a proposed release date soon. Spoiler alert: it's > sooner than you think.
> We'd like to make this change effective immediately, but we also don't > want to just issue this change by fiat. So we're requesting comments > and feedback on this change. Do you like the idea? Why or why not? > Will this policy make it more likely you'll contribute? Less likely?
On Tue, Apr 19, 2011 at 4:22 PM, Jacob Kaplan-Moss <ja...@jacobian.org> wrote: > Hi folks --
> The core team has come to a rough consensus and we're planning to drop > this backport-everything policy. Instead, we'll only backport critical > patches. That is, we'd only backport patches for:
> * Security issues. > * Data-loss bugs. > * Crashing bugs. > * Major functionality bugs in newly-introduced features.
> In other words, we'd basically only backport bugs that would have > prevented a release in the first place.
Is this a policy against backporting fixes by default, or against the backports themselves?
Let me put it in another way with an example: you don't want to do the backport, and it's your policy, and it sounds like a good idea. But I'm using 1.3 in production and there's a bugfix I really want, so I do the backport (and write the code, tests, docs). If I submit this to the issue tracker, is there a chance my patch will get into the next minor release, or you won't even consider it?
The answer for this should be included in the documentation, it's useful for end-users and also for people reviewing patches
> I'm using 1.3 in production and there's a bugfix I really want, so I > do the backport (and write the code, tests, docs). If I submit this to > the issue tracker, is there a chance my patch will get into the next > minor release, or you won't even consider it?
Even with an accepted (or even ready-for-checkin) ticket containing all what's needed committers have some non-trivial amount of work to do. As I understand intention of the new policy it's this work that they want to decrease.
In such situation at my company we just patch our internal Django package. In fact we often patch an immediate problem locally first and only then bring the patch to discussion upstream. Much less pressure for everyone. I think you should either live with official stable releases or be ready to maintain your own if you choose to live on trunk.
Another issue with regards to backporting bug fixes that Jacob didn't mention is the fact that bug fixes often introduce subtle regressions and backwards incompatibilities.
Personally, I find that backporting a fix very often takes only a minute, by appropriate use of DVCS features (e.g. hg transplant) and some scripting. (This doesn't include the time for running tests again, and doesn't include cases where the merge fails or the patch has to be effectively re-written). However, the problems caused by introducing these regressions onto supposedly stable branches can be much more time consuming and tricky.
By keeping backporting of bugs down to a minimum, we avoid these problems, and help keep our stable branch genuinely stable. Not introducing regressions is much more important than fixing existing bugs, since most people will have worked around existing bugs where necessary for their needs anyway.
Regards,
Luke
-- "In my opinion, we don't devote nearly enough scientific research to finding a cure for jerks." (Calvin and Hobbes)
> Another issue with regards to backporting bug fixes that Jacob didn't > mention is the fact that bug fixes often introduce subtle regressions > and backwards incompatibilities.
> Personally, I find that backporting a fix very often takes only a > minute, by appropriate use of DVCS features (e.g. hg transplant) and > some scripting. (This doesn't include the time for running tests again, > and doesn't include cases where the merge fails or the patch has to be > effectively re-written). However, the problems caused by introducing > these regressions onto supposedly stable branches can be much more time > consuming and tricky.
> By keeping backporting of bugs down to a minimum, we avoid these > problems, and help keep our stable branch genuinely stable. Not > introducing regressions is much more important than fixing existing > bugs, since most people will have worked around existing bugs where > necessary for their needs anyway.
+1 to all of this. IMO the bigger issue is the regressions that inevitably get introduced when we backport all bugfixes, not the extra committer time to do backports (which I also agree I haven't found problematic).
+1, I agree with Carl and Luke. The issue here is that for non-showstopper bugs, users have probably found (or may even be relying on!) the existing behavior. Keeping the "stable" branch more stable by only changing things when there's a serious issue seems to be a positive improvement.
I think this point means that even community supplied patches to minor issues probably don't belong in the stable release. It's easy enough to maintain a patched local version for your own particular itches. Better stability and fewer regressions from patches related to minor issues seems like a fair tradeoff.
I am in favor of a more open policy regarding backporting docs fixes. We expect users to be developing using the stable docs, and if we have identified a deficiency there, we should fix it so that they will benefit before the next major release.
That's certainly a move in the right direction so +1 from me too. The
problem of
backporting correlates with how much time passes between any release
i.e. long time between releases gives bigger pita with backporting.
Even more so because you have several version control systems etc.
etc. (not going to rant about that fact :). What if we'd move release
dates so close that maybe backporting isn't an issue anymore?
What's really stopping us from ditching all but one DVCS and have
commit IDs/tags/whatever on a weekly basis that we advertise as "good"
or "stable" or whatever? Is Django so big and users are so cemented in
how to think about and use it that we must at all cost carry forward
the several decades old notion of having a fat release every now and
then (even if it goes down to several weeks, the procedures won't
change, just the backport pita might get a little smaller imo)? Don't
know, just thinking aloud ...
On Wed, Apr 20, 2011 at 3:32 PM, Markus Gattol <markus.gat...@gmail.com> wrote: > That's certainly a move in the right direction so +1 from me too. The > problem of > backporting correlates with how much time passes between any release > i.e. long time between releases gives bigger pita with backporting. > Even more so because you have several version control systems etc. > etc. (not going to rant about that fact :). What if we'd move release > dates so close that maybe backporting isn't an issue anymore?
There is a lot of effort in making a release -- ensuring that it is well-tested and stable. While it might be possible to shorten cycles, I don't think that removes the need for support of old versions. Anyway, a faster release cycle while keeping the existing support policy would effectively shorten the time to upgrade and place pressure on downstream cycles.
> What's really stopping us from ditching all but one DVCS and have > commit IDs/tags/whatever on a weekly basis that we advertise as "good" > or "stable" or whatever? Is Django so big and users are so cemented in > how to think about and use it that we must at all cost carry forward > the several decades old notion of having a fat release every now and > then (even if it goes down to several weeks, the procedures won't > change, just the backport pita might get a little smaller imo)? Don't > know, just thinking aloud ...
It's funny, early in the history of the project, we tried to argue that position and regularly got requests for "stable" releases despite protests that trunk was, in fact, stable. Even so, the need for releases is more about community cohesion and less about technical limits. Support gathers around releases. And there are many people running django in prod who are not active contributors -- the answer to these people can, with releases, be "read the release notes". Without releases, what is a reasonable answer?
On Wed, Apr 20, 2011 at 3:56 PM, Paul McMillan <p...@mcmillan.ws> wrote: > +1, I agree with Carl and Luke. The issue here is that for > non-showstopper bugs, users have probably found (or may even be > relying on!) the existing behavior. Keeping the "stable" branch more > stable by only changing things when there's a serious issue seems to > be a positive improvement.
Okay, I do think the regression issue makes a much stronger argument than the developer time issue.
I'd be more comfortable if the policy stated that any new bugs introduced by the last release would be backported to that release. It's possible that "major functionality bugs in newly-introduced features" will equate to virtually the same thing, but I'm not clear what would constitute a major functionality bug (it sounds big, and like it might be a difficult criterion to meet).
On the other hand, if we've all been living with a trivial bug since 1.0, we can probably live with it for just a little longer, and backporting certainly doesn't sound worth the potential risk of a regression.
> I'd be more comfortable if the policy stated that any new bugs > introduced by the last release would be backported to that release. > It's possible that "major functionality bugs in newly-introduced > features" will equate to virtually the same thing, but I'm not clear > what would constitute a major functionality bug (it sounds big, and like > it might be a difficult criterion to meet).
I like this. In general we treat regressions as release blockers anyway, so this fits with the "backport if it would have blocked the release" spirit of the proposal.
> Okay, I do think the regression issue makes a much stronger argument > than the developer time issue.
> I'd be more comfortable if the policy stated that any new bugs > introduced by the last release would be backported to that release. > It's possible that "major functionality bugs in newly-introduced > features" will equate to virtually the same thing, but I'm not clear > what would constitute a major functionality bug (it sounds big, and like > it might be a difficult criterion to meet).
Hmm, Jacob didn't specifically mention regressions, though in our discussions on django-core we did include them. Was that a deliberate change Jacob? Or perhaps regressions are covered by this sentence in your email:
> In other words, we'd basically only backport bugs that would have > prevented a release in the first place.
which certainly does include regressions introduced since the last release.
Luke
-- "I regret I wasn't born with opposable toes." (Calvin and Hobbes)
On Wed, Apr 20, 2011 at 5:24 PM, Luke Plant <L.Plant...@cantab.net> wrote: > Hmm, Jacob didn't specifically mention regressions, though in our > discussions on django-core we did include them.
Yup, sorry - was moving too fast. Regressions, clearly, get backported -- if something worked in 1.2, it should work in 1.3 unless we've deliberately removed it.
