The locmem patch and development progress
Daniel Brandt
Check out http://code.djangoproject.com/ticket/3012 for a pretty
critical patch to the locmem cache backend. The patch is accepted and
ready for checkin, and more than two weeks have passed since. The bug
itself was filed in february. What am I missing here? I ran into this
problem every single day, until I found that ticket and the patch.
I love django.. I have spent roughly 3½ months working with it with
far better results than I initiallty thought and I see no good
alternative to switch to.
A lot of times when people complain about that it takes a long time to
get something from being a ticket in trac to be commited to the tree,
the reply is that it has to do with keeping django stable, assuring
trunk is usable at all times. A nice mindset, sure, but what more is
needed from the ticket above to get it commited? It has already been
approved!
I do not wish to step on anyones toes here, but this is a major
concern for me. I am about to move a really large existing site to
django in the next 6 months and I know I will probably write and
submit patches to django during that time, but I am starting to loose
the motivation to contribute back. What is the use of writing patches
when it takes such a long time before it's committed or even
concidered for inclusion, even for critical stuff like the locmem
patch above. It is demotivating, and I think more people than me think
so too. What is needed to speed up the development while maintaning
your standards? Or perhaps another approach to the code base is
needed? More committers is needed, I'm sure, but those need to grow
out of the community effort, proving themselves, and that obviously
takes a lot of time.
With the quick growth of django users lately I'm sure the amount of
reported bugs and submitted patches will grow, and django development
needs to get up to speed to stay on top of the situation.
As it stands, I don't think it's clear enough to outsiders if the
percieved slowness of development is intentional or due to not enough
people helping.
If anyone gets upset reading this. please accept my apologies, I am
not out to troll. I am just trying to get a grip on how my help would
be most effective as I dig deeper into django.
Regards,
Daniel
Michael Radziej
>
> Something has been bugging me for a while..
>
> Check out http://code.djangoproject.com/ticket/3012 for a pretty
> critical patch to the locmem cache backend. The patch is accepted and
> ready for checkin, and more than two weeks have passed since. The bug
> itself was filed in february. What am I missing here? I ran into this
> problem every single day, until I found that ticket and the patch.
I can only tell you that two weeks with Easter in between isn't really that
long. It used to be a lot longer, so Django has become much better ;-)
Anyway, I can only recommend that you simply apply the patch to your own
installation (and report your results in the ticket). A patch management
system like quilt can help if you need several patches. I've been doing this
since I ran into similar problems (with different bugs, of course), and
never looked back.
Michael
--
noris network AG - Deutschherrnstraße 15-19 - D-90429 Nürnberg -
Tel +49-911-9352-0 - Fax +49-911-9352-100
http://www.noris.de - The IT-Outsourcing Company
Vorstand: Ingo Kraupa (Vorsitzender), Joachim Astel, Hansjochen Klenk -
Vorsitzender des Aufsichtsrats: Stefan Schnabel - AG Nürnberg HRB 17689
Jacob Kaplan-Moss
> Something has been bugging me for a while..
This type of complaint seems to come up every few months. I'm always tempted to
ignore it because I have a hard time responding in a way that doesn't end up
making me sound like a dick; usually that temptation wins out.
I'm gonna give an answer a try this time. I'll try as hard as I can to not be a
jerk -- that's totally not my intention -- but if I fail I hope you'll forgive
me.
I completely feel your pain, by the way. I only maintain one open source
project, but I use literally hundreds. I've filed tickets and bugs against many
other projects, and as you might expect the results run the gamut from "fuck
you"[1] to being checked in instantly. So I understand the temptation to try to
drum up more forward motion in any way possible.
Thing is, despite being cathartic, these "why hasn't my bug been fixed?" emails
really don't accomplish anything.
Like the vast majority open-source projects, Django is 100% volunteer-driven.
Every single bit of work is done by somebody in his or her spare time. We all
have some combination of full-time jobs, wives, families and hobbies competing
for our time. Obviously spending time on mailing lists is Yet Another Time Suck.
I know this isn't your intention, but when I see an email like yours I feel like
I'm being reprimanded for having a real life. I suspect most other developers
feel the same way.
I think it's very easy to underestimate the amount of work required in
checking in a patch. It seems like a simple handful-of-lines patch like yours
should be a no-brainer, but there's a whole bunch of steps I (or any
other bug fixer) has to go through before we can check a fix in:
1. Read through the patch -- every single line -- to make sure it matches
our coding standards, doesn't contain obvious errors or types, doesn't
change a stable API, etc.
2. Asses the impact of the patch on the rest of the system. There are
often subtle or unseen consequences of even simple patches.
To use yours as an example: if applied, the locmem cache would now use
picking instead of simple object reference to store cached objects. This
(a) means the patch will slow down the performance and (b) will
cause any code relying on the ability of locmem to cache unpickleable
objects. Obviously those aren't deal-breakers, but I have to look
for things like that.
Even a one-line patch can cause ripples throughout the codebase,
and they have to be considered before the patch is checked in
3. Look for any related bugs in the tracker, and assess how this fix
might affect them. There can often be a handful of related bugs that
all need to fixed as one.
4. Proofread any associated documentation, or possibly write some if
the patch doesn't include it.
5. Check that the patch includes tests and that the tests indeed
cover the new code paths introduced.
6. Apply the patch.
7. Run the regression tests against every supported version of Python
with every database backend available.
8. Finally, check the patch in.
I'm not complaining about this! I find checking in patches to be the **best**
part of working on Django, but it's a major time-suck. If I can't set aside at
least an hour to look at bugs, I likely just won't get anything done.
And adding new developers to a project is *hard*. We want to be completely sure
that we trust someone before giving them commit access. Sure, we could switch to
a model like PHP where anyone with a patch gets commit access, but then we'd
risk turning out like, well, PHP.
Finally: this is open source. Nothing prevents you from modifying your
code in any way you want; you don't need my permission or the checkin
of your patch in order to use it.
Once again I'm very sorry if any of the above comes off as dickish. I
really do feel your pain, and I wish more than anything that I could
spend more time on Django (while still being able to pay the
mortgage).
Working in open source at any level can require almost saintlike patience.
Jacob
[1] No prizes for guessing which project this was.
Jonathan Daugherty
# with every database backend available.
Do you have a resident buildbot? That could be used to run the
regression tests on (all pythons) x (all databases). It would still
take time, of course, but it could at least be automated[1][2].
(How is it done currently, if not with that sort of automation?)
HTH,
[1] http://buildbot.sourceforge.net/
[2] http://www.python.org/dev/buildbot/
--
Jonathan Daugherty
http://www.b-tree.org
Nicola Larosa
> I've filed tickets and bugs against many other projects, and as you
> might expect the results run the gamut from "fuck you"[1] to being
> checked in instantly.
> [1] No prizes for guessing which project this was.
I'll have a shot anyway: RoR? ;-}
http://www.flickr.com/photos/doesrails/128015501/in/pool-canadaonrails/
> We all have some combination of full-time jobs, wives,
That would be "spouses", even in the case all Django committers should be
heterosexual males. ;-P
> Once again I'm very sorry if any of the above comes off as dickish.
It doesn't to me. You may want to save a copy of this message, it may come
out useful next month. ;-)
--
Nicola Larosa - nicola...@gmail.com
Crypto is like an ATM that only lets you get money after you authenticate
yourself with your card and PIN. DRM is like some kind of nefarious goon
hired by the bank to follow you around after you get your money out,
controlling how you spend it. -- Cory Doctorow, December 2006
Michael Radziej
>
> # 7. Run the regression tests against every supported version of Python
> # with every database backend available.
>
> Do you have a resident buildbot? That could be used to run the
> regression tests on (all pythons) x (all databases). It would still
> take time, of course, but it could at least be automated[1][2].
Really: I think it would be a great help if somebody puts this together
and simply tested each commit. I thought about doing this on myself, but didn't
because of ... lack of time.
;-)
Daniel Brandt
>
> On 4/18/07, Daniel Brandt <daniel...@gmail.com> wrote:
> > Something has been bugging me for a while..
>
> This type of complaint seems to come up every few months. I'm always tempted to
> ignore it because I have a hard time responding in a way that doesn't end up
> making me sound like a dick; usually that temptation wins out.
>
> I'm gonna give an answer a try this time. I'll try as hard as I can to not be a
> jerk -- that's totally not my intention -- but if I fail I hope you'll forgive
> me.
First of all: Thank you a lot for taking the time to respond..
<..>
> Like the vast majority open-source projects, Django is 100% volunteer-driven.
> Every single bit of work is done by somebody in his or her spare time. We all
> have some combination of full-time jobs, wives, families and hobbies competing
> for our time. Obviously spending time on mailing lists is Yet Another Time Suck.
>
> I know this isn't your intention, but when I see an email like yours I feel like
> I'm being reprimanded for having a real life. I suspect most other developers
> feel the same way.
I understand exactly what you are saying here (family man myself),
open source is a "hard hat zone", and it wares you out at times taking
all the comments and keep on working.
> I think it's very easy to underestimate the amount of work required in
> checking in a patch. It seems like a simple handful-of-lines patch like yours
> should be a no-brainer, but there's a whole bunch of steps I (or any
> other bug fixer) has to go through before we can check a fix in:
Just to be clear, I'm not the author of the patch, I simply found it useful.
<..>
> And adding new developers to a project is *hard*. We want to be completely sure
> that we trust someone before giving them commit access. Sure, we could switch to
> a model like PHP where anyone with a patch gets commit access, but then we'd
> risk turning out like, well, PHP.
>
> Finally: this is open source. Nothing prevents you from modifying your
> code in any way you want; you don't need my permission or the checkin
> of your patch in order to use it.
Fully agree.. What I am interested in finding out is if this really is
a good direction to turn, maintaning what essentially is a private
fork, and then use it in a production environment? The issue with it
being in the "official" django tree is that there are alot more people
out there using that codebase and improving on it. Keeping my own
fork would mean less to no peer review.
> Once again I'm very sorry if any of the above comes off as dickish. I
> really do feel your pain, and I wish more than anything that I could
> spend more time on Django (while still being able to pay the
> mortgage).
Man, I'd pay your mortgage if I could as a thank you for helping to
bring dajngo about. Thank you again for taking the time to respond,
this answered of things I've been thinking about..
D
Vinay Sajip
> > checking in a patch. It seems like a simple handful-of-lines patch like yours
> > should be a no-brainer, but there's a whole bunch of steps I (or any
> > other bug fixer) has to go through before we can check a fix in:
>
> > that we trust someone before giving them commit access. Sure, we could switch to
> > a model like PHP where anyone with a patch gets commit access, but then we'd
> > risk turning out like, well, PHP.
>
> > Finally: this is open source. Nothing prevents you from modifying your
> > code in any way you want; you don't need my permission or the checkin
> > of your patch in order to use it.
>
> Fully agree.. What I am interested in finding out is if this really is
> a good direction to turn, maintaning what essentially is a private
> fork, and then use it in a production environment? The issue with it
> being in the "official" django tree is that there are alot more people
> out there using that codebase and improving on it. Keeping my own
> fork would mean less to no peer review.
>
I think all the points being made are valid. There are people who want
to help, and to try to reduce the bottlenecks and the pressure on the
core developers wherever they can. We all acknowledge what a hard job
it is to keep on top of things with a limited number of hours in the
day.
What's the thinking on documentation-only patches? While they are just
as worthy of review and critical assessment as code patches, there is
less of a concern about affecting trunk stability, and less impact
analysis work needed. Could there be a more streamlined process to get
them reviewed and into trunk? I couldn't see anything in the current
tracker that allows such patches to be identified and fast-tracked.
Another thing I've noticed is that tickets with small, low-risk
patches (which in theory should be easier and quicker to close) are
not that easy to single out in the tracker, sharing equal space in the
Trac reports with weightier patches which are harder to assess and
finalise. Closing out the easier tickets quickly would benefit
everyone by improving the signal/noise ratio - how could this be
addressed?
I've only recently started using Django (yay!) and been trying to help
out triaging tickets, suggesting patches etc. in accordance with the
documented guidelines. Hope I'm not stepping on any toes. Please set
me straight if I've missed anything obvious.
Regards,
Vinay Sajip
Malcolm Tredinnick
Not true. We try to be very careful about the documentation we commit to
the tree. Almost no documentation patch goes into the tree without
changes -- usually it gets edited or corrected once by the person doing
the original commit and once again by Adrian as he edits all of the doc
changes.
> Could there be a more streamlined process to get
> them reviewed and into trunk? I couldn't see anything in the current
> tracker that allows such patches to be identified and fast-tracked.
The core developers all have a good intuitive understanding of which
areas and tickets are going to be more time consuming than others. If
you sit down and work out the numbers, you'll find that most critical
documentation patches do go in very quickly, simply because they *are*
easy to review, fix and commit.
There are some documentation patches still uncommitted because either
they require more work, aren't necessarily the right approach or are
documenting things that are unstable or not going to be around and we
haven't quite made a decision yet. So these types of areas do get
implicitly fast tracked.
> Another thing I've noticed is that tickets with small, low-risk
> patches (which in theory should be easier and quicker to close) are
> not that easy to single out in the tracker, sharing equal space in the
> Trac reports with weightier patches which are harder to assess and
> finalise. Closing out the easier tickets quickly would benefit
> everyone by improving the signal/noise ratio - how could this be
> addressed?
It gets addressed already. Smaller changes do get evaluated faster than
larger ones. Small does not necessarily mean low-risk, though.
Regards,
Malcolm
Vinay Sajip
> > as worthy of review and critical assessment as code patches, there is
> > less of a concern about affecting trunk stability, and less impact
> > analysis work needed.
>
> Not true. We try to be very careful about the documentation we commit to
> the tree. Almost no documentation patch goes into the tree without
> changes -- usually it gets edited or corrected once by the person doing
> the original commit and once again by Adrian as he edits all of the doc
> changes.
Sorry, which part of "While [documentation patches] are just as
worthy of review and critical assessment as code patches, there is
less of a concern about affecting trunk stability, and less impact
analysis work needed." is not true? Nothing in my post was intended to
suggest compromising documentation quality in any way. We all know
that a key component of Django's success is its great documentation.
> > Could there be a more streamlined process to get
> > them reviewed and into trunk? I couldn't see anything in the current
> > tracker that allows such patches to be identified and fast-tracked.
>
> The core developers all have a good intuitive understanding of which
> areas and tickets are going to be more time consuming than others. If
> you sit down and work out the numbers, you'll find that most critical
> documentation patches do go in very quickly, simply because they *are*
> easy to review, fix and commit.
Fair enough - I was just thinking this would help triagers (esp. new
triagers) to do their bit. For example, as a new triager I would be
more confident patching documentation errors and minor omissions while
finding my way around the framework. Having e.g. documentation-only
tickets via a custom query would seem to make this activity easier. Of
course committers always have the final say - we just try to help move
things along.
> There are some documentation patches still uncommitted because either
> they require more work, aren't necessarily the right approach or are
> documenting things that are unstable or not going to be around and we
> haven't quite made a decision yet. So these types of areas do get
> implicitly fast tracked.
>
> > Another thing I've noticed is that tickets with small, low-risk
> > patches (which in theory should be easier and quicker to close) are
> > not that easy to single out in the tracker, sharing equal space in the
> > Trac reports with weightier patches which are harder to assess and
> > finalise. Closing out the easier tickets quickly would benefit
> > everyone by improving the signal/noise ratio - how could this be
> > addressed?
>
> It gets addressed already. Smaller changes do get evaluated faster than
> larger ones. Small does not necessarily mean low-risk, though.
I agree. I meant "small and low-risk", not "small, therefore low-
risk". I notice there have been some other posts recently in the same
vein - talking about
"easy pickings" and "low-hanging fruit".
Regards,
Vinay
Malcolm Tredinnick
> > > What's the thinking on documentation-only patches? While they are just
> > > as worthy of review and critical assessment as code patches, there is
> > > less of a concern about affecting trunk stability, and less impact
> > > analysis work needed.
> >
> > Not true. We try to be very careful about the documentation we commit to
> > the tree. Almost no documentation patch goes into the tree without
> > changes -- usually it gets edited or corrected once by the person doing
> > the original commit and once again by Adrian as he edits all of the doc
> > changes.
>
> Sorry, which part of "While [documentation patches] are just as
> worthy of review and critical assessment as code patches, there is
> less of a concern about affecting trunk stability, and less impact
> analysis work needed." is not true?
Let's try to avoid making a big issue out of this, but since you asked:
he last part is the problem: committing bad documentation leads to
confusion (we've managed to do that before) and so a fair bit of
thinking *is* required, sometimes more than for other fixes.
> Nothing in my post was intended to
> suggest compromising documentation quality in any way. We all know
> that a key component of Django's success is its great documentation.
>
> > > Could there be a more streamlined process to get
> > > them reviewed and into trunk? I couldn't see anything in the current
> > > tracker that allows such patches to be identified and fast-tracked.
> >
> > The core developers all have a good intuitive understanding of which
> > areas and tickets are going to be more time consuming than others. If
> > you sit down and work out the numbers, you'll find that most critical
> > documentation patches do go in very quickly, simply because they *are*
> > easy to review, fix and commit.
>
> Fair enough - I was just thinking this would help triagers (esp. new
> triagers) to do their bit. For example, as a new triager I would be
> more confident patching documentation errors and minor omissions while
> finding my way around the framework. Having e.g. documentation-only
> tickets via a custom query would seem to make this activity easier. Of
> course committers always have the final say - we just try to help move
> things along.
Okay, I understand you better now: I thought you were suggesting a way
to make committers look at certain tickets. Now I realise you are
talking about the initial bug fixing work. My apologies.
You can already do what you are seeking here: go to "View tickets"
construct whatever query you want and then save it as a bookmark. If you
want to work on documentation patches, you can make that bookmark. If
somebody else wants to work on internationalisation or newforms or
whatever, they can make their own bookmark for that.
Regards,
Malcolm
Simon G.
Speaking as a triager, I originally thought that an "easy-fix"
category would help solve these issues.
However, after doing quite a bit of ticket twiddling, these easy fix/
low hanging fruit don't hang around very long anyway. The tend to get
closed pretty quickly, even without triage (i.e. the core developers
seem to find them easily). Otherwise, I generally slot them into
"Ready for Checkin", if it's a few lines/simple bug fix thing that
doesn't need much discussion, and they get dealt with from there.
It might be nice, however, to have some way of flagging tickets -
there's a few tickets around that are RTC, have patches but have been
sitting around for months. it's hard to tell if this is because
they're forgotten, or someone's pondering them, or they need work, or
no-one's had the time to check them in. This locmem one is a case in
point here.
Maybe we could tag these somehow ("outstanding"?) so that the next
time a dev. gets some ticket time, they can glance over them and give
some feedback (wontfix, pls. improve patch, will-do-it-when-I-have-
time, etc).
--Simon G.
Malcolm Tredinnick
[...]
> Maybe we could tag these somehow ("outstanding"?) so that the next
> time a dev. gets some ticket time, they can glance over them and give
> some feedback (wontfix, pls. improve patch, will-do-it-when-I-have-
> time, etc).
If it's marked "ready to checkin" and hasn't been checked in then it
means we're still thinking about it or needs more time to do the final
checking in that we might have right now. Otherwise it would be back in
accepted. I don't see how "oustanding" adds anything to appearing at the
top of the "ready to checkin" list.
As you might have noticed, I'm personally reluctant to add lots of
fine-grained statuses like this because I'm not sure that it really adds
much to the workflow. Particularly in the "ready to checkin" phase, the
list of tickets fits on a screen or two, so it's an easy report to scan.
Regards,
Malcolm
Philipp...@gmail.com
As an bug reporter, I have a small suggestion to make: use a kernel or
gcc-like terminology for the branches. The main page could list 0.96
and (trunk or dev or 0.97) and clearly state that the 0.96 branch is
only accepting bugfixes and that the trunk branch is under heavy dev,
with fixes delayed. Later on the trunk branch would switch to the
bugfix state and people could come out of the wood to push their pet
patch. I'm oversimplifying a bit there but you get the idea.
It wouldn't change anything to the core devs' work but would make the
whole process much easier to understand from the outside.
my 2 cents,
Philippe
jedie
+1
User can use a stable SVN version (with bugfix state) and there exist
a heavy dev branch for experimentals. This branch can have a "feature
freeze state". In this time the branch stabilized and a new release
can be born ;)
Simon G.
stable (except for major security fixes), and the active development
is going on in trunk? This has been documented:
http://www.djangoproject.com/weblog/2007/apr/06/changes/ - people
should be using the 0.96 release on production websites. SVN is for
development.
Don't get me wrong, I certainly don't want to scare of anyone who
wants to contribute, but I don't see that helping. We've already got a
fairly complicated repository (with branches out the wazoo), and I
think it'll just complicate things even further without giving any
benefits.
--Simon
Philipp...@gmail.com
the website whether the current dev branch (svn trunk) is in heavy
dev, feature freeze or bugfixes only. It's only a matter of
communication, I certainly did not advocate maintaining a new "stable
but cuting edge" svn branch.
Regards,
Philippe