Evidence Based Scheduling
Pierre Phaneuf
Simon Law
> A new scheduling-related article on Joel on Software:
>
> http://www.joelonsoftware.com/items/2007/10/26.html
Yes. Joel has finally taken Painless Software Scheduling and moved it
into FogBugz 6.
--
Cheers,
Simon Law http://www.law.yi.org/~sfllaw/
Pierre Phaneuf
> Yes. Joel has finally taken Painless Software Scheduling and moved it
> into FogBugz 6.
It also seems to be improved compared to his manual scheduling that he
was promoting in Painless Software Scheduling, much in the same ways
that we improved upon it when we made Schedulator.
Avery Pennarun
I wouldn't say "much in the same ways". I think he has at least one
similar goal but has gone in an almost completely different direction.
One nice thing about Fogbugz 6's approach is that it's super-simple:
you don't need to provide any more information in your estimates than
you ever did (no loadfactor, no magicday, no vacation times, etc) and
now you can generate pretty charts. That's pretty cool. In general
I'm a fan of the most minimal interface possible.
His solution to deciding on a loadfactor is simply to measure the
difference between estimated and actual time to completion for each
bug, which simplifies things nicely. He also measures not just the
average difference (like schedulator does) but the variance, which can
be quite powerful.
The solution to the MagicDay is a little more borderline: if you find
a bug, fix it as part of the bug you were working on at the time.
Thus, random bug fixing is essentially added to your loadfactor. That
works (and note how it solves one of schedulator's problems, which is
that the MagicDay seems to be slightly variable between people),
although it can be a bit messy.
But at the core, what Fogbugz 6 does is purely estimation of the
release date of your project, based on the assumptions that:
- All your projects and tasks are listed in fogbugz
- Vacation time isn't going to upset your schedule too much
- When you find a new bug, you won't assign it to someone else on the team
And the results of these assumptions are not actually certain: they
work for Joel, but nobody else has tested them yet. On the other
hand, drawing lots of pretty graphs, even wrong ones, is a certifiably
great way to get people to buy your new product.
Schedulator's goals are a bit different, because it's also a
*self-monitoring* tool and you can use it to improve your performance,
not just measure it. Why is my loadfactor so high compared to this
person? Where do my schedule slippages come from?
Schedulator4 also lets you import bugs from any tracking system, and
makes it easy to create lightweight tasks that are much simpler than
bugs in fogbugz (maybe fogbugz6 has lightweight tasks, I'm not sure).
Still, it might be that fogbugz6 does 99% of what most people who use
schedulator want it to do, with much less learning curve, and with
prettier graphs. I'm not sure. It's also commercial, where
schedulator is free.
At the moment, we have a license for fogbugz at work (and with
software assurance, that includes fogbugz6). But we have a very, very
small team, so scheduling hasn't been a real issue yet. We use even
schedulator pretty lightly so far.
Have fun,
Avery
Peter McCurdy
presentation: http://joelonsoftware.com/items/2007/10/24austindemo.html
The snazzy graphs seem particularly important, if only for helping
convince upper-management-type people that there's some value in
there, and making the whole release estimation process a bit more
obvious. So you can say "see, when we add more features, the release
date moves" and so on.
Peter.
Pierre Phaneuf
> > > Yes. Joel has finally taken Painless Software Scheduling and moved it
> > > into FogBugz 6.
> >
> > It also seems to be improved compared to his manual scheduling that he
> > was promoting in Painless Software Scheduling, much in the same ways
> > that we improved upon it when we made Schedulator.
>
> I wouldn't say "much in the same ways". I think he has at least one
> similar goal but has gone in an almost completely different direction.
I think I meant that to be singular, as in "it's not just the Painless
Software Scheduling put into Fogbugz, but also a bunch of extra
stuff".
Some of that "extra stuff" is similarish, for example the horizontal
timeline for each developers in section 4 ("Manage your projects
actively") reminds me of the milestone summary, and the slider for the
priority is like the "fix bugs up to priority X" that was also in the
summary.
But other stuff is different, like the distribution stuff, which is
actually very interesting, IMHO. We've been trying to find out ways to
help developers make better estimates, and this might be one way.
> The solution to the MagicDay is a little more borderline: if you find
> a bug, fix it as part of the bug you were working on at the time.
> Thus, random bug fixing is essentially added to your loadfactor. That
> works (and note how it solves one of schedulator's problems, which is
> that the MagicDay seems to be slightly variable between people),
> although it can be a bit messy.
I don't think that's really how he addresses the MagicDay.
He says there are actually two ways you can deal with new things (be
them related or not), and that it both gives the same result: either
you make it part of the bug you were working on at the time (and it
inflates the load factor), or you file separate bugs.
I think the most direct equivalent to our magic day is the probability
curve, with the "probability of shipping" being similar to the magic
number we multiplied by. It's just that instead of picking just one
number and showing you what is predicted with that number, he shows a
*bunch* of predictions with different numbers.
> But at the core, what Fogbugz 6 does is purely estimation of the
> release date of your project, based on the assumptions that:
> - All your projects and tasks are listed in fogbugz
> - Vacation time isn't going to upset your schedule too much
> - When you find a new bug, you won't assign it to someone else on the team
I didn't try the software itself, and it doesn't directly mention it,
but there is a reference made about taking vacations and holidays into
account, in the mapping the durations of bugs into real days (at the
end of "Simulate the future").
And like I mentioned, I don't see why when a new bug is found, you
shouldn't file it and assign it to whoever (it might be yourself, of
course). The "Scope creep" part tells you how he copes with slippage,
mostly by telling people to add a buffer, and use that graph to see
when you're heading for a disaster. It's an interesting graph (and I
like the whole idea of using probabilities instead of making a single
prediction, in general), but I'm not sure this in particular is a
better way.
He does recommend something intriguing: when fixing a bug in a new
feature, charge them to the original task instead of filing a new bug.
This makes the loadfactor equivalent integrate bug fixing as well.
The killer, for me anyway, is that everything needs to be in Fogbugz.
Maybe that's a good idea, it's not *that* long to file new bugs
(although there's a bunch of annoying things with Trac that Fogbugz
certainly did better, I do have to say that I can file tickets at a
pretty damned good rate, if I ever want to!), but it certainly is
somewhat restrictive.
For a pure software development team, it might be okay, but as soon as
you start integrating people at the edge (like QA, managers producing
specs or whatever), it gets funkier and funkier to work within a
single system.
> Schedulator's goals are a bit different, because it's also a
> *self-monitoring* tool and you can use it to improve your performance,
> not just measure it. Why is my loadfactor so high compared to this
> person? Where do my schedule slippages come from?
I think the graph in the "Track elapsed time" section does this job
pretty nicely, and in fact, *better* than Schedulator.
The slope of the line (what is that line called, BTW?) tells you your
loadfactor, and you can compare that, just like you'd compare your
loadfactor with someone else's. But also, you can look at the
distribution of your estimates, which Schedulator just averages out,
and you can tell whether someone is better at estimating than you are.
I'd venture that it's probably better to have someone who's
*consistently* over- or under-estimating (having a velocity of 2, bug
after bug, say) than someone who's closer to the mark, but all over
the place (0.6 to 1.4).
> Schedulator4 also lets you import bugs from any tracking system, and
> makes it easy to create lightweight tasks that are much simpler than
> bugs in fogbugz (maybe fogbugz6 has lightweight tasks, I'm not sure).
That's one of my favourite feature of Schedulator, as a pusher (as a
programmer, I don't really care, because all of my stuff *is* in the
bug tracker!).
> Still, it might be that fogbugz6 does 99% of what most people who use
> schedulator want it to do, with much less learning curve, and with
> prettier graphs. I'm not sure. It's also commercial, where
> schedulator is free.
I think there might be ideas to steal, there, and that the killer one
would actually be an hybrid, but I'm not sure of the exact mix.
For example, it really looks like the EBS stuff uses an actual
timesheet tool, where you "start" working on a bug, and "stop" the
clock when you go off for lunch or go home, and since it's integrated
inside of Fogbugz, it probably stops the clock automatically when the
bug is resolved, for example. That's definitely a different approach
than what we did, and I think it's got both pros and cons. Among other
things, it probably is less of a pain if you can "clock in" right from
the bug, which Schedulator wouldn't be able to do (you could clock in
from your schedule page, though).
Pierre Phaneuf
> The snazzy graphs seem particularly important, if only for helping
> convince upper-management-type people that there's some value in
> there, and making the whole release estimation process a bit more
> obvious. So you can say "see, when we add more features, the release
> date moves" and so on.
That's definitely a *very* strong feature, IMHO. I'm finding that I
can convince people that my schedule is correct more easily by
basically making the display prettier (with the exact same data, of
course). That is patentedly retarded, of course, but is the way
reality works, and I can't really argue about it...
Avery Pennarun
> Avery wrote:
> > The solution to the MagicDay is a little more borderline: if you find
> > a bug, fix it as part of the bug you were working on at the time.
> > Thus, random bug fixing is essentially added to your loadfactor. That
> > works (and note how it solves one of schedulator's problems, which is
> > that the MagicDay seems to be slightly variable between people),
> > although it can be a bit messy.
>
> I don't think that's really how he addresses the MagicDay.
>
> He says there are actually two ways you can deal with new things (be
> them related or not), and that it both gives the same result: either
> you make it part of the bug you were working on at the time (and it
> inflates the load factor), or you file separate bugs.
>
> I think the most direct equivalent to our magic day is the probability
> curve, with the "probability of shipping" being similar to the magic
> number we multiplied by.
Not really, and this is the tricky part. The MagicDay is intended to
deal with the inevitable creation/discovery of *new* bugs (or feature
requests) as you work on the bugs you already have. The probability
curve only deals with the misestimation of *existing* bugs (which in
schedulator we aim to *average* out to zero - maybe not a good idea
:)).
Now, if you take all the random bugs you discover and integrate them
into your bug estimates, then you've included them in the probability
curve. But if you file *new* bugs for them (for example, if they're
major changes in direction for a feature or something), you screw up
that graph: all your estimates will be fine, but the project will
still be late.
Joel's use of the "extra buffer time", though, is essentially
equivalent to the MagicDay. It's just not automatic. The MagicDay
sort of doles out your buffer at a certain rate, so you know if you
exceed that rate, you're getting into trouble, and if you're below
that rate, you're just lucky so far :) But at least his article deals
with it, now that you mention it, yes.
> He does recommend something intriguing: when fixing a bug in a new
> feature, charge them to the original task instead of filing a new bug.
> This makes the loadfactor equivalent integrate bug fixing as well.
In fact I already used my loadfactor for that for small bugs; I don't
want to misestimate existing bugs just to deal with other stuff I run
into along the way, and I'm too lazy to file separate bugs for every
little thing that comes up. So if (like Joel) you don't believe in
loadfactors, this part actually fits the schedulator model exactly.
(If you're Joel, just always set your loadfactor to 1.0.)
> The killer, for me anyway, is that everything needs to be in Fogbugz.
> Maybe that's a good idea, it's not *that* long to file new bugs
> (although there's a bunch of annoying things with Trac that Fogbugz
> certainly did better, I do have to say that I can file tickets at a
> pretty damned good rate, if I ever want to!), but it certainly is
> somewhat restrictive.
I've often thought that a turbo-fogbugz-bug-creator applet might be a
good time investment :) If that's the only reason not to use fogbugz,
it's not a very convincing one. I'm more concerned with how to run a
shared open source project; you certainly can't buy a fogbugz license
for every single potential developer on the Internet, so something
like google code is more appropriate at that point.
One of the reasons I think WvStreams development didn't really take
off outside NITI (until now... sort of :)) is that outsiders couldn't
see the bug list. I don't want to do that again with my new projects.
> For a pure software development team, it might be okay, but as soon as
> you start integrating people at the edge (like QA, managers producing
> specs or whatever), it gets funkier and funkier to work within a
> single system.
I'm pretty sure they don't need anything like fogbugz *or*
schedulator, really. Product managers and "people people" need more
of a "promise-based" system instead of a "prediction-based" system:
http://www.advogato.org/person/apenwarr/diary/82.html
I don't know for sure about QA, but it's another story again, I agree.
> > Schedulator's goals are a bit different, because it's also a
> > *self-monitoring* tool and you can use it to improve your performance,
> > not just measure it. Why is my loadfactor so high compared to this
> > person? Where do my schedule slippages come from?
>
> I think the graph in the "Track elapsed time" section does this job
> pretty nicely, and in fact, *better* than Schedulator.
The graph looks pretty awesome, yes. My problem is that I can sort of
"see trends" without the pictures, so graphs seldom help me, and thus
I tend to not bother producing graphs often enough. But I'm sure it
would help a lot of people.
> I'd venture that it's probably better to have someone who's
> *consistently* over- or under-estimating (having a velocity of 2, bug
> after bug, say) than someone who's closer to the mark, but all over
> the place (0.6 to 1.4).
Granted. But the unfortunate thing about being wrong is that it's
almost impossible to do it predictably :) I'd rather just learn to
converge on the right answer over time.
Also, even if you're all over the place, if you divide your tasks into
*enough* pieces, it'll average out.
> For example, it really looks like the EBS stuff uses an actual
> timesheet tool, where you "start" working on a bug, and "stop" the
> clock when you go off for lunch or go home, and since it's integrated
> inside of Fogbugz, it probably stops the clock automatically when the
> bug is resolved, for example. That's definitely a different approach
> than what we did, and I think it's got both pros and cons. Among other
> things, it probably is less of a pain if you can "clock in" right from
> the bug, which Schedulator wouldn't be able to do (you could clock in
> from your schedule page, though).
I find that with schedulator, my work revolves around the schedule
more than around the BTS anyway (since I might have non-BTS tasks,
multiple BTSes, etc), so that wouldn't be the end of the world. And
it would be relatively easy to have a "clock in" button that records
the start time, then auto-terminates at the resolve-time of the bug
according to the BTS. Just one more schedulator plugin, I imagine.
This idea works much better if you assume 1.0 loadfactors, though, I
think. Or does it? Well, you'd have to correct out the loadfactor, I
suppose.
> > The snazzy graphs seem particularly important, if only for helping
> > convince upper-management-type people that there's some value in
> > there, and making the whole release estimation process a bit more
> > obvious. So you can say "see, when we add more features, the release
> > date moves" and so on.
>
> That's definitely a *very* strong feature, IMHO. I'm finding that I
> can convince people that my schedule is correct more easily by
> basically making the display prettier (with the exact same data, of
> course). That is patentedly retarded, of course, but is the way
> reality works, and I can't really argue about it...
Now now, it's not so much retarded as "illiterate":
http://alumnit.ca/~apenwarr/log/?m=200610#10
Graphs are valuable for sure.
In general, I think what Joel has stumbled on here might be the
scheduling equivalent of bayesian filtering in spam: stop trying to be
clever and just do the statistics. I'd like to know if it actually
works for anyone but Joel, which we're likely to find out in a few
months after people are using his program. If it *does* work, I think
stealing some ideas is very relevant.
Note that schedulator4 plugins could track your complete schedule
change history and thus do any of the evidence-based scheduling bits
we want. It's just a matter of coding them in.
Have fun,
Avery
Peter McCurdy
> But other stuff is different, like the distribution stuff, which is
> actually very interesting, IMHO. We've been trying to find out ways to
> help developers make better estimates, and this might be one way.
I agree, this is what I'd love to have, especially if I could have
gotten it for all my historical stuff at NITI.
> I didn't try the software itself, and it doesn't directly mention it,
> but there is a reference made about taking vacations and holidays into
> account, in the mapping the durations of bugs into real days (at the
> end of "Simulate the future").
He mentions vacations briefly in the video, but not in enough detail
to know what's going on. It does look like he's got a full-on
timesheet app included in there, so it's probably a part of that.
> The killer, for me anyway, is that everything needs to be in Fogbugz.
> Maybe that's a good idea, it's not *that* long to file new bugs
> (although there's a bunch of annoying things with Trac that Fogbugz
> certainly did better, I do have to say that I can file tickets at a
> pretty damned good rate, if I ever want to!), but it certainly is
> somewhat restrictive.
Ah. You should definitely watch the video. His goal in FogBugz 6 was
to make it easier to create bugs in FogBugz than it would be to write
them down in Notepad. You just click on the "new" button, then start
typing; when you press Enter, the bug is filed (with all non-subject
fields taken from the current filter), and it starts you writing a new
bug. Interestingly, it looks like they had to write something awfully
close to MESH, but that isn't quite as good in a few ways.
Peter.
Pierre Phaneuf
> Joel's use of the "extra buffer time", though, is essentially
> equivalent to the MagicDay. It's just not automatic. The MagicDay
> sort of doles out your buffer at a certain rate, so you know if you
> exceed that rate, you're getting into trouble, and if you're below
> that rate, you're just lucky so far :) But at least his article deals
> with it, now that you mention it, yes.
True. As I mentioned, I guess the way he deals with it is to try to
predict if he's in trouble with the trend of the ship date
distribution over time graph, but that's a bit iffy (I don't like that
it doesn't raise alarms like the MagicDay does).
> > He does recommend something intriguing: when fixing a bug in a new
> > feature, charge them to the original task instead of filing a new bug.
> > This makes the loadfactor equivalent integrate bug fixing as well.
>
> In fact I already used my loadfactor for that for small bugs; I don't
> want to misestimate existing bugs just to deal with other stuff I run
> into along the way, and I'm too lazy to file separate bugs for every
> little thing that comes up. So if (like Joel) you don't believe in
> loadfactors, this part actually fits the schedulator model exactly.
> (If you're Joel, just always set your loadfactor to 1.0.)
It's a bit more subtle than the loadfactor, I think, since it's per
bug. The trick part is actually increases the time elapsed on the "old
bug", so you see that you were more wrong than you thought at the
time. This effectively does the same as what you say for predicting
the release, but for improving your estimations, it could be a big
difference, IMHO.
> I've often thought that a turbo-fogbugz-bug-creator applet might be a
> good time investment :) If that's the only reason not to use fogbugz,
> it's not a very convincing one. I'm more concerned with how to run a
> shared open source project; you certainly can't buy a fogbugz license
> for every single potential developer on the Internet, so something
> like google code is more appropriate at that point.
Oh, I don't use Trac *because* it's faster to file new bugs than
Fogbugz, but rather because it's sanity-to-cost ratio is high (say,
better than Bugzilla's!) and that the maximum cost threshold I'm
working with is very low (couldn't buy, pretty much).
For an open source project, Fogbugz would be ridiculously expensive,
yes. Google Code is a good compromise, Trac might be another one,
depending on your exact needs.
I think I had a bookmark to the Fogbugz "new bug" page with as much
pre-filled as I could make it, it was pretty good, but could have been
better. For example:
> One of the reasons I think WvStreams development didn't really take
> off outside NITI (until now... sort of :)) is that outsiders couldn't
> see the bug list. I don't want to do that again with my new projects.
That's another point for having the schedule outside of any single bug
tracker, I agree.
> I'm pretty sure they don't need anything like fogbugz *or*
> schedulator, really. Product managers and "people people" need more
> of a "promise-based" system instead of a "prediction-based" system:
> http://www.advogato.org/person/apenwarr/diary/82.html
>
> I don't know for sure about QA, but it's another story again, I agree.
Yeah, I know that it doesn't work for some types of people, of course
(I used to make schedule pages for those people, but purposely full of
lies, so that the summary would be very angry and red when these
people were involved, telling me I had to deal with them in their own
ways), but I was thinking more of "other" people using a support
ticket system (like Enzo, say), or a QA department with their own
hypothetical test framework (I've seen systems that are similar to a
bug tracker, but specifically designed for testing, with test cases
and the concept of a "run", it might make sense to integrate that sort
of thing in the schedule).
> The graph looks pretty awesome, yes. My problem is that I can sort of
> "see trends" without the pictures, so graphs seldom help me, and thus
> I tend to not bother producing graphs often enough. But I'm sure it
> would help a lot of people.
Oh, I could probably do it without a graph too, just that since we
don't keep track of by how much we were wrong for each bug, we simply
don't know.
> > I'd venture that it's probably better to have someone who's
> > *consistently* over- or under-estimating (having a velocity of 2, bug
> > after bug, say) than someone who's closer to the mark, but all over
> > the place (0.6 to 1.4).
>
> Granted. But the unfortunate thing about being wrong is that it's
> almost impossible to do it predictably :) I'd rather just learn to
> converge on the right answer over time.
Well, I'd say that sometimes, it is (and that's one thing the
loadfactor covered just fine). For example, someone working longer (or
shorter) hours can still be making excellent estimates.
> Also, even if you're all over the place, if you divide your tasks into
> *enough* pieces, it'll average out.
That's an "if" there, which I agree you should bloody do, of course,
but it's pretty nifty with the EBS that with a bad person who doesn't
divide their tasks enough will *actually* show up in the overall
result, making the probability curve shallower and widening the
distribution...
You can look for that yourself if you look at the size of the
estimates or glancing at the summary could do it (big spaces between
lone bugs on a row: bad!), but translating the "badness" into a wider
distribution makes the danger very concrete and visible.
Again, we knew to watch out for that, we didn't need graphs or the
distributions, but that's not the case for everyone, and people don't
always think about it (we thought very long and hard about this whole
schedulation business, it makes us not very good examples of typical
users, I think).
> I find that with schedulator, my work revolves around the schedule
> more than around the BTS anyway (since I might have non-BTS tasks,
> multiple BTSes, etc), so that wouldn't be the end of the world. And
> it would be relatively easy to have a "clock in" button that records
> the start time, then auto-terminates at the resolve-time of the bug
> according to the BTS. Just one more schedulator plugin, I imagine.
Yeah, I think this could work just fine... It's probably more
complicated in reality, because you'd have to have ways to correct
mistakes (I stayed "clocked in" all night because I forgot to stop it
when I left for home, oops!), but that's more or less it.
> This idea works much better if you assume 1.0 loadfactors, though, I
> think. Or does it? Well, you'd have to correct out the loadfactor, I
> suppose.
I think he's pretty much accepted the idea of loadfactors different
than 1.0 with this (which is one of the biggest reasons for me
disagreeing with simply stating "oh, they've just put Painless
Software Schedules in Fogbugz"). He's keeping it in a different form
(probabilities for each velocities, instead of a single averaged out
number), but it's really similar in overall effect.
Someone working long hours, for example, would end up having more
velocities greater than 1, just like they would have a lower
loadfactor, giving more or less the same effect. If you're a
super-good estimator and you consistently work 10 hours of pure
productivity a day, you'll get this 1.25 velocity for every single bug
you do, just as if you had put a 0.8 loadfactor.
> Now now, it's not so much retarded as "illiterate":
> http://alumnit.ca/~apenwarr/log/?m=200610#10
I'm rude, what can I say. :-)
> In general, I think what Joel has stumbled on here might be the
> scheduling equivalent of bayesian filtering in spam: stop trying to be
> clever and just do the statistics. I'd like to know if it actually
> works for anyone but Joel, which we're likely to find out in a few
> months after people are using his program. If it *does* work, I think
> stealing some ideas is very relevant.
Me, what I wonder about is how updating estimates factors into things?
Someone who consistently thinks stuff is going to be easy, and still
is very diligent about updating his estimate when he finds out it's
harder is somewhat predictable (and happens in real life too)...
The more bayesian-style thing to do would be to ignore updated
estimates, and let the Monte Carlo simulation actually integrate how
overly optimistic a developer is. That sounds a little bit nuts to me,
but I guess that would make it even more like Bayesian spam filtering
as well. :-)
> Note that schedulator4 plugins could track your complete schedule
> change history and thus do any of the evidence-based scheduling bits
> we want. It's just a matter of coding them in.
The Monte Carlo bit is more of a core calculation than something that
could be plugged in, I'd say (unless the core calculation is done by a
plugin already?).
Pierre Phaneuf
> > But other stuff is different, like the distribution stuff, which is
> > actually very interesting, IMHO. We've been trying to find out ways to
> > help developers make better estimates, and this might be one way.
>
> I agree, this is what I'd love to have, especially if I could have
> gotten it for all my historical stuff at NITI.
Yeah, me too! See my closing comment about changing estimates in my
previous message, I'd be curious to know you opinion on that...
> He mentions vacations briefly in the video, but not in enough detail
> to know what's going on. It does look like he's got a full-on
> timesheet app included in there, so it's probably a part of that.
I don't have sound at work, so I was planning on watching it tonight...
> Ah. You should definitely watch the video. His goal in FogBugz 6 was
> to make it easier to create bugs in FogBugz than it would be to write
> them down in Notepad. You just click on the "new" button, then start
> typing; when you press Enter, the bug is filed (with all non-subject
> fields taken from the current filter), and it starts you writing a new
> bug. Interestingly, it looks like they had to write something awfully
> close to MESH, but that isn't quite as good in a few ways.
That's a very good goal to have, I think. I had to split a ticket into
no less than 30 tickets or so, last week (the QA guy thought it'd be
easier to just file one gigantic ticket with every single issue he
found). *sigh*
Peter McCurdy
>
> > > But other stuff is different, like the distribution stuff, which is
> > > actually very interesting, IMHO. We've been trying to find out ways to
> > > help developers make better estimates, and this might be one way.
> >
> > I agree, this is what I'd love to have, especially if I could have
> > gotten it for all my historical stuff at NITI.
>
> Yeah, me too! See my closing comment about changing estimates in my
> previous message, I'd be curious to know you opinion on that...
Well, I don't know for sure, but I'm almost 100% certain that all
FogBugz does is looks at the original estimate you (or someone else)
made and the final elapsed time. That's all the information that was
available in older versions of FogBugz, and it's very unlikely that
they changed the database schema for some minor improvement.
Furthermore, it's otherwise hard to say what estimate you should use.
The final estimate is guaranteed to be 100% accurate all the time, as
it'll just be how long it took you to do it, so it's not very
interesting. So if you don't use the first estimate as your baseline,
what do you use?
One minor issue I have with the whole EBS thing is that Joel talks
about having people estimate bugs for other people in his talk. This
seems doomed to failure, but I think he's using it as a crutch to work
around a few tricky bits.
> That's a very good goal to have, I think. I had to split a ticket into
> no less than 30 tickets or so, last week (the QA guy thought it'd be
> easier to just file one gigantic ticket with every single issue he
> found). *sigh*
Well, it was probably easier for *him*....
Peter.
Pierre Phaneuf
> > Yeah, me too! See my closing comment about changing estimates in my
> > previous message, I'd be curious to know you opinion on that...
>
> Well, I don't know for sure, but I'm almost 100% certain that all
> FogBugz does is looks at the original estimate you (or someone else)
> made and the final elapsed time. That's all the information that was
> available in older versions of FogBugz, and it's very unlikely that
> they changed the database schema for some minor improvement.
Hmm, Fogbugz did have a "original estimate" and "current estimate", on
top of the "elapsed time", that was one of the big things in Painless
Software Schedules!
> Furthermore, it's otherwise hard to say what estimate you should use.
> The final estimate is guaranteed to be 100% accurate all the time, as
> it'll just be how long it took you to do it, so it's not very
> interesting. So if you don't use the first estimate as your baseline,
> what do you use?
That's quite true.
> One minor issue I have with the whole EBS thing is that Joel talks
> about having people estimate bugs for other people in his talk. This
> seems doomed to failure, but I think he's using it as a crutch to work
> around a few tricky bits.
That's weird, because in the article, the very first point in the
final "While we're at it" section is about how only the programmer
doing the work can create the estimate! I'd personally have the but
tracker delete the estimate if a bug is being re-assigned... (which
would also provide a trick for gamers to change their "original
estimate")
> > That's a very good goal to have, I think. I had to split a ticket into
> > no less than 30 tickets or so, last week (the QA guy thought it'd be
> > easier to just file one gigantic ticket with every single issue he
> > found). *sigh*
>
> Well, it was probably easier for *him*....
Yeah, that's the attitude: I'll save 12 seconds of my time at the cost
of whole minutes of a bunch of other people's time...
Avery Pennarun
> bug. The trick part is actually increases the time elapsed on the "old
> bug", so you see that you were more wrong than you thought at the
> time. This effectively does the same as what you say for predicting
> the release, but for improving your estimations, it could be a big
> difference, IMHO.
Right. In general, splitting your equation into more independent
factors is better for updating your model of how things work, but
doesn't particularly help statistical (historical) models.
Historical vs. model-based predictions:
http://www.advogato.org/person/apenwarr/diary.html?start=97
> Yeah, I know that it doesn't work for some types of people, of course
> (I used to make schedule pages for those people, but purposely full of
> lies, so that the summary would be very angry and red when these
> people were involved, telling me I had to deal with them in their own
> ways), but I was thinking more of "other" people using a support
> ticket system (like Enzo, say), or a QA department with their own
> hypothetical test framework (I've seen systems that are similar to a
> bug tracker, but specifically designed for testing, with test cases
> and the concept of a "run", it might make sense to integrate that sort
> of thing in the schedule).
Schedulator4 attempts to include a type of deadline-based scheduler
for this sort of person (basically, tasks have due dates, and
schedulator should schedule them far enough back to be "safely likely"
of achieving that due date). It's all implemented except the actual
scheduling algorithm for such tasks. Heh.
As for QA, it would be an interesting extension: essentially, a "run"
is a bug that should have the same estimate every time. Really you
just want to keep a timesheet and even the *estimate* is calculated
automatically from the history of previous instances of that run.
Interesting, but perhaps of limited value. I mean, it's apparently
hard to explain to a lot of people just the *idea* that if it took you
two weeks last time, it'll probably take you two weeks this time.
> Oh, I could probably do it without a graph too, just that since we
> don't keep track of by how much we were wrong for each bug, we simply
> don't know.
Actually we do: we track OrigEst and CurrEst for every bug. Also,
Schedulator4 has a plugin that happens to track (partly because I was
too lazy to expire old entries) the entire CurrEst history of each
bug. It's a "simple" matter of graphing it after that.
> Well, I'd say that sometimes, it is (and that's one thing the
> loadfactor covered just fine). For example, someone working longer (or
> shorter) hours can still be making excellent estimates.
True, but that's perhaps easier shown by adjusting the number of
workinghours rather than the loadfactor. Schedulator4 has both, just
in case, and I suspect that fogbugz6 has a per-person workinghours
setting. (Fogbugz5 has a global one, but I think it maxes out at 40
hours per week because Joel is pushy :))
> That's an "if" there, which I agree you should bloody do, of course,
> but it's pretty nifty with the EBS that with a bad person who doesn't
> divide their tasks enough will *actually* show up in the overall
> result, making the probability curve shallower and widening the
> distribution...
Very neat, yes.
> Yeah, I think this could work just fine... It's probably more
> complicated in reality, because you'd have to have ways to correct
> mistakes (I stayed "clocked in" all night because I forgot to stop it
> when I left for home, oops!), but that's more or less it.
Actually, if your workinghours setting works correctly, that's exactly
what you *should* do. Clock in when you start the bug. Clock out
when you finish. No exceptions (unless you switch bugs in the
middle). The system should know you don't work overnight or on
weekends and thus not assume that you did.
In any case, correcting it is easy, at least in schedulator: just
update the elapsed time.
> Me, what I wonder about is how updating estimates factors into things?
> Someone who consistently thinks stuff is going to be easy, and still
> is very diligent about updating his estimate when he finds out it's
> harder is somewhat predictable (and happens in real life too)...
I'd say it's not that important. I imagine most bugs spend most of
their lifetime with currest=origest, elapsed=0. In other words,
they're waiting in the queue. Once you start, you might update
currest, but it's not going to last long enough to make a serious dent
in your statistics (which will still be determined largely by the
other bugs waiting in the queue).
Of course, you update the statistics engine based on the difference
between origest and elapsed(==currest), so you still use currest for
that. And sometimes people make typos in OrigEst, so trusting that
completely for current predictions isn't so good. I'd probably just
use CurrEst all the time for my real calculations (and schedulator
always has).
Or you could drop the concept of OrigEst altogether, along with the
concept that currest==elapsed when you've finished. Then you can fix
any Est typos without a problem, and update the statistics by
comparing Est with Elapsed. Hmm, that's rather nice.
> > Note that schedulator4 plugins could track your complete schedule
> > change history and thus do any of the evidence-based scheduling bits
> > we want. It's just a matter of coding them in.
>
> The Monte Carlo bit is more of a core calculation than something that
> could be plugged in, I'd say (unless the core calculation is done by a
> plugin already?).
Ah, I see what you mean.
The problem here is that Schedulator tries to provide you with more
information than Fogbugz does. If you just want release-prediction
graphs, you could do that as a plugin, since schedulator just collects
the data and hands it to you. But if you want your personal schedule,
time-per-bug is a critical part of the schedulator algorithm - and
monte carlo simulation doesn't do you much good anyway.
You could perhaps have a monte-carlo-engine plugin for doing the
release predictions, then feed your *average* velocity from that back
into schedulator to set your LoadFactor. (This discards the
variability, but knowing the variability doesn't help when you're
talking about individual bugs anyway.)
Have fun,
Avery
Avery Pennarun
> One minor issue I have with the whole EBS thing is that Joel talks
> about having people estimate bugs for other people in his talk. This
> seems doomed to failure, but I think he's using it as a crutch to work
> around a few tricky bits.
I haven't watched the talk, but I can make the theory almost sound
reasonable: once we have an automatic historical model for converting
estimates into predictions + variability, why *not* have your manager
make predictions for you?
Theoretically, every almost-reasonable-but-mostly-stupid thing you do
will only increase your variability. The predictions aren't
necessarily more wrong, just less precise (and they'd *tell* you how
much less precise!). And maybe a particular manager is actually a
good estimator.
This is the danger with historical predictions: they only tell you
what will happen, not how to make it happen better. EBS is definitely
working on the first, not the second. Admittedly, the first is the
more pressing problem.
Have fun,
Avery
Pierre Phaneuf
> Schedulator4 attempts to include a type of deadline-based scheduler
> for this sort of person (basically, tasks have due dates, and
> schedulator should schedule them far enough back to be "safely likely"
> of achieving that due date). It's all implemented except the actual
> scheduling algorithm for such tasks. Heh.
Yeah, I remember you mentioning it... How do you know how far back is
"safely likely"? :-)
> As for QA, it would be an interesting extension: essentially, a "run"
> is a bug that should have the same estimate every time. Really you
> just want to keep a timesheet and even the *estimate* is calculated
> automatically from the history of previous instances of that run.
> Interesting, but perhaps of limited value.
I think that this isn't really the domain of Schedulator, but this
hypothetical test case tracking system would definitely be connected
to Schedulator, much like a bug tracker is now.
I don't really want to think about this a lot, but I feel it would be
different enough to warrant some amount of thinking. For example, a
given test case can more easily be swung from one tester to the other
without introducing too much variance in how long it'll actually take.
Also, a single person can do a certain number of test cases at once
(but not too many), assuming enough hardware is available, again
without changing how long a given test case takes all that much.
This could all be taken into account more or less easily, but would
have to be catered for specifically. Working on two test cases would
be allowed (not stopping the clock on the first one), and it could
split a given time between however many test cases you were working on
at the same time (if during an hour you worked on three test cases,
they each get 20 minutes), which would take into account the increased
efficiency. Or maybe it could just let more than one clock run at the
same time, and it'll look like the QA team really overworks itself?
Anyway, you see how this is just different enough that you probably
don't want the same software to handle it "upstream" of Schedulator,
but I think it could produce something that could be fed into
Schedulator, still.
> I mean, it's apparently
> hard to explain to a lot of people just the *idea* that if it took you
> two weeks last time, it'll probably take you two weeks this time.
Hahaha! ;-)
> Actually we do: we track OrigEst and CurrEst for every bug. Also,
> Schedulator4 has a plugin that happens to track (partly because I was
> too lazy to expire old entries) the entire CurrEst history of each
> bug. It's a "simple" matter of graphing it after that.
Ah, that's handy!
> > Well, I'd say that sometimes, it is (and that's one thing the
> > loadfactor covered just fine). For example, someone working longer (or
> > shorter) hours can still be making excellent estimates.
>
> True, but that's perhaps easier shown by adjusting the number of
> workinghours rather than the loadfactor. Schedulator4 has both, just
> in case, and I suspect that fogbugz6 has a per-person workinghours
> setting. (Fogbugz5 has a global one, but I think it maxes out at 40
> hours per week because Joel is pushy :))
Maybe, I didn't use the new Schedulator enough to warrant playing
around with the working hours. That's something I think Schedulator
does better than this EBS thing: you could set different loadfactors
for different periods of time, when you knew you'd work longer hours
for a while, for example.
> > Yeah, I think this could work just fine... It's probably more
> > complicated in reality, because you'd have to have ways to correct
> > mistakes (I stayed "clocked in" all night because I forgot to stop it
> > when I left for home, oops!), but that's more or less it.
>
> Actually, if your workinghours setting works correctly, that's exactly
> what you *should* do. Clock in when you start the bug. Clock out
> when you finish. No exceptions (unless you switch bugs in the
> middle). The system should know you don't work overnight or on
> weekends and thus not assume that you did.
Well, uh, I *did* work overnight or on weekends sometimes. :-P
> I'd say it's not that important. I imagine most bugs spend most of
> their lifetime with currest=origest, elapsed=0. In other words,
> they're waiting in the queue. Once you start, you might update
> currest, but it's not going to last long enough to make a serious dent
> in your statistics (which will still be determined largely by the
> other bugs waiting in the queue).
That's what I was thinking about, that it wouldn't really make a
difference if your estimate is wrong for the bug you're currently
working on. Just let it be, and record the elapsed time correctly (and
when the bug is resolved, copy the elapsed time into the current
estimate).
> Of course, you update the statistics engine based on the difference
> between origest and elapsed(==currest), so you still use currest for
> that. And sometimes people make typos in OrigEst, so trusting that
> completely for current predictions isn't so good. I'd probably just
> use CurrEst all the time for my real calculations (and schedulator
> always has).
That's the thing: I agree that changing the original estimate because
of a typo is okay, but otherwise, I think that you just want to keep
it the way it was the first time, as this more accurately reflects
one's skills as an estimator? The idea here is that the less you can
make yourself look good in the distribution of your errors, the more
reliable (or at least, conservative) the schedule is.
> Or you could drop the concept of OrigEst altogether, along with the
> concept that currest==elapsed when you've finished. Then you can fix
> any Est typos without a problem, and update the statistics by
> comparing Est with Elapsed. Hmm, that's rather nice.
By "currest==elapsed" (equal?!?), you mean "currest=elapsed" (assign),
right? Yeah, that's just about what I think would be best.
That, along with discouraging updating estimates. When you find out
it's taking longer, just work on it longer, and save the time it
*actually* took in the elapsed.
It would still be possible to game the system a little bit by filing
separate bugs (with their own estimate) for issues you find while
working on a bug, but that's the kind of behaviour that ought to be
encouraged anyway.
If you were really pushy, as a bug tracker author, you could make the
estimate field read-only once it's been set (much like the origest in
Fogbugz), and when a bug is re-assigned, remove the estimate and any
elapsed time. Kind of rude, though...
> The problem here is that Schedulator tries to provide you with more
> information than Fogbugz does. If you just want release-prediction
> graphs, you could do that as a plugin, since schedulator just collects
> the data and hands it to you. But if you want your personal schedule,
> time-per-bug is a critical part of the schedulator algorithm - and
> monte carlo simulation doesn't do you much good anyway.
True, the EBS thing completely skips (ab)using the Student Syndrome
effect for each individual developer.
> You could perhaps have a monte-carlo-engine plugin for doing the
> release predictions, then feed your *average* velocity from that back
> into schedulator to set your LoadFactor. (This discards the
> variability, but knowing the variability doesn't help when you're
> talking about individual bugs anyway.)
That's something that I thought about for a long time, because that's
actually exactly how I'd set my loadfactor: I would keep a start date
about three months back, and (just after finishing a bug) binary
search for the loadfactor that makes my last bug done *now*.
This was one of my very first thought when I saw the EBS thing, but
remember I also mentioned that being able to set different loadfactors
for different periods as being a nice feature of Schedulator. Maybe
it's not really worth it, and making it all as automatic as possible
is the best way to go (the "Bayesian filter approach" we talked about
earlier)...
Pierre Phaneuf
> I haven't watched the talk, but I can make the theory almost sound
> reasonable: once we have an automatic historical model for converting
> estimates into predictions + variability, why *not* have your manager
> make predictions for you?
>
> Theoretically, every almost-reasonable-but-mostly-stupid thing you do
> will only increase your variability. The predictions aren't
> necessarily more wrong, just less precise (and they'd *tell* you how
> much less precise!). And maybe a particular manager is actually a
> good estimator.
I watched it yesterday, and what Joel was suggesting a manager would
do is to find a developer that has a small variability, have him sit
down with the developer that has a large variability and have them
work out more sensible estimates. The way I understood this, Fogbugz
remembers who made the estimate (of course), and uses that to
calculate the variability (so if the developer with the small
variability updated the other developer's estimates, the variability
of the latter developer's bugs would be reduced in the next
simulation, because it would be using the historical velocities of
that better estimator).
> This is the danger with historical predictions: they only tell you
> what will happen, not how to make it happen better. EBS is definitely
> working on the first, not the second. Admittedly, the first is the
> more pressing problem.
EBS seems like it can do just as much as Schedulator for the second,
but again, that's not all that much either... And I still find that
the emphasis on giving developers tools to help themselves improve
their estimation skills is a ridiculously good investment!
Avery Pennarun
> On Oct 29, 2007 2:00 PM, Avery Pennarun <apen...@gmail.com> wrote:
> > Schedulator4 attempts to include a type of deadline-based scheduler
> > for this sort of person (basically, tasks have due dates, and
> > schedulator should schedule them far enough back to be "safely likely"
> > of achieving that due date). It's all implemented except the actual
> > scheduling algorithm for such tasks. Heh.
>
> Yeah, I remember you mentioning it... How do you know how far back is
> "safely likely"? :-)
I'd probably use some multiple of the estimated time. You obviously
don't want to schedule a deadline task at the last possible moment,
just in case it goes overtime, but perhaps a MagicDay's worth would be
enough. Or maybe the 95% likely threshold, if you're doing EBS.
> > Actually, if your workinghours setting works correctly, that's exactly
> > what you *should* do. Clock in when you start the bug. Clock out
> > when you finish. No exceptions (unless you switch bugs in the
> > middle). The system should know you don't work overnight or on
> > weekends and thus not assume that you did.
>
> Well, uh, I *did* work overnight or on weekends sometimes. :-P
But you don't include that in your schedule, or it'll start making
predictions *assuming* you worked evenings and weekends. In an ideal
schedulator world, you'd take a few days off to compensate after
working weekends. Or whatever.
In Schedulator, you do this by counting the actual number of working
hours, but setting your LoadFactor appropriately.
In Joel's world, where LoadFactor is always 1.0, then you just
*always* worked 8 hours Monday to Friday, whether that reflects
reality or not. If you *actually* worked 12 hours, I don't care; you
still only get to add scatter 8 per day through your assigned tasks.
Fogbugz6 will (I assume) simply do that for you automatically. The
math works out in the end.
> > Or you could drop the concept of OrigEst altogether, along with the
> > concept that currest==elapsed when you've finished. Then you can fix
> > any Est typos without a problem, and update the statistics by
> > comparing Est with Elapsed. Hmm, that's rather nice.
>
> By "currest==elapsed" (equal?!?), you mean "currest=elapsed" (assign),
> right? Yeah, that's just about what I think would be best.
>
> That, along with discouraging updating estimates. When you find out
> it's taking longer, just work on it longer, and save the time it
> *actually* took in the elapsed.
Well, the idea isn't necessarily that you've assigned "currest :=
elapsed", but that they somehow *end up* being equal by the end.
Schedulator traditionally just whined at you if they didn't end up
being equal, because it didn't know which of the two you forgot to
update.
(Fogbugz6, since it updates the elapsed time for you, makes that part obvious.)
Anyway, the key innovation here is *not* updating your estimate when
you find out you're wrong. Feels a little weird to us old schedulator
fogies, but probably seems perfectly normal to most people :)
> True, the EBS thing completely skips (ab)using the Student Syndrome
> effect for each individual developer.
It's just an orthogonal feature, is all. Not really conflicting.
> That's something that I thought about for a long time, because that's
> actually exactly how I'd set my loadfactor: I would keep a start date
> about three months back, and (just after finishing a bug) binary
> search for the loadfactor that makes my last bug done *now*.
That's actually a dangerous strategy, depending on your mentality: it
doesn't necessarily result in a loadfactor that converges on a
particular number. Instead, it might just creep slowly upward. If
you're using the schedulator to gently apply pressure to make you work
harder (something I'll call the "Andrew factor" for posterity :)), and
you fall a bit behind, adjusting in this way simply means the next bug
has *less* pressure and thus you might fall behind *again*, and so on,
until eventually you're spending 90% of your time reading Slashdot.
Or whatever crappy news site the kids read nowadays.
> This was one of my very first thought when I saw the EBS thing, but
> remember I also mentioned that being able to set different loadfactors
> for different periods as being a nice feature of Schedulator. Maybe
> it's not really worth it, and making it all as automatic as possible
> is the best way to go (the "Bayesian filter approach" we talked about
> earlier)...
I really do like being able to set a specific loadfactor for certain
periods: basically when I would set myself up for a "sprint" (in the
not-exactly-Agile sense), I'd list off all the tasks for the project,
multiply by the MagicDay, set my loadfactor scarily low, and promise a
due date. If I used my normal loadfactor in which I actually allowed
interruptions, Schedulator would have been pretty useless for keeping
me on schedule in that situation. On the other hand, I don't think
most people used that feature.
I think it's really a feature/functionality tradeoff here. The
default loadfactor should probably be automatic, and then experts
could figure out how to adjust it the way they want.
Have fun,
Avery