[erlang-questions] Software Estimation and Progress Tracking

[ke...@funceng.com]

Hello All,

I recently wrote a blog post on software estimation and progress
tracking for functional languages. Unfortunately, the post was not
terribly substantive and posed more questions than it answered. Since
the Erlang community seems to have a lot of good practical experience on
various sizes of commercial projects, I wondered if anyone here had
experience with estimation and progress, particularly as it differs (or
perhaps doesn't) when using Erlang as opposed to imperative languages.

If you have any thoughts, please respond on this list or better yet,
just add a comment to the blog post:

http://funceng.com/2013/09/18/estimation-and-progress/

Thanks,

Kevin

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Michael Truog]

Some links that seem relevant are:
http://www.slideshare.net/Arbow/comparing-cpp-and-erlang-for-motorola-telecoms-software
http://www.erlang.org/faq/how_do_i.html#id55052

[Jesper Louis Andersen]

On Thu, Sep 19, 2013 at 6:44 AM, <ke...@funceng.com> wrote:

I recently wrote a blog post on software estimation and progress tracking for functional languages. Unfortunately, the post was not terribly substantive and posed more questions than it answered. Since the Erlang community seems to have a lot of good practical experience on various sizes of commercial projects, I wondered if anyone here had experience with estimation and progress, particularly as it differs (or perhaps doesn't) when using Erlang as opposed to imperative languages.

The underlying problem of software estimation is this:

"Please estimate how many years it takes to make a fusion reactor from now, solving all power problems on the earth in one fell swoop."

The thing is, you can't. People understand this to be research and hence they have no qualms at throwing effort at the project in the hope it brings you closer to the ultimate goal, albeit slowly.

Software is not "construction". It turns out to be "research". Regarding it as "construction" by imposing estimates and progress tracking is usually futile due to this dichotomy and the fact people missed it was "research".

So, will the research happen faster in Erlang than in other languages? This is hard to answer without being concrete. My loose experience is that languages working at higher level in general provides shorter paths to enlightenment[0]. Since functional languages are often working at a higher level there is a considerable overlap. But don't underestimate something like Python, Ruby or Javascript which also works at the same high level. It is not always the case that you will be faster due to the trait of being functional.

Compared to "traditional" languages like C or C++, the lack of proper garbage collection is the key thing which make you research faster. Compared to Java, it is the succinctness of Erlang. Compared to many other systems, like Python or Ruby, it is the toolset in Erlang to describe concurrency and the ability to utilize the machine via parallelism. Compared to Golang, it is probably the process isolation and the ability to run large complex systems with hundreds of applications in the same memory space without panics[1].

Bringing it back to estimation, I fear that we will just adapt and expect more features to delivered in shorter time, so our estimation of progress and effort will still be off, even if we get things done faster. For some reason, there is a bias toward "fast delivery" with no regard toward "quality of product". This is a depressing work environment, though: every deliverable will be too late, always. And this leads to a psychological phobia of estimation :)

[0] Since software is research, the goal is to seek enlightenment and understand something deeper than before.

[1] Though this is a hunch, not experience.

[Ulf Wiger]

On 19 Sep 2013, at 11:59, Jesper Louis Andersen <jesper.lou...@erlang-solutions.com> wrote:

Software is not "construction". It turns out to be "research". Regarding it as "construction" by imposing estimates and progress tracking is usually futile due to this dichotomy and the fact people missed it was "research".

Yes, but in my experience, this is how most research funding is determined as well: "We estimate that we will invent X in month 47 of the project, after which we will discover Y based on X in month 64, having spent N number of man-months in the process."

BR,

Ulf

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.

http://feuerlabs.com

[Jesper Louis Andersen]

On Thu, Sep 19, 2013 at 12:19 PM, Ulf Wiger <u...@feuerlabs.com> wrote:

Yes, but in my experience, this is how most research funding is determined as well: "We estimate that we will invent X in month 47 of the project, after which we will discover Y based on X in month 64, having spent N number of man-months in the process."

Indeed.

Which is why researchers do the research first and put it in a drawer. Then go seek the funding for the research. Thus they hit a 100% success rate, which is impressive. And it makes it consistent so they will get more funding for more research.

[Valentin Micic]

Well, if you do not know what you're doing, then you can just as well call it a research.

[Joe Armstrong]

Once upon a very long time ago we did a project to compare the efficiency of

Erlang to PLEX.

We implemented "the same things" (TM)  in Erlang and PLEX and counted total man hours

We did this for several different things.

Erlang was "better" by a factor of 3 or 25 (in total man hours) - the weighted average was a factor 8

They asked "what is the smart programmer effect"

We said "we don't know"

We revised the figure 8 down to 3 to allow for "the smart programmer effect" - this was

too high to be credible, so we revised it down to 1.6. (the factors 3 and 1.6 where just plucked

out of the air with no justification)

Experiments that show that Erlang is N times better than "something else" won't be believed if

N is too high.

The second point to remember is that you *never* implement exactly the same thing

in two different languages (or very rarely) - the second time you do something

you have presumably learnt from the mistakes made the first time you do something.

If you implement the same thing N times in the same language, each implementation should take

less effort and code than the last time you did it. What can you learn from this?

The difference in programmer productivity can vary by a factor of 80 - (really it's infinity,

because some programmers *never* get some code right, so the factor 80 discounts the

totally failed efforts) - So given a productivity factor you have to normalize it by  a factor

that depends upon the skill and experience of the programmer.

There are people who claim that they can make models estimating how long a software projects take.

But even they say that such models have to be tuned, and are only applicable to projects

which are broadly similar. After you've done almost the same thing half a dozen times

it might be possible to estimate how long a similar project might take.

The problem is we don't do similar things over and over again. Each new unsolved problem

is precisely that,  a new unsolved problem.

Most time isn't spent programming anyway -  programmer time is spent:

    a) fixing broken stuff that should not be broken

    b) trying to figure out what problem the customer actually wants solving

    c) writing experimental code to test some idea

    d) googling for some obscure fact that is needed to solve a) or b)

    e) writing and testing production code

e) is actually pretty easy once a) - d) are fixed. But most measurements of productivity only measure

lines of code in e) and man hours.

I've been in this game for many years now, and I have the impression that a) is taking a larger and

larger percentage of my time. 30 years ago there was far less software, but the software there was

usually worked without any problems - the code was a lot smaller and consequently easier to understand.

Again in the last 30 years programs have got hundreds to thousands of times larger (in terms of code lines)

but programming languages haven't got that much better and our brains have not gotten any smarter. So

the gap between what we can build and what we can understand is growing rapidly.

Extrapolating a bit I guess a) is going to increase - so in a few years we'll have incredibly smart

devices which almost work, and when broke nobody will able to fix, and programmers will spend 100%

of their time fixing broken stuff that should not be broken.

And no I have to figure out why firefox has suddenly stopped working - something is broken ...

Cheers

/Joe

On Thu, Sep 19, 2013 at 6:44 AM, <ke...@funceng.com> wrote:

[Mahesh Paolini-Subramanya]

There are - at least - four orthogonal areas in which your software gets developed, each of which has different metrics for estimation, tracking, progress, etc., etc., etc.  To throw some semantics at this

1) Technical

When the specifics of the solution are clear, and it pretty much boils down to implementation. 

"I need to replace the valve on my Hot Tub" (With the same model, I've already done it once before, etc., etc.)

2) Engineering

When you need to solve the problem first, before implementing the solution.  There _is_ a body of kno

"I need to install my Hot Tub" (Hmmm. There is no water line going out there. How do I get one? What about the electrical circuit? Do I need architectural permission? etc.)

3) Science

You need to invent a new class of solutions for the problem at hand. 

"I need to install my Hot Tub in an underground bunker in the marshlands of Florida" (How do I build an underground bunker in the marsh? Maybe I can freeze the ground and pour concrete? How do I keep the concrete from sinking? Hmmm. Time to start running experiments…"

4) Art

You need to get the intangibles correct, viz., is it maintainable? Supportable? Documented? Elegant? 

"Will some future Significant Other like my paranoid underground-bunker hot-tub?"

Note that each of the above are _different_.  

- Its fairly easy to Manage / Maintain / Monitor "Technical" work.  

- There is a reasonable body of knowledge that helps in doing the same for "Engineering" stuff.  

- For Science, its all pretty clearly made-up (the fusion reactor!), 

- For "Art", well, it really _is_ in the eye of the beholder.  (Good Documentation? For whom? What do you mean by "Good"? *I* understand it! And so does Jane!)

Which brings me back to the original point - much as we would like it to be that way, software pretty much never fits neatly into one of the buckets above - it is some combination of the four, with different parameters for {T, E, S, A}.  

Whats worse, there is a time-variant aspect to this too - and the parameters are inter-related.  e.g., different "Engineering" solutions have different "Technical" impacts.  In short, your development process is actually f(T, E, S, A, Time)

All this being said, there is a time-honored Academic way of solving f(T, E, S, A, Time), which basically consists of wishing-away the unknowns (or make unrealistic assumptions about them) and then spend an in-ordinate amount of time on the remaining parameters.  With some appropriate tweaking of parameters, one can quite successfully make this match some "real-world" results, which can then be trumpeted widely.  Any failures can be blamed on the actors (ho-ho. "Actors". In an erlang post. I crack myself up.) who were clearly not qualified..

This may very well be the best option…

Cheers

Mahesh Paolini-Subramanya

That Tall Bald Indian Guy...

Google+  | Blog   | Twitter  | LinkedIn

[Varuna]

On Thu, Sep 19, 2013 at 6:44 AM, <ke...@funceng.com> wrote:
> I recently wrote a blog post on software estimation and progress tracking for
> functional languages. Unfortunately, the post was not terribly substantive and

> ......

> Bringing it back to estimation, I fear that we will just adapt and expect more
> features to delivered in shorter time, so our estimation of progress and
> effort will still be off, even if we get things done faster. For some reason,
> there is a bias toward "fast delivery" with no regard toward "quality of
> product". This is a depressing work environment, though: every deliverable
> will be too late, always. And this leads to a psychological phobia of
> estimation :)

> ...

Not sure if SWEBOK Guide at (http://www.swebok.org) would be of any help,
nevertheless, there is good information under the sections Software Engineering
Economics, Software Engineering Management, and Software Engineering Models and
Methods.

With regards,
Varuna