Evidence

[Jimmy P]

<ponder>

So I've been thinking recently about evidence recently. Specifically the role evidence can or should play in the decision we make as part of the process of building software.

In bygone era's, back in the 70's at places like IBM and in the US defence department, very serious people built very serious software. As they did so they also collected very serious metrics on its quality and cost etc. They then analysed it after the fact and made claims about relationships between team size and quality etc etc. 

A good example of this work can be found in the Mythical Man Month which is where most of my exposure to this kind of evidence in software comes from. "Wow!" I thought as I read that book and the references to the hard data they had collected. "This is all a very impressive undertaking!" But despite the fact I agree with most of the claims made in that book. I have found few of them very useful. It's probably more accurate to say that I knew and understood the claims before I knew they were based on hard evidence. But the end result is the same.

Skip forward to today's discussion on the best ways to build software....

Most of the conversations we have today are usually simply based around opinion. These conversations are easy to have. I find them useful. But at times they leave me feeling a little guilty that any conclusions I draw from them might be unsound or predicated on a false premise.

A step up from opinion might be anecdotal evidence. Widely regarded as dangerous and in many contexts only accepted in cases were there is a complete lack of any other evidence. But that seems to be the case a lot for us, therefore I wonder if anecdotal evidence is actually a good basis for decisions in designing and building software? Though this has never sat well with me so I did some reading on why anecdotal evidence presents problems. Enter Wikipedia:

"Because of the small sample, there is a larger chance that it may be unreliable due to cherry-picked or otherwise non-representative samples of typical cases.^[1][2] Anecdotal evidence is considered dubious support of a claim"

Oh dear that seems like a problem right? Until you move further down the page and find this: 

"...in medicine, published anecdotal evidence by a trained observer (a doctor) is called a case report, and is subjected to formal peer review.^[12] Although such evidence is not seen as conclusive, it is sometimes regarded as an invitation to more rigorous scientific study of the phenomenon in question.^[13] For instance, one study found that 35 of 47 anecdotal reports of drug side effects were later sustained as "clearly correct."^[14]"

Wow that's cool right?

Maybe we just need to add peer review to the presentations and blog articles we write... Well hangon I guess both of those mediums are a form of peer review right? I wonder how the compare to a medical case study?

I'd love to hear other peoples thoughts on the thoughts I've posed here? Perhaps someone has some scientific training that they could bring to bear on my musings?

</ponder>

Cheers,

Jim P

@pjimmy

[Liam McLennan]

I think we can do better than anecdotal evidence. I always challenge people to articulate why they hold their opinion, mostly as a way to weed out dogma.

--
--
You received this message because you are subscribed to the "ozaltdotnet" group.
To post: send an email to ozalt...@googlegroups.com
To unsubscribe: send an email to ozaltdotnet...@googlegroups.com
For more options, visit this group at
http://groups.google.com.au/group/ozaltdotnet?hl=en-GB
 
Find us on the web at http://ozalt.net
---
You received this message because you are subscribed to the Google Groups "ozaltdotnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ozaltdotnet...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrew Stone]

Mr P, 

I agree with your sentiment and also yours too Liam.

One thing that springs to mind, is that I guess RFCs are trying be part of such a process: http://en.wikipedia.org/wiki/Request_for_Comments 

But they don't seem to cover much related to software itself more a coverage of protocols and the like?

Stonie.

[Jimmy P]

@liam: what do you accept as adequate justification of an opinion?

@stonie: yeah man RFC's are perhaps an interesting example of peer review? They are wholly synthetic ideas though right? Not like a case study?

Cheers,

Jim P

@pjimmy

[Liam McLennan]

Any logical argument. For software it often relates to code quality metrics.

The effectiveness of this depends on a degree of shared understanding. The argument "X is good because it is a pure function" relies on a shared understanding that pure functions are good. Going back to first principles constantly is not practical.

[Andrew Stone]

I agree with Liam's advice.

@ They are wholly synthetic ideas though right? Not like a case study?

 

@Mr P, depends what you mean by "synthetic"? We work within a man made environment (like a particular stack\compiler\os for example) therefore you are often bound by the decisions that those before you have made.

In practical terms, I tend to debate with as many _smart_ people as possible about "X" within a time box that is practical for the task, hopefully gaining a head start... then experiment and derive my own proof through doing, building and testing. I can't risk my reputation or my company reputation by believing marketeers or Fanbois (preaching to the converted here I'm sure).

Worked well for me so far, I also recognise that things change quickly in software, some good, some bad... there are lots of false leads along the way. I try and challenge my own ideas as much as I can. 

I would love for us to have a more rigorous scientific approach to software, hopefully we will. For now it seems to me that building great software contains applied science, art and craftsmanship. and in my case at least... lots of caffeine. :) 

Stonie.

[Andrew Clancy]

There are plenty of studies out there if you care to google it, and I agree: dyor, developers should prove new technologies & paradigms to themselves (especially when like this one the movement is massive and only growing), however I've had this type of argument too many times with engineers who are stuck in the mud over the years. Orm (vs stored ), tdd, agile, wpf vs winforms, nosql etc, where the concepts may have been new but bleedingly obvious as to the benefits.
Btw if you need some evidence: I'm an aussie working in London finance, and all of the banks are converting their trading platforms to this type of stack

[Jonathan Parker]

I think lack of regulation is the problem. If we don't start self regulating then we will be left with enforced government regulation.

And when I say 'we' I mean everyone involved in the software development process, not just developers.

[Andrew Clancy]

This I was in the wrong thread there ;) thought I was on the angular one ...

[Stephen Price]

The proof is in the pudding, so to speak. 

We need unit tests that validate said arguments. Executable arguments. 

Leads me to wondering what kind of code coverage we'll end up with. :)

[Stephen Price]

Oh dear. It still made sense. :p

[Andrew Stone]

Oh dear. It still made sense. :p

Yeah that's what I thought! :) This mailing list has not seen so much action in years! I think Andrew can be forgiven for the misplacement! :)

[Andrew Clancy]

I guess got lack of much time (I might expand later) I'll say:
Coding is a highly creative industry, many breakthroughs and revelations have been made off the back of single or few developers going after an idea that simply makes sense to them, without evidence. Developers are a very smart bunch.
However, the discussion on the other thread simply isn't anecdotal, for those with their finger on the pulse it is a large and growing paradigm, and the evidence is hard to miss.
I guess I also feel, for me this type of list is for discussion of ideas, take them with a grain of salt and research the ones you find interesting

[Andrew Clancy]

*got=for. Stupid swype

[Andrew Clancy]

Indeed. Discussion is good, it keeps us thinking& opens our minds. Twitter doesn't match this medium for that

[Jimmy P]

Related http://evidence-based-software-engineering.eventbrite.com.au/