On Thu, Feb 12, 2009 at 11:08 PM, Obie Fernandez

We've discussed the potential of a somewhat objective audit before.
One of the things I'm personally still unclear on is whether doing
these audits is something that works well in a distributed (aka
"meatcloud") fashion, or whether it works better in a centralized
fashion.  Does the emergent intelligence of crowds (the same emergent
intelligence that drives large-scale open source to success) sift out
the identify of the high quality software teams, or can such high
quality only really be discerned by other high-quality software teams?

If the latter is the case there's a clear bootstrapping issue.

Regardless, because it may nonetheless be the case that the builders
of good software can only be recognized by rare quality practitioners,
there's the unfortunate ballast of experience:  every certification or
large-scale auditing process in software that I can recall has, for
all practical purposes, failed due to either widespread skepticism,
outright corruption, or inherent bias.

It's been my limited experience in auditing software and working with
(and after) other developers, that there are a number of objective
metrics one can put into play.  Any of them can be "gamed", as has
been hinted earlier in this discussions.  In aggregate, however, it is
very difficult to game the majority of them.

Furthermore, In reality, where we happen to live, there seems to
actually be a power law at work:  the vast majority of software and
teams plying our trade are so bad that they fail nearly every metric
we could reasonably apply.  Only the vanishing minority of products
and practitioners pass even a single gameable metric.  Yet we
ultimately have little means of discerning for ourselves, for our
clients (or for differentiating for our future clients) the difference
between one or the other.

To be able to deploy a standard of comparison (whether distributed or
centrally administered and computed) that forces products or teams to
either score in the vast gutter of mediocrity, or work diligently to
at least game performance on a number of axes (when such gaming is,
realistically, often more difficult than just working to better
oneself) to separate from the pack -- would imply a vast improvement
in the current and long-standing state of affairs.

In my limited experience the difference is usually striking and
obvious:  "this is trash" vs. "wow, this is pretty good".  In the
Rails community the latter is really only elicited by the products of
about 10-20 shops (those shops Obie was trying to catalog in one of
his solicitations this evening).  It would be great if we could get
that number up to 50, 100, or even 1000.  I'd like to believe that
putting a workable set of metrics out there would eventially bias
progress in that direction.

Best,
Rick