A couple of Railsconfs ago, Courtenay and I did indeed discuss
certification and I advocated an organization to set guidelines for
what the characteristics and measurements that set apart a successful
and disciplined Rails shop apart from the unwashed masses. Rather than
trying to certify individuals (too easy to game!) I think it would be
useful to have a certification process for organizations that involved
an actual extensive interview process and audit of code and practices,
with associated scorecards and registration in some sort of official
directory. Yes, you would have to pay handsomely to get this
certification, and the result would not necessarily be what you
expect. The details for this is all very rough and frankly I don't
know the value of it, just an idea that's been in the back of my mind
for a long time so I'm  putting it out there for debate.

The name and concept is somewhat influenced by CMM, which I suppose is
anathema to most Agilistas. I'm the first to admit that I don't know
much about CMM other than it provides a scoring system for
organizations.

In a Rails context, you would establish a scorecard that categorized
your shop on a scale of 0 to 3 (or whatever, just thinking out loud
here)

RMM0 "aka Cowboy level"
- No formal development process
- No test coverage
- No standardized business practices
- Static analysis failures

RMM1 and RMM2 would have to be something in-between. RMM1 is
considered negative. RMM2 is considered positive.

RMM3 "aka Master level" (Purposely exclusive territory here, I can
imagine that only a handful of shops in the world could achieve this
level!)
- Agile software development practices
- 100% test coverage WITH THE APPROPRIATE TYPES OF TESTS (For
instance, at Hashrocket lately we are doing much less unit testing at
the MVC level because automated acceptance and integration testing
with Cucumber is so powerful and effective.
- 100% pair-programming (muahaha)
- Formal and standardized business practices
- Institutionalized continuous learning and process improvement
- Positive customer testimonials
- Successful deployment of Rails application(s) with substantial scaling demands

Incidentally, I'm bringing this up for discussion, but I might be
interested in a joint-venture along these lines with an out of work
senior Rails developer that feels like taking the idea and running
with it. (Rick?) In fact I can envision the idea expanding to the
point where being a RMM auditor could be a profitable little part-time
gig for Rails freelancers with the right personality type and
enthusiasm.

Obie Fernandez
CEO & Founder | Hashrocket
904.435.1671 office
404.934.9201 mobile

Hashrocket, Inc.
320 N 1st Street
Suite 712
Jacksonville Beach, FL 32250

http://hashrocket.com
http://obiefernandez.com

Who decides what's appropriate?  According to what metrics?  Is this
testing all possible branches?  And if you're doing integration
testing, how do you cover all possible datasets?  This just seems way
too hard to quantify and too easy to game.

Again, this is way too easy to game; you could hire a seasoned expert
and a "net-loss" programmer and stick them together, and you'd have
"pair programming."  And if this means that XP is the One True Agile
Process, that cuts Crystal Clear, Scrum, DSDM and others completely
out of the loop, and that can't be right...

Again, may have little or nothing to do with the quality of the work
done.  There have been plenty of technical snowjobs done that the
client didn't discover until well after.

Honestly, I think the only thing that counts is the positive
testimonials of your peers; by which I mean that the next engineer
coming in after you've left, taking over your code, should be happy
with it and can stand up and defend it as if it were his own.

There have been places I've come into where I've seen good quality
code holding the system together, and I didn't have to ask who had
written it; it was obvious just from looking at it that these guys
were from a pro consultant shop.

Ironically, what really made it obvious to me was not that they'd used
DSLs, metaprogramming or self-modifying code; it was that they clearly
knew when it was appropriate to use those tools and when not to.  In
some cases they specifically went with a lower tech solution because
it was more maintainable, even though it wasn't quite as cool as what
they wishfully described in the comments.  They'd clearly considered
their audience, and were writing with the next guy in mind.

See, this is so nebulous, and so hard.  Scaling is not something you
can do in the context of a six month project, and the nature of a
software application is to have bottlenecks and features that pop up
in the oddest places in the strangest ways.  You can quote the CAP
theorum and wave memcached around, but I don't see how you can say
"substantial scaling" without quantifying it further.

The reason that CMM is so reviled amongst programmers is because the
system is inherently corrupt.  The CMM evaluators are paid by the
people who are being evaluated, and they're the ones who make an
industry out of giving people high marks for having a process.  Having
a high CMM process doesn't mean you actually produce software more
efficiently (although there's a Construx presentation that says
differently, I believe they're taking the evaluator's reports and
repackaging it) -- it just means that you have a well defined process,
and (at the highest level) can predict the number of bugs you get out
of that process.

Will.

The auditor, based on guidelines established by a group like this one?
Believe me I realize the challenges in making something like this work
in practice. The auditors would have to have "know it when I see it"
capabilities for judging code quality.

I run one of the only shops that I know of that does proper and
effective pair-programming ALWAYS, even onsite with clients, etc.
Hence the evil laugh... the only other shop that I know for a fact
operates similarly is Pivotal. I admit that it might not be vital it
is to attach this criterion to RMM -- on the other hand, good Agile
practices are indispensible nowadays.

Flawed reasoning. Pair-programming is included in XP but is really its
own thing apart from the particular Agile implementation you choose.
Besides, XP is the only Agile practice that I'm aware of which
"manages down" to the developer level and dictates how they operate.
Scrum and others manage up in terms of releases and stakeholders.
Right? We pick and choose whatever works from XP, Scrum, etc..

Dunno about that. You might be able to snow your clients, but you
shouldn't be able to snow the auditor.

Agreed in principle. In practice, the client devs taking over are
usually not as good as we are and don't have a basis to judge, nor do
I see a good system for tallying that data. Actually, we've known
client devs to fuck up our code almost immediately. One of the worst
feelings in this business... :-/

That's what I would want the auditor to do.

True.

I agree. Needs a lot more thought, but in theory there must be some
way of establishing whether a shop has delivered real, scalable
solutions rather than just toy apps.

Totally. What if you had to pay a non-refundable fee to RMM upfront in
order to get evaluated. We're not such a huge "industry" that we
couldn't use that money to compensate a neutral and properly-qualified
auditor -- one that wouldn't be subject to coercion or bribing.

On Thu, Feb 12, 2009 at 11:08 PM, Obie Fernandez

We've discussed the potential of a somewhat objective audit before.
One of the things I'm personally still unclear on is whether doing
these audits is something that works well in a distributed (aka
"meatcloud") fashion, or whether it works better in a centralized
fashion.  Does the emergent intelligence of crowds (the same emergent
intelligence that drives large-scale open source to success) sift out
the identify of the high quality software teams, or can such high
quality only really be discerned by other high-quality software teams?

If the latter is the case there's a clear bootstrapping issue.

Regardless, because it may nonetheless be the case that the builders
of good software can only be recognized by rare quality practitioners,
there's the unfortunate ballast of experience:  every certification or
large-scale auditing process in software that I can recall has, for
all practical purposes, failed due to either widespread skepticism,
outright corruption, or inherent bias.

It's been my limited experience in auditing software and working with
(and after) other developers, that there are a number of objective
metrics one can put into play.  Any of them can be "gamed", as has
been hinted earlier in this discussions.  In aggregate, however, it is
very difficult to game the majority of them.

Furthermore, In reality, where we happen to live, there seems to
actually be a power law at work:  the vast majority of software and
teams plying our trade are so bad that they fail nearly every metric
we could reasonably apply.  Only the vanishing minority of products
and practitioners pass even a single gameable metric.  Yet we
ultimately have little means of discerning for ourselves, for our
clients (or for differentiating for our future clients) the difference
between one or the other.

To be able to deploy a standard of comparison (whether distributed or
centrally administered and computed) that forces products or teams to
either score in the vast gutter of mediocrity, or work diligently to
at least game performance on a number of axes (when such gaming is,
realistically, often more difficult than just working to better
oneself) to separate from the pack -- would imply a vast improvement
in the current and long-standing state of affairs.

In my limited experience the difference is usually striking and
obvious:  "this is trash" vs. "wow, this is pretty good".  In the
Rails community the latter is really only elicited by the products of
about 10-20 shops (those shops Obie was trying to catalog in one of
his solicitations this evening).  It would be great if we could get
that number up to 50, 100, or even 1000.  I'd like to believe that
putting a workable set of metrics out there would eventially bias
progress in that direction.

Best,
Rick

On Thu, Feb 12, 2009 at 11:08 PM, Obie Fernandez

We've discussed the potential of a somewhat objective audit before.
One of the things I'm personally still unclear on is whether doing
these audits is something that works well in a distributed (aka
"meatcloud") fashion, or whether it works better in a centralized
fashion.  Does the emergent intelligence of crowds (the same emergent
intelligence that drives large-scale open source to success) sift out
the identify of the high quality software teams, or can such high
quality only really be discerned by other high-quality software teams?

If the latter is the case there's a clear bootstrapping issue.

Regardless, because it may nonetheless be the case that the builders
of good software can only be recognized by rare quality practitioners,
there's the unfortunate ballast of experience:  every certification or
large-scale auditing process in software that I can recall has, for
all practical purposes, failed due to either widespread skepticism,
outright corruption, or inherent bias.

It's been my limited experience in auditing software and working with
(and after) other developers, that there are a number of objective
metrics one can put into play.  Any of them can be "gamed", as has
been hinted earlier in this discussions.  In aggregate, however, it is
very difficult to game the majority of them.

Furthermore, In reality, where we happen to live, there seems to
actually be a power law at work:  the vast majority of software and
teams plying our trade are so bad that they fail nearly every metric
we could reasonably apply.  Only the vanishing minority of products
and practitioners pass even a single gameable metric.  Yet we
ultimately have little means of discerning for ourselves, for our
clients (or for differentiating for our future clients) the difference
between one or the other.

To be able to deploy a standard of comparison (whether distributed or
centrally administered and computed) that forces products or teams to
either score in the vast gutter of mediocrity, or work diligently to
at least game performance on a number of axes (when such gaming is,
realistically, often more difficult than just working to better
oneself) to separate from the pack -- would imply a vast improvement
in the current and long-standing state of affairs.

In my limited experience the difference is usually striking and
obvious:  "this is trash" vs. "wow, this is pretty good".  In the
Rails community the latter is really only elicited by the products of
about 10-20 shops (those shops Obie was trying to catalog in one of
his solicitations this evening).  It would be great if we could get
that number up to 50, 100, or even 1000.  I'd like to believe that
putting a workable set of metrics out there would eventially bias
progress in that direction.

Best,
Rick

It will not work on a distributed basis because of issues related to
privacy and confidentiality. I simply can't publish 90% of the work we
do in any significant way because my clients would freak out. On the
other hand, getting an auditor in here with a signed NDA for a day or
two to review our code and processes seems doable.

Why? Because it would take an objective auditor to step up to the
plate? Because he would need to be vetted by some initial group, which
itself would have nobody to audit them? I don't think that challenge
is unsurmountable.

I wouldn't consider that a reason not to try.

That's right.

I suspect that's everywhere, not just where you live. We are asked to
rescue atrocities on a regular basis.

Clearly published standards would go a long way towards starting to
establish criteria for judgment. It will be a challenge to write the
criteria descriptions in such a way that it educated interested
parties about what they should be seeking. If it succeeded at that,
the project would be an educational resource and net positive for the
overall community. Wouldn't it?

Right! I fully expect that if such a system was put into place, that
only a handful of companies would qualify for the top-level RMM
certification. Maybe a slightly larger would qualify for the RMM level
right below top-level, and feel that it is a selling point. To the
extent that the rest of our "industry" admired the people at the top,
they might work towards qualifying. Idealistic, yes...

Yes, yes, yes! Amen, brother! Thanks for clearly expressing what I was
suggesting!

Cheers,
Obie

On Thu, Feb 12, 2009 at 11:43 PM, Obie Fernandez

:-)

My suggestion, then, would be that the process be two-tiered.  The
first tier is a suite of basically automated metrics that most
software in the field will probably fail spectacularly:  test
coverage, cyclomatic complexity, lack of revision control :-),
revision control code churn, class/module/function coupling, etc.
There's a maintained, publicly available package of tools along with a
set of accompanying standards documents.  Anyone can pull the suite
tools, run them over their software, and derive an aggregate metric
that's gameable, but tends to indicate that at least something is
probably going right if the scores are high.  The tools needed to do
this are in large part already available and open source.

The second tier would be an NDA-agreeable, rigorous professional audit
of code and/or practices, conducted by expert(s) in developing and
assessing software and practices.  I'm presuming here that we dodge
the bootstrap process by saying (a) those who care probably can
identify the proper auditors, and (b) if we make the audit process
open enough then a bad bootstrapping could be supplanted by better
experts following the same recipe.

The gateway to the second tier is a reasonable score on the first
tier, as well as presumably some sort of cost barrier -- to both
defray costs and to squelch overwhelming demand to use the auditing
resource. In theory one could get a limited per-application expert
audit or a more comprehensive team-and-process audit.

The net result, presuming the two tiers are evolved over time to try
to promote and identify the best techniques and methodologies, would
be to provide a way for developers to tune their processes cheaply, to
really hone their craft (more expensively), and to be able to
differentiate quality from hackery.

Best,
Rick

We've been discussing this sort of thing internally for a few months.

I strongly believe that Rails is being held back by a shortage of
skilled developers.

One of the most common questions our sales people hear is "Do you know
any good developers?" :-)

One small nitpick: I don't think it's fair to exclude the skilled
single developers out of the highest level certification because they
cannot pair program. I think pairing works, but good code is good code
at the end of the day. A strong organization with multiple developers
working on a project is a benefit, to be sure, but I don't see it as
part of a skills/deeds based certification.

I'd say the same thing about Positive customer testimonials. After
all, a dev shop with a solid balance sheet is a benefit to many
customers, but are we going to require open books before
certification? :-)

--
-- Tom Mornini
-- Founder and CTO
-- Engine Yard, Inc.

On Feb 12, 7:45 pm, Obie Fernandez <obiefernan...@gmail.com> wrote:

I don't think it requires two people at one table to type "rails
foobar" and "cap deploy".

My gut feeling reading through Obie's "Hashrocket Licensing
Certification" post is that Expert Rails == Expert Agile. No ifs/buts/
maybes. Or perhaps Expert Rails requires Expert Agile. I disagree. As
the owner of a Rails consultancy, Mocra, where we wholeheartedly sell
agile to clients, I nonetheless rebuke the notion that pair
programming is a requirement for an Expert Rails Developer or even an
Expert Rails Consultancy.

The aim of being an "Expert Developer" is to get the job done upfront
and long term. An Expert Rails Developer would know how to use Rails
to do this. An Expert Agile Developer would use Agile processes to do
this. My guess is that Scrum (tm) and other agile communities have
their certifications and two-day mastery courses/certifications all
bedded down.

In my opinion, any Rails/Ruby/JavaScript/Unix/TDD/BDD super combo
certification should be independent of development process. I don't
think any "Learning Rails" or "Expert Rails" books have chapters on
how to pair program, and at best their references to testing is a
summary of the test APIs. I don't think DHH pair programs.

How many people write unit tests for their JavaScript? 5-10% maybe. I
believe that its more important that an Expert Rails developer is an
Expert JavaScript developer than that they subscribe to any
development process/philosophy. Most Rails developers still treat
their JavaScript like a toy. Or like icing on a cake: you slap it on
the top and hope it hasn't melted in the morning.

I do pair program. Some times. And other times I don't. I always TDD
as long as I can figure out how the f@#$ to do it for the given
rubygem/cocoa GUI/rails webapp/javascript badge that I'm working on.
How many people write integration tests for RubyGems or unit tests for
JavaScript? Both are more critical skills for an Expert Rails
developer than the ability to work with a 2nd person 8 hrs a day x 5
days a week.

Penultimately, if there is to be the notion of a "consultancy"
certification: if you have 5 developers when you spend the 1 mth
attempting to get certified, and over the next 12 mths you lose 1 and
hire 10 others, what then? Does the staff who left carry their
certification with them? Do the new 10 staff automatically get the
consultancy's rating?

Finally, how does an "official" "We're an Awesome Rails consultancy"
badge on their website change the pure dynamics of capitalism? Good
companies will get referrals, get more business and stay around. Bad
companies will just get maintenance jobs on old PHP sites. If we have
a "We're so Awesome" official institution then we're going to take our
eyes off the ball - writing awesome code - and get back to dark days
of Rails in the beginning when every 3rd blog post was defending
Rails. Except now we'll be attacking or defending the "We're so
Awesome" brand: people who have the official mark will wish to defend
their expensive certification, and people who don't care about it will
spend their time rubbishing the very concept. Our community will look
silly. More importantly we won't be Awesome anymore.

Cheers
Dr NIc

On Feb 13, 4:45 pm, "tmorn...@engineyard.com" <tmorn...@gmail.com>
wrote:

I think this whole process is too subjective to try to wrap rules around it.
 The successful business are the ones who keep clients and attract new
business.  The ones who aren't will either be called out by their peers or
not attract new business.
The problem with formalization is the assertion that there is a golden path
to software development.  Unfortunately, this isn't true because practices
will have to vary depending on the client industry and economic conditions.

From my experience, your clients don't really care how you operate.  They
care that there is an appearance that they are getting decent quality
software at a decent price in a decent timeframe.  Coming up with lists with
guidelines only benefits those who actually understand what all that jargon
means.

+1 to both Dr. Nic and Bryan. It appears once more that larger shops will
gain favoritism and be the ones determining these guidelines. Process (how
you do what you do) and performance (providing a result) should be
separated. Just because a shop isn't agile (or only has one person) does not
mean that they do not produce high-quality software.

Also, going on the subject of testing, as Bryan well knows, that's a
controversial topic in of itself. Do we test views? Would a certification
impose that on us? What level of unit testing is acceptable? Salesforce goes
for 70%+ test coverage (with 100% passing tests). They're huge, have lot's
of money, and changed their industry. Should that then be the metric, even
though they aren't doing Rails?

No client has ever come to us and asked about certification, insurance, etc.
They look at what we've done, talk to our clients, and have candid
conversations with me.

Having said that, large enterprises might care about certifications, but I
have yet to have any of our huge clients ask for one.

Sincerely,

Robert Dempsey, CEO
Atlantic Dominion Solutions

http://adsdevshop.com
http://twitter.com/rdempsey
Phone: 321-274-4684

On Feb 12, 10:45 pm, Obie Fernandez <obiefernan...@gmail.com> wrote:

This statement is quite worrisome, and sounds like it's trying to
limit competition in the Rails space

Obie - can you clarify what you mean by this?

- Robert Dempsey

On Feb 13, 2009, at 8:56 AM, Robert Dempsey wrote:

Agreed.

Clients do ask about insurance in contracts and whatnot. But no one's  
ever demanded certification. To be fair, none exists for Rails, so  
there's nothing to ask for. But even when we were an all-Microsoft  
shop, we were never Microsoft Certified. But no one cared. We did  
great work and that's what mattered.

I say, let the markets decide.

Colin

Colin A. Bartlett
Kinetic Web Solutions
http://blog.kineticweb.com

Sorry, that was only meant to say that you can't *buy* a top-level RMM
grading. That's all.

Maybe we need a Rails Immaturity Model instead! :)
http://en.wikipedia.org/wiki/Capability_Immaturity_Model

I agree with Colin. Microsoft Certification is valuable to some firms
because of the marketing power.  The Microsoft brand is one of the
most powerful in the world, beating out Coke and GE in some cases.
So, companies go through the hoops and pay the big bucks to become
associated with the brand.  It often does little to ensure success of
any given project.

Rails, on the other hand, explicitly and purposefully does not have
such dominant brand awareness in end-users or clients. In
http://www.loudthinking.com/posts/6-why-theres-no-rails-inc  David
says, "The growth of the Rails ecosystem has been staggering. There
are so many shops out there offering Rails consulting and training. I
believe part of that proliferation is due to the fact that there's no
core-group monopoly that can dominate the market."  You could easily
drop in "Rails certification monopoly" in place of core-group monopoly
and have a similar growth impacting effect.

RMM could be a good idea for shops that want to improve.  Continuous
improvement is a good thing and RMM could provide a framework for
that.  Yet the certification piece would be completely and 100% a
marketing tool granted by the Rails-shop certification monopoly.  In
that light, it starts to sound like Microsoft Gold Partner
certification without the power of the brand.

_ Scott

I am a -1 on the whole certification idea.  I work for a large company
where certifications are almost required in order to get new business
(ISO 9000, CMMI, etc).  It is very costly to get certificated and to
keep the certification.  There is a whole industry around doing CMMI
based training and assessment.  While our site is CMMI level 5, I know
of projects that don't follow CMMI level 1 principles.

Obie, I would read this paper about CMMI and Agile to get an idea of
how the very process oriented companies look at Agile.  It would be
interesting to see where Hashrocket might rank within the CMMI scale
(at least a 3 and maybe a 5 in my mind)
http://www.sei.cmu.edu/news-at-sei/whats-new/cmmiagile.html

Maybe a certification is overkill but a website where companies can
register and document the level of process they implement allowing for
customers to provide feedback so they could get ranked.  If that kind
of site took off, it would give prospective customers a way to compare
apples to apples.

Just my 2 cents...
Chris

On Feb 13, 9:38 am, Obie Fernandez <obiefernan...@gmail.com> wrote:

This is a terrible idea. Anytime you create a certification model what
you'll find is that people/shops will only be working towards that
model. The strength in the Ruby community has been innovation and the
freedom to try new things and ideas. If clients begin to believe that
the only good shops are those that are RMM compliant you will quickly
see a death to this spirit of innovation.

- Brian Cardarella

On Feb 13, 10:28 am, Chris W <chris.d.willi...@gmail.com> wrote:

Let's disconnect the concept of a Rails Maturity Model and what that
would entail from the concept of official certification, since
certification implies a bunch of bad stuff and politics.

On Feb 13, 7:26 am, Colin Bartlett <colinbartl...@gmail.com> wrote:

I agree with Colin.  Great results is what matters and a free market
can sort that out.

In "Why there's no Rails Inc" http://www.loudthinking.com/posts/6-why-theres-no-rails-inc,
David said:
"The growth of the Rails ecosystem has been staggering. There are so
many shops out there offering Rails consulting and training. I believe
part of that proliferation is due to the fact that there's no core-
group monopoly that can dominate the market."  This proposal sounds
like you'd like to substitute 'core-group monopoly' with 'Rails-
certification monopoly'.

In any industry, certification is all about and only about marketing.
Firms go through the hoops and pay big bucks to be a Microsoft Gold
partner (which requires certification) because the Microsoft brand is
so strong - stronger than Coke and GE in many cases.  RMM could be a
good idea for firms that want a framework for continuos improvement,
certainly.  Yet the RMM certification piece seems like Microsoft Gold
without the power of a brand.

Well summarized. My feelings exactly.

..nap

On Feb 13, 2009, at 10:50 AM, smeade wrote:

You can try to disconnect the word "certification" but the idea is
still implied and the end will still be the same.

I think a better idea would be to follow the model of the medical
community. Have some sort of apprenticeship process for individual
developers. Let the shops worry about the skills of those that are
being hired and let the results build a reputation that speak to the
clients.

This RMM idea feels too much like a 4-star rating process that
companies can brag about.

- Brian Cardarella

On Feb 13, 10:49 am, Obie Fernandez <obiefernan...@gmail.com> wrote:

With the example of CMM/CMMI out in the wild as an example, the notion  
that you can disconnect any sort of maturity model from "bad stuff and  
politics" seems rather questionable.

Mike

On Feb 13, 2009, at 9:49 AM, Obie Fernandez wrote:

In a previous life I had some pretty negative experiences with both
individual certifications, as well as certifications of
organizations.  As a longtime Microsoft developer I obtained a number
of certifications (MCSD, MCSDBA, MCAD) and while they helped me to get
some more cash out of my employer they were pretty useless.  I can't
tell you how many people I interviewed that held all sorts of
certifications but couldn't complete the most basic coding
assignment.  At an organizational level my employer at the time was
also a Microsoft Certified Partner.  The only reason for applying and
obtaining that level of certification was to get more business.  Being
certified didn't make us anymore qualified to build great software
then a guy working out of his basement it just meant we had the money
to buy into the program and send our consultants to get their certs,
and he didn't.

These past experiences make it difficult for me to value any sort of
certification.  I see certifications as a way for folks to make money
by giving those who wish to be certified a false sense of
accomplishment and belonging.  I don't see a way for it to happen that
doesn't involve some seriously negative ramifications for all those
involved.  Sure, it might make someone some money, and sure it might
make a big corporate client feel all warm and fuzzy by working with a
"Rails Certified Master", but at the end of the day, it won't make
anyone more likely to succeed at building and delivering great
software.  Building great software is too hard to be compiled down to
a checklist, and I, for one, am 100% against the idea of putting
something together that implies otherwise.

Cheers,
Steve

+1

Jonathan

--
Jonathan Weiss
http://blog.innerewut.de
http://twitter.com/jweiss