Myopic View of DevOps Misses the Mark

[Guillaume FORTAINE]

Hello,

I would greatly appreciate to have the point of view of the DevOps
community about these various questions, if
possible, please :

http://pleasediscuss.com/andimann/20100326/myopic-devops-misses-the-mark/

a) Who handles ongoing support, especially software update for the
unrestrained sprawl of non-standard systems and components.

b) Who ensures each new application doesn’t interfere with existing
and especially legacy systems (and networks, storage, etc.)?

c) Who handles integration with common production systems that cannot
be encapsulated in a VM, like storage arrays (NAS, SAN), networking
fabrics, facilities, etc.

d) Who handles impact analysis, change control and rollback planning
to ensure deployment risk is understood and mitigated?

e) Who is responsible for cost containment and asset rationalization,
when devops keeps rolling out new systems and applications?

f) Who ensures reporting, compliance, data updates, log maintenance,
Db administration, etc. are built into the applications, and
integrated with standard management tools?

g) Who will assure functional isolation, role-based access controls,
change auditing, event management, and configuration control to secure
applications, data, and compliance?

I look forward to your answer,

Best Regards,

Guillaume FORTAINE
gfor...@gfortaine.biz
+33(0)631.092.519

[Jon Wood]

On Sunday, 26 June 2011 at 10:21, Guillaume FORTAINE wrote:

Hello,

I would greatly appreciate to have the point of view of the DevOps
community about these various questions, if
possible, please :

The answer to all those questions is the same - your sysadmin team. "Devops" is almost by definition not a team in itself, but greater collaboration between teams. Just because your developers and sysadmins have decided to work closer together doesn't mean that everyone stops specialising, and the ops team suddenly doesn't have any time to maintain your infrastructure because they're too busy building autocompletion into your software.

These considerations don't stop being made, but with teams that are working closely they can be made with a better insight into everybody's requirements, rather then the ops team passing down decrees from on high.

Jon

[Kief Morris]

On Sun, Jun 26, 2011 at 10:27 AM, Jon Wood <j...@blankpad.net> wrote:
> On Sunday, 26 June 2011 at 10:21, Guillaume FORTAINE wrote:
>>
>> Hello,
>>
>> I would greatly appreciate to have the point of view of the DevOps
>> community about these various questions, if
>> possible, please :
>
> The answer to all those questions is the same - your sysadmin team. "Devops"
> is almost by definition not a team in itself, but greater collaboration
> between teams. Just because your developers and sysadmins have decided to
> work closer together doesn't mean that everyone stops specialising, and the
> ops team suddenly doesn't have any time to maintain your infrastructure
> because they're too busy building autocompletion into your software.
> These considerations don't stop being made, but with teams that are working
> closely they can be made with a better insight into everybody's
> requirements, rather then the ops team passing down decrees from on high.
> Jon

I would say the answer is simply "the team". I agree the ops team will
have much of the specialist knowledge and skills around these areas,
but the idea of DevOps is for everyone to have awareness and take
ownership of these concerns, rather than focusing on their own
immediate concerns.

Kief

[Kief Morris]

On Sun, Jun 26, 2011 at 10:21 AM, Guillaume FORTAINE
<gfor...@gfortaine.biz> wrote:
> I would greatly appreciate to have the point of view of the DevOps
> community about these various questions, if
> possible, please :
>
> http://pleasediscuss.com/andimann/20100326/myopic-devops-misses-the-mark/

I agree with Andi that much of the writing about DevOps has come from
the development perspective, but Ops people who can ditch the attitude
of "[ops] manage what they are given, and keep the business running,
regardless of the mess that gets thrown over the wall at them" will
benefit greatly from DevOps. Andi is dead-on with "apps team have to
pull their weight too, by addressing gaps ..."

DevOps is an opportunity for the ops people to get involved at the
start of the development process and make sure that those various
support and operational concerns are baked in, along with making sure
that infrastructure and support changes are well understood, prepared,
and tested from the start.

This is a vast improvement over having to wait until a "finished"
product is thrown over the wall at them, and then working around the
mess to make it work.

Rather than getting our backs up about DevOps as a threat, we ops
people should seize the opportunity.

Kief

[Andrew Clay Shafer]

>
> I agree with Andi that much of the writing about DevOps has come from
> the development perspective, but Ops people who can ditch the attitude
> of "[ops] manage what they are given, and keep the business running,
> regardless of the mess that gets thrown over the wall at them" will
> benefit greatly from DevOps. Andi is dead-on with "apps team have to
> pull their weight too, by addressing gaps ..."
>

I would appreciate it if someone could point out the devops writing
that is dev centric.

I feel and hear that the opposite is true.

I responded to Andi when he first posted that.
http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/

[James Turnbull]

Agreed. The DevOpsDays events I've attended the number of dev attendees
were usually 10% or less. One of the key problems I see is that more
engagement, education and exposure is needed in the dev side of the house.

Regards

James Turnbull

--
Author of:
* Pro Puppet (http://tinyurl.com/ppuppet)
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

[Kit Plummer]

On Sunday, June 26, 2011 at 6:27 PM, James Turnbull wrote:

> Andrew Clay Shafer wrote:
> > > I agree with Andi that much of the writing about DevOps has come from
> > > the development perspective, but Ops people who can ditch the attitude
> > > of "[ops] manage what they are given, and keep the business running,
> > > regardless of the mess that gets thrown over the wall at them" will
> > > benefit greatly from DevOps. Andi is dead-on with "apps team have to
> > > pull their weight too, by addressing gaps ..."
> >
> > I would appreciate it if someone could point out the devops writing
> > that is dev centric.
> >
> > I feel and hear that the opposite is true.
> >
> > I responded to Andi when he first posted that.
> > http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/
>
> Agreed. The DevOpsDays events I've attended the number of dev attendees
> were usually 10% or less. One of the key problems I see is that more
> engagement, education and exposure is needed in the dev side of the house.
>
> Regards
>
> James Turnbull

Yep. I believe where some of the confusions lies, with regard to the Dev-to-Ops slant, is the recent hububification of "infrastructure as code". To me this just means bringing dev practices (CI, SCM, etc.) to operational tasks - which is really, really good stuff. But, it definitely isn't bringing devs to the operational table for the required collaboration.

Kit

[Guillaume FORTAINE]

Dear Mister Shafer,

Thank you for your reply.

> I responded to Andi when he first posted that.
> http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/

I have read with interest your blog post (and by the way, just bought
the book from James Turnbull "Pro Linux System Administration" that is
quoted in it :) ).

Clearly, I still don't understand what is the value-add of "DevOps"
with the current definition of its proponents and it seems the people
at LISA (Large Installation System Administration) think the same
thing :

http://everythingsysadmin.com/2010/10/devops-aint-nothing-we-havent.html

"Ok, now you know the background for what I really wanted to say. I
was inspired recently because a mailing list I'm on people were
mocking DevOps as "nothing new". The people on this list are all in
the top 10th percentile of system administrators; the cutting edge
that other IT teams hope to emulate."

"So congratulations, DevOps folks! I'm glad that you are popularizing
what some of us have been doing for years."

http://www.usenix.org/event/lisa11/cfp/

"While DevOps is new, it embodies themes long popular at LISA:
automation, performance, scaling, collaboration, and cooperation."

Maybe will I meet some of you at LISA '11 whose theme will be "DevOps:
New Challenges, Proven Values" ?

Best Regards,

Guillaume FORTAINE
gfor...@gfortaine.biz
+33(0)631.092.519

[Damon Edwards]

Guillame,

I'm not sure which "current definition of [DevOps] proponents" you are referring to, but I think people who sign up for this google group generally feel that there is value in building a community that discusses experiences and shares ideas for identifying and removing bottlenecks, inefficiencies, and conflicts that commonly plague IT organizations (especially those organizations who build and run revenue producing web-based services). 

Do you have a specific question or specific DevOps concept that you don't think is valuable? Quoting other people's old blog posts (especially when both of those people have evolved their DevOps opinions since then), doesn't explain where you are coming from or what you are getting at.

-Damon

[John Vincent]

> Clearly, I still don't understand what is the value-add of "DevOps"
> with the current definition of its proponents and it seems the people
> at LISA (Large Installation System Administration) think the same
> thing :
>
> http://everythingsysadmin.com/2010/10/devops-aint-nothing-we-havent.html
>

1) I wouldn't call a blog post by one person (even if it *IS* Tom L.)
representative of "the people at LISA". I've been a LISA attendee in
the past and I don't entirely agree with Tom's "old" article.
2) I don't know that everything has to have a value-add. DevOps isn't
something that can be bottled, labeled and sold as magic in a jar.

There are some things about DevOps that are new and some things that
are old. Countless articles and blog posts have been written about
different aspects of DevOps - cultural and technical and even some
that differ from traditional SA or Development work (devops isn't just
about SAs). Automation doesn't buy you immediate agility.
Cross-training teams doesn't buy you cooperation between those teams.

I'm honestly confused as to what value-add there needs to be.

--
John E. Vincent
http://about.me/lusis

[gareth rushgrove]

On 27 June 2011 14:38, Guillaume FORTAINE <gfor...@gfortaine.biz> wrote:
>
> Clearly, I still don't understand what is the value-add of "DevOps"
> with the current definition of its proponents and it seems the people
> at LISA (Large Installation System Administration) think the same
> thing :
>
> http://everythingsysadmin.com/2010/10/devops-aint-nothing-we-havent.html
>

I think one of the problems here is that the content you're pointing
to is pretty old. Your referenced articles are from March and October
last year. This might seem recent, but the whole coalescence around
the term devops is but 2 years old or so. Both articles referenced
also appear to be written from the point of view of someone who read
about it on the internet as well, rather than someone involved in the
active conversations either online or even better in person. Basically
it's the internets fault :)

The comments on both posts do what I think is a good job of adding to
the conversation the posts start though.

> "Ok, now you know the background for what I really wanted to say. I
> was inspired recently because a mailing list I'm on people were
> mocking DevOps as "nothing new". The people on this list are all in
> the top 10th percentile of system administrators; the cutting edge
> that other IT teams hope to emulate."
>

Here's something important though. A group of systems administrators
talking amongst themselves about how the stuff they are doing is
awesome. But does that list also have the top whatever percentile of
developers? A health scattering of testers? Because if not it's going
to be a one-sided conversation. I'm not saying that's bad or
unhelpful. I'm saying if you're looking for reasons why devops appears
popular it's incredibly inclusive nature is definitely part of it.

> "So congratulations, DevOps folks! I'm glad that you are popularizing
> what some of us have been doing for years."
>

And in any case the top few percent is one thing, popularizing
something to everyone else based on demonstrable benefits is hard
work. Even back in January Damon Edwards was talking of the DevOps
Echo Chamber and how to escape it.

http://www.agileweboperations.com/2011_escaping_devops_echo_chamber

Devops is really just a banner under which start people have been
discussing approaches to common problems and in many cases
popularizing them. Oh, and hacking on code (lots and lots of code. By
my calculation at least half the code on github is written by John
Vincent and Jordan Sissel). And talking to folks outside their
discipline, and in many cases finding common ground.

G

>
> http://www.usenix.org/event/lisa11/cfp/
>

So we move forward 9 months (who knows, maybe thanks to the comments)
from the above referenced blog post and LISA have an entire conference
under the devops banner.

> "While DevOps is new, it embodies themes long popular at LISA:
> automation, performance, scaling, collaboration, and cooperation."
>
>
> Maybe will I meet some of you at LISA '11 whose theme will be "DevOps:
> New Challenges, Proven Values" ?
>
> Best Regards,
>
> Guillaume FORTAINE
> gfor...@gfortaine.biz
> +33(0)631.092.519
>
>
> On Mon, Jun 27, 2011 at 3:13 AM, Andrew Clay Shafer
> <andrew...@gmail.com> wrote:
>>
>>>
>>> I agree with Andi that much of the writing about DevOps has come from
>>> the development perspective, but Ops people who can ditch the attitude
>>> of "[ops] manage what they are given, and keep the business running,
>>> regardless of the mess that gets thrown over the wall at them" will
>>> benefit greatly from DevOps. Andi is dead-on with "apps team have to
>>> pull their weight too, by addressing gaps ..."
>>>
>>
>> I would appreciate it if someone could point out the devops writing
>> that is dev centric.
>>
>> I feel and hear that the opposite is true.
>>
>> I responded to Andi when he first posted that.
>> http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/
>>
>

--
Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com

[Ernest Mueller]

dev...@googlegroups.com wrote on 06/26/2011 08:27:38 PM:

> From: James Turnbull <ja...@lovedthanlost.net>
> Andrew Clay Shafer wrote:
> >> I agree with Andi that much of the writing about DevOps has come from
> >> the development perspective, but Ops people who can ditch the attitude
> >> of "[ops] manage what they are given, and keep the business running,
> >> regardless of the mess that gets thrown over the wall at them" will
> >> benefit greatly from DevOps. Andi is dead-on with "apps team have to
> >> pull their weight too, by addressing gaps ..."

> > I would appreciate it if someone could point out the devops writing
> > that is dev centric.
> >
> > I feel and hear that the opposite is true.
> >
> > I responded to Andi when he first posted that.
> > http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/
>
> Agreed. The DevOpsDays events I've attended the number of dev attendees
> were usually 10% or less. One of the key problems I see is that more
> engagement, education and exposure is needed in the dev side of the house.

Absolutely, this is a big red herring. Lack of dev participation in DevOps is one of the key things we need to work on, IMO. I have been to a lot of DevOps events and they have all been close to too Ops-focused and definitely predominantly attended by ops. Except for the dangerously deluded "NoOps" people, I don't know who is even claiming this.

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

[John Vincent]

I want to slighty counter this argument about not enough dev
involvement. I've been meaning to bring this up but the amount of
developer involvement is, in my experience, pretty much regional in
nature. I've been to meeting where developers outweighed system admins
(in head count. I never see a developer built like me ;>) and I've
seen the reverse. I see quite a bit of systems code written by
developers to make their life easier.

Also Jordan Sissel has like 20 times more code on github than I do.
We're just from the same school of hate-driven development (his
phrase).

--

[gareth rushgrove]

On 27 June 2011 15:21, Ernest Mueller <ernest....@ni.com> wrote:
>>
>> Agreed. The DevOpsDays events I've attended the number of dev attendees
>> were usually 10% or less. One of the key problems I see is that more
>> engagement, education and exposure is needed in the dev side of the house.
>
> Absolutely, this is a big red herring. Lack of dev participation in DevOps
> is one of the key things we need to work on, IMO. I have been to a lot of
> DevOps events and they have all been close to too Ops-focused and definitely
> predominantly attended by ops. Except for the dangerously deluded "NoOps"
> people, I don't know who is even claiming this.
>

That's pretty interesting. I'd say most of the London events I've
organised have had a much higher developer attendance. 50:50 at times
even. I don't know if that's because I'm predominantly a developer and
have tended to organise more developer centric events or whether it's
just a London thing. The Hamburg devopsdays event as well certainly
wasn't only 10% developers.

Also given the typical topics up for discussion some developers can
impersonate ops folk pretty well in my experience.

I'd be interested in seeing if this is actually a trend. If anyone
else who's organising or attending events wants to try and gather some
data? I'll start asking at London events.

Gareth

> Ernest
> ______________________
> UN-altered REPRODUCTION and DISSEMINATION of
> this IMPORTANT information is ENCOURAGED.

--

[Ernest Mueller]

Yeah, my perspective is limited to Austin and California. Last week at DevOpsDays San Jose there was a hand poll taken and IIRC it was more than 2:1 ops to dev...

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

gareth rushgrove ---06/27/2011 09:32:46 AM---On 27 June 2011 15:21, Ernest Mueller <ernest....@ni.com> wrote: >>

[Andrew Clay Shafer]

>
> Clearly, I still don't understand what is the value-add of "DevOps"
> with the current definition of its proponents and it seems the people
> at LISA (Large Installation System Administration) think the same
> thing :
>

First, everything is derivative. devops in general is nothing new per
se, but I see it as the confluence of ideas, technologies and
techniques coming together in specific ways that can be
transformative. IMHO most of the devops community and their ideas
represent an evolution of the ideas championed at LISA for years
flavored with more webops bias and less academic/hpc bias.

Second, Tom's post and the comments capture certain perspectives at a
point in time. That's not the same thing as 'the people at LISA'.
Tom's a reasonable guy with an impressive body of work. I'm sure if we
were all sitting around a table together with Tom now, we'd probably
agree on most points.

Third, I don't know what you are using as a definition of devops, but
I personally don't think these are solved problems. devops is not a
destination, at best it is the beginning of a journey. devops is a
community of practice which is growing the body of knowledge together.
To me devops as a cultural movement is about recognizing that there
are smart people out there who have been faced with similar problems
and learning what I can from them. I would hope people would do that
anyway. You can question the name, you can question the origin of the
practices, but ultimately that is a distraction from making real
progress. Unless you know everything already... :)

[Kit Plummer]

But John, there is a difference between SysAdmins who write code and developers who implement business functionality.

Kit

--
Kit Plummer :: MaestroDev :: 520.360.4729

On Monday, June 27, 2011 at 7:28 AM, John Vincent wrote:

> I want to slighty counter this argument about not enough dev
> involvement. I've been meaning to bring this up but the amount of
> developer involvement is, in my experience, pretty much regional in
> nature. I've been to meeting where developers outweighed system admins
> (in head count. I never see a developer built like me ;>) and I've
> seen the reverse. I see quite a bit of systems code written by
> developers to make their life easier.
>
> Also Jordan Sissel has like 20 times more code on github than I do.
> We're just from the same school of hate-driven development (his
> phrase).
>

> On Mon, Jun 27, 2011 at 10:21 AM, Ernest Mueller <ernest....@ni.com (mailto:ernest....@ni.com)> wrote:
> > dev...@googlegroups.com (mailto:dev...@googlegroups.com) wrote on 06/26/2011 08:27:38 PM:

[James Turnbull]

gareth rushgrove wrote:
> That's pretty interesting. I'd say most of the London events I've
> organised have had a much higher developer attendance. 50:50 at times
> even. I don't know if that's because I'm predominantly a developer and
> have tended to organise more developer centric events or whether it's
> just a London thing. The Hamburg devopsdays event as well certainly
> wasn't only 10% developers.
>

So the last two DevOpsDays in Mountain View and the first DevOpsDays
Downunder had pretty small dev turnouts and I know Patrick has commented
on similar at other events.

> Also given the typical topics up for discussion some developers can
> impersonate ops folk pretty well in my experience.

This is something we need to change too - we need to have more topics
(not packaging for example :) ) that are primarily about the Dev in
DevOps not primarily about the Ops in DevOps.

Cheers

James

[Kit Plummer]

Yep.  One topic (as a software engineer) I'd like to see more discussion about is the notion of "self-service environment" - meaning the ability of a developer/QA to rapidly instantiate a development machine(s) that mimics the operational environment (e.g. OS, VM, framework/library, application configurations across n-machines).

I really love what Patrick has discussed here, using Vagrant:

http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/

I think there is a ton of potential in Vagrant for helping bridge Dev and Ops.  Is there a similar connection that can be made for provisioning dev environments to IaaS providers (mcollective, knife, etc.)?

One thing that I consistently don't see in organizations I talk to is the documented delivery or deployment plan/workflow/pipeline/composition.  I believe in most cases, devs don't have much understanding of what happens to their code once committed.

Lastly, another area where I think there is lots of room for conversation is in code instrumentation - specifically in measuring operational lifecycles/transactions between distributed components/sub-systems (beyond just the front-end).

Great discussion guys!

Kit

[Damon Edwards]

For about a year now I've been doing the informal "raise your hand" poll (highly scientific). I've been seeing more like 60% "ops" 40% "dev" on average... I'm putting the roles in quotes because there are all kinds of circumstances where the lines are blurred and self-identification is always tricky.

But some interesting caveats...

1. There are a good number of exceptions where it is 90% ops / 10% dev... But I have never seen the reverse.

2. I feel like the majority of DevOps "content" producers (consistent bloggers, active on lists, creating projects, organizing events, etc.) are coming from a ops (or majority ops) background.

All that being said, I do believe that we need to do a better job of attacking DevOps problems from a Dev perspective (as Ben Rockwood recently called the Dev > Ops perspective). Lots of focus has gone into bringing best practices from Dev into Ops... And not nearly as much about how Dev roles/behavior/culture needs to adapt.

[patrick.debois]

My 2cents on this:

1. Population dev vs ops:

- my observation is that the attendees of a devops meetup/devopsdays
gravitate towards the background of the organizers. It's not strange
because the selection of the topics, friends, mouth-to-mouth advertising
will reflect their interests
- in the past bringing the topic to developer/agile conferences used to
be near to impossible, now I see devops being discussed in Ruby, Php
conferences, Xpdays, Agile 2011, GotoCon, Devoxx.
- this encourages me, I never set out to start a complete separate
circuit of conferences, but somehow it needed to get bootstrapped.
Actually I hope that devops would become a track of all dev/agile
conferences ; just like there are talks for frontend developers, backend
developers, database people, QA people, there is room for devops
oriented people
- A great way to get developers interested is actually to go present at
other conferences and make them aware, much like horse/water
- At Ops is currently at the end of the chain, they feel the pain more.
But the pain is there with Dev team too. But they can get away with
grumbling.

2. It's bringing more dev into ops (and vice versa)

- Making devs aware of operational issues is certainly something devs
can learn from ops
- Ops can clearly learn from devs when it comes to testing,
Agile/Lean/Kanban folks in development

2. Are we all devs , ops, or QA

My current view is that I would be looking for what some people call
T-shaped people . People how a deep technical expertise in either dev,
ops, qa and are *willing* to extend and collaborate. Working with only
generalists will slow down your learning curve and results (YMMV)

And the discussion is not only for Sec, QA, DBA, Windows. Hence I
started the devopsdays Goteborg about devops being 'inclusive'.

3. How to get others more involved:

- metrics, monitoring, logging are all great ways to provide feedback;
but it's not enough to just point them out to devs. They need to be
valuable, f.i. during their testing
- using them for retrospectives (not only inside ops) and sharing the
learning experience to the beginning of the project chain
- the same is for automation: devs care too about reproducibility;
that's why people love things like chef/puppet and vagrant. And from my
experience it takes them less time to learn as they already used to
programming

My guess is that f.i. if we find a way to integrate the use of all these
things in their current workflow/IDE there will a much wider adoption.
In the past the set of tools Ops and Devs had been working on had been
different , now they start to be able to use the same tools. Maybe they
are not managing 1000 servers and you might find their environment
negligible as opposed to your production environment, but I believe this
is where part of the cross over will start to happen.

- why not integrating stuff in their Eclipse/Favorite IDE and make them
accustomed to the tools ops is using
- managing their development desktop with the same tools
- allowing them to deploy in the same way as you would do in production
- give them the same metrics, monitoring for their dev environment
- provide them 'self-servicing' ways to explore new options
- expose logfiles, information they can use for problem solving
- help in getting them do *their work* and they will listen

Tools don't make a difference per s�, but they can change the interest
and behavior , eventually leading to culture change.

Most of all, pair, mingle, eat sushi, or whatever. Talk to them and
don't be set aback when they don't seem interested. Instead of thinking,
them vs us. Think of what *you* can do better in each-other's usual
world , to make things frictionless from both perspectives.

Patrick

[Kief Morris]

I wonder if part of what's going on here is that many of the people driving devops are a certain flavour of ops person, pulling development oriented practices into their work. This is why there's a perception that devops is dominated by ops people, and at the same time a perception among many ops people that devops is 'developer centric' practices being pushed in their direction.

[Anthony Goddard]

Hells yeah.

On Jun 27, 2011, at 1:53 PM, patrick.debois wrote:

Tools don't make a difference per sé, but they can change the interest and behavior , eventually leading to culture change.

[John Vincent]

That's an interesting perspective. Look at things traditionally.
Developers have always "deployed" code (whether locally or to their
own servers on side projects). Tools like capistrano and fabric are
developer initiated, not operations. It's not THAT big of a leap for
them to be brought into the production deploy arena.

Meanwhile Operations people are not typically developers in the bigger
sense of the word. Sure we've been python and perl junkies, shell
script monkies and the occasional tcl folk. Yeah we could write a PHP
frontend to stop having to do things for people ourselves. But agile
never factored into it for the most part. Perl *DOES* have a rich
history of testing behind it but we've always been able to say "I
don't do java" and boot it back to the developers.

I would argue sysadmins/operations teams are having to step out of the
traditional comfort zone a bit more than developers. In that sense
right now, the operations side of the house is undergoing a massive
cultural shift. I would wager in 5 years time (if not sooner)
operations folks will simply HAVE to know more than traditional
scripting languages. Yes, we have to be developers. It might not be
the same problem domain as the official development side of the house
but companies won't tolerate a sysadmin of any sufficient seniority
who isn't willing (or capable of learning) the new model. It's all
about getting the most for your money as a business.

And don't think that developers don't have to step up to the plate
either. Everyone is in the same boat. Comfort zones are changing and
you either keep up or get left behind.

On Mon, Jun 27, 2011 at 2:51 PM, Kief Morris <kiefd...@gmail.com> wrote:
> I wonder if part of what's going on here is that many of the people driving devops are a certain flavour of ops person, pulling development oriented practices into their work. This is why there's a perception that devops is dominated by ops people, and at the same time a perception among many ops people that devops is 'developer centric' practices being pushed in their direction.
>

--

[Kit Plummer]

Again, I'm feeling like the "kind of development" thang is getting lost.

Ops will likely never be the ones committing to implement a user story, feature, bug fix, or the like. So to say that Ops "have to be developers" is true, but must be caveated with "in the scope of QA/deployment/delivery/operations.

I also think there's a bit of scope required, from a complexity perspective. And...though I don't really want to...it does bring back NoOps to the conversation. There are always going to be "projects" where dev and ops are likely the same thing (read that as NoOps). PaaS environments are somewhat proving this to be true - ala Heroku making it easy to do so. But, in projects of any scale/complexity there is a need to separate roles and responsibilities. This doesn't mean that devs can't help ops write Puppet modules, or that Ops can't help a dev implement an async email notification feature. It just means that at the end of the day one side or the other will be responsible.

Kit

--
Kit Plummer :: MaestroDev :: 520.360.4729

[Thom Allen]

John,

I agree, and as a developer, I know I need to step up and have  broader understanding of the operations role. I was recently in a situation where I had to configure, setup and manage the dev/stage/production servers for a project. This did a couple of things for me. One, it reminded me how little I knew about the OS. I can't get away with just knowing how to move and copy files. I had no idea about best practices for resource management, security and monitoring. This exercise also convinced me that as a developer, I need to know more about the domain my applications are playing in rather than relying on operations. This is partially why the DevOps movement caught my eye.

--Thom

[John Vincent]

On Mon, Jun 27, 2011 at 3:25 PM, Kit Plummer <kplu...@maestrodev.com> wrote:
> Again, I'm feeling like the "kind of development" thang is getting lost.
>
> Ops will likely never be the ones committing to implement a user story, feature, bug fix, or the like. So to say that Ops "have to be developers" is true, but must be caveated with "in the scope of QA/deployment/delivery/operations.
>

I'd probably disagree here. If we take the John Willis perspective,
right now operations teams are spending 80% of the time doing bullshit
firefighting muckity muck type work. The other 20% is providing value
to the business. As we start to automate more and the feedback loop
becomes smaller, operations folks will have more time on their hands.
More time to add value to the business. True they might not dive right
in and start implementing new features but they will.

I know one example does not a trend make but the company I was
previously at had almost this exact thing start happening. We had a
senior operations guy who took it upon himself to learn java for a
couple of reasons. One was to make the codebase more understanding of
the environment and all the automation and tools created by
operations. The other was to deal with bugs that never seemed to
register on the radar because there were acceptable work arounds. (Yes
there was a greater problem on the second one).

My point is that, while right now I'm writing glue apps (like the
mentioned email gateway or a sinatra web service that does X), I'm
just as much a part of the development team as anyone else. I just
don't write any java code.

I think we need to be careful that we don't mix title with role.
Titles are typically fixed but roles change based on need. One day I
was a websphere administrator writing jacl scripts for a deploy
console. The next day I was knee deep dealing with DB2 bufferpool hit
ratios. Yet another day I'm standing up a new firewall.

We don't seem to have this roadblock when it comes to operations folks
doing networking or security work. Why then, do we have this mental
roadblock about operations folks doing development work? It's not as
if we're incapable.

P.S. Love you long time, Kit ;)

[Thom Allen]

Kit,

I don't think I would expect Ops to have the depth of experience a seasoned developer has. But I know from experience, it would be nice if there was more than a casual understanding. And as a developer, I know I need more than a casual understanding of the environment I will be working in. For example, you mention PaaS environments. Over the past few weeks I have failed to deploy applications to a cloud based infrastructure because I didn't understand the basic configuration needs, and I didn't have an expert counterpart to help me. That's a failure on my part to think I didn't need any help.

--Thom

[Kit Plummer]

No worries John. Don't get me wrong, I want Ops to better understand my environment - and I love working with you guys. ;)

Thom, yep been there...but, that's you and me. I would never expect my junior guys - who are doing the core "business-value-development" stuff to have to worry about that. Back to my scope point. Not everyone on-the-chain should/needs to be collaborating. I suppose some of this is me thinking as a "Systems" guy too...and not just a developer.

Software architecture, which is mainly influenced by more senior dev and ops folks, is where a lot of the DevOps collaboration should happen. Throwing this out there - with a bit of instigation intended.

Kit

--
Kit Plummer :: MaestroDev :: 520.360.4729

On Monday, June 27, 2011 at 12:37 PM, Thom Allen wrote:

> Kit,
>
> I don't think I would expect Ops to have the depth of experience a seasoned developer has. But I know from experience, it would be nice if there was more than a casual understanding. And as a developer, I know I need more than a casual understanding of the environment I will be working in. For example, you mention PaaS environments. Over the past few weeks I have failed to deploy applications to a cloud based infrastructure because I didn't understand the basic configuration needs, and I didn't have an expert counterpart to help me. That's a failure on my part to think I didn't need any help.
>
> --Thom
>
>
> On Mon, Jun 27, 2011 at 1:25 PM, Kit Plummer <kplu...@maestrodev.com (mailto:kplu...@maestrodev.com)> wrote:
> > Again, I'm feeling like the "kind of development" thang is getting lost.
> >
> > Ops will likely never be the ones committing to implement a user story, feature, bug fix, or the like. So to say that Ops "have to be developers" is true, but must be caveated with "in the scope of QA/deployment/delivery/operations.
> >
> > I also think there's a bit of scope required, from a complexity perspective. And...though I don't really want to...it does bring back NoOps to the conversation. There are always going to be "projects" where dev and ops are likely the same thing (read that as NoOps). PaaS environments are somewhat proving this to be true - ala Heroku making it easy to do so. But, in projects of any scale/complexity there is a need to separate roles and responsibilities. This doesn't mean that devs can't help ops write Puppet modules, or that Ops can't help a dev implement an async email notification feature. It just means that at the end of the day one side or the other will be responsible.
> >
> > Kit
> >
> > --

> > Kit Plummer :: MaestroDev :: 520.360.4729 (tel:520.360.4729)

[Nathaniel Eliot]

At the risk of counter-instigation:

*Everybody* should be trained in collaboration, and at minimum should understand the forces their counterparts deal with. I don't expect programmers to know the ins and outs of Linux, any more than they expect me to be an amazing Ruby program. But understanding what their pain points are ASAP, and understanding how they can help me in my job, has made me a much better op than I otherwise might be. And some of the best developers to work with are those who understand what deployments and being on call mean for an admin (and why we're allergic to "just a small change" to production machines at 4:45PM on a Friday).

Far better to have all your people experienced in collaboration from day one, than to make it a rarefied skill that is only practiced at "the top". Otherwise, where are you going to pull from when the need for expert collaborators arise? How well your company react when the worst happens, and it's just three juniors in different departments who are available when a fire breaks out?

--
Nathaniel Eliot
T9 Productions

[Lindsay Holmwood]

On 28 June 2011 01:21, James Turnbull <ja...@lovedthanlost.net> wrote:
>
> So the last two DevOpsDays in Mountain View and the first DevOpsDays
> Downunder had pretty small dev turnouts and I know Patrick has commented on
> similar at other events.
>

That's not entirely accurate.

I did a snap poll on the first day of DevOps Down Under and there was
a 50/50 dev/ops split.

I did a lot of work to encourage devs to attend, but the talk content
was skewed towards ops, so there was a lot of attrition and by the
second day it had worn down to ~ 20/80 dev/ops.

It's an interesting hidden bias - we talk a lot about attracting devs,
but sometimes our conversations are too ops focused and end up driving
devs away.

Lindsay

--
w: http://holmwood.id.au/~lindsay/
t: @auxesis

[gareth rushgrove]

On 27 June 2011 16:21, James Turnbull <ja...@lovedthanlost.net> wrote:
>
>> Also given the typical topics up for discussion some developers can
>> impersonate ops folk pretty well in my experience.
>
> This is something we need to change too - we need to have more topics (not
> packaging for example :) ) that are primarily about the Dev in DevOps not
> primarily about the Ops in DevOps.
>

I'm basically the worst possible example here. I'm a developer who
gets worked up about packaging and ganglia. So what topics do people
think appeal to developers? I'm asking squarely with my organising
london events hat on.

G

> Cheers
>
> James

[gareth rushgrove]

On 27 June 2011 16:35, Kit Plummer <kplu...@maestrodev.com> wrote:
>
> Yep.  One topic (as a software engineer) I'd like to see more discussion
> about is the notion of "self-service environment" - meaning the ability of a
> developer/QA to rapidly instantiate a development machine(s) that mimics the
> operational environment (e.g. OS, VM, framework/library, application
> configurations across n-machines).
>
> I really love what Patrick has discussed here, using Vagrant:
>
> http://www.jedi.be/blog/2011/03/28/using-vagrant-as-a-team/
>

Does the world eventually become one of:

* developer provisioned vagrant(like) instances and then the same
chef/puppet/whatever recipes/manifests/whatevers use to provision
EC2/OpenStack instances
* local and remote running platform as a service, eg. Cloudfoundry.
Potentially running as above.

?

> One thing that I consistently don't see in organizations I talk to is the
> documented delivery or deployment plan/workflow/pipeline/composition.  I
> believe in most cases, devs don't have much understanding of what happens to
> their code once committed.
>

It's nearly alway documented in code. But that particular part of the
codebase isn't always shared as widely as it should be. From a
developer point of view it's also not always treated like the rest of
the code with tests and the like. Or developers try and take over with
a badly hand rolled version of rsync or the like.

G

> Lastly, another area where I think there is lots of room for conversation is
> in code instrumentation - specifically in measuring operational
> lifecycles/transactions between distributed components/sub-systems (beyond
> just the front-end).
>

> Great discussion guys!
>
> Kit
>
>

--

[gareth rushgrove]

On 27 June 2011 20:25, Kit Plummer <kplu...@maestrodev.com> wrote:
>
> I also think there's a bit of scope required, from a complexity perspective.
> And...though I don't really want to...it does bring back NoOps to the conversation.
> There are always going to be "projects" where dev and ops are likely the same thing
> (read that as NoOps). PaaS environments are somewhat proving this to be true - ala
> Heroku making it easy to do so.

I always think about Heroku as more outsourcing operations. Running a
PAAS internally is interesting though in that it introduces a hard API
in-between dev and ops, something your average devops folk are
traditionally against. I've never been sure whether NoOps really means
no operations people inside your organisation, or some mythical fairy
land where software and hardware always work 100% of the time.

G

--

[Guillaume FORTAINE]

Hello,

@ Damon Edwards

> where you are coming from

I come from Lille :) And according to your Linkedin's profile, you
come from San Francisco :) But this question is nonsense given the
Globalization of Information Systems ;)

> what you are getting at.

I am interested in approaching Operational Excellence. Hence, my
interest in DevOps that could be a mean to help me in this is never
ended quest.

> 1. There are a good number of exceptions where it is 90% ops / 10% dev... But I have never seen the reverse.
>
> 2. I feel like the majority of DevOps "content" producers (consistent bloggers, active on lists, creating projects, organizing events, etc.) are coming from a ops (or majority ops) background.
>

I don't know if, we, French people, are contrarian but the last major
DevOps events in France were organized by regional JUGs (Java User
Group). 4 to be more precise (Breizh JUG, Ch'ti JUG, Paris JUG, Mars
JUG) :

http://parisdevops.fr/blog/2011/06/08/devops-debarque-chez-vous.html

@ John Vincent

> 1) I wouldn't call a blog post by one person (even if it *IS* Tom L.)

> representative of "the people at LISA".

One can reasonably call representative the point of view of the
program co-chair of LISA. Though, I agree with Andrew Shafer that, to
quote him, "Tom Limoncelli's post and the comments capture certain

perspectives at a point in time."

> 2) I don't know that everything has to have a value-add.

Everything needs to have value-add. One can't only have as leitmotiv :
"Let's people be happy (but give them a bunch of money)". It doesn't
work like this ;)

According to my understanding, there is a clear value-add in DevOps in
helping to solve the question : How ensure a match between the
Information System and the business objectives ?

@ gareth rushgrove

> So we move forward 9 months (who knows, maybe thanks to the comments)
> from the above referenced blog post and LISA have an entire conference
> under the devops banner.

Maybe will we be able to drink a beer together there to discuss further ? :)

@ Andrew Shafer

> Third, I don't know what you are using as a definition of devops,

"Let's people be happy (but give them a bunch of money)".

> Unless you know everything already... :)

No one can know everything :) But by aggregating the knowledge of
everybody, you could, at least, have a better overview ;) Hence, my
DevOpSpace project :

http://www.linkedin.com/company/devopspace

@ Patrick Debois

Thank you for your feedback. Question : Why didn't you manage to
successfully promote DevOps in Europe ?

I look forward to your answer,

[Scott Smith]

In my experience, the split is more about personnel than time. In larger groups, anyway. Like it's 80% of the people are doing bullshit firefighting and 20% are doing interesting work. They both work out to be the same, but yeah.

Regardless, I think a lot of this is due to the fact that the "long tail," if you will, of ops people really don't seem to give a shit about the work beyond what's been put in front of their face from 9-6, or whenever they're oncall.

-scott

[James Bailey]

On 28 June 2011 05:28, Scott Smith <sc...@ohlol.net> wrote:
> In my experience, the split is more about personnel than time. In larger
> groups, anyway. Like it's 80% of the people are doing bullshit firefighting
> and 20% are doing interesting work. They both work out to be the same, but
> yeah.
> Regardless, I think a lot of this is due to the fact that the "long tail,"
> if you will, of ops people really don't seem to give a shit about the work
> beyond what's been put in front of their face from 9-6, or whenever they're
> oncall.

There is coming a time when the long tail won't have a chance to
adapt. 100 years ago we had 1,000s of people working in cotton mills
now those jobs are gone replaced by automated milling machines. 50
years ago we had 1,000 of people building cars those jobs are now now
replaced with robotic assembly lines. Currently we have 1,000s of
people administering systems do we seriously expect those escape the
laws of progress.

Jim :)

[Patrick Debois]

@ Patrick Debois

Thank you for your feedback. Question : Why didn't you manage to
successfully promote DevOps in Europe ?

LOL! Define successful :)

I'd love to play the perfection game with you/others on this. The perfection game is a simple format to give positive feedback, which is not as simple as it sounds. This “protocol” was devised by Jim & Michele McCarthy in their “Software for your Head” book.

• Give a score out of 10. The number itself doesn’t have a lot of meaning, we use the number to see if we get closer to perfection (10/10) in each iteration.
• What we did to earn those points. It’s important to let the proposer know this, so that they keep or reinforce their strong points.
• What we should do to earn 10/10. This has to be clear, actionable advice. This is the difficult part. When they first start using the perfection game, people usually “sugarcoat” negative feedback to make it look like positive advice. E.g. “Your proposal would be perfect if it wasn’t like this“. What can I do with such feedback? Nothing much.

Looking forward to your feedback,

Patrick

[Guillaume FORTAINE]

Dear Mister Debois,

Thank you for your reply.

> LOL! Define successful :)

To make a living thanks to this new field that you've introduced :)

> Give a score out of 10. The number itself doesn’t have a lot of meaning, we
> use the number to see if we get closer to perfection (10/10) in each
> iteration.

I would give a score of 5. The middle because good progress has been
done (2 years ago, the devops word didn't exist , now there are nearly
1700 members on the Linkedin group :
http://www.linkedin.com/groups?home=&gid=2825397 ), but one can go
further.

> What we did to earn those points. It’s important to let the proposer know
> this, so that they keep or reinforce their strong points.

To bring in more executive profiles into the party. I would say that
10% of the attendance of a devopsday should include managers from
corporations. So each time this number is reached, a point is earned
:) Without key decisions makers, trying to inject a new Culture inside
an enterprise is nonsense.

> What we should do to earn 10/10. This has to be clear, actionable advice.
> This is the difficult part. When they first start using the perfection game,
> people usually “sugarcoat” negative feedback to make it look like positive
> advice. E.g. “Your proposal would be perfect if it wasn’t like this“. What
> can I do with such feedback? Nothing much.

Given that more than 60% of the enterprises are using ITIL, DevOps
should clearly be lean enough to not fear words like "Agile ITIL" :

http://blogs.rpath.com/wpmu/closing-the-gap/2010/04/08/is-agile-itil-an-oxymoron/

instead of saying :

http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/

"I’m not going to really get into what I think about most of ITIL"

So my conclusion is that by making ITIL Agile, DevOps could bring in
more top executives into the party thus significantly expanding to
finally be able to reach a 10/10 :)

Best Regards,

Guillaume FORTAINE
gfor...@gfortaine.biz
+33(0)631.092.519

[Ernest Mueller]

dev...@googlegroups.com wrote on 06/28/2011 07:32:07 AM:
> Given that more than 60% of the enterprises are using ITIL, DevOps
> should clearly be lean enough to not fear words like "Agile ITIL" :
>
> http://blogs.rpath.com/wpmu/closing-the-gap/2010/04/08/is-agile-
> itil-an-oxymoron/
>
> instead of saying :
>
> http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/
>
> "I’m not going to really get into what I think about most of ITIL"
>
> So my conclusion is that by making ITIL Agile, DevOps could bring in
> more top executives into the party thus significantly expanding to
> finally be able to reach a 10/10 :)

That's like making a pig more kissable by putting lipstick on it.

You could make the same argument that agile should have been marketed as "faster waterfall!" My, that would be nice and comforting to the existing power structure.

But what this does is effectively dilute and betray the message. DevOps isn't ITIL. If someone is allowed to think of it as ITIL, they will do stupid things when they implement it.

This is, however,a helpful data point to us, that we need to work on defining a good coherent DevOps message in the service of "escaping the echo chamber," because the longer we don't, the more confused outsiders, analysts, etc. will decide on their own bastardized interpretations of what it is and spread disinformation that will be harmful to the effort in the long run.

[Anthony Goddard]

Maybe I'm being naive, but I wonder how many folk share my same story..

Ops was fun, but hanging with devs all the time and tinkering with dev made me want to transition from ops to dev. I already used a bunch of "devops" practices but nothing like the well thought out and sensible practices that Devops encouraged. None of my ops colleagues had any interest in writing software to do anything (a few bash / perl scripts maybe) and I was a lonely sysadmin in a world of exciting ruby developers (cue the violin here).

Then devops came along and suddenly I realized I could have my cake and eat it too. I could keep doing ops and keep writing code, they weren't mutually exclusive. To be honest, the most valuable thing in the beginning was being able to search the #devops hashtag. That lead me to other like-minded folk (+ colleagues who did have an interest in writing software) and eventually to things like devopsdays where I met some awesome people, learned some new tricks and became better at what I do. I'm now a very happy ops guy who does a healthy amount of dev and I absolutely have a refreshing & renewed interest in the future of my field.

(cue the dramatic, skeptical, suspenseful music here)

When selling the idea of devops, I've found management at all levels to be supporting and encouraging. All I've seen from 'enterprise' folk - whether they're vendors or senior managers from other organizations has been them trying to misuse the term, sell me more crap I don't need and add "devops" to their ever-expanding list of buzzwords they don't understand.

(mute the soundtrack)

I guess everyone has a different definition and I can only speak from experience - from where I stand, the introduction of "Devops" into our field has been 100% successful. 10/10.

:)

Ant

[James Bailey]

On 28 June 2011 14:01, Ernest Mueller <ernest....@ni.com> wrote:
> dev...@googlegroups.com wrote on 06/28/2011 07:32:07 AM:
>> Given that more than 60% of the enterprises are using ITIL, DevOps
>> should clearly be lean enough to not fear words like "Agile ITIL" :
>>
>> http://blogs.rpath.com/wpmu/closing-the-gap/2010/04/08/is-agile-
>> itil-an-oxymoron/
>>
>> instead of saying :
>>
>> http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/
>>
>> "I’m not going to really get into what I think about most of ITIL"
>>
>> So my conclusion is that by making ITIL Agile, DevOps could bring in
>> more top executives into the party thus significantly expanding to
>> finally be able to reach a 10/10 :)
>
> That's like making a pig more kissable by putting lipstick on it.
>
> You could make the same argument that agile should have been marketed as
> "faster waterfall!" My, that would be nice and comforting to the existing
> power structure.
>
> But what this does is effectively dilute and betray the message. DevOps
> isn't ITIL. If someone is allowed to think of it as ITIL, they will do
> stupid things when they implement it.

I do believe that DevOps can be implemented within an ITIL framework
and I do think that the two can be supportive rather than
antagonistic. The problem is most managers charged with implementing
ITIL see ITIL as the alpha and omega of IT practice and talking DevOps
as a challenge to their authority and methodology. A similar problem
exists within project management using Prince2 when the the PRojects
IN Controlled Environments of the Prince2 framework becomes
inseparable from the Waterfall development process. There lies the
problem the confusion of framework and process.

>
> This is, however,a helpful data point to us, that we need to work on
> defining a good coherent DevOps message in the service of "escaping the echo
> chamber," because the longer we don't, the more confused outsiders,
> analysts, etc. will decide on their own bastardized interpretations of what
> it is and spread disinformation that will be harmful to the effort in the
> long run.

I think DevOps is a limiting term, there is a need to bring the whole
of the business into the process. Limiting this brave new world to the
interaction of Dev and Ops is ultimately going to fail in the
5,000-15,000 employee organisations I contract in, they are not
technology companies, they are not even tech literate, they are not
innovative, they are above all conservative and the power for change
resides in the equally conservative middle manager. For all their
talk of the importance of the employee and "investing in people" they
are deeply mistrustful at best of new ideas coming from the ops/dev
team. Usually they just nod politely to our ideas and ignore them.

ITIL was around for 10-15 years before it achieved mass adoption by
this kind of organisation. DevOps has been around what 2-3, though I
do realise many of us have been quietly and stealthy working this way
for many years.

For me the challenge of Devops is not the practice but finding a way
of conversing with management to make the DevOps process fit within
the ITIL and Project management frameworks.

Someone said that DevOps was about people, processes and tools in that
order. However it seems that most places I work the people that matter
are not ready for that conversation.

Peace Jim :)

[John Vincent]

Guillame,

I apologize if this comes off with the wrong tone but I think you're
sadly mistaken. No one (that I'm aware of outside of you?) is saying

"Let's people be happy (but give them a bunch of money)".

The money to be made in "selling" DevOps is not institutional (in that
it's a service sold by the likes of IBM or CA). Understand that DevOps
is a movement started by practitioners and not salesmen.

This community of practitioners is (in my personal opinion) one of
meritocracy. We write code, we share ideas and we promote. If one
feels they must assign some sort of tally to the impact of devops in a
business then it's sad to say they'll probably be very frustrated. I
said this in a previous blog post but I'll gladly say it again. The
people you see involved in this community are the current and future
executives that you are saying we should court. Think on that for a
moment. We don't need to be "sold" on DevOps because it's what we know
and do. After 17ish years in this industry, I'll not put up with
things like an inability to automate, lack of meaningful metrics or
backbitting and buck passing between my teams any longer. The business
value is perfectly clear to me. We make more money because we've
stopped pissing around with inconsequential minutia.

I'm very sorry if you expected something else.

--

[Sean OMeara]

+1 lusis

[Will Thames]

As with Sean, I agree with John 100% but would also add that there's
plenty of money to be made through devops - by allowing our businesses
to deliver improvements to their products more quickly and efficiently
then the business becomes more efficient with a tighter feedback loop.
As Jez mentioned earlier, continuous delivery and devops should not be
confused, both of these are separate and complementary ways to improve
efficiency - waiting until the end of the development cycle to say
'I'm not releasing this because it doesn't meet this requirement I
forgot to tell you about' is nowhere near as good as 'let's discuss
the user stories we'll deliver in this sprint'.

Plenty of brilliant organisations have been doing devops for years.
The devops focus is allowing the rest of us to find out what the best
organisations are doing and bring some of those best practices into
our own businesses!

Will

[Will Thames]

Interesting blogpost from appdynamics includes a great quote from John Willis:

"DevOps is about making your business more agile — it’s about going
from ‘Ah-Ha’ to ‘Cha-Ching’ as fast as possible."

http://www.appdynamics.com/blog/2011/06/28/making-the-business-more-agile/

Will

[Ernest Mueller]

+1, well said.

E

Sent from my iPhone

[Damon Edwards]

If anyone needs to make the argument to their boss as to why it's worth investing time and money in DevOps problems, I wrote this post about the topic:

http://dev2ops.org/blog/2010/11/7/devops-is-not-a-technology-problem-devops-is-a-business-prob.html

It's an older post but I still use the points and the graphics (the "ah-ha" to "ka-ching" graphics are in there) all the time. Making a business more agile (i.e. being able to react quicker to external forces or internal ideas) is a fundamental goal that no MBA is going to disagree with. It's up there with "make more with less" in terms of fool proof arguments. The only trick is that you now have to convince them that 1) you can do it 2) you can do enough of it, quick enough.

It's also got a graphic that shows DevOps in relation to ITIL (building on one of the more common ITIL v3 graphics)

[Nikolay Sturm]

* Ernest Mueller [2011-06-28]:

> This is, however,a helpful data point to us, that we need to work on
> defining a good coherent DevOps message in the service of "escaping
> the echo chamber," because the longer we don't, the more confused
> outsiders, analysts, etc. will decide on their own bastardized
> interpretations of what it is and spread disinformation that will be
> harmful to the effort in the long run.

A big problem I see whith the current state of devops is that it seems
to be like a shell which people fill with different contents, depending
on their background and needs (infrastructure as code, kanban,
dev-op-whatever cooperation, you name it).

If I compare this with agile software development, I feel agile being
much more like a plant. It has a core (pit?), which consists of a few
principles for software development, and from that grows a large body of
ideas and practices.

DevOpsDays Europe will be about inclusion, I wish for it to also be
about exclusion. Where are the limits of devops? What do we want to
concern ourself with and what is outside the bounds of devops? What is
the essential core?

If devops becomes anything for anyone, it will basically be nothing.

cheers,

Nikolay

--
"It's all part of my Can't-Do approach to life." Wally

[Damon Edwards]

When you think about DevOps as a problem statement (or a problem
umbrella) it's actually pretty easy to gain consensus.... Something
like "DevOps problems are the bottlenecks, inefficiencies, and
conflicts that plague the development to operations lifecycle and, in
turn, plague the business it supports".

When people don't like that and demand a "DevOps = X" statement... I
just fold in Adam Jacob's statement: "DevOps is a cultural and
professional movement". They say "movement to do what?". And I go
right back to "solve the bottlenecks, inefficiencies, and conflicts
that plague the...."

It's pretty easy to get a general consensus around the idea that:
1) DevOps is a cultural and professional movement and that is it
2) a general statement of the problem types that the movement wants to
solve (there sort of already is this consensus in the loosest way)

I'm not really sure why people get fixated on having DevOps be some
prescriptive methodology to be followed in only one way. I don't see
what the movement gains from that and I think it is going to lead to
endless infighting, naval gazing, and co-option.

Look at how well that worked out for "The Cloud". Countless hours have
been spent fighting about "what is a Cloud?". Who gives a shit. It's
nice we have a lightening rod to bring people together for a bit, now
let's get on with solving real problems. By the way, endless
infighting, naval gazing, and co-option is also pretty descriptive of
the state of the Agile movement.

If there was a general consensus that DevOps is a cultural and
professional movement and then some sort of general problem statement
and we focus our efforts on holding that, then who cares what vendors
or other people do? Anyone can claim they have the answers... and
maybe they do or maybe they don't. But either way they are just part
of the conversation and aren't hurting anything. Everyone learns,
Everyone wins.

If our goal is as altruistic as we say it is, being exclusive or
demanding that there is only One True DevOps Way™ is totally
counterproductive.

[Jez Humble]

Showing how ITIL and devops can work together is one of my goals for the next year or two. I have an article in cutter in a couple of months discussing this. Also check out Gene Kim's work. Here is one of my blog entries on this topic: http://continuousdelivery.com/2010/11/continuous-delivery-and-itil-change-management/

Sent from my iPhone

[Matthew Jones]

In my experience, devops to one company is completely different to devops in another. The deciding factor for success... culture.

I've seen it work brilliantly in small and agile software development firms, but not so much in universities and other larger institutions.

Devops works in organisations because the people on the ground want it to work.

@matthew_jones

[Jez Humble]

Most of my recent experience is with large orgs. In my experience, it's just not as simple as 

Sent from my iPhone

[Jez Humble]

Balls

As I was saying, the people on the ground wanting it to work is necessary but not sufficient. Furthermore, it's possible to win people round (i've seen it happen). Culture can be changed in large orgs. Part of making that happen involves convincing senior people not to block it because of perfectly reasonable concerns around governance and compliance. Since that's the pain I'm experiencing now, it seems reasonable to attack it so that the bottoms up approach can actually work.

Viva bottoms up!

Sent from my iPhone

[Guillaume FORTAINE]

Hello,

@ Ernest Mueller

> This is, however,a helpful data point to us, that we need to work on
> defining a good coherent DevOps message in the service of "escaping the echo
> chamber," because the longer we don't, the more confused outsiders,
> analysts, etc. will decide on their own bastardized interpretations of what
> it is and spread disinformation that will be harmful to the effort in the
> long run.

+1, well said.

@ John Vincent

> I apologize if this comes off with the wrong tone but I think you're
> sadly mistaken. No one (that I'm aware of outside of you?) is saying

> "Let's people be happy (but give them a bunch of money)".

http://www.agileweboperations.com/2011_escaping_devops_echo_chamber

"All too often, DevOps conversations focus on our personal complaints
rather than actual business problems. Just because we find something a
headache or unpleasant doesn’t necessarily mean that our perceived
problem matters to the business. To put it bluntly, we are paid to
have those headaches. Happiness would be nice, but it’s not much of a
business justification. DevOps needs a different argument if we want
to be heard."

@ James Bailey

> For me the challenge of Devops is not the practice but finding a way
> of conversing with management to make the DevOps process fit within
> the ITIL and Project management frameworks.

Too much widsom in just a single email.

@ Jez Humble

> Showing how ITIL and devops can work together is one of my goals for the next year or two. I have an article in cutter in a couple of months discussing this. Also check out Gene Kim's work. Here is one of my blog entries on this topic: http://continuousdelivery.com/2010/11/continuous-delivery-and-itil-change-management/
>

Thank you for this great post. Especially, I would greatly appreciate
to invite you to a further reading of the one entitled "Devops: It is
Not About ITIL, It is About Proficiency" :

http://theagileexecutive.com/2010/06/29/devops-it-is-not-about-itil-it-is-about-proficiency/

"TIL should really be thought of as a control model and a common
language for process integration.

If your ITIL process are bureaucratic and heavyweight, they’ll
restrict or choke continiuous deployments. If they are lightweight and
modeled for agility, they’ll faciliate the desired control even when
revved to high velocity."

By the way, I have just bought your book "Continuous Delivery".

Otherwise, you are more than welcome to Share on the document that I
have created on DevOpSpace Documents and entitled "Managing a
Planet-Scale Information System, the DevOps way" :

https://docs.google.com/document/d/19iyHH0YSpmvP4lANA9fI8j2Q72CK46VWb8se8TwUqq0/edit?hl=fr&authkey=COHJy9gD

Best Regards,

Guillaume FORTAINE
gfor...@gfortaine.biz
+33(0)631.092.519

[Philip J. Hollenback]

On Wed, 29 Jun 2011 08:27 +1000, "Matthew Jones" <ma...@geekle.id.au>
wrote:

> In my experience, devops to one company is completely different to
> devops in another. The deciding factor for success... culture. I've
> seen it work brilliantly in small and agile software development
> firms, but not so much in universities and other larger institutions.
>
> Devops works in organisations because the people on the ground want it
> to work.

The issue of implementing devops in a major organization is a tough one.
I wrote a
blog post about this very subject a few month back:

http://www.hollenback.net/index.php/DevOpsBlocker

I work in a very large internet company, and the inertia is simply
astounding. It's extremely hard to implement devops-compatible changes.

I mainly try to work on the communication aspect - I figure if I can
foster more discussion between developers and operations folks, that
will hopefully lead to better understanding (and more compassion) on
both sides. I've also had quite a bit of success getting operations
people to use
development tools (for example, formal code review).

P.
--
Philip J. Hollenback
phi...@pobox.com
www.hollenback.net

[Kief Morris]

+1

I think of DevOps is described as "Exploring ways to remove the
bottlenecks, inefficiencies, ...", etc, then it makes it clear it's
not intended to offer a prescriptive Solution to the problem, or even
a "framework".

It's also important to keep in mind that there isn't a one size fits
all when it comes to DevOps. When dealing with many large
organizations, fitting DevOps into an ITIL perspective will help fit
it into the organization. In other situations, although ITIL contains
some concepts and ways of framing things that have value, it also
brings a lot of baggage.

A strength of capital A Agile is that it doesn't provide a
prescriptive solution, but rather a set of principles that can be
applied in many ways. A problem with Agile is that a number of
prescriptive frameworks have emerged which, although they clearly
satisfy a need, do encourage the idea that there is One True Agile
Way.

DevOps doesn't have either of these (yet?), or at least, a consensus
hasn't yet formed on how to articulate the principles.

Kief

[Nikolay Sturm]

* Damon Edwards [2011-06-28]:

> When you think about DevOps as a problem statement (or a problem
> umbrella) it's actually pretty easy to gain consensus.... Something
> like "DevOps problems are the bottlenecks, inefficiencies, and
> conflicts that plague the development to operations lifecycle and, in
> turn, plague the business it supports".

Here you provide the exclusion I was asking for. Such a statement
excludes, e.g. most classical sysadmin work from the devops sphere as
you limit devops to the area of software development and its operation.
I don't know whether or not this was intended or is just a
misunderstanding on my side.

The important aspect for me is that I wouldn't agree with this
particular exclusion.

To me devops has two main pillars:
- cooperation of all parties involved with a
- focus on creating (it-related) business capabilities

Thinking of devops in these terms, I can see its specific value as
compared to agile software development, the lean movement etc. and it
becomes obvious, that it is an inclusive movement, that wants to change
the way we all work together.

Ironically, this sets a much broader stage after having asked for the
limits of devops in my previous mail.

Not sure whether this makes any sense to anyone or was obvious to
everyone, but thanks Damon for making me think about this. I am looking
forward to the discussions in Gothenburg.

cheers,

Nikolay

[Ernest Mueller]

The description "the development to operations lifecycle" only excludes classical sysadmin to the degree to which classical sysadmins have forgotten that their entire existence is owed to the need to support the applications the business depends on.

I know a lot of sysadmins who consider "the machines" to be the sum total of their job, and don't really care about the apps running on them or what they are used for. This is one of the unforuntate antipatterns that hopefully DevOps will solve for these folks before it leads to their obsolescence.

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

Nikolay Sturm ---06/29/2011 02:57:50 AM---* Damon Edwards [2011-06-28]: > When you think about DevOps as a problem statement (or a problem

[Nikolay Sturm]

* Ernest Mueller [2011-06-29]:

> The description "the development to operations lifecycle" only
> excludes classical sysadmin to the degree to which classical sysadmins
> have forgotten that their entire existence is owed to the need to
> support the applications the business depends on.

Sigh, I really have a hard time getting my point across. :/

Do you consider installing and operating 3rd party software like a
mailserver part of "the development to operations lifecycle"? What about
providing print services or file system space to users?

I was of the impression that all these tasks would be exlcuded from
devops with Damon's problem statement. Please correct me if I am wrong.

If I understood Damon correctly and devops indeed just wants to concern
itself with the in-house software value chain, then that's fine. But I
don't believe this to fix the core problem.

To give a little background, until recently I worked as a sysadmin in a
company selling appliances. We basically just operated 3rd party
software. In this scenario we faced the same lack of cooperation, that
operations face in non-devops web shops, but differently. In our case,
projects where thrown over the wall. Other departments would come to us
and say 'we have a new project going, here are your tasks, please
implement them'.

In my opinion, there is exactly the same chasm between any department
and IT as there is between devs and ops in a web shop.

My question is, whether people want to include this general chasm in the
devops problem statement. Or do you guys see a fundamental difference
between these cases?

cheers,

Nikolay

[Ernest Mueller]

No fundamental difference, the customers you are delivering services to are just internal. Many of the DevOps tenets are equally valid. There are some (CI!) that get wonky of course since in a "mail server" case the dev has usually all been done by a software provider, but heck even "dev code" nowadays is often 50%+ third party libraries and widgets. But I find that if you go back to the concept behind the practice you can implement it easily. ("Big dev CI engine" is really "rapid delivery of small changes" which becomes "don't wait a year to patch your systems, continuously patch with testing in place to find issues".)

The pressure to DevOps has come from SaaS shops because the innate failure in the way we've been doing things has more direct financial incentive to fix. The improved concepts can filter back to everyone.

In fact, I started on the DevOps path here at NI when I was in our IT department - that was somewhat interrupted because when we went into SaaS products they pulled me out of IT into the product side, and I in turn pulled my DevOps A team over with me. But most of my seminal DevOps "ah-ha" moments were from my time in IT.

Example: We had an Atlassian Confluence upgrade project including underlying tech stack changes - we needed a long time to get the new systems, get everything all loaded up and tuned, and then the devs could start porting macros and plugins. We wanted to waterfall while they wanted to iterate. I realized that we could iterate on the systems side as part of their sprints; the old sysadmin fear that "I'll only get one shot at this before they are on the systems and I can't change anything" was mitigated by their refactoring discipline, so "Hey in sprint 2 we're switching app servers too" wasn't traumatic.

Example: I was PMing a company-wide core network upgrade. We had to take down all our systems in a complex order (app, db, SAN, network), do the work, and spend literally 8 hours bringing them all back up. I asked each layer (network, storage, UNIX, etc.) what they were going to do to test, and they were surprised. "Well, once it's all up, the users will test the apps, and we'll see if anything went wrong! Plus we have, you know, Sitescope monitors..." I realized "what does a unit test mean for systems" is an unsolved problem and we had a lot of success with the resulting process and baked-in testing.

As with agile, you have principles, methods, and practices. Some of the lower level practices will change from type of shop to type of shop, but the higher level concepts hold in any service delivery context.

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

Nikolay Sturm ---06/29/2011 10:29:34 AM---* Ernest Mueller [2011-06-29]: > The description "the development to operations lifecycle" only

From: Nikolay Sturm <goo...@erisiandiscord.de>
To: dev...@googlegroups.com
Date: 06/29/2011 10:29 AM
Subject: Re: Myopic View of DevOps Misses the Mark
Sent by: dev...@googlegroups.com

---

[Kit Plummer]

Nikolay,

I definitely think your environment is fair game. In fact I believe DevOps should be applicable to anything where delivery of software is involved - regardless of what the definition of Ops is. For example, I come from an embedded development environment where my code was delivered to hardware peeps (typically electrical engineers). Indeed there is the same chasm there between SW and HW teams. The solution...one team with SW and HW engineers.

I also believe the rise of mobile applications is creating another channel to cross - with the app stores being a new version of Ops. To think that "Ops" can only mean the people who run webservers is rather shallow.

Kit

[Paul Graydon]

With the increasing reliance on cloud services, Paas, iaas and saas etc, it's only a myopic sysadmin who isn't looking to the future and wondering what's next for them. I'm not unduly concerned for myself, but I see sysadmin / ops as problem solvers. As services get abstracted away from hardware the problem changes. It continually surprises me to find sysadmins who are that myopic, especially given they manage to keep up with the pace of changing hardware, software and good practice.
For my current job the cloud is impractical, except as a private one someday but even then I can see the increasing trend in my work as moving away from hardware as my primary 'problem' to services and our own software. Increasing automation from config management, and the increasingly homogenous environment results in less time firefighting.
Surely there is only so much time one can spend playing solitaire before wondering if there is something else to be improving, or another type of problem to solve?

On Jun 29, 2011 4:20 AM, "Ernest Mueller" <ernest....@ni.com> wrote:

[Scott McCarty]

Just one side note, Ernst brought up a good point (probably not on purpose), he mentioned CI. CI and CD in a fully or almost fully commercial off the shelf (COTS) environment suffers from the same problems. If you are going to continuously patch servers, you need a person in between, and a suite of tests. The tests might rely on DNS, MAIL, Kerberos, etc, but it is very, similar.

I think COTS environments with Finance, HR, BI being your customer are fair game too, but there is obviously a big challenge there. I once heard our CIO call the CFO and head of HR his business partners, and trying to convince business partners of a need for better cooperation almost obviously has to be one of our goals as advocates of a collaborative mentality like DevOps.

Best Regards
Scott M

--
Scott McCarty
scott....@gmail.com
http://crunchtools.com
@fatherlinux

[Marius Sturm]

Hey Damon,
I agree with you, that including and excluding is not the right way.
Not because of endless infighting, I don't care if some guys on some
mailinglists fight for the realreal DevOps Way. But I think it can
be very hard to re-include some topics that were excluded in the past.
So excluding topics in a public movement needs to be done in a very
wise way.

But my question is, how would you benchmark DevOps?

In my opinion, it is very hard to measure DevOps benefits if it is not
nailed down to a concrete problem-space and concrete approach to solve
these problems.
You can only measure how well your company's culture is saturated with
the DevOps idea. So any resulting process changes are maybe a win for
the company, maybe not, but for sure they are not directly related to
the DevOps migration.

This is a big advantage of the agile or lean methodology, you can run a
software project and compare it with the conventional method, there
will be a delta for some metrics that can be evaluated. I think this is
the reason why agile is taken serious by the industry.

Best regards,
Marius

> demanding that there is only One True DevOps Way� is totally

[Zac Stevens]

On Thu, Jun 30, 2011 at 12:17 PM, Marius Sturm
<marius...@googlemail.com> wrote:
> But my question is, how would you benchmark DevOps?

I'd respond with a different question: _Why_ would you benchmark DevOps?

This discussion (and I don't just mean this mailing list thread,
although it provides a good example!) clearly suffers from the term
"DevOps" implying different things to different people.

To me, the DevOps "movement" is cultural, and advocates better
cooperation between the teams involved in turning an idea into
something valuable. From that perspective, discussion about how to
measure the value of DevOps is mind-boggling.

Perhaps I'm inexperienced (or lucky), but I've never met an executive
who admitted believing that her company would be stronger if her staff
refused to cooperate with each other. The value of teamwork and
cooperation is self-evident - or, at least, not openly questioned.

The flip side is that many of the technology and process changes
associated with DevOps discussions _can_ (and should) be measured and
justified in business terms. We can make business cases to justify
investments in our monitoring systems, or automated testing, or
configuration management. But none of those things is "DevOps", and
different organisations will adopt them to different degrees, in
different sequences.

As a result of people working together, we come up with ideas to
improve the way we work. The most important improvements for my
company may not be the same as yours, or anybody elses. What matters
is that we continue to learn, and improve.

In organisations with strong project governance, perhaps the "DevOps
Benchmark" is the number of project proposals co-sponsored by Ops and
Dev(/Sec/Net/QA/etc) teams. That said, I feel that a company that
_needs_ to measure its "DevOps success" is likely to have a corporate
culture impeding cooperation, beyond the common "dev vs ops" dynamic.

Zac

[ghbrow...@gmail.com]

Actually that was somewhat discussed at mountain view. It was argued (and I think I agree) is the best way to gauge devops adoption is to evaluate company morale.

I have seriously considered the idea of having a practice at every standup to ask the question (with a show of hands) "was yesterday a good/bad/indifferent day for you?" and seriously record that as a metric.

> demanding that there is only One True DevOps Way™ is totally

[Radu Brumariu]

On Thu, Jun 30, 2011 at 4:58 AM, Zac Stevens <z...@cryptocracy.com> wrote:
> On Thu, Jun 30, 2011 at 12:17 PM, Marius Sturm
> <marius...@googlemail.com> wrote:
>> But my question is, how would you benchmark DevOps?
>
> I'd respond with a different question: _Why_ would you benchmark DevOps?

Because you want to see if the processes/practices/policies you're
implementing have the effect you were counting on.

To me, DevOps has already a negative buzz to it -- at every conference
I have been at and it has been talked about, the whole conversation
could be summarized into a few statements.

"Dev and Ops separate are bad. Don't throw things over the wall. Don't
have a wall. blablabla"

If the final goal is to have smooth operations of a product, in my
opinion, the things to do is to hire competent people, keep the size
of the ops team small, don't give in to crap like ITIL, make ops
report into the dev structure, make manager responsible of production
issues.

Ultimately, having smooth operations is about both ops and dev being
taking ownership of the product and working together on a common goal.
It's hard to implement that in processes and policies it's more work
ethics. And it is entirely boring to make a 300 slides presentation on
something that could be summarized in a few sentences.

Radu

[Zac Stevens]

Hi Radu,

On Thu, Jun 30, 2011 at 3:38 PM, Radu Brumariu <bru...@gmail.com> wrote:
> On Thu, Jun 30, 2011 at 4:58 AM, Zac Stevens <z...@cryptocracy.com> wrote:
>> I'd respond with a different question: _Why_ would you benchmark DevOps?
>
> Because you want to see if the processes/practices/policies you're
> implementing have the effect you were counting on.

Ahh, but that's just my point. What processes, practices, or policies
are prescribed by DevOps? I absolutely agree that concrete changes
that are predicted to have a measurable impact should be measured.

> To me, DevOps has already a negative buzz to it -- at every conference
> I have been at and it has been talked about, the whole conversation
> could be summarized into a few statements.
>
> "Dev and Ops separate are bad. Don't throw things over the wall. Don't
> have a wall. blablabla"

I'm inclined to agree.

Perhaps it's just my interpretation of "DevOps", but I don't think the
core messages are in any way novel or revolutionary. I have some
empathy for people saying "But we've been doing this for years! What
a load of rubbish!" while thinking that attitude misses the point.
Loads of people _haven't_ been doing this for years.

If it's so obvious that it's bad for Dev and Ops to be at loggerheads,
why does it appear to be so common?

> If the final goal is to have smooth operations of a product, in my
> opinion, the things to do is to hire competent people, keep the size
> of the ops team small, don't give in to crap like ITIL, make ops
> report into the dev structure, make manager responsible of production
> issues.

I personally share all of your views (and would live by them for my
own company), but I think you'd agree they're not the only way.

In any event, there's a lot of places that have already given into
ITIL, or that have grown large ops teams. Just because that's their
situation doesn't mean they don't want to improve. I'm looking
forward to reading more from Jez Humble on the subject of DevOps and
ITIL, and I'd love to hear people with larger teams talking about what
they're doing.

Do scale and formal process necessarily imply inefficiency and
adversarial relationships? It may be the default outcome, but I don't
believe it's a certainty.

> Ultimately, having smooth operations is about both ops and dev being
> taking ownership of the product and working together on a common goal.
> It's hard to implement that in processes and policies it's more work
> ethics. And it is entirely boring to make a 300 slides presentation on
> something that could be summarized in a few sentences.

I agree that this is largely about attitudes and worth ethics, and
that those won't spring from policies and processes. People are hard,
and you can't legislate common sense.

What if we consider "DevOps" as a metaphorical hashtag? A label that
is applied to ideas and suggestions which support all the good stuff
we wish was "business as usual" everywhere. An umbrella,under which
successful organisations can share the lessons they've learned about
fostering ownership and cooperation, and where organisations with room
for improvement can look for suggestions.

I think it would be absolutely brilliant if the term "DevOps" was dead
in five years, because the attitudes it implies had become the norm.
I rather doubt it will happen so soon.

Zac

[Kit Plummer]

On Thursday, June 30, 2011 at 8:53 AM, Zac Stevens wrote:

> Hi Radu,
>
> On Thu, Jun 30, 2011 at 3:38 PM, Radu Brumariu <bru...@gmail.com (mailto:bru...@gmail.com)> wrote:

> > On Thu, Jun 30, 2011 at 4:58 AM, Zac Stevens <z...@cryptocracy.com (mailto:z...@cryptocracy.com)> wrote:
> > > I'd respond with a different question: _Why_ would you benchmark DevOps?
> >
> > Because you want to see if the processes/practices/policies you're
> > implementing have the effect you were counting on.
>
> Ahh, but that's just my point. What processes, practices, or policies
> are prescribed by DevOps? I absolutely agree that concrete changes
> that are predicted to have a measurable impact should be measured.
>
> > To me, DevOps has already a negative buzz to it -- at every conference
> > I have been at and it has been talked about, the whole conversation
> > could be summarized into a few statements.
> >
> > "Dev and Ops separate are bad. Don't throw things over the wall. Don't
> > have a wall. blablabla"
>
> I'm inclined to agree.
>
> Perhaps it's just my interpretation of "DevOps", but I don't think the
> core messages are in any way novel or revolutionary. I have some
> empathy for people saying "But we've been doing this for years! What
> a load of rubbish!" while thinking that attitude misses the point.
> Loads of people _haven't_ been doing this for years.
>
> If it's so obvious that it's bad for Dev and Ops to be at loggerheads,
> why does it appear to be so common?

And here is why I believe it important to get DevOps (or whatever name is applied to the notion) to be more inclusive of all stakeholders (including QA) - as well as including tools and pragmatic practices in the discussion. Sure, if it is just about "what should be", then it is a really easy solution.

Really, the problem is that people haven't been doing all of these things for years...and if they say they have they are full of shit. Don't get me wrong, there are good Devs and good Ops who have been working together to deliver software cheaper, faster and better - but I've yet to see it as part of lifeblood of any organizational whole. I do agree that practices like "infrastructure-as-code" have grow in a lot of places, but that's just one functional piece of the whole continuum that is dev-to-ops.

>
> > If the final goal is to have smooth operations of a product, in my
> > opinion, the things to do is to hire competent people, keep the size
> > of the ops team small, don't give in to crap like ITIL, make ops
> > report into the dev structure, make manager responsible of production
> > issues.
>
> I personally share all of your views (and would live by them for my
> own company), but I think you'd agree they're not the only way.
>
> In any event, there's a lot of places that have already given into
> ITIL, or that have grown large ops teams. Just because that's their
> situation doesn't mean they don't want to improve. I'm looking
> forward to reading more from Jez Humble on the subject of DevOps and
> ITIL, and I'd love to hear people with larger teams talking about what
> they're doing.
>
> Do scale and formal process necessarily imply inefficiency and
> adversarial relationships? It may be the default outcome, but I don't
> believe it's a certainty.

That's a good question. Obviously it depends on the complexity and operational requirements. NASA's software process is/was as slow as could possibly be, based on intended inefficiencies and adversarial relationships injected into the flow.

In the same way that Agile is really difficult to measure, DevOps will suffer. Until it is academically possible to compare two like projects/streams - it will be difficult to overcome this particular conversation. And, there are other intangible attributes, like employee happiness, that come into play as a result of improved operational flow (simply because there is less stress involved in delivering what the customer wants).

>
> > Ultimately, having smooth operations is about both ops and dev being
> > taking ownership of the product and working together on a common goal.
> > It's hard to implement that in processes and policies it's more work
> > ethics. And it is entirely boring to make a 300 slides presentation on
> > something that could be summarized in a few sentences.
>
> I agree that this is largely about attitudes and worth ethics, and
> that those won't spring from policies and processes. People are hard,
> and you can't legislate common sense.

I disagree. A combination of stick and carrot is required. Change doesn't just happen.

>
> What if we consider "DevOps" as a metaphorical hashtag? A label that
> is applied to ideas and suggestions which support all the good stuff
> we wish was "business as usual" everywhere. An umbrella,under which
> successful organisations can share the lessons they've learned about
> fostering ownership and cooperation, and where organisations with room
> for improvement can look for suggestions.
>

I'm a huge fan of "Practices of an Agile Developer" because it strips away most of the Agile non-sense and captures what is truly useful to me as a software engineer. I think it would be incredibly useful to capture the ~40 best DevOps practices (based on a voting scheme) and compare them to the book.

http://pragprog.com/titles/pad/practices-of-an-agile-developer

>
> I think it would be absolutely brilliant if the term "DevOps" was dead
> in five years, because the attitudes it implies had become the norm.
> I rather doubt it will happen so soon.

Yep. I'm not too fond of the need to apply silly adjectives to verbs, ala "lean *",

>
>
> Zac

[Radu Brumariu]

> Ahh, but that's just my point.  What processes, practices, or policies
> are prescribed by DevOps?  I absolutely agree that concrete changes
> that are predicted to have a measurable impact should be measured.

I really cannot answer that, as I said, to me DevOps is more about
work ethics than anything else.

> If it's so obvious that it's bad for Dev and Ops to be at loggerheads,
> why does it appear to be so common?

From personal experience, I believe that to be the effect of politics
that trickle down from upper management to the grunts. I have seen
people coming with excellent work attitude only to be swallowed by
their business unit culture. It's not their fault, they're just trying
to fit in.

> I personally share all of your views (and would live by them for my
> own company), but I think you'd agree they're not the only way.
>
> In any event, there's a lot of places that have already given into
> ITIL, or that have grown large ops teams.  Just because that's their
> situation doesn't mean they don't want to improve.  I'm looking
> forward to reading more from Jez Humble on the subject of DevOps and
> ITIL, and I'd love to hear people with larger teams talking about what
> they're doing.
>
> Do scale and formal process necessarily imply inefficiency and
> adversarial relationships?  It may be the default outcome, but I don't
> believe it's a certainty.

So, just to be clear, I haven't read all the ITIL documents, but I
believe that to be more geared towards IT rather than production
operations -- which is my main focus ( I should have stated this
before, sorry )
I think, the more processes you have ( e.g. the more situations you
try to cover ) the harder it gets for the people doing the real work
to do their job effectively.
I am not saying that the efficiency is necessarily in reverse
proportion w/ the weight of the processes, but that a small set of
very clear cut rules, that leave room for individual thinking and
decisions, will last longer.

> I think it would be absolutely brilliant if the term "DevOps" was dead
> in five years, because the attitudes it implies had become the norm.
> I rather doubt it will happen so soon.

I was rather complaining that every talk that made reference to
DevOps, sounded in the same way, covered the same questions, etc --
basically, brought nothing new to the table. And talks that are tought
to be technical, and relevant become more of a launchpad for various
people trying to make a name for themselves, by repeating what they've
heard at a previous conference.

Regarding large companies and such -- I believe that the way market
moves today has a fluidity that was not even fathomable 30 years ago.
There must be some Darwinian law for operations and I believe that
even those big companies would eventually have to move towards
something that makes sense in the market. Just slower.

That's why I think smaller operational teams are the key. Even in a
big company, if your operations is small 4-5 people, you're bound to
evolve faster.

Radu

[Mark J. Reed]

Having Ops report into the Dev management structure may make sense for
a company whose entire business is built around software that they
develop. It makes less for a company which hosts many different
websites using a wide variety of software stacks with applications
developed by many different groups inside and outside the company.

If there is any myopia in DevOps, I think it's around this - the idea
that it only applies to startups or SaaS/"Web 2.0" companies or
whatever you want to call this niche. Are those of us in large
companies just doomed to keep our Walls of Confusion, with ITIL as the
mortar?

I really think that the SaaS shops are the ones that need the movement
the least. If you're a 5-person startup with all your infrastructure
in someone else's cloud, DevOps is so clearly the way to go that it
hardly merits a name. But if you're a 500-person organization
supporting 100 different websites with their own development and
editorial teams on your own hardware infrastructure, you have a
different set of problems. Nonetheless, DevOps has something to say
and a way to help with those different problems. The principles and
goals are the important things; specific practices will follow from
those principles and goals, and will vary between different
organizations and solution spaces. If we attempt to codify the
practices, we'll end up with a much smaller scope of applicability
than DevOps deserves.

I do agree that DevOps as a Thing should have a limited lifespan, but
I think 5 years is a bit aggressive; after all, people are still
discovering Agile even at this late date. A rallying point is a good
thing, and I think DevOps will be around for a while, even if most of
us will have the reaction "What, that old thing? You aren't doing
that already?"

[Paul Graydon]

I strongly disagree there. I'm the first full time 'ops' guy at my
workplace. The company has been going close on 10 years now, and last
year processed around $1bn worth of transactions through its systems.
Prior to me all the ops guys were 50/50 dev (new apps & maintenance) and
ops. Previous ops guys could routinely be seen delving in to code to
investigate and fix problems, tune performance etc, alongside usual code
as an infrastructure style tasks. With a bunch of the work outsourced
to far remote programmers it's always been necessary for operations to
be able to do that. Coming in as a non-dev has taken a little bit of
work for all parties as practices have had to shift to suit someone who
can't write Java (though I rather swiftly got used to parsing java stack
traces), but has a stronger grasp of the sysadmin/ops end of the work.
What we've done in our company isn't a far cry from what's been done in
other sister companies either.
On a slightly bizarre twist, our parent company has just started a
'DevOps' team, who's sole responsibilities are 'deploying applications
and acting as first line of support' (that's more or less verbatim, and
there is no indication anywhere that they might have other duties) Not
a hint of any development work or even monitoring. Still trying to
figure out how that's 'devops'. Sounds too much like someone picked up
on the buzz word and implemented without bothering to understand it.

Paul

[Kit Plummer]

You're right to disagree in this sense. It wasn't my point to say that Ops aren't doing dev work or that Devs aren't doing ops work somewhere in the world. But, if you're disagreeing that that the typical organizational structure is one where there isn't an intentional functional divide (and administrative) between product engineering and IT I don't know how to counter. However, having ops peeps respond to problems - well that's not DevOps either. The intent is to get Ops working together with Devs earlier in the cycle, to make it possible to optimize the delivery and maintenance process.

[kurtmilne]

Great thread.

1 - I do think we can benchmark devops. Ask folks who have "been
there" "What did you do to create performance breakthrough"? "How did
you measure results"? Then build some hypothesis about what works.
(culture, process, work procedures, tools, org structure, context)
Then collect data on practices and performance from a bunch of
companies, and use statistics to test the hypothesis. Does the
presence of those practices that create breakthrough correlate with
better results accross many companies? We can do some cross tab
analysis - work only for certain size? Only certain org structure?
Only web apps? If we find more than noise - then we can start to
define what "high impact" dev ops practices are, and when/where they
should be applied.

2 - Is it time for a DevOps mainifesto? Machines build machines. No
GUI based configuration. Test driven design. Rebuild from recipe
instead of repair. Everyone owns uptime/security/quality etc. etc.
Lets get 10 big brains in a room at Lake Tahoe and create one.

3 - Traditional enterprise IT is begging for this. I just spoke with
a guy who runs IT at a "traditional" shop. He is looking for anything
that will help his guys see what a modern datacenter/ops team actually
does. Remeber, lots of shops struggle with mainframe guys who won't
go to a noSQL class. Oh - we are supposed to backup all three tiers
and test the backup? My sense is that there are more shops "out of
the loop" than "in the loop" on this.

Kurt

[John Allspaw]

I've been on vacation and barely following this thread, so pardon if I'm restating what smart folks have already said.

Asking for a definitive or canonical methodology from something that is nothing more than a cultural and professional change in perspective isn't a worthwhile approach.  DevOps isn't a methodology. 

It's a short word that means "collaboration and communication between dev and ops that support the business". You know you're on the right track by running this against anything written that includes the word "devops".

<code>
sed 's/devops/collaboration and communication between dev and ops that support the business/g'
</code>

I understand that calling a team or staff "devops" is helpful to communicate the sort of culture there is, and for groups that do that to further the above goal - rock on. But in my mind, it really doesn't matter. Like John Vincent mentioned, one size does not fit all.

That unease you have in your stomach that wants DevOps to be a methodology, a prescription, a recipe, a procedure, or a manifesto....that exists because getting two groups with different-but-related responsibilities and domain expertise to respect each other's challenges, goals, and culture is hard to do. That means you'll have to encourage those humans to talk to each other and work together, despite their sometimes differing points of view.

And that can't happen with a list of checkboxes, unless that list comes from a worksheet written by a therapist. And if you're lucky enough to have a therapist on staff, then good for you - you're going to be fine.

There's enough examples in conference videos, blogs, and articles from places like Flickr, Etsy, Wealthfront, Facebook, Dealnews, Shopzilla, Google, Amazon, Heroku, Opscode, and OmniTI (to mention a few) about how these successful organizations work as well as they do, engineering-wise (yes that includes Ops), and not all of them do the same things, or have the same conventions or responsibilities. This is ok, but is likely to irk people looking for patterns to match so that they can replicate in their own back yard.

But all of those organizations do have one thing in common: they encourage and pursue collaboration and communication between Development and Operations. Which is the non-precriptive, non-procedural, non-product, non-technology thing that is commonly known as "DevOps".

Carry on,
allspaw

--
John Allspaw

[Sean OMeara]

Assuming Tumblr can stay fixed long enough for you to load this. Here's part one of a 2 part anecdote:

http://goo.gl/7wx2A

[kurtmilne]

Kit - thanks for recommending this book. Bought it. Looks
interesting. I like the format -- many practices x ~few hundred words
per practice.

[gareth rushgrove]

On 30 June 2011 17:12, Kit Plummer <kitpl...@gmail.com> wrote:
>
> I'm a huge fan of "Practices of an Agile Developer" because it strips away most of the Agile non-sense and captures what is truly useful to me as a software engineer. I think it would be incredibly useful to capture the ~40 best DevOps practices (based on a voting scheme) and compare them to the book.
>
> http://pragprog.com/titles/pad/practices-of-an-agile-developer
>>

I'm also a fan of that particular book, and trying to create something
similar would be far more useful to my mind than a punchy 'manifesto'.
I've also seen that approach help cut through some of the meta
discussions around agile (or Agile). Practical without being preachy
or trying to be new.

Here's some samples for anyone not familiar with the format. 750 words
or so on a topic, following a simple set of headings and not being so
specific that the advice dates quickly.

http://media.pragprog.com/titles/pad/CodeAndDebug.pdf

Who fancies a bit of writing then? I'd certainly be interested in
contributing some time writing or editing or looking after a github
repo and a pdf build script.

Chapters on Monitoring, Deployment, Post-mortems, etc. Topics like
Infrastructure as code, the joy of system packages, sharing on call
duties, etc.

If this thread demonstrates anything it's that as a relatively small
group of people we can happily spend time writing lots on a subject
we're obviously passionate about.

G

--
Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com

[Elijah Wright]

On Thu, Jun 30, 2011 at 6:58 AM, Zac Stevens <z...@cryptocracy.com> wrote:

> Perhaps I'm inexperienced (or lucky), but I've never met an executive
> who admitted believing that her company would be stronger if her staff
> refused to cooperate with each other.  The value of teamwork and
> cooperation is self-evident - or, at least, not openly questioned.

I have. This is one of the classic ways that a leader can be highly screwed up.

There tends to be a trail of hardworking bodies left behind, while
they 'escape', scot-free of any lasting consequence.

Put another way, this is one of the possible implementations of the
Peter Principle.

I'm not sure that the person in question would *say* that conflict is
exactly what he wanted - but his
observed goal over the course of years was to bring people into
conflict to see who would
burn out and who would produce Awesomeness as a result. It can work -
but the human cost
is rather horrific to watch. Borderline sociopathy? "I am not a
psychiatrist, YMMV"...

best,

--elijah