system administration vs. automation

[Philip J. Hollenback]

Chris Siebenmann on how continuing automation will make traditional
system administration increasingly untenable:

http://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutomationDownsideII

P.
--
Philip J. Hollenback
www.hollenback.net
@philiph

[Kit Plummer]

It's a good write up.  But, kind off missed the point of both automation and monkey-maintenance.  

As with "infrastructure-as-code" the idea is that one has a manifest of exactly what is happening with the deployment/maintenance of infrastructure.  Most importantly, that manifest is also a living document which captures every change.

Automation isn't intended to replace anymore - good or bad employees - but it is intended to allow them to focus on more important things, like maintaining and managing the coded infrastructure.  I can't help but think of Rails' argument that convention is better than configuration.  Just like in this discussion, there's the 80% that will benefit from the conventions.  There will always be the 20% that push beyond the value of convention and require the ability to do their own configuration.  Again, automation just provides an opportunity for orgs the 20% be masters and 80% be monkeys (including dumb developers, lame project managers, and ignorant help-deskers :)).

Monkey-maintenance as in all industries is intended to remove the necessity for the "masters" to do the basic "is the power-cord plugged in?" tasks.  Automation indeed promotes the shift of basic work towards those with less skills and experience.  The ultimate benefit is organizations needs less masters and don't have to deal with human generated changes to dynamic processes which were most likely the cause of the problem in the first place.  Sure, stuff breaks, and we need knowledgeable peeps to be able to troubleshoot and ensure that the problem doesn't happen again.  But, over time the less knowledgeable will gain the requisite knowledge, especially if they are integrated into the design and implementation of the infrastructure.

Caveat.  I'm a software engineer, building automation tools - with the hope that the automation will allow me to develop more, and operators to provide me better metrics rather than solve problems I likely induced in the first place.

Kit

-- 
Kit Plummer :: MaestroDev :: 520.360.4729

[Ernest Mueller]

At 04:06 PM 2/20/2012, Philip J. Hollenback wrote:
>Chris Siebenmann on how continuing automation will make traditional
>system administration increasingly untenable:
>
>http://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutomationDownsideII

Yeah, I don't think a lot of that concern. Reminds me of when I took
over the Web Systems team at NI after a guy who "didn't believe in
automation because then you won't know how the systems work." Of
course since people were getting paged 200 times a week they were
quitting, and the systems were super unreliable. But we knew really
well how to go roll logs manually. Because, you know, that helps our
business achieve its objectives somehow.

Ernest

P.S. I got through that whole thing without cursing! Aren't you proud of me?

[Phil Cryer]

Relevant tweet, still makes me laugh.
https://twitter.com/#!/DEVOPS_BORAT/status/171729255439413248

P

--
http://philcryer.com

[John Ryding]

Kit,

You hit the nail on the head - the point of automation (and technology in general) is to abstract away low level details so people can focus on solving more complex problems.

I understand where these anti-DevOps arguments are coming from - people's jobs will be going away for the ones that don't adapt. Cloud is replacing the wall of hardware procurement, software is being created to deploy entire topologies of systems with a click of a button. I believe that the people that are really good at what they do with their sysadmin work aren't going to lose their job, they will just evolve into a new role where they become a knowledge resource and/or designer of complex systems. We have a guy on our team who is an amazing sysadmin and is an awesome knowledge resource. As a developer, I've thrown out ideas that he thought were really stupid and ended up proposing a better solution that I never would have thought of because he had a lot more experience than me.

If anyone is interested in more anti-DevOps arguments, this reddit post has a bunch: http://www.reddit.com/r/sysadmin/comments/pp860/why_i_hate_the_word_devops/

-John Ryding

Kit Plummer ---02/20/2012 05:43:25 PM---It's a good write up.  But, kind off missed the point of both automation and monkey-maintenance.

[Philip J. Hollenback]

Also check out the reddit comments on a recent blog post of mine:

http://www.reddit.com/r/sysadmin/comments/pbnv3/devops_is_here_whether_you_like_it_or_not/

I was essentially arguing the same point in that post: adapt or die.

P.

On Wed, Feb 22, 2012, at 12:03 PM, John Ryding wrote:
>
> Kit,
>
> You hit the nail on the head - the point of automation (and technology in
> general) is to abstract away low level details so people can focus on
> solving more complex problems.
>
> I understand where these anti-DevOps arguments are coming from - people's
> jobs will be going away for the ones that don't adapt. Cloud is replacing
> the wall of hardware procurement, software is being created to deploy
> entire topologies of systems with a click of a button. I believe that the
> people that are really good at what they do with their sysadmin work
> aren't
> going to lose their job, they will just evolve into a new role where they
> become a knowledge resource and/or designer of complex systems. We have a
> guy on our team who is an amazing sysadmin and is an awesome knowledge
> resource. As a developer, I've thrown out ideas that he thought were
> really
> stupid and ended up proposing a better solution that I never would have
> thought of because he had a lot more experience than me.
>
> If anyone is interested in more anti-DevOps arguments, this reddit post
> has
> a bunch:
> http://www.reddit.com/r/sysadmin/comments/pp860/why_i_hate_the_word_devops/
>
> -John Ryding
>
>
>

> From: Kit Plummer <kplu...@maestrodev.com>
> To: dev...@googlegroups.com,
> Date: 02/20/2012 05:43 PM
> Subject: Re: system administration vs. automation
> Sent by: dev...@googlegroups.com
>
>
>

> Email had 1 attachment:
> + graycol.gif
> 1k (image/gif)

[John Allspaw]

On the topic of automation and what I think to be a misunderstanding of the limits and repurcussions of it: I have a draft of what is likely to be a lengthy blog post on the topic of automation's role in web operations and development. But I can try to summarize here:

• Automation, used maturely, extends and augments the abilities of humans. It doesn't remove tasks, it increases the number of them.
• Automation doesn't *remove* human error, normally assumed. It actually *increases* the surface area by which fundamental surprises can take place, by introducing new dynamics (ex: automated changes generally require consideration on timing the orchestration, which isn't a consideration when the change is done manually)
• Automation underscores the "Law of Stretched Systems", because more work can be performed with less initiating tasks.
• Automation changes the role of the engineer, from a manual operator to a supervisor. This means that work doesn't go away; it simply changes. Usually this also means additional monitoring burdens.
• Automation *increases* the cognitive load and mental burden on operations (lowercase 'o', meaning all of engineering/development/operations) because the possibility for data overload (ex: what is the status/behavior of 1000 machines getting a configuration change within a minute, as opposed to one at a time over 3 days?), and functional resonance (ex: will any of the changes we make affect interconnected component behavior, like thundering herd effects on caching?)

A lot of these concerns are covered in Lisa Bainbridge's paper, "Ironies of Automation", which is quite old but still incredibly apt: http://www.bainbrdg.demon.co.uk/Papers/Ironies.html 

as well as Woods' "Decomposing Automation: Apparent Simplicity, Real. Complexity" 

http://t.co/Ts7ber1D

Regardless, I think that while I see the logic in a "automation will cause people to lose their jobs" and "automation is the answer to everything" because our adorable field of internet operations is only (arguably) 12 years old, I think that we'll one day catch up to the thinking of other fields such as aviation and manufacturing. Which is to say: we'll exploit automation for it's benefits, but become extremely aware of its limitations, dangers, and utility in specific cases.

A blunt and limited observation would also be that as time goes by, I'm challenged by hiring *more* people as time goes by, not less. Related to the increasing popularity of automation? Discuss.

-j

--
John Allspaw

[John Ryding]

> A blunt and limited observation would also be that as time goes by, I'm challenged by hiring *more* people as time goes by, not less. Related to the increasing popularity of automation? Discuss.

I completely agree with this statement. Simplifying the creation of these systems may results in MORE people getting hired to manage and grow the underlying infrastructure. This is because the amount of end users of the automation increases thanks to the reduction of complexity - where before only 5 people were able to set up a system because of 50 manual steps, now 100+ people can set the system up because all they have to do is click a button. As such, new problems arise for the underlying infrastructure team - scaling issues, new use cases introduced from a larger variety of users, etc. Like John said in his previous post:

> Automation, used maturely, extends and augments the abilities of humans. It doesn't remove tasks, it increases the number of them.

-John Ryding

John Allspaw ---02/22/2012 01:00:29 PM---On the topic of automation and what I think to be a misunderstanding of the limits and repurcussions

[Sascha]

I can't believe people fear for their jobs. I've never met a single
competent engineer in danger of having no work. Ever. There is
always work for the intelligent and competent and the more you
automate the lame stuff, the more you get to work on the fun stuff.

That is all.

On Feb 22, 12:22 pm, John Ryding <jryd...@us.ibm.com> wrote:
> > A blunt and limited observation would also be that as time goes by, I'm
>
> challenged by hiring *more* people as time goes by, not less. Related to
> the increasing popularity of automation? Discuss.
>
> I completely agree with this statement. Simplifying the creation of these
> systems may results in MORE people getting hired to manage and grow the
> underlying infrastructure. This is because the amount of end users of the
> automation increases thanks to the reduction of complexity - where before
> only 5 people were able to set up a system because of 50 manual steps, now
> 100+ people can set the system up because all they have to do is click a
> button. As such, new problems arise for the underlying infrastructure team
> - scaling issues, new use cases introduced from a larger variety of users,
> etc. Like John said in his previous post:
>
> > Automation, used maturely, extends and augments the abilities of humans.
>
> It doesn't remove tasks, it increases the number of them.
>
> -John Ryding
>

> From:   John Allspaw <alls...@gmail.com>
> To:     dev...@googlegroups.com,
> Date:   02/22/2012 01:00 PM
> Subject:        Re: system administration vs. automation
> Sent by:        dev...@googlegroups.com
>

> On Wed, Feb 22, 2012 at 12:03 PM, John Ryding <jryd...@us.ibm.com> wrote:
>
>   Kit,
>
>   You hit the nail on the head - the point of automation (and technology in
>   general) is to abstract away low level details so people can focus on
>   solving more complex problems.
>
>   I understand where these anti-DevOps arguments are coming from - people's
>   jobs will be going away for the ones that don't adapt. Cloud is replacing
>   the wall of hardware procurement, software is being created to deploy
>   entire topologies of systems with a click of a button. I believe that the
>   people that are really good at what they do with their sysadmin work
>   aren't going to lose their job, they will just evolve into a new role
>   where they become a knowledge resource and/or designer of complex
>   systems. We have a guy on our team who is an amazing sysadmin and is an
>   awesome knowledge resource. As a developer, I've thrown out ideas that he
>   thought were really stupid and ended up proposing a better solution that
>   I never would have thought of because he had a lot more experience than
>   me.
>
>   If anyone is interested in more anti-DevOps arguments, this reddit post
>   has a bunch:

>  http://www.reddit.com/r/sysadmin/comments/pp860/why_i_hate_the_word_d...
>
>   -John Ryding

>
>   Inactive hide details for Kit Plummer ---02/20/2012 05:43:25 PM---It's a

>   good write up.  But, kind off missed the point of bothKit Plummer

>   ---02/20/2012 05:43:25 PM---It's a good write up.  But, kind off missed
>   the point of both automation and monkey-maintenance.
>

>   From: Kit Plummer <kplum...@maestrodev.com>
>   To: dev...@googlegroups.com,
>   Date: 02/20/2012 05:43 PM
>   Subject: Re: system administration vs. automation
>   Sent by: dev...@googlegroups.com
>

>  graycol.gif
> < 1KViewDownload

[Scott Smith]

Fear is rarely something rationalizable

[Jesse Gonzalez]

I've been an system administrator in academia, small business, for a large hosting company then became an architect and and now find myself living as a devops engineer for the same said company.

Every role I've had there has always been some series of commands or a long drawn out process you'd have to run to do some job, and before I left those roles, the process was automated, usually via some bash or perl (back then). My mantra was "I'd rather watch work than do it". An interview candidate used that line recently and my response was let's hire this guy.

In academia every semester there were new students, and those students needed shell accounts, email accounts, AD accounts and the like. Amazed at how the many before me did that manually, I wouldn't have it, too much room for error. I never feared the loss of a job due to automating tasks that previously took 5 weeks to accomplish and upon completion take 5 minutes. I'd ultimately end up working on more complex projects, get exposure to more technology, gain some recognition, etc.

Working for a small company I automated work for other people, using Apache FOP to format invoices from some text source to PDFs. It was a welcomed by the team, not because they didn't have to do it anymore, but more so that they had the opportunity to work on other tasks.

All I'm saying is that while we move to automate more and more, there will always be work. Always. Addressing stability issues, fixing known edge case bugs, make monitoring suck less, ... The list goes on and on. Anyone that fears automation will replace their job probably should dust off their resume and start looking now. They probably don't "have it".

Jesse Gonzalez
jesse.g...@rackspace.com

[Scott McCarty]

I have never even worked at a place where there wasn't 80 hours of work per week, no matter how much automation we put in place or how many people we continued to hire.

Best Regards
Scott M

[cshl]

this is brilliant. please be sure to post a link to your blog post
when it goes live.

On Feb 22, 12:59 pm, John Allspaw <alls...@gmail.com> wrote:
> On the topic of automation and what I think to be a misunderstanding of the
> limits and repurcussions of it: I have a draft of what is likely to be a
> lengthy blog post on the topic of automation's role in web operations and
> development. But I can try to summarize here:
>

>    - Automation, used maturely, extends and augments the abilities of

>    humans. It doesn't remove tasks, it increases the number of them.

>    - Automation doesn't *remove* human error, normally assumed. It actually

>    *increases* the surface area by which fundamental surprises can take place,
>    by introducing new dynamics (ex: automated changes generally require
>    consideration on timing the orchestration, which isn't a consideration when
>    the change is done manually)

>    - Automation underscores the "Law of Stretched Systems", because more

>    work can be performed with less initiating tasks.

>    - Automation changes the role of the engineer, from a manual operator to

>    a supervisor. This means that work doesn't go away; it simply changes.
>    Usually this also means additional monitoring burdens.

>    - Automation *increases* the cognitive load and mental burden on

>    operations (lowercase 'o', meaning all of
>    engineering/development/operations) because the possibility for data
>    overload (ex: what is the status/behavior of 1000 machines getting a
>    configuration change within a minute, as opposed to one at a time over 3
>    days?), and functional resonance (ex: will any of the changes we make
>    affect interconnected component behavior, like thundering herd effects on
>    caching?)
>

> A lot of these concerns are covered in Lisa Bainbridge's paper, "*Ironies
> of Automation"*, which is quite old but still incredibly apt:http://www.bainbrdg.demon.co.uk/Papers/Ironies.html
>
> as well as Woods' "*Decomposing Automation: Apparent Simplicity, Real.
> Complexity" *
> *http://t.co/Ts7ber1D*
> *
> *

> >http://www.reddit.com/r/sysadmin/comments/pp860/why_i_hate_the_word_d...

>
> > -John Ryding
>
> > [image: Inactive hide details for Kit Plummer ---02/20/2012 05:43:25
> > PM---It's a good write up. But, kind off missed the point of both]Kit
> > Plummer ---02/20/2012 05:43:25 PM---It's a good write up.  But, kind off
> > missed the point of both automation and monkey-maintenance.
>

> > From: Kit Plummer <kplum...@maestrodev.com>
> > To: dev...@googlegroups.com,
> > Date: 02/20/2012 05:43 PM
> > Subject: Re: system administration vs. automation
> > Sent by: dev...@googlegroups.com

> > ------------------------------

> > *http://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutomationDownsideII*<http://utcc.utoronto.ca/~cks/space/blog/sysadmin/AutomationDownsideII>

>
> > P.
> > --
> > Philip J. Hollenback

> > *www.hollenback.net*<http://www.hollenback.net/>