The relationship between deployments and operations

[Philip J. Hollenback]

Maybe it's time to change the subject? Mark Imbriaco posited a few days
back that deployment and operations are orthogonal. Then he
successfully trolled me (in a good way!) to write a blog post about it:

http://www.hollenback.net/WhoDoesDeployments

Curious what everyone thinks.

P.
--
Philip J. Hollenback
www.hollenback.net
@philiph

[Joe McDonagh]

I disagree with the main theme; I think that deployments are a sub-set
of operations. It just so happens that your infrastructure is so huge,
you need multiple people to manage this sub-set of operations.

--
Joe McDonagh

AIM: YoosingYoonickz Google: Joseph.E.McDonagh
IRC: joe-mac Skype: therealjoemac

[Scott Smith]

It's only a subset if you make it so.

[Matt Ryanczak]

On 03/19/2012 01:31 PM, Scott Smith wrote:
> It's only a subset if you make it so.

Indeed. I think the most successful release management that I have seen
has been in shops where releases are managed by a team composed of
members of both the operations and development. Ia big shop where
release is likely a full time job, why not have a group dedicated to it?

I'd be interested in hearing more about the scope of a release
management group. How does that scope change when releasing new vs old
applications? For instance, in the case of new releases, who enables
monitoring and similar ongoing activities? Is that part of release or is
it a ticket to the NOC after the fact? If new monitoring tools are
required, are they part of the release?

[Joe McDonagh]

By that logic, any organization with an infrastructure large enough to
warrant separate teams for monitoring or CM would make that not part of
ops either.

On 3/19/12 2:19 PM, Matt Ryanczak wrote:
> On 03/19/2012 01:31 PM, Scott Smith wrote:
>> It's only a subset if you make it so.

[Ramin K]

On Monday, March 19, 2012 8:32:33 AM UTC-7, Phil Hollenback wrote:

Maybe it's time to change the subject?  Mark Imbriaco posited a few days
back that deployment and operations are orthogonal.  Then he
successfully trolled me (in a good way!) to write a blog post about it:

http://www.hollenback.net/WhoDoesDeployments

Curious what everyone thinks.

I went the opposite direction and made deployment part of Development. Taking the last 18 months into account I'm not sure I'd ever let Ops own code deployment again. In my experience letting Ops own code deployment causes two problems. It insulates the agents of change, developers, from the consequences of their acts. It also encourages Ops to see themselves as final gatekeepers or line of defense which results in all the usual us vs them problems. When Dev is responsible for code pushes "throwing code over the wall" becomes a statement with no meaning. Responsibility of the system is now jointly owned without having to spend any time driving that home.

As with all implementations YMMV depending on your environment, but it's something to think about.

Ramin

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Interesting. What's the relationship and linkages between Ops and Dev?
Obviously with Dev owning deployment they still need to understand how
Production works and Ops still need to know how the application(s) work
to monitor/manage/run them? How do you achieve that?

Regards

James Turnbull

- --
Author of:
* Pro Puppet (http://tinyurl.com/ppuppet)
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPZ8QyAAoJECFa/lDkFHAyaI0H/2KKml9eopsTo5apdWXZ3/fy
Ynntzeoe5aUQB9Z7eV2HJe4KWz6zNsO2MgWy+RSB+yAR2ReHE4yTBexdwoIClYiv
Yxmw+HrS8P7bFxHB0AoW3Y5k0Iq/GUtVvDk/FAIWgicYWdtAstm3YU/Tihm19UoG
VgUmyevP5ogNL7GIl3OJ8DLCs9wIZimu745/x7FCTCap9q5ra+hXBkNe7fVbkWMx
ntaDrfV0iknXDLdftSLeWExd5VGDjwAk/L22yIOa7mkMqQgEMREFRuFCfr+yW43c
k1EksO9SSPYDX4NXU4eepEXKhnydkF32aBroJ/3I41FQBzIhzUu8aUmmqkRb9hc=
=6ENS
-----END PGP SIGNATURE-----

[Roger Gregory]

I'd agree with jointly owned, and further that removing the "us vs them" culture is key to better product. But I still believe there is a difference in posture for each (development and ops) which is important.

In our case, I've tightly woven Dev and Ops. It makes for a nice fit, and while developers have control over deployment, the Ops side provides critical insight and perspective (or so I'd like to think).

I think you're right in that everyone feeling responsible is the magic sauce..

-r

[Miles Fidelman]

James Turnbull wrote:
>
> Interesting. What's the relationship and linkages between Ops and Dev?
> Obviously with Dev owning deployment they still need to understand how
> Production works and Ops still need to know how the application(s) work
> to monitor/manage/run them? How do you achieve that?
>

There is the model that developers also gets to wire deployments into
monitoring and alerting, and wear the pager. That tends to provide a
pretty direct feedback loop.

--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra

[Roger Gregory]

I'll weigh in here..

Interesting. What's the relationship and linkages between Ops and Dev?
Obviously with Dev owning deployment they still need to understand how
Production works and Ops still need to know how the application(s) work
to monitor/manage/run them? How do you achieve that?

Development 'owns' code releases which they generally coordinate on a project basis. Deployment is via a kick-script, which deploys the relevant release and generates notifications for the development team. Dev reports up through ops (so to speak -- I consider myself much more on the Ops side than dev and they report to me). No deployment takes place outside that process, but there are no restrictions (other than we all frown upon unnecessary Friday deployments).

My team is responsible both for the uptime of the sites as well as feature development and enhancement, so there's a nice sense of direct ownership and accountability for keeping things running.

I think it's a great way to mix -- each 'side' has specializations, but each works together toward s common goal. That said, I could easily see how if separated there could be a tension between the two teams..

- Roger 

[Ramin K]

Good question and much harder to answer than I had anticipated.

Coarsely Dev owns site/product code while Ops owns infrastructure code and both of us attempt to do releases without affecting the other. We all have access to each other code trees, but it's rarely used unless it's a simple case. If I want to make a major change in the the site tree or deployment process I find it better to consult with a Dev about what I want and why, letting them build the solution and vice versa when it comes to Puppet, OS builds, or monitoring. Coordination happens most consistently per major project the same as Roger mentioned.

Also each team tries to broadcast intentions to the other team as soon as possible. I am a big fan of signaling work needed before it's needed. You're less likely to surprise someone, you can nearly eliminate last minute scrambles, I may do some research while watching tv or ask around on IRC, someone may get and itch and just decide to do it during a lull, etc.

Case in point we're planning to switch to 12.04 LTS in May so we can stop maintaining a number of packages. As far as we can tell Dev shouldn't need to do anything, but we've prepped them to be aware of the change. We're planning around their release schedule to keep blocks to their progress to a minimum. On the Dev side Node.js was requested to create some additional site functionality a month before they started writing code. This allowed us to launch node on stage before it was needed and work out some of our operational issues like CM, packaging, and monitoring. After the Ops pass Dev did their own operational pass for deployment order and code. Both teams raised tickets against each other, Dev for the best way to start/stop/keep node running and Ops for a nice status page to monitor.

Both teams dark launch when possible to make the changes minimal during a push and we probably break stage more often than most as failures in coordination hit there.

I don't believe ownership of any deploy process can imbue knowledge on the team that holds it. After all once you have deploy working most of us ignore it until it breaks or new functionality is needed. My opinion is that Development understands production far more by working within it rather than being walled off from it. Ops is also forced to be more proactive about what is going to production when we can no longer hold it up because we didn't get involved earlier. Breaking my own reactive habits of years of break/fix work has been personally frustrating at times.

And finally a big caveat, we're a small 15 person startup with an average age of 34. The majority of the company has 10+ year of experience in their current role. I think what we're doing can be replicated and is a useful point of view other teams, but how much of it is the process and how much is the team I can not say. I did like hearing of at least one other  team that is acting similarly.

Ramin

[Spike Morelli]

I've seen a bit of both and a mix and I quite like the mixed approach.

In the bit of both, I've seen smaller orgs giving the ops team the job to deploy and that sometimes did result in throwing it over the wall/ops being the gatekeeper, but other times (in more devops envs) it worked just fine. Technically speaking I think that's fine if you can dodge the poor-culture bullet. In some smallish java shops I've seen devs doing releases which more often than not just meant shipping some war files over.

In some bigger shops (50 devs in several teams, 10+ sys ppl and 6 release folks) it worked very much like Philip described in his blog post and I'm not honestly sure we could have made it work otherwise. In that scenario a sys eng and a release person would sit with the dev team throughout the project and advise devs when necessary. Release folks were quite apart from ops in terms of skills and interest, not really into unix or infrastructure and very focused on scripting and artefacts management, so a merger was not an option. And it honestly worked pretty well that way, there was a lot of conversation going on at all times and the divide was less of a cultural one and more of an interest/specialisation one. Does that mean that in a big enough corp it makes sense to have CM and monitoring teams? Maybe, maybe not, I've began to think that team can be perceived as interest groups and as long as you keep good comm channels it's just helpful for people to huddle up around what they like most and are most proficient with.

About the mix env. What happens there is that the *first* release of one app is gated on ops and the subsequent ones can be carried out by devs independently. This has worked really well for us and it's an interesting model to me.

One quick note about devs carrying pagers, we tried that out in a couple places I worked at and while it nice in theory in practice most times it wasn't useful. What we always ended up with was a dev on call that we could escalate to if needed, but otherwise not directly getting paged by the monitoring system.

hope that helps,

Spike

--

http://www.spikelab.org/

http://twitter.com/spikelab

http://uk.linkedin.com/in/spikemorelli

[John Ryding]

Spike,

Why do you think the pager duty failed? Was it because there were no post-mortems done after the outage to explain to development what went wrong?

I know I'm mostly guessing from the little information you provided, but I would be interested to hear your thoughts.

Sent from my iPad