Sizing a DevOps team

[Ray Nugent]

While I know this may not be the best venue for this question I think there is enough expertise here to provide an informed opinion :-)

I'm trying to develop a process for sizing a DevOps team. I have a good idea how to size a Dev team but not this new construct. Any advice?

Thanks

--
Ray Nugent

[Damon Edwards]

Hi Ray,

My first question would be: What do you consider a "DevOps team" and what do you want it to do? (Do you mean systems/release engineering? Do you mean a special team focused on project management or process improvement? Process architecture? Other? Some combo?)

It's my view that DevOps isn't a role but a world view, best practices, and tooling improvements that needs to find it's way into all roles across the development to operations lifecycle. But maybe that's not how others are seeing it.

-Damon

Damon Edwards

[ DTO Solutions, Inc.  | 1840 Gateway Drive - Suite #200, San Mateo CA 94404 | o: 1.650.292.9660 x705 | m: 1.415.830.5856 | f: 1.415.358.8435 ]

[Ernest Mueller]

I think it's fine to discuss this here, it's not like there's another good
devops forum to discuss non-toolchain questions...

Hmmm, I guess the question is relative to what? A "machines per admin"
kind of metric? Or "Ops per Dev?" I think there's a lot of variables
there. (And what do you mean by DevOps...)

Our team, which is writing the Web side stuff for product integration, is
as follows. Our use cases are
a) Web-enabled features for desktop hardware/software to connect to
b) SaaS products leveraging our product code

Staffing:
Various not easily countable developers writing functional code for our
software products
3 devs writing the "Web side" of the equation - auth, licensing, user
portal, proxies, etc.
1 systems engineer doing development to automate stuff
1 systems engineer doing systems work
1 operations guy to take pages - eventually as $ allows growing to the
"minimum stable" follow-the-sun number of 6 operations folks, two for each
time zone (24x7 support plus mutual coverage - we use Hungary and Malaysia
as follow the sun sites since we have installations there). Fixers, not
implementers.

We could use some additional devs as we grow... Starting something up from
scratch, we needed a pretty strong mix of ops to devs, nearly 50/50; as the
environment matures we can probably have more devs to the ops (also depends
on how much is automated).

I am not quite sure if it counts as DevOps, but my previous Web
applications administration team that tried to work closely with our Web
programmers on our Web site was:

20 or so business analysts
50 or so programmers
7 admins/ops
3 Web designers

But that was spreading pretty thin. It depends on a lot of factors but I'd
say running less than 1:5 ops to dev means you're probably not effectively
going to do devops. It'll vary if you have devs performing tasks an ops
person would otherwise, of course, which is very common...

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

From: Ray Nugent <ranu...@gmail.com>

To: devops-t...@googlegroups.com

Date: 04/15/2010 02:07 PM

Subject: Sizing a DevOps team

Sent by: devops-t...@googlegroups.com

[Kris Buytaert]

On Thu, 2010-04-15 at 14:30 -0500, Ernest Mueller wrote:
> I think it's fine to discuss this here, it's not like there's another good
> devops forum to discuss non-toolchain questions...
>

agile-system-...@googlegroups.com

Which is the initial Devops mailinglist ...

[Vladimir Vuksan]

Sizing depends on the skills and abilities of people on team. It also
depends what kind of environment you are trying to manage. DevOps at least
how I interpret is a way of tearing down walls between development and
operations. So you could have a lean ops team if developers are willing to
assume some of the ops tasks. That may be impractical if the
infrastructure is complicated. This all depends on what the problem is.

[Ernest Mueller]

Oh, really? Cool, I'll go join. Seems like pretty low traffic though.

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

From: Kris Buytaert <Kris.B...@gmail.com>

To: devops-t...@googlegroups.com

Date: 04/15/2010 02:33 PM

Subject: Re: Sizing a DevOps team

Sent by: devops-t...@googlegroups.com

agile-system-...@googlegroups.com

--
To unsubscribe, reply using "remove me" as the subject.

[Scott Smith]

On 4/15/10 12:33 PM, Vladimir Vuksan wrote:
> and operations. So you could have a lean ops team if developers are
> willing to assume some of the ops tasks.

Sweet, can we have them make coffee and book our hotel rooms for conferences too?

Seriously though, this isn't about taking two peoples' roles and turning them into one.

-scott

[Vladimir Vuksan]

I am tempted to answer with an equally stupid remark but I will refrain. I
am not saying that two peoples' roles should be turned into one. This all
depends on the complexity of your app, your team's abilities, level of
traffic, customer requirements etc. Since resources are not unlimited you
have to make choices. That said I think not having an ops person is a bad
idea however that doesn't mean you can't run a business without one.

[Scott Smith]

On 4/15/10 2:38 PM, Vladimir Vuksan wrote:
> I am tempted to answer with an equally stupid remark but I will refrain.

OK, I'll refrain from tongue-in-cheek remarks. :)

> I am not saying that two peoples' roles should be turned into one. This
> all depends on the complexity of your app, your team's abilities, level
> of traffic, customer requirements etc. Since resources are not unlimited
> you have to make choices.

Sure, but this should be a given. Nothing is a simple matter in the real world.

-scott

[peco]

Using the number of apps that will be supported could be a vector used
for sizing devops. Of course that leads to the question of what is
considered an application but not getting into this one on this
thread :)

If you have to support 50 vs 500 vs 5000 apps you will need different
staffing. I think server count is not a complete measure. There are a
ton of one-function weby apps that run on 500+ machines with 1-2 ops
guys.

Another thing to consider is the support tiering what each level is
responsible for. If you level 3 engineers building the systems will
also need to work on end user issues/troubleshoot and tickets you will
need more of them.

So the overall vectors for sizing I think are in order of importance
applications, users, systems

Thoughts?

Cheers
peco

PS: If you are releasing a production application consider having at
least one devops/ops. If they are twiddling their thumbs on production
ops they can help with code repositories, continuous integration,
ticketing systems, testing frameworks (and world peace if there is
time left)

[Ernest Mueller]

I think it's a fair question what roles really need to be 'dev' vs 'ops'.
In reality, what you have is a collection of people with varying skill
sets. One person may not code at all, another may know nothing but code,
but there are many mixed-skill individuals out there.

Is ops a dedicated role in all circumstances? No. Heck, when I was at
FedEx I was surprised by our large dev group not having any "system admins"
- just the couple devs that had UNIX of DBA knowledge (or moved too slow
when volunteers were solicited, I guess) did the admin stuff.

What are the reasons to separate out dev and ops, and are they valid or not
valid?
1. "Separation of duties" in terms of not letting someone push their own
code and also control production. A noble goal, but doesn't it apply to
"ops code"? In reality, having a certain defined set of people who can
authorize release of code is probably sufficient, even if they are 'devs'.
2. Don't want to waste money by wasting programmer time on system admin;
programmers are expensive and ops guys are barely over blue collar ITT
graduates! I think we know this isn't true any more; system engineers are
more highly paid on average according to the 2010 salary survey I just
read. You have high and low end admins and high and low end programmers.
3. Specialization - if your org is large enough, there's some benefit to
the "group of ops guys" noodling together, but the endgame there is
corrosive and why we're talking about having ops hand in hand with dev now
anyway.

If developers "shouldn't" do ops... Should ops people not develop? I think
if you have sufficient resources of people with the right skill for the job
working on dev and ops, it doesn't matter if it's the same people,
different people, or what. Are there other drivers I'm missing?

Ernest
______________________
UN-altered REPRODUCTION and DISSEMINATION of
this IMPORTANT information is ENCOURAGED.

From: Vladimir Vuksan <vli...@veus.hr>

To: devops-t...@googlegroups.com

Date: 04/15/2010 04:38 PM

Subject: Re: Sizing a DevOps team

Sent by: devops-t...@googlegroups.com

[Ray Nugent]

First of all thanks to everyone for their input. What I got out of this is that DevOps is still an undefined role. Possibly not even real at this point. My goal was to try to see if I could come up with a way to  "calculate" the ratio of ops folks to dev folks. I realize there is a lot of "it depends" in this effort but I thought I'd check with practitioners (I'm more a dev guy myself) for some real world input. My gut feel is 3 devs per 1 ops. What do you guys think?

Ray

--
Ray Nugent
www.smartscalesystems.com

[Diego Parrilla Santamaría]

I think it's imposible to size something if we don't have enough
measures taken from empiric tests. “You can’t control what you can't
measure.” DeMarco dixit.

First we have to define the roles, goals and tasks, and then take measures.

Diego Parrilla | VP Product Management | Abiquo | +34 620 57 81 46 |
diego.p...@abiquo.com

[Joshua Timberman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Apr 16, 2010, at 12:18 AM, Ray Nugent wrote:

> First of all thanks to everyone for their input. What I got out of
> this is that DevOps is still an undefined role. Possibly not even
> real at this point. My goal was to try to see if I could come up
> with a way to "calculate" the ratio of ops folks to dev folks. I
> realize there is a lot of "it depends" in this effort but I thought
> I'd check with practitioners (I'm more a dev guy myself) for some
> real world input. My gut feel is 3 devs per 1 ops. What do you guys
> think?

As Damon said, DevOps isn't about a 'role', its about a cultural
change between teams to boost development and operations collaboration.

The real metric you need to look at is the number of applications
you're supporting, and how many developers and admins it takes to
support and deploy those applications. Its a complexities of scale
issue. Keep in mind in a DevOps friendly environment, the
infrastructure itself is an application.

When the infrastructure is an application, you need "developers" who
can manage it, and thats where sysadmins who use software development
practices are valuable.

- --
Joshua Timberman
http://twitter.com/jtimberman

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkvJ0aUACgkQTGP9Ng9XhV6fbQCfdHiVZYiIXIGgMLWAC1POOROg
k9YAoJg1Ejlp21KJcdct+LZ/pJQAJ/sQ
=rFAe
-----END PGP SIGNATURE-----


--
Subscription settings: http://groups.google.com/group/devops-toolchain/subscribe?hl=en

[jallspaw]

100% agree with Joshua.

- ops guys:server ratio is useless
- ops guys:developers ratio is useless

whereas:
- ops guys: infra/app complexity has value
- developers: infra/app complexity has value

Folks looking for a 'rule of thumb' on either of those two will have a
tough and long time looking for something that shouldn't be there. Not
really the way to go about it.

There is no "devops" role. Unless "people who communicate and
collaborate" is a specific role in your company. If so, that's one
messed-up company. :)

[James Bailey]

If you are sizing teams then take into consideration Dunbar number
theory[0] and this lovely little article on group size[1], I have
seen this happen so many times, in so many walks of life. The simple
fact of group size can have an amazing influence on group dynamics and
efficiency.

Quick rules of thumb keep for teams in sizes of 5-7, big dip
performance around 12-15 then a nice rise in effectiveness until
around 50-80 when the group begins to split and unravel again.

Humans we maybe be able to use tools but we still dumb monkeys. :)

[0] http://en.wikipedia.org/wiki/Dunbar%27s_number
[1] http://web.archive.org/web/20051214125132/www.lifewithalacrity.com/2004/03/the_dunbar_numb.html

Peace Jim