Internet operations ISN'T a part of IT?

50 views
Skip to first unread message

John Vincent

unread,
Mar 7, 2012, 2:38:39 PM3/7/12
to dev...@googlegroups.com
In another thread Gareth started, the following comment was made:

" I would add that Internet operations is not IT/IS. Internet
operations is an extension of engineering, not information services. I
think this is an important distinction to be made in large
organizations. They often lump anything with a CPU under the IT
department. The roles are very different."

I looked at that for quite some time and I simply cannot accept that
as anything approaching common sense.

When I worked for the local print media outlet here in Atlanta, there
were two sides of the house - business and IT (or IS in other
companies). To everyone NOT in "computing", everything is IT -
development, system administration, security, network administration,
desktop support. I would wager this is the case in MOST companies
where technology is not your primary export.

Additionally, this particular attitude feels like a dangerous semantic
game. I guess by now everyone knows how I feel about standing up
targets. Does anyone else agree with that statement?

Please note I'm not calling out the person who MADE the statement. My
goal is not to insult that person. I'm just trying to wrap my head
around it.

--
John E. Vincent
http://about.me/lusis

Matt Ryanczak

unread,
Mar 7, 2012, 2:44:20 PM3/7/12
to dev...@googlegroups.com
As the person who made that statement I'd love to hear more about why
you think this is a dangerous concept.

I agree that "To everyone NOT in "computing", everything is IT" and I
think that this is a dangerous thing. When supporting your product
(let's say web application) is lumped in with Windows licenses and
printer toner by your business leadership you have a problem.

Christopher Webber

unread,
Mar 7, 2012, 2:50:53 PM3/7/12
to dev...@googlegroups.com
This pretty easily goes both ways, but, I think that fundamentally, one of the big issues is that "IT" doesn't think and behave like Operations even though that is quickly becoming a huge part of most IT shops. For example, the desktop support team ends up being pretty small and most apps become web apps, at least here on campus. If most/all of your enterprise apps are web apps, how do you not make that transition?

-- cwebber

James Turnbull

unread,
Mar 7, 2012, 2:51:42 PM3/7/12
to dev...@googlegroups.com
Matt Ryanczak wrote:
> As the person who made that statement I'd love to hear more about why
> you think this is a dangerous concept.
>
> I agree that "To everyone NOT in "computing", everything is IT" and I
> think that this is a dangerous thing. When supporting your product
> (let's say web application) is lumped in with Windows licenses and
> printer toner by your business leadership you have a problem.

Yes - a problem of elitism. I find this sort of thinking really
dangerous. Let me play back I heard when you said this:

"We have the elite Web Engineers who run web-facing services over here."

"Who are those guys over there?"

"Oh toner monkeys/operators/NOC staff/rack-n-stackers. Ignore them."

IT's job is to facilitate the business doing business. All of IT. It's
awesome your web services run. Kudos. But what about the Windows box
doing your Accounts Receivable? Or the Procurement systems. Or Finance.
Or HR. If these functions fail maybe your business doesn't fail in the
same immediate public way it does with your web services but it fails
all the same.

Regards

James Turnbull

--
Author of:
* Pro Puppet (http://tinyurl.com/ppuppet)
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Pro Nagios 2.0 (http://tinyurl.com/pronagios)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

John Vincent

unread,
Mar 7, 2012, 2:52:50 PM3/7/12
to dev...@googlegroups.com
On Wed, Mar 7, 2012 at 2:44 PM, Matt Ryanczak <ryan...@gmail.com> wrote:
> As the person who made that statement I'd love to hear more about why you
> think this is a dangerous concept.
>

The danger is one of ivory towerism, new straw men for people to
attack, the idea that somehow you don't have to play by the same
rules?

> I agree that "To everyone NOT in "computing", everything is IT" and I think
> that this is a dangerous thing. When supporting your product (let's say web
> application) is lumped in with Windows licenses and printer toner by your
> business leadership you have a problem.

I disagree here. Yes, I've seen this lead to budget issues among other
things but the rest of the business really doesn't care. Really they
don't. They don't get the nuances of different types of IT. Hell, they
don't even get the nuances of non-IT departments.

And here's the thing, it doesn't really matter if they do or not.
Unless your business *IS* IT, you don't need to know. Sure the
communication could be smoother but does the business need to know or
even care that, say facilities actually has janitors vs. security
guards vs. HVAC maintenance? No. What they care about is that the
essential service is being delivered.

The same thing hold true for IT. If your responsibility is developing
an internal web application, your end responsibility is not really any
different than desktop support - you're providing a service.

Trust me, I've been on the flip side of this. At a previous company,
our "CTO" was called "Director of Office Automation". Yes, it was as
bad as it sounds. We're talking about the role of moving from pen and
paper to typewriters and copiers. But creating a whole NEW
division/department to interface with the business makes no sense.


>
>
> On 03/07/2012 02:38 PM, John Vincent wrote:
>>
>> In another thread Gareth started, the following comment was made:
>>
>> " I would add that Internet operations is not IT/IS. Internet
>> operations is an extension of engineering, not information services. I
>> think this is an important distinction to be made in large
>> organizations. They often lump anything with a CPU under the IT
>> department. The roles are very different."
>>
>> I looked at that for quite some time and I simply cannot accept that
>> as anything approaching common sense.
>>
>> When I worked for the local print media outlet here in Atlanta, there
>> were two sides of the house - business and IT (or IS in other
>> companies). To everyone NOT in "computing", everything is IT -
>> development, system administration, security, network administration,
>> desktop support. I would wager this is the case in MOST companies
>> where technology is not your primary export.
>>
>> Additionally, this particular attitude feels like a dangerous semantic
>> game. I guess by now everyone knows how I feel about standing up
>> targets. Does anyone else agree with that statement?
>>
>> Please note I'm not calling out the person who MADE the statement. My
>> goal is not to insult that person. I'm just trying to wrap my head
>> around it.
>>
>

--

Paul Graydon

unread,
Mar 7, 2012, 3:04:07 PM3/7/12
to dev...@googlegroups.com
On 03/07/2012 09:51 AM, James Turnbull wrote:
> Yes - a problem of elitism. I find this sort of thinking really
> dangerous. Let me play back I heard when you said this:
>
> "We have the elite Web Engineers who run web-facing services over here."
>
> "Who are those guys over there?"
>
> "Oh toner monkeys/operators/NOC staff/rack-n-stackers. Ignore them."
>
> IT's job is to facilitate the business doing business. All of IT. It's
> awesome your web services run. Kudos. But what about the Windows box
> doing your Accounts Receivable? Or the Procurement systems. Or Finance.
> Or HR. If these functions fail maybe your business doesn't fail in the
> same immediate public way it does with your web services but it fails
> all the same.
>
90% of my time is server and infrastructure focused, but that 10% of the
time? Changing toners, fixing printers, resolving problems on Windows
workstations (even though I only use Linux on my workstation and have
for so many years my Windows skills are rusty.) The junior in our
department probably splits more 80/20 for that, and I'll admit I tend to
'encourage' him to go fix some of the Windows problems ;) but they're
all still critical as far as I'm concerned:

Uptime is not servers, or services, uptime is business.

Christopher Little

unread,
Mar 7, 2012, 3:07:45 PM3/7/12
to dev...@googlegroups.com
Uptime is not servers, or services, uptime is business. 

That's probably the shortest, sharpest definition I've heard.

Matt Ryanczak

unread,
Mar 7, 2012, 4:00:48 PM3/7/12
to dev...@googlegroups.com
On 03/07/2012 02:51 PM, James Turnbull wrote:
> Yes - a problem of elitism. I find this sort of thinking really
> dangerous. Let me play back I heard when you said this:
>
> "We have the elite Web Engineers who run web-facing services over here."
>
> "Who are those guys over there?"
>
> "Oh toner monkeys/operators/NOC staff/rack-n-stackers. Ignore them."

I certainly don't mean to diminish one role over the other. They are
different roles imho. I'll be the first to admit that I probably have
tunnel vision, pretty much all I do is Internet operations and that has
been my focus for many years now. I was responsible for IT at one time,
The IT people I managed were part of a larger engineering organization
along with software engineers, systems engineers and QA people. The IT
people mostly did internal IT stuff. I could never find someone who
could (or would) both run and exchange server, fix a PC and engineer a
good Linux based clustering solution or write code.

So here I sit in my Internet operations bubble wondering.....
How does a business map devops methodologies to corporate IT?
How does a company that makes cheese wheels, or door stops leverage
devops methodologies?
Perhaps the question is how does devops map to business operations in a
generic sense?

btw, this is a very thought provoking conversation. thanks!

John Vincent

unread,
Mar 7, 2012, 4:12:56 PM3/7/12
to dev...@googlegroups.com
On Wed, Mar 7, 2012 at 4:00 PM, Matt Ryanczak <ryan...@gmail.com> wrote:
> On 03/07/2012 02:51 PM, James Turnbull wrote:
>>
>> Yes - a problem of elitism.  I find this sort of thinking really
>> dangerous. Let me play back I heard when you said this:
>>
>> "We have the elite Web Engineers who run web-facing services over here."
>>
>> "Who are those guys over there?"
>>
>> "Oh toner monkeys/operators/NOC staff/rack-n-stackers. Ignore them."
>
>
> I certainly don't mean to diminish one role over the other. They are
> different roles imho. I'll be the first to admit that I probably have tunnel
> vision, pretty much all I do is Internet operations and that has been my
> focus for many years now. I was responsible for IT at one time, The IT
> people I managed were part of a larger engineering organization along with
> software engineers, systems engineers and QA people. The IT people mostly
> did internal IT stuff. I could never find someone who could (or would) both
> run and exchange server, fix a PC and engineer a good Linux based clustering
> solution or write code.

But really isn't this simply a question about responsibility and
specialization? I don't expect my Exchange guys to be linux experts
per se because that's a discipline unto itself. I do however expect
folks to be cross-trained for back up purposes.

>
> So here I sit in my Internet operations bubble wondering.....
> How does a business map devops methodologies to corporate IT?
> How does a company that makes cheese wheels, or door stops leverage devops
> methodologies?
> Perhaps the question is how does devops map to business operations in a
> generic sense?
>

There are quite a few stories of non-tech companies managing systems
with puppet for instance. I wish I could recall one I heard about last
year. One of the problems we have in general is that what I'll call
traditional ops (for the sake of this conversation) are largely using
"legacy" platforms. The tool chains did not start with HPUX and
Window. They started with Linux and are being expanded to account for
those.

Combine that with the fact that, in a windows world, your approach is
typically one of golden images and difficult to automate components. I
mean I've managed to pull of some amazing shit with the Exchange
server cli tools but I don't want to do it ever again.

IT has no inherent value to a non-technical company (in the general
sense). IT is a cost sink. It's firefighters and folks with poor
hygiene. ANYTHING you do at that point to build a better relationship
is valuable. If it means you can cut staffing costs or reduce time to
deliver a service, it provides more value than they see now.

> btw, this is a very thought provoking conversation. thanks!

Whew. I was worried for a minute. I didn't want you to think I was
jumping on you. I was just legitimately concerned if I was in some
sort of bubble myself in disagreeing.

Joe Miller

unread,
Mar 7, 2012, 4:16:10 PM3/7/12
to dev...@googlegroups.com
I have met many people who could both fix a PC and engineer a good linux based clustering solution, but I also need to be aware of my fiduciary duties to the business. It's fine to have this senior engineer fix someone's PC once in a while but I really should go out and find someone cheaper to do it. In a startup, this "many hats" thing is a bit different than a large organization too.

That's not to say the cheaper role is providing less value to the business and I think @lusis is right that at the end of the day no one outside of IT cares how the service is delivered, just make sure it gets delivered and do so at a reasonable cost.

Also, there's plenty of room for new thinking and innovation in this desktop support (my term) world. Just look at what Facebook did with "IT Vending machines":  http://www.geek.com/articles/gadgets/facebook-introduces-vending-machines-for-computer-accessories-20110711/

Matt Ryanczak

unread,
Mar 7, 2012, 5:30:05 PM3/7/12
to dev...@googlegroups.com
On 03/07/2012 04:12 PM, John Vincent wrote:
> There are quite a few stories of non-tech companies managing systems
> with puppet for instance. I wish I could recall one I heard about last
> year. One of the problems we have in general is that what I'll call
> traditional ops (for the sake of this conversation) are largely using
> "legacy" platforms. The tool chains did not start with HPUX and
> Window. They started with Linux and are being expanded to account for
> those.

Is devops a set of tools? Methodologies? Both?

Can we define devops? Wikipedia says: "In computing, "DevOps" is an
emerging set of principles, methods and practices for communication,
collaboration and integration between software development
(application/software engineering) and IT operations (systems
administration/infrastructure) professionals.[1] It has developed in
response to the emerging understanding of the interdependence and
importance of both the development and operations disciplines in meeting
an organization's goal of rapidly producing software products and services."

This definition does a good job of defining what I think (thought?)
devops to be. Do people think this is a reasonable definition? Perhaps
it is not? Wikipedia is not exactly perfect :) I'm sure we can help
shape what the definition should be.

>> btw, this is a very thought provoking conversation. thanks!
>
> Whew. I was worried for a minute. I didn't want you to think I was
> jumping on you. I was just legitimately concerned if I was in some
> sort of bubble myself in disagreeing.

This type of conversation is exactly why I joined this list! I'm trying
to define what devops means to me and our industry as well business in
general. Everyone's thoughts have been very helpful toward that end.

Joe Miller

unread,
Mar 7, 2012, 5:39:00 PM3/7/12
to dev...@googlegroups.com
Perhaps we just invented a new term -- BizOps or BizIT.  An emerging set of principles for how the rest of the Business collaborates and engages with IT ? =)

Ken Sheppardson

unread,
Mar 7, 2012, 6:37:48 PM3/7/12
to dev...@googlegroups.com
For me this goes hand-in-hand with the classic dilemma of whether IT lives under the CFO/finance or the CTO/VP Engineering. The distinction that always helps me figure out what lives where is: Who is your customer?

If you're focused inward, that is, on addressing the needs of the companies employees (ala HR), living in the organization on the finance/business operations side makes total sense to me. That can be everything from changing toner cartridges to managing SaaS solutions to developing and deploying internal tools.

If you're focused outward, building and supporting the company's products, that's the other side of the shop in my opinion.

I think the size of the company and the nature of your business determines the relative size of those two efforts and whether it's one team or two. For a new web startup, the person writing Chef recipes to build boxes on EC2 is likely also setting up the WiFi and buying ink. That "group" probably lives in Engineering until the finance/ops side realizes they need better control over and visibility into stuff like ink costs. In a more mature web, outwardly focused business I think it's inevitable you have two groups. "Help desk"-style work splits off into finance/ops, customer-centric systems development lives in a separate organization, getting tasked now and then to help with internal projects as necessary. Eventually you grow to the point where you have to start building some inwardly-focussed development capability... or you learn to outsource and rely on SaaS solutions. If your not an information-centric business, all your "IT" will probably live in finance/ops under one roof until and unless you realize/decide that sales, marketing, and customer service all need some dedicated support, then voila, you again end up evolving into an organization with two groups: one focused inward, the other outward.

Ken

-- 
Ken Sheppardson
Director, Development Operations
CrowdFlower  |  415-625-3320  |  2111 Mission Street, Suite 302, San Francisco, CA 94110 

Jim Bird

unread,
Mar 7, 2012, 7:29:38 PM3/7/12
to dev...@googlegroups.com

Scott Smith

unread,
Mar 7, 2012, 8:53:17 PM3/7/12
to dev...@googlegroups.com
I think you mean to say "UNSUSCRIBE ME NOW!!!!"

Ernest Mueller

unread,
Mar 7, 2012, 9:00:19 PM3/7/12
to dev...@googlegroups.com
At 01:38 PM 3/7/2012, John Vincent wrote:
>In another thread Gareth started, the following comment was made:
>
>" I would add that Internet operations is not IT/IS. Internet
>operations is an extension of engineering, not information services. I
>think this is an important distinction to be made in large
>organizations. They often lump anything with a CPU under the IT
>department. The roles are very different."
>
>I looked at that for quite some time and I simply cannot accept that
>as anything approaching common sense.

It's not that hard. It's been that way in all the shops I've been in.

At NI,when starting a SaaS product line, we started a DevOps team in
R&D rather than utilize IT. Now, I had managed a Web Ops team in IT
that managed the corporate Web site, but for an actual product the
theory was different. We knew having a group in IT working on the
product wouldn't get enough focus on the product because of the
different customer bases.

At Bazaarvoice, we have various internal IT teams, but those are
different from the operations teams that work on the customer facing
product. They do the Exchange and HR systems and whatnot, we do the
AWS/Rackspace/Akamai kind of stuff.

At Towery Publishing, we had a web division and a IT group. They
reported to the same person (me), but they were kept separate.

Different teams make sense. Sometimes they are working with different
technologies, but even when they're not, their focus is
different. Working on shipping product for external customers shares
very little in common with running internal facing services. In fact,
I would ask "why *would* they be the same?" This is a disease - we
group people into teams by what technology they know? That is aligned
with business goals how?

An IT department has a budget out of G&A that is usually figured as a
percent of total. An engineering department has COGS figured into
their product.

Can you come up with any reason why IT and product Web Operations
should be in the same group that stands up to a minute of really
thinking about it? I can't.

Ernest

Ernest Mueller

unread,
Mar 7, 2012, 9:06:35 PM3/7/12
to dev...@googlegroups.com
At 04:39 PM 3/7/2012, Joe Miller wrote:
>Perhaps we just invented a new term -- BizOps or BizIT. An emerging
>set of principles for how the rest of the Business collaborates and
>engages with IT ? =)

Interestingly enough our IT department at Bazaarvoice is called
"BizTech," Business Technologies. Different from TechOps, part of Engineering.

This is a bit of a tempest in a teacup. Corporate IT needs to
collaborate with the business and do DevOpsey things. Web Operations
needs to collaborate with the business and do DevOpsey things. They
are largely doing different things, with different parts of the
business, on different cadences, with difference customers, using
different budgets and usually different technologies. They are not
different in quality or technique, but that doesn't make it useful
for them to be the same.

Ernest

John Vincent

unread,
Mar 7, 2012, 11:12:37 PM3/7/12
to dev...@googlegroups.com
Combining responses with your previous email here

I think the problem is that we're confusing roles, responsibilities,
budget codes and perception.

In my mind all of these groups and teams and initiatives are still
Information Technology or Information Systems. You can slice it and
dice it anyway you want. You can complain and rail against the gods
but it's like the hacker vs. cracker debate. The battle you're trying
to fight is one of perception and semantics - both are ones that are
incredibly difficult.

The reality of corporate structure is that there has to be a a
hierarchy. Things have to be classified for budgets and what not. It's
simply easier to divide departments along ontological lines (or is it
taxonomic? A debate for another day). These guys deal with
"technology". That's IT. These guys deal with "money". That's finance.

Yes, it's unfair in the sense that it doesn't respect the nuances WE
as practitioners recognize. As I've said, a lot of time it's doesn't
make a lick of sense. At one company, the guys doing telco and
provisioning of connectivity to our remote sites were under the
helpdesk manager. This was despite the fact that they also configured
firewalls and CPE. Why were they not part of the Networking team which
actually fell in with the sysadmins?

That's the point I'm really trying to make. The battle should not be
one trying to differentiate yourself by title or responsibility.
Differentiate yourself based on merit.

dwe...@gmail.com

unread,
Mar 7, 2012, 11:56:41 PM3/7/12
to dev...@googlegroups.com
On Wed, 07 Mar 2012, Ernest Mueller wrote:

> Can you come up with any reason why IT and product Web Operations should
> be in the same group that stands up to a minute of really thinking about
> it? I can't.

Who supports the internal employees when they have issues with the web
services? Who supports the webops' workstations? How much would IT
benefit from the metrics collection techniques of webops? How much
would webops gain from the debugging and customer service skills in IT?

It depends on how narrowly you define 'group.' Should they all be in the
same meetings or have the same team lead? Probably not. That's true for
any large team though. I would argue that IT and Ops have complimentary
skill sets and produce better work when they work together.

Doug

Jez Humble

unread,
Mar 8, 2012, 1:36:49 AM3/8/12
to dev...@googlegroups.com, dev...@googlegroups.com
Isn't that called "agile"?

Sent in a hurry from a device without a proper keyboard.

Damon Edwards

unread,
Mar 8, 2012, 4:06:14 AM3/8/12
to dev...@googlegroups.com
This is all gets simple real quick when you ask the question "why?" (the central DevOps question in my opinion).

Why are you employed at your company?

Meaning... What is the purpose of your job? What is the value you produce for your company's customers?

Saying that everyone who works on those computer thingys should be lumped together, expected to follow the exact same operating model, and are identical in every way is bad management in a company of any significant scale. This makes as much sense as lumping a company's corporate facilities management staff and manufacturing staff together because they both spending their days doing physical stuff, keeping the machines running, worrying about power, worrying about stress injuries, etc.

OK, that might be a terrible replying-after-midnight analogy... But hopefully the point I'm trying to make got across.

There are people who use technology to provide services that support the business operations that, in turn, go out and bring in revenue (CRM systems, email, warehouse support systems, logistics systems, laptops, file servers, accounting systems, etc..)

There are those who use technology to directly produce revenue by providing a service directly to paying customers. They ARE the "factory floor"... The direct link that connects the company to someone's wallet. They take a business idea and transform it into a running feature for a customer (yes, that includes dev and ops type activities).

Of course both of these groups can share best practices, use the same cool tools, go to DevOps Days, and collect similar pay checks. From a egalitarian point of view we can all be careful not to say one is more "valuable" than the other. Both are definitely "necessary". Both should be proud of the skill their jobs require.

But let's get real... From the perspective of the business those activities have very different purposes, require a different set of capabilities, have different priorities, and require different management and funding techniques.

Ironically, lumping all things IT together will have the same negative, siloing effects on a business that the dev and ops division has inside a "technology organization"... It's just happening at a more macro level! This time we are the ones inside the silo protecting our own interests and not seeing that we are making the larger system (i.e. the business) more inefficient and ineffective.

Ernest Mueller

unread,
Mar 8, 2012, 7:59:56 AM3/8/12
to dev...@googlegroups.com
At 10:12 PM 3/7/2012, John Vincent wrote:

>I think the problem is that we're confusing roles, responsibilities,
>budget codes and perception.
>
>In my mind all of these groups and teams and initiatives are still
>Information Technology or Information Systems. You can slice it and
>dice it anyway you want. You can complain and rail against the gods
>but it's like the hacker vs. cracker debate. The battle you're trying
>to fight is one of perception and semantics - both are ones that are
>incredibly difficult.

I guess my point is that conceptually, both are "IT", but in
many/most places, no, product focused Web ops is not part of the
otherwise internally focused IT department. You seem to not be
arguing that they *should* be together, which is what I thought you
were saying initially, just that "well you lose that battle so they
*are*," which might happen but isn't true for the shops I've been in.

Ernest

John Vincent

unread,
Mar 8, 2012, 10:24:48 AM3/8/12
to dev...@googlegroups.com

Interesting. I would have expect NI to be one of those shops where it
WAS lumped together.

In the sense of an org chart though (in my experience) everyone has to
report to a C-level at some point. You can break it down to Directors,
Assistant Directors, VP, SVP whatever but in the end you report to the
CIO (or CFO in some cases where there's not been a CIO) and that
person's purview is IT. So maybe it's semantics. Maybe that person
understands the intricacies and you get a better result. I have yet to
work in a shop like that. As Damon said, my experience has been either
your supporting the factory floor or you ARE the factory floor.

A good example is when I was at Community Loans. We had a pretty big
development team. Their job was to develop our lending system. However
they were still of the "supporting the factory floor" variety because
it was the internal application our stores used. All though it was
internal, it was our primary system and it was all web-based. You
could call the sysadmin side of the house "web operations" and it
would have been totally accurate. From the business side of the house
though, it was indistinguishable from the guys who supported the PCs
in the field, the DBAs who managed DB2, the guys writing the code, the
guys managing the servers at our corporate office or the folks
provisioning the VPN at each site. It was ALL "the IT department" and
all the different fiefdoms reported up to the CIO at some point. And
yes, that perception DID cause some problems especially around
budgeting because it was "the IT budget" from the outside.

Here's the thing. Those SAME things existed in other departments.
AR/AP/Collections whatever all fell under finance. The thing is it's
easy for people to understand those subtleties. Funds in vs. Funds out
is still the same base consruct - Money. IT is specialized beyond that
to a degree and it's not immediately apparent to outsiders what those
differences are.

You and I know that managing a web property operationally is different
than managing a VPN. Conceptually some things port but we have
specialization for those things because they are domain experts in
their own right. If DevOps teaches us anything it's that SysAdmins
don't understand development in the same way a Developer does.
Developers don't understand Operations in the same way a SysAdmin
does. Yes, there is overlap but we're not trying to throw out the
distinction between departments/groups of IT specialization per se.

Sorry for the ramble. It made sense when I was typing it ;)

Nathaniel Eliot

unread,
Mar 8, 2012, 10:42:37 AM3/8/12
to dev...@googlegroups.com
On Thu, Mar 8, 2012 at 9:24 AM, John Vincent <lusi...@gmail.com> wrote:
> Interesting. I would have expect NI to be one of those shops where it
> WAS lumped together.

I only contracted at NI for six months, so take this with a grain of salt:

They have a good understanding of how technology duties break down
according to business need, because they are fundamentally a
broad-spectrum technology company. Ernest Mueller's relentless
evangelism for DevOps was what introduced me to it, but his work and
his team was an example of the exception proving the rule: NI was also
one of the most siloed organizations I've worked at. I quickly learned
to juggle disparate tasks, to avoid being deadlocked waiting on
someone from Team X, to get back to me about task Y that I wasn't
allowed to do myself.

Lumping disparate needs together is also bad, but it's in many ways
the opposite of silos. Lumping is far more typical of small,
non-technological companies (Mom & Pop's with a web store and a
network printer) than larger/more technical companies (who tend toward
bureaucracy in the name of fiscal accountability).

--
Nathaniel Eliot
T9 Productions

Ernest Mueller

unread,
Mar 8, 2012, 8:55:30 PM3/8/12
to dev...@googlegroups.com
At 09:42 AM 3/8/2012, Nathaniel Eliot wrote:
>On Thu, Mar 8, 2012 at 9:24 AM, John Vincent <lusi...@gmail.com> wrote:
> > Interesting. I would have expect NI to be one of those shops where it
> > WAS lumped together.
>
>I only contracted at NI for six months, so take this with a grain of salt:
>
>They have a good understanding of how technology duties break down
>according to business need, because they are fundamentally a
>broad-spectrum technology company. Ernest Mueller's relentless
>evangelism for DevOps was what introduced me to it, but his work and
>his team was an example of the exception proving the rule: NI was also
>one of the most siloed organizations I've worked at. I quickly learned
>to juggle disparate tasks, to avoid being deadlocked waiting on
>someone from Team X, to get back to me about task Y that I wasn't
>allowed to do myself.

Well, exactly. So I managed the Web Systems team at NI for many
years. It was in IT, and responsible for our Web site (hundreds of
apps, a pretty big one). The IT group siloed harder and harder over
the years I was there - all divided up by function. UNIX admins,
Windows admins, DBAs, network, groupware. They collaborated less and less.

It made being a Web Ops group very difficult. We were trying to
interface with all the various dev teams divided by business area
(ecomm, ecrm, support, advisors, etc.) as well as all the infra teams
divided by technology. The devs wanted us to interface with the rest
of infrastructure for them so they wouldn't be subjected to another
long lecture about SAN internals. Many a time we said "just give us
our own DBA so we don't have to deal with them!" In the end, it was
the standard story of the insular admin team, with a flavor of mainly
having the mindset of serving the internal customer, so an ERP
problem affecting a dozen people was more important than a Web site
problem costing revenue. Nate was on a similar team spawned to
mirror ours, an internal apps administration team, and they felt the
same pains. Our relationships with the devs and the business were
pretty good, actually better than with the other infra teams,
inspiring my OpsOps post...
http://theagileadmin.com/2010/03/12/before-devops-dont-you-need-opsops/

My experiences on that team were what made the burning need for
DevOps so clear, and when we tuned into the movement at Opscamp 2009
I realized it was what we had been trying to get to. So I have to
admit, our different approach with our SaaS products was somewhat
influenced by my input. They wanted to start a team in R&D to do
SaaS, and I made no bones about saying that it wasn't going to work
unless we integrated dev and ops and had everything integral to our
product's success in the product team's hands. That meant no IT. We
didn't all report up to the same person except the CEO; engineering
goes up a different path than IT/CFO. And by keeping pimp hand
strong, we got success and shipped products.

The devs we had realized the difference. I remember one day, one dev
was trying to switch all of his calls from http to https. He
approached me about it, and I showed him where to change that in our
PIE model and how to load in the cert and we put it in source control
and we were done. Then he called the DBAs to try to get the call
from Oracle ERP to our systems switched. Thus ensued two weeks of
calls and meetings to beg DBAs to listen to us and to make a cronned
curl call https instead of http. When it was done, he turned to me
and said "Hey... I think I get what you're talking about with this
DevOps stuff."

It was my hope that by showing the results we could get with the
DevOps approach on a greenfield team we could then get IT to adopt a
different approach as well. Might happen one day, but I decided to
move on to do DevOps in a larger product focused organization instead.

Ernest

John Vincent

unread,
Mar 8, 2012, 9:11:57 PM3/8/12
to dev...@googlegroups.com

That's pretty much one of the most awesome stories I have. I think it
also clears up some of the confusion in talking past each other.

So I have no problem with project based full stack teams. That
actually makes perfect sense. It's probably a hard sell fiscally if
you say "this person is the DBA for this project ONLY" because you're
essentially saying "there is a one to one mapping of DBAs to
projects". But as it interfaces to the business as a whole I still
stand by my original statements ;)

Nathaniel Eliot

unread,
Mar 9, 2012, 1:02:19 AM3/9/12
to dev...@googlegroups.com
On Thu, Mar 8, 2012 at 8:11 PM, John Vincent <lusi...@gmail.com> wrote:
> So I have no problem with project based full stack teams. That
> actually makes perfect sense. It's probably a hard sell fiscally if
> you say "this person is the DBA for this project ONLY" because you're
> essentially saying "there is a one to one mapping of DBAs to
> projects". But as it interfaces to the business as a whole I still
> stand by my original statements ;)

Full-stack teams work better (in smaller frameworks) with full-stack
developers. We don't have a DBA at Infochimps; that role is largely
shared between Travis and I, with backup from most other technical
employees depending on specific circumstance. This may mean less depth
of knowledge than a 100% DBA brings to the job, but that's implicit in
"less than one DBA per team" anyway. We overcome it with lots of talk,
research, and an architecture that encourages experimentation.

And by full-stack developer, I don't (necessarily) mean someone with a
lot of experience in all areas. A DBA who has no database problems to
solve can write shell scripts, or do something else of use to the
team. What's needed is a willingness to step outside of your
specialty, if you're the most available brain for the job.

Roger Gregory

unread,
Mar 20, 2012, 12:54:46 PM3/20/12
to dev...@googlegroups.com
In our case (mid-sized newspaper), we've outsourced traditional IT completely -- helpdesk and engineering/networking functions are handled by a partner company.

Internal Application support (accounting systems, print publication system, etc) are handled by an internal "applications" team.  Web-facing systems are via a standalone team as well (which handle Internet Operations).

We've specifically distanced ourselves from the more traditional IT support functions (workstations, printers, etc)...

-r
Reply all
Reply to author
Forward
0 new messages