Meeting notes for January 17, 2011: "Effectively Organizing Devops"

[Igal Koshevoy]

We had an outstanding meeting, I'm grateful to everyone that came out
on the holiday to participate. We had nearly three-hours of wonderful
discussion covering many experiences, challenges and solutions. I got
a lot out of this and hope the rest of you did as well.

I incorporated many of your additions and clarifications into my
slides, although I'm afraid that short of writing a book, it's hard to
capture the depth of discussion that each bullet point incited:
http://pragmaticraft.com/files/effectively_organizing_devops.pdf

Anyway, I hope that the presentation will also give you some
inspiration for other presentations, lightning talks and discussions
that you can volunteer to do at future meetings. *hint* :]

See you next month!

-igal

[Justin]

I am inspired by the spirit of this group and I want to contribute
what I can. Here are my notes from the meeting. I hope they help.

What is dev ops.....
Extending agile metaphor beyond development.
Instead of using the bucket brigade build a better fire engine.
Automating away operations problems.
Bringing operations to the development level of the organization.

Someplace like Microsoft there is a wall between the systems guys and
development guys.

Configuration management?

Past attempts. Why are things the way they are?
Never assume. "I have seen this and this, therefore I understand."

Advertise your ignorance.

Reign in as many people as possible. People generally don't volunteer
information, you need to ask.

"bucket brigade problem"

Everyone has a opinion about something they feel they understand.

Selling dev ops is tougher then anything.
Changing how an organization works means people will rise and
fall.
All might not necessarily be happy.

Advertising the presence of technical debt.
Not as well advertised.
Def. We can fix this right now but it is not a robust or clean
solution.
Your building a rats nest. Usually a temporary fix. More work
then they want to pay right now.
Generally something you will have to fix later.
Technical debt for operations-Not using any kind a
configuration management at all.
There isn't much credit or praise for solving the problem long
term.
Figuring out what they can do to be a star that isn't putting
out fire
Many times they don't know any better.
They want to be a hero, let them be a hero someplace useful.

Rockstar(cowboy, trailblazers) programmers are hard to work with.
This produces bad culture of undocumented code. Black box
apps.
Can lead to unmaintainable pace of change.

How to deal with the cultural difference between developers and IT?
IT people are Nazis. Developers want to kill my servers.
"We create apps Vs we run servers", what is the goal? We create
and deliver services.
A lot of this is self perpetuating. At the very least keep the
other side updated goes a long way.
If operations has to say no include a reason why. Proposing
alternatives open communications.
Large companies groups may not have the same goals.
Can be very much context driven.

Put a face on the other side and start a conversation. It is a
people thing.

How to distribute credit?
Make sure people are working on teams. Spread the credit evenly.

Quality assurance is a separate team altogether from development or
operations
Seek automation at every level.
Safety, efficiency, repeatability, training, foundation for
the future.
documentation- manifests describe the system.
If the ISP goes away you can redeploy to a new ISP with
minimal pain.
People can contribute quickly. No (little) black magic.

What is a tool worth where automation is not possible?
Open office skills is not a secretarial skill you can sell.
Stakeholder buy in?

Setup puppet/chef the first time.
Specialized shell scripts cannot scale well. API can.
Incremental value is much less with shell scripts. They can become
brittle code.

Start small. Work your way up.
Using native packaging format for whatever your system is.

Giant hairy shell scripts.
How much does it cost to get rid of it?
Delivering tangible value in short iterations
Compartmentalize

Infrastructure as code
http://www.youtube.com/watch?v=LKENuz-DKTg&feature=youtube_gdata_player

Consult a DBS sometimes but they live in a different land. Therefore
they may be using completely different tools.

Dev test product branches are used in version control.

Sandbox development. Checkout manifest onto a virtualized machine.
Blowup from there.

Puppet for development setup/stack.

Modularize- break things up.
Monitoring becomes important. Hey the system works.

Change management
Weekly meetings?
Ad hoc meetings with just involved parties.
Just let is flow downhill. Asynchronous workflow, emails, tickets,
etc.

Make it easy to report an issue
Keep issue tracker healthy. Can lead to multiple submits of same
issue.
Evaluate quickly to determine urgency and have way to expedite urgent
work.
Rockstars can do well here.
Schedule work and resource.
Make sure affected parties are involved.
Sure iterations with quick feedback from frontline.
* break into smaller steps
Discuss and plan mitigation.
Review and trial, return for improvements.
Apply work and monitor results

People that like databases will scream with hate towards ORM. But the
payoff is great with ORMs because they provide for much greater
developer productivity.

What is useful? The minimum viable product for the next step.

Rubrics cube as you get closer to solving it looks further and further
from solved.
-Masters of the rubric cube can see the patterns of solutions

Don't optimize and scale until later because you are guessing at that
point.
Some things may be obvious. But wait until you get scale that is
hard to think about.

Tech changes
Automated testing
Automated provisioning
Automated setup
Automated maintenance
Automated deploys
Automated exception reporting...
Automated

Stop blame games. "The buck stops here" (as in bucking off
responsibility).
This does not mean you should not report problems.
Re-evaluate workflow.