Meeting notes 8/7/09

[Jen Browne]

VeloSLC 8/7/09

Friday, August 07, 2009
10:19 AM

Problems
· Configuration drift
· Wall of confusion
§ Silo'd information seems intentional at times
□ knowledge hoarding
□ Basic understanding of each others' jobs
□ Exposing fundamentals across the wall can reduce translation
issues
§ Issue augmented by Tribes
§ Create boundary objects
□ Puppet
□ Applications delivered from dev team with config management piece
· Change is the root cause of most outages
§ Build a culture of:
□ Learning
□ Respect
□ Opportunity
□ Engagement
□ Investment
§ Get the wrong people off the bus
§ Get yourself off the bus
· Hero culture

Techniques
· Version control
§ Git is best
§ Not the same as change control, which is process
· Deployment
§ Can be somewhat decoupled from version control
§ Seeing what's currently on servers is essential
§ Tag every release with What, Why, When and Who
· Correlation
§ If a unit test fails, it should tell you with a block of the
failure
§ Build a dashboard that shows you a visual representation of
metrics
□ Metrics all on same timeline
□ Cactus, etc.
□ Tiered by services
□ Shared metrics with the business is essential
· Shared version control
§ Same repository for both
□ Ops team
□ Dev team
§ Always ship trunk (master)
· Conditional branches in code
§ Allows you to turn features off
§ Don't use for different environments
□ Can make things not work in different environments
□ Different config files per environment could solve problem
§ Dark launches
□ Features are on but not visible in UI
® Runs queries but throws them away
□ Could be enabled for subset of users
□ Can give indication of impact on infrastructure
· Automate everything
§ Start with what you have
§ Config management is auditing configs against spec
§ Automate incrementally
§ At the push of a button, deploy new servers
§ 2 stages
□ Bare metal to running services (infrastructure)
□ ~~blurred area~~ (particularly when you use both Puppet and
ControlTier)
□ Ongoing deployment (applications)
§ Disaster recovery becomes a snap
· Continuous integration

Culture
· Pair Dev & Ops
§ Firefighting is one scenario
§ Similar to paired programming
§ Can be for specific projects
· Create a culture of learning and respect
· "There is only us."
· Do experiments
§ Fail fast - fail safe
§ Test-driven development
□ Skill that takes time to develop
□ Can be slow and frustrating at first
□ TATFT (google search)
§ Spike
□ Write a solution that works
□ Throw it away
□ Write using TDD
§ Some would advocate this:
□ Write monitoring code that fails first
□ Rewrite until it's green
□ Something to think about - might work
§ Mitigates failures

How to start:
· Metrics
· Version control
· Homogenize systems
· Build out deployment
· Monitoring
· Dashboard

Next meeting:
· September 4, 2009
· 10-noon
· Place tbd
· Meet 10-noon on the first Friday of every month