On Wed, Dec 7, 2011 at 2:28 PM, peco <pecomeis...@gmail.com> wrote:
> At the heart of successful devOps is good team communication and
> collaboration. It becomes especially important as the dev & ops teams
> become more distributed and grow in count.
> What gems are people using for effective dev&ops collab? I am
> especially interested in the troubleshooting, issue tracking,
> production issue solving aspect of things. A ticketing system is not
> quite the answer although it would certainly participate in the flow.
> Email is not quite it either.
> Troubleshooting a big system involving several teams requires crisp
> clear and timely echange of detailed tech info and analysis, is there
> anything good out there?

> Cheers

On Wednesday, December 7, 2011 at 2:28 PM, peco wrote:
> At the heart of successful devOps is good team communication and
> collaboration. It becomes especially important as the dev & ops teams
> become more distributed and grow in count.
> What gems are people using for effective dev&ops collab? I am
> especially interested in the troubleshooting, issue tracking,
> production issue solving aspect of things. A ticketing system is not
> quite the answer although it would certainly participate in the flow.
> Email is not quite it either.
> Troubleshooting a big system involving several teams requires crisp
> clear and timely echange of detailed tech info and analysis, is there
> anything good out there?

> Cheers

> I will have to +1 to most of the Atlassian products in my opinion.  I worked with all their products and have to say that they are well worth the price you have to pay for them and their prices are pretty reasonable.  They have tight integration with each component of their product line and works well across the whole organization to connect many aspects of the development lifecycle.  I worked with many organizations in the past and present ranging from large fortune 500s to small startups and Atlassian has been instrumental to the success of all of our projects.  I do not work for Atlassian if someone wants to ask. :-)

> On Wednesday, December 7, 2011 at 2:28 PM, peco wrote:

>> At the heart of successful devOps is good team communication and
>> collaboration. It becomes especially important as the dev & ops teams
>> become more distributed and grow in count.
>> What gems are people using for effective dev&ops collab? I am
>> especially interested in the troubleshooting, issue tracking,
>> production issue solving aspect of things. A ticketing system is not
>> quite the answer although it would certainly participate in the flow.
>> Email is not quite it either.
>> Troubleshooting a big system involving several teams requires crisp
>> clear and timely echange of detailed tech info and analysis, is there
>> anything good out there?

>> Cheers

On Wed, Dec 7, 2011 at 3:12 PM, Adam Rosien <a...@rosien.net> wrote:
> I think chat/IRC/Jabber is the best tool for what you're describing. You
> can archive it, have bots perform tasks or relate status, and separate
> rooms for various groups are all great features.

> .. Adam

> On Wed, Dec 7, 2011 at 2:28 PM, peco <pecomeis...@gmail.com> wrote:

>> At the heart of successful devOps is good team communication and
>> collaboration. It becomes especially important as the dev & ops teams
>> become more distributed and grow in count.
>> What gems are people using for effective dev&ops collab? I am
>> especially interested in the troubleshooting, issue tracking,
>> production issue solving aspect of things. A ticketing system is not
>> quite the answer although it would certainly participate in the flow.
>> Email is not quite it either.
>> Troubleshooting a big system involving several teams requires crisp
>> clear and timely echange of detailed tech info and analysis, is there
>> anything good out there?

>> Cheers

> Hmm, I don't think DevOps collaboration is just for super emergent issues,
> so let me answer with a longer list.

>    - We use ZenDesk as a service desk for request management. It has
>    problem/incident functionality too but it's not really good for working a
>    long term issue with multiple participants because is just a big threaded
>    discussion, we end up doing problem management on a Confluence wiki page
>    that may have subsidiary bug reports etc.
>    - We use Campfire for chat and also log system changes from some
>    automated sources into it.
>    - We use Confluence wiki pages for release management.
>    - We use a customized version of HP's PPM for bug tracking (dev and
>    ops both)
>    - We use Perforce for source control (dev and ops both)
>    - We use TFS with telerik's TeamPulse addon (did use JIRA+Greenhopper
>    but had whole-department level change) for task cards and burndown (dev and
>    ops both)
>    - We use WebEx for phone bridges (too often, in an emergent situation
>    people don't type into chat enough, and it allows you to say "I'M TALKING
>    TO  YOU!"
>    - We have a mature and custom test procedure/test execution repository
>    from our product side

> I'm not really too happy with our suite of process tools overall.  I don't
> want some big ol' nasty suite, but I'd like something more integrated that
> could do basic change management (request and tracking), problem/incident
> management, and request management all in one without having to just make
> one off wiki pages.

> Specific shortcomings:
> 1. Change logging to Campfire is good for communication but probably isn't
> a real auditable change log. And begs the question of change requests.
> We're not all 3l33t and Facebooky about "make a code change and push to
> production that day," we have ISO processes and whatnot we're working with.
> 2. Tasks that are planned in an iteration work well in TFS, but other
> unplanned tasks end up going into email and being dropped by both devs and
> ops. Considering having both devs and ops use ZenDesk to track "random
> other tasks" but a more GTD system would be nice.
> 3. Where's something that does problem/incident well at all?  Haven't seen
> it.
> 4. Integration.  Project management to tasks, problems/incidents to bugs,
> anything to wiki pages - it's a mess.

> For emergent issues that get large and go longer than an immediate fix it
> pretty much goes: (using ICS for roles)  alert or report -> ticket in
> ZenDesk -> chat with anyone who happens to be online in Campfire -> trying
> to pull others in via email and phone tree -> Mix of Campfire and WebEx ->
> emails that get merged into a wiki doc -> specific bug reports if needed or
> ops intervention logged in Campfire -> fixes checked into Perforce ->
> change requests to release manager who also uses a wiki doc -> moves to
> dev/test requested in ZenDesk (or just in chat if the issue is an ongoing
> sev1 kind of thing) -> test executions -> release coordinated in Campfire.

> I like the look of Datadog's "facebook for operations" product for collab
> too.

> Ernest
> ______________________
> UN-altered REPRODUCTION and DISSEMINATION of
> this IMPORTANT information is ENCOURAGED.

> [image: Inactive hide details for Will Lowe ---12/07/2011 05:35:53 PM---+1
> on Jira and IRC/Campfire/IM. But opening and closing Jira t]Will Lowe
> ---12/07/2011 05:35:53 PM---+1 on Jira and IRC/Campfire/IM.  But opening
> and closing Jira tickets is too slow during a real fire

> From: Will Lowe <will.l...@gmail.com>
> To: devops-toolchain@googlegroups.com
> Date: 12/07/2011 05:35 PM
> Subject: Re: Tools for DevOps collaboration
> Sent by: devops-toolchain@googlegroups.com
> ------------------------------

> +1 on Jira and IRC/Campfire/IM.  But opening and closing Jira tickets is
> too slow during a real firedrill.

> For those we've found a shared Google Spreadsheet to be pretty handy.  A
> bunch of rows like this:

> | Issue Summary | Architecture/Product Component | Fix Owner | Status |
> Notes | ...

> ... goes a long way towards keeping the entire team focused during a big
> problem.

> Will

> On Wed, Dec 7, 2011 at 3:12 PM, Adam Rosien <*a...@rosien.net*<a...@rosien.net>>
> wrote:

>    I think chat/IRC/Jabber is the best tool for what you're describing.
>    You can archive it, have bots perform tasks or relate status, and separate
>    rooms for various groups are all great features.

>    .. Adam

>    On Wed, Dec 7, 2011 at 2:28 PM, peco <*pecomeis...@gmail.com*<pecomeis...@gmail.com>>
>    wrote:
>       At the heart of successful devOps is good team communication and
>       collaboration. It becomes especially important as the dev & ops
>       teams
>       become more distributed and grow in count.
>       What gems are people using for effective dev&ops collab? I am
>       especially interested in the troubleshooting, issue tracking,
>       production issue solving aspect of things. A ticketing system is not
>       quite the answer although it would certainly participate in the
>       flow.
>       Email is not quite it either.
>       Troubleshooting a big system involving several teams requires crisp
>       clear and timely echange of detailed tech info and analysis, is
>       there
>       anything good out there?

>       Cheers

On Wednesday, December 7, 2011 at 3:32 PM, Scott Smith wrote:
> They all have their pros and cons. I choose to judge based on two things: API quality and the number of clicks required to manipulate a ticket.

> Right now github issues is winning my vote, hands down. Notifications suck and labels are super basic, but the web interface is concise when you need to use it. Can't say the same about anything else out there, really.

> Afaik Jira still doesn't have a REST API soooo...

> On Dec 7, 2011, at 6:15 PM, Stan Chan <stanc...@gmail.com (mailto:stanc...@gmail.com)> wrote:

> > I will have to +1 to most of the Atlassian products in my opinion.  I worked with all their products and have to say that they are well worth the price you have to pay for them and their prices are pretty reasonable.  They have tight integration with each component of their product line and works well across the whole organization to connect many aspects of the development lifecycle.  I worked with many organizations in the past and present ranging from large fortune 500s to small startups and Atlassian has been instrumental to the success of all of our projects.  I do not work for Atlassian if someone wants to ask. :-)

> > On Wednesday, December 7, 2011 at 2:28 PM, peco wrote:

> > > At the heart of successful devOps is good team communication and
> > > collaboration. It becomes especially important as the dev & ops teams
> > > become more distributed and grow in count.
> > > What gems are people using for effective dev&ops collab? I am
> > > especially interested in the troubleshooting, issue tracking,
> > > production issue solving aspect of things. A ticketing system is not
> > > quite the answer although it would certainly participate in the flow.
> > > Email is not quite it either.
> > > Troubleshooting a big system involving several teams requires crisp
> > > clear and timely echange of detailed tech info and analysis, is there
> > > anything good out there?

> > > Cheers

On Wed, Dec 7, 2011 at 15:12, Adam Rosien <a...@rosien.net> wrote:
> I think chat/IRC/Jabber is the best tool for what you're describing. You can
> archive it, have bots perform tasks or relate status, and separate rooms for
> various groups are all great features.

> .. Adam

> On Wed, Dec 7, 2011 at 2:28 PM, peco <pecomeis...@gmail.com> wrote:

>> At the heart of successful devOps is good team communication and
>> collaboration. It becomes especially important as the dev & ops teams
>> become more distributed and grow in count.
>> What gems are people using for effective dev&ops collab? I am
>> especially interested in the troubleshooting, issue tracking,
>> production issue solving aspect of things. A ticketing system is not
>> quite the answer although it would certainly participate in the flow.
>> Email is not quite it either.
>> Troubleshooting a big system involving several teams requires crisp
>> clear and timely echange of detailed tech info and analysis, is there
>> anything good out there?

>> Cheers

> For those we've found a shared Google Spreadsheet to be pretty handy.  A bunch of rows like this:

> | Issue Summary | Architecture/Product Component | Fix Owner | Status | Notes | ...

> ... goes a long way towards keeping the entire team focused during a big problem.

> Will

On Thu, Dec 8, 2011 at 4:42 AM, Adam Rosien <a...@rosien.net> wrote:
> I think chat/IRC/Jabber is the best tool for what you're describing. You can
> archive it, have bots perform tasks or relate status, and separate rooms for
> various groups are all great features.

> What gems are people using for effective dev&ops collar?

> great topic Peco, we've just started trying to improve our comms/stack so I'm glad to see this thread popping up.

> On 7 Dec 2011, at 22:28, peco wrote:

>> What gems are people using for effective dev&ops collar?

> spoken word. Seriously, I appreciate the importance of good tools, but nothing will replace the clarity of the message and emotions that you get in a face to face meeting. I currently manage a distributed team so we don't get as much face to face time, but a couple of us are on site and most times we'll just get a room for 10 mins and talk things over. This includes discussing blockers, new projects and so forth.

> Other than that our stack looks like:
> * Github pull request to share code (say when we add monitoring code to their apps or modify config related stuff)
> * Github wiki pages for project planning, postmortems/restrospectives and deployments documents (I have templates to fill out to simplify the job)
> * campfire for chat (mostly because it's easier on some of the non eng folks)
> * rally for scrum and kanban, altho I would strongly not recommend it, but I would recommend something that allows you to do kanban for ops and scrum for dev. I prefer this to more traditional BTSs simply because the flow is generally simpler, especially if the product is well done there's less overhead than in many ticketing systems (and btw you really want something supporting swim lanes)
> * skype/something to use voice. This is the next best thing to face to face so I recommend you try and push for it and I say push because ime engineers will gravitate toward text based comms whenever possible.
> * email for some comms, but we try to avoid it as it generally takes longer, cause more friction being devoid of emotion and falls through the crack as ops tend to get swamped with cron emails and that kind of stuff

> I'd like to conclude by saying that in my experience the thing that went the farthest in improving our comms was creating process around it and communicating (no joke) the importance of it. I honestly doubt there will ever be a tool that encompasses all the comms needs in a devops team so the best chance is to let people know how important it is for them to take initiative and talk to people.

> hope this helps,

> Spike
> --
> http://www.spikelab.org/
> http://twitter.com/spikelab
> http://uk.linkedin.com/in/spikemorelli

> Just gonna address what you said about cron mail. If you're getting email cron output, either something IS wrong or you are DOING something wrong.

On Saturday, December 10, 2011 at 12:05 PM, Spike Morelli wrote:
> Fair point, I should have been clearer (poor communication on my side heh ). When I said "cron email" I meant a largish volume of emails from automated systems. I appreciate that if the message is not important it should not have been sent in the first place and in that sense you could say "you're doing something wrong", but in my experience this is easier in theory than in practice. What I've frequently seen is stuff configured to report lower impact events via email, nagios disk full 80% or log exceptions from the dev environment for example, and when you have a largish pool of machines that will generate enough emails to make ops, and to a less extent devs, less thrilled by email conversations.