Train Simulator Laserjet

[Ling Baus]

FurthermoreI believe if Gary did DevOps for LaserJet firmware, you can do it for anything. (See how Darren Hague applied DevOps to SAP here, and how John Kordyback did it for a mainframe COBOL application here).

Their problem was that the firmware group was having tremendous difficulty keeping up with the demand for new, innovative features, despite having somewhere between 400 and 800 developers working across the globe, supporting 10MM+ lines of code.

They were completing two software releases per year, with the majority of their time being spent porting their code to new products. Gary estimated that only 5% of their time was spent creating or supporting new features.

I think this is a profound observation. In other words, no one can learn anything if everyone is getting sufficiently slow feedback (e.g., 6 weeks). Learning can only occur if we can see the cause-effect linkage between the work we do and the outcomes.

This created many problems, including an ever-increasing number of branches that had to be actively maintained (as well as branches of branches), each creating a unique build, requiring separate testing.

They eliminated separate branches for all the products, putting all LaserJet models into trunk. Instead of using branches and compile-time #ifdefs, printer capabilities are established at run-time, using an XML configuration file.

One time, they had a broken build for 5 days, which created enormous problems, because no developers could commit code for new features. Why? It turns out that some new tests that raised the quality bar were causing the test suite to fail.

Gary Gruver: To me DevOps in the Enterprise is all about addressing the biggest weakness of most Agile implementations in the Enterprise. All to often I see Enterprise implementations of Agile that focus on scrum at the team level and completely ignore the Enterprise level definition of done. They talk about how many scrum teams they have and how well they are doing scrum. What they completely miss is one of the key values of Agile, which is always keeping the code base stable and close to releasable. To me DevOps in the Enterprise is all about driving that discipline into large organizations with different groups and applications that have to work together. At its core it is about defining and driving a more robust Enterprise level definition of done.

First is what I would call the Google model of taking developers into production. In this situation, the architecture is fairly clean and the management chain has very clear understanding of how hard it is to deliver and run supportable Enterprise level Software. In this model the management team has declared that the development team will own operational support of the new application for at least the first 9 months in production. After that time they can set up a meeting with operations to see if the team is willing to take over on-going support. If not the development team is responsible for operations. This creates the type of closed loop learning that will really get you rethinking development shortcuts and your definition of done. It makes sure that the development team is considering and prioritizing all the operational aspects of the product.

The second approach that is probably more practical in most organizations is working to drive an Enterprise operational like environment back upstream as close to the developers as possible. This approach forces the resolution of Operational and Enterprise level issues early and often. This can be done through Enterprise level Continuous Integration, Continuous Delivery and/or Continuous Deployment. In my mind, Enterprise level Continuous Integration is more related to what we did at HP with over 400 developers committing code on an ongoing basis that had to work across multiple products at the same time. What I have learned since is that when you try to implement a similar large scale operational feedback process for Enterprise level software you really start crossing the line into Continuous Delivery where the deployment aspects become much more important. Creating an Enterprise level DevOps environment upstream as close to the developers as possible really requires investing in creating all the Continuous Delivery capabilities that Jez Humble and David Farley documented in their excellent book. Once you have all those things in place developers realize real-time what it takes to have a clear definition of done in an Operational like environment. The final step in my mind ( it is subtle) is going all the way to Continuous Deployment where you get the additional advantages they describe of small batch sizes moving into production. The reason I make this subtle distinction is that some customers and businesses feel it is more important to batch up deployments into less frequent releases to provide more consistency for their customers. The key though is to use Continuous Delivery techniques to reach the point where it is easy to release more frequently than the business wants and remove the delivery of Software as a constraint for the business.

At HP the DevOps challenge was integrating code from over 400 developers around the world and making sure it still worked on all the different printers, copiers, and scanners under development and in the field. The operational piece was fairly straight forward compared to Enterprise Software. You just FTP the new code to the printer and made sure it worked. The challenge was making sure it worked and would stay working across that many different devices. Ideally from a DevOps perspective we would have wanted every check-in automatically tested against a real printer for every different device supported. Given that we had over 15,000 hours of tests that we were running daily this would have cost way too much and required turning every tree in Idaho into paper for testing. Therefore, we created an extensive deployment pipeline that heavily depended on simulator testing on VMs, and emulator testing that included our custom HW to create real-time operational environment feedback to our developers. Since the simulator and emulator testing were much cheaper and more frequent we heavily relied on them for our DevOps type environment. That said anytime we found a failure on real hardware that did not show up earlier we worked to improve our emulator and simulator testing capabilities and coverage.

My biggest learning in moving over to Enterprise Software was that the deployment process was much more complex and less reliable than the FTP to a printer I had grown to know and love. It is so difficult to get an entire website up and working with everything operationally correct that a lot of teams start depending on dedicated environments or defining done based on their subcomponent of the website. To me this is directly opposite of what definition of done should be in an Agile Enterprise. The DevOps perspective provides the discipline for integrating the entire Enterprise system that must work together in production as early and often as possible. The Enterprise system should be built up in a structured and logical manner ( ). Without it you are just kidding yourself that your Enterprise is Agile no matter how many scrum teams you have going.

If you are focused on improving productivity in a large Enterprise it is important to remember that dedicated environments and branches are evil. They just create way too much overhead and delay the feedback that comes from integrating early and often. Additionally this feedback needs to come from as operational like an environment as soon as possible. DevOps in the Enterprise needs to be real but as the story shows seeing is believing.

3a8082e126