Notes for 2012-10-15: conditional execution, VMs and image builders, packaging, continuous integration and testing tools/services

[Igal Koshevoy]

We had a great meeting last night. Although we had no agenda, we still ended up with three hours of great group discussions. Below are some of the topics we covered. Feel free to add comments, content or corrections.

Conditional execution A.K.A. idempotence

• You should be able to run your configuration management tool (Chef, Puppet, etc) multiple times and it should only make changes when there are changes to be made. So if you reapply the same configuration to a system, it should do absolutely nothing, otherwise there's something wrong and you must fix this.
• Tools like Chef and Puppet describe the state of the system, so if they say that the system should have a certain package installed, the tool is smart enough to only install the package if it's not there.
• These tools provide features to help you avoid redoing work. For example, when using Chef to describe a remote file, you can add a checksum so that it's only downloaded if the file isn't present or doesn't match the checksum of the existing one. Or use the conditional execution "not_if" or "only_if" in a resource (e.g. a file, package, command, etc) to describe shell or Ruby code to check whether the resource should be installed/downloaded/run, etc.

Virtual machines and image builders

• Vagrant "a free and open source project. The vision of the project is to create a tool to transparently manage all the complex parts of modern development within a virtual environment without affecting the everyday workflow of the developer too much."
• Vagrant's VMs can be configured using Chef, Puppet or shell commands. You can embed Chef recipes or Puppet manifests into your project along with Vagrants configuration, or have Vagrant talk to a Chef or Puppet server to configure your VM. A common approach is to ship a project that relies on a base image of an OS with a minimal installation, and then use the configuration management system to install and setup everything else. However, because it can take a while for the configuration management system to download and install everything, it can make sense to snapshot the VM after this process is done and ship these images to users so they can fire up a VM and have it ready quickly, rather than waiting for the software to finish installing. The configuration management would still be run, but would merely be making changes to this cached snapshot, which will usually be faster.
• Various tools for creating VM images:
• VMBuilder is "a Python-based software package for creating VM images of free software GNU/Linux-based operating systems. It's currently being developed by the Ubuntu Virtualization Team [...] It makes building and installing VMs a breeze. The vanilla / trunk release supports Ubuntu on Xen, VirtualBox, VMware, KVM and Amazon EC2."
• VeeWee is "the tool to easily build vagrant base boxes or KVM, VirtualBox and [VMWare] Fusion images".
• BoxGrinder is "a set of projects that help you grind out appliances for multiple virtualization and Cloud providers"

File and package caches

• It's in your interest to have your systems pull files and packages from servers you administer, rather than from other people's machines on the Internet. For example, if your configurations rely on installing Ubuntu packages, Maven JARs, Ruby gems, Puppet modules from the Puppet Forge, or even software from Github, then you're relying on an external system that may go down or change in surprising ways without notice. At the very least, have your system behave sanely if these misbehave, like halting execution rather than trashing a system. As you get more systems, rebuild them more often, or deploy more often, this becomes a bigger concern.
• Pulp "is a Python application for managing software repositories and their associated content, such as packages, errata, and distributions. It can replicate software repositories from a variety of supported sources, such as http/https, file system, ISO, and RHN, to a local on-site repository. It provides mechanisms for systems to gain access to these repositories, providing centralized software installation."

Continuous integration, and testing tools and services

• Continuous Integration "in software engineering [...] implements continuous processes of applying quality control [through] small pieces of effort, applied frequently." In practice, this usually describes some system that runs jobs like tests or builds when it sees commits made to a special source code repository or branch.
• Travis CI: "A [free] hosted continuous integration service for the open source community". They have servers that can watch your source code repositories, run tests when commits are made, and alert you when the tests break. The service is free for open source projects and easy to setup because it provides baked-in support for all major programming languages, databases, etc. They're working on a commercial product for closed source projects. However, it's very much oriented towards testing and their only platform is 64-bit Ubuntu. FYI, the Travis VMs use Vagrant.
• Jenkins: "An extendable open source continuous integration server". You run this server yourself and can configure it in effectively unlimited ways, e.g. have it compile OS packages, add comments to code review boards to note whether tests passed, or have it run tests against pretty much anything.
• Justin Stoller gave us a great overview of some interesting talks he saw recently at the Jenkins User Conference. They also offer a complete list of talks, abstracts and slides.
• Trunk Gating with Gerrit, Jenkins and Zuul: See how the OpenStack project uses Gerrit, Jenkins and Zuul to perform trunk gating and how you can apply the same approach to ensure only high quality code is merged into your project's source control system. This talk will focus particularly on Zuul, a new free software tool that uses the Jenkins API to perform several kinds of project automation tasks, including testing interrelated changes in parallel before merging them. More info on Zuul.
• Rebuilding the Airplane at 10,000m: Implementing Continuous Deployment with Jenkins and Gerritt -- Speed or stability, pick one. We want both, and in this talk, I will discuss the experimentation and transition phase made from slow brittle processes, to much more continuous deployments with Jenkins and Gerritt collaborating to provide faster developer feedback on the quality of their changes. Along the way I will share pitfalls, useful tools and practices to make the transition to continuous deployment easier for others. Slides.
• Beyond Selectors: Using Jenkins to Display Selenium Failures and UX Compliance -- Jenkins is more then a CI tool, it can be used as a complete platform to test your software in as many ways as possible. At Groupon, we have created two unique plugins that dramatically increase the usefulness of Jenkins. First, a plugin which will match failed Selenium screenshots with the JUnit error pages, displaying screenshots on the same page as the error stack trace. Second, we have created a plugin that will compare current screenshots with screenshots from a known quality source and determine if the overall UX of a page is correct. Jenkins allows us to go beyond functional testing, it now allows us to test the experience of a webpage. Slides.
• Sikuli: "is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). Sikuli includes Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API's support. You can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or even an iphone or android application running in a simulator or via VNC." Here's a very cool video of Sikuli in action.
• Selenium "automates browsers. That's it. What you do with that power is entirely up to you. Primarily it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well." There's also a local Selenium Users Group, although lately they've mostly been talking about testing in general.

-igal