Meeting notes for 2012/08/20: IPMI, Puppet modules; Samba, Hiera, bricks and mortar modules; Jenkins/Hudson, CI, Gerrit, pre-commit; etc!

[Igal Koshevoy]

We had a another good meeting and about 30 people participated. Below are the highlights of what we discussed. If you have questions, corrections or additions, please feel free to reply to this post.

Corey Osman talked about IPMI:

• He explained the benefits of using the IPMI toolset for managing remote server hardware. Servers with IPMI have a tiny computer inside them, called a BMC (Baseboard Management Controller), which is always running if the server is plugged in, even if the server itself is off. You connect the BMC's special management port to the network and then use special cleint software to talk it, instructing it to do things like reboot the computer, mount a remote DVD as if it was connected directly and boot from it so you can install an operating system, or change the BIOS settings.
• He demonstrated his new "rubyipmi" library, which provides a nice Ruby API for interacting with hardware. 
• He also showed us his new "bmclib" Puppet module for automating these interactions through configuration management. 

Spencer Krum and William Van Hevelingen presented a whirlwind tour of information:

• They showed us the system that they use at Portland State University's Computer Action Team (CAT) to manage Samba, a system for sharing UNIX files with Windows computers. You can see their Puppet module.
• They're using Puppet Hiera to extract all their configuration variables. If you're using Puppet and haven't tried Hiera, you really need to because it simulates the standard way that Chef and other configuration management tools separate configuration variables from resources in a sensible way. The manifests just loop through this configuration data and turn it into Puppet resources, e.g. Hiera reads a YAML file (it's like XML but human-readable) to get a list of file server shares and their parameters, like paths and servers, and turns these into Puppet resources, which it can then apply to the system.
• They also talked about their brick-and-mortar Puppet modules approach, where the "bricks" are generic modules that do things like manage Samba, while "mortar" modules are things that manage their specific Samba implementation by gluing the "brick" modules together.
• They demonstrated their magical "puppet-sync" tool which is a clever Git hook that lets them setup new Puppet environments by just pushing code to special branches. This makes it very easy to create Puppet environments, e.g. for isolated development and testing. You can learn more about this here and here.

Adam Gandelman of Canonical's team for Ubuntu Server (my favorite server OS) talked about testing OpenStrack with their OS:

• OpenStack is an open source Infrastructure-as-a-Service cloud computing platform. It's become wildly popular, which is great, but is also undergoing intense development by a huge number of contributors, which makes it difficult to keep track of and ensure quality. A major challenge is making sure that the code flowing into OpenStack actually works. Adam's written a bunch of tests that setup clean environments on bare metal with Ubuntu MAAS (Metal-as-a-Service) and Ubuntu Juju, and run integration tests that ensure OpenStack installs and is able to run a sample application. The easier way to test this stuff is to use a "post-commit" approach, where tests are run after the code has been merged into the master branch of the Git repository, but unfortunately this is little more than a sanity check. Adam is setting up a "pre-commit" system which runs tests against proposed changes before they're merged and tells those involved in the project whether they passed, so they can better decide whether to merge them in.
• He's using Jenkins, a continuous integration (CI) platform to run the tests. OpenStack developers push code and this is picked up by the Gerrit code review system for the OpenStack project. His Jenkins instance listens to Gerrit using a plugin ("Gerrit Trigger"?) for proposed changes, executes tests against them, and leaves a comment on Gerrit with the human readable results. He's strung together a bunch of steps using the "Parameterized Build" plugin so he can isolate failures.
• A gentleman from Yahoo! recommended the "Build Pipeline" for visualizing interrelated Jenkins jobs.
• Another recommendation was to replace the multiple jobs with one, because they're all ultimately producing a single result, a Gerrit comment about the tests.
• Another recommendation was to use a high level language (e.g. Ruby, Python, etc) or a built tool (e.g. make, rake, ant) so that Jenkins just installs that program and runs it to perform all the work needed to prepare the system, run the tests, and generate the human-readable report. This is in contrast to having multiple Jeninks jobs and writing complicated shell scripts in a Jenkins HTML form textarea to do the work.

-igal