BDD Testing For Ansible

[Sharan]

Hey all. Recently, I started using Ansible in one of my projects. And I should say that, Ansible is easy to understand, get a firm grip and get things rolling up easily. One thing that slowed me down was that, after I write each and every task, I ensure that the task is doing its job in the right way by going to the remote machine and verifying manually. I don't know if that is a newbie's worry of not sure if that task is written rightly! But, I felt that some kind of a test (for example TTD / BDD style) would ensure that the playbook is helping me make my tests go from red to green. To make it clear, I am not worried that the ansible modules are erroneous. I know they are tested thoroughly before they get into the main branch. My botheration is about whether the things I put together as a task / play does things correctly.

After doing some research, I also found that writing tests for Infrastructure as Code is not a moot thing because even this is code. But is there any BDD Testing framework that Ansible can be coupled with? Or even in the first place, is it required? Am I doing a wrong practice of checking the remote machine manually after running each task? 

Any suggestions on the following are welcomed!

1. Best practice in developing infrastructure as code

2. How to test if the scripts are doing what they are intended to do

3. Any BDD framework available for the above purpose

Thanks in advance.

[Ivan S. Freitas]

Hi Sharan,

On Fri, Apr 10, 2015 at 3:00 PM, Sharan <sharan.m...@gmail.com> wrote:
> Hey all. Recently, I started using Ansible in one of my projects. And I
> should say that, Ansible is easy to understand, get a firm grip and get
> things rolling up easily. One thing that slowed me down was that, after I
> write each and every task, I ensure that the task is doing its job in the
> right way by going to the remote machine and verifying manually. I don't
> know if that is a newbie's worry of not sure if that task is written
> rightly! But, I felt that some kind of a test (for example TTD / BDD style)
> would ensure that the playbook is helping me make my tests go from red to
> green. To make it clear, I am not worried that the ansible modules are
> erroneous. I know they are tested thoroughly before they get into the main
> branch. My botheration is about whether the things I put together as a task
> / play does things correctly.

I'm on a similar situation. Even though I consider most of my
infrastructure's ansible description stable, I also looked for
alternatives to implement testing in the infrastructure itself, not
only to test the correctness of what I have done but also for (in the
near future, I hope) integrate it in a "continuous delivery" approach
to the infrastructure phases of deployment.

However I don't bother much with testing things of ansible itself, as
I consider almost all of the actions' behavior to be fair stable and
predictable. For example, usually what is wrong is not the
plabook/roles/tasks involved in configuring the firewall, but the
error lies in my iptables rules. And there is also the parts I cannot
control with ansible for the moment. Thus, instead of focusing on
testing each little piece itself, I choose to consider the
infrastructure as a black box and test the behavior of each component.

> After doing some research, I also found that writing tests for
> Infrastructure as Code is not a moot thing because even this is code. But is
> there any BDD Testing framework that Ansible can be coupled with? Or even in
> the first place, is it required? Am I doing a wrong practice of checking the
> remote machine manually after running each task?

I've seen two approaches: serverspec and BDD (cucumber, more
specifically). The former required ruby installed remotely (AFAIR), so
it was scraped right away (we don't use ruby at all at work). On the
BDD approach, I started using behave [1], since I can, to some extent,
integrate it with the developing api of ansible. Right now I'm
working on a test suite of basic behavior (ssh reachability, checking
if the pages are accessible and returning the correct http code, and
so on), and I have integrated it with ansible for the inventory part
(getting the ip/dns addresses directly from ansible's inventory) and
with other python libraries (paramiko, requests). It's small but
useful for now, but I'm also interested in other opions/experiences
about testing infrastructure, specially ones more closely integrated
with ansible.

[1] http://pythonhosted.org/behave/index.html

--
Ivan Sichmann Freitas

[Christian A DeKonink]

I use serverspec for TDD for my Infrastructure. You don't need to install anything on the client. All of my clients run cents 6 minimal install. I run server spec from an ops server, on that server I installed ruby.

First I create all of the tests, then I create the plays to make the tests pass. Serverspec is very easy to use.

I also use the following enhancements so I can use my browser to easily see which tests passed/failed

https://github.com/vincentbernat/serverspec-example

This show me results formatted into red/green squares to quickly see where each test passed or failed.