Server acceptance unit testing

[John Warburton]

Hi Everyone

We want to unit test our servers with something like serverspec, but we do not have the coding skills in the team to write ruby/rspec, and we want a low barrier to entry for writing tests

 

Our site isn't small with 3.5K servers, 200 modules, 400 manifests covering 3K resources. Each server averages about 650 resources

 

I was wondering what others do for server acceptance testing given a similar set of requirements. Based on my quick reviews on what is out there, Jenkins with perl's Test::Harness looks a good fit, which is a little scary

 

Am I missing something?

 

Thanks

 

John

 

Requirements:

• Use industry standard test harness

• Use industry standard formatter / dashboard (Jenkins?)
• Lightweight, not monolithic. One test = one file

• Command line based

• Independent of central server for initiating checks

• Roll up checks into arbitrary domains such as role, profile, location, environment

• Can be run on production servers

• Check actual state, not puppet results (we already do that)
• Run not just at build, but all the time and alert on deviations through monitoring system
• Some checks run on servers, but some run elsewhere that check other systems (accurate data centre inventory records, console access, DNS reverse address exists, etc)
• Do not want to be restricted by language to write tests in. We have a wide range of programming abilities in team (averaging little or none), and expect most checks to be written in shell, and if they have to be - perl/python. Very rarely ruby

 

 

Reviewed:

• Serverspec
• Ruby / rspec. No chance of widespread adoption

• Beaker
• Ruby / rspec. No chance of widespread adoption
• Does allow shell outs...
• Bigger than a test harness - whole test environment creation. Probably more than what we'd need
• Focus seems to test puppet modules - cannot run in production and test arbitrary states

• xUnit
• Language specific? Difficult to have tests in different languages
• Need to deploy each language specific test runner
• Jenkins can eat xUnit test results

• Perl
• Test::Harness by default exists on every server
• Different languages supported if in different directories
• Jenkins eats TAP test results

[Nan Liu]

On Mon, Jun 16, 2014 at 10:52 PM, John Warburton <jwarb...@gmail.com> wrote:

Hi Everyone

We want to unit test our servers with something like serverspec, but we do not have the coding skills in the team to write ruby/rspec, and we want a low barrier to entry for writing tests

 

Our site isn't small with 3.5K servers, 200 modules, 400 manifests covering 3K resources. Each server averages about 650 resources

 

I was wondering what others do for server acceptance testing given a similar set of requirements. Based on my quick reviews on what is out there, Jenkins with perl's Test::Harness looks a good fit, which is a little scary

 

Am I missing something?

If you are looking for something that's low barrier of entry and shell is your target, have you considered something like bats?
https://github.com/sstephenson/bats

Nan 

[Eric Shamow]

There is Beaker, which is an internal (but open) PL testing framework:

https://github.com/puppetlabs/beaker/wiki

Beaker has an rspec variant, but it also has its own DSL which is much, much simpler.

-Eric

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACqVBqB_kX8VUYwb2BHwaTpfCi%2B%2BFqxH3J4ono67tdU_3ve5Aw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Eric Shamow

Methodologies Lead

Puppet Labs

[John Warburton]

On 17 June 2014 16:13, Nan Liu <nan...@gmail.com> wrote:

If you are looking for something that's low barrier of entry and shell is your target, have you considered something like bats?
https://github.com/sstephenson/bats

Thanks Nan - that escaped me. Will take a look. The best I could find was "Sh Unit 2" which seemed more complicated

John

[John Warburton]

On 17 June 2014 16:17, Eric Shamow <er...@puppetlabs.com> wrote:

There is Beaker, which is an internal (but open) PL testing framework:

https://github.com/puppetlabs/beaker/wiki

Beaker has an rspec variant, but it also has its own DSL which is much, much simpler.

 

Thanks Eric

I've looked at it, and the language is still a barrier

However, we are likely to adopt it in the test pipeline where we have a gap testing new modules after simple syntax checks and catalog compiles

John