Beaker and mock services

[Alex Harvey]

Hi all,

I am investigating whether or not I can use Beaker to do acceptance testing on roles and profiles. 

I've had a look at Liam Bennett's excellent blog posts -

http://tech.opentable.co.uk/blog/2014/09/01/testing-puppet-with-beaker-pt-dot-3-testing-roles/

http://www.slideshare.net/liamjbennett/cfgmgmt2015-testing-with-beaker

I need to handle a situation in my tests where, say, a role that I am testing will apply a base class which will cause the node, for instance, to join a FreeIPA domain.  But I don't want Beaker to actually build a FreeIPA box.  And I don't want my short-lived node to join a real FreeIPA domain.  

I would hope that Beaker could either build Mock Services

https://en.wikipedia.org/wiki/Mock_object

Or better still, tell Beaker to expect the base class to try to apply the FreeIPA class, and just pretend it succeeds.  Just as you can stub out methods in rspec etc.

Has anyone done anything like this before?

Kind regards,

Alex Harvey

[David Schmitt]

Hi,

On a general note, I'd say that acceptance tests should be end-to-end
tests. That would imply, that unless you're actually testing the real
domain joining, what value would you get out of the test?


Cheers, David

[Justin Stoller]

Have you looked into rspec-puppet (http://rspec-puppet.com/)? It should let you do things like apply a class to a node (the role class) and then inspect the catalog generated to see if it includes the base class in question, without having to spin up a new machine (as long as you correctly stub your facts) or worry about a machine actually connecting to a FreeIPA domain.

After you have a strong base of rspec-puppet tests I would use Beaker to ensure that the roles can *actually* spin up a FreeIPA box and join a similarly spun up domain, in a way that can *relatively* quickly smoke test small changes at a time before pushing them to a staging environment.

HTH,

Justin

Kind regards,

Alex Harvey

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5493914c-ecc9-42e4-ad90-4151e0e75fbc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dylan Ratcliffe]

I agree with the other comments, start with RSpec, where you can easily mock things, then move on to acceptance testing. The reason you cannot easily mock things with acceptance testing is that currently Beaker is executing Puppet Apply on a piece of code and then using Serverspec to check over the status of the server afterwards by running commands over SSH. It does not actually have access to the compiled catalog in Ruby like RSpec testing does. As a result it's very hard to tell it to ignore a failure of a certain resource.

You could always comment it out for your acceptance tests, not ideal though.

[Gareth Rushgrove]

You can actually do something like that :)

Beaker is going to run you Puppet code on the node(s) it spins up,
probably using apply manifest like so:

apply_manifest(pp, :catch_failures=>true)

If this sees an error while doing so (because your FreeIP class
doesn't apply) then it will throw an error. However...

If you check the API docs you can see:

http://www.rubydoc.info/github/puppetlabs/beaker/Beaker/DSL/Helpers/PuppetHelpers#apply_manifest_on-instance_method

apply_manifest(pp, :accept_all_exit_codes => true)

This will run your puppet code and not throw an error.

Now, whether this works for you really depends on if any other classes
depend on the FreeIPA class. And you can't easily test for idempotent
runs easily with this.

Another approach if you can change the puppet code is to introduce a
switch in the profile, for instance if a fact exists like MOCK_FREEIPA
you could not include the class. And then drop that fact into your
beaker test environment.

Also see the other points about testing at different levels but you
can absolutely do it in your acceptance tests.

Gareth

> Kind regards,
> Alex Harvey
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/5493914c-ecc9-42e4-ad90-4151e0e75fbc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Gareth Rushgrove
@garethr

devopsweekly.com
morethanseven.net
garethrushgrove.com

[Alex Harvey]

Thanks Gareth.

Both of these solutions are a bit messy but it is good to know of all the options.  One of the big issues I would encounter immediately I used :accept_all_exit_codes is that the FreeIPA code takes several minutes to time out if the server can't be contacted.  Or if I moved all this logic associated with testing into the code, that would work, but it is not really where test logic belongs, and it would rather clutter my base classes.

I feel that there is some space here for a new approach to acceptance testing.  I am not sure that I agree that acceptance tests should be end-to-end tests.  Consider:

• spinning up a FreeIPA system - not to mention all the other external services that a base class might depend upon - takes a long time.  The longer tests take to run, the less likely people will actually run them.
  
• not all developers will have workstations that can run such resource-intensive tests.
• I actually don't want to test that every single role can join the FreeIPA domain - I only want to test that once, and preferably in tests associated with my FreeIPA modules and classes.

Another approach might be to could create a long-lived acceptance test environment that only contains a DNS, FreeIPA, etc, however:

• this would introduce a management burden - sysadmins needed to keep them running, clean up junk related to broken tests etc.
  
• It would also introduce a cost - we would have to pay Amazon to always have these instances running.

I actually think deploying into a real UAT environment is the only real end-to-end test of the code that matters before it goes to production.

Now, if I am testing a web server role, rspec-puppet allows me to do very useful testing on catalogs - but if I am testing a front end web server role, I only want to know that it builds a properly configured Apache.  I suppose, therefore, that I could just test the front end web server profile; although it would be good to also test as much of the integration as is practical.

What if Beaker was extended to provide a framework for provisioning fakes - if it accepted plugins somehow for a mocked up FreeIPA service etc.  Would that be unrealistically complicated?  In the development world, developers must be mocking up these sorts of fakes all the time.  How do they do it - do they just continually reinvent the wheel or are there tools for this?

[David Schmitt]

fakes/mocks in s/w development have a very limited scope: usually only one or two methods with pre-defined return values. If you really do not want to test integration with a certain service, perhaps making that explicit and having a test version of a profile that does not include the FreeIPA support makes sense? It would also ensure that other services (like SSO on the apache ??) do not depend too heavily on FreeIPA.

It reminds me a bit of Aspect Oriented Programming, where cross-cutting concerns are extracted from the code and re-injected by the compiler in a uniform way across the board.


Cheers, D.

[Alex Harvey]

On Thursday, October 22, 2015 at 8:11:46 PM UTC+11, David Schmitt wrote:

fakes/mocks in s/w development have a very limited scope: usually only one or two methods with pre-defined return values. If you really do not want to test integration with a certain service, perhaps making that explicit and having a test version of a profile that does not include the FreeIPA support makes sense? It would also ensure that other services (like SSO on the apache ??) do not depend too heavily on FreeIPA.

It reminds me a bit of Aspect Oriented Programming, where cross-cutting concerns are extracted from the code and re-injected by the compiler in a uniform way across the board.

Actually, I have worked on a number of projects in my time where developers did mock up some reasonably complex fakes for integration testing.

I have been thinking about this more.  At the moment, integration tests are limited because Beaker does not allow us to modify either the source manifest code or the compiled catalogs under test.  We had a flow like:

Beaker --> provisioner (vagrant, vSphere etc) --> Serverspec

Could we instead have something like:

Beaker --> Rspec-puppet (possibly producing modified catalogs) --> provisioner --> Serverspec

We have so much awesome functionality in Rspec-puppet but the big gotcha is that we have no way of testing the catalogs that it produces on real systems.

Could I be onto something here?

[Alex Harvey]

Hi all

What if we added a feature to the Puppet Labs spec helper rake tasks allowing us to selectively merge in modified versions of some files to the fixtures directory, rather than currently only supporting symlinking in entire directories?

I want to use a modified version of one file manifests/base.pp but I want everything else in manifests to be used as is.  Currently, the spec helper allows me via .fixtures.yml to symlink into fixtures the entire directory of manifest code.

If there was a feature that allowed me to make everything in manifests available as-is except for base.pp then I could commit a modified file site_manifests/base.pp in fixtures that removes the class for joining the FreeIPA domain.

Any other ideas?

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/u87-bbzZei8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/de2ab5b7-a9d9-405d-9211-9f1f5b1bb72f%40googlegroups.com.