beaker ec2 example, should this work?

[Brett Swift]

I've seen how the puppetdb module uses ec2 to execute beaker tests.  I've tried setting this up as well and am getting some errors.

Is there a working example of using the different hypervisors?

I see this: https://github.com/puppetlabs/beaker/wiki/Creating-A-Test-Environment#ec2-support   but the documentation doesn't suggest it's a polished feature :)

It might be helpful to document a project that has an example of all the hypervisors, as each one will have different required / optional parameters in the nodeset configuration file. 

Indented below is the error when running  `beaker --hosts spec/acceptance/nodesets/ec2-west-el6-64mda-el6-64a.cfg`  - I pulled that example from the puppetdb project,  and as per the documentation have a .fog file set up..  I was hoping to get an authentication or configuration error on first run. 

Beaker::Hypervisor, found some ec2 boxes to create

Failed: errored in CLI.provision

#<TypeError: no implicit conversion of Symbol into Integer>

/Projects/live_modules/puppet-shawlib/.bundle/ruby/2.1.0/gems/beaker-1.19.1/bin/beaker:6

/Users/bswift/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15

/Users/bswift/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15 

I don't have much to go on there.. 

[Ken Barber]

The exception seems a little truncated, is that it? BTW Instead of
annotating what has happened in a paragraph, just provide the full
example in your shell in a gist or something, it says the message much
more simply :-). Sort of like this:
https://gist.github.com/kbarber/97f45eba0f922497901a. Or better yet,
if you can create a repository with the basics of the above so I can
take a look that would be much much easier.

I must warn you, the aws_sdk.rb code was written in a massive hurry,
and thus has very little error protection so if everything isn't spot
on, it might break.

ken.

[Ken Barber]

FWIW also - I wouldn't test the state of the puppetdb module testing
pieces, we've been waiting for our QA & Module teams to automate those
tests for a about a year now :-(. But honestly, they haven't been ran
for quite some time - so I'd be dubious.

The PuppetDB source code itself (ie. puppetdb server, not the module)
however does use the AWS EC2 test stuff, and thats a better example to
use.

ken.

[brett...@gmail.com]

I was just testing the host config file from puppetdb coupled with the documentation on the beaker documentation. I was actually going to omit the error message. That's actually all of it except for the json output of the compiled beaker configs. I can send the full output in the morning.

It looks like the Google Compute Engine docs are more complete... It doesn't matter where it runs. Mostly looking for a free tier cloud to get started with. I'm not sure aws micro would even be big enough anyways. But it'd be cool to get it working.

Thanks



Brett
Sent from my iPhone

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/AzcyYneW820/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DgHWf5zD3j8GD8%2BYXsf5HCwYPvSXcH2Veka2O2_b5w_w%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Ken Barber]

> I was just testing the host config file from puppetdb coupled with the documentation on the beaker documentation.

Those docs honestly look old, they are still mentioning blimpy which I
effectively deprecated/superseded with the aws_sdk driver.

> I was actually going to omit the error message. That's actually all of it except for the json output of the compiled beaker configs. I can send the full output in the morning.

Send the full output and the configuration and I can take a closer
look. Anything less, I'll probably struggle.

> It looks like the Google Compute Engine docs are more complete... It doesn't matter where it runs. Mostly looking for a free tier cloud to get started with. I'm not sure aws micro would even be big enough anyways. But it'd be cool to get it working.

Sure, well we use EC2 heavily so I'm happy to help you there, I know
some people use Google Compute Engine also, but I have no intimate
knowledge of how this one works.

ken.

[Justin Stoller]

On Mon, Oct 6, 2014 at 6:44 PM, Ken Barber <k...@puppetlabs.com> wrote:

> I was just testing the host config file from puppetdb coupled with the documentation on the beaker documentation.

Those docs honestly look old, they are still mentioning blimpy which I
effectively deprecated/superseded with the aws_sdk driver.

> I was actually going to omit the error message.  That's actually all of it except for the json output of the compiled beaker configs.   I can send the full output  in the morning.

Send the full output and the configuration and I can take a closer
look. Anything less, I'll probably struggle.

You should also include a redacted version of your ~/.fog

 

> It looks like the Google Compute Engine docs  are more complete...  It doesn't matter where it runs. Mostly looking for a free tier cloud to get started with.   I'm not sure aws micro would even be big enough anyways. But it'd be cool to get it working.

Sure, well we use EC2 heavily so I'm happy to help you there, I know
some people use Google Compute Engine also, but I have no intimate
knowledge of how this one works.

ken.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTng8oLjKCaVS6RV%2BjBSHqWAgYSatP69fpxcNWF1Upmz%2BA%40mail.gmail.com.

[Ken Barber]

Actually Brett, maybe this is a better approach. I've got a working
new project here showing beaker + beaker-rspec with EC2 support:

https://github.com/kbarber/sample-beaker

And you can see how I've launched it here:

https://gist.github.com/kbarber/850a7d88fce409592bab

Perhaps a better example will set you straight :-). It does fail
incidentally, but it is kind of meant to as an example. Perhaps you
can start with this project skeleton and modify to taste.

Now as Justin mentioned, you do need a ~/.fog file - this was
primarily to be compatible with the old Blimpy driver, but alas, we
don't use Fog any more. The file should look something like:

:default:
:aws_access_key_id: AAAAAA
:aws_secret_access_key: BBBBBB

(And obviously match your own EC2 access keys and secrets)

Also pay close attention to the config/image_templates/ec2.yaml file
... these map names of images to AMIs today, and the AMIs provided are
the ones we use for our own testing, but you might want to maintain
your own list. This is entirely up to you, just be aware each AMI
needs a small amount of pre-setup if you want to create your own (that
is certain minimal things might need 'baking' into the image, but
nothing drastic). Of course, you are free to use the ones here, but if
you need customisation I'd suggest forking your own images :-).

Let me know how you get on with that. I think generally speaking all
of this needs an overhaul in regards to usability, a lot of this
awkward layout is due to backwards compatibility from legacy elements.
That aside, once you get the fundamental elements right it should be
okay.

ken.

[Brett Swift]

That's great Ken. 

I'll have a look.   My .fog file was correct but I was missing that ec2.yaml.  

I get the user experience thing, it'll evolve and I'll help if I can.   

Would I be right to assume you built your images with packer?   

Thanks! 

[Ken Barber]

> That's great Ken.
>
> I'll have a look. My .fog file was correct but I was missing that
> ec2.yaml.
>
> I get the user experience thing, it'll evolve and I'll help if I can.
>
> Would I be right to assume you built your images with packer?

All of those images predate packer, but we're using packer now. For
example my colleague Wyatt is about to add Debian 7 testing using a
packer template he has developed. So going forward yes, packer is the
trick.

ken.

[Brett Swift]

Nice.  I'll look out for your packer project.  I've been using it to build our vagrant boxes, but haven't yet built any for vSphere or external cloud. 

I'm getting this error: 

ruby/2.1.0/gems/aws-sdk-1.42.0/lib/aws/core/resource.rb:238:in `rescue in block in define_attribute_getter': unable to find the image (AWS::Core::Resource::NotFound)

Are your AMI's public? 

Gist: 

https://gist.github.com/brettswift/176b802bfe31dae369e9

Using your sample project, if I include the puppetlabs_spec_helper in the Gemfile, it doesn't error but completes rather quickly saying there are no tests (obviously).   But it doesn't bawk on any ec2 configuration. 

[Ken Barber]

> Nice. I'll look out for your packer project. I've been using it to build
> our vagrant boxes, but haven't yet built any for vSphere or external cloud.

Its not my project, its a private project we have. So I can't speak
around if we'll release it or not - just to be clear :-).

> I'm getting this error:
> ruby/2.1.0/gems/aws-sdk-1.42.0/lib/aws/core/resource.rb:238:in `rescue in
> block in define_attribute_getter': unable to find the image
> (AWS::Core::Resource::NotFound)
>
> Are your AMI's public?
> Gist:
> https://gist.github.com/brettswift/176b802bfe31dae369e9

I would presume not? Honestly, I would look into creating your own
most probably. Not because they don't work, just that you could
probably do a better quality job and then own the process yourself.
Start with a good basis, like the endorsed Centos AMI's for example:

http://wiki.centos.org/Cloud/AWS

Then apply the necessary customisations on top with packer using the
amazon-ebs builder as a good 'entry level' way of doing it:
http://www.packer.io/docs/builders/amazon-ebs.html

> Using your sample project, if I include the puppetlabs_spec_helper in the
> Gemfile, it doesn't error but completes rather quickly saying there are no
> tests (obviously). But it doesn't bawk on any ec2 configuration.

So puppetlabs_spec_helper assists with the unit testing side of things
and has nothing directly to do with beaker, so look into rspec-puppet
for the long story around that. puppetlabs_spec_helper provides a
number of utilities and helps bridge the testing parts with various
different versions of puppet basically, but the main piece of work
that users should focus on is rspec-puppet.

Basically from a user perspective the helper gives you a rake task:
`rake spec` and a file like so:
https://github.com/puppetlabs/puppetlabs-puppetdb/blob/master/.fixtures.yml
that will help you automatically retrieve the other modules your
module depends on during testing time. As opposed to just running
`rspec spec/unit` for example. So yeah, chalk and cheese ...

ken.

[brett...@gmail.com]

Hmmm. I'm pretty sure the puppelabs_spec_helper gave me a beaker target....

Brett
Sent from my iPhone

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/AzcyYneW820/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTnCuq80Zj%2ByuzNQcqUErz7viufhhdPHsHpyicQkNkKqTg%40mail.gmail.com.

[Ken Barber]

>>> Using your sample project, if I include the puppetlabs_spec_helper in the
>>> Gemfile, it doesn't error but completes rather quickly saying there are no
>>> tests (obviously). But it doesn't bawk on any ec2 configuration.
>>
>> So puppetlabs_spec_helper assists with the unit testing side of things
>> and has nothing directly to do with beaker, so look into rspec-puppet
>> for the long story around that. puppetlabs_spec_helper provides a
>> number of utilities and helps bridge the testing parts with various
>> different versions of puppet basically, but the main piece of work
>> that users should focus on is rspec-puppet.
>>
>> Basically from a user perspective the helper gives you a rake task:
>> `rake spec` and a file like so:
>> https://github.com/puppetlabs/puppetlabs-puppetdb/blob/master/.fixtures.yml
>> that will help you automatically retrieve the other modules your
>> module depends on during testing time. As opposed to just running
>> `rspec spec/unit` for example. So yeah, chalk and cheese ...
>

> Hmmm. I'm pretty sure the puppelabs_spec_helper gave me a beaker target....

Yep you are right, it was added in January by blkperl. I didn't realize this.

So in response to your original question:

>>> Using your sample project, if I include the puppetlabs_spec_helper in the
>>> Gemfile, it doesn't error but completes rather quickly saying there are no
>>> tests (obviously). But it doesn't bawk on any ec2 configuration.

Perhaps its because its using the `default.yml` definition that it is
not complaining. Perhaps that AMI is public and fine. Check
`spec/acceptance/nodesets/default.yml` in your project for what this
contains.

All the rake task seems to do is run beaker with --color and the
spec/acceptance directory as a path, so beakers defaults everywhere
else kick in. Which makes sense, in this case there are several
BEAKER_* style variables that are used to setup the details in CI for
example. This is because CI software such as Jenkins uses environment
variables as a primary API between the Jenkins job data and the jobs
scripts being executed.

https://github.com/puppetlabs/beaker/wiki/How-to-Write-a-Beaker-Test-for-a-Module

The BEAKER_set allows for a basic way of executing a particular
nodeset for example, which is rather nice. Perfect for a matrix job in
Jenkins to iterate across your distros.

ken.

[brett...@gmail.com]

The default.yaml is in the gist I supplied. I used the el6 and centos6 64 from your sample project.

An Ami search on aws by the name or Ami ID didn't turn up any results which is why I thought they were private.

I'll try building an Ami with packer. Are there any conventions that need to go into the VM required by beaker?

Brett
Sent from my iPhone

> --
> You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/AzcyYneW820/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNT%3DPeu-CEU50yFTgoyWhywHAPbqz6tbD_PVTKznv-FDPqw%40mail.gmail.com.

[Ken Barber]

> The default.yaml is in the gist I supplied. I used the el6 and centos6 64 from your sample project.

Taking a look again, its even more confusing than that, the platform
name you have used is pointing at one of the very very old AMI's. The
newer ones we actually use are private, but I don't have the ability
to make it public. Looks like its locked up in someone elses older
amazon account. Sorry about that. This is why we are slowly replacing
these older ones, the management of them has slipped, my fault mainly
- we'll work on that for PuppetDB.

> An Ami search on aws by the name or Ami ID didn't turn up any results which is why I thought they were private.
>
> I'll try building an Ami with packer. Are there any conventions that need to go into the VM required by beaker?

I think the main requirement to make your life easier, is to allow
logging in from root (which can be often disabled) this is often done
by changing just sshd_config (and hupping/restarting sshd), and
removing the /root/.ssh/authorized_keys files also. Some distros
support doing this kind of thing via cloud-init as well, so you can
pass something like:

#cloud-config
disable-root: false

Into the user_data field of packer for example, but not this will not
persist on its own, you can change the configuration in /etc/cloud/
and bake that setting into the image. This usually affects
Debian/Ubuntu images I believe, I haven't built one of the newer
centos images in a while so try firing up the image and checking if it
has /etc/cloud/ in its build, this usually indicates its got some sort
of cloud-init. Having said that, someone has a patch up for beaker to
allow logging in as non-root and 'fixing' this:

https://github.com/puppetlabs/beaker/pull/478

Beyond that it depends on the tests and helpers you want to use. Both
'curl' and 'git' are good tools to have around on an image, since a
lot of tests can utilize these commands, having said that some tools
on top of the image can most certainly be installed by beaker in the
spec_helper_acceptance.rb there is a section for doing this _before_
the suite runs.

In fact, the base beaker stuff will attempt to install items for you,
like timing is important so the code goes to lengths to make sure the
the image has the right tools before it uses ntp to update the time.
Having the tooling there (I believe it uses ntpdate? but look at your
log output from beaker it will tell you) should make that exercise
mildly faster.

Starting simple, and trying to run tests against an image is usually a
good move in general, and being additive when you need it. Like I said
you have two choices most of the time - bake in a tool, or do it
during the testing run _before_ suite ... its really dependant on the
wider problem you are going to solve with the image. Baking items into
the image will of course speed up the exercise at test time, at the
cost of forcing you to remake it when it needs a change. Starting
conservatively with what goes into the image I think is a good idea.

After that you'll start to see the wider problems when you run your
tests a lot ... such as the flakiness of public mirrors :-). But thats
a story for another day :-).

ken.