crazy ideas about minitest and simple_cuke

[Bryan Berry]

using the location  files/default/minitest/*_test.rb   to store my tests doesn't really work for me

I only want to run the tests that apply to the recipes that are in my run_list, not to all recipes that belong to the cookbooks in my run_lists

accordingly, simple_cuke's locations for tests

files/default/suite  also doesn't quite work for me

Do we really need 'files/default' ? 

I have the following in mind, the following layout would be preferable, even thought it means hacking the remote_directory resource to support a path besides files/

tests/{minitest,simple_cuke,cuken,test_foo}/<recipe_name>/<good_stuff_here>

I have modified the mintest-chef-cookbook so that is fetches from files/default/minitest/<recipe_name>/*_test.rb

https://github.com/bryanwb/minitest-handler-cookbook

it is conceivable to me that I may have tests from minitest, simple_cuke, and some other testing scheme yet to be invented all in the same cookbook. 

Tomorrow I hope to try adding some cucumber features . . . wish me luck!

[Bryan Berry]

I have been playing more w/ simple_cuke but am running into some roadblocks from my complete ignorance of cucumber.


https://github.com/bryanwb/chef-ark/blob/master/files/default/tests/features/test/test_steps.rb

Feature: ark dump

  

Scenario: File should be in its proper place

  When I run `ls /usr/local/foobar/`

  Then the output should match /jaxrpc.jar/

Sadly this fails on my hacked version of simple_cuke as it isn't a valid ruby file

https://github.com/bryanwb/simple_cuke

/var/cache/suite/features/test_steps.rb:1: syntax error, unexpected ':', expecting $end

Feature: Ark Dump

        ^ (SyntaxError)

Igor, btw my use of cucumber kept failing until I deleted the date line in the term-ansicolor.gemspec

I think there is value in segmenting the cucumber tests by recipe as the cookbook author will not know which roles the eventual users apply to them. 

time to go read some more about cucumber ;)

[ranjib dey]

save the feature file as .feature , not .rb.

--
You received this message because you are subscribed to the Google Groups "chef-testing" group.
To post to this group, send email to chef-t...@googlegroups.com.
To unsubscribe from this group, send email to chef-testing...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/chef-testing?hl=en.

[Igor Afonov]

Hi Bryan,

I think that our approaches to testing are a little bit orthogonal:) 

Let me explain you my workflow and my intentions behind simple_cuke: I've created it to describe & test the configuration of nodes. These are pure acceptance tests - they know nothing about implementation and they care only about resulting configuration. And I think in this case cucumber is perfect choice and using it at post run callback works very well.

My features set (20 scenarios and growing) main mission is make sure that the node configuration is adequate and up to date after each deploy/converge. It is especially valuable during node bootstrap (I don't know why - maybe I'm unlucky person but the most scary errors I've did were found during bootstrap.) Testing & describing nodes with cucumber for me is a kind of additional monitoring system.

I think in described case features should be stored in one place. You can put them in sperate folders but for cucumber there is no difference - it will load everything. I use tags to match features and nodes. Features without tags run everywhere while features that are marked with a node role would be run only on specified nodes. For now this is enough for me but if it is needed this mechanism could be improved and could match nodes by names etc.

I don't think that using cucumber in this configuration is a good idea to test individual recipes. I think it could potentially work for testing your own very custom recipes that are not intended to be cross-platform, but it won't work for recipes that you're going to redistribute because writing pure acceptance test for cross-platform environment could be very tricky, complex and error-prone. The only way to overcome cross-platform problems would be using chef resources in cucumber steps to do assertions - but for me it seems to be too complex and it would require too much effort to implement it.

Regarding your difficulties with cucumber - I really recommend you to take a look into this book http://pragprog.com/book/hwcuc/the-cucumber-book. It is great because it doesn't focus on web development/rails but instead shows how you can test anything with cucumber.

[Bryan McLellan]

On Sun, Apr 1, 2012 at 3:26 PM, Bryan Berry <bryan...@gmail.com> wrote:
> using the location  files/default/minitest/*_test.rb   to store my tests
> doesn't really work for me
>
> I only want to run the tests that apply to the recipes that are in my
> run_list, not to all recipes that belong to the cookbooks in my run_lists

Then do something like this?:

# Search through all cookbooks in the run list for tests
node[:recipes].each do |recipe|
# recipes is actually a list of cookbooks and recipes with :: as a delimiter
cookbook_name, recipe_name = recipe.split('::')
recipe_name = "default" if recipe_name.nil?
remote_file "tests-#{cookbook_name}-#{recipe_name}" do
source "tests/minitest/#{recipe_name}.rb"
cookbook cookbook_name
path "#{node[:minitest][:path]}/#{cookbook_name}"
purge true
ignore_failure true
end
end

> Do we really need 'files/default' ?

Chef is pretty specific about what folders get to remain part of a
cookbook. Even 'knife cookbook site share' uses the
Chef::CookbookLoader to make a tarball, which ultimately means a root
directory of 'tests' is not included in the upload to the community.
Uploading the cookbook to Hosted Chef will lose new top level
directories as well.

I suppose someone could extend the 'root_files' type therein to
support directories, but this would have to be done and I didn't want
to wait for this support to start working on performing integration
tests on cookbooks.

Bryan

[Bryan Berry]

hey btm,

exactly, that is pretty much the code I added to your cookbook to run the tests form my recipe

Igor, 

agreed, our approaches are orthogonal. I am thinking about it from the point of view of a cookbook maintainer rather than to test my overrall infrastructure. Still not sure whether minitest nor cucumber will better serve my needs. Have to learn more about cucumber to find out ;)

thanks for your work on simple_cuke

[Bryan McLellan]

On Tue, Apr 3, 2012 at 1:25 AM, Bryan Berry <bryan...@gmail.com> wrote:
> agreed, our approaches are orthogonal. I am thinking about it from the point
> of view of a cookbook maintainer rather than to test my overrall
> infrastructure. Still not sure whether minitest nor cucumber will better
> serve my needs. Have to learn more about cucumber to find out ;)

We're currently implementing both in our internal project for
integration testing the Opscode cookbooks. Minitest checks the little
bits get set on the system and cucumber ensures that the service works
correctly from outside.

Bryan

[Michael Johnston]

Bryan,

What tools/setup are you using to test from the outside? Does cucumber run on your dev machine, on the node being tested, or on another node that consumes the service being tested?

On Wednesday, April 4, 2012 4:02:05 PM UTC-7, Bryan McLellan wrote: