Organizing thoughts
116 views
Skip to first unread message
Wagner, Jason
Feb 29, 2012, 10:45:52 AM2/29/12
to chef-t...@googlegroups.com
I haven't had much time to look into this recently, but trying to organize what the flow is, what pieces are needed, and what is out there and how it fits. Consider the following conversation provoking, not a proposal as such.
Hypothetical Workflow
The "check everything and deploy" task would:
Run specs on all cookbooks.
Make lightweight nodes, apply the role, and run role/features.
Launch the cluster and run all cluster/features, inspect minitests.
Hierarchy
Clusters have many Nodes have many Recipes
For now, I'm purposefully assuming Nodes == Roles, Recipes == Cookbooks for the workflow to see how far we can get with a minimal number of entities in play.
Recipes/Cookbooks
Goal: Testing and quality checking very quickly without the need to apply it to a node.
rspec-chef, chefspec
BDD. Not how much practical overlap there is.
foodcritic
Code quality
minitest-chef-handler
Tests cookbooks applied to a system post-convergence? I'm unsure where this fits, but haven't looked at it much. Not really accounting for it in any further thoughts.
Roles/Nodes
Goal: Confirm that a node with a given role has expected behavior.
Toft + cucumber/cucumber-nagios
I've been using this as an 80% solution, though the fact that it runs chef-solo does cause some problems-- no searches or use environments. Might be addressable with a lightweight mock search object.
Cluster
Goal: Given a whole-system description, does the entire cluster have expected behaviors.
Ironfan
https://github.com/infochimps-labs/ironfan/wiki/core_concepts expresses some interesting things. Ironfan-ci bears some looking. The silverware cookbook *might* resolve the ordering and dependency issue that Marius mentioned. I'm waiving off any thought of cluster level stuff until I see what Ironfan does. No point in duplicating effort unnecessarily, and they are pretty far down the road. There are references in the doc to a lot of the cluster level stuff we've talked about.
Integrating the current tools
Using the current suite of tools out there (possibly with slight modifications), create a gem of tools, rake tasks, knife plugins and dependencies.
Extend the Chef repository conventions with:
/cookbook/*/spec -- run with rspec-chef and/or chefspec
/cookbook/*/features/steps -- cookbook specific steps to be used in roles/features
/roles/features -- cucumber behaviors for roles
When I have some time (next week... maybe), I might take a look at getting rspec-chef, chefspec, and/or footcritic as rake tasks, pending thoughts from other folks of course.
The toft/cucumber combination is working fairly well for me, but needs to mock out some chef features so chef-solo looks like chef-client. I've also only looked at the LXC usage, not Vagrant. Assembling a better library of cucumber steps would be great, too, Toft has a few and cucumber-nagios is useful. The convention includes /cookbook/*/features/steps specifically for adding recipe specific steps.
Questions
How do environments interact with this?
If different environments cause different expectations, how do the tests account for that?
Does testing certificates or databags make sense outside of the context of recipes/cookbooks?
Testing environments? (make sure all environments have certain property X, constraints, etc)
Any tools outside of the chef ecosystem of note?
Other thoughts?
--
Jason Wagner
Hypothetical Workflow
The "check everything and deploy" task would:
Run specs on all cookbooks.
Make lightweight nodes, apply the role, and run role/features.
Launch the cluster and run all cluster/features, inspect minitests.
Hierarchy
Clusters have many Nodes have many Recipes
For now, I'm purposefully assuming Nodes == Roles, Recipes == Cookbooks for the workflow to see how far we can get with a minimal number of entities in play.
Recipes/Cookbooks
Goal: Testing and quality checking very quickly without the need to apply it to a node.
rspec-chef, chefspec
BDD. Not how much practical overlap there is.
foodcritic
Code quality
minitest-chef-handler
Tests cookbooks applied to a system post-convergence? I'm unsure where this fits, but haven't looked at it much. Not really accounting for it in any further thoughts.
Roles/Nodes
Goal: Confirm that a node with a given role has expected behavior.
Toft + cucumber/cucumber-nagios
I've been using this as an 80% solution, though the fact that it runs chef-solo does cause some problems-- no searches or use environments. Might be addressable with a lightweight mock search object.
Cluster
Goal: Given a whole-system description, does the entire cluster have expected behaviors.
Ironfan
https://github.com/infochimps-labs/ironfan/wiki/core_concepts expresses some interesting things. Ironfan-ci bears some looking. The silverware cookbook *might* resolve the ordering and dependency issue that Marius mentioned. I'm waiving off any thought of cluster level stuff until I see what Ironfan does. No point in duplicating effort unnecessarily, and they are pretty far down the road. There are references in the doc to a lot of the cluster level stuff we've talked about.
Integrating the current tools
Using the current suite of tools out there (possibly with slight modifications), create a gem of tools, rake tasks, knife plugins and dependencies.
Extend the Chef repository conventions with:
/cookbook/*/spec -- run with rspec-chef and/or chefspec
/cookbook/*/features/steps -- cookbook specific steps to be used in roles/features
/roles/features -- cucumber behaviors for roles
When I have some time (next week... maybe), I might take a look at getting rspec-chef, chefspec, and/or footcritic as rake tasks, pending thoughts from other folks of course.
The toft/cucumber combination is working fairly well for me, but needs to mock out some chef features so chef-solo looks like chef-client. I've also only looked at the LXC usage, not Vagrant. Assembling a better library of cucumber steps would be great, too, Toft has a few and cucumber-nagios is useful. The convention includes /cookbook/*/features/steps specifically for adding recipe specific steps.
Questions
How do environments interact with this?
If different environments cause different expectations, how do the tests account for that?
Does testing certificates or databags make sense outside of the context of recipes/cookbooks?
Testing environments? (make sure all environments have certain property X, constraints, etc)
Any tools outside of the chef ecosystem of note?
Other thoughts?
--
Jason Wagner
Wagner, Jason
Mar 16, 2012, 7:40:42 AM3/16/12
to chef-t...@googlegroups.com
Just a note, but I've reprioritized the time I spend on this. I'll be looking into it after the OpenStack Essex release occurs in the next few months.
This group is pretty quiet. Has anyone had more thoughts or run across any problems that they would want to be addressed from a Chef testing system?--
Jason Wagner
This group is pretty quiet. Has anyone had more thoughts or run across any problems that they would want to be addressed from a Chef testing system?
Jason Wagner
Jay Perry
Mar 23, 2012, 10:14:39 AM3/23/12
to chef-t...@googlegroups.com
Can't you use minitest-chef-handler to validate roles/nodes post-convergence? This could be combined with vagrant. Thoughts?
Reply all
Reply to author
Forward
0 new messages