Re: Chef testing talk on twitter
Robert J Berger
I absolutely agree that testing is key. Shortening the cookbook writing/testing feedback loop is what drove me down this road.
From what I've seen, the Infochimps are closest at the moment. Spiceweasel and a couple other similar projects just seem like a glorified shell script, which is what everyone is trying to get away from. I presume because you are using cluster-chef v2 already, you concur that it's the closest baseline out there?
From what I can tell, they have been using it internally for a long time. Their efforts to open source it were there, but they were not putting much effort into a "community". They weren't against it, but they were mainly scratching their own itch (Image of Chimps is too much).
I'd like to understand the Infochimps' plan for Ironfan. The rspec tests wouldn't even run last week due to missing test fixtures, which is a concerning property for the foundation of a test system. There's a lot of churn in it right now, though, so it's a fast moving target.
I think bouncing around ideas and requirements would be a good start. Start on the chef list and see if there's enough interest to split off?
What are your requirements and expectations for infrastructure testing?
- Some unit / sanity checks that can be run before commiting to Git and pushing to the chef server. Like foodcritic++ and ideally some set of Cucumber/Rspec like tests that can be run without running a full VM run.
- Would catch stupid syntax errors, databag access and attribute errors, Smart checking against chef node attributes from chef server
- Standalone Cookbook tests that would use toth or something else to test using Vagrant / VM with some mocking if needed. https://github.com/acrmp/chefspec seems good for some of this
- Cluster level testing with local Vagrant or other VMs
- Should be able to be done in a CI if desired
- Post chef-client run Cucumber/RSpec type tests that can be a form of regression test for everything you are testing for, chef-attributes, files, processes running etc.
- This could be part of monitoring or just a mechanism to confirm that the chef run did all you wanted beyond what your monitoring system shows.
On Tue, Feb 21, 2012 at 1:54 AM, Robert J Berger <rbe...@runa.com> wrote:What you describe is very much what we would also like to have. We're using cluster_chef (aka Ironfan) already (version2).We're in the same position, not crtical path for us ether, but its clear that having testing as part of the chef process is key to long term sanity.Its also clear that a lot of other folks are thinking and poking around with this, so it seems like it would be good to try for some synergy of efforts.Not sure what's the right way to go. Maybe a Mailing list and IRC channel?__________________Robert J BergerOn Feb 20, 2012, at 2:56 PM, Wagner, Jason wrote:Hello, Robert
What type of problems were you looking at addressing?
I timeboxed an attempt to turn Toft into a cloud provider in cluster-chef, but the rebranding to ironfan and the fact that rake spec doesn't run shied me away for now.
The plan was:
- extend cluster-chef with Toft/lxc and OpenStack as clouds
- add SmokeTest and Test DSL statements to facet to run cucumber features.
- add knife cluster tasks for local testing
- add smoke testing to cluster launch
- very rough investigation of cross-node notificationsNone of this is critical path for us at the moment, but it is a long term goal to have it. If there's a group looking at something similar, I'd like to be involved.
Thanks!
--
Jason Wagner
Wagner, Jason
Hi, I'm cc'ing Marius as we were having the exact same conversation last week.__________________Robert J Berger - CTORuna Inc.
On Feb 21, 2012, at 6:24 AM, Wagner, Jason wrote:I absolutely agree that testing is key. Shortening the cookbook writing/testing feedback loop is what drove me down this road.
Shortening the edit, push, test cycle is important, but also all the other good things of BDD/TDD, that allow you to feel free to refactor and know that you are not reintroducing old bugs and better catch new bugs.
Yes, I previously had my own hack of scripts to bring up a cluster by building knife commands and such. cluster_chef was a big improvement. We're looking forward to V3 aka Ironfan. It looks like they ahve done some testing and ci ideas in Ironfan. We're going to have a mini-tutorial at BigDataCamp next week in Santa Clara. Flip Fromer will be giving it. Are you in the Bay Area or will you be here for the Strata Conf or something?
From what I've seen, the Infochimps are closest at the moment. Spiceweasel and a couple other similar projects just seem like a glorified shell script, which is what everyone is trying to get away from. I presume because you are using cluster-chef v2 already, you concur that it's the closest baseline out there?
I'm on the opposite coast, in the DC area. Not sure I'll be hitting anything that way for a while.
From what I can tell, they have been using it internally for a long time. Their efforts to open source it were there, but they were not putting much effort into a "community". They weren't against it, but they were mainly scratching their own itch (Image of Chimps is too much).
I'd like to understand the Infochimps' plan for Ironfan. The rspec tests wouldn't even run last week due to missing test fixtures, which is a concerning property for the foundation of a test system. There's a lot of churn in it right now, though, so it's a fast moving target.
Nathan from Infochimps was at the Chef Summit, and saw the interest in crowbar, general need for orchistration and my nudging must have caused him to go back and push Infochimps to launch Cluster_chef as a product / service. At the same time, Opscode asked them not to have "chef" in the name due to trademark complications. Opscode wants to encourage them to keep going, but they can't allow Chef® to become the Xerox® or Klenex® of configuration management. But that caused the renaming to Ironfan.It looks like they are open sourcing a lot more than what they had before and they are in the middle of refactoring and renaming a bunch of stuff. We should ask when it will be appropriate to try using it since it does seem like its still in flux...
I sent an email earlier today to flip. With hindsight, you probably would have been better to talk to them.
I find the chef-list to be impossible to follow, at least in the form of the digests I get. Do you know of a better way to follow it? All the conversations are jumbled together....I think bouncing around ideas and requirements would be a good start. Start on the chef list and see if there's enough interest to split off?
I think its worth starting something dedicated to this... I just created a google group for now....
What are your requirements and expectations for infrastructure testing?
- Some unit / sanity checks that can be run before commiting to Git and pushing to the chef server. Like foodcritic++ and ideally some set of Cucumber/Rspec like tests that can be run without running a full VM run.
- Would catch stupid syntax errors, databag access and attribute errors, Smart checking against chef node attributes from chef server
- Standalone Cookbook tests that would use toth or something else to test using Vagrant / VM with some mocking if needed. https://github.com/acrmp/chefspec seems good for some of this
- Cluster level testing with local Vagrant or other VMs
- Should be able to be done in a CI if desired
- Post chef-client run Cucumber/RSpec type tests that can be a form of regression test for everything you are testing for, chef-attributes, files, processes running etc.
- This could be part of monitoring or just a mechanism to confirm that the chef run did all you wanted beyond what your monitoring system shows.
So there looks to be two complementary but separable goals here- validating the production and validating product. The rspec, foodcritic, and such validate that the method acts as expected. I was more focused on validating that the product acted as expected via cucumber-nagios and toft-- sort of a smoke test/front line, one-shot monitoring.
Or perhaps I need sleep.
So there's unit-y tests with rspec, static analysis with foodcritic, and integration-y tests with rspec/cucumber, which potentially transition to a monitoring capability. Ironfan seems to impact the integration far more than the unit/static analysis phases.
So can this be modeled like a build tool? Chef-maven, anyone? Knife is a tool for targetted tasks, does this seems to chain a bunch of knife commands together into a build workflow?
Need to sleep on it. </babbling>
Jason Wagner
Marius Ducea
out ironfan in the next days or so. One of the important things we are
trying to achieve is dependencies and ordering (aka: wait for a dns
server to be fully configured and only after that add the dns records,
etc.) Do you guys know if Ironfan supports dependencies, or how would
you do something like that?
Thanks,
- Marius -
On 2/21/12 8:00 PM, Wagner, Jason wrote:
> Hi Marius!
>
> On Tue, Feb 21, 2012 at 3:39 PM, Robert J Berger <rbe...@runa.com
> <mailto:rbe...@runa.com>> wrote:
>
> Hi, I'm cc'ing Marius as we were having the exact same conversation
> last week.
>
> __________________
> Robert J Berger - CTO
> Runa Inc.
> +1 408-838-8896 <tel:%2B1%20408-838-8896>
> * Some unit / sanity checks that can be run before commiting to
> Git and pushing to the chef server. Like foodcritic++ and
> ideally some set of Cucumber/Rspec like tests that can be run
> without running a full VM run.
> o Would catch stupid syntax errors, databag access and
> attribute errors, Smart checking against chef node
> attributes from chef server
> * Standalone Cookbook tests that would use toth or something else
> to test using Vagrant / VM with some mocking if needed.
> https://github.com/acrmp/chefspec seems good for some of this
> * Cluster level testing with local Vagrant or other VMs
> o Should be able to be done in a CI if desired
> * Post chef-client run Cucumber/RSpec type tests that can be a
> form of regression test for everything you are testing for,
> chef-attributes, files, processes running etc.
> o This could be part of monitoring or just a mechanism to
> confirm that the chef run did all you wanted beyond what
> your monitoring system shows.
>
> Matches very well with my list. I missed foodcritic somehow, but I like.
>
> So there looks to be two complementary but separable goals here-
> validating the production and validating product. The rspec,
> foodcritic, and such validate that the method acts as expected. I was
> more focused on validating that the product acted as expected via
> cucumber-nagios and toft-- sort of a smoke test/front line, one-shot
> monitoring.
>
> Or perhaps I need sleep.
>
> So there's unit-y tests with rspec, static analysis with foodcritic, and
> integration-y tests with rspec/cucumber, which potentially transition to
> a monitoring capability. Ironfan seems to impact the integration far
> more than the unit/static analysis phases.
>
> So can this be modeled like a build tool? Chef-maven, anyone? Knife is
> a tool for targetted tasks, does this seems to chain a bunch of knife
> commands together into a build workflow?
>
> Need to sleep on it. </babbling>
> --
> Jason Wagner
> ja...@tensorwrench.com <mailto:ja...@tensorwrench.com>
>
Wagner, Jason
There are hard could be modeled as a dependency graph between facets. Right now it launches all servers in parallel, waits for launches to finish, then launches all convergences in parallel. It might be sufficient to:
while unlaunched servers > 0
if no servers with satisified dependencies, error
converge servers with satisified dependencies in parallel
join
end
--
Jason Wagner
Robert J. Berger
________________________