Updating TDI Book

[Stephen Nelson-Smith]

Hello all,

I'm working on updates and expansions for the O'Reilly Test-driven
Infrastructure book. It's fair to say the book has received a mixed
reception. I guess it depends what people were expecting from it.
Obviously it isn't a book about Chef - and readers expecting a through
coverage of everything about Chef were disappointed. Hopefully,
they'll be happier with TDG, out in the summer.

However, amongst the critics, the general view is that the book needs
more worked examples and more meat around the actual testing. I think
that's fair, although to be honest the original remit was really just
an exploration of the overall ideas, and a call for people to start to
explore this space. At the time of writing, the number of tools
available for this kind of endeavour was pretty small. Since then, I
think interest in the subject has grown.

So - as I look to expand and improve, I'd greatly welcome ideas and
requests for coverage. If you've read the book, excellent. If not,
let me know - we can get you a copy to read.

Thanks,

S.

[Robert J Berger]

That's exciting to hear! I think that could be a really good focus for getting testing and Chef to the next level and communicating the why's and hows...

Since a lot of us are just getting started on injecting testing into our workflow, it might be great to have some way that we can start from some common frameworks or patterns, try them out / use them in our own environments and then submit appropriate results into your book...

I don't know if it would be worth having a wiki or some way to help crowdsource parts of the book this way... But it could be a virtuous cycle to try for something like that...

Rob

[Brad Knowles]

On Mar 6, 2012, at 5:43 PM, Stephen Nelson-Smith wrote:

> So - as I look to expand and improve, I'd greatly welcome ideas and
> requests for coverage. If you've read the book, excellent. If not,
> let me know - we can get you a copy to read.

I'd like to see some real-world examples of test-driven development with regards to cookbooks, starting fairly simply but working up to the level of some of the most complex cookbooks out there -- perhaps even take one of the more complex cookbooks out there and use it as another example.

I'd also like to see detailed discussion of minitest and Rspec in comparison & contrast with a tool like cucumber, and how tools like Jenkins (or other CI platforms) can also integrated into that.

Right now, we're not doing any TDD at all, and I'd like to fix that.

But that means retrofitting a lot of testing on top of many existing cookbooks (over 50 cookbooks, comprising over 50k lines of "infrastructure as code"), and that also means using the right tool for the job -- whatever that might mean with regards to any testing, development, or CI tools that could potentially be involved.

And yes, I'd love to read a copy of the current book, as well as being involved in the next one.

--
Brad Knowles <bkno...@ihiji.com>
SAGE Level IV, Chef Level 0.0.1

[Bryan Berry]

SNS,

I read the book and enjoyed it, as you are already aware

My biggest issue with the book was getting the examples to run. I had a number of problems getting the examples to run, as have others. Improving that experience would go a long way.

As Brad_K points out, it would be great to test an existing cookbook. I recommend ntp

Like Brad_K, I would also like to be involved in TDG :)

keep up the great work!

--
You received this message because you are subscribed to the Google Groups "chef-testing" group.
To post to this group, send email to chef-t...@googlegroups.com.
To unsubscribe from this group, send email to chef-testing...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/chef-testing?hl=en.

[Mike Adolphs]

Hi Stephen,

On Wed, Mar 7, 2012 at 12:43 AM, Stephen Nelson-Smith <sane...@gmail.com> wrote:

...

However, amongst the critics, the general view is that the book needs
more worked examples and more meat around the actual testing.

Your book's a well-written introduction for cucumber-chef I've overall enjoyed reading.

What I didn't like about it though was that you've only scratched the surface in respect of testing and that cucumber-chef is just way to ec2-centric. If you read the synopsis e.g. at the german Amazon store (http://www.amazon.de/dp/1449304818) - don't worry, synopsis' in english - it's way to unclear that Amazon's ec2 is a requirement to run the examples. The title didn't show this either.

At my company for example we're not using ec2 at all for privacy reasons and I'm mostly interested in solutions which could be integrated in a VirtualBox/Vagrant/VMware environment. As Brad already said taking build servers like Jenkins into account would be helpful as well.

Good to hear that you keep working on the topic!

Best regards,

Mike

 

--

Mike Adolphs

Stitenstrasse 24

23554 Luebeck

- Germany -

Mail. mi...@fooforge.com

Web. http://fooforge.com/

Github. https://github.com/fooforge/

XING. http://www.xing.com/profile/Mike_Adolphs

[Brad Knowles]

On Mar 7, 2012, at 12:18 AM, Bryan Berry wrote:

> As Brad_K points out, it would be great to test an existing cookbook. I recommend ntp

Oh, if you're going to do NTP, then I'll resurrect all my old contacts with the NTP Public Services Project and the NTP Forum, and I assure you that we will do everything we possibly can to make the cookbook as correct and full-featured as possible.

You might even help me convince them that they need to use Chef to manage their entire development/build architecture, which could help drive chef to support many additional platforms that are not currently covered.

There's lots of potential for serious depth there.

[Brad Knowles]

On Mar 7, 2012, at 3:42 AM, Mike Adolphs wrote:

> At my company for example we're not using ec2 at all for privacy reasons and I'm mostly interested in solutions which could be integrated in a VirtualBox/Vagrant/VMware environment.

Oh, crap. I didn't know that it was ec2-specific. We use Rackspace, and if ec2 is a requirement for cucumber-chef, then it would not be possible for us.

Yeah, we definitely would want to cover multiple chef implementation environments.

Thanks for the info!

[Miah Johnson]

On Wednesday, March 7, 2012 8:27:12 AM UTC-8, Brad Knowles wrote:

Oh, crap.  I didn't know that it was ec2-specific.  We use Rackspace, and if ec2 is a requirement for cucumber-chef, then it would not be possible for us.

Yeah, we definitely would want to cover multiple chef implementation environments.

Not only that, it was tailored specifically to Hosted-Chef, which means I've never used it. I have looked at chefspec and used foodcritic quite a bit though.

I did enjoy the book, but the rather niche requirements to get into cucumber-chef disappointed me.

-Miah

[Stephen Nelson-Smith]

Hi there,

Apologies for the slow response - I've been super-busy!

On Wednesday, 7 March 2012 00:09:03 UTC, Robert J Berger wrote:

That's exciting to hear! I think that could be a really good focus for getting testing and Chef to the next level and communicating the why's and hows...

Absolutely.  I'd love it to be the core around which we organize ideas, patterns, innovation etc.  That's the great thing about these constantly evolving books.

 

Since a lot of us are just getting started on injecting testing into our workflow, it might be great to have some way that we can start from some common frameworks or patterns, try them out / use them in our own environments and then submit appropriate results into your book...

Yep absolutely.  Somewhere where we can collaborate and discuss would be perfect.  This list is a great start.

 

I don't know if it would be worth having a wiki or some way to help crowdsource parts of the book this way... But it could be a virtuous cycle to try for something like that...

Yes, that would be good... *thinks*

S.

[Stephen Nelson-Smith]

Hi Brad,

On Wednesday, 7 March 2012 02:47:02 UTC, Brad Knowles wrote:

On Mar 6, 2012, at 5:43 PM, Stephen Nelson-Smith wrote:

> So - as I look to expand and improve, I'd greatly welcome ideas and
> requests for coverage.  If you've read the book, excellent.  If not,
> let me know - we can get you a copy to read.

I'd like to see some real-world examples of test-driven development with regards to cookbooks, starting fairly simply but working up to the level of some of the most complex cookbooks out there -- perhaps even take one of the more complex cookbooks out there and use it as another example.

The book goes through a fairly trivial example.  It certainly needs a more complex example.  Any suggestions or requests for which cookbooks / scenarios would make a good example?  It can't be so complicated as to take up a huge amount of space and require significant domain expertise, but it does need to surface some complexity, I think.

I was thinking something like a load-balanced media wiki, or something like that.

 

I'd also like to see detailed discussion of minitest and Rspec in comparison & contrast with a tool like cucumber, and how tools like Jenkins (or other CI platforms) can also integrated into that.

Right - I'm working on that right now.  Doing a compare and contrast of the various approaches.  More on this shortly.

 

Right now, we're not doing any TDD at all, and I'd like to fix that.

But that means retrofitting a lot of testing on top of many existing cookbooks (over 50 cookbooks, comprising over 50k lines of "infrastructure as code"), and that also means using the right tool for the job -- whatever that might mean with regards to any testing, development, or CI tools that could potentially be involved.

In this respect this is actually  no different from trying to retrofit tests to any legacy code... but maybe this could be covered.

 

And yes, I'd love to read a copy of the current book, as well as being involved in the next one.

Drop me an email privately, and I'll sort you out with a copy.

S.

[Stephen Nelson-Smith]

Hi,

What I didn't like about it though was that you've only scratched the surface in respect of testing and that cucumber-chef is just way to ec2-centric. If you read the synopsis e.g. at the german Amazon store (http://www.amazon.de/dp/1449304818) - don't worry, synopsis' in english - it's way to unclear that Amazon's ec2 is a requirement to run the examples. The title didn't show this either.

Agree.  So, when I wrote cucumber-chef, there were some specific design decisions for that.  Now we have more tools, this can be expanded.  Of course, if there's sufficient interest, we could make cucumber-chef support other approaches.  Vagrant would be an option, although I'm yet to be convinced that this scales.  What do you think would be the priority?

 

At my company for example we're not using ec2 at all for privacy reasons and I'm mostly interested in solutions which could be integrated in a VirtualBox/Vagrant/VMware environment. As Brad already said taking build servers like Jenkins into account would be helpful as well.

Aha! You just answered... so the problem I have with VirtualBox is this:  If you want to spin up a complicated infrastrcture - with say three or four nodes - you have two problems immediately.  Firstly, many people won't have enough memory in your generic workstation to do this comfortably.  Secondly, it's slow to start up.  The obvious next step is to set up a dedicated VirtualBox machine.  But now you've set up an external dependency - users need to buy/procure/set up a powerful machine in order to infrastructure.

My solution to these problems was to use Linux containers (fast, cheap) and EC2 (fast, easy).  However, I appreciate it's platform-specific, and making it support alternative platforms would be good.

Thanks for the feedback.

S.

[Stephen Nelson-Smith]

Hi Brad,

On Wednesday, 7 March 2012 16:27:12 UTC, Brad Knowles wrote:

On Mar 7, 2012, at 3:42 AM, Mike Adolphs wrote:

> At my company for example we're not using ec2 at all for privacy reasons and I'm mostly interested in solutions which could be integrated in a VirtualBox/Vagrant/VMware environment.

Oh, crap.  I didn't know that it was ec2-specific.  We use Rackspace, and if ec2 is a requirement for cucumber-chef, then it would not be possible for us.

It just uses the Fog API.  That could trivially switch out to Rackspace.

S.

[Stephen Nelson-Smith]

Hello Miah,

On Friday, 16 March 2012 23:10:14 UTC, Miah Johnson wrote:

On Wednesday, March 7, 2012 8:27:12 AM UTC-8, Brad Knowles wrote:

Oh, crap.  I didn't know that it was ec2-specific.  We use Rackspace, and if ec2 is a requirement for cucumber-chef, then it would not be possible for us.

Yeah, we definitely would want to cover multiple chef implementation environments.

Not only that, it was tailored specifically to Hosted-Chef, which means I've never used it. I have looked at chefspec and used foodcritic quite a bit though.

They're entirely different beasts.  Excellent tools, relevant and valuable, but not trying to solve the same problem.  As for Hosted-Chef, yes... and again, for the same reasons as I enumerate above.  I wanted cucumber-chef to be easy to get going with.  Basically - install the gem, and populate a knife.rb with SAAS credentials for EC2 and Hosted Chef.

What would you want to use?  Cucumber-chef would work with Open Source Chef - it doesn't rely on any Hosted Chef features (RBAC, Organizations).  So as long as you construct your knife.rb to use your own Chef server, it would just work.  Maybe I should give an example of doing that?

I've never used Chef solo - how much demand would there be to cover that?

I did enjoy the book, but the rather niche requirements to get into cucumber-chef disappointed me.

Agreed.  We can fix that :)

S.

[Wagner, Jason]

I was having a lot of success with Toft (https://github.com/exceedhl/toft) for setting up LXC containers using Chef Solo.  It has Vagrant wrappers for setting up VMs to build the containers on, but I haven't used that.  There were some important caveats:

- All testing needs are Linux of the same distro/version.
- I had to condition out things that rely on searching and use a default value, since any feature that relies on the server breaks Chef Solo.

I don't think it would be too hard to stub out a search provider or use a Chef Server. Both of which are things I want to look at in the next month or two as that priority bubbles to the top of our needs.

I read and liked the TDD Infrastructure book, but the EC2 and Hosted Chef dependencies put me off using Cucumber Chef. If it's just using fog and the knife config, I'll probably have to re-evaluate how much it would take to meet the needs.

I would argue that chefspec and similar tools could be a very strong part of the Chef TDD cycle. For me, part of TDD is the rapidity of the feedback. The quicker that tests show me red/green lights, the less temptation there is to skirt good practices or get distracted. Things like chefspec will produce quicker, first-pass feedback in some situations.

S.

--

You received this message because you are subscribed to the Google Groups "chef-testing" group.

To view this discussion on the web visit https://groups.google.com/d/msg/chef-testing/-/J6lqO90oGJMJ.

To post to this group, send email to chef-t...@googlegroups.com.
To unsubscribe from this group, send email to chef-testing...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/chef-testing?hl=en.

--
Jason Wagner

ja...@tensorwrench.com