Beaker - what’s your perspective?

[Gene Liverman]

Hi friends! I’m trying to better understand the community perspective on Beaker and its supplemental gems. I’m particularly interested in hearing your thoughts on the state of its maintenance and what, if anything, you’d like to see change in that regard. I’m looking for both positive and negative opinions and impressions. 

--

Gene Liverman
Sr. Site Reliability Engineer
gene.l...@puppet.com

[Trevor Vaughan]

So, I chimed in over in Slack but wanted to go ahead and respond here with a summary of what we've been talking about there so that it'll be preserved for the future and searchable.

This is a summary of multiple views and anyone participating in that discussion should feel free to correct my biased opinions in here (I like Beaker). I have a presentation that I did on exactly what I use Beaker for and why I like it from last year's conference at https://www.youtube.com/watch?v=4iBEIMQkBCk. The associated repository can be found at https://github.com/trevor-vaughan/puppetize_2019_multi_node_beaker for those that want a full working example.

Who Uses it (module count is just modules, not necessarily modules with tests though the vast majority of the SIMP modules have tests):

• Voxpupuli
• 127 Forge Modules
  
• The System Integrity Management Project
• 106 Forge Modules
• A handful of other community folks
• I'd love to see a full analysis of all forge modules and what type of testing they use but I don't have time to dig into that right now (Gene?)
  

The Pros:

• Beaker generally works as it is for both single node and multi-node (my main use case) testing.
• See the video as to why multi-node testing is important
  
• It preserves the rspec syntax that makes the output of the tests easy to understand for non-technical folks as well as easy to trace for technical folks.
  
• It has the ability to be extended relatively easily in Ruby
• It works with most major cloud providers (and Vagrant)
• It hasn't really taken a lot of care and feeding recently to keep it chugging along

The Cons:

• It's not well documented (at all)
• When the project was modularized a couple of years ago, the documentation was thrown to the four winds with each of the modules and the care and feeding of the docs pretty much dried up.
• The DSL is inconsistent. Some methods are 'on(host)' others are 'host.thing()' which is pretty darn confusing
• Since it hasn't had a ton of internal care and feeding, it hasn't kept up with all of the things that Bolt can do. On the other hand, it also seems to have solved some issues that are currently being faced by the next generation of proposed testing tech.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2BmGaMcuF0OrPjjkSqJdb%2Bd_Nq9vSN0c3k8wP4L1v2SSZ-7Htw%40mail.gmail.com.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Dan White]

Well put, Trevor. 

I have never used it because I have found it impossible to set up from scratch. 

"Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us."

Bill Waterson (Calvin & Hobbes)

On Apr 20, 2020, at 10:46 AM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CANs%2BFoXCtGHEbo8ENO9Cd_RcAJBjKKHqg67OTpUDvUMoF69kaA%40mail.gmail.com.

[Bram Vogelaar]

I too find it excruciating to use, I have described it in the past as having a rage inducing cli, and the lack of documentation makes it hard to use. Which has driven me to use test-kitchen for doing puppet acceptance testing. (with the added benefit of being able to use inspec profiles).

In my world view the uptake has been very small outside the vox pupuli and puppet supported modules because of this. After Litmus was released i have never considered Beaker again for any real testing needs, since test-kitchen is very very single node only by design.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/62A0053C-FE8E-44F4-9883-AB9918DE418B%40icloud.com.

[Konrad Scherer]

On 4/21/20 3:33 AM, Bram Vogelaar wrote:
> I too find it excruciating to use, I have described it in the past as having a rage inducing cli, and the lack of
> documentation makes it hard to use. Which has driven me to use test-kitchen for doing puppet acceptance testing. (with
> the added benefit of being able to use inspec profiles).
> In my world view the uptake has been very small outside the vox pupuli and puppet supported modules because of this.
> After Litmus was released i have never considered Beaker again for any real testing needs, since test-kitchen is very
> very single node only by design.

My experience was similar. I tried to figure out beaker but was unable to make progress due to the lack of documentation
and examples (almost two years ago now). With test-kitchen and kitchen-puppet I was able to do my testing and it is my
preferred runtime testing framework. I also found the integration with InSpec very useful.

--
Konrad Scherer, MTS, Linux Products Group, Wind River

[Trevor Vaughan]

Bram and Konrad,

I'm personally interested in your perspective on multi-node testing.

In my experience with our stack of around 100 modules, I've found that single node testing is marginally better than spec testing in that it tells you if a service actually starts. However, my project has found that multi-node testing is the only way to know if most services (outside of node-only services) actually function properly.

For instance, we found that perfectly valid rsyslog configurations would cause the daemon to start and appear to be functioning properly but that remote queueing, forwarding to multiple hosts, etc... was horribly broken. This was only possible to troubleshoot with multi-node environments via beaker (without writing something like beaker ourselves from scratch).

I understand that we're the minority but I think that it's a gap that has significant real-world implications across the board.

Note, we do use InSpec for compliance testing and have actually integrated it with Beaker in the simp-beaker-helpers gem.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e3571a4a-036c-5cea-7ae7-c92e8c3d08eb%40windriver.com.