Provider testing

[David Lutterkort]

One thing that just occurred to me is that since we have pretty tight contracts around a provider's behavior, it ought to be possible to build a test harness that is data driven, and therefore simplifies testing that a provider does what it is supposed to do.

I've mocked up what that could look like in YAML in this gist - it would be strictly black box testing, but might be better than (a) having no tests at all or (b) making provider writers dive into test frameworks. I've mocked this up in YAML, but it would also possible to come up with some sort of DSL for this; depends on what value that DSL would add though.

David

[David Schmitt]

I've been thinking about this in the context of the resource API work I've been doing, and the main issue seems to be around the requirement of having a real system to run those tests on, whereas quite a bit of the code is data-transformational in nature and should not require actual system interaction for testing. For providers talking to APIs, something like webmock or vcr can be useful, but I don't know a good general stubbing layer for system interaction.

The other fundamental problem this type of testing runs into is that a harness like this can test the correct interface behaviour, but not the effects of the code. That is, the harness can test that get and set accept and return the right values, but it'll not be able to validate that the resultant system state is what it should be (e.g. was the correct service started).

That said, I still firmly believe that there is quite a bit of value to be had in a test harness like this. Adding some quickcheck style intelligence, or generative testing, could go a long way towards making the interface testing less tedious. It'll not fully replace well-thought out unit and acceptance tests though.

--
You received this message because you are subscribed to the Google Groups "libral" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libral+unsubscribe@googlegroups.com.
To post to this group, send email to lib...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libral/CAHN%2BA%2BXF5S1dUmuEr1dOfxoZazhUQ6xMja%3Dnvj9zxkdyDpiKMQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[David Lutterkort]

I had a go at putting something together in this PR - I moved away from writing tests in pure YAML since that seemed too clunky and opted for writing my own crappy DSL. One of the shortcomings of pure YAML is that you want to allow some light computation (like getting the current user's name from ENV) that would just require some homegrown expression language.

The sample test gives a hint of where this could go; though I am now leaning more towards using an existing test framework with lots of predefined helpers. The homegrown crappy DSL will invariably become too hard to understand, I fear.

But even in this rudimentary form, it's been really helpful in uncovering problems, both with the provider, as with the general framework; one fun question I stumbled over: validating a resource with 'ensure=absent' against the provider schema is not really what anybody should be required to do.

David

To view this discussion on the web visit https://groups.google.com/d/msgid/libral/CALF7fHYKRe6UrNgWAQAL1rGv_kY0Q4h-1EfkZX1QRFt5%3Dq203w%40mail.gmail.com.

[David Schmitt]

On 5 April 2017 at 23:46, David Lutterkort <lut...@puppet.com> wrote:

I had a go at putting something together in this PR - I moved away from writing tests in pure YAML since that seemed too clunky and opted for writing my own crappy DSL. One of the shortcomings of pure YAML is that you want to allow some light computation (like getting the current user's name from ENV) that would just require some homegrown expression language.

The sample test gives a hint of where this could go; though I am now leaning more towards using an existing test framework with lots of predefined helpers. The homegrown crappy DSL will invariably become too hard to understand, I fear.

Yes, please go with a established framework. e.g. rspec is well-used in the modules community. There will have to be more than one level of testing and test tooling anyways:

* full-system interaction/integration testing. At least the developer will have to specify existing resources and valid configurations for the test suite

* unit testing the individual parts of the provider. This will have to happen in the provider's native language's ecosystem, as it'll require mocking/stubbing at a very intimate level.

But even in this rudimentary form, it's been really helpful in uncovering problems, both with the provider, as with the general framework; one fun question I stumbled over: validating a resource with 'ensure=absent' against the provider schema is not really what anybody should be required to do.

Yeah, I expect quite a few checks that we could do automatically (e.g. idempotency, query consistency, etc) with little to no investment from the developer.

David