Jira (PUP-8234) layer 3 data sources can not be overriden for unit testing purposes

1 view
Skip to first unread message

Brian Vanderbusch (JIRA)

unread,
Dec 7, 2017, 1:47:02 PM12/7/17
to puppe...@googlegroups.com
Brian Vanderbusch created an issue
 
Puppet / Bug PUP-8234
layer 3 data sources can not be overriden for unit testing purposes
Issue Type: Bug Bug
Affects Versions: PUP 5.3.3
Assignee: Thomas Hallgren
Components: Hiera & Lookup, Test Infrastructure
Created: 2017/12/07 10:46 AM
Priority: Normal Normal
Reporter: Brian Vanderbusch

I've been very unseccesful in attempting to unit test a module that includes a hiera-eyaml data source defined in my layer 3 hiera.yaml defined hierarchy.

I've tried overriding the EYAML_CONFIG, configurign rspec `hiera_config` to use a different hiera.yaml in my test fixture, providing facts/params values directly in my .spec, and even providing a dockerized environment complete with fake keys in an attempt to replicate the conditions of all real environemnts in a a way that unit testing can be done.

It seems that when it comes to overrides, nothing is capable of overriding the data-sources defined in the modules `hiera.yaml`. This means that any data source which relies on the puppetserver will basically make unit testing impossible.

I've opened issues in both rspec-puppet and hiera-eyaml, but it's likely the heart of this problem may rest in puppet core language, so I'm opening an issue here as well.

Here are the issues opened with rspec-puppet and hiera-eyaml:

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Thomas Hallgren (JIRA)

unread,
Dec 7, 2017, 5:11:02 PM12/7/17
to puppe...@googlegroups.com
Thomas Hallgren commented on Bug PUP-8234
 
Re: layer 3 data sources can not be overriden for unit testing purposes

Brian Vanderbusch, you write: "It seems that when it comes to overrides, nothing is capable of overriding the data-sources defined in the modules `hiera.yaml`". I'm not sure what you mean by overrides and perhaps I misunderstand completely but there is no such concept in the hiera configuration. We have three layers, global, environment, and module. They are searched in order and depending on merge strategy, the search may either stop at first found (this is the default and might be perceived as override), or be a merge of all data found for a specific key.

Can you please provide an environment that exhibits the problem or problems in this ticket together with expected and actual results? A reproducer is always clarifying and also great to have when verifying once the problem is fixed.

Brian Vanderbusch (JIRA)

unread,
Dec 7, 2017, 8:45:02 PM12/7/17
to puppe...@googlegroups.com

Unfortunately all the examples I have at the moment are private and
proprietary.

I left some specific repro/technical description s in the GitHub issues I
referenced though.

Basically, I have data sources in a puppet 5 module that require the puppet
server to be involved. Specifically, one of these is the now native eyaml
lookup.

U fortunately, whenever I give rspec a value for hiera_config to hiera.yaml
file that doesn't contain eyaml sources in the hierarchy, that value is
considered a layer 2 hiera configuration, and therefore overridden by the
module's hiera.yaml file.

All I'm trying to do is write unit tests, but I have no way to provide fake
data without puppet trying to execute those external lookups, even though
they aren't needed because I manually provided values to APL.

On Dec 7, 2017 4:11 PM, "Thomas Hallgren (JIRA)" <

Thomas Hallgren (JIRA)

unread,
Dec 7, 2017, 9:44:02 PM12/7/17
to puppe...@googlegroups.com

This can be done in one of two ways. Either you provide a simple and concise sample that exhibits the problem you describe, or I will need to create such a sample based on guesses of what your environment might look like and what you are trying to accomplish. The advantages with the former is that it doesn't involve the guessing part and once it works, it will have solved your specific problem. I'm not asking for your proprietary production environment, just a sample that contains the parts needed to show the problem.

Brian Vanderbusch (JIRA)

unread,
Dec 7, 2017, 10:07:02 PM12/7/17
to puppe...@googlegroups.com

This I can do, but I figured that surely by now someone had written a
single example of how to unit test a module that is now native to puppet.

I figured that first and foremost this was a documentation issue.

It seems like hiera-eyaml is now a native puppet lib, so wouldn't there be
some rspec-puppet test somewhere I could start with just to make sure I'm
not screwing something up out of my own ignorance?

I mean, it has been tested, right?

On Dec 7, 2017 8:44 PM, "Thomas Hallgren (JIRA)" <

Thomas Hallgren (JIRA)

unread,
Dec 7, 2017, 10:21:02 PM12/7/17
to puppe...@googlegroups.com

No, hiera-eyaml is not a native puppet lib. Puppet just provides an interface to the hiera eyaml gem. There are several unit tests for that interface. Both for version 5 and for version 3. Tests include environment and module configurations with different options (different certs in different modules).

Brian Vanderbusch (JIRA)

unread,
Dec 7, 2017, 10:50:03 PM12/7/17
to puppe...@googlegroups.com

I'll start here tomorrow, and close the ticket, or update it with a simple
repro depending on how things go. Thanks for pointing in the right
direction.

On Dec 7, 2017 9:21 PM, "Thomas Hallgren (JIRA)" <

Henrik Lindberg (JIRA)

unread,
Dec 8, 2017, 3:55:02 AM12/8/17
to puppe...@googlegroups.com

I read this ticket as that there is a problem with the testing infrastructure (and subsequently in the hiera implementation to facilitate testing). In essence I think there should be an official mechanism that allows convenient mocking of a module's hiera.yaml. That is now somewhat difficult.

Next level down gets trickier - you want to test your module's hiera.yaml, and it has references to data and needs keys. When those are secret it also becomes difficult to test. You could do this with a different data set containing only non-secret things, but you want to make sure that eyaml (or similar) is actually decrypting - i.e. you want your mocking to be as close as possible to the real thing. It is however only possible to test a module's real hiera config and data to a degree if it contains secrets. I imagine some kind of production acceptance testing could be an alternative.

Brian Vanderbusch (JIRA)

unread,
Dec 8, 2017, 10:10:02 AM12/8/17
to puppe...@googlegroups.com

You're right. I was trying to keep the thread going when I should have been sleeping.

The problem/goal isn't to test the eyaml data-sources, it's to unit test the rest of the module. I have 2 class params out of about 40 that have eyaml data sources. However, when I use any fact in my hiera.yaml that is part of the hierarchy for the eyaml data source, then my unit tests fail, even if i'm not testing the eyaml params, and even if I provided absolute string values.

Basically, i've only been able to run the most simple unit tests on my module because everything i try winds up trying to execute the eyaml data source.

I'm going to put together a working example for you. Should have it done in an hour or so.

Brian Vanderbusch (JIRA)

unread,
Dec 8, 2017, 1:51:02 PM12/8/17
to puppe...@googlegroups.com

Ok, i managed to provide a repro, here: https://github.com/LongLiveCHIEF/eyaml-rspec-example-failure

The master branch is project state before adding a eyaml data source, and then there's a branch for `add-eyaml-data-source` which contains the reproduced error.

I may have actually made some progress on the pinpoint the issue in the process though, as I discovered that even my branch passed, until i added the very last commit, which added a `lookup` hiera interpolation in one of the hiera data sources: https://github.com/LongLiveCHIEF/eyaml-rspec-example-failure/blob/add-eyaml-data-source/data/sector/alpha.yaml#L5

For convenience, I've dockerized this entire reproduction, and assuming you have docker installed, you can run it simply with:

```
git clone g...@github.com:LongLiveCHIEF/eyaml-rspec-example-failure.git
cd eyaml-rspec-example-failure
make build
make test
// will build the entire env, install all testing deps/puppet/etc, and then execute pdk test unit
```
Then to view the repoduction, just checkout the `add-eyaml-data-source` and run `make test` again.

However, I found that when I initially tried to reproduce the issue, I wasn't getting any errors, even using a fake pkcs7 key pair. (note, I'm not testing any of the eyaml decryption outcomes)

so, if you are on the `add-eyaml-data-source` branch, and rewind one commit, and run `make test` again... everything works.

This docker environment contains both a set of "fake" keys, to simulate what i'm doing to test, as well as the real keys used to create the encrypted value in `secrets/sector/alpha.eyaml`.

I guess what my next question is, and I using the `lookup` improperly in the `data/sector/alpha.yaml` file? Or am i using it correctly, and I have now uncovered the function to mock to prevent the `bad interpolation` error?

Brian Vanderbusch (JIRA)

unread,
Dec 8, 2017, 1:52:03 PM12/8/17
to puppe...@googlegroups.com

p.s. ignore the README content in the example repository. It was generated by pdk and means absolutely nothing.

Brian Vanderbusch (JIRA)

unread,
Dec 11, 2017, 11:54:02 AM12/11/17
to puppe...@googlegroups.com

Thomas Hallgrenis it possible to change the title of the issue? Looks like with the repro I've submitted, I've narrowed it down to issues with the given params in rspec not being used with in-data lookup functions.

Maybe it would even be better to close this issue and determine if we should open a new one here, or if this is perhaps a hiera-eyaml issue?

Henrik Lindberg (JIRA)

unread,
Dec 11, 2017, 12:13:03 PM12/11/17
to puppe...@googlegroups.com

It is fine to change the title of this ticket (go ahead). I also don't mind it being open until some kind of resolution is made regarding new ticket elsewhere etc.

Brian Vanderbusch (JIRA)

unread,
Dec 11, 2017, 12:16:03 PM12/11/17
to puppe...@googlegroups.com

I can't figure out how to change the title. I was guessing that's something only you can do?

Henrik Lindberg (JIRA)

unread,
Dec 12, 2017, 10:52:03 AM12/12/17
to puppe...@googlegroups.com

I can change it by clicking on it - what do you want it to say?

Thomas Hallgren (JIRA)

unread,
Jan 22, 2018, 7:47:02 AM1/22/18
to puppe...@googlegroups.com

Brian Vanderbusch, are you unable to click on the title and change it? If so, let me know what would you like the title to be and I'll change it for you.

Josh Cooper (JIRA)

unread,
Jan 23, 2018, 8:15:03 PM1/23/18
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Jan 23, 2018, 8:15:05 PM1/23/18
to puppe...@googlegroups.com

David McTavish (Jira)

unread,
Dec 6, 2021, 2:36:02 PM12/6/21
to puppe...@googlegroups.com
David McTavish updated an issue
Change By: David McTavish
Labels: final_triage unit_test
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

David McTavish (Jira)

unread,
Dec 6, 2021, 2:36:02 PM12/6/21
to puppe...@googlegroups.com

Josh Cooper (Jira)

unread,
Jan 13, 2022, 1:45:02 AM1/13/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-8234
 
Re: layer 3 data sources can not be overriden for unit testing purposes

This ticket hasn't been updated in while and it's not entirely clear what the issue is, so I'm going to close. If this is still an issue, please reopen with instructions on how to reproduce.

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages