Jira (PUP-10478) Add getvar Hiera backend

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Enis Inan Summary: Add getvar lookup-key Hiera backend Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Enis Inan

Configuring this would look something like {code} - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data {code} where {{path}} resolves to a {{Hash}} value whose key you're looking up. The allowable {{path}}s are {{trusted.*}} and {{facts.*}} (so anything in the {{trusted}} or {{facts}} hash). All other paths will raise an error. This is to prevent people from using (unsafe) top-scope variables to get Hiera data (bad practice). However, we will still support unsafe access via an {{unsafe}} option. Configuring that would look something like {code} - name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Enis Inan (Jira)]

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Enis Inan

Configuring this would look something like {code} - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data {code}

where {{path}} resolves to a {{Hash}} value whose key you're looking up. The allowable {{path}}s paths are {{trusted.*}} and {{facts.*}} (so anything in the {{trusted}} or {{facts}} hash). All other paths will raise an error. This is to prevent people from using (unsafe) top-scope variables to get Hiera data (bad practice). However, we will still support unsafe access via an {{unsafe}} option. Configuring that would look something like

{code} - name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Enis Inan

Configuring this would look something like {code} - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data {code}

where {{path}} resolves to a {{Hash}} value whose key you're looking up. The allowable {{path}}s are {{trusted.*}} and {{facts.*}} (so anything in the {{trusted}} or {{facts}} hash). All other paths will raise an error. This is to prevent people from using (unsafe) top-scope variables to get Hiera data (bad practice). However, we will still support unsafe access via an {{unsafe}} option. Configuring that would look something like

{code} - name: "unsafe hiera data"     lookup_key: getvar     path: foo.bar     unsafe: true {code}

Add Comment

[Enis Inan (Jira)]

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Enis Inan

Configuring this would look something like {code} - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data {code}

where {{path}} resolves to a {{Hash}} value whose key you're looking up. The allowable paths are {{trusted.*}} and {{facts.*}} (so anything in the {{trusted}} or {{facts}} hash hashes or anything in those hashes ). All other paths will raise an error. This is to prevent people from using (unsafe) top-scope variables to get Hiera data (bad practice). However, we will still support unsafe access via an {{unsafe}} option. Configuring that would look something like

{code} - name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Enis Inan

Configuring this would look something like {code} - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data {code}

where {{path}} resolves to a {{Hash}} value whose key you're looking up. The allowable paths are {{ trusted.* path }} and must start with {{ facts.*}} (so the {{ trusted}} or {{facts}} hashes or anything in since those hashes) are the top-level variables that we've deemed safe to use . All other paths will raise an error. This is People who want to prevent people from using ( use other, unsafe ) top- scope level variables to get Hiera data (bad practice). However, we will still support unsafe access can do so via an {{unsafe}} option. Configuring that would look something like

{code} - name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Enis Inan (Jira)]

Enis Inan updated an issue

Puppet / PUP-10478 Add getvar Hiera backend

Change By: Enis Inan Epic Link: PIE-241

Add Comment

[Rob Braden (Jira)]

Rob Braden updated an issue

Puppet / PUP-10478 Add getvar Hiera backend

Change By: Rob Braden Fix Version/s: PUP 6.15.0

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend

Change By: Reid Vandewiele

Configuring this would look something like {code} - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data {code}

where {{path}}  (or each entry in {{paths}}) resolves to a {{Hash}} value whose key you're looking up containing hiera lookup keys, e . g. { code:java} {   "cygwin::enable": false,   "ntp::servers": [     "time.nist.gov",     "clock.sjc.he.net"   ],   "ntp::restrict": [     "127.0.0.1"   ],   "ntp::disable_monitor": true,   "resource_api::agent::api_version": "1.4.2",   "resource_api::server::api_version": "1.4.2",   "syslog::server": "rsyslog.ops.puppetlabs.net" } {code}  {{ path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error. People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like

{code} - name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like

{code}- --hierarchy:  - name: "ServiceNow hiera data"   lookup_key: getvar   path: trusted.external.servicenow.hiera_data   - name: "User fact Hiera data"    lookup_key: getvar    path: facts.hiera.data {code} where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys, e.g. {code:java}

{   "cygwin::enable": false,   "ntp::servers": [     "time.nist.gov",     "clock.sjc.he.net"   ],   "ntp::restrict": [     "127.0.0.1"   ],   "ntp::disable_monitor": true,   "resource_api::agent::api_version": "1.4.2",   "resource_api::server::api_version": "1.4.2",   "syslog::server": "rsyslog.ops.puppetlabs.net" } {code}  {{path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error. People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like {code}- name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like

{code:yaml} --- hierarchy:

- name: "ServiceNow hiera data"     lookup_key: getvar     path: trusted.external.servicenow.hiera_data   - name: "User fact Hiera data"     lookup_key: getvar

path: facts.hiera.data {code} where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys, e.g.

{code:json}{

"cygwin::enable": false,   "ntp::servers": [     "time.nist.gov",     "clock.sjc.he.net"   ],   "ntp::restrict": [     "127.0.0.1"   ],   "ntp::disable_monitor": true,   "resource_api::agent::api_version": "1.4.2",   "resource_api::server::api_version": "1.4.2",   "syslog::server": "rsyslog.ops.puppetlabs.net" } {code}  {{path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error. People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like {code}- name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like {code :yaml } --- hierarchy:  - name: "ServiceNow hiera data"     lookup_key: getvar     path: trusted.external.servicenow.hiera_data  - name: "User fact Hiera data"    lookup_key: getvar    path: facts.hiera.data

{code} where {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys, e.g.

{code: java json }{

"cygwin::enable": false,   "ntp::servers": [     "time.nist.gov",     "clock.sjc.he.net"   ],   "ntp::restrict": [     "127.0.0.1"   ],   "ntp::disable_monitor": true,   "resource_api::agent::api_version": "1.4.2",   "resource_api::server::api_version": "1.4.2",   "syslog::server": "rsyslog.ops.puppetlabs.net" } {code}  {{path}} must start with {{trusted}} or {{facts}} since those are the top-level variables that we've deemed safe to use. All other paths will raise an error. People who want to use other, unsafe top-level variables can do so via an {{unsafe}} option. Configuring that would look something like {code}- name: "unsafe hiera data"   lookup_key: getvar   path: foo.bar   unsafe: true {code}

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like {code:yaml} --- hierarchy:

- name: " ServiceNow hiera Node-specific Hiera data , via trusted and via fact "     lookup_key: getvar     path     paths :       - " trusted.external.servicenow. hiera_data hieradata"   - "trusted.external.unicorn.hieradata"       - "facts.hieradata"   - name: " User fact Hiera data Some yaml stuff "     lookup_key     data_hash : getvar yaml     path     paths : facts       - "environments/%{trusted . hiera extensions . data pp_environment}.yaml"       - "common.yaml" {code} where for the {{ getvar}} Hiera layer, {{ path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys , e . g. For example: {code:json}{

"cygwin::enable": false,   "ntp::servers": [     "time.nist.gov",     "clock.sjc.he.net"   ],   "ntp::restrict": [     "127.0.0.1"   ],   "ntp::disable_monitor": true,   "resource_api::agent::api_version": "1.4.2",   "resource_api::server::api_version": "1.4.2",   "syslog::server": "rsyslog.ops.puppetlabs.net" }

{code}   To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}} since those are the top-level , because per Hiera best practices, only variables that we've deemed safe not subject to use evaluation-order dependencies or calling scope should be referenced in hiera . All other paths will raise an error yaml . Thus, the following hierarchy configuration should fail: People who want to use other, unsafe top {code:yaml} - level variables can do so via an -- hierarchy:   - name: "Example failure"     lookup_key: getvar     path: "class_scope_variable.hieradata" { code} The error emitted should look something like: { unsafe code } } ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option , which dictates that variables consulted must exist and be in topscope . If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false. {code} Configuring that the backend to allow the use of non-topscope variables would look something like this: {code :yaml } - -- hierarchy:   - name: " unsafe hiera data Example failure "   lookup_key: getvar   path: foo "class_scope_variable . bar hieradata"   unsafe     options : true       topscope_variables_only: false {code}

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like {code:yaml}

hierarchy:   - name: "Node-specific Hiera data, via trusted and via fact"     lookup_key: getvar     paths:       - "trusted.external.servicenow.hieradata"       - "trusted.external.unicorn.hieradata"       - "facts.hieradata"   - name: "Some yaml stuff"     data_hash: yaml     paths:       - "environments/%{trusted.extensions.pp_environment}.yaml"       - "common.yaml" {code} where for the {{getvar}} Hiera layer, {{path}} (or each entry in {{paths}}) resolves to a {{Hash}} value containing hiera lookup keys. For example:

{code:json}{   "cygwin::enable": false,   "ntp::servers": [     "time.nist.gov",     "clock.sjc.he.net"   ],   "ntp::restrict": [     "127.0.0.1"   ],   "ntp::disable_monitor": true,   "resource_api::agent::api_version": "1.4.2",   "resource_api::server::api_version": "1.4.2",   "syslog::server": "rsyslog.ops.puppetlabs.net" } {code}

To provide best-practices guardrails, by default {{path}} must start with a valid top-scope variable such as {{trusted}} or {{facts}}, because per Hiera best practices, only variables not subject to evaluation-order dependencies or calling scope should be referenced in hiera.yaml. Thus, the following hierarchy configuration should fail: {code:yaml} --- hierarchy:

- name: "Example failure"     lookup_key: getvar     path: "class_scope_variable.hieradata" {code} The error emitted should look something like:

{code} ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false. {code} Configuring the backend to allow the use of non-topscope variables would look like this: {code:yaml} --- hierarchy:   - name: "Example failure (bypassed) "     lookup_key: getvar     path: "class_scope_variable.hieradata"     options:       topscope_variables_only: false {code}

Add Comment

[Reid Vandewiele (Jira)]

{ code panel } {{ ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml : getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false. }} { code panel }

Configuring the backend to allow the use of non-topscope variables would look like this: {code:yaml} --- hierarchy:   - name: "Example failure (bypassed)"     lookup_key: getvar     path: "class_scope_variable.hieradata"     options:       topscope_variables_only: false {code}

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like {code:yaml}hierarchy:   - name: "Node-specific Hiera data, via trusted and via fact"

lookup_key     data_hash : getvar

lookup_key     data_hash : getvar

path: "class_scope_variable.hieradata" {code} The error emitted should look something like:

{panel}

{{ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml: getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false.}}

{panel}

Configuring the backend to allow the use of non-topscope variables would look like this: {code:yaml}--- hierarchy:   - name: "Example failure (bypassed)"

lookup_key     data_hash : getvar

path: "class_scope_variable.hieradata"     options:       topscope_variables_only: false {code}

The backend type should be {{data_hash}}. The most technically appropriate type might be {{lookup_key}}, but there are UX considerations here and the name {{data_hash}} helps communicate to the user how this works—the data returned should be like what you find in a yaml file, which is type {{data_hash}}.

Add Comment

[Henrik Lindberg (Jira)]

Henrik Lindberg commented on PUP-10478

Re: Add getvar Hiera backend This does not have to go into puppet, and is not a good idea in general as it makes the result of lookup depend on order of evaluation - it creates a hard to debug and reason about configuration. Features like this was deliberately removed from the hiera 5 implementation because of the mess they create. If you really want to do this, you can write your own backend function (in puppet language for example), it has access to puppet variables in scope. Don't however try to make `paths` be anything other than file paths - there are other options for passing other things to backends - such as "options", and "uri/uris". The file paths are automatically handled with change monitoring etc. If all you want to do is to produce a hash with values from facts, that is super simple to do in a custom backend function. Suggest closing this ticket with Won't Do.

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele updated an issue

Puppet / PUP-10478 Add getvar Hiera backend Change By: Reid Vandewiele

Configuring this would look something like {code:yaml}

hierarchy:   - name: "Node-specific Hiera data, via trusted and via fact"

data_hash: getvar     paths     uris :

data_hash: getvar     path     uri : "class_scope_variable.hieradata"

{code} The error emitted should look something like: {panel} {{ERROR: /etc/puppetlabs/code/environments/production/hiera.yaml: getvar hierarchy level "Example failure" is configured to consult missing or non-topscope variable "class_scope_variable". This is prohibited by the topscope_variables_only=true option, which dictates that variables consulted must exist and be in topscope. If you would like to consult "class_scope_variable" anyway, you must set topscope_variables_only=false.}} {panel} Configuring the backend to allow the use of non-topscope variables would look like this: {code:yaml}--- hierarchy:   - name: "Example failure (bypassed)"

data_hash: getvar     path     uri : "class_scope_variable.hieradata"

options:       topscope_variables_only: false {code} The backend type should be {{data_hash}}. The most technically appropriate type might be {{lookup_key}}, but there are UX considerations here and the name {{data_hash}} helps communicate to the user how this works—the data returned should be like what you find in a yaml file, which is type {{data_hash}}.

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele commented on PUP-10478

Re: Add getvar Hiera backend

Hey Henrik! There was a long discussion in Slack about this that didn't get carried into the ticket description. Re: “makes the result of lookup depend on order of evaluation”: You're right, and we are aware of that concern. We had a long Slack discussion about best practices, user awareness of this kind of issue, what appropriate guard-rails should be, or how aggressively we should protect against users doing this. There is nowhere else in Hiera that programmatically protects against the use of local-scope variables in the hierarchy as far as I know, so the suggested default option topscope_variables_only=true and associated error message actually provides more protection for the proposed getvar backend than we have anywhere else. We considered permitting only $facts or $trusted to be used here by default as well. The current suggestion of any top-scope variable without opting in to what Puppet tells you is a bad idea is because it's usually a safe bet that top-scope variables are set before classes are evaluated, and therefore reasonably permissible. Re: “Don’t however try to make `paths` be anything other than file paths” That sounds like a valuable implementation note to be aware of. For now, updated the description to use "uris". Re: “you can write your own backend function […] that is super simple to do in a custom backend function” This is where we started, actually, and was one of the options considered and discussed in the Slack thread. The main counter-point is disagreement that for users writing a customer back-end function is that simple. It's simple to write a function, but more complicated to know that you can write a backend, choose an appropriate backend type, and put it together. Not that hard, but definitely a barrier. The original impetus for this ticket was implementing an integration with ServiceNow through the trusted_external_command feature. If we want to do this, and only do it ourselves, for ServiceNow, then we just write a servicenow bakcend. But if we want users to be able to do this more generally, including for in-house use of trusted_external_command, we should provide a generalized backend for this in Puppet.

Add Comment

[Maggie Dreyer (Jira)]

Maggie Dreyer commented on PUP-10478

Re: Add getvar Hiera backend

Can we write a supported custom backend that users can use and install if they want to use it? The main argument is against having it be built into Puppet, since we've been moving towards having less and less in core over the last couple of years. Seems like the usability should be similar if it's built-in vs installing an external plugin. Isn't that they way people use hiera-eyaml now?

Add Comment

[Reid Vandewiele (Jira)]

Reid Vandewiele commented on PUP-10478

Re: Add getvar Hiera backend

hiera-eyaml is built in. Works out-of-box, no plugins or additional installation required. The primary use case motivating the discussion around providing it out-of-box (I don’t care if it’s technically core or technically one of the modules we ship with core) is: Provide a way to supply Hiera data through trusted_external_command. Like for eyaml, the ideal solution to this use case should not require users to identify and install extra plugins. It so happens that there are several other ways users inject data into Puppet for which the same motivation (I want this data to go to Hiera) may also exist. It seems like we can solve our primary use case through a generalized backend. Alternative approaches could be to modify the definition of how trusted_external_command works. I haven't explored any design ideas for that yet. We would need to be careful not to overcomplicate the interface if we want to jump that way.

Add Comment

[Henrik Lindberg (Jira)]

Henrik Lindberg commented on PUP-10478

Re: Add getvar Hiera backend

Hiera eyaml is a module that is shipped with puppet agent - there is just a tiny bit of support for it in puppet's repo because puppet's hiera5 implementation needed to override the hiera3 version of it. I.e the actual eyaml logic is in a module. What is suggested here should be in a separate module as well. When hiera 3 is gone completely (in hiera eyaml module as well) then the puppet core specials for it can be cleaned up. We had to support getting variables in scope since hiera 5 needed to be backwards compatible in this respect. What the hiera 5 implementation does is that it monitors if any values that affect the context has changed in such a way that the hierarchy may be affected and if that is detected then the cache is evicted. This leads to worse performance. Thus - using variables in scope leads to worse performance. When using a solution like the proposed you also have to use the lookup cli with `compile` which makes it slower. Without `compile` it would not produce what is produced in real use. A far better solution would be to allow a hiera.yaml to specify a function that is called to produce variables that are derived from initial topscope variables. Those variables (produced by function returning a hash) would be available for interpolation in hiera yaml, and in hiera data, but not be available as variables in the puppet language. This way, there would be no dynamic changes to the context and no messy order of evaluation problems to confuse people. (This was one of the features I wanted to add to hiera 5).

Add Comment

[Rob Braden (Jira)]

Rob Braden updated an issue

Puppet / PUP-10478 Add getvar Hiera backend

Change By: Rob Braden Team: Froyo

Add Comment

[Enis Inan (Jira)]

Enis Inan commented on PUP-10478

Re: Add getvar Hiera backend

Maggie Dreyer We implemented getvar in the CMDB module – https://github.com/puppetlabs/puppetlabs-servicenow_cmdb_integration/blob/master/lib/puppet/functions/servicenow_cmdb_integration/getvar.rb – so it's not urgent. But it would probably be nice-to-have to facilitate Hiera lookups on the trusted['external'] variable. /cc Reid Vandewiele

Add Comment

[Bryan Jen (Jira)]

Bryan Jen updated an issue

Puppet / PUP-10478 Add getvar Hiera backend

Change By: Bryan Jen Epic Link: PIE- 241 578

Add Comment