representing undef in hiera?

[Christopher Wood]

How would I represent an undef in hiera such that a class parameter will be populated as undef? Like this, but in hiera:

#class testing::cwood1 ( $param1 = undef ) {
class testing::cwood1 ( $param1 ) {
if str2bool($param1) {
notify { 'mn':
message => "param1 is ${param1}"
}
}
}

I'm dealing with a config file template where most options are always turned off. I'm also trying to get our puppet agent runs to fail with catalog compilation errors if they can't look up a hiera value, so there are no default values in the class parameters. (Exploring if catalog compilation failure is safer than the possibility of populating a config with default values.)

If I could list a bunch of obvious 'undef' style items in hiera that might be handier than listing 'false', given that the meaning of 'configitem=false' is different in this config than an absent configitem.

Details:

In the node yaml doing ~ or null both give me the same result.

testing::cwood1::param1: ~

But that parameter isn't looked up.

Feb 9 10:18:55 pt1 puppet-agent[22696]: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass param1 to Class[Testing::Cwood1] at /etc/puppet/environments/production/modules/testing/manifests/init.pp:3 on node pt1.hostopia.com

I've checked that the yaml is readable by ruby and the appropriate key shows up with pp as

"testing::cwood1::param1"=>nil}

so not sure what is up here.

When I use the empty string in the yaml the puppet catalog applies but I do not see anything in the log (so the class parameter was accepted).

testing::cwood1::param1: ''

Not really sure if I'm just mucking about with edge cases and should special-case my template or what. (http://yaml.org/type/null.html)

[jcbollinger]

On Monday, February 9, 2015 at 10:37:39 AM UTC-6, Christopher Wood wrote:
 

I'm dealing with a config file template where most options are always turned off. I'm also trying to get our puppet agent runs to fail with catalog compilation errors if they can't look up a hiera value, so there are no default values in the class parameters. (Exploring if catalog compilation failure is safer than the possibility of populating a config with default values.)

If I could list a bunch of obvious 'undef' style items in hiera that might be handier than listing 'false', given that the meaning of 'configitem=false' is different in this config than an absent configitem.

You're having trouble because you're being inconsistent.  If it is meaningful and supported for a class parameter to be absent, then why are you trying to force it to be present in the data?  On the other hand, if you mean to force a value to be specified for the parameter, then why are you trying to affirmatively specify no value?

Hiera aside, I don't much care for your design here.  Either allow the parameter to be unspecified, including via Hiera, or designate an actual value the class parameter can take to specify that the corresponding config option is to be omitted from the target file ("absent" would probably be the conventional choice for that).


John

[Christopher Wood]

On Tue, Feb 10, 2015 at 06:52:05AM -0800, jcbollinger wrote:
> On Monday, February 9, 2015 at 10:37:39 AM UTC-6, Christopher Wood wrote:
>  
>
> I'm dealing with a config file template where most options are always
> turned off. I'm also trying to get our puppet agent runs to fail with
> catalog compilation errors if they can't look up a hiera value, so there
> are no default values in the class parameters. (Exploring if catalog
> compilation failure is safer than the possibility of populating a config
> with default values.)
>
> If I could list a bunch of obvious 'undef' style items in hiera that
> might be handier than listing 'false', given that the meaning of
> 'configitem=false' is different in this config than an absent
> configitem.
>
> You're having trouble because you're being inconsistent.  If it is
> meaningful and supported for a class parameter to be absent, then why are

I don't think I phrased that too well. Inside the config file any given item may be present in its default or non-default values but it's customary here that defaults are left out of this config file. Think never specifying "Listen 80" in httpd.conf and having Apache listen on port 80 anyway.

> you trying to force it to be present in the data?  On the other hand, if
> you mean to force a value to be specified for the parameter, then why are
> you trying to affirmatively specify no value?

There are two behaviours when a class parameter hiera lookup fails to find anything, the one with a default value and the one without a default value.

class this ( $param = undef ) {
# stuff
}

class that ( $param ) {
# stuff
}

With the former, the true/false handling would leave a config parameter out of the config file. With the latter, the failure mode is catalog compilation breakage, something which fails an agent run but whose consequences are more immediately visible. These days I prefer making the results of affirmative decisions immediately visible to presuming the continued viability of a past default.

> Hiera aside, I don't much care for your design here.  Either allow the
> parameter to be unspecified, including via Hiera, or designate an actual
> value the class parameter can take to specify that the corresponding
> config option is to be omitted from the target file ("absent" would
> probably be the conventional choice for that).

I thought so too, and then wondered if it's worth keeping a consistent interface between modules. If one class parameter default is undef and another is '80', that means the admin building a new role gets to figure out which parameters require their attention and which can be left as defaults. "Pay attention to all the parameters or catalog compilation fails" seems a safer message, especially with so many separate hiera keys across our installation.

[puppet exp] $ grep -rh '^[a-z]' hieradata | awk '{print $1}' | sort -u | wc -l
390
[puppet exp] $ grep -rh '^[a-z]' hieradata | awk '{print $1}' | wc -l
2561

(Not all 390 separate keys apply to each server, of course.)

> John
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/puppet-users/8c7bf563-8155-483b-921e-9c2ee51f0c80%40googlegroups.com.
> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com
> 2. https://groups.google.com/d/msgid/puppet-users/8c7bf563-8155-483b-921e-9c2ee51f0c80%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[jcbollinger]

On Tuesday, February 10, 2015 at 9:48:50 AM UTC-6, Christopher Wood wrote:

On Tue, Feb 10, 2015 at 06:52:05AM -0800, jcbollinger wrote:
>    On Monday, February 9, 2015 at 10:37:39 AM UTC-6, Christopher Wood wrote:
>     
>
>      I'm dealing with a config file template where most options are always
>      turned off. I'm also trying to get our puppet agent runs to fail with
>      catalog compilation errors if they can't look up a hiera value, so there
>      are no default values in the class parameters. (Exploring if catalog
>      compilation failure is safer than the possibility of populating a config
>      with default values.)
>
>      If I could list a bunch of obvious 'undef' style items in hiera that
>      might be handier than listing 'false', given that the meaning of
>      'configitem=false' is different in this config than an absent
>      configitem.
>
>    You're having trouble because you're being inconsistent.  If it is
>    meaningful and supported for a class parameter to be absent, then why are

I don't think I phrased that too well. Inside the config file any given item may be present in its default or non-default values but it's customary here that defaults are left out of this config file. Think never specifying "Listen 80" in httpd.conf and having Apache listen on port 80 anyway.

I understand.  Really.  And my question stands.  In your example, if you're ok with a value for the 'port' parameter not being specified to your class, as embodied by having some kind of appropriate handling in the event that Hiera should somehow affirmatively declare that parameter unspecified, then 'port' is an optional parameter.  Optional parameters are implemented in Puppet by assigning default [non-]values to them in the parameter list.

 

>    you trying to force it to be present in the data?  On the other hand, if
>    you mean to force a value to be specified for the parameter, then why are
>    you trying to affirmatively specify no value?

There are two behaviours when a class parameter hiera lookup fails to find anything, the one with a default value and the one without a default value.

class this ( $param = undef ) {
  # stuff
}

class that ( $param ) {
  # stuff
}

With the former, the true/false handling would leave a config parameter out of the config file. With the latter, the failure mode is catalog compilation breakage, something which fails an agent run but whose consequences are more immediately visible. These days I prefer making the results of affirmative decisions immediately visible to presuming the continued viability of a past default.

That's reasonable.  The fact remains that Hiera does not inherently provide any generic way to affirmatively declare that you're not specifying a parameter value.  Even if it did, it would be unlikely to have the semantics you're after, because -- I repeat -- what you're asking is inconsistent.  If a parameter is required, then there is no practical difference between its value being accidentally unspecified and its value being affirmatively unspecified: either way, a required parameter is unspecified.

Note well: Puppet DSL's 'undef' keyword does not represent a value.  It represents the absence of any value.  If you write a resource-style class declaration that associates keyword 'undef' with a class parameter, it is equivalent to not specifying that parameter at all.  If you assign the keyword 'undef' as a class parameter default, it marks the parameter optional with no default value -- if no actual value is given for that parameter then it will not have a defined value.  If you override a resource parameter to be undef, it makes the resource be treated as if no value were explicitly assigned (in particular, that parameter takes its default value, if any).

 

>    Hiera aside, I don't much care for your design here.  Either allow the
>    parameter to be unspecified, including via Hiera, or designate an actual
>    value the class parameter can take to specify that the corresponding
>    config option is to be omitted from the target file ("absent" would
>    probably be the conventional choice for that).

I thought so too, and then wondered if it's worth keeping a consistent interface between modules. If one class parameter default is undef and another is '80', that means the admin building a new role gets to figure out which parameters require their attention and which can be left as defaults. "Pay attention to all the parameters or catalog compilation fails" seems a safer message, especially with so many separate hiera keys across our installation.

Your admins are responsible for every parameter anyway.  You can make Puppet enforce that they give at least nominal attention to each, but I don't think it gains you much ground.  Were I an admin working under such a system, I would be inclined to bootstrap a new role by copying data from one or more existing ones, or maybe to create a set of default data for the purpose.  Either one renders your effort moot.  It will be hard to prevent admins working around your enforcement approach, and putting such a regime in place sends a message that you do not trust them to do their jobs correctly.  (And even if you don't, I advise you to avoid sending such a message.)  If you ask me, the message should be "Ensure all the parameters have correct values, else machines will be misconfigured".  Focus your people on the result rather than on the process, and test the result.

With that said, if you want to enforce that there be some kind of physical manifestation of a value [non-]choice for every class parameter, enforced by compilation failure when such a manifestation is not present, then you can avoid all class parameter defaults, and implement some actual value that consistently and generically represents non-specification of any real value.  Something unlikely to be desired as a real value for any class parameter would be wise, such as maybe '__absent__' or '__default__'.  You can move any actual default values into the class bodies.

John

[Clayton O'Neill]

In my experience using tilde as the value will pass through the equivalent of undef.  That said, I agree with all the comments about making optional arguments actually optional, and not forcing them to be defined in hiera.

For example:

---
key: ~

[James Olin Oden]

Hey Clayton is this documented somewhere (i.e. the us of ~ like this,
or the use of ~ as a value in hiera file?). Thanks...James

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-users/972e276e-c908-4666-a7ed-3d7de084f84c%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[James Olin Oden]

Never, mind I was looking in the wrong place. It's in the YAML spec
(or it's a YAML thing). Tilde represents NULL (or is equivalent
too):

http://www.yaml.org/spec/1.2/spec.html#id2805071

Thanks...James

[ess]

On Monday, February 9, 2015 at 11:37:39 AM UTC-5, Christopher Wood wrote:

How would I represent an undef in hiera such that a class parameter will be populated as undef?

John's doing a better job than I would of talking about the overall theory, but we ran into something similar and I found this bug / feature that you might find interesting if you haven't already seen it:

https://projects.puppetlabs.com/issues/20923

We are using the puppetlabs-lvm module and needed to set the size value to be undef / use extents for a specific filesystem.  Since it didn't seem possible to do that with Hiera, we moved that functionality back into code.

[Christopher Wood]

(It seems my gears ground slowly on this thing, It took me this long to think about it.)

Aha, now I get it, I think. In my case I was trying to have all the possible choices (absence, 80, 8080, for example) represented in one "place" (hiera). And nil/undef is not the correct concept for the "absence" choice.

This was my misconception, it turns out. I took 'undef' to be intended as 'put nothing here' rather than 'I did not specifically select anything'.

>  
>
> >    Hiera aside, I don't much care for your design here.  Either allow
> the
> >    parameter to be unspecified, including via Hiera, or designate an
> actual
> >    value the class parameter can take to specify that the
> corresponding
> >    config option is to be omitted from the target file ("absent" would
> >    probably be the conventional choice for that).
>
> I thought so too, and then wondered if it's worth keeping a consistent
> interface between modules. If one class parameter default is undef and
> another is '80', that means the admin building a new role gets to figure
> out which parameters require their attention and which can be left as
> defaults. "Pay attention to all the parameters or catalog compilation
> fails" seems a safer message, especially with so many separate hiera
> keys across our installation.
>
> Your admins are responsible for every parameter anyway.  You can make
> Puppet enforce that they give at least nominal attention to each, but I
> don't think it gains you much ground.  Were I an admin working under such
> a system, I would be inclined to bootstrap a new role by copying data from
> one or more existing ones, or maybe to create a set of default data for
> the purpose.  Either one renders your effort moot.  It will be hard to
> prevent admins working around your enforcement approach, and putting such
> a regime in place sends a message that you do not trust them to do their
> jobs correctly.  (And even if you don't, I advise you to avoid sending

The "admin building a new role", I should clarify, mainly means me. After a few hitches with implicit things in hiera that I should have made explicit I'm trying to figure out how to expose my errors earlier in the process. I already have a few new tests in place but I feel it's not yet enough.

I appreciate your time, this is much clearer for me now.

> such a message.)  If you ask me, the message should be "Ensure all the
> parameters have correct values, else machines will be misconfigured". 
> Focus your people on the result rather than on the process, and test the
> result.
>
> With that said, if you want to enforce that there be some kind of physical
> manifestation of a value [non-]choice for every class parameter, enforced
> by compilation failure when such a manifestation is not present, then you
> can avoid all class parameter defaults, and implement some actual value
> that consistently and generically represents non-specification of any real
> value.  Something unlikely to be desired as a real value for any class
> parameter would be wise, such as maybe '__absent__' or '__default__'.  You
> can move any actual default values into the class bodies.
>
> John
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> [2]https://groups.google.com/d/msgid/puppet-users/2fab11a3-4167-48f2-8813-1e606ed38360%40googlegroups.com.

> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com

> 2. https://groups.google.com/d/msgid/puppet-users/2fab11a3-4167-48f2-8813-1e606ed38360%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout

[Christopher Wood]

That's where I wound up too.

I did file PUP-3992 but that ended up being an error message suggestion rather than having a better point.

https://tickets.puppetlabs.com/browse/PUP-3992

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [1]puppet-users...@googlegroups.com.
> To view this discussion on the web visit

> [2]https://groups.google.com/d/msgid/puppet-users/a4c38428-6027-498f-905b-edb807c1e134%40googlegroups.com.

> For more options, visit [3]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:puppet-users...@googlegroups.com

> 2. https://groups.google.com/d/msgid/puppet-users/a4c38428-6027-498f-905b-edb807c1e134%40googlegroups.com?utm_medium=email&utm_source=footer
> 3. https://groups.google.com/d/optout