Hiera-puppet with Puppet 3 - FakeScope

[Stefan Goethals]

Hello,

I am testing Puppet 3 with the built in hiera bindings.

The yaml backend seems to work ok, though with renamed variables compared to using the hiera() functions.

What does not seem to work for me is the puppet backend.

I have been using it without problems in puppet 2.7.x.

What i expect to see is the 'Debug: hiera(): Looking for data in p3t::data'like when using hiera().

What i see is this 'Debug: hiera(): Looking for data in fake::data'

As a result the file modules/p3t/manifests/data.pp is never queried when using the 'class p3t ( $somevar = '' )' syntax.

The code is the following.

hiera.yaml

---

:hierarchy:

  - common


:backends:
  - puppet


:puppet:

  datasource: data

 

modules/p3t/manifests/init.pp

# 2 tries... (1 at a time of course..)


class p3t  ( $somevar = '' )
class p3t  ( $somevar = hiera('somevar','') )


{
  notify { "test: $somevar": }
}

puppept_backend.rb

      def hierarchy(scope, override)
        begin

          data_class = Config[:puppet][:datasource] || "data"
        rescue

          data_class = "data"
        end

        
        # added Debug output here...
        Hiera.debug( " scope :#{scope}: " )
        Hiera.debug( " scope.resource :#{scope.resource}: " )
        Hiera.debug( " scope.resource.name :#{scope.resource.name}: " )


        calling_class = scope.resource.name.to_s.downcase

        calling_module = calling_class.split("::").first

server output

# result of using class p3t  ( $somevar = '' ) 


Debug: Automatically imported p3t from p3t into production

Debug: hiera(): Hiera Puppet backend starting
Debug: hiera(): Looking up p3t::somevar in Puppet backend

Debug: hiera():  scope :#<Puppet::Indirector::Hiera::FakeScope:0x7f03abf1fdf0>: 
Debug: hiera():  scope.resource :#<struct #<Class:0x7f03abf22de8> name="fake">: 
Debug: hiera():  scope.resource.name :fake: 

Debug: hiera(): Looking for data in data::common
Debug: hiera(): Found data in class data::common

Debug: hiera(): Looking for data in fake::data
Debug: hiera(): Found data in class fake::data

Compiled catalog for pm3.lan.super-visions.com in environment production in 0.15 seconds




# and using class p3t  ( $somevar = hiera('somevar','') )


Debug: Automatically imported p3t from p3t into production

Debug: hiera(): Hiera Puppet backend starting
Debug: hiera(): Looking up p3t::somevar in Puppet backend

Debug: hiera():  scope :#<Puppet::Indirector::Hiera::FakeScope:0x7f79f7558390>: 
Debug: hiera():  scope.resource :#<struct #<Class:0x7f79f7560b58> name="fake">: 
Debug: hiera():  scope.resource.name :fake: 

Debug: hiera(): Looking for data in data::common
Debug: hiera(): Found data in class data::common

Debug: hiera(): Looking for data in fake::data
Debug: hiera(): Found data in class fake::data

Debug: hiera(): Looking up somevar in Puppet backend
Debug: hiera():  scope :Scope(Class[P3t]): 

Debug: hiera():  scope.resource :Class[P3t]: 
Debug: hiera():  scope.resource.name :P3t: 
Debug: hiera(): Looking for data in data::common

Debug: importing '/etc/puppet-modules/p3t/manifests/data.pp' in environment production
Debug: hiera(): Looking for data in p3t::data
Compiled catalog for pm3.lan.super-visions.com in environment production in 0.16 seconds

Regards,

Stefan - Zipkid - Goethals. 

[Andy Parker]

On Thu, Sep 6, 2012 at 6:45 AM, Stefan Goethals <ste...@zipkid.com> wrote:
> Hello,
>
> I am testing Puppet 3 with the built in hiera bindings.
>

Great! Thanks for trying it out and giving some feedback

> The yaml backend seems to work ok, though with renamed variables compared to
> using the hiera() functions.
>

Yes. The data bindings will construct the fully qualified name of the
variable (so class pt3 ($var) {} looks up pt3::var) in order to make
sure that they are kept unique.

> What does not seem to work for me is the puppet backend.
>

This is because of the name of the variables that are looked up. The
puppet backend will try to lookup pt3::data the variable named
pt3::var. This is a known limitation and the data bindings were not
meant to support the puppet backend.

> I have been using it without problems in puppet 2.7.x.
>

The way you were using it in 2.7.x should continue to work. And your
output below indicates that it still does.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To post to this group, send email to puppe...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-dev?hl=en.

[Stefan Goethals]

Hello,

"This is a known limitation and the data bindings were not
meant to support the puppet backend."

??? Can you elaborate on this please ?

The puppet backend just makes Hiera awesome for supplying defaults.

Puppet not supporting the puppet backend for Hiera sounds weird to me.

Regards,

Stefan - Zipkid - Goethals.

[Andy Parker]

On Fri, Sep 7, 2012 at 1:36 AM, Stefan Goethals <ste...@zipkid.com> wrote:
> Hello,
>

> "This is a known limitation and the data bindings were not
> meant to support the puppet backend."
>
> ??? Can you elaborate on this please ?
>
> The puppet backend just makes Hiera awesome for supplying defaults.
> Puppet not supporting the puppet backend for Hiera sounds weird to me.
>

You are right that it allowed a way of supplying defaults that didn't
have to live inside the class, but that is the same problem that the
data bindings are trying to solve. With the puppet backend you would
have done:

class example::data {
$var = 'foo'
}

class example($param = hiera('var')) { }

That will still work. The point of the data bindings, however, was to
get rid of these explicit calls to hiera. So now you can just put the
default inline:

class example($param = 'foo') { }

That way there is no tying of the manifest code to Hiera. Module
authors can put sane defaults in for parameters, and consumers of the
module can use data bindings to set a value for them. Both the puppet
backend and the data bindings were trying to achieve similar ends:
separate data from code. The data bindings do this by extending the
language and providing an extension point in puppet where you can plug
in your own data lookup services. The puppet backend was a hack that
depended on knowledge about puppet that leaked through to the
implementation of the backend, which was never supposed to be
available. We made the decision to not continue leaking this
information in the data bindings code so that it would be less tied to
the internals of puppet. That had the effect of cutting off the puppet
backend, which we did not consider a problem since we wanted to
promote the pure data bindings going forward.

I'm not sure if this clears anything up. Does it make a bit more sense?

[R.I.Pienaar]

this makes sense in the simple case.

but what if you had something like this:

class example::data {
case $::ofamily {
"RedHat": { $service = "httpd" }
"Debian": { $service = "apache2" }
default: { fail("Please specify a service name using hiera") }
}
}

Here we clearly model different behaviours depending on the context and
clearly state what we do not support via default but still alow the
module user to then decide they would provide this override data
through whatever hierarchy make sense for their site, ENC or just
hardcoding it.

> separate data from code. The data bindings do this by extending the
> language and providing an extension point in puppet where you can
> plug in your own data lookup services. The puppet backend was a hack that
> depended on knowledge about puppet that leaked through to the
> implementation of the backend, which was never supposed to be
> available. We made the decision to not continue leaking this
> information in the data bindings code so that it would be less tied
> to the internals of puppet. That had the effect of cutting off the
> puppet backend, which we did not consider a problem since we wanted to
> promote the pure data bindings going forward.

I think the puppet backend far from being a hack was actually a major
win for module authors and our ability to create reusable modules that
work out of the box.

I think the 'hack' aspect comes from the fact that puppet doesn't cleanly
expose some of the data this backend would need hence the fake scope thing
but conceptually I think if it was realised better this backend still has
solid reason to exist and function

Either way, if we are not going to support a puppet backend the pull
request 1111 needs to remove that backend.

[Andy Parker]

On Fri, Sep 7, 2012 at 9:10 AM, R.I.Pienaar <r...@devco.net> wrote:
>
> but what if you had something like this:
>
> class example::data {
> case $::ofamily {
> "RedHat": { $service = "httpd" }
> "Debian": { $service = "apache2" }
> default: { fail("Please specify a service name using hiera") }
> }
> }
>
> Here we clearly model different behaviours depending on the context and
> clearly state what we do not support via default but still alow the
> module user to then decide they would provide this override data
> through whatever hierarchy make sense for their site, ENC or just
> hardcoding it.
>

The data bindings can handle that by having the Debian and Redhat
cases in hiera (all of the facts are available to the data binding
service) and then the class would have:

class example($service = "UNKNOWN") {
if ($service == "UNKNOWN") {
fail("Please specify a service name")
}
}

I don't think this is quite as clear as the data class form, since
there is now no single place to look to understand the behavior.

>> separate data from code. The data bindings do this by extending the
>> language and providing an extension point in puppet where you can
>> plug in your own data lookup services. The puppet backend was a hack that
>> depended on knowledge about puppet that leaked through to the
>> implementation of the backend, which was never supposed to be
>> available. We made the decision to not continue leaking this
>> information in the data bindings code so that it would be less tied
>> to the internals of puppet. That had the effect of cutting off the
>> puppet backend, which we did not consider a problem since we wanted to
>> promote the pure data bindings going forward.
>
> I think the puppet backend far from being a hack was actually a major
> win for module authors and our ability to create reusable modules that
> work out of the box.
>

True. And the data bindings are trying to take a next step in that direction.

> I think the 'hack' aspect comes from the fact that puppet doesn't cleanly
> expose some of the data this backend would need hence the fake scope thing
> but conceptually I think if it was realised better this backend still has
> solid reason to exist and function
>
> Either way, if we are not going to support a puppet backend the pull
> request 1111 needs to remove that backend.
>

I'm not sure if we want to remove it yet, since it still works and a
lot of people are still using it. We would probably deprecate it
first. I would only want to do that if there was enough consensus that
the data bindings provide enough of an alternative solution.

So we are going to continue supporting the puppet backend via the
hiera functions. The data bindings will "work" with it in the sense
that it doesn't blow up (as it did the first time we tried a setup
with it enabled). At some point during the 3.x cycle it will probably
be deprecated as it becomes clear that its use cases are covered by
the data bindings, which might take some changes to the data bindings.

[R.I.Pienaar]

----- Original Message -----
> From: "Andy Parker" <an...@puppetlabs.com>
> To: puppe...@googlegroups.com
> Sent: Friday, September 7, 2012 5:53:34 PM
> Subject: Re: [Puppet-dev] Hiera-puppet with Puppet 3 - FakeScope
>

> On Fri, Sep 7, 2012 at 9:10 AM, R.I.Pienaar <r...@devco.net> wrote:
> >
> > but what if you had something like this:
> >
> > class example::data {
> > case $::ofamily {
> > "RedHat": { $service = "httpd" }
> > "Debian": { $service = "apache2" }
> > default: { fail("Please specify a service name using hiera")
> > }
> > }
> > }
> >
> > Here we clearly model different behaviours depending on the context
> > and
> > clearly state what we do not support via default but still alow the
> > module user to then decide they would provide this override data
> > through whatever hierarchy make sense for their site, ENC or just
> > hardcoding it.
> >
>
> The data bindings can handle that by having the Debian and Redhat
> cases in hiera (all of the facts are available to the data binding
> service) and then the class would have:
>
> class example($service = "UNKNOWN") {
> if ($service == "UNKNOWN") {
> fail("Please specify a service name")
> }
> }
>
> I don't think this is quite as clear as the data class form, since
> there is now no single place to look to understand the behavior.

this assumes everyone has operating system hierarchies, the data class
example specifically makes the class multi os aware regardless of whatever
hierarchies users have provided.

In your example you'd either need to have a similar case in the example
class - thus coupling the data processing into the logic class - or
force everyone to have OS hierarchies AND supplying all the data you
need.

The point of a in-puppet hierarchy is that it's there by default and
works by default as a last resort in the case that a user doesn't provide
the data.

It's also a good crutch in the case where people want to do crazy logic
on their data using complex if/case/math which they couldn't do using the
yaml/json/whatever data sources as those clearly can't contain any kind
of logic other than that provided by the fact based selection which is
roughly equivalent to a case statement

consider:

class exim::data {
$deliver_queue_load_max = $cpu_cores * 1.2
}

Here we want exim to not deliver mail when the server is too loaded
but what defines 'too loaded' depends on the amount of cores the machine
has. It would be pretty painful to express this in pure-data hiera
as you'd need to dedicate an entire hierarchy to the amount of cores
present, quite a pain

I'd think the only sane alternative is:

class example($service=$example::data::service) {...}

which roughly achieves the same as the hiera-puppet backend in this
specific example, not particularly nice though I'd think we can do
better

OK

[Stefan Goethals]

Hello,

Volcane has explained it pretty well.

What i was really looking forward to with Puppet 3 - Hiera was being able to create modules with defaults for different OS/Versions built right in via the hiera-puppet backend without any dependency on external hiera data thus creating real 'standalone' modules that don't target one single OS.

The way it seems to be implemented 'forces' the addition and 'complicated' distribution method for modules to even remotely achieve this.

This implementation feels like a step back.

Regards,

Stefan Goethals.

[Erik Dalén]

Can't this be solved by this pattern:

class example::params {

case $::ofamily {
"RedHat": { $service = "httpd" }
"Debian": { $service = "apache2" }
default: { fail("Please specify a service name using hiera") }
}
}

class example (
$service = $example::params::service
) inherits example::params {
...
}

Which has the benefit of working in 2.6.3+ and not just 3.0+
Might not look as nice though.

--
Erik Dalén

[Stefan Goethals]

The question here is not really how to work around this but rather what the new feature should add instead of take away.

The thing you do there with the 'inherits' is exactly the behaviour i 'expect' from hiera-puppet but less the 'inherits' call.

Regards,

Stefan.

[Andy Parker]

On Mon, Sep 10, 2012 at 3:33 AM, Stefan Goethals <ste...@zipkid.com> wrote:
> The question here is not really how to work around this but rather what the
> new feature should add instead of take away.

The new features adds the ability to have data looked up by a fully
qualified name. This allows module authors to ship defaults with the
module and module consumers to use another system to redefine what
those defaults should be in their environment.

> The thing you do there with the 'inherits' is exactly the behaviour i
> 'expect' from hiera-puppet but less the 'inherits' call.
>

Is making it more explicit in this manner a bad thing? It seems to me
that keeping things more explicit like this is mainly an improvement
because it doesn't hide as many details.