Status of Data in modules

[Eric Sorenson]

Thanks to everyone who kicked the tires on the experimental data in modules feature included in Puppet 3.3.0. We got a lot of feedback, some cool proof-of-concept modules, and a definitive conclusion to the experiment.

The idea of including a module-specific hiera backend is centered around one primary use case: replacing the 'params class pattern', a common idiom in Puppet modules that's described in the [Using Parameterized Classes][param-classes] guide. The problem that most testers ran into though is that for non-trivial modules they ended up having to re-implement the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious ways. The solutions to this led to more contortions until we'd ended up with the ability to execute parser functions in the right-hand-side of a yaml value. So something which started out trying to help separate data from code ended up putting code back into data!

Additionally, even after multiple attempts to simplify the surface area and user experience with the bindings system (described in ARM-9) that underlay the data-in-modules implementation, users still found its complexity daunting. There are some important bits of scaffolding (like an actual type system for Puppet!) that will prove valuable as more of the future parser and evaluator work that Henrik is building makes its way into the product, but in the final analysis the data in modules feature was the wrong vehicle to introduce them.

Refocusing on the problems users were trying to solve (and here I have to give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] module) and the problems with the 'params' pattern lent some clarity. We've gotten into a situation of disparity with regard to hiera and data bindings, because data bindings enable module _users_ to use their site-wide hiera data but don't provide moduel _authors_ the same affordance. But rather than introduce additional complexity, we can close the gap for existing code patterns.

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet backend for a value of 'classname::variable' in the file 'modules/classname/manifests/params.pp', which simplifies class definition and provides consistency with other hiera backends. As a module author, you'd still leave your logic for variables in params.pp, but they'd be implicitly looked up via data bindings as the class is declared, after consulting site-wide hiera.
- remove the user-facing '--binder' functionality
- fix known problems with the hiera-puppet lookups ([Redmine 15746][15746], namely, but if there are others that are important to you please speak up!)

To show how this would work, I'll rework the ['smart parameter defaults' example][param-classes] I linked above, with my commentary behind `##` comments:

# /etc/puppet/modules/webserver/manifests/params.pp

class webserver::params { ## nothing changes here...
$packages = $operatingsystem ? {
/(?i-mx:ubuntu|debian)/ => 'apache2',
/(?i-mx:centos|fedora|redhat)/ => 'httpd',
}
$vhost_dir = $operatingsystem ? {
/(?i-mx:ubuntu|debian)/ => '/etc/apache2/sites-enabled',
/(?i-mx:centos|fedora|redhat)/ => '/etc/httpd/conf.d',
}
}

# /etc/puppet/modules/webserver/manifests/init.pp

class webserver( ## inheritance is gone, and
$packages, ## data bindings look up the defaults
$vhost_dir ## as webserver::params::vhost_dir
) {

package { $packages: ensure => present }

file { 'vhost_dir':
path => $vhost_dir,
ensure => directory,
mode => '0750',
owner => 'www-data',
group => 'root',
}
}

# /etc/puppet/manifests/site.pp

node default {
class { 'webserver': } ## no params needed, they're in hiera

## then in one of my site-wide hiera layers, I can override
## the value without modifying the module or class declaration

# /etc/puppet/hieradata/snowflake.domain.com.yaml
webserver::vhost_dir: '/some/other/dir'

This way the module author (who probably has the most work to do and needs the expressiveness of the DSL) can provide default data, but site administrators can still override it using mechanisms they're already using.

Note too that this is the next iteration, not necessarily the end state. It's super important to get this right because the whole community is going to have to live with it for a long time; those of you out here on the bleeding edge willing to risk some skin to make something awesome are critical to making that happen.


Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles


[puppetlabs-ntp]: https://github.com/apenney/puppetlabs-ntp/tree/data-in-modules
[startrek]: https://github.com/pro-puppet/puppet-module-startrek
[param-classes]: http://docs.puppetlabs.com/guides/parameterized_classes.html#appendix-smart-parameter-defaults
[15746]: https://projects.puppetlabs.com/issues/15746

[Nan Liu]

On Fri, Oct 11, 2013 at 1:09 PM, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Thanks to everyone who kicked the tires on the experimental data in modules feature included in Puppet 3.3.0. We got a lot of feedback, some cool proof-of-concept modules, and a definitive conclusion to the experiment.

Thanks for sending a summary.

 

The idea of including a module-specific hiera backend is centered around one primary use case: replacing the 'params class pattern', a common idiom in Puppet modules that's described in the [Using Parameterized Classes][param-classes] guide. The problem that most testers ran into though is that for non-trivial modules they ended up having to re-implement the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious ways. The solutions to this led to more contortions until we'd ended up with the ability to execute parser functions in the right-hand-side of a yaml value. So something which started out trying to help separate data from code ended up putting code back into data!

Additionally, even after multiple attempts to simplify the surface area and user experience with the bindings system (described in ARM-9) that underlay the data-in-modules implementation, users still found its complexity daunting. There are some important bits of scaffolding (like an actual type system for Puppet!) that will prove valuable as more of the future parser and evaluator work that Henrik is building makes its way into the product, but in the final analysis the data in modules feature was the wrong vehicle to introduce them.

Yep, in trivial cases hiera data layer can approximate conditional in params.pp, but the I can see how the complexity ramps up rapidly.

Refocusing on the problems users were trying to solve (and here I have to give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] module) and the problems with the 'params' pattern lent some clarity. We've gotten into a situation of disparity with regard to hiera and data bindings, because data bindings enable module _users_ to use their site-wide hiera data but don't provide moduel _authors_ the same affordance. But rather than introduce additional complexity, we can close the gap for existing code patterns.

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet backend for a value of 'classname::variable' in the file 'modules/classname/manifests/params.pp', which simplifies class definition and provides consistency with other hiera backends. As a module author, you'd still leave your logic for variables in params.pp, but they'd be implicitly looked up via data bindings as the class is declared, after consulting site-wide hiera.

So this is only limited to class variables? and this is still compatible with inherits params class (to ease migration)?

Totally understand the need for proof of concept, should there be experimental branch v.s. production branch (i.e. Linux kernel)? Would appreciate an official notice when a final pattern is decided for long term support. 

Thanks,

Nan 

[Spencer Krum]

Thanks for sending this out Eric. When will there be a release of Puppet with this functionality released? I'm excited to kick the tires on it.

Thanks,

Spencer

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.

--
Spencer Krum
(619)-980-7820

[Dan Bode]

On Fri, Oct 11, 2013 at 11:09 AM, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Thanks to everyone who kicked the tires on the experimental data in modules feature included in Puppet 3.3.0. We got a lot of feedback, some cool proof-of-concept modules, and a definitive conclusion to the experiment.

The idea of including a module-specific hiera backend is centered around one primary use case: replacing the 'params class pattern', a common idiom in Puppet modules that's described in the [Using Parameterized Classes][param-classes] guide. The problem that most testers ran into though is that for non-trivial modules they ended up having to re-implement the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious ways. The solutions to this led to more contortions until we'd ended up with the ability to execute parser functions in the right-hand-side of a yaml value. So something which started out trying to help separate data from code ended up putting code back into data!

Additionally, even after multiple attempts to simplify the surface area and user experience with the bindings system (described in ARM-9) that underlay the data-in-modules implementation, users still found its complexity daunting. There are some important bits of scaffolding (like an actual type system for Puppet!) that will prove valuable as more of the future parser and evaluator work that Henrik is building makes its way into the product, but in the final analysis the data in modules feature was the wrong vehicle to introduce them.

Refocusing on the problems users were trying to solve (and here I have to give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] module) and the problems with the 'params' pattern lent some clarity. We've gotten into a situation of disparity with regard to hiera and data bindings, because data bindings enable module _users_ to use their site-wide hiera data but don't provide moduel _authors_ the same affordance. But rather than introduce additional complexity, we can close the gap for existing code patterns.

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet backend for a value of 'classname::variable' in the file 'modules/classname/manifests/params.pp', which simplifies class definition and provides consistency with other hiera backends. As a module author, you'd still leave your logic for variables in params.pp, but they'd be implicitly looked up via data bindings as the class is declared, after consulting site-wide hiera.

+1

Really happy to see this solved in a way that will not lead to complex migrations to Puppet 4.

Although, to play devil's advocate, two concerns:

the special nature of params as a namespace suffix:

- how do users know not to use this namespace for anything else?

- What if user declares resources in params? Does this fail? Do they always get realized when anything else from that namespace is applied?

the magic mapping from <x>::parameter <x>::params::parameter may be something hard to grok for new users who are not already familiar with the params pattern. This is probably solvable with documentation and --debug logging, but still worth noting.

 

[Erik Dalén]

On 11 October 2013 20:09, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Thanks to everyone who kicked the tires on the experimental data in modules feature included in Puppet 3.3.0. We got a lot of feedback, some cool proof-of-concept modules, and a definitive conclusion to the experiment.

The idea of including a module-specific hiera backend is centered around one primary use case: replacing the 'params class pattern', a common idiom in Puppet modules that's described in the [Using Parameterized Classes][param-classes] guide. The problem that most testers ran into though is that for non-trivial modules they ended up having to re-implement the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious ways. The solutions to this led to more contortions until we'd ended up with the ability to execute parser functions in the right-hand-side of a yaml value. So something which started out trying to help separate data from code ended up putting code back into data!

Additionally, even after multiple attempts to simplify the surface area and user experience with the bindings system (described in ARM-9) that underlay the data-in-modules implementation, users still found its complexity daunting. There are some important bits of scaffolding (like an actual type system for Puppet!) that will prove valuable as more of the future parser and evaluator work that Henrik is building makes its way into the product, but in the final analysis the data in modules feature was the wrong vehicle to introduce them.

Refocusing on the problems users were trying to solve (and here I have to give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] module) and the problems with the 'params' pattern lent some clarity. We've gotten into a situation of disparity with regard to hiera and data bindings, because data bindings enable module _users_ to use their site-wide hiera data but don't provide moduel _authors_ the same affordance. But rather than introduce additional complexity, we can close the gap for existing code patterns.

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet backend for a value of 'classname::variable' in the file 'modules/classname/manifests/params.pp', which simplifies class definition and provides consistency with other hiera backends. As a module author, you'd still leave your logic for variables in params.pp, but they'd be implicitly looked up via data bindings as the class is declared, after consulting site-wide hiera.

so if foo looks in foo::params, does foo::bar look in foo::bar::params?

If you want to share a variable between them do you recommend having foo::bar::params inherit foo::params or just include it and do $shared_variable=$::foo::params::shared_variable?

How do you then allow site admins to override that value in both foo and foo::bar without duplicating the key in hiera?

I'm not really sure it is only module authors that need the expressiveness of the DSL. As an example see my puppetdbquery hiera backend. So with that a site administrator can inpus something like ntp::client::servers::_puppetdbquery: 'class[ntp::server]' to get their ntp clients to use the hosts for their ntp servers. But really that could be solved by allowing function calls within hiera values instead in a more general way.

 

Note too that this is the next iteration, not necessarily the end state. It's super important to get this right because the whole community is going to have to live with it for a long time; those of you out here on the bleeding edge willing to risk some skin to make something awesome are critical to making that happen.


Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles


[puppetlabs-ntp]: https://github.com/apenney/puppetlabs-ntp/tree/data-in-modules
[startrek]: https://github.com/pro-puppet/puppet-module-startrek
[param-classes]: http://docs.puppetlabs.com/guides/parameterized_classes.html#appendix-smart-parameter-defaults
[15746]: https://projects.puppetlabs.com/issues/15746

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.

--
Erik Dalén

[Eric Sorenson]

On Oct 11, 2013, at 12:01 PM, Spencer Krum <krum.s...@gmail.com> wrote:

> Thanks for sending this out Eric. When will there be a release of Puppet with this functionality released? I'm excited to kick the tires on it.

Should be in Puppet 3.4, slated for release near the end of November.

[Chuck]

I see the best aspect of data in modules is that it allows the clear separation of variables per module in hiera.  This is important because module developers don't need access to a central global hiera that is "static".  For our use we need to break variables down by environment and datacenter, hiera is great for this.  And data in modules created a very clear scope of control for module authors.  We were basically treating data in modules as a distributed hiera v1 which works really well.  The only addition that would have been nice is the adding of the classname in front of all variables as you are proposing for your params.pp.

Managing a "central" hiera directory structure can be painful when you have 20 - 50 developers that need to create hiera variables.  I really think data in modules helps this considerably.  I agree that about moving logic into hiera is not beneficial as you just end up with more code.

[Chuck]

I see hiera data in modules just being a very useful extension of the current hiera implementation.

Why

1)  separate out the variables my dev teams use into the modules so that it is easier to package their changes.  Promoting their module includes all of their heir data.

2)  In an enterprise environment this allows other modules to access that data when the modules are not totally independent.  Eg.. I have a apache_module_v1 and a apache_business_unit_v2 that uses the apache module.

3)  access controls by adding hiera into modules as opposed to a main hiera directory structure.  The developers just need access to their modules. 

4)  We version our modules so that they call all be active at the same time and the old modules only have maintenance updates.  Hiera in modules means the central hiera is not touched by the developers.

        eg.  apache_module_v1

               apache_module_v2

               apache_module_v3

What would be nice but not necessary:

1) defined variable automatically have classname added to avoid global conflicts.

eg.

Module: apache

  variable: port

  becomes global hiera:   apache::port

[Alessandro Franceschi]

Thanks for the update Eric, very useful to understand the ongoing works on data in modules.

On Friday, October 11, 2013 9:01:19 PM UTC+2, Dan Bode wrote:

On Fri, Oct 11, 2013 at 11:09 AM, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Thanks to everyone who kicked the tires on the experimental data in modules feature included in Puppet 3.3.0. We got a lot of feedback, some cool proof-of-concept modules, and a definitive conclusion to the experiment.

The idea of including a module-specific hiera backend is centered around one primary use case: replacing the 'params class pattern', a common idiom in Puppet modules that's described in the [Using Parameterized Classes][param-classes] guide. The problem that most testers ran into though is that for non-trivial modules they ended up having to re-implement the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious ways. The solutions to this led to more contortions until we'd ended up with the ability to execute parser functions in the right-hand-side of a yaml value. So something which started out trying to help separate data from code ended up putting code back into data!

Additionally, even after multiple attempts to simplify the surface area and user experience with the bindings system (described in ARM-9) that underlay the data-in-modules implementation, users still found its complexity daunting. There are some important bits of scaffolding (like an actual type system for Puppet!) that will prove valuable as more of the future parser and evaluator work that Henrik is building makes its way into the product, but in the final analysis the data in modules feature was the wrong vehicle to introduce them.

Refocusing on the problems users were trying to solve (and here I have to give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] module) and the problems with the 'params' pattern lent some clarity. We've gotten into a situation of disparity with regard to hiera and data bindings, because data bindings enable module _users_ to use their site-wide hiera data but don't provide moduel _authors_ the same affordance. But rather than introduce additional complexity, we can close the gap for existing code patterns.

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet backend for a value of 'classname::variable' in the file 'modules/classname/manifests/params.pp', which simplifies class definition and provides consistency with other hiera backends. As a module author, you'd still leave your logic for variables in params.pp, but they'd be implicitly looked up via data bindings as the class is declared, after consulting site-wide hiera.

+1

Really happy to see this solved in a way that will not lead to complex migrations to Puppet 4.

Although, to play devil's advocate, two concerns:

the special nature of params as a namespace suffix:

- how do users know not to use this namespace for anything else?

- What if user declares resources in params? Does this fail? Do they always get realized when anything else from that namespace is applied?

the magic mapping from <x>::parameter <x>::params::parameter may be something hard to grok for new users who are not already familiar with the params pattern. This is probably solvable with documentation and --debug logging, but still worth noting.

Yes worth noting, but as you said proper documentation might suffice.
Anyway +1 also for me on the lookup to params.pp (Dan this, + Puppet 3 data bindings, reminds me  https://github.com/example42/puppi/blob/master/lib/puppet/parser/functions/params_lookup.rb ;-)

I'd like to add to the complexity cases , in case you haven't still sorted it out, the situations where some of the modules' params change according to user provided values to other params.

For example look here:
https://github.com/stdmod/puppet-elasticsearch/blob/master/manifests/init.pp#L124
the path of the configuration dir ($dir param here) change according to the value of the $install param,

and in order to manage this the code in the linked line is necessary.
I wonder how such a case could be managed with data in modules (without replicating a similar logic).

 

al

[Erik Dalén]

Isn't that possible to solve by allowing one hiera value to use another hiera value for interpolation?

https://projects.puppetlabs.com/issues/21367

--
Erik Dalén

[Alessandro Franceschi]

Didn't know of this function which actually looks interesting.

Is the hiera value usable in the same hierarchy structure ?

I mean, would it be possibile to have an hiera.yaml like:

---
version: 3
hierarchy:
  - category: 'osfamily'
  - category: 'operatingsystem'
  - category: '^{install}'
  - category: 'environment'
  - category: 'common'
      paths:
        - 'is_virtual/${is_virtual}'
        - 'common'

Is so, then it might actually work. 

[John Julien]

On Friday, October 11, 2013 7:12:32 PM UTC-5, Chuck wrote:

What would be nice but not necessary:

1) defined variable automatically have classname added to avoid global conflicts.

eg.

Module: apache

  variable: port

  becomes global hiera:   apache::port

+1

Defining a variable in a module but having it's scope be global seems counter intuitive and could lead to conflicts.  If someone wants a topscope variable they should probably define it in a topscope location (ENC, Centralized Hiera Files, etc)

[Trevor Vaughan]

I also would like to know if this is going to get mainlined.

It would be an excellent addition for overall module reuse.

Trevor

On Tue, Dec 31, 2013 at 3:31 AM, Fabio Sangiovanni <fsangi...@gmail.com> wrote:

Hi everybody,

is there any news about this topic?
I think it would be great to know Puppetlabs' official position on the matter (if any), and the status of activities about it (issues on tickets.puppetlabs.com don't say much).

Thanks!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f312d16c-0695-48a9-a630-2016fff1ba91%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Eric Sorenson]

On Jan 3, 2014, at 5:11 AM, Fabio Sangiovanni <fsangi...@gmail.com> wrote:

> Ok, I get it. First rule of data in modules: you don't talk about data in modules.
>

Hah! No, I just don't like to reply until I have something substantial to report. Did you see this thread from a couple of weeks ago? It was only on puppet-dev so if you're following this in puppet-users you may have missed it: https://groups.google.com/d/topic/puppet-dev/f0KrpOtfKRY/discussion

Next steps out of that were that I'm pulling together a google doc, which I haven't finished yet due to holidays.

I'm excited that people are working with the module_data implementation. The main technical concern I have is forward and backward compatibility: how should a module express that it needs a particular implementation of DIM?

[R.I.Pienaar]

----- Original Message -----
> From: "Eric Sorenson" <eric.s...@puppetlabs.com>
> To: puppet...@googlegroups.com
> Cc: puppe...@googlegroups.com
> Sent: Friday, January 3, 2014 5:09:54 PM
> Subject: [Puppet-dev] Re: [Puppet Users] Status of Data in modules
>
>
>
> On Jan 3, 2014, at 5:11 AM, Fabio Sangiovanni <fsangi...@gmail.com> wrote:
>
> > Ok, I get it. First rule of data in modules: you don't talk about data in
> > modules.
> >
>
> Hah! No, I just don't like to reply until I have something substantial to
> report. Did you see this thread from a couple of weeks ago? It was only on
> puppet-dev so if you're following this in puppet-users you may have missed
> it: https://groups.google.com/d/topic/puppet-dev/f0KrpOtfKRY/discussion
>
> Next steps out of that were that I'm pulling together a google doc, which I
> haven't finished yet due to holidays.
>
> I'm excited that people are working with the module_data implementation. The
> main technical concern I have is forward and backward compatibility: how
> should a module express that it needs a particular implementation of DIM?

the same way the module declares it needs puppet > 3, or the new parser and
any other related things. These are not hiera questions, but once you solve
the underlying problem that a module needs this would just slot into that.

At the moment my module is broken on 3.4 because you've changed some internals
sadly.