Data binding for defines

[Alessandro Franceschi]

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

Naming is straightforward, a define like:
apache::vhost { 'example.com': }

could have its params looked in this namespace:
apache::vhost::example.com::<parameter>

My poor understanding of Puppet code points me to:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource.rb#L343
and makes me guess that such a change should not be too difficult.

I see many wonderful use cases for such a feature and no apparent cons.

What do you think?

I've opened a ticket about this:

https://tickets.puppetlabs.com/browse/PUP-2528

al

[Andy Parker]

On Thu, May 15, 2014 at 7:46 AM, Alessandro Franceschi <a...@lab42.it> wrote:

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I tried to find the original ticket about data binding to see if there had been a previous discussion and came up with http://projects.puppetlabs.com/issues/11608 and http://projects.puppetlabs.com/issues/8235. There is also the commit that brought it in: https://github.com/puppetlabs/puppet/commit/3ac5b50ce66948d9488bd394c169022d55ecf0d1

I wasn't able to figure out if there had been any discussion on this topic either.

 

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

Naming is straightforward, a define like:
apache::vhost { 'example.com': }

could have its params looked in this namespace:
apache::vhost::example.com::<parameter>

The problem with using that key for lookup is that it clashes with a class with the same name:

class a::b($param) {} => a::b::param

collides with

define a($param) {}

a { b: }                           => a::b::param

A better approach would possibly be to follow the resource reference syntax and have the key be A[b]::param or even A[b][param], which is the syntax in the language for puppet 4 to access resource parameter values (see https://tickets.puppetlabs.com/browse/PUP-488)

My poor understanding of Puppet code points me to:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource.rb#L343
and makes me guess that such a change should not be too difficult.

I see many wonderful use cases for such a feature and no apparent cons.

What do you think?

I've opened a ticket about this:

https://tickets.puppetlabs.com/browse/PUP-2528

al

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/6f9e8c23-53c2-420f-b7af-9b9fe7d5c1b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 22-24 in San Francisco

Register by May 30th to take advantage of the Early Adopter discount —save $349!

[Alessandro Franceschi]

On Thursday, May 15, 2014 6:08:44 PM UTC+2, Andy Parker wrote:

On Thu, May 15, 2014 at 7:46 AM, Alessandro Franceschi <a...@lab42.it> wrote:

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I tried to find the original ticket about data binding to see if there had been a previous discussion and came up with http://projects.puppetlabs.com/issues/11608 and http://projects.puppetlabs.com/issues/8235. There is also the commit that brought it in: https://github.com/puppetlabs/puppet/commit/3ac5b50ce66948d9488bd394c169022d55ecf0d1

I wasn't able to figure out if there had been any discussion on this topic either.

 

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

Naming is straightforward, a define like:
apache::vhost { 'example.com': }

could have its params looked in this namespace:
apache::vhost::example.com::<parameter>

The problem with using that key for lookup is that it clashes with a class with the same name:

class a::b($param) {} => a::b::param

collides with

define a($param) {}

a { b: }                           => a::b::param

A better approach would possibly be to follow the resource reference syntax and have the key be A[b]::param or even A[b][param], which is the syntax in the language for puppet 4 to access resource parameter values (see https://tickets.puppetlabs.com/browse/PUP-488)

Good point. Any sane naming is ok.

What do you think of such a feature? Doable? Useful?

[John Bollinger]

On Thursday, May 15, 2014 9:46:40 AM UTC-5, Alessandro Franceschi wrote:

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

I'm skeptical about their value.  We already have data bindings for classes, by which resource parameters can (indirectly) be injected, and I am inclined to think that classes are the right level of abstraction.  On the other hand, we also have create_resources(), which already can give you resource-level data binding if you want to use it that way.

I do not favor drawing distinctions between defined resource types and native resource types.  On that basis, I would argue for data binding either for all resource types or for none, and against data binding for only defined types.

I am concerned about the impact.  It is already somewhat costly for Puppet to evaluate data bindings for class parameters, and adding bindings for resource parameters (even just for resources of defined types) will magnify that.  Note that the cost scales with the aggregate number of defined parameters for all declared resources, independent whether any data are actually bound.  In fact, the cases were no data are bound are the most costly, because hiera must then search the entire hierarchy.

 

Naming is straightforward, a define like:
apache::vhost { 'example.com': }

could have its params looked in this namespace:
apache::vhost::example.com::<parameter>

My poor understanding of Puppet code points me to:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource.rb#L343
and makes me guess that such a change should not be too difficult.

I agree, the change probably would not be difficult to implement -- just costly.  It might be interesting to have some data on what proportion of invocations of that method follow the "return nil" branch, avoiding a lookup.  As a wild guess, I'll say at least 75% in a typical catalog compilation run (maybe we can start a pool :-)).  Even if it were only 25%, though, that would translate into a 33% increase in the number of (expensive) lookups.
 

I see many wonderful use cases for such a feature and no apparent cons.

I see no particularly good use cases and several cons, but I'm prepared to be amazed by your vision.  Would you care to elaborate?

John

[Alessandro Franceschi]

On Thursday, May 15, 2014 9:55:05 PM UTC+2, John Bollinger wrote:

On Thursday, May 15, 2014 9:46:40 AM UTC-5, Alessandro Franceschi wrote:

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

I'm skeptical about their value.  We already have data bindings for classes, by which resource parameters can (indirectly) be injected, and I am inclined to think that classes are the right level of abstraction.  On the other hand, we also have create_resources(), which already can give you resource-level data binding if you want to use it that way.

I do not favor drawing distinctions between defined resource types and native resource types.  On that basis, I would argue for data binding either for all resource types or for none, and against data binding for only defined types.

I am concerned about the impact.  It is already somewhat costly for Puppet to evaluate data bindings for class parameters, and adding bindings for resource parameters (even just for resources of defined types) will magnify that.  Note that the cost scales with the aggregate number of defined parameters for all declared resources, independent whether any data are actually bound.  In fact, the cases were no data are bound are the most costly, because hiera must then search the entire hierarchy.

Yes, these are valid and convincing points.

Anyway if we find data binding useful for classes and can bear the performance overhead, I suppose we can do the same for defined types.

 

 

Naming is straightforward, a define like:
apache::vhost { 'example.com': }

could have its params looked in this namespace:
apache::vhost::example.com::<parameter>

My poor understanding of Puppet code points me to:
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource.rb#L343
and makes me guess that such a change should not be too difficult.

I agree, the change probably would not be difficult to implement -- just costly.  It might be interesting to have some data on what proportion of invocations of that method follow the "return nil" branch, avoiding a lookup.  As a wild guess, I'll say at least 75% in a typical catalog compilation run (maybe we can start a pool :-)).  Even if it were only 25%, though, that would translate into a 33% increase in the number of (expensive) lookups.
 

I see many wonderful use cases for such a feature and no apparent cons.

I see no particularly good use cases and several cons, but I'm prepared to be amazed by your vision.  Would you care to elaborate?

Yes, why not. Let me digress a bit then. It could be useful to verify if the following idea seems cool only to me.

First, a premise.

I've written a lot of modules which are ageing and have large parts which are very similar. I don't have the time to keep them all updated and there are around better alternatives for many applications, even if sometimes when I try to use modules from others I find myself screaming in despair.

For a long time I wondered how I could bypass all my code duplication, at the same time I was more and more convinced that an application module should be considered as a sort of library, to be used as is, without modifications, (reusability), which have standard interfaces (naming standards) and should not be opinionated in how in manages its application.

According to me opinionated modules can stay at higher abstraction levels (profiles/stacks/whatever... they have to deliver a working setup, so some choice on how things should be done is needed there) but the component modules should just expose a clear and consistent interface on how to manage the application they are made for.

Also I've seen that most of the times I found myself just with the need of having a configuration file as wanted, eventually managing correlations and integrations among (component) modules in custom higher abstraction classes with dedicated templates and resources where all the logic could be defined as I needed.

Finally I think that exposing as modules parameters each (or many) of the managed application configuration entries is a dead end.

When I see a define like this: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp with dozens of application specific parameters and the virtual host file content managed with:

    content => template('apache/vhost.conf.erb'),

instead of something like:

define apache::vhost(
    $docroot,
    $template = 'apache/vhost.conf.erb',

[...]

    content => template($template),

I ask me a lot of weird questions. But I'm digressing. My point is that a reusable module should allow total freedom on how its configurations should be managed: with a file-based approach (source/template/ now epp_template...), with a setting-based approach, (augeas, file_line..) with concat or whatever. User should decide what's best for him, not the module author. 
Also I think that a single parameter that expects an hash ( options_hash, config_file_options_hash, options, config... name it as you want) whose key values can be freely used in a [custom] template is better than dozens of parameters, totally unmaintainable, one for each possible configuration entry of an application (and for each one, as you pointed, a data binding lookup).

Given this premise (longer than expected, sorry), I've recently thought about a single module: tp (stays for tiny puppet) which should be able to manage essential features of any application.

Something you can use in manifests in a similar fashion:

To install an application (the tp module contains all the data to do it right on different OS) 

  tp::install { 'redis': }  

To configure it via a template:

  tp::conf { 'redis.conf':
    application => 'redis',

    template    => 'site/redis/redis.conf.erb',
  }  

Or, alternatively (using in the title any sane separator):

  tp::conf { 'redis--redis.conf':

    template    => 'site/redis/redis.conf.erb',

  }  

To configure it via the fileserver:

  tp::conf { 'redis.conf':
    application => 'redis',

    source      => 'puppet:///modules/site/redis/redis.conf',
  }  

But also have something like (to manage single lines in a configuration file):

  tp::line { 'redis::redis.conf::port':

    value => '1234',

  }  

or even something like:

  tp::concat { 'redis':

     target   => 'redis.conf',

     order    => 10,

     content => 'port 1234',

  }  

And manage, if overrides are needed, any internally used parameter with something like:

  tp::settings { 'redis':

    lookup_strategy => 'merge', # Such a settings should define if the module's data has to be merged or not with user data
    settings => {

      config_dir_path => '/opt/redis/conf',

      tcp_port        => '3242',

      pid_file_path   => '/opt/redis/run/redis.pid',

    },

Also, it could be nice to have a face that allows commands like:

puppet tp check redis  # To check is redis is running, based on settings like service name , pid file, port ... (Incidentally this would make integration tests a "bit" easier and quicker)

puppet tp info redis # To show info about how redis is managed by Puppet or how it is working

Basically such a tp module would be a sum of most of my existing modules main features + the puppi module (https://github.com/example42/puppi,  for the functionalities not related to application deployments).

Now, the real added value of such a thing would be a (Hiera like) set of yaml files where all the default settings for as many applications as possible for different operating systems can be defined in order to make usage of the previous defines possible with a wide variety of applications and allow, at the same time, extreme customization options.

To do this (hey, this is the topic of this thread), it would have been nice to have a data in modules approach (as based on defines parameters) and have in files like:

tp/data/redis/default.yaml , tp/data/redis/osfamily/RedHat.yaml all the data needed for all the managed applications, such as :
---
  tp::settings::redis::port: '6379'
  tp::settings::redis::config_file_path: '/etc/redis/redis.conf'

Or, alternatively, data expressed in a format like :

---

  redis::packages: (or tp::install::redis::packages...)

    redis:

      ensure: present

      alias: 'redis'

  redis::services:

    redis:

      ensure: running

      enable: true

      alias: 'redis'

      require: Package[redis]

  redis::files:

    '/etc/redis/redis.conf':

      ensure: present

      owner: root

      group: root

      mode: 0644

      alias: 'redis.conf'

      notify: Service[redis]

    '/etc/redis':

      ensure: directory

      owner: root

      group: root

      mode: 0755

      alias: 'redis.dir'

      notify: Service[redis]

In this case the tp::install code could be as simple as this POC:

define tp::install (

  $packages  = { } ,

  $services  = { } ,

  $files     = { } ,

  $execs     = { } ,

  $users     = { } ,

  $configs   = { } ,

  ) {

  if $packages {

    create_resources('package', $packages)

  }

  if $services {

    create_resources('service', $services)

  }

  if $files {

    create_resources('file', $files)

  }

  if $execs {

    create_resources('exec', $execs)

  }

  if $users {

    create_resources('user', $users)

  }

}

Anyway, however is organized the internal data, and the relevant code (instead of the create_resource we could use a lambda and cycle over the various hashes) having the possibility to use Hiera for the backend would be a good thing also because it would make easier for user to override the default module data with custom one.

I guess there are better ways to obtain the same (suggestions welcomed) rather than enabling  data bindings for defined types  and using data in modules (both features being not existing in Puppet core). For example with a custom function, but I have to figure out how to do it in the right way.

... but, what were we talking about? :-)

[David Schmitt]

Hi,

I very much like the idea for data binding on all parameters, but this
statement is really not universally true. I've got catalogs with only
tens of classes but thousands of resources. That works out to 100x more
hiera calls. :-/


Regards, David

[Andy Parker]

John makes a very good point about the performance cost. The impact of putting data bindings on everything is made even worse by the fact that it is an all or nothing price you have to pay, there isn't any way to use it for just a subset of your catalog.

As a little number collection, would some of you mind doing a little experiment and posting the numbers?

Run "puppet apply -e 'notice(hiera("somekey", 1), hiera("otherkey", 1))' --profile --debug"

Make sure that it is using your real hiera config with your real data files or else it won't be an accurate reflection of the cost. You'll get output that should contain something like this:

Debug: PROFILE [apply] 2.2 Compile: Created settings scope: took 0.0103 seconds

Debug: hiera(): Hiera YAML backend starting

Debug: hiera(): Looking up somekey in YAML backend

Debug: hiera(): Looking for data source common

Debug: hiera(): Cannot find datafile /var/lib/hiera/common.yaml, skipping

Debug: PROFILE [apply] 2.3.1 Called hiera: took 0.0019 seconds

Debug: hiera(): Looking up otherkey in YAML backend

Debug: hiera(): Looking for data source common

Debug: hiera(): Cannot find datafile /var/lib/hiera/common.yaml, skipping

Debug: PROFILE [apply] 2.3.2 Called hiera: took 0.0002 seconds

I don't have hiera setup on this machine so it didn't do anything, but you can see how much overhead *per-parameter* data bindings introduces. So in a system with no hierarchy, and no data files (and in fact no hiera config) it adds about 2 usec per parameter. 

If some of you could post results as well as estimates of the number of defined types and parameters that you have in a catalog that would be great.

 

Regards, David

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5375A109.7020209%40dasz.at.

For more options, visit https://groups.google.com/d/optout.

[Joshua Hoblitt]

On 05/15/2014 12:55 PM, John Bollinger wrote:
>
> On Thursday, May 15, 2014 9:46:40 AM UTC-5, Alessandro Franceschi wrote:
>> >
>> > I wonder what do you thing about a feature request to have data bindings
>> > also for defines' parameters.
>> >
>
> I'm skeptical about their value. We already have data bindings for
> classes, by which resource parameters can (indirectly) be injected, and I
> am inclined to think that classes are the right level of abstraction. On
> the other hand, we also have create_resources(), which already can give you
> resource-level data binding if you want to use it that way.

I personally think that data binding for defined types would be
incredibly useful. I presently have to deal with the situation by
declaring them via my ENC (smart parameters in Foreman).

However, why not make data bindings completely orthogonal by extending
it to all resource types? I realize that's taking things pretty far
into the meta realm but my impression is that there are already a fair
number of folks using stub classes to realize resources defined as hiera
data already.

-Josh

--

[Henrik Lindberg]

Note that this is per defined parameter per type since it obviously need
to process them all. Not per actually used parameter.

- henrik

[Alessandro Franceschi]

Andy,

did this on a vagrant box with eyaml and file backends.

The hierarchy has 7 levels.

Debug: PROFILE [apply] 1 Setup server facts for compiling: took 0.0000 seconds

Debug: PROFILE [apply] 2.1 Compile: Set node parameters: took 0.0000 seconds

Debug: PROFILE [apply] 2.2 Compile: Created settings scope: took 0.0100 seconds

Debug: hiera(): [eyaml_backend]: Hiera eYAML backend starting

Debug: hiera(): [eyaml_backend]: Set option: datadir = /vagrant/hiera/

Debug: hiera(): [eyaml_backend]: Set option: pkcs7_private_key = /vagrant/keys/private_key.pkcs7.pem

Debug: hiera(): [eyaml_backend]: Set option: pkcs7_public_key = /vagrant/keys/public_key.pkcs7.pem

Debug: hiera(): [eyaml_backend]: Looking up somekey in eYAML backend

Debug: hiera(): [eyaml_backend]: Looking for data source it-mil1/dev/nodes/dnsns001

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/nodes/dnsns001.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source it-mil1/dev/roles/dnsns

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/roles/dnsns.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source it-mil1/dev/global

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/global.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source nodes/dnsns001

Debug: hiera(): [eyaml_backend]: Looking for data source roles/dnsns

Debug: hiera(): Cannot find datafile /vagrant/hiera/roles/dnsns.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source dev

Debug: hiera(): Cannot find datafile /vagrant/hiera/dev.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source global

Debug: hiera(): Hiera File backend starting

Debug: hiera(): Looking up somekey in File backend

Debug: hiera(): Hiera File_backend: looking for data source 'it-mil1/dev/nodes/dnsns001'

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/nodes/dnsns001.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'it-mil1/dev/roles/dnsns'

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/roles/dnsns.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'it-mil1/dev/global'

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/global.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'nodes/dnsns001'

Debug: hiera(): Cannot find datafile /vagrant/hiera/nodes/dnsns001.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'roles/dnsns'

Debug: hiera(): Cannot find datafile /vagrant/hiera/roles/dnsns.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'dev'

Debug: hiera(): Cannot find datafile /vagrant/hiera/dev.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'global'

Debug: PROFILE [apply] 2.3.1 Called hiera: took 0.0336 seconds

Debug: hiera(): [eyaml_backend]: Set option: datadir = /vagrant/hiera/

Debug: hiera(): [eyaml_backend]: Set option: pkcs7_private_key = /vagrant/keys/private_key.pkcs7.pem

Debug: hiera(): [eyaml_backend]: Set option: pkcs7_public_key = /vagrant/keys/public_key.pkcs7.pem

Debug: hiera(): [eyaml_backend]: Looking up otherkey in eYAML backend

Debug: hiera(): [eyaml_backend]: Looking for data source it-mil1/dev/nodes/dnsns001

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/nodes/dnsns001.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source it-mil1/dev/roles/dnsns

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/roles/dnsns.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source it-mil1/dev/global

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/global.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source nodes/dnsns001

Debug: hiera(): [eyaml_backend]: Looking for data source roles/dnsns

Debug: hiera(): Cannot find datafile /vagrant/hiera/roles/dnsns.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source dev

Debug: hiera(): Cannot find datafile /vagrant/hiera/dev.eyaml, skipping

Debug: hiera(): [eyaml_backend]: Looking for data source global

Debug: hiera(): Looking up otherkey in File backend

Debug: hiera(): Hiera File_backend: looking for data source 'it-mil1/dev/nodes/dnsns001'

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/nodes/dnsns001.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'it-mil1/dev/roles/dnsns'

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/roles/dnsns.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'it-mil1/dev/global'

Debug: hiera(): Cannot find datafile /vagrant/hiera/it-mil1/dev/global.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'nodes/dnsns001'

Debug: hiera(): Cannot find datafile /vagrant/hiera/nodes/dnsns001.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'roles/dnsns'

Debug: hiera(): Cannot find datafile /vagrant/hiera/roles/dnsns.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'dev'

Debug: hiera(): Cannot find datafile /vagrant/hiera/dev.d, skipping

Debug: hiera(): Hiera File_backend: looking for data source 'global'

Debug: PROFILE [apply] 2.3.2 Called hiera: took 0.0156 seconds

Regards, David

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5375A109.7020209%40dasz.at.

For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 2014-16-05 21:35, Alessandro Franceschi wrote:
> Andy,
> did this on a vagrant box with eyaml and file backends.
> The hierarchy has 7 levels.
>

> Debug: PROFILE [apply] 2.3.2 Called hiera: took 0.0156 seconds
>

If types on average have 10 parameters/properties, and you have 200 of
them, that means the overhead is something like 30 seconds for this
hierarchy. (Ballpark figure obviously) since it depends on where in the
hierarchy they are, and if they have values or not set in the manifests.

Out of curiosity, why not use the Puppet Resource Default mechanism and
only lookup in hiera what you actually need and set those values as
defaults? Is it too limited for what you want to do? If so how?

- henrik

[Trevor Vaughan]

Henrik,

Do you know if it would be possible to cache all non-fact-based Hiera data and only traverse the tree for data that has actually changed?

That could save a LOT of time at compile.

I suspect that there would need to be a cache invalidator similar to that in the Puppet core that has been discussed so much recently.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/ll5q5d%24kqo%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Alessandro Franceschi]

On Friday, May 16, 2014 9:50:04 PM UTC+2, henrik lindberg wrote:

On 2014-16-05 21:35, Alessandro Franceschi wrote:
> Andy,
> did this on a vagrant box with eyaml and file backends.
> The hierarchy has 7 levels.
>

> Debug: PROFILE [apply] 2.3.2 Called hiera: took 0.0156 seconds
>

If types on average have 10 parameters/properties, and you have 200 of
them, that means the overhead is something like 30 seconds for this
hierarchy. (Ballpark figure obviously) since it depends on where in the
hierarchy they are, and if they have values or not set in the manifests.

Yes, I definitively see the perfomance problems here.

Incidentally this should trigger an alarm for whoever abuses with the number of parameters in classes (me, included).

I take the occasion (sorry for OT)  to ask you an advice for what is my actual need, which made me ask for such a feature and which might be approached in an alternate way.
I described in a previous post in this thread what I'd like to do. Basically have an in-module hiera datadir .

An alternative approach to data bindings on defines + data in module can be to create a custom function that makes the hiera lookup on the module local data directory.
I've seen that the hiera function accepts an optional parameter that defines an extra data source in hierarchy to add to the configured one. Can his be an array of datasources? Is it based only on the datadir configured in the local hiera.yaml?  In this case the datadir should be in the module itself.

Is is possible to set an alternative path for the hiera.yaml when calling the hiera function ?
(A sort of data in modules, based on a function and not on data binding indirection as done in Rip's module).

Any hint, also regarding totally different approaches for this problem would be very welcomed.

 

Out of curiosity, why not use the Puppet Resource Default mechanism and
only lookup in hiera what you actually need and set those values as
defaults? Is it too limited for what you want to do? If so how?

Well, if we don't lookup values defined in resources defaults we can't override them. It might be limitating (and in the case of the tp module mentioned earlier, i would not make sense to have resource defaults).

al

- henrik

[John Bollinger]

On Thursday, May 15, 2014 7:17:38 PM UTC-5, Alessandro Franceschi wrote:

On Thursday, May 15, 2014 9:55:05 PM UTC+2, John Bollinger wrote:

On Thursday, May 15, 2014 9:46:40 AM UTC-5, Alessandro Franceschi wrote:

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

I'm skeptical about their value.  We already have data bindings for classes, by which resource parameters can (indirectly) be injected, and I am inclined to think that classes are the right level of abstraction.  On the other hand, we also have create_resources(), which already can give you resource-level data binding if you want to use it that way.

I do not favor drawing distinctions between defined resource types and native resource types.  On that basis, I would argue for data binding either for all resource types or for none, and against data binding for only defined types.

I am concerned about the impact.  It is already somewhat costly for Puppet to evaluate data bindings for class parameters, and adding bindings for resource parameters (even just for resources of defined types) will magnify that.  Note that the cost scales with the aggregate number of defined parameters for all declared resources, independent whether any data are actually bound.  In fact, the cases were no data are bound are the most costly, because hiera must then search the entire hierarchy.

Yes, these are valid and convincing points.

Anyway if we find data binding useful for classes and can bear the performance overhead, I suppose we can do the same for defined types.

In some cases we cannot bear the overhead even just for classes.  People who tried to use hiera-gpg with Puppet 3 discovered that pretty quickly.  Even if there were only a few encrypted data, the encrypted file gets decrypted for every class parameter that has no value specified in a higher-priority back end (which usually is most of them), making catalog compilation take forever.

We're not now in a comfort zone with respect to data binding cost -- we're near and sometimes over the edge of acceptable cost.
 

I see many wonderful use cases for such a feature and no apparent cons.

I see no particularly good use cases and several cons, but I'm prepared to be amazed by your vision.  Would you care to elaborate?

Yes, why not. Let me digress a bit then. It could be useful to verify if the following idea seems cool only to me.

First, a premise.

[... omitted for brevity ...]
 

My point is that a reusable module should allow total freedom on how its configurations should be managed: with a file-based approach (source/template/ now epp_template...), with a setting-based approach, (augeas, file_line..) with concat or whatever. User should decide what's best for him, not the module author. 

I accept the premise up to this point, with some reservations.  In particular, I think it applies mainly to fairly low-level modules, such as those managing fairly narrow programs or services.  Once you move into modules that compose those lower-level units, I don't think you can reliably stick to the premised approach any longer.  On the other hand, it may be that such mid- or high-level modules are not well suited for reuse anyway.

 

Also I think that a single parameter that expects an hash ( options_hash, config_file_options_hash, options, config... name it as you want) whose key values can be freely used in a [custom] template is better than dozens of parameters, totally unmaintainable, one for each possible configuration entry of an application (and for each one, as you pointed, a data binding lookup).

There is a trade-off here between expressiveness of the class/resource interface and its cost.  On the expressiveness side, I prefer separate parameters for resource types, at least up to a point.  Declarations are so much easier to read that way.  I'm not so fond of types exposing dozens of parameters, however; generally, I'm inclined to think that such types should be decomposed.  The same applies to classes to some extent, but I don't worry much about that end because I don't generally want to see resource-like class declarations at all.

 

[...]

In fact that's related to, but not quite, the original topic of the thread.  You can have something along those lines (at least the latter) without extending classes' automated data binding feature to resources.  I tried -- evidently unsuccessfully -- to make that point in my first response.  You put your data into a hash, nested to whatever depth it needs to be for the degree of consolidation you want, and your generic 'tp' module retrieves it via an ordinary hiera() call.  That will work in today's Puppet.

The question, then, is not so much about whether it is useful to be able to bind data to specific resources, but about whether Puppet should do so automatically.

I have misgivings, too, about exposing module implementation details in the form a data-binding interface applicable to internally-declared resources, but I'll leave that aside for now.

 

In this case the tp::install code could be as simple as this POC:

define tp::install (

  $packages  = { } ,

  $services  = { } ,

  $files     = { } ,

  $execs     = { } ,

  $users     = { } ,

  $configs   = { } ,

  ) {

  if $packages {

    create_resources('package', $packages)

  }

  if $services {

    create_resources('service', $services)

  }

  if $files {

    create_resources('file', $files)

  }

  if $execs {

    create_resources('exec', $execs)

  }

  if $users {

    create_resources('user', $users)

  }

}

Or like this:

define tp::install () {
  $config = hiera("tp::config::$title", {})

  if $config['packages'] {
    create_resources('package', $config['packages'])
  }

  # etc...
}

You could even go further by bundling all the data for all tp-managed components into one higher-level hash, and looking it up just once, recording it in a variable of class tp for later access by the various defined types of the module.

 

Anyway, however is organized the internal data, and the relevant code (instead of the create_resource we could use a lambda and cycle over the various hashes) having the possibility to use Hiera for the backend would be a good thing also because it would make easier for user to override the default module data with custom one.

I agree that providing the data via Hiera is a good approach.  I wouldn't recommend anything else.

 

I guess there are better ways to obtain the same (suggestions welcomed) rather than enabling  data bindings for defined types  and using data in modules (both features being not existing in Puppet core). For example with a custom function, but I have to figure out how to do it in the right way.

See above for one way.


John

[Alessandro Franceschi]

On 19 May 2014, at 17:37, John Bollinger <john.bo...@stjude.org> wrote:

On Thursday, May 15, 2014 7:17:38 PM UTC-5, Alessandro Franceschi wrote:

On Thursday, May 15, 2014 9:55:05 PM UTC+2, John Bollinger wrote:

On Thursday, May 15, 2014 9:46:40 AM UTC-5, Alessandro Franceschi wrote:

Hallo everybody,

I've tried to look around in the group for past discussions about this topic but haven't found any.
If this has been already debated , please forgive me and point me to the right direction.

I wonder what do you thing about a feature request to have data bindings also for defines' parameters.

I'm skeptical about their value.  We already have data bindings for classes, by which resource parameters can (indirectly) be injected, and I am inclined to think that classes are the right level of abstraction.  On the other hand, we also have create_resources(), which already can give you resource-level data binding if you want to use it that way.

I do not favor drawing distinctions between defined resource types and native resource types.  On that basis, I would argue for data binding either for all resource types or for none, and against data binding for only defined types.

I am concerned about the impact.  It is already somewhat costly for Puppet to evaluate data bindings for class parameters, and adding bindings for resource parameters (even just for resources of defined types) will magnify that.  Note that the cost scales with the aggregate number of defined parameters for all declared resources, independent whether any data are actually bound.  In fact, the cases were no data are bound are the most costly, because hiera must then search the entire hierarchy.

Yes, these are valid and convincing points.

Anyway if we find data binding useful for classes and can bear the performance overhead, I suppose we can do the same for defined types.

In some cases we cannot bear the overhead even just for classes.  People who tried to use hiera-gpg with Puppet 3 discovered that pretty quickly.  Even if there were only a few encrypted data, the encrypted file gets decrypted for every class parameter that has no value specified in a higher-priority back end (which usually is most of them), making catalog compilation take forever.

We're not now in a comfort zone with respect to data binding cost -- we're near and sometimes over the edge of acceptable cost.
 

I see many wonderful use cases for such a feature and no apparent cons.

I see no particularly good use cases and several cons, but I'm prepared to be amazed by your vision.  Would you care to elaborate?

Yes, why not. Let me digress a bit then. It could be useful to verify if the following idea seems cool only to me.

First, a premise.

[... omitted for brevity ...]
 

My point is that a reusable module should allow total freedom on how its configurations should be managed: with a file-based approach (source/template/ now epp_template...), with a setting-based approach, (augeas, file_line..) with concat or whatever. User should decide what's best for him, not the module author. 

I accept the premise up to this point, with some reservations.  In particular, I think it applies mainly to fairly low-level modules, such as those managing fairly narrow programs or services.  Once you move into modules that compose those lower-level units, I don't think you can reliably stick to the premised approach any longer.  On the other hand, it may be that such mid- or high-level modules are not well suited for reuse anyway.

Just bear in mind that I don’t expect this tp module to substitute any existing module. I suppose it can be used in cases where you have simple applications to manage (ie the typical package/service/configfile ones) or when you don’t need specific defines to manage elements of an application (apache modules, mysql grants…) and just want to manage a few files over a vanilla installation of an application.

John, if it were that simple I wouldn’t have asked here.

The hierarchy of such a tp module has to be module specific and should not depend on how data is managed in users’ hiera.yaml.

Default data for the managed applications should be placed in the same tp module and be based on a module specific hierarchy, it would contain references to osfamily/operatingsystem/etc facts that can’t be forced into the users’ own local hierarchies (besides that fact that imho in a sane /etc/puppet/hiera.yaml file there should not be references to OS related facts) .

The hiera function allows the possibility to add a datasource to the default hierarchy, so I might try to work around it  (even if I’d need an array of extra datasources rather that just one, and I’m not sure this is supported, and I would need to change the datadir on the fly, which is almost surely not supported), or maybe create a custom function that mimicks the hiera functionality in some way.

  if $config['packages'] {
    create_resources('package', $config['packages'])
  }

  # etc...
}

You could even go further by bundling all the data for all tp-managed components into one higher-level hash, and looking it up just once, recording it in a variable of class tp for later access by the various defined types of the module.

How data would be organized and namespaced is a secondary problem after all, my main concern is how to organize in a hiera-like style a bunch of yaml files inside a module and get data from them.

Using directly hiera instead of writing a custom function that emulates it seemed the most logical approach, but it still doesn’t seem currently possible.

 

Anyway, however is organized the internal data, and the relevant code (instead of the create_resource we could use a lambda and cycle over the various hashes) having the possibility to use Hiera for the backend would be a good thing also because it would make easier for user to override the default module data with custom one.

I agree that providing the data via Hiera is a good approach.  I wouldn't recommend anything else.

 

I guess there are better ways to obtain the same (suggestions welcomed) rather than enabling  data bindings for defined types  and using data in modules (both features being not existing in Puppet core). For example with a custom function, but I have to figure out how to do it in the right way.

See above for one way.

Thanks anyway for the attempt, hopefully now is clearer why I can’t do this with existing functionalities.

Al

John

-- 
You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/4lFhfChM9XM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/285608ae-a5f7-4091-b108-5a81309d3fe7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alessandro Franceschi

site { 'Example42 Puppet modules’:
  url       => 'http://www.example42.com',
  before => Service[‘puppet’],
}

[Joshua Hoblitt]

It seems to me that there are two reoccurring basic themes in this
thread of folks expressing either:

* eagerness to push more data into hiera
or
* sentiment that hiera is already a performance bottleneck

The first is language / design pattern issue while the later is an
engineering problem. Has much effort been spent looking into
accelerating hiera's resolution speed?

-Josh

--

[Alessandro Franceschi]

Good point, I think some efforts were done in field but evidently they might not be enough.
As for the opportunity to extend to defines the data binding functionality I definitively agree, now, that it could be a severe performance issue so I wonder if it might make sense to consider a sort of meta-parameter that would enable such functionality on demand: disabled by default but activable by users or authors.

Such a parameter might be applied also to classes.

If not possible or advisable on a per define/class basis it could make sense as a configuration option, so that who doesn't use data bindings can disable it. 

Just guessing...

Al

[John Bollinger]

On Monday, May 19, 2014 11:42:57 AM UTC-5, Alessandro Franceschi wrote:

The hierarchy of such a tp module has to be module specific and should not depend on how data is managed in users’ hiera.yaml.

Default data for the managed applications should be placed in the same tp module and be based on a module specific hierarchy, it would contain references to osfamily/operatingsystem/etc facts that can’t be forced into the users’ own local hierarchies (besides that fact that imho in a sane /etc/puppet/hiera.yaml file there should not be references to OS related facts) .

Ok, but you're throwing a curve there.  Your original suggestion / request had none of those constraints.  Would those concerns be adequately addressed if R.I.'s data in modules were in the core product?  Alternatively, would it be acceptable for the 'tp' module to depend on a module providing that feature?  Or to provide that or something equivalent itself?

> Thanks anyway for the attempt, hopefully now is clearer why I can’t do this with existing functionalities.

Your requirements are clearer, yes.  What's not quite clear yet is whether you want to avoid your data binding falling back to general data when resource parameters are not found in module-specific data.  If you do want to avoid that then I think you're right that existing functionalities won't do what you want, but then I'm certain that you were wrong earlier about what you want being achievable by a simple modification.

On the other hand, I think you could put a custom function into module 'tp' that would read data from (only) a module-specific data source, along lines similar to hiera.  If you used such a function instead of hiera(), then could a solution along the lines I described work for you?


John

[John Bollinger]

On Monday, May 19, 2014 11:47:41 AM UTC-5, Joshua Hoblitt wrote:

It seems to me that there are two reoccurring basic themes in this
thread of folks expressing either:

* eagerness to push more data into hiera

No, I don't think that's a good characterization.  There is no particular barrier now to putting any and all relevant data into hiera.  The theme here is about whether Puppet should provide for binding that data to resource instances directly and automatically, or whether it should continue to rely on classes to serve as intermediaries for that purpose.

 

    or
* sentiment that hiera is already a performance bottleneck

That, on the other hand, is certainly a prominent theme.  It has been my main focus in arguing against resource-level automated data binding, but it is not the only concern there.  As I also said, I think classes are the right level of abstraction for data binding, and I have misgivings about opening up a new avenue for mucking with module internals.  Along those lines, what I'd really like to see is a tighter data binding focus, revolving around a distinction expressible in DSL between public module components and internal ones.  Given such a computationally-observable distinction, I would specifically like to see

1. Module-specific data along the lines R.I. proposed, or similar.
2. Private classes of a module skipping general data sources, and relying directly and only on module-specific data.

If I understand Alessandro correctly, that might serve as a sound foundation for what he wants to do, too.


John

[Alessandro Franceschi]

On Tuesday, May 20, 2014 6:25:10 PM UTC+2, John Bollinger wrote:

On Monday, May 19, 2014 11:42:57 AM UTC-5, Alessandro Franceschi wrote:

The hierarchy of such a tp module has to be module specific and should not depend on how data is managed in users’ hiera.yaml.

Default data for the managed applications should be placed in the same tp module and be based on a module specific hierarchy, it would contain references to osfamily/operatingsystem/etc facts that can’t be forced into the users’ own local hierarchies (besides that fact that imho in a sane /etc/puppet/hiera.yaml file there should not be references to OS related facts) .

Ok, but you're throwing a curve there.  Your original suggestion / request had none of those constraints.  Would those concerns be adequately addressed if R.I.'s data in modules were in the core product?  Alternatively, would it be acceptable for the 'tp' module to depend on a module providing that feature?  Or to provide that or something equivalent itself?

The constraints were somehow implicit in how I described the expected behaviour, I suppose.
The module can have other dependences, if needed.
I've started to do something around it, still far from working as expected, it's more a readme driven development, and I'm evaluation alternative approaches both for the defines and the parameters to use and how to structure the internal data.
Some experiments are online https://github.com/example42/puppet-tp feel free to comment / suggest, as everything there is under discussion.
I'd like to use internally hiera in some way, but I don't think it's still possible, so I'm going to use a custom "tp_lookup" function that is going to mimics its basic functionality. 
If you have any suggestion on how to use directly Hiera that would be welcomed, as usual. 

 

> Thanks anyway for the attempt, hopefully now is clearer why I can’t do this with existing functionalities.

Your requirements are clearer, yes.  What's not quite clear yet is whether you want to avoid your data binding falling back to general data when resource parameters are not found in module-specific data.  If you do want to avoid that then I think you're right that existing functionalities won't do what you want, but then I'm certain that you were wrong earlier about what you want being achievable by a simple modification.

Well, if data binding for defines is not a possibile approach, the module will have its internal default values for  parameters and accept (of course) users' overrides.
I still have to figure out how to merge user provided data with internal default data ( I think it make sense to expose a parameter that let the user define the merge behaviour)  
 

On the other hand, I think you could put a custom function into module 'tp' that would read data from (only) a module-specific data source, along lines similar to hiera.  If you used such a function instead of hiera(), then could a solution along the lines I described work for you?

Yep, the tp_lookup function mentioned earlier is expected to do what I'd have preferred to do via Hiera data bindings (on defined params) + module in data.

thanks,
al 

John

[John Bollinger]

On Monday, May 26, 2014 11:01:59 AM UTC-5, Alessandro Franceschi wrote:

On Tuesday, May 20, 2014 6:25:10 PM UTC+2, John Bollinger wrote:

[Al:]
 

If you have any suggestion on how to use directly Hiera that would be welcomed, as usual. 

 

> Thanks anyway for the attempt, hopefully now is clearer why I can’t do this with existing functionalities.

Your requirements are clearer, yes.  What's not quite clear yet is whether you want to avoid your data binding falling back to general data when resource parameters are not found in module-specific data.  If you do want to avoid that then I think you're right that existing functionalities won't do what you want, but then I'm certain that you were wrong earlier about what you want being achievable by a simple modification.

Well, if data binding for defines is not a possibile approach, the module will have its internal default values for  parameters and accept (of course) users' overrides.
I still have to figure out how to merge user provided data with internal default data ( I think it make sense to expose a parameter that let the user define the merge behaviour)  
 

To what extent would you still have this problem if you could use hiera directly?  I'm thinking that the issue there might be only that there are parameters that you wish to actively prevent the user from overriding.  If that's the only thing, then would you be willing to give up active enforcement for documentation (i.e. telling users "don't do that")?

 

On the other hand, I think you could put a custom function into module 'tp' that would read data from (only) a module-specific data source, along lines similar to hiera.  If you used such a function instead of hiera(), then could a solution along the lines I described work for you?

Yep, the tp_lookup function mentioned earlier is expected to do what I'd have preferred to do via Hiera data bindings (on defined params) + module in data.

So, if you're willing to depend on other modules, then I think ripienaar/module-data combined with explicit lookups can do a lot of what you're after.  It gives you an always-on (doesn't depend on the user modifying hiera.yaml), in-module data source from which you can look up the needed data.  For defined types and current Puppet, that necessarily involves explicit lookups, as we already discussed.

If that combination leaves essential features unserved, then perhaps at least R.I.'s module can give you ideas for how to build your own -- maybe just the same thing but defaulting to highest priority.


John

[Daniele Sluijters]

> So, if you're willing to depend on other modules, then I think ripienaar/module-data combined with explicit lookups can do a lot of what you're after.

Honestly, that should just be part of Puppet core. It's such an incredibly useful feature and would go a long way to help people out in such scenario's. It might not be the utopia Alessandro is shooting for but it's certainly a good compromise.

[Trevor Vaughan]

+1 Data in modules is absolutely necessary!

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/579481e0-f2bd-4162-b846-6602824c3c4e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alessandro Franceschi]

Hi all,

let me reply to this old thread to announce the release of Tiny Puppet, a module that allows installation and configuration of virtually any application on any OS.

This post was originally started to find a way to solve a problem at the core of Tiny Puppet working concept (based on defines and not classes). I then followed another approach, based on a custom function, as suggested by John in one of the replies, which mimics Hiera behaviour and gets inspiration from the data in module concept.

For more details on Tiny Puppet refer to the post on Puppet Users and the links there:
https://groups.google.com/forum/#!topic/puppet-users/_ZhYQBC8UP0

Incidentally, I'm quite proud and excited by this achievement and would be grateful to hear any feedback or comment about it (eventually directly in the Puppet Users announce post).

Best regards and wishes for a great new year to everybody,

Alessandro Franceschi