NextGen Modules: Lessons learned

[Alessandro Franceschi]

I've posted a blog post about my current ideas on NextGen modules and how I evaluate now some choices made in the past.

The Article is here:
http://www.example42.com/?q=NextGen_Modules_Lessons_learned 

It contains also some insights on how and when could be the next iteration of Example42 modules, which somehow is a follow up to what has been discussed in
https://groups.google.com/forum/#!topic/example42-puppet-modules/NqtPD3c0KLc

Since I'd like to involve more people as maintainers of some specific modules (for example the ones related to application I don't use or know well) your opinion on what has been written in the post and on the general matter of the evolution of these modules is very welcomed, here.

[Javier B]

El lunes, 2 de septiembre de 2013 10:44:42 UTC-3, Alessandro Franceschi escribió:

I've posted a blog post about my current ideas on NextGen modules and how I evaluate now some choices made in the past.

The Article is here:
http://www.example42.com/?q=NextGen_Modules_Lessons_learned 

Ciao Alessandro,

I've read your article, and here I am, sharing my thoughts, as requested :)

First thing first: good article!

Looking toward the future of the ex42 modules, I think you clearly summarize the current situation and it's also clear you (we all?) should think of a roadmap on the issues you arise in the article.

---
In the case of data bindings and params_lookup(), I think that ex42 modules have grown up to the level where, for a given feature, you need to set a working "transition release version", where you use the new Y feature while still supporting the old X feature that will be deprecated in the next release cycle (commonly, providing a warning about that).

This is in the same spirit that puppet guys did with, i.e., variable naming in templates (warning about the usage of vars without "@").

In the case of params_lookup() we can begin by adding a deprecation warning in the function, working like this (just a though to make my point clear). When we call param_lookup('bla'),

 * If the var 'bla' is found in the format 'module::bla', we return the value.
 * If the var 'bla' is found in the format 'module_bla' or other of the supported formats, we rise a deprecation warning and return the value.

  That way, people using the module will know that in the next semantic version bump of the package (let's say from 1.y.z to 2.0.0), the params_lookup function will not be supported, and they've been warned in advance.

  Probably you need to begin having 2 branches, "current" and "nexgen", to manage all these? More on this at the end.

---
  Regarding "templates vs. options", although I like the idea of a generic "options" array/hash, I think it misses the point on variable sanitizing/using inside the module, and variables passed as parameters are better for that. I mean, when you write

some_class{ 'name':
  mainp => 'value'
}

class some_class (
  mainp = ''
) {

  $var = $mainp ? {
    '' => undef,
    default => 'somevalue'
  }

$var value depends specifically of the value of "mainp". Sure, you can do exactly the same using "options['mainp']", but in that case, the only thing you gain is using a "single parameter" instead of a bunch of them. The logic of $var REQUIRES that a value for a parameter named "mainp" (or whatever, but specific) exist.

And the same reasoning goes to variable vs options[] when refering to templates (on some vars you apply ".each", "if"s, and many similar things that depend specifically on the var used).

My thinking is that whe need to provide BOTH things and we need to add, at some point in the template,  $options expansion in a simple way (maybe as augeas does)?

template things
...
...
param = <%= @var %>
...
some
more
stuff
...
<% scope.lookupvar('module::options').each do |opt_v| -%>
<%= opt_v -%>
<% end -%>

That way, everything that is specific can be given and manipulated as desired in the module, yet anything that is not specific can be still be used and provided by the user of the module.

This reminds me of the way the define "example42-iptables::rule" handle the parameter "rule", as a "catch all" parameter that can be used to set iptables rules that are not covered by the other parameters it accepts (destination, source, port, etc.).

As usual, I'm not sure if I'm being clear :)

---
Regarding "dependency_class": +1!.

---
 Regarding "what's next", I think ex42 mods are the most cohesive group of modules I've ever seen. Even many "puppetlabs" modules I've seen are not reusable at all. Sure, they are very good at setting up app X, but they are not even similar to any other puppetlabs module and, in that sense, ex42 modules REALLY set the bar high.

As I said at the beginning, I think ex42 modules have grown to the point where different branches should be maintained, kind of stable, nexgen, others?, to ease adoption of the modules by newcomers without slowing down the adoption of new features.

I would like to see these modules grow and will be really glad to help in anything I can.

Well, my 2 cents.

Regards,

   Javier

[Alessandro Franceschi]

On Thursday, September 5, 2013 4:02:58 PM UTC+2, Javier B wrote:

El lunes, 2 de septiembre de 2013 10:44:42 UTC-3, Alessandro Franceschi escribió:

I've posted a blog post about my current ideas on NextGen modules and how I evaluate now some choices made in the past.

The Article is here:
http://www.example42.com/?q=NextGen_Modules_Lessons_learned 

Ciao Alessandro,

I've read your article, and here I am, sharing my thoughts, as requested :)

Thanks, really. Replies like this are immensely useful.

First thing first: good article!

Looking toward the future of the ex42 modules, I think you clearly summarize the current situation and it's also clear you (we all?) should think of a roadmap on the issues you arise in the article.

A note on how I'd like to bring on the developments, FYI.

I'd like to keep on defining the design structure of the modules, but on doing this I definitively welcome proposals and correctlons from who is interested. Once "standard", modules layouts and basic logic is defined I'd try definitively to involve directly other authors in maintaining specific modules, as you guys from Netmanagers, and others, are already doing for some nextgen modules.

Actually I'd prefer to follow this approach even more , as I don't see any sense in trying to maintain a module for an application I don't use or know well.
 

---
In the case of data bindings and params_lookup(), I think that ex42 modules have grown up to the level where, for a given feature, you need to set a working "transition release version", where you use the new Y feature while still supporting the old X feature that will be deprecated in the next release cycle (commonly, providing a warning about that).

+1

 

This is in the same spirit that puppet guys did with, i.e., variable naming in templates (warning about the usage of vars without "@").

In the case of params_lookup() we can begin by adding a deprecation warning in the function, working like this (just a though to make my point clear). When we call param_lookup('bla'),

 * If the var 'bla' is found in the format 'module::bla', we return the value.
 * If the var 'bla' is found in the format 'module_bla' or other of the supported formats, we rise a deprecation warning and return the value.

  That way, people using the module will know that in the next semantic version bump of the package (let's say from 1.y.z to 2.0.0), the params_lookup function will not be supported, and they've been warned in advance.

+1
Even this might end up in quite a lot or warnings.

Also there a pair of things that leaves me doubts, I don't know how to replicate in "pure" Hiera + Puppet 3 data bindings
- The global option of params lookup

- The lookup of a top scope variable (I mean, with Hiera's puppet backend you can define a class where to look for variables, but does this work also on the top scope? What's the exact configuration syntax? Any idea? )

 

  Probably you need to begin having 2 branches, "current" and "nexgen", to manage all these? More on this at the end.

Silly Hamletic doubt of the day:
Will "NextGen" modules always be the newer ones or they will keep on being the "2nd gen" modules, and the "3rd gen" ones will have a different name (or just called 3.x).?

Well, more or less I think the same.
I would keep parameters for all the options that:
- Are quite important or qualifying for a given application (ie: allowed hosts for nrpe, syslog server, in any, for rsyslog... )
- Are used also to set value of Puppet variables or define in to include extra resources (ie: use_ssl for apache).

I just don't like the idea of having a class parameter for any (or almost) single configuration entry of an application.
(if I look at some ex42 modules, besides the too large list of common params, I also don't like some cases where the number of params has grown out of control)

 

---
Regarding "dependency_class": +1!.

---
 Regarding "what's next", I think ex42 mods are the most cohesive group of modules I've ever seen. Even many "puppetlabs" modules I've seen are not reusable at all. Sure, they are very good at setting up app X, but they are not even similar to any other puppetlabs module and, in that sense, ex42 modules REALLY set the bar high.

As I said at the beginning, I think ex42 modules have grown to the point where different branches should be maintained, kind of stable, nexgen, others?, to ease adoption of the modules by newcomers without slowing down the adoption of new features.

Yes, I think I will keep on having the same modules and manage in branches the current  and the future gen.

 

I would like to see these modules grow and will be really glad to help in anything I can.

You're *very* welcomed.

I add here some ideas on what I've been thinking about lately, regarding the new modules, still no sample code, just some random points to mumble about:

- Monitoring / Firewalling management. Currently they require a lot of params, almost just made for these functions, and the monitor / firewall meta modules implementation sucks. I was thinking at a pair of approaches for the future:

1 - Only 2 params for each function: monitor / monitor_hash , the first one is a boolean, the second an hash.
In the monitor_hash there's everything:

- what to monitor like  port, protocol, process name, service name... 

- the "monitor_tools" to sue
- eventually some tool specific setting with a proper namespace (ie: nagios::service_template, eye::cpu_low ...).
There's has been on GitHub some discussion with mburger and others on this, and is definitively important to have the option to add monitor tool specific params.
Technically speaking is already possible to define in the module the default hash (with correct parameters for the different OS) and merge it with the one the user may provide and with the future parser is possible to make operations on hashes, like filtering out keys or making operations on them, that could be useful to manage the tool specific keys.

2- Think about a way to provide monitoring abstraction outside the module, so that it's not made heavier but all the required params and might eventually be module-neutral. The basic cons here are that you might need to replicate and keep aligned data and that's never a good thing.

Also the whole monitor/firewall resources according to the tool, might be created directly using the injected data and the create_resources function, which applies well for cases like this.

Finally monitor could become a type, and the different monitor tools just providers (as it happens with PuppetLabs' firewall module). But the fact that AFAIK nobody has already done something like this, and some glimpses of self awareness, make me think that is not  an easy thing to do.

- Modules and stacks.

Besides the dependency_class part, which is just a piece of the whole problem of modules interoperability, I think we should explore more layout designs for stacks and be more aggressive in having modules that manage only the application they have to manage. This is not always done on all ex42 modules (generally there are params that let you override some defaults that are needed to make the module do something useful out of the box, but that's just a workaround).
With established stacks, eventually distributed with the modules they use, it's much easier to keep things separated, as you can move into the stack and out of the module all the integration parts.

Also I will probably start to use the puppetdbquery module as it's just great and it opens easily to the huge amount of information in puppetdb. I'm thinking about stacks that automatically manage autoscaling fetching info about other nodes resources (exported or not) directly on puppetdb.

I still haven't anything done here, just exploring what's around. 

- Hiera2 and data in modules + future parser.
Here I think we are at the beginning of a new revolution in modules design, and for this reason I don't want to rush into doing 3rd gen modules... I prefer to keep on working on the nextgen ones, as they seem to work well for current setups (besides the overhead of params_lookup in Puppet3 and the too wide number of params) and explore different module layouts to find what works better for the future.

Any idea on these and similar topics is gold.

al

[Javier B]

El jueves, 5 de septiembre de 2013 13:32:13 UTC-3, Alessandro Franceschi escribió:

A note on how I'd like to bring on the developments, FYI.

I'd like to keep on defining the design structure of the modules, but on doing this I definitively welcome proposals and correctlons from who is interested. Once "standard", modules layouts and basic logic is defined I'd try definitively to involve directly other authors in maintaining specific modules, as you guys from Netmanagers, and others, are already doing for some nextgen modules.

Works for me :)

 

Also there a pair of things that leaves me doubts, I don't know how to replicate in "pure" Hiera + Puppet 3 data bindings

- The global option of params lookup

- The lookup of a top scope variable (I mean, with Hiera's puppet backend you can define a class where to look for variables, but does this work also on the top scope? What's the exact configuration syntax? Any idea? )

"I understand" (though haven't yet tried it :) that the fully qualifying of variables does the magic here. It would become something like:

$do_monitor = $::monitor or $::module::monitor

--
On branches and versions

  Probably you need to begin having 2 branches, "current" and "nexgen", to manage all these? More on this at the end.

Silly Hamletic doubt of the day:
Will "NextGen" modules always be the newer ones or they will keep on being the "2nd gen" modules, and the "3rd gen" ones will have a different name (or just called 3.x).?

I was thinking it in the Debian's way ("sid" will always be unstable) or in the kernel's "next" branch, so "nextgen" would always be "the future", current would be 2.x, next one 3.x and so on?

You decide here... It's OK with me even if you decide to use "types of salamis around the world" as codenames :D

---
  Regarding "templates vs. options"

 

[etc.,etc.]

 

Well, more or less I think the same.
I would keep parameters for all the options that:
- Are quite important or qualifying for a given application (ie: allowed hosts for nrpe, syslog server, in any, for rsyslog... )
- Are used also to set value of Puppet variables or define in to include extra resources (ie: use_ssl for apache).

I just don't like the idea of having a class parameter for any (or almost) single configuration entry of an application.
(if I look at some ex42 modules, besides the too large list of common params, I also don't like some cases where the number of params has grown out of control)

+1

 

---
 Regarding "what's next",

 

Yes, I think I will keep on having the same modules and manage in branches the current  and the future gen.

 

I add here some ideas on what I've been thinking about lately, regarding the new modules, still no sample code, just some random points to mumble about:

- Monitoring / Firewalling management. Currently they require a lot of params, almost just made for these functions, and the monitor / firewall meta modules implementation sucks. I was thinking at a pair of approaches for the future:

1 - Only 2 params for each function: monitor / monitor_hash , the first one is a boolean, the second an hash.
In the monitor_hash there's everything:

- what to monitor like  port, protocol, process name, service name... 

- the "monitor_tools" to sue
- eventually some tool specific setting with a proper namespace (ie: nagios::service_template, eye::cpu_low ...).
There's has been on GitHub some discussion with mburger and others on this, and is definitively important to have the option to add monitor tool specific params.
Technically speaking is already possible to define in the module the default hash (with correct parameters for the different OS) and merge it with the one the user may provide and with the future parser is possible to make operations on hashes, like filtering out keys or making operations on them, that could be useful to manage the tool specific keys.

Can you provide the link that discussion?

I see here the same issue discussed before on the "options" hash:  with a hash you lose control on its content inside your recipe, unless you "evaluate" this hash's content in a fixed way inside the recipe. Then, you are just getting rid of having a lot of parameters in the module declaration, but just "hiding" them all inside the hash.

Still I'm not sure of the benefits of each choice.

2- Think about a way to provide monitoring abstraction outside the module, so that it's not made heavier but all the required params and might eventually be module-neutral. The basic cons here are that you might need to replicate and keep aligned data and that's never a good thing.

Also the whole monitor/firewall resources according to the tool, might be created directly using the injected data and the create_resources function, which applies well for cases like this.

Finally monitor could become a type, and the different monitor tools just providers (as it happens with PuppetLabs' firewall module). But the fact that AFAIK nobody has already done something like this, and some glimpses of self awareness, make me think that is not  an easy thing to do.

I agree that firewalling and monitoring should become a type. Although I'm a complete newbie in ruby, would like to help if you have some ideas regarding this :)
 

- Modules and stacks.

Besides the dependency_class part, which is just a piece of the whole problem of modules interoperability, I think we should explore more layout designs for stacks and be more aggressive in having modules that manage only the application they have to manage. This is not always done on all ex42 modules (generally there are params that let you override some defaults that are needed to make the module do something useful out of the box, but that's just a workaround).
With established stacks, eventually distributed with the modules they use, it's much easier to keep things separated, as you can move into the stack and out of the module all the integration parts.

+1

- Hiera2 and data in modules + future parser.

Here I think we are at the beginning of a new revolution in modules design, and for this reason I don't want to rush into doing 3rd gen modules... I prefer to keep on working on the nextgen ones, as they seem to work well for current setups (besides the overhead of params_lookup in Puppet3 and the too wide number of params) and explore different module layouts to find what works better for the future.

+1

Javier

[Frode Egeland]

I'll just chime in on the naming..

I think the 'NextGen' name is confusing. I get the Debian 'sid' analogy, but I don't think it's clear enough.
Maybe we should have branches that correspond to the major puppet version they are developed for, so '2.x', '3.x', and so on? Or 'version_2', 'version_3', etc..?

It'd be good if we could decide quickly, as (as seen in my other post to this mailing list) I'm having some issues with the params_lookup function and how it works (or rather, doesn't work) with hiera in puppet 3.x.

It it's not fixable, I'd need somewhere to commit my changes (pull requests) - I'm thinking I'll be removing params_lookup and using the default hiera behaviour, then if the param is still empty, look up from params.pp.

Cheers,

Frode

[Alessandro Franceschi]

On Monday, September 9, 2013 4:08:04 AM UTC+2, egeland wrote:

I'll just chime in on the naming..

I think the 'NextGen' name is confusing. I get the Debian 'sid' analogy, but I don't think it's clear enough.
Maybe we should have branches that correspond to the major puppet version they are developed for, so '2.x', '3.x', and so on? Or 'version_2', 'version_3', etc..?

Yes, I'm too more inclined to leave nextgen as the version 2 and call the next one 3.

 

It'd be good if we could decide quickly, as (as seen in my other post to this mailing list) I'm having some issues with the params_lookup function and how it works (or rather, doesn't work) with hiera in puppet 3.x.

It it's not fixable, I'd need somewhere to commit my changes (pull requests) - I'm thinking I'll be removing params_lookup and using the default hiera behaviour, then if the param is still empty, look up from params.pp.

We have to fix this, obviously.

I'd suggest anyway, to start using hiera variables in data bindings format (module::parameter), since that's the "default" and that is what is going to work in the future.

[Michael Jerger]

Hi,

I've followed your discussion and think, that we are discussing typically programming language design topics.

I think, hashes are some sort of concept of "object serialisation". So in languages like java or clojure we would be able to define (and hand over as parameter) a firewall parameter object belonging to the firewall module.

The Java example is sth. like:

package example42.firewall
class FirewallParameter {
  public boolean firewall            = false;
  public String  firewall_src        = "0/0";
  public String  firewall_dst        = "0/0";
  public Integer firewall_port       = 22;
  public String  firewall_protocol   = "tcp";
}

These java classes can be serialized (write to a binary stream) and deserialized (create object & read from stream). In clojure the serialisation format is even not a binary stream but a json like string.

The difference to the discussed hash solution is, that java or clojure

• binds the ability to serialize and deserialize to the defined class - so
  
• clients has no additional need to code the correct serialisation and
• parameter verification can be written to FirewallParameter class directly.
• allows to clients and the firewall component to directly address (read or write) values like sshFirewallParameter.firewall_src = "1.2.3.4";

Without the ability to encapsulate serialisation information, serialization code and the way of convenient access I see no chance to avoid copy / past redundancy and heavy to analyse incompatibility.

Still I'm not sure of the benefits of each choice.

2- Think about a way to provide monitoring abstraction outside the module, 

so that it's not made heavier but all the required params and might 
eventually be module-neutral. The basic cons here are that you might need 
to replicate and keep aligned data and that's never a good thing.

Also the whole monitor/firewall resources according to the tool, might be 
created directly using the injected data and the create_resources function, 
which applies well for cases like this.

Finally monitor could become a type, and the different monitor tools just 
providers (as it happens with PuppetLabs' firewall module). But the fact 
that AFAIK nobody has already done something like this, and some glimpses 
of self awareness, make me think that is not  an easy thing to do.

I agree that firewalling and monitoring should become a type. Although I'm 
a complete newbie in ruby, would like to help if you have some ideas 
regarding this :)

I think that's a good way - so we will have a general module "firewall" and "monitoring" collecting all the common needed information / parameters.

I propose also to provide a central class for common configuration.


Michael

[Alessandro Franceschi]

https://github.com/example42/puppet-mcollective/pull/9

 

I see here the same issue discussed before on the "options" hash:  with a 
hash you lose control on its content inside your recipe, unless you 
"evaluate" this hash's content in a fixed way inside the recipe. Then, you 
are just getting rid of having a lot of parameters in the module 
declaration, but just "hiding" them all inside the hash.

I think, hashes are some sort of concept of "object serialisation". So in languages like java or clojure we would be able to define (and hand over as parameter) a firewall parameter object belonging to the firewall module.

The Java example is sth. like:

package example42.firewall
class FirewallParameter {
  public boolean firewall            = false;
  public String  firewall_src        = "0/0";
  public String  firewall_dst        = "0/0";
  public Integer firewall_port       = 22;
  public String  firewall_protocol   = "tcp";
}

These java classes can be serialized (write to a binary stream) and deserialized (create object & read from stream). In clojure the serialisation format is even not a binary stream but a json like string.

The difference to the discussed hash solution is, that java or clojure

• binds the ability to serialize and deserialize to the defined class - so
  
• clients has no additional need to code the correct serialisation and
• parameter verification can be written to FirewallParameter class directly.
• allows to clients and the firewall component to directly address (read or write) values like sshFirewallParameter.firewall_src = "1.2.3.4";

Without the ability to encapsulate serialisation information, serialization code and the way of convenient access I see no chance to avoid copy / past redundancy and heavy to analyse incompatibility.

Not sure of what are your final conclusions about the opportunity to use hashes to provide configuration key pairs.
What I can say is that we can already access values in a hash in a template and also in Puppet manifests, and that the future parser will allow better manipulations of hashes:
http://docs.puppetlabs.com/puppet/3/reference/lang_experimental_3_2.html#available-functions
Also I thank that they are almost unavoidable when dealing with an arbitrary, not predefined and unknown number of parameters, as could be some monitoring tool specific ones.

Still I'm not sure of the benefits of each choice.

2- Think about a way to provide monitoring abstraction outside the module, 

so that it's not made heavier but all the required params and might 
eventually be module-neutral. The basic cons here are that you might need 
to replicate and keep aligned data and that's never a good thing.

Also the whole monitor/firewall resources according to the tool, might be 
created directly using the injected data and the create_resources function, 
which applies well for cases like this.

Finally monitor could become a type, and the different monitor tools just 
providers (as it happens with PuppetLabs' firewall module). But the fact 
that AFAIK nobody has already done something like this, and some glimpses 
of self awareness, make me think that is not  an easy thing to do.

I agree that firewalling and monitoring should become a type. Although I'm 
a complete newbie in ruby, would like to help if you have some ideas 
regarding this :)

I think that's a good way - so we will have a general module "firewall" and "monitoring" collecting all the common needed information / parameters.

 

I propose also to provide a central class for common configuration.

What you mean with common configuration?
Would you make some examples of this central class?