Rethinking Puppet modules

[Alessandro Franceschi]

Hi all,
I've written a blog post with my opinions on the current modules ecosystem:
http://www.example42.com/2014/05/31/rethinking-modules-part-1/

The post talks about :

• What are the reusability features a module should have, imho
• The distinction between application (component) modules and higher abstraction modules
• What are the challenges and reusability options for higher abstraction modules

It also raises a pair of points of the challenges that, according to me, we should face in our future modules: 

• Patterns to extend reusability of higher abstraction layer modules 
• Standardisation in the component application modules

I'm truly interested in hearing others' opinions on these topics, in order to understand if such points are important only for me or they reflect common concerns.

Any feedback welcomed

Alessandro

[jcbollinger]

I have never accepted the premise that "reusable" as applied to Puppet modules has to mean "usable in any context, to achieve any end related to the module's area of application, via any conceivable style," which is pretty much how I read your reusability criteria for component modules.  Certainly a module that satisfies those criteria can serve pretty much any purpose within its domain, but what matters to me at any given time is whether a module can serve my purpose, whatever that happens to be.

If some third-party modules are (re)usable for me but not for (say) Felix, and vise versa, then that doesn't change the fact that both our jobs are made easier by not having to write everything from scratch.  If that means I need to study a module a bit before I decide whether to use it, then fine -- I ought to do that anyway.

Moreover, I think it is harmful to the module ecosystem to promote a position that only Cadillac modules are "reusable," because that sets the bar very high for potential contributors.  I don't think it helps anyone to tell people that their contributions are unwelcome (which branding them "not reusable" does).  Most people who write modules do so for their own purposes first and foremost, and if the community is unwelcoming to them then they will just keep their work to themselves.


I agree, however, that there is a useful distinction between "component modules" and higher-level modules, and also that some higher-level modules are conceivably reusable.  I furthermore agree that the criteria you set out all tend to make such a higher-level module either applicable to more use cases or easier to integrate.

On the other hand, I tend to think that higher-level modules will less frequently be used together on the same nodes than are component modules, so that integration considerations are less of an issue.  I also tend to think that reusable higher-level modules will be fewer than component modules, so that standardization is less of a consideration for them.  Of course, I think I ascribe less importance overall to standardization (of Puppet modules) than you do.


I have recently decided, too, that more of the responsibility for module reusability has been placed on module authors than is needful or useful.  Puppet itself could and should provide more support than it does.  In particular, I would like to see more flexible data binding (e.g. a mechanism for data aliases and/or indirect data names, a mechanism for slicing data, improved DSL-based data injection, etc.), and more flexible class / module resolution (e.g. aliases / indirection).  These sorts of features would improve all modules' reusability by opening more flexible approaches to integrating modules into a site's manifest set.


John

[Trevor Vaughan]

+1 to everything that John has said.

While many Forge modules are not useful to me out of the box, I have had good luck modifying them to meet my needs at any given time.

I've also been able to learn new ways of doing things (and not doing things) due to the willingness of people to share.

I do think that some of the recent proposals around allowing multiple modules that cover the same material to exist on the same system are a good thing and could help with reusability in the future. In this case, you would get chunks of reusability that work together but that happily coexist with other isolated chunks.

It's good to have these types of discussions though. Knowing different users' ideas of perfection can encourage different automated methods of attempting to attain perfection in the future (puppet-lint for example).

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e30a76b9-3f1a-4408-8cc4-d4c2a2207874%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Alessandro Franceschi]

First of all, let me thank you, John, for replying to this post and giving a sense to the discussion.

At times I wonder if PuppetLabs' guys read the forum and, if they do, why they don't express an opinion on these matters.
I might be obsessive about modules, but I think that it's here where we play the most important game on user code reusability. 

On Friday, June 20, 2014 5:04:28 PM UTC+2, jcbollinger wrote:

On Thursday, June 19, 2014 11:09:22 AM UTC-5, Alessandro Franceschi wrote:

Hi all,
I've written a blog post with my opinions on the current modules ecosystem:
http://www.example42.com/2014/05/31/rethinking-modules-part-1/

The post talks about :

• What are the reusability features a module should have, imho
• The distinction between application (component) modules and higher abstraction modules
• What are the challenges and reusability options for higher abstraction modules

It also raises a pair of points of the challenges that, according to me, we should face in our future modules: 

• Patterns to extend reusability of higher abstraction layer modules 
• Standardisation in the component application modules

I'm truly interested in hearing others' opinions on these topics, in order to understand if such points are important only for me or they reflect common concerns.

I have never accepted the premise that "reusable" as applied to Puppet modules has to mean "usable in any context, to achieve any end related to the module's area of application, via any conceivable style," which is pretty much how I read your reusability criteria for component modules.  Certainly a module that satisfies those criteria can serve pretty much any purpose within its domain, but what matters to me at any given time is whether a module can serve my purpose, whatever that happens to be.

Reusability is definitively a relative concept, and I agree it can have different nuances, still, IMHO, there are a few reusability patterns that are easy to introduce and add a great value (for example the possibility, for the module's user to provide custom templates for configuration files instead of the ones proposed by default by the module. This is easy to introduce and should definitively be common practice).

 

If some third-party modules are (re)usable for me but not for (say) Felix, and vise versa, then that doesn't change the fact that both our jobs are made easier by not having to write everything from scratch.  If that means I need to study a module a bit before I decide whether to use it, then fine -- I ought to do that anyway.

Moreover, I think it is harmful to the module ecosystem to promote a position that only Cadillac modules are "reusable," because that sets the bar very high for potential contributors.  I don't think it helps anyone to tell people that their contributions are unwelcome (which branding them "not reusable" does).  Most people who write modules do so for their own purposes first and foremost, and if the community is unwelcoming to them then they will just keep their work to themselves.

I understand the point, but at the same time I find it harmful also to have hundreds of modules, that are published on the forge but work only for a specific case and are practically useless for others.

It's not a matter of preventing or discouraging users from publishing their work, but to have a good overall quality of public modules and to encourage good practices. I'd rather work on trying to convey and promote basic reusability patterns that can be used and embraced by everybody.

Also, I don't think that a reusable module is necessarily a good one or viceversa, actually in most the cases reusability features add complexity and overheads.

 

I agree, however, that there is a useful distinction between "component modules" and higher-level modules, and also that some higher-level modules are conceivably reusable.  I furthermore agree that the criteria you set out all tend to make such a higher-level module either applicable to more use cases or easier to integrate.

On the other hand, I tend to think that higher-level modules will less frequently be used together on the same nodes than are component modules, so that integration considerations are less of an issue.  I also tend to think that reusable higher-level modules will be fewer than component modules, so that standardization is less of a consideration for them.  Of course, I think I ascribe less importance overall to standardization (of Puppet modules) than you do.


I have recently decided, too, that more of the responsibility for module reusability has been placed on module authors than is needful or useful.  Puppet itself could and should provide more support than it does.  In particular, I would like to see more flexible data binding (e.g. a mechanism for data aliases and/or indirect data names, a mechanism for slicing data, improved DSL-based data injection, etc.), and more flexible class / module resolution (e.g. aliases / indirection).  These sorts of features would improve all modules' reusability by opening more flexible approaches to integrating modules into a site's manifest set.

All good points that diverge a bit from the discussed topics and are not alternative.
I suppose priorities are rather subjective, after all.

Incidentally, I've published the second part of the blog post this thread is about:

http://www.example42.com/2014/06/22/rethinking-modules-part-2/
it describes better that Tiny Puppet idea I introduced in another thread we talked about: https://groups.google.com/forum/#!topic/puppet-dev/4lFhfChM9XM 
(and yes, at the end I mimicked Hiera in the module with a custom function)

Al

[Alessandro Franceschi]

On Friday, June 20, 2014 8:01:49 PM UTC+2, Trevor Vaughan wrote:

+1 to everything that John has said.

While many Forge modules are not useful to me out of the box, I have had good luck modifying them to meet my needs at any given time.

Well, in my very personal opinion, if someone takes a module of mine and has to modify it in order to make it meet his needs, I've lost.

I think, as a general rule, that every Puppet setup should have at least 2 separated directories in the modulepath.
A shared, common, one, with public modules downloaded from the Forge or other sources, and not modified.

A site, local, one, with local modules and eventually the public modules that have been downloaded and changed (the less of these, the better) for local needs.

I've also been able to learn new ways of doing things (and not doing things) due to the willingness of people to share.

I do think that some of the recent proposals around allowing multiple modules that cover the same material to exist on the same system are a good thing and could help with reusability in the future. In this case, you would get chunks of reusability that work together but that happily coexist with other isolated chunks.

Are you referring to these?
https://groups.google.com/forum/#!topic/puppet-dev/em-WGLCrdIw
https://groups.google.com/forum/#!topic/puppet-dev/dkdXaUt8yDg 

It's good to have these types of discussions though. Knowing different users' ideas of perfection can encourage different automated methods of attempting to attain perfection in the future (puppet-lint for example).

Indeed, and actually this was the reasoning behind this post, as sometimes I wonder how some things I consider obvious or necessary really matter and are considered useful by others.

I generally try to figure out solutions following the current status of the Puppet language (I consider myself an end user) also considering the inevitable lapse of time that occurs from a feature development to its usage at large.
Still, proposals as the ones expressed in the linked threads are definitively worth attention, thanks

Al

[Pablo Fernandez]

Hi,

I think you (example42) hit the nail in the head in the "rethinking
modules part 1" with one tiny exception... but that error was multiplied
by x100 in the second part of your dissertation.

The exception I am talking about is the desirability to pass a template
as a parameter: I agree you should be able to pass a different template,
but I think that should be the exception for extreme cases. The idea of
the parameters is precisely to avoid passing a file!!

In my ideal world, where I manage several heterogeneous systems, I want
to be able to glance at one file and see how one node or cluster is
configured: that would be the hiera data. Having a complex hierarchy
with three or four files to check is already a burden... but it's quite
acceptable for the sake of avoiding duplicities. But if I have to check
every different template that is passed to the modules to understand
what is going on, this capability is completely lost.

I am an advocate of the current trend: make good modules that serve
everyone (with a restricted purpose), and improve them so that it really
serves everyone and they don't to fork (contribute!!). Then higher
abstraction modules with wider purpose, configurable so that it serves
many people too, and there you go. So, exactly what you described in the
part 1... but not in the part 2, sorry.

My two cents!
BR/Pablo

[Alessandro Franceschi]

On Monday, June 23, 2014 6:49:00 PM UTC+2, pablo.f...@cscs.ch wrote:

Hi,

I think you (example42) hit the nail in the head in the "rethinking
modules part 1" with one tiny exception... but that error was multiplied
by x100 in the second part of your dissertation.

The exception I am talking about is the desirability to pass a template
as a parameter: I agree you should be able to pass a different template,
but I think that should be the exception for extreme cases. The idea of
the parameters is precisely to avoid passing a file!!

In my ideal world, where I manage several heterogeneous systems, I want
to be able to glance at one file and see how one node or cluster is
configured: that would be the hiera data. Having a complex hierarchy
with three or four files to check is already a burden... but it's quite
acceptable for the sake of avoiding duplicities. But if I have to check
every different template that is passed to the modules to understand
what is going on, this capability is completely lost.

You might be misunderstanding the point about the possibility to pass a template as a reusability option.

I'm not saying or advocating the idea of having a template for each node, I just think it's a needed reusability feature.
If a class or a define exposes parameters to allow the modification of different variables in a template that's perfectly ok for me, that's the intended usage of templates.

As a side note I'm a bit perplex about the proliferation of parameters in classes or defines in order to match single application specific configuration entries.
When I look at this, for example:
https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp
my backbone freezes.
A huge amount of define parameters, that map Apache directives, and not even a parameter that allows me to provide a custom template.

So that, when I want to create a virtualhost file in the way I want, I have to study and accomodate all the parameters to obtain what I need, when I could just pass a custom template and use a few of the core parameters (Virtualhost name, document root and so on) to populate it dynamically.

Also, if you want a pure data driven setup, where you place on hiera hashes of data with all the stuff you need, you can still have this using a template and a single parameters ( such as the options_hash here: https://github.com/example42/puppet-tp/blob/master/manifests/conf.pp#L19 ) here you can place any application specific setting without the need to add parameters to the define (in your template, of course, you have to use and manage this hash).

The intended usage of defines like tp::conf which rely entirely on external templates or files, is inside higher abstraction modules, where you can expose the parameters you want and which provide a specific template that may use the exposed parameters, keeping static what you don't need to modify in your nodes.

 

I am an advocate of the current trend: make good modules that serve
everyone (with a restricted purpose), and improve them so that it really
serves everyone and they don't to fork (contribute!!). Then higher
abstraction modules with wider purpose, configurable so that it serves
many people too, and there you go. So, exactly what you described in the
part 1... but not in the part 2, sorry.

Hope my clarification exposes better the rationale behind the second part.

Thank you

Al

 

My two cents!
BR/Pablo

[jcbollinger]

On Monday, June 23, 2014 10:34:00 AM UTC-5, Alessandro Franceschi wrote:

First of all, let me thank you, John, for replying to this post and giving a sense to the discussion.

At times I wonder if PuppetLabs' guys read the forum and, if they do, why they don't express an opinion on these matters.
I might be obsessive about modules, but I think that it's here where we play the most important game on user code reusability. 

On Friday, June 20, 2014 5:04:28 PM UTC+2, jcbollinger wrote:

I have never accepted the premise that "reusable" as applied to Puppet modules has to mean "usable in any context, to achieve any end related to the module's area of application, via any conceivable style," which is pretty much how I read your reusability criteria for component modules.  Certainly a module that satisfies those criteria can serve pretty much any purpose within its domain, but what matters to me at any given time is whether a module can serve my purpose, whatever that happens to be.

Reusability is definitively a relative concept, and I agree it can have different nuances, still, IMHO, there are a few reusability patterns that are easy to introduce and add a great value (for example the possibility, for the module's user to provide custom templates for configuration files instead of the ones proposed by default by the module. This is easy to introduce and should definitively be common practice).

I think it's great that you are thinking about such questions, discussing them, and promoting your conclusions.  And I agree that there are reusability patterns that can be applied fairly easily and add substantial value.  But I suspect that if we compared lists, mine would differ from yours.
 

 

If some third-party modules are (re)usable for me but not for (say) Felix, and vise versa, then that doesn't change the fact that both our jobs are made easier by not having to write everything from scratch.  If that means I need to study a module a bit before I decide whether to use it, then fine -- I ought to do that anyway.

Moreover, I think it is harmful to the module ecosystem to promote a position that only Cadillac modules are "reusable," because that sets the bar very high for potential contributors.  I don't think it helps anyone to tell people that their contributions are unwelcome (which branding them "not reusable" does).  Most people who write modules do so for their own purposes first and foremost, and if the community is unwelcoming to them then they will just keep their work to themselves.

I understand the point, but at the same time I find it harmful also to have hundreds of modules, that are published on the forge but work only for a specific case and are practically useless for others.

I disagree.  It is not at all a problem -- in itself -- to have hundreds of narrowly-scoped modules on the forge.  More modules means more chances to find something that serves my needs, or something I can learn from.  The concern is presumably about how easy or hard it is to find 'the' right module for me, but I don't think the solution is to try to ensure that all modules published there meet specific, high API and functionality criteria.

Rather, the solution is the same as ever in an open-source community: documentation, reputation (both of the code and of the author), popularity (mostly of the code), and support and recommendations from the community, especially (but not exclusively) from thought leaders.  I daresay, for example, that many people choose your modules in large part just because they are yours.  You and your modules have strong reputations, which are reinforced when users find that your modules work well for them (or undermined in the unlikely event that they find otherwise).

Perhaps the Forge platform could be enhanced to make those tools easier to apply in its context, but at least some of them are already reasonably accessible.

 

It's not a matter of preventing or discouraging users from publishing their work, but to have a good overall quality of public modules and to encourage good practices. I'd rather work on trying to convey and promote basic reusability patterns that can be used and embraced by everybody.

I understand that it's not your intention to discourage authors, and as I said, I think it's great that you are thinking about these things and promoting the practices you find good.  But if the point of publishing a module on the forge is for others to be able to (re)use it, then how we characterize whether a module is "reusable" has deep influence on participation and the community dynamic.  Reusability is not a boolean condition, it is a qualitative, analog measure.

 

Also, I don't think that a reusable module is necessarily a good one or viceversa, actually in most the cases reusability features add complexity and overheads.

I agree.  But if you say that my module is not "reusable" then why should I not take that as a rebuff -- an assertion that my module is not useful to the community and therefore should not be shared?  If someone else were to use it in any way, then would that not be "reuse"?  If there's a reasonable chance of that happening, then does that not on its face prove that the module is indeed reusable?

John

[jcbollinger]

On Monday, June 23, 2014 10:52:48 AM UTC-5, Alessandro Franceschi wrote:

On Friday, June 20, 2014 8:01:49 PM UTC+2, Trevor Vaughan wrote:

+1 to everything that John has said.

While many Forge modules are not useful to me out of the box, I have had good luck modifying them to meet my needs at any given time.

Well, in my very personal opinion, if someone takes a module of mine and has to modify it in order to make it meet his needs, I've lost.

We have a deep difference of opinion there.  The way I see it, if someone uses a module of mine to serve his needs, with or without modification, then he and I have both won.  And if he gives me his modifications (on suitable terms) then I have won twice.  In a sense, I can be a bigger winner if users do modify my module than if they don't.

I think this may be near the core of our more general difference of opinion about what it means for a module to be 'reusable'.


John

[Felix Frank]

Hi guys,

On 06/24/2014 03:48 PM, jcbollinger wrote:
>
> I understand the point, but at the same time I find it harmful also
> to have hundreds of modules, that are published on the forge but
> work only for a specific case and are practically useless for others.
>
>
>
> I disagree. It is not at all a problem -- in itself -- to have hundreds
> of narrowly-scoped modules on the forge. More modules means more
> chances to find something that serves my needs, or something I can learn
> from. The concern is presumably about how easy or hard it is to find
> 'the' right module for me, but I don't think the solution is to try to
> ensure that all modules published there meet specific, high API and
> functionality criteria.
>
> Rather, the solution is the same as ever in an open-source community:
> documentation, reputation (both of the code and of the author),
> popularity (mostly of the code), and support and recommendations from
> the community, especially (but not exclusively) from thought leaders. I

> daresay, for example, that many people choose /your/ modules in large

> part just because they are yours. You and your modules have strong
> reputations, which are reinforced when users find that your modules work
> well for them (or undermined in the unlikely event that they find
> otherwise).
>
> Perhaps the Forge platform could be enhanced to make those tools easier
> to apply in its context, but at least some of them are already
> reasonably accessible.

I have followed this thread with great interest, and although I didn't
take the time to read the blog posts, I have to say that I'm pretty much
100% with Alessandro on this one.

My own dives into the Forge have been short-lived so far, but the
current experience is detrimental, I feel. The multitude of solutions
and the fact that there may actually be two or more modules that each
suit *some* specific needs of mine, with no visible impetus towards
merging similar modules into a "one fits all" solution makes me despair
early. It is my fear that many potential users are repelled by the same
experience.

It would be my hope that each problem that modules solve could culminate
in one module that strikes just the right balance between being generic
enough to serve (most) everyone without being overladen with features
that almost nobody needs. (Such go-to modules would likely be taken
under the umbrella of "supported" modules.)

That is not to say that there should be no ecosystem, we just cannot
have too many dead leaves and branches scattered everywhere. Those
should be recycled into better core modules by some means.

If I read him correctly, Alessandro is currently questing for a way to
make just this unification process possible, although that's possibly
just my interpretation. (But a common set of practices *will* make the
merging of functionality much easier.) I support this.

Best,
Felix

[Pablo Fernandez]

Hi,

On 23/06/14 19:15, Alessandro Franceschi wrote:

As a side note I'm a bit perplex about the proliferation of parameters in classes or defines in order to match single application specific configuration entries.

When I look at this, for example:
https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp
my backbone freezes.
A huge amount of define parameters, that map Apache directives, and not even a parameter that allows me to provide a custom template.

I agree... it sucks! But I think it sucks more to the person that wrote the module than to those using it. When you see a declaration of a vhost, written in puppet language, it's clear and self-explanatory. But if you pass a template instead, the clarity is lost: you need to look for it elsewhere, it's much more complicated to understand a template than a puppet hash, and you need to seek each variable to see where its value is coming from.

The specific problem with vhosts (and other of the kind) is a problem with the language itself: Hiera does not search parameters of defined types, and you need to find your way around with a class that does a simple create_resources to create those resources of that defined type (and probably set defaults with parameters). It could be worse, though :)

Also, if you want a pure data driven setup, where you place on hiera hashes of data with all the stuff you need, you can still have this using a template and a single parameters ( such as the options_hash )

I have mixed feelings about this: if it's not a parameter, you don't have automatic hiera lookups, and additionally you have to properly document the structure of the hash. Parameters are almost self-explanatory... but I agree that there are some modules where everything is a parameter, and it's difficult to handle. But I think I would rather have that than doing a template do all the work and have to parameters (template and options_hash)

Hope my clarification exposes better the rationale behind the second part.

Yes, I understand your approach... but I don't think it's going very far (given the current trend): Puppet has chosen the "multi-parameter" route because they want to be able to plug in different data feeds. Neither a config_hash nor a template are really feed-able.

As a side note, a problem with "forking" the model: when people do fancy stuff (like the truth_enforcer - I loved the idea BTW - or using special libraries as requirements) regular users often can't profit from their modules. Too much effort lost from both sides!! I think PuppetLabs (they have the steering wheel) is more-or-less clear with how they envision the "ideal" module. As long as we stick to that, our modules will live longer and be used more often. This should not prevent us from thinking of alternatives, obviously.

BR/Pablo

[Alessandro Franceschi]

On Tuesday, June 24, 2014 3:48:17 PM UTC+2, jcbollinger wrote:

On Monday, June 23, 2014 10:34:00 AM UTC-5, Alessandro Franceschi wrote:

First of all, let me thank you, John, for replying to this post and giving a sense to the discussion.

At times I wonder if PuppetLabs' guys read the forum and, if they do, why they don't express an opinion on these matters.
I might be obsessive about modules, but I think that it's here where we play the most important game on user code reusability. 

On Friday, June 20, 2014 5:04:28 PM UTC+2, jcbollinger wrote:

I have never accepted the premise that "reusable" as applied to Puppet modules has to mean "usable in any context, to achieve any end related to the module's area of application, via any conceivable style," which is pretty much how I read your reusability criteria for component modules.  Certainly a module that satisfies those criteria can serve pretty much any purpose within its domain, but what matters to me at any given time is whether a module can serve my purpose, whatever that happens to be.

Reusability is definitively a relative concept, and I agree it can have different nuances, still, IMHO, there are a few reusability patterns that are easy to introduce and add a great value (for example the possibility, for the module's user to provide custom templates for configuration files instead of the ones proposed by default by the module. This is easy to introduce and should definitively be common practice).

I think it's great that you are thinking about such questions, discussing them, and promoting your conclusions.  And I agree that there are reusability patterns that can be applied fairly easily and add substantial value.  But I suspect that if we compared lists, mine would differ from yours.
 

Maybe. But I can change idea in front of valid arguments :-)

 

 

If some third-party modules are (re)usable for me but not for (say) Felix, and vise versa, then that doesn't change the fact that both our jobs are made easier by not having to write everything from scratch.  If that means I need to study a module a bit before I decide whether to use it, then fine -- I ought to do that anyway.

Moreover, I think it is harmful to the module ecosystem to promote a position that only Cadillac modules are "reusable," because that sets the bar very high for potential contributors.  I don't think it helps anyone to tell people that their contributions are unwelcome (which branding them "not reusable" does).  Most people who write modules do so for their own purposes first and foremost, and if the community is unwelcoming to them then they will just keep their work to themselves.

I understand the point, but at the same time I find it harmful also to have hundreds of modules, that are published on the forge but work only for a specific case and are practically useless for others.

I disagree.  It is not at all a problem -- in itself -- to have hundreds of narrowly-scoped modules on the forge.  More modules means more chances to find something that serves my needs, or something I can learn from.  The concern is presumably about how easy or hard it is to find 'the' right module for me, but I don't think the solution is to try to ensure that all modules published there meet specific, high API and functionality criteria.

Rather, the solution is the same as ever in an open-source community: documentation, reputation (both of the code and of the author), popularity (mostly of the code), and support and recommendations from the community, especially (but not exclusively) from thought leaders.  I daresay, for example, that many people choose your modules in large part just because they are yours.  You and your modules have strong reputations, which are reinforced when users find that your modules work well for them (or undermined in the unlikely event that they find otherwise).

Perhaps the Forge platform could be enhanced to make those tools easier to apply in its context, but at least some of them are already reasonably accessible.

 

It's not a matter of preventing or discouraging users from publishing their work, but to have a good overall quality of public modules and to encourage good practices. I'd rather work on trying to convey and promote basic reusability patterns that can be used and embraced by everybody.

I understand that it's not your intention to discourage authors, and as I said, I think it's great that you are thinking about these things and promoting the practices you find good.  But if the point of publishing a module on the forge is for others to be able to (re)use it, then how we characterize whether a module is "reusable" has deep influence on participation and the community dynamic.  Reusability is not a boolean condition, it is a qualitative, analog measure.

Absolutely agree. Reusability has many nuances and degrees.

 

 

Also, I don't think that a reusable module is necessarily a good one or viceversa, actually in most the cases reusability features add complexity and overheads.

I agree.  But if you say that my module is not "reusable" then why should I not take that as a rebuff -- an assertion that my module is not useful to the community and therefore should not be shared?  If someone else were to use it in any way, then would that not be "reuse"?  If there's a reasonable chance of that happening, then does that not on its face prove that the module is indeed reusable?

John

Before giving my 2c I quote from another answer from you:

> We have a deep difference of opinion there.  The way I see it, if someone uses a module of mine to serve his needs, with or without modification, then he and I have both won.  And if he gives me his modifications (on suitable terms) then I have won twice.  In a sense, I can be a bigger winner if users do modify my module than if they don't.

> I think this may be near the core of our more general difference of opinion about what it means for a module to be 'reusable'. 

Well, IMHO the primary condition of reusability is that you can use a module without changing anything inside it. 
If such a change can be of general use, may be applied to other cases and it's committed back to the upstream source, that's perfect. Exactly the win2win condition you mention. I suppose nobody has the ambition, for sure not me, to make a perfectly reusable module out of the box, also given the nuances of reusability we were mentioning.

I personally would discourage, and consider it not good for the overall sanity of a Puppet setup, to get an existing module, fork it locally and work on it to accomodate custom needs.
That's a quite common approach, but it really generates a lot of extra , reinventing the wheel, work.

Let's compare this to a language like PHP, how many people do take php modules or components from, say, Pear, and MODIFY them to accomodate local needs?
I don't think that's a common habit.
That's why I think that component modules should be considered as libraries, which we can be used, patched, improved, but are still considered as public and reusable piece of code. 

Incidentally, about modules reusability you can read more extensive coverage (from my own perspective) in this pdf: https://bitly.com/1qbxWZy
Yes, that's also a subtle promotional move, but I'm really interested in your lucid feedback on such topics.

Al

[Alessandro Franceschi]

Cool. Any contribution to https://github.com/example42/puppet-tp is welcomed.
I'm currently figuring out what's the best way to organise and structure the module data and have to fix the tp_lookup function that is the core of the hiera-like retrieval mechanism. 
Once these basic parts are fixed I will start to populate aggressively the data directory (with data for different applications) and then take care of the puppet tp face.

Anyway , just to make it clearer, the intended usage of tp is to ease and standardise the management of software that can be installed with normal native packages (eventually providing a custom repo via the dependency_class parameter).
It's not and will never be a complete replacement of other modules, rather I think of it as a complementary tool, to be used instead of full modules for some cases (where you deal with just packages , services and files resources) or together with an existing module (which might provide some application specific resources (such as mysql::grant) that a generic tp module will never have) in other cases.
It's intended to be used in higher abstraction modules where can stay the logic of how different resources are grouped ad configured.
Think of it as the tool that can let us define the shape of the single lego pieces, but these pieces have to be assembled together somewhere else.

al

[Alessandro Franceschi]

On Tuesday, June 24, 2014 4:27:41 PM UTC+2, pablo.f...@cscs.ch wrote:

Hi,

On 23/06/14 19:15, Alessandro Franceschi wrote:

As a side note I'm a bit perplex about the proliferation of parameters in classes or defines in order to match single application specific configuration entries.

When I look at this, for example:
https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost.pp
my backbone freezes.
A huge amount of define parameters, that map Apache directives, and not even a parameter that allows me to provide a custom template.

I agree... it sucks! But I think it sucks more to the person that wrote the module than to those using it. When you see a declaration of a vhost, written in puppet language, it's clear and self-explanatory. But if you pass a template instead, the clarity is lost: you need to look for it elsewhere, it's much more complicated to understand a template than a puppet hash, and you need to seek each variable to see where its value is coming from.

The specific problem with vhosts (and other of the kind) is a problem with the language itself: Hiera does not search parameters of defined types, and you need to find your way around with a class that does a simple create_resources to create those resources of that defined type (and probably set defaults with parameters). It could be worse, though :) 

Also, if you want a pure data driven setup, where you place on hiera hashes of data with all the stuff you need, you can still have this using a template and a single parameters ( such as the options_hash )

I have mixed feelings about this: if it's not a parameter, you don't have automatic hiera lookups, and additionally you have to properly document the structure of the hash. Parameters are almost self-explanatory... but I agree that there are some modules where everything is a parameter, and it's difficult to handle. But I think I would rather have that than doing a template do all the work and have to parameters (template and options_hash)

Sometimes I wonder if a totally data driven approach is something that is always the right choice.

Given that it's mostly a matter of personal preference, infrastructure design, skills and so on, I feel like that sometimes modules' author forget that Puppet users are most of the times system administrators that know exactly how to configure their files and need Puppet to replicate the configurations they have. So having to study a module in order to achieve the intended output on the generated configurations might be frustrating.

Anyway if you want all your data in hiera, also the application specific settings, nodody prevents you from placing in hiera a key with this data and placing it in a variable that you use to feed the options_hash parameter.

Really, I have seen and tested different approaches on how to organise things with Puppet: all the data on Hiera, no Hieraat all; class groupings in an ENC, Hiera, a site module; nodes classification using nodes' inheritance, a nodeless setup, an ENC; file managed or settings managed configurations.. I don't have any real reason to say that one approach is better than another, and actually my intention is to find ways (modules) that can adapt to different usage patterns.

Hope my clarification exposes better the rationale behind the second part.

Yes, I understand your approach... but I don't think it's going very far (given the current trend): Puppet has chosen the "multi-parameter" route because they want to be able to plug in different data feeds. Neither a config_hash nor a template are really feed-able.

Well, if this is the path I think it has to be walked with care, at least for scalability issues:
- If you use augeas, file_line or similar to manage per-setting configurations you risk to multiply the number of resources for each node. 

- If you add many parameters to your classes, besides the extra time needed for the data binding lookups, you have higher module's maintenance costs

I'm not saying this should not be done at all, but at the same time that should not be the unique way.

Choice, IMHO, reusability is all about freedom of choice.

Users' choice, not authors.

Al