Thoughts on Module Namespace Conflicts

[Trevor Vaughan]

I was talking with a few folks today about potential resolutions to module namespace issues.

== Fundamental Issue ==

puppetlabs_apache -- Installs To --> apache

example42_apache -- Installs To --> apache

theforeman_apache -- Installs To --> apache

You get my point...

== Current Solutions ==

Right now, there are two ways to solve this problem if you want some of your nodes to use puppetlabs_apache and others to use theforeman_apache.

1) Modify the module and force the namespace:

  - puppetlabs_apache -- Installs To --> puppetlabs_apache

  - theforeman_apache -- Installs To --> theforeman_apache

  * Isssue: You can't just use outside code at will. You have to modify *everything* that uses the above code.

2) You have to create a separate environment for each host group (potentially each host) that you want to have use the differing code.

   * Issue: This is a LOT of overhead for a couple of modules and may have other ramifications in terms of performance as you get up in number.

So, would it be possible to allow multiple versions of a module to exist in the same namespace but only use one during a given run?

== The First Proposal ==

Inspired by RVM Gemsets, how about allowing modules to declare which version of a given module they will use?

/etc/puppet/modules/apache/puppetlabs/{files,modules,manifests}

/etc/puppet/modules/apache/theforeman/{files,modules,manifests}
/etc/puppet/modules/apache/<author>/{files,modules,manifests}

Then, use could be dictated by something like: include apache@puppetlabs.

Unfortunately, this really comes down to the parser being able to understand the following:

include apache => See if there are any @'s and use that one

include apache@puppetlabs => include the Puppetlabs apache

include apache@puppetlabs && include apache@theforeman => Fail, conflict

== Alternative ==

One possible alternative is to just use the metadata.json file to dictate which module will be used when loading other modules.

Again, if there is a conflict, that is a failure but *only* if the code attempts to use both at the same time.

The benefit here is that it should make things very unambiguous while the drawback (if it really is) is that you *must* put a metadata.json file in every module that you create.

This is just a set of thoughts that I hope get us moving in a direction where this type of thing is possible and I look forward to hearing what people think (good, bad, or ugly)!

Thanks,

Trevor

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[Henrik Lindberg]

On 2015-10-02 1:23, Trevor Vaughan wrote:
> I was talking with a few folks today about potential resolutions to
> module namespace issues.
>
> == Fundamental Issue ==
>
> puppetlabs_apache -- Installs To --> apache
> example42_apache -- Installs To --> apache
> theforeman_apache -- Installs To --> apache
>
> You get my point...
>
> == Current Solutions ==
>
> Right now, there are two ways to solve this problem if you want some of
> your nodes to use puppetlabs_apache and others to use theforeman_apache.
>
> 1) Modify the module and force the namespace:
> - puppetlabs_apache -- Installs To --> puppetlabs_apache
> - theforeman_apache -- Installs To --> theforeman_apache
>
> * Isssue: You can't just use outside code at will. You have to modify
> *everything* that uses the above code.
>
> 2) You have to create a separate environment for each host group
> (potentially each host) that you want to have use the differing code.
>
> * Issue: This is a LOT of overhead for a couple of modules and may
> have other ramifications in terms of performance as you get up in number.
>
> So, would it be possible to allow multiple versions of a module to exist
> in the same namespace but only use one during a given run?
>

Basically no, this is not possible without adding some kind of mechanism
to filter out modules on the modulepath. This will need to be done per
environment anyway. Remember that modules can contribute all sorts of
extensions to puppet (faces, indirections, facts, functions, types, and
with future parser also bindings). For performance reasons (and
bootstrapping, etc.) these are scanned when the environment loads (some
scans are lazy, but may take place before manifests really starts to be
evaluated).

- henrik

> tvau...@onyxpoint.com <mailto:tvau...@onyxpoint.com>

>
> -- This account not approved for unencrypted proprietary information --
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com
> <mailto:puppet-dev+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Walter Heck]

I'm personally more of a fan of what some programming languages call interfaces; a sort of contract if you will that a module implements. 

So for instance olindata::galera could look for any class that implements the IMysql interface. That way, I don't really need to worry if the end-user is using puppetlabs::mysql or example42::mysql, as long as they implement the IMysql interface I can guarantee it does what I need. 

This would have pretty heavy impact on how you write code right now, but it would be a rather elegant solution imho.

interface IMysql::server identified by '9ff3d80b-b02d-4994-b4da-e1ff205304ea' {

  Service['mysql']

  Package['mysql-server']

  File['apache2.conf']

}

class puppetlabs::mysql implements IMysql['9ff3d80b-b02d-4994-b4da-e1ff205304ea'] {

  [.. SNIP..]

  # file, service, package here like normal

}

class example42::mysql implements IMysql['9ff3d80b-b02d-4994-b4da-e1ff205304ea'] {

  [.. SNIP..]

  # file, service, package here like normal

}

class olindata::galera {

  package { 'galera':

    require => Package['IMysql::server::mysql-server']

  }

}

Or something along those lines. Question remains where the IMysql::server interface then needs to be declared, and how we would manage the GUID registration of interfaces.

Comments, thoughts?

Walter

> <mailto:puppet-dev+unsub...@googlegroups.com>.

[Trevor Vaughan]

To Henrik: Yes, this make sense, it would be nice to have some easy way to keep the purely Puppet DSL portions separate though.

Walter: This would be nice, but variables are the...uh...variable here.

Many people only expose the variables that they need while others expose every variable they can find. It would be an interesting experiment to see if it would work though.

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/d171f0ce-4d7b-41fb-8722-98e0c8244e92%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

[Heck, Walter]

Hi Trevor,

On Tue, Feb 10, 2015 at 9:17 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Walter: This would be nice, but variables are the...uh...variable here.

Many people only expose the variables that they need while others expose every variable they can find. It would be an interesting experiment to see if it would work though.

Well, in for instance Delphi interfaces could also have properties, so I don't think that needs to be a problem. The interface wouldn't determine the value of the property though, just that it should exist in a class implementing that interface.

See for some Delphi examples here: http://www.delphibasics.co.uk/Article.asp?Name=Interface 

--

Best regards, 

Walter Heck 

CEO / Founder OlinData - Open Source Training & Consulting

Check out our upcoming trainings

[Henrik Lindberg]

On 2015-10-02 19:52, Walter Heck wrote:
> I'm personally more of a fan of what some programming languages call
> interfaces; a sort of contract if you will that a module implements.
>

Yes that is good - does not solve the issue of modules having the same
name though - either the module name must change, or modules must be
installed with both author and name and then referenced the same way.

- henrik

> > tvau...@onyxpoint.com <javascript:> <mailto:tvau...@onyxpoint.com
> <javascript:>>

> >
> > -- This account not approved for unencrypted proprietary
> information --
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Puppet Developers" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send

> > an email to puppet-dev+...@googlegroups.com <javascript:>
> > <mailto:puppet-dev+...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com>
>
> >

> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/

> <http://puppet-on-the-edge.blogspot.se/>

>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/d171f0ce-4d7b-41fb-8722-98e0c8244e92%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-dev/d171f0ce-4d7b-41fb-8722-98e0c8244e92%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Henrik Lindberg]

On 2015-10-02 21:17, Trevor Vaughan wrote:
> To Henrik: Yes, this make sense, it would be nice to have some easy way
> to keep the purely Puppet DSL portions separate though.
>

Yeah, but that is the way it works - so way to much to change to make
that happen.

I guess it would be possible to have a special implementation of a
module that is a proxy for a real selected module, and that the real
module is not on the real module path - only the proxy. Still quite
complex to achieve. Since the concept of a module is not that well
encapsulated.

Another approach is to differentiate between available modules (more
like a repo) and modules being used in an environment - having a
distinct step in between that configures the modulepath for the runtime.
Still the same issue with where the description of which modules to use
is kept - probably ends up with the same amount of configuration as when
just using directory environments.

You could write something yourself that does that for you - i.e.
something that creates the environments you need/want.

- henrik

> > * Isssue: You can't just use outside code at will.. You

> /etc/puppet/modules/apache/__puppetlabs/{files,modules,__manifests}
> >
> /etc/puppet/modules/apache/__theforeman/{files,modules,__manifests}
> >
> /etc/puppet/modules/apache/<__author>/{files,modules,__manifests}

> > (410) 541-6699 <tel:%28410%29%20541-6699>
> > tvau...@onyxpoint.com <mailto:tvau...@onyxpoint.com__>

> >
> > -- This account not approved for unencrypted proprietary information --
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Puppet Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send

> > an email to puppet-dev+...@__googlegroups.com
> > <mailto:puppet-dev+_...@googlegroups.com>.

> > To view this discussion on the web visit

> >https://groups.google.com/d/__msgid/puppet-dev/CANs%__2BFoU2unExPve9ig%__3DinW9obDpwii7gBi6W4RkHycViibJ__p-g%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com>
>
> > <https://groups.google.com/d/__msgid/puppet-dev/CANs%__2BFoU2unExPve9ig%__3DinW9obDpwii7gBi6W4RkHycViibJ__p-g%40mail.gmail.com?utm___medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com?utm_medium=email&utm_source=footer>__>.
>
> > For more options, visithttps://groups.google.com/d/__optout
> <https://groups.google.com/d/optout>.

>
>
> --
>
> Visit my Blog "Puppet on the Edge"

> http://puppet-on-the-edge.__blogspot.se/

> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/d171f0ce-4d7b-41fb-8722-98e0c8244e92%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-dev/d171f0ce-4d7b-41fb-8722-98e0c8244e92%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --

> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699
> tvau...@onyxpoint.com <mailto:tvau...@onyxpoint.com>
>
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com

> <mailto:puppet-dev+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXFyivsBttxKa23YFEe6jLMANYKbnAn5tjJany%2BbDq3hQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXFyivsBttxKa23YFEe6jLMANYKbnAn5tjJany%2BbDq3hQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Heck, Walter]

Hi Henrik,

On Tue, Feb 10, 2015 at 9:30 PM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

On 2015-10-02 19:52, Walter Heck wrote:

I'm personally more of a fan of what some programming languages call
interfaces; a sort of contract if you will that a module implements.

Yes that is good - does not solve the issue of modules having the same name though - either the module name must change, or modules must be installed with both author and name and then referenced the same way.

But with this you wouldn't need to have more then one module with the same name? All you'd need is a single module that satisfies all the interfaces the rest of your code asks for. I would think that if this was implemented, the community would gravitate towards a single IMysql interface which can then be implemented by different mysql modules in different ways.

cheers,

[Hunter Haugen]

This reminds me of ARM-17 that I tried to model after Haskell/Python/Clojure https://github.com/puppetlabs/armatures/pull/64 . It stalled out due to lack of time, though it may have some relevant content.

-Hunter

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAOfYMj4uk2OUtQ5tbG6waX0K4FQRfUST8Q2hf-sfp5Pz%3DsADJw%40mail.gmail.com.

[Dominic Cleal]

On 10/02/15 00:23, Trevor Vaughan wrote:
> I was talking with a few folks today about potential resolutions to
> module namespace issues.
>
> == Fundamental Issue ==
>
> puppetlabs_apache -- Installs To --> apache
> example42_apache -- Installs To --> apache
> theforeman_apache -- Installs To --> apache
>
> You get my point...

I do, but just for the avoidance of doubt, and because I still get asked
about it regularly, theforeman/apache is dead and has been replaced by
puppetlabs/apache for a couple of releases now. Long live pl-apache :-)

--
Dominic Cleal
Red Hat Engineering

[Trevor Vaughan]

Heh, that's probably because it's the third hit in the list when you search for Apache on the Forge.

Is there a way for people to mark Forge modules as unmaintained?

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/54DB2DE9.20001%40redhat.com.

For more options, visit https://groups.google.com/d/optout.

[Felix Frank]

On 02/10/2015 01:23 AM, Trevor Vaughan wrote:
> 2) You have to create a separate environment for each host group
> (potentially each host) that you want to have use the differing code.
>
> * Issue: This is a LOT of overhead for a couple of modules and may
> have other ramifications in terms of performance as you get up in number.

Hi,

call me slow, but is this not the exact problem that users of r10k do
*not* face?

Yes, I've never tried it, for shame.

Cheers,
Felix

[Trevor Vaughan]

r10k definitely helps with this but you still have to maintain the configurations for each of your environments.

So, 50 hosts with one different/conflicting module each == 50 branches with one line different in a file. (yeah, straw man). Also, you still can't mix and match your conflicting namespace modules. This means if you *really* want to use two different 'apache' modules, you have to rename one of them and all referencing calls.

It *is* certainly the way to go right now though.

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/54DBE144.8030606%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.

[Dominic Cleal]

On 11/02/15 18:13, Trevor Vaughan wrote:
> Heh, that's probably because it's the third hit in the list when you
> search for Apache on the Forge.
>
> Is there a way for people to mark Forge modules as unmaintained?

Not as far as I know. I ought to upload a new release with a big
warning in the README. All I've done for the meantime is add a
"deprecated" tag, which isn't a big help.

There's a similar issue I have with augeasproviders. Again I've yet to
update the README in domcleal/augeasproviders to indicate that
herculesteam/augeasproviders is the maintained version, but it would be
great to be able to mark this as such in the Forge, or even redirect to
another module.

[R.I.Pienaar]

redirects would be nice, I have the same problem with ripienaar/concat and
puppetlabs/concat

[Erik Dalén]

A related ticket to the interface and redirect functionality is https://tickets.puppetlabs.com/browse/FORGE-111

There I just suggest the feature to be able to say that module B provides the interface of module A version X. So either module can satisfy a dependency on module A version X.

Then if you want to have some sort of generic interface that other modules can implement, you could have a sort of virtual module with he version being the interface version. And then other modules provide or depend on that virtual module. The meaning of virtual module here is the same as for virtual packages in package managers, like "ruby-interpreter" or "smtp-server" in Debian, and the implementation would also be pretty similar.

Regarding having multiple variants or versions of a module being installed at the same time I think we should change the environment loader API to provide a list of modules instead of a list of module paths. Then plugins could implement any sort of behaviour for this without jumping through hoops.

Basically I think that would mean deprecating this: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/node/environment.rb#L144-L150

And changing this method to return list of paths to individual modules instead of names of modules: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/node/environment.rb#L233-L241

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/100231099.15755.1423733538852.JavaMail.zimbra%40devco.net.

[Trevor Vaughan]

And this is why he gets awards...

+1

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAAAzDLe5OC2Y44_P2ovuCFnZ2iw3uZEN-Sgic_pjcamYyNVHrQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Henrik Lindberg]

On 2015-12-02 15:12, Erik Dalén wrote:
> A related ticket to the interface and redirect functionality is
> https://tickets.puppetlabs.com/browse/FORGE-111
>
> There I just suggest the feature to be able to say that module B
> provides the interface of module A version X. So either module can
> satisfy a dependency on module A version X.
> Then if you want to have some sort of generic interface that other
> modules can implement, you could have a sort of virtual module with he
> version being the interface version. And then other modules provide or
> depend on that virtual module. The meaning of virtual module here is the
> same as for virtual packages in package managers, like
> "ruby-interpreter" or "smtp-server" in Debian, and the implementation
> would also be pretty similar.
>
>
> Regarding having multiple variants or versions of a module being
> installed at the same time I think we should change the environment
> loader API to provide a list of modules instead of a list of module
> paths. Then plugins could implement any sort of behaviour for this
> without jumping through hoops.
>

I have claimed on many occasions that an environment is just like a
module (with higher privileges / precedence). The set of available
modules should be more of a repository kind of thing that the loader
resolves modules from.

Essentially this is how the new loaders work (used only for 4x functions
so far).

- henrik

[Wil Cooley]

On Mon, Feb 9, 2015 at 4:23 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

I was talking with a few folks today about potential resolutions to module namespace issues.

== Fundamental Issue ==

puppetlabs_apache -- Installs To --> apache

example42_apache -- Installs To --> apache

theforeman_apache -- Installs To --> apache

You get my point...

... 

== The First Proposal ==

Inspired by RVM Gemsets, how about allowing modules to declare which version of a given module they will use?

/etc/puppet/modules/apache/puppetlabs/{files,modules,manifests}

/etc/puppet/modules/apache/theforeman/{files,modules,manifests}
/etc/puppet/modules/apache/<author>/{files,modules,manifests}

Then, use could be dictated by something like: include apache@puppetlabs.

Unfortunately, this really comes down to the parser being able to understand the following:

include apache => See if there are any @'s and use that one

include apache@puppetlabs => include the Puppetlabs apache

include apache@puppetlabs && include apache@theforeman => Fail, conflict

== Alternative ==

One possible alternative is to just use the metadata.json file to dictate which module will be used when loading other modules.

Again, if there is a conflict, that is a failure but *only* if the code attempts to use both at the same time.

The benefit here is that it should make things very unambiguous while the drawback (if it really is) is that you *must* put a metadata.json file in every module that you create.

This is just a set of thoughts that I hope get us moving in a direction where this type of thing is possible and I look forward to hearing what people think (good, bad, or ugly)!

== Alternative #2: Namespaced Module-Sets ==

Let's call it "Namespaced Module-Sets"; it's similar to the first proposal but kinda turned on its end. The concept of the module path is extended with optional names for any element in the module path. These names can then be used as a namespace prefix to address classes within a particular module-set. From within the module-set, references to classes do not need to be qualified (or perhaps, within a named set must not be) and these references are local-first, meaning that the set is searched before the module path is searched.

With this scheme, 3rd party module authors do not need to change the dependencies of their modules nor the names by which they refer to these dependencies. Local devadmins have the ability to use multiple modules that present internally the same top-level namespace and to mix in modules that are dependent on specific versions of those.

Example: I'll use YAML to represent the namespaced module path, but I'm not necessarily presenting this as the way it would actually be configured.

---

modulepath:

  # Our local modules, without an attached name

  - path: /etc/puppet/modules/local

  # PuppetLabs modules and modules that depend on them

  - path: /etc/puppet/modules/puppetlabs

    name: puppetlabs

One could have a node defined to use our local "apache" module and another node defined to use "puppetlabs::apache", which is installed in /etc/puppet/modules/puppetlabs. Now one could also use puppetlabs/passenger by installing it to /etc/puppet/modules/puppetlabs. Because scope references are local-first, passenger/manifests/init.pp's "include '::apache'" refers to "::puppetlabs::apache".

The biggest problem I see with this scheme is with Ruby components that are not Puppet-namespaced. Within a module-set there would be no problem, as the local-first would still apply, but if, e.g., functions with the same name were in different module-sets, then the language would need to be extended to support namespaces in these.

And there are a number of other details that would need to be worked out, including but not limited to:

  * Hiera lookups; I have a feeling that there is some complexity here that is escaping me

  * Would it fall back to looking in named module-sets for unqualified names? If there were an unnamed module-set after a named module-set, which would be searched first?

  * My feeling is that the use of named module-sets and the names chosen for them should be a local-only decision. That is, distributed modules should never assume anything about the existence of module-set names, either by convention or technical restriction. (So, for example, one would only expect to use them within a role or profile or node definition.) There are probably gray areas with this that I am not seeing.

  * A variation would be to not allow unnamed module-sets at the local level, but this would be a breaking change, whereas the above should be backwards compatible.

  * Additional complexity for r10k/pmt/etc. probably?

In some ways, this is a lot like environments, but unlike environments, which are unique at the node-level, these allow the use of multiple module-sets and are scoping for entities at the manifest-level. As I understand environments, they can (and probably usually should) be relatively isolated from each other; the purpose of namespaced module-sets is to allow addressing without isolation. Maybe this is what Henrik's last message in this thread is getting at when he says:

I have claimed on many occasions that an environment is just like a module (with higher privileges / precedence). The set of available modules should be more of a repository kind of thing that the loader resolves modules from.

Wil

[Henrik Lindberg]

On 2015-20-02 20:51, Wil Cooley wrote:
> In some ways, this is a lot like environments, but unlike environments,
> which are unique at the node-level, these allow the use of multiple
> module-sets and are scoping for entities at the manifest-level. As I
> understand environments, they can (and probably usually should) be
> relatively isolated from each other; the purpose of namespaced
> module-sets is to allow addressing without isolation. Maybe this is what
> Henrik's last message in this thread is getting at when he says:
>
> I have claimed on many occasions that an environment is just like a
> module (with higher privileges / precedence). The set of available
> modules should be more of a repository kind of thing that the loader
> resolves modules from.

I meant that an environment can be seen as a module. Instead of having a
modulepath, it should list all the modules that it requires (those
modules in turn can require other modules). When a module is needed it
should be resolved against a repository of modules. Such a repository
could be implemented many ways; one being that it is just the modules
now on the modulepath.

The main difference is that the resolution from name to actual module is
now free from implementation concerns.

Currently the dependencies that are expressed are between the containers
of logic (the module depends on another specific module with a specific
implementations e.g. puppetlabs/stdlib). This is simple but not very
flexible. It would be far better if the requirements were established
based on what is actually required from that module - say that it needs
a function 'foo'. Then that could be declared and any module that
happens to make that function available is a potential replacement. The
same is applicable to other elements of the puppet language, classes,
defines, types, etc.

Such a scheme requires something to select the modules that are actually
wanted out of all potential modules. This could be the role of the
environment module - it would have dependencies on the modules that
should be used. It is not limited to that however since any module can
act as a configuration (by having explicit dependencies on other
modules) - such modules does not have to contain anything but the
dependencies. They become reusable parts of configuration thus making it
easier to set up many different environments.

While this increases flexibility it also adds complexity. It is also
not the complete solution to all the things discussed in this thread,
but I believe it could be the foundation for most of them.

[Spencer Krum]

The use case I think for having multiple modules of the same name
available is quite limited. I don't think anyone would claim best
practice if they had two apache modules in their modulepath.

I also think there is another component which is versioning. Right now
there is no system for a profile module to know what version of e.g.
rabbitmq is installed. This means it just assumes one and fails if the
rabbitmq class takes different parameters now. One terrible workaround
for this is for classes to accept a 'rabbitmq_module_version' parameter
and then switch on it inside the class. This workaround is actually
used.

So I think as long as we're discussing how to make module owner
detectable, requireable, whatever, that we should involve versioning as
well, because bumping minor or major versions of utility modules like
postgres, mysql, rabbitmq happens a lot.

One case that actually happens where there is a want for two different
modules by the same name, is when an organization is migrating. An org
that is moving from example42-apache to puppetlabs-apache isn't gonna
want to have a flag day where everything is broken. They don't really
want to play move hosts one by one out of an environment into a testing
environment then into a 'done' environment. This is where being able to
have two modules of the same name could be really useful.

Erik proposed allowing modules to specify an interface, then other
modules could implement that interface. I don't think that is a likely
outcome. I think when we have two modules that both manage 'apache' the
only reason one hasn't totally taken over the other is because they have
different underlying ideas about how to manage apache. I.e. what the
inputs are and what the scope of the outputs are.

We on the openstack infrastructure team have even gone so far as to fork
an old version of the apache module to openstackinfra-httpd. This is
0.0.4 of the pl-apache module and takes wildly different inputs. There
are three reasons for the fork. One, this module takes a template, fills
it out, and dumps it in a vhost. Clean and simple. The way we think it
should be. Two, now that we have forked we can update the codebase,
clean it up, and fix bugs. Three, with the module now living as 'httpd'
in our module path, we, or others consuming our infra, can use a modern
version of the apache module for whatever they need.


As a final thought. One way to deal with this problem is to socialize
the requirement of a new metadata file (yay) called something like
manifests/properties.pp. This file would have all the data in
metadata.json. But since it is readable by puppet we could end up with
code that looks like this in our profiles:

class profile () {

case $::apache::properties::author {
'puppetlabs': {
// do some apache stuff the puppetlabs way
}
'example42': {
// do some apache stuff the example42 way
}
'default': { fail("please stop writing your own apache module") }
}

}

And the technique for handling multiple versions of the same module is
basically exactly the same. The cool part of this is that it requires no
code changes to puppet core. The bad part is we'd have to socialize it
and thats really hard.


--
Spencer Krum
ni...@spencerkrum.com

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-dev+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/puppet-dev/mc8jqc%24hus%241%40ger.gmane.org.

[Spencer Krum]

I'll resurrect this old thread to give an update. We landed a function
in stdlib to do some of what I discussed in my last post, by inspecting
module metadata directly.

The function:
https://github.com/puppetlabs/puppetlabs-stdlib#load_module_metadata
A blog post I wrote about some possible uses:
http://scienceofficersblog.blogspot.com/2015/08/inspecting-puppet-module-metadata.html

--
Spencer Krum
ni...@spencerkrum.com

> https://groups.google.com/d/msgid/puppet-dev/1424730766.2492445.231441093.2BCFFDD7%40webmail.messagingengine.com.