Moving non-core types and providers out of puppet

[Josh Cooper]

We've been talking about moving non-core types and providers out of the puppet repo for many years[1], and we are going to make every effort to make that happen for Puppet 6, finally! I've written up a proposal in the form of a PR to the puppet-specifications repo[2]. Please take a look and comment on the PR.

Thanks!

Josh

[1] https://groups.google.com/d/msg/puppet-dev/4y9ZvbbkKBs/z1KdGGQyGWkJ

[2] https://github.com/puppetlabs/puppet-specifications/pull/106

[Josh Cooper]

Some people raised concerns about vendoring modules in the puppet-agent package[1]. I wanted to provide some more context about the "why", and some refinements to the "how".

First, new users should be able to download the puppet-agent package and manage basic resources for the platform they're running on. We don't want users to download an engine and be forced to assemble a car. Commonly used modules like stdlib, archive, sudo, inifile, powershell, etc should be part of the initial experience.

On the other hand, advanced puppet users don't want vendored modules conflicting with modules they're already managing using r10k, librarian-puppet, etc. Another way to think about it is, if you're going through the trouble of specifying your dependencies in a Puppetfile, then you want the same results no matter which puppet-agent version you're using. If you start to rely on vendored module magic to satisfy dependencies, then failures will happen as the puppet-agent package is updated over time.

It's also important to consider serverless vs server-based deployments. When running serverless puppet (apply, resource, describe, etc), modules only need to be installed on the local system. However, for server-based puppet, modules need to be installed on the server and downloaded to the agent. In this scenario, an agent's vendored modules are not useful.

We also can't assume that vendored modules on the server are applicable to the agents you're trying to manage. For example, you may have a RHEL server, but Windows, OSX, etc agents.

So we have different constraints for new users vs advanced users, and serverless vs server-based deployments. Is there a solution that makes Puppet more accessible for new users while not getting in the way for advanced users?

I'm proposing we add the vendored modules directory to the basemodulepath with the least precedence. So on *nix, basemodulepath would default to: 

    /etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules:/opt/puppetlabs/puppet/vendor_modules

On Windows, we'd do something similar by appending $installdir\Puppet Labs\Puppet\puppet\vendor_modules to the basemodulepath. Note the exact location depends on the installation directory, typically C:\Program Files.

For serverless puppet, e.g. puppet apply, we would only load modules from the vendored directory as a last resort. This way if you `puppet module install puppetlabs-stdlib` you'd always use that version no matter what version is vendored in the puppet-agent package.

With server-based puppet, the server would also only load from the vendored directory as a last resort. That way if you installed the module following the roles and profiles pattern, and set your per-environment:

    modulepath=site:modules:$basemodulepath

Then the server would always prefer the per-environment module over the vendored one.

The server's compiler and fileserver would resolve modules the same way, since they both call Puppet::Node::Environment#modules. This is important so that agents always pluginsync the same version of the type (and its provider) as was used during compilation. And the agent would need to always prefer its pluginsync libdir over any modules that may be locally installed (vendored or otherwise).

Finally, if you really don't want to use vendored modules, then you can override basemodulepath, omitting the "vendored_modules" directory, and everything will work as it did before.

Josh

[1] https://github.com/puppetlabs/puppet-specifications/pull/106

--

Josh Cooper | Software Engineer

jo...@puppet.com | @coopjn

[Trevor Vaughan]

I would ask that you only vendor items that are super generic and that you namespace them such that they won't interfere with people already written codebases.

The following seem relatively safe since they've been in use for quite some time:

• stdlib
• archive (maybe)
• inifile
• powershell (maybe)

However, there are a LOT of sudo, etc... implementations which mean that you're going to pretty much break everyone's environments out of the box and cause a great number of headaches.

Furthermore, what happens when I want the next version of 'stdlib' with the new shiny but you've baked in a version that now makes me work around everything?

If you want a 'new user pack', make it just that, an extra package that I can install that will drop a bunch of vendored modules somewhere and provide me with instructions on setting it up.

I do not feel that putting this in 'puppet-agent' itself is a good idea *at all*, particularly considering that we still can't build puppet-agent from source to override any shenanigans that may arise from these good intentions.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CA%2Bu97unFNeUF_5oxumMFTER5hkgR_45FSPcKuE6ogo-9KCjnLw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Trevor Vaughan]

Oh, and you're possibly setting yourself up for SERVER-94 hell if you're not EXTREMELY careful.

[Josh Cooper]

On Thu, Apr 26, 2018 at 7:42 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

I would ask that you only vendor items that are super generic and that you namespace them such that they won't interfere with people already written codebases.

For modules that were previously in core puppet, we are plan on appending "_core" to the module name to avoid conflicts with already written modules of the same name. For example, this might happen if you have a "manifest" module with the same name as a "T&P" module. But I understand your concern with things like sudo, which have never been vendored before.

 

The following seem relatively safe since they've been in use for quite some time:

• stdlib
• archive (maybe)
• inifile
• powershell (maybe)

However, there are a LOT of sudo, etc... implementations which mean that you're going to pretty much break everyone's environments out of the box and cause a great number of headaches.

Can you be specific about how this would break everyone? Is it that you can't install a module with the same name as a module that is already installed? Or are you concerned the vendored module will be visible in the search path and conflict with another version?

Furthermore, what happens when I want the next version of 'stdlib' with the new shiny but you've baked in a version that now makes me work around everything?

One of the primary goals is if you express an explicit dependency on stdlib, e.g. in your Puppetfile

    mod 'puppetlabs-stdlib', '4.25.1'

then that version should always be used no matter what version might be vendored. This is true for both puppet apply and puppet server/agent scenarios.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoX49%3D4b04dEpZD4rud%2BUHL5unQDUmz%3DFVE83LGavtfmwg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Josh Cooper]

On Thu, Apr 26, 2018 at 7:42 PM, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

Oh, and you're possibly setting yourself up for SERVER-94 hell if you're not EXTREMELY careful.

Yep, I'm aware of the issues with environment isolation, as I filed the initial redmine ticket more than 5 years ago describing the problems with sharing a libdir between master and agent, and how code can leak across environments[1]. The proposal doesn't break environment isolation provided you're explicitly managing your module dependencies and running `puppet generate types`, which you need to do anyways to load different versions of types in different environments.

Josh

[1] https://projects.puppetlabs.com/issues/18154

[Trevor Vaughan]

Interesting. Maybe I missed this detail.

So, if I had 5 different 'vendored' locations across my filesystem, and they were all in my environment configuration, the Puppetfile would dictate which items from these locations would actually be used?

I think I can get behind that.

My concerns were both of what you pointed out. First, naming conflicts and second version conflicts from the search path.

If there are multiple search paths, would it just figure out which to work with in terms of version matching?

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CA%2Bu97u%3D%3Dy3E%3DKAaBZ4HK1G_hUVonnTcXcTDyFRH3PvHxHp_fOw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.