Directory Environments

[Joshua Partlow]

Hi folks,

(TL;DR: in 3.5.0+ environments will change to be named directories with a specific structure and configurable defaults, all to be found in a configured 'environmentpath')

Andy, Henrik and I have been working on the new environment system intended to eventually replace the current options for defining static and dynamic environments with a single enumerable alternative.  I'm going to try and summarize where that work stands for 3.5.0 and what will be left for 3.6.0.

Prior to the new work, environments could be specified statically, dynamically or by default in the following three ways:

Static environments could be explicitly set as sections in puppet.conf, where the section name was the environment name and certain settings in that section would take priority over main, run mode or application default settings if puppet's main environment setting had been set to match, yet in turn be over ridden by command line settings.  The settings which have an effect within a static environment section are 'modulepath', 'manifest', 'templatedir', and 'config_version'.

Dynamic environments could be set by including $environment for interpolation in these key settings, for instance in the [main] 'modulepath' and 'manifest' settings.

And the default case is simply the default 'environment' (production) and the default 'modulepath' and 'manifest' settings.

Starting in Puppet 3.5.0 we are adding a new way of specifying environments which allows both for explicit enumeration of existing environments and dynamic addition of environments without adjusting the master configuration.  In this system each environment will simply be a directory with a specific structure located in a given path.  Hence we are calling them directory environments for now to distinguish them from their legacy kin.

The new system relies on a new setting 'environmentpath' in puppet.conf.  The 'environmentpath' may be set to one or more directories that will be searched for environment directories.  Its default is "$confdir/environments".

Each directory in the path may contain a number of subdirectories, one for each environment, where the name of the subdirectory is the name of the environment (valid environment names match the regular expression /\w+/).  Inside each environment's directory there are two more directories: `modules` for environment specific modules and `manifests` for the environment's manifests (this is where `site.pp` would be placed).

  environments

   |- my_env

   |   |- modules

   |   |   \- stdlib

   |   \- manifests

   |       \- site.pp

   ...

If your 'environmentpath' setting contains multiple directories, then an environment will be matched from the first path directory it can be located in.  So environments found earlier in the path will shadow environments that are located later in the path.

In addition, there is a new setting 'basemodulepath' which is intended to provide a path to global modules available across all environments.  It has the same defaults as the current [main] 'modulepath' setting "$confdir/modules:/usr/share/puppet/modules", and 'modulepath' now defaults to "$basemodulepath".

This setting is appended to each directory environment's modulepath.  So for the above 'my_env' example, its modulepath would be "$confdir/environments/my_env/modules:$basemodulepath".

In 3.5.0, the 'basemodulepath' setting can be interpolated into other legacy environments as well.

The new directory environments will be functional as described above, but not feature complete in 3.5.0.  The plan is to finish them in 3.6.0, and deprecate static, dynamic and the default environment settings, which will then be removed in 4.0.

The main directory environment feature scheduled to be added in 3.6.0 is to allow for a per environment configuration file that can customize modulepath and manifest directory settings within each directory environment.  This would be in the form of an environment.conf file, following the same format as puppet.conf.  In addition to setting 'modulepath' and 'manifest', 'config_version' could be set here as well.  This should give parity to the current static environment sections in puppet.conf. (https://jira.puppetlabs.com/browse/PUP-1596)

In 3.6.0, [main] 'modulepath', 'manifest', 'config_version' would be deprecated.  The default manifest would be found in "$confdir/environments/production/manifests/site.pp", and packaging would be modified to construct this base default environment.  Use of static environment sections and interpolating $environment within puppet.conf would be deprecated as well.

You can also enumerate the new directory environments via the v2.0 API endpoint (/v2.0/environments) that Andy and Dan worked on. (see https://github.com/puppetlabs/puppet/blob/master/api/docs/http_environments.md) Command line enumeration of environments is targeted for 4.0.

The epic for the environment work is: https://jira.puppetlabs.com/browse/PUP-536

Please let me know if I made any mistakes about existing functionality, or you have any questions or suggestions for us with regards to the directory environments.

thanks,

Josh P.

--

Josh Partlow

jpar...@puppetlabs.com

Developer, Puppet Labs

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14

Register now and save $350! 

[Jason Antman]

My heart stopped when I read this, though after reading the tickets, I think I'm a little more clear on it, and less concerned.

Just to confirm, the existing functionality will continue to work until the new directory-based environments are feature complete, correct?

My specific concerns are around how I currently have environments configured; the salient parts are:
   modulepath = /etc/puppet/environments/$environment/modules:/etc/puppet/environments/$environment/modules-community
i.e. I have 2 modulepaths within each environment
   config_version = /etc/puppet/environments/$environment/config_version.sh $environment
we have a Jenkins workflow wrapped around testing branches and promoting them through dev, test and prod. This is heavily dependent on all reports including config_version, which includes the current HEAD commit hash of the environment.

Thanks,
Jason

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CADxAQ5pzqpo20nuLY5LxFzrEf53V_m-8wYHNq9%2BCvo1T1b-Arw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Daniele Sluijters]

Hey,

As far as I can see I see no reason why your setup wouldn't work, at least not once it's feature complete in 3.6.0.

All in all, this sounds great to me!

-- 

Daniele Sluijters

[Spencer Krum]

I want to echo Jason that I don't see how multiple module directories will work in this scheme. My modules go into two buckets 'internal' and 'public'.

Can you describe that?

Thanks,
Spencer

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/7d0ae273-1ceb-44f3-9797-ed4289026db3%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Spencer Krum
(619)-980-7820

[Andy Parker]

On Fri, Feb 21, 2014 at 5:13 AM, Jason Antman <ja...@jasonantman.com> wrote:

My heart stopped when I read this, though after reading the tickets, I think I'm a little more clear on it, and less concerned.

Yes, all existing environments will continue to work in 3.5.0. The new environment system is still somewhat experimental in 3.5 and should be feature complete in 3.6. At that point the current environments will still work, but puppet will start issuing deprecations warnings.

 

Just to confirm, the existing functionality will continue to work until the new directory-based environments are feature complete, correct?

My specific concerns are around how I currently have environments configured; the salient parts are:
   modulepath = /etc/puppet/environments/$environment/modules:/etc/puppet/environments/$environment/modules-community
i.e. I have 2 modulepaths within each environment

The modulepath will be possible once we do PUP-1596. 

 

   config_version = /etc/puppet/environments/$environment/config_version.sh $environment
we have a Jenkins workflow wrapped around testing branches and promoting them through dev, test and prod. This is heavily dependent on all reports including config_version, which includes the current HEAD commit hash of the environment.

Ok, I hadn't thought about that use of config_version (passing the environment as a parameter to the script). I was hoping to remove the $environment interpolation (PUP-1114) as a way to clearly move away from the current dynamic environment pattern, but maybe that won't be possible.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/53075117.3060708%40jasonantman.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

[Andy Parker]

On Fri, Feb 21, 2014 at 8:15 AM, Spencer Krum <krum.s...@gmail.com> wrote:

I want to echo Jason that I don't see how multiple module directories will work in this scheme. My modules go into two buckets 'internal' and 'public'.

Can you describe that?

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CADt6FWNm1cS_GXrsieVLtMfMZvKS7ngU14SvFC-4oud_YBrmfw%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

[James Sweeny]

--

You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CADxAQ5pzqpo20nuLY5LxFzrEf53V_m-8wYHNq9%2BCvo1T1b-Arw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Josh/Andy/Henrik,

Could you please provide some background on why this change is being made, and what the benefits to the user will be? If this is just to make it easier to support future APIs, I'm curious to know why we need to change the UI.

Thanks,

- James

--

James Sweeny
Professional Services

http://puppetlabs.com/

[Jason Antman]

On 02/21/2014 11:53 AM, Andy Parker wrote:

On Fri, Feb 21, 2014 at 5:13 AM, Jason Antman <ja...@jasonantman.com> wrote:

My heart stopped when I read this, though after reading the tickets, I think I'm a little more clear on it, and less concerned.

Yes, all existing environments will continue to work in 3.5.0. The new environment system is still somewhat experimental in 3.5 and should be feature complete in 3.6. At that point the current environments will still work, but puppet will start issuing deprecations warnings.

 

Just to confirm, the existing functionality will continue to work until the new directory-based environments are feature complete, correct?

My specific concerns are around how I currently have environments configured; the salient parts are:
   modulepath = /etc/puppet/environments/$environment/modules:/etc/puppet/environments/$environment/modules-community
i.e. I have 2 modulepaths within each environment

The modulepath will be possible once we do PUP-1596.

http://reviews.ddtc.cmgdigital.com/r/7402/

Ok. Fair 'nuff.

 

   config_version = /etc/puppet/environments/$environment/config_version.sh $environment
we have a Jenkins workflow wrapped around testing branches and promoting them through dev, test and prod. This is heavily dependent on all reports including config_version, which includes the current HEAD commit hash of the environment.

Ok, I hadn't thought about that use of config_version (passing the environment as a parameter to the script). I was hoping to remove the $environment interpolation (PUP-1114) as a way to clearly move away from the current dynamic environment pattern, but maybe that won't be possible.

The output of our config_version script is "<environment name>/<full git hash>". Passing $environment to config_version was, IIRC, a way to make a portable config_version script that didn't have to "guess" the environment from its current path. I suppose that if environments were required to be in a given path, and config_version lived in that path, it would be a non-issue.

Thanks for replying and clearing that up.

-Jason

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANhgQXu0p3hHYrEeJy6i6gyePOQa5uPOhi3rOGvht6WyKwa9OA%40mail.gmail.com.

[Andy Parker]

On Fri, Feb 21, 2014 at 8:55 AM, James Sweeny <james....@puppetlabs.com> wrote:

On Fri, Feb 21, 2014 at 2:41 AM, Joshua Partlow <joshua....@puppetlabs.com> wrote:

Hi folks,

(TL;DR: in 3.5.0+ environments will change to be named directories with a specific structure and configurable defaults, all to be found in a configured 'environmentpath')

snip

 

You can also enumerate the new directory environments via the v2.0 API endpoint (/v2.0/environments) that Andy and Dan worked on. (see https://github.com/puppetlabs/puppet/blob/master/api/docs/http_environments.md) Command line enumeration of environments is targeted for 4.0.

The epic for the environment work is: https://jira.puppetlabs.com/browse/PUP-536

Please let me know if I made any mistakes about existing functionality, or you have any questions or suggestions for us with regards to the directory environments.

snip some more

 

Hi Josh/Andy/Henrik,

Could you please provide some background on why this change is being made, and what the benefits to the user will be? If this is just to make it easier to support future APIs, I'm curious to know why we need to change the UI.

I'm not sure which parts of the description you are referring to, but I'll take a stab at giving some background on parts of it. The original request came from the Puppet Labs PE team, where they want to be able to make the PE UIs aware of the environments. In order to do that there needs to be a list of environments for them to present to the user. The current way that puppet handles environments make this very difficult, or impossible, since there is no single list of environments that it can just pull up. There has been talk in the past about separating environments out into something separate from the puppet.conf file. Those ideas about separating out environments lead to a system where we can enumerate all of the environments that are available and so provide the functionality that was requested.

Now you might wonder why not just do something like doing some querying of the filesystem for module directories and manifests with a $environment set to *? The reason is that even that wouldn't get us a correct list of environments. There are too many variations and corner cases. Also there is another problem in which *any* name, no matter if there is something that shows up on the filesystem, can end up being a valid environment. So that would have just led down a path of corner cases, unclear definitions of what constitutes an environment, and misery.

The reason for the layout of directories is because it closely matches what was already a common practice for dynamic environments. So really what we are doing is more closely modeling what was already done.

The reason for the /v2.0/environments REST endpoint is because we've wanted, for a long time, to separate the internals of how puppet is implemented, from the external facing API. The current endpoints that feed into the indirector all have to follow a specific pattern, don't provide any layers between the network and the internals, and have caused us a lot of pain (security issues, usability issues, code maintenance issues). Since this work was to provide a way of querying the master for all of the environments, it seemed like a natural time to get a new system in place. I felt it was a little absurd that in order to query for the list of environments from the current API, you'd have to construct a path that includes an environment and a key that isn't needed (since all indirector paths are of the form environment/indirection/key). So the new API is a way of breaking free and letting us follow well trodden API design principles. 

Note: v2.0, as opposed to v2,  was decided upon because it isn't a valid environment name, which means we can unambiguously distinguish it from an environment and therefore a request for the old system.

As for benefits to users. I think there will be many, but I'll just stick to how the new environments are laid out on disk. By laying out environments like this it creates nicely self-contained segments of puppet code. Each environment can be understood on its own by simply looking at what is in that directory. It puts environments on a similar level of modules. It adheres to a common structure, is found and loaded in a common way, and contains information to describe itself. I foresee a future in which we can even manipulate environments (or rather the code and configuration that installed into environments) as artifacts that are versioned, promoted, etc. just like we can with modules.

I hope that gave some background to where these changes came from. If not, let me know :)

 

Thanks,

- James

--

James Sweeny
Professional Services

http://puppetlabs.com/

Join us at PuppetConf 2014, September 23-24 in San Francisco - http://bit.ly/pupconf14

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAKDACKtqykjQnJEJzFMO%2B8du%3D-AOwRiU9V-jKJrFceb0m1Hb%3DA%40mail.gmail.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

[Jeremy T. Bouse]

My take on this is that I don't see it will affect my dynamic environment usage for my personal puppet environment as I'm using r10k fairly easily.

My deployment is available at https://github.com/UGNS/standard-modules and contains the modules deployed from a Puppetfile into the modules/ sub-directory but also contains a scripts/ and manifiests/ sub-directories so the only exception appears to be the scripts/ directory which for me contains my config version and autosigner scripts. Along with Jason my config version script takes the environment as an argument but given that the environment name is part of the path I could simply modify it to use the path rather than requiring an argument so long as we can still specify a config version script with path taken relative to the environment directory itself then it.

[Eric Sorenson]

James Sweeny wrote:
> Could you please provide some background on why this change is being
> made, and what the benefits to the user will be? If this is just to
> make it easier to support future APIs, I'm curious to know why we need
> to change the UI.

Good question James. The ultimate goal is to support the awesome
workflows that involve environments, a'la r10k, with a well-designed
backend implementation. Right now, dynamic environments work kind of by
accident, not because Puppet was designed to support them, and they fall
apart pretty easily. So the benefit to the user is that the best
practises and tooling the community's developed around code promotion,
testing, versioning get codified and baked in.

--
Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[Dominic Cleal]

On 21/02/14 08:41, Joshua Partlow wrote:
> Hi folks,
>
> (TL;DR: in 3.5.0+ environments will change to be named directories with
> a specific structure and configurable defaults, all to be found in a
> configured 'environmentpath')

Thanks for the detailed explanation Josh, this was really useful. It
sounds like a good direction. My only concern is the complete removal
of static and dynamic environments in Puppet 4.0, which could be
problematic for users who have built workflows around the ability to
define complex environment modulepaths.

I've been doing a little regression testing and came across the
following bug with "puppet apply --modulepath":
https://tickets.puppetlabs.com/browse/PUP-1765

Do you have any thoughts on the correct way to resolve this? The
current implementation appears unintuitive.

My inclination is that the combined loader should treat settings that
have been overridden on the command line with higher priority than the
environmentpath loader, but environments in puppet.conf at a lower
priority. (I suspect in practice, this could be tricky to implement.)

--
Dominic Cleal
Red Hat Engineering

[Dominic Cleal]

On 25/02/14 15:59, Dominic Cleal wrote:
> On 21/02/14 08:41, Joshua Partlow wrote:
>> Hi folks,
>>
>> (TL;DR: in 3.5.0+ environments will change to be named directories with
>> a specific structure and configurable defaults, all to be found in a
>> configured 'environmentpath')
>
> Thanks for the detailed explanation Josh, this was really useful. It
> sounds like a good direction. My only concern is the complete removal
> of static and dynamic environments in Puppet 4.0, which could be
> problematic for users who have built workflows around the ability to
> define complex environment modulepaths.

Ignore that comment please, I missed the references to PUP-1596.