Modulepath and manifest in environment.conf should interpolate $environment too

[Bostjan Skufca]

Hi all,

this change I have not yet implemented, but I believe it might be nice to have.

Currently in environment.conf we can specify modulepath with relative or absolute paths to module directories, like this:

modulepath = modules/:/some/other/location/env_name/

manifest = /path/to/env_name/manifest2/

It would be splendid if we could define modulepath paths with $environment as variable part of path, like this:

modulepath = /path/to/$environment/modules

manifest = /path/to/$environment/manifest2/

Would there be any interest for this feature?

If this is implemented for modulepath setting, maybe it should be appropriate to implement it for manifest setting too?

b.

PS: Explanation why I find this useful, or from which use case I derived it would be useful:

I have two puppets running on each system. Primary is for managing whole system + secondary puppet, and secondary puppet is just for managing primary puppet. This is useful for puppet upgrades - if something goes wrong, secondary puppet still works and fixes primary puppet if required, and vice versa.

For these two I use the same module repository (each puppet has own module though) and I find it convenient to not have to clone module repository for the second time, for each environment.

[Bostjan Skufca]

To elaborate a bit further:

I've delved into puppet code and figured out that to enable such behaviour a single line needs to be changed, like this:

------------------------------------------------------------

--- puppet/settings.rb.DIST 2015-03-08 23:26:56.792628995 +0000

+++ puppet/settings.rb 2015-03-08 23:36:50.313658881 +0000

@@ -1263,7 +1263,7 @@

   # @api public

   class ChainedValues

     ENVIRONMENT_SETTING = "environment".freeze

-    ENVIRONMENT_INTERPOLATION_ALLOWED = ['config_version'].freeze

+    ENVIRONMENT_INTERPOLATION_ALLOWED = ['config_version', 'manifest', 'modulepath'].freeze

 

     # @see Puppet::Settings.values

     # @api private

------------------------------------------------------------

I would like to address this question to more seasoned puppet developers:

Is there a reason interpolation is not enabled for these settings, some design-based reason?

Tnx,

b.

[Henrik Lindberg]

On 2015-08-03 20:27, Bostjan Skufca wrote:
> Hi all,
>
> this change I have not yet implemented, but I believe it might be nice
> to have.
>
> Currently in environment.conf we can specify modulepath with relative or
> absolute paths to module directories, like this:
> modulepath = modules/:/some/other/location/env_name/
> manifest = /path/to/env_name/manifest2/
>
> It would be splendid if we could define modulepath paths with
> $environment as variable part of path, like this:
> modulepath = /path/to/$environment/modules
> manifest = /path/to/$environment/manifest2/
>
> Would there be any interest for this feature?
> If this is implemented for modulepath setting, maybe it should be
> appropriate to implement it for manifest setting too?
>

This scares me a bit. It looks like it has potential to open the can of
worms known as dynamic environments we managed to put the lid on with
directory environments. Will have to discuss if this can used to do
harmful things. If allowed it could only be allowed inside an absolute path.

Will see what Joshua Partlow has to say as well.

- henrik

> b.
>
>
> PS: Explanation why I find this useful, or from which use case I derived
> it would be useful:
> I have two puppets running on each system. Primary is for managing whole
> system + secondary puppet, and secondary puppet is just for managing
> primary puppet. This is useful for puppet upgrades - if something goes
> wrong, secondary puppet still works and fixes primary puppet if
> required, and vice versa.
> For these two I use the same module repository (each puppet has own
> module though) and I find it convenient to not have to clone module
> repository for the second time, for each environment.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-dev+...@googlegroups.com
> <mailto:puppet-dev+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-dev/d32b2c66-4dd2-4bf7-b0d7-05194b65dde4%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-dev/d32b2c66-4dd2-4bf7-b0d7-05194b65dde4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[Felix Frank]

On 03/09/2015 02:16 PM, Henrik Lindberg wrote:
>> It would be splendid if we could define modulepath paths with
>> $environment as variable part of path, like this:
>> modulepath = /path/to/$environment/modules
>> manifest = /path/to/$environment/manifest2/
>>
>> Would there be any interest for this feature?
>> If this is implemented for modulepath setting, maybe it should be
>> appropriate to implement it for manifest setting too?
>>
>
> This scares me a bit. It looks like it has potential to open the can of
> worms known as dynamic environments we managed to put the lid on with
> directory environments. Will have to discuss if this can used to do
> harmful things. If allowed it could only be allowed inside an absolute
> path.

Agreed, there might be security implications.

I also fail to see the value in that. Do you mean to allow an
environment to extend itself to a whole different file system tree?
Wouldn't that just be horrible for organizing things?

Felix

[Henrik Lindberg]

The use case was to use one location for two master instances without
having to have two copies, and without having to have a modulepath that
includes the name of the environment (which is possible now).

Not sure that use case is worth the potential problems.

- henrik

[Joshua Partlow]

I'm not really following the use case either, Bostjan.  I'm not understanding what benefit having the $environment interpolate is giving you here, since $environment is already selected to get to environment.conf, the path is already pre-determined.

It was PUP-3162 where we ultimately decided to whitelist for $environment interpolation, and the only setting that made the cut was config_version.  I don't think we want to change that because it makes the pathing harder to think about and may have other unforeseen consequences in the settings and environment  handling.

-Josh

- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/mdkgbj%242b4%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.

--

Josh Partlow

jpar...@puppetlabs.com

Developer, Puppet Labs

[Bostjan Skufca]

On Monday, 9 March 2015 14:54:14 UTC+1, Felix Frank wrote:

On 03/09/2015 02:16 PM, Henrik Lindberg wrote:
>> It would be splendid if we could define modulepath paths with
>> $environment as variable part of path, like this:
>> modulepath = /path/to/$environment/modules
>> manifest = /path/to/$environment/manifest2/
>>
>> Would there be any interest for this feature?
>> If this is implemented for modulepath setting, maybe it should be
>> appropriate to implement it for manifest setting too?
>>
>
> This scares me a bit. It looks like it has potential to open the can of
> worms known as dynamic environments we managed to put the lid on with
> directory environments. Will have to discuss if this can used to do
> harmful things. If allowed it could only be allowed inside an absolute
> path.

Agreed, there might be security implications.

Probably, I did not look into that. But since node-2-environment mapping is mostly done by masters now (is this recommended setup now?), I do not think it would be that severe. 

 

I also fail to see the value in that. Do you mean to allow an
environment to extend itself to a whole different file system tree?

Yes and no, see next message for additional explanation of my setup.

 

Wouldn't that just be horrible for organizing things?

It can. But so can PHP be abused for creating insecure aplications. And "rm -rf" wipes your system without warning too :)

But, as I percieved the happening in the recent past, PuppetLabs had hard time steering users towards sane module / node definition organization, and can see a value of not destroying what has been achieved.

b.

[Bostjan Skufca]

On Monday, 9 March 2015 17:05:22 UTC+1, henrik lindberg wrote:

On 2015-09-03 14:54, Felix Frank wrote:
> I also fail to see the value in that. Do you mean to allow an
> environment to extend itself to a whole different file system tree?
> Wouldn't that just be horrible for organizing things?
>

The use case was to use one location for two master instances without
having to have two copies, and without having to have a modulepath that
includes the name of the environment (which is possible now).

Spot on.

+ extend "modulepath" to "modulepath and manifest config setting".

(only after starting to pursue modulepath interpolation I realised having interpolated "manifest" setting would be splendid too.

b.

[Bostjan Skufca]

On Monday, 9 March 2015 17:33:27 UTC+1, Joshua Partlow wrote:

On Mon, Mar 9, 2015 at 9:05 AM, Henrik Lindberg <henrik....@cloudsmith.com> wrote:

On 2015-09-03 14:54, Felix Frank wrote:

On 03/09/2015 02:16 PM, Henrik Lindberg wrote:

It would be splendid if we could define modulepath paths with
$environment as variable part of path, like this:
modulepath = /path/to/$environment/modules
manifest = /path/to/$environment/manifest2/

Would there be any interest for this feature?
If this is implemented for modulepath setting, maybe it should be
appropriate to implement it for manifest setting too?

This scares me a bit. It looks like it has potential to open the can of
worms known as dynamic environments we managed to put the lid on with
directory environments. Will have to discuss if this can used to do
harmful things. If allowed it could only be allowed inside an absolute
path.

Agreed, there might be security implications.

I also fail to see the value in that. Do you mean to allow an
environment to extend itself to a whole different file system tree?
Wouldn't that just be horrible for organizing things?

The use case was to use one location for two master instances without having to have two copies, and without having to have a modulepath that includes the name of the environment (which is possible now).

Not sure that use case is worth the potential problems.

I'm not really following the use case either, Bostjan.  I'm not understanding what benefit having the $environment interpolate is giving you here, since $environment is already selected to get to environment.conf, the path is already pre-determined.

The main motivation for my setup is:

- if something happens to puppet upgrade and puppet fails to work, I do not want to ssh into nodes to manually fix the problem.

- Even "mcollective" is not proper solution here (usable for quickfix, yes), as some nodes might be down and will have to catch up eventually

To achieve this, I have two puppets (installed in dedicated separate dirs under /usr/local/puppet-1 and /usr/local/puppet-2). If, for the sake of argument, we forget that puppet-1 manages the whole system except itself, you can see the redundancy here (puppet-2 is served a special config catalog that only includes puppet-1 installation/upgrades). Each puppet definition is in own puppet module, so nothing is shared there.

Up to this point it is all nice, and works, if you presume you use separate manifests for each puppet, for the same node. The annoying thing becomes when I have to manage manifests in separate git repos, at least partially.

If all above seems overcomplicated and overengineered, there might be something about me being more lazy than others and thus putting in more work upfront so I do not have to do it later (fixing broken puppet installations manually). If so, please tell me:)

It was PUP-3162 where we ultimately decided to whitelist for $environment interpolation, and the only setting that made the cut was config_version.  I don't think we want to change that because it makes the pathing harder to think about and may have other unforeseen consequences in the settings and environment  handling.

I can understand that.

I believe we are talking about two contradicting concepts here:

a) directory environments assume all environment-related content is under main environment directory, except for common modules (basemodulepath)

b) power users know what they are doing and would like to interpolate variables to optimize their workflow

I already work around that when installing puppet, as it gets patched with mentioned changes automagically. And these changes are only needed on master, though.

ALTERNATE SOLUTION

Since I posted original message, I have come up with a better solution actually. Forget the interpolation patch. I've introduced new puppet configuration variable, called "environment_conf_filename", which enables customization of "environment.conf" filename.

One puppet master now uses environment.conf-1, the other environment.conf-2, from the same directory (and same git repo).

Content of these files now looks like this:

### environment.conf-1

manifest = manifests-1   <-- full node definitions, including module for puppet-2

modulepath = modules/

### environment.conf-2

manifest = manifests-2     <- separate manifest, that only includes definition of puppet-1 installation for all nodes

modulepath = modules/

Now all content can reside in the same git repository, and puppet masters still do not interfere with each other, which is splendid.

Since solution is already done (works for me), I am moving this thread to "comment/criticise my setup", ok? If you have opinion about this, please speak up.

And thanks for responses,

b.

[Adrien Thebo]

The change you've proposed seems extremely specific to your use case; do you see this feature being usable for anyone outside of your environment?

 

And thanks for responses,

b.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/5ee4b679-af17-438d-9a5f-81c01241949b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Adrien Thebo | Puppet Labs

[Bostjan Skufca]

I do not want to assume anything about how others use tools that are given to them. What I do know is that I like tools that provide flexibility and can be adapted to my needs. This particular case here is something that feels quite natural, especially if you consider how much code needs to be changed to achieve greater flexibility. I think we are not talking major changes here, merely desinhibition.

Come to think about it again, it would actually make more sense to create something which is similar to what ENC does for nodes. Therefore, I propose:

EEL - External Environment Locator

Proposed new master setting:
environment_terminus = /path/to/external/environment/locator.sh

It takes one argument:
- requested environment name (once node and ENC agree on node environment)

It returns one path to directory or file:
- if what is returned is a directory, that is used as environment path
- if what is returned is a file, then that is treated as path to environment.conf

After that, all the normal rules apply.

Maybe I should spin this into a separate thread;)

B.

[Bostjan Skufca]

On Wednesday, 11 March 2015 03:41:45 UTC+1, Bostjan Skufca wrote:

EEL - External Environment Locator

Proposed new master setting:
environment_terminus = /path/to/external/environment/locator.sh

I was too fast for my own good.

This is what I intended to actually write as an example:

[master]

env_terminus = exec

external_env = /path/to/external/environment/locator.sh 

b.

PS: this is a sample, adjusted to be the most similar to node_terminus, which's behaviour I am trying to mimic. If someday this actually gets implemented, I have no preference about actual configuration keys used (external_env should be called env_terminus_exec_path if you ask me).