Feature proposal: external environment locator (EEL)

[Bostjan Skufca]

Hi all,

I would like to propose a new feature for puppet master: External Environment Locator (EEL)

The main objective would be to make locating puppet environment locations more flexible compared to directory environments. Basic functionality would facilitate usage of external program to convert environment name to corresponding path (directory or environment configuration file).

To be able to achieve this, two new settings would need to be introduced:

1. env_terminus - possibla values: 'directory' and 'exec', 'directory' being the default one (retains current directory environments functionality)

2. external_env - path to executable, if 'exec' is used in the setting above

Sample configuration:

-----------------------------------------

[master]

env_terminus = exec

external_env = /path/to/external/environment/locator.sh 

------------------------------------------

EEL (locator.sh) requirements would be defined with the following specification:

It takes one argument: 

- requested environment name (once node and ENC agree on node environment)

It returns one path, to directory or to file: 

- if returned path is a directory, that is used as environment path, seeking environment.conf inside it, or using defaults (based on returned path) if not found;

- if returned path is a file, then that is treated as path to environment.conf. which should be opened and parsed for additional environment configuration. If file does not exist, or some settings are not set, then dirname() is used to determine path to environment and defaults are based on that value (moduledir and manifest)

After that, all the normal rules apply.

I would like to hear your opinion about this, and whether this would be useful for you.

Best regards,

b.

[Trevor Vaughan]

I don't have a problem with this as a feature but I also would not use it.

It does seem like it would be reasonably easy to implement and maintain.

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/13559aa4-2760-43dc-a619-26e27630a8b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvau...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

[John Bollinger]

On Wednesday, March 11, 2015 at 11:56:44 AM UTC-5, Bostjan Skufca wrote:

Hi all,

I would like to propose a new feature for puppet master: External Environment Locator (EEL)

The main objective would be to make locating puppet environment locations more flexible compared to directory environments. Basic functionality would facilitate usage of external program to convert environment name to corresponding path (directory or environment configuration file).

Greater flexibility is not a justification in itself.  Just because I wouldn't use such a feature doesn't mean it's not worthy, but I would like to hear a bit about how you imagine using it, and in what other situations you imagine it being useful.


John

[Felix Frank]

On 03/11/2015 05:56 PM, Bostjan Skufca wrote:
>
> I would like to hear your opinion about this, and whether this would
> be useful for you.
>

Hi,

I don't believe that such a feature would be useful to a majority of
users, myself included.

I do see two disadvantages to adding this:
1. It adds complexity, of which there is no shortage as far as
environments are concerned, at least that's my gut feeling.
2. It adds configuration options, so there is additional complexity for
the end user as well.

I'm not saying we should remove as many configuration settings and
options as possible, but we shouldn't go overboard with new ways for
users to shoot themselves in the foot.

Cheers,
Felix

[Bostjan Skufca]

John, please see here (in 9th message on this other thread I explained my use case), this is a direct link to that message:

https://groups.google.com/d/msg/puppet-dev/Akk8vZBPuRs/sB71tK5ow1UJ

b.

[Aaron Stone]

Neat idea. I would consider using this for automatic git-checkout of project branches, rather than preemptively checking out all branches by cron or git-hook.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/56c3e851-b315-48e9-a2f9-ed3e0b2e4c16%40googlegroups.com.

[Bostjan Skufca]

On Thursday, 12 March 2015 23:58:03 UTC+1, Aaron Stone wrote:

Neat idea. I would consider using this for automatic git-checkout of project branches, rather than preemptively checking out all branches by cron or git-hook.

I haven't thought about this before, but now it seems very feasible way of eliminating an redundant step(s).

BTW what did you actually have in mind:

a) creating branch in git repo and node assignment automatically clones git repo to new location? or

b) particular environment is git-pulled on every invocation?

c) both?

b.

[Bostjan Skufca]

On Thursday, 12 March 2015 23:31:58 UTC+1, Felix Frank wrote:

On 03/11/2015 05:56 PM, Bostjan Skufca wrote:
>
> I would like to hear your opinion about this, and whether this would
> be useful for you.
>

Hi,

I don't believe that such a feature would be useful to a majority of
users, myself included.

I do see two disadvantages to adding this:
1. It adds complexity, of which there is no shortage as far as
environments are concerned, at least that's my gut feeling.

I agree up to a point here, but see the rest of the post below.

2. It adds configuration options, so there is additional complexity for
the end user as well.

I'm not saying we should remove as many configuration settings and
options as possible, but we shouldn't go overboard with new ways for
users to shoot themselves in the foot.

For such an occasion I like to say that, yes, doing "rm -rf /" is bad, but as a sysadmin I still want to have this tool available. :)

Maybe puppet is becoming complex enough that Puppet Labs should think about classifying configuration settings into two categories: basic and advanced. This one (EEL) would definitely fall into advanced category, and directory environments should be advocated/documented as primary way to create/use environments.

Anyway, what I am trying to say is:

- puppet should not be holding power users back as a sacrifice for being nice to novices;

- puppet should strike a good balance between both extremes, by means of feature implementation and documentation organisation;

- achieving such a balance is not easy, I admit;

- puppet is a power tool, and irresponsible users that rush into using it, should not stand in the way of progress

b.

[Daniele Sluijters]

Hi,

It would really help if you elaborated on your use cases in this thread. I read the one message you linked but I'm still not sure about it and I don't feel like reading the whole other thread to get context. If you propose a feature part of that proposal should be a rationale and intended use cases. Preferably with a comparison of how this makes things better compared to the old setup.

On Friday, 13 March 2015 16:52:13 UTC+1, Bostjan Skufca wrote:

On Thursday, 12 March 2015 23:31:58 UTC+1, Felix Frank wrote:

On 03/11/2015 05:56 PM, Bostjan Skufca wrote:
>
> I would like to hear your opinion about this, and whether this would
> be useful for you.
>

Hi,

I don't believe that such a feature would be useful to a majority of
users, myself included.

I do see two disadvantages to adding this:
1. It adds complexity, of which there is no shortage as far as
environments are concerned, at least that's my gut feeling.

I agree up to a point here, but see the rest of the post below.

2. It adds configuration options, so there is additional complexity for
the end user as well.

I'm not saying we should remove as many configuration settings and
options as possible, but we shouldn't go overboard with new ways for
users to shoot themselves in the foot.

For such an occasion I like to say that, yes, doing "rm -rf /" is bad, but as a sysadmin I still want to have this tool available. :)

Maybe puppet is becoming complex enough that Puppet Labs should think about classifying configuration settings into two categories: basic and advanced. This one (EEL) would definitely fall into advanced category, and directory environments should be advocated/documented as primary way to create/use environments.

Anyway, what I am trying to say is:

- puppet should not be holding power users back as a sacrifice for being nice to novices;

That's a very broad argument that can be applied to the whole universe. It's not backed up by facts or any data, it's just a "thing you can say".

- puppet should strike a good balance between both extremes, by means of feature implementation and documentation organisation; 

You can't strike a balance between both extremes by having both, you need to meet in the middle. Which means maybe not all the features and maybe not the greatest documentation but enough of both so people don't shoot themselves. 

- achieving such a balance is not easy, I admit;

Balance means weighing things against each other. Having both all the features and excellent documentation and making sure that people can't shoot themselves is very very very hard. It's precisely because of this that we decide to reject features at times. 

- puppet is a power tool, and irresponsible users that rush into using it, should not stand in the way of progress

Again, that's just a thing you can say. It sounds very impressive, but what does it mean and how does it apply to reality? Irresponsible users or not, we're the ones that get flack for it, that need to support it, that have crying users and get a bad name because "Puppet nuked my system". It will never be their fault, it will always be the fault of the system. 

b.

[John Bollinger]

So I guess, then, you're talking about this:

The main motivation for my setup is:

- if something happens to puppet upgrade and puppet fails to work, I do not want to ssh into nodes to manually fix the problem.

- Even "mcollective" is not proper solution here (usable for quickfix, yes), as some nodes might be down and will have to catch up eventually

To achieve this, I have two puppets (installed in dedicated separate dirs under /usr/local/puppet-1 and /usr/local/puppet-2). If, for the sake of argument, we forget that puppet-1 manages the whole system except itself, you can see the redundancy here (puppet-2 is served a special config catalog that only includes puppet-1 installation/upgrades). Each puppet definition is in own puppet module, so nothing is shared there.

Up to this point it is all nice, and works, if you presume you use separate manifests for each puppet, for the same node. The annoying thing becomes when I have to manage manifests in separate git repos, at least partially.

I think I'm having trouble understanding this scenario, or how the proposal helps, or both, so correct me where I'm wrong:

• I guess you have separate agents on each machine and separate masters to serve them, at least currently, for I can't imagine why you would want to go this direction if you were running masterless for even one of the two Puppets you describe.
• I infer that you want to consolidate the masters but not the agents, for consolidating the agents would eliminate the resiliency benefit that inspired this approach in the first place.
• The problem you then encounter, I take it, is that you have collisions between the environment names associated with the (erstwhile) two masters.
• The solution you propose is essentially to introduce namespacing for environments, via dynamically-selected environment (root) directories.

If all that's correct, then why does the ENC ability to designate node environments not sufficiently meet your needs?  Instead of using same-named environments in different directories, you could instead use different environment names to separate the environment group for one purpose from the environment group for the other.  Example: "production" and "production-puppet".  That has the additional advantage of keeping all the external classification logic in the same place, instead of splitting it across two separate programs.


John

[Bostjan Skufca]

On Monday, 16 March 2015 14:55:23 UTC+1, John Bollinger wrote:

On Thursday, March 12, 2015 at 5:33:35 PM UTC-5, Bostjan Skufca wrote:

On Thursday, 12 March 2015 20:30:23 UTC+1, John Bollinger wrote:

On Wednesday, March 11, 2015 at 11:56:44 AM UTC-5, Bostjan Skufca wrote:

Hi all,

I would like to propose a new feature for puppet master: External Environment Locator (EEL)

The main objective would be to make locating puppet environment locations more flexible compared to directory environments. Basic functionality would facilitate usage of external program to convert environment name to corresponding path (directory or environment configuration file).

Greater flexibility is not a justification in itself.  Just because I wouldn't use such a feature doesn't mean it's not worthy, but I would like to hear a bit about how you imagine using it, and in what other situations you imagine it being useful.

John, please see here (in 9th message on this other thread I explained my use case), this is a direct link to that message:

https://groups.google.com/d/msg/puppet-dev/Akk8vZBPuRs/sB71tK5ow1UJ

So I guess, then, you're talking about this:

The main motivation for my setup is:

- if something happens to puppet upgrade and puppet fails to work, I do not want to ssh into nodes to manually fix the problem.

- Even "mcollective" is not proper solution here (usable for quickfix, yes), as some nodes might be down and will have to catch up eventually

To achieve this, I have two puppets (installed in dedicated separate dirs under /usr/local/puppet-1 and /usr/local/puppet-2). If, for the sake of argument, we forget that puppet-1 manages the whole system except itself, you can see the redundancy here (puppet-2 is served a special config catalog that only includes puppet-1 installation/upgrades). Each puppet definition is in own puppet module, so nothing is shared there.

Up to this point it is all nice, and works, if you presume you use separate manifests for each puppet, for the same node. The annoying thing becomes when I have to manage manifests in separate git repos, at least partially.

I think I'm having trouble understanding this scenario, or how the proposal helps, or both, so correct me where I'm wrong:

• I guess you have separate agents on each machine and separate masters to serve them, at least currently, for I can't imagine why you would want to go this direction if you were running masterless for even one of the two Puppets you describe.

I never said anything about masterless. You are right about two puppets and separate masters. Going masterless just shifts deployment work from one to multiple locations, though it could be done.

• I infer that you want to consolidate the masters but not the agents, for consolidating the agents would eliminate the resiliency benefit that inspired this approach in the first place.

Correct.

 

• The problem you then encounter, I take it, is that you have collisions between the environment names associated with the (erstwhile) two masters.

This was never a problem, as masters are separate processes (for resiliency reasons) and having identically named environments was never an issue.

 

• The solution you propose is essentially to introduce namespacing for environments, via dynamically-selected environment (root) directories.

I would not agree on this point, as namespacing suggests you can reuse the same name in different contexts within the same process. This is not the case here. What is proposed here is environment location discovery service - SUPPOSEDLY it should return the same path every time the same environment name is passed to it. I say supposedly because that would be the intended use case, but I imagine that this locator might return whatever user wants it to return, though currently I can not imagine a use case where one would want to serve the same environment from multiple separate directories.

What is written in the previous paragraph only applies to single puppet master process.

If there is another puppet master process on the same system that resolves environments to paths in different manner, that is no concern to the other puppet master.

If all that's correct, then why does the ENC ability to designate node environments not sufficiently meet your needs?  Instead of using same-named environments in different directories, you could instead use different environment names to separate the environment group for one purpose from the environment group for the other.  Example: "production" and "production-puppet".  That has the additional advantage of keeping all the external classification logic in the same place, instead of splitting it across two separate programs.

This might be solvable with single puppet master process and multiple environments, as you have suggested, but again this undermines redundancy (single puppet master is easy to manage manually if something happens, multiple - not so much fun anymore) and duplicates environment data (two clones of the same repos) which I am also trying to reduce.

b.

[Bostjan Skufca]

On Monday, 16 March 2015 14:01:58 UTC+1, Daniele Sluijters wrote:

Hi,

It would really help if you elaborated on your use cases in this thread. I read the one message you linked but I'm still not sure about it and I don't feel like reading the whole other thread to get context. If you propose a feature part of that proposal should be a rationale and intended use cases. Preferably with a comparison of how this makes things better compared to the old setup.

I specifically linked to a post (and not the whole thread), it should be more or less context-free. I would rather not copy-paste the whole thing here, as next message (not yours specifically) would accuse me of copy-pasting instead of linking:). Aaron Stone provided another use case in this thread, couple of posts ago.

Here is my use case summarized:

- each node has two puppet agents installed, they manage one another

- each of these agents talks to separate a master process

- both agents request the same environment: one master returns the whole system catalog, the other just bits to manage primary puppet

- on puppet masters server this means duplicate environment data/module repository clones

- this is inefficient as secondary puppet only manages first puppet and thus utilizes one separate module from whole repository

- both puppet masters use the same ENC already

With this feature, I could downsize the management to:

- single environment repository, with environment.conf for primary and environment.conf-secondary for secondary puppet master process

- (please note that I have already patched puppet master so that it supports configurable name for "environment.conf")

- both would use the same modules repository

- they would have separate manifest definitions (one defines the full system, the other is quite simple, just puppet), kept in separate directories, but managed within the same git repo

What is gained by this feature, for me?

- If node is in environment X, I can go to /etc/puppet/env/X and do a single git pull in there, and configuration is deployed

- without this, I would need to do additional step: go to /etc/puppet-SECONDARY/env/X and do another git pull

This is my use case, and this feature would be only one part of the whole equation.

Another use case would be environment location verification:

- suppose you have multiple departments using puppet environments

- environments are prefixed with dept1_, dept2_ and so on

- departments upload their environments into their dedicated directories, one per department, say /etc/puppet/env/dept1/dept1_prod would be the final path to environment, and department1 has access to this directory: /etc/puppet/env/dept1

- you have you puppet master configured with multiple directory environment paths: .../env/dept1:.../env/dept2:...

- as long as everyone is playing nice, everything works

- now imagine that dept1 accidentally creates environment dept2_prod, which should belong into dept2/... subtree

- with current implementation (dir environments) it is impossible to check if environment directory is actually the one that was initially indended

With EEL, you could easily check this, and correctly ignore .../env/dept1/dept2_prod environment, as environment prefix "dept2_" would be invalid in .../env/dept1/ directory.

 

On Friday, 13 March 2015 16:52:13 UTC+1, Bostjan Skufca wrote:

For such an occasion I like to say that, yes, doing "rm -rf /" is bad, but as a sysadmin I still want to have this tool available. :)

Maybe puppet is becoming complex enough that Puppet Labs should think about classifying configuration settings into two categories: basic and advanced. This one (EEL) would definitely fall into advanced category, and directory environments should be advocated/documented as primary way to create/use environments.

Anyway, what I am trying to say is:

- puppet should not be holding power users back as a sacrifice for being nice to novices;

That's a very broad argument that can be applied to the whole universe. It's not backed up by facts or any data, it's just a "thing you can say".

I agree this is broad. But it still applies if features are rejected (your words, see below) because users can shoot themselves in the foot.

 

- puppet should strike a good balance between both extremes, by means of feature implementation and documentation organisation; 

You can't strike a balance between both extremes by having both, you need to meet in the middle. Which means maybe not all the features and maybe not the greatest documentation but enough of both so people don't shoot themselves. 

I thoroughly disagree here. Take git, for example. It provides both, the so-called "porcelain" interface, which covers majority of general use cases, and for power users there is the "plumbing" interface, which can be used to achieve advanced integration or functionality.

However I do recognize the danger of becoming everything-but-the-kitchen-sink implementation of configuration management, but I do not see this feature as step towards that direction.

 

- achieving such a balance is not easy, I admit;

Balance means weighing things against each other. Having both all the features and excellent documentation and making sure that people can't shoot themselves is very very very hard. It's precisely because of this that we decide to reject features at times.

I understand.

 

- puppet is a power tool, and irresponsible users that rush into using it, should not stand in the way of progress

Again, that's just a thing you can say. It sounds very impressive, but what does it mean and how does it apply to reality? Irresponsible users or not, we're the ones that get flack for it, that need to support it, that have crying users and get a bad name because "Puppet nuked my system". It will never be their fault, it will always be the fault of the system.

Huh? :)

Users that have systems in production they do not understand, and then slapping puppet on top of it using modules they do not understand, bricking the service (or the system) and yelling "Puppet nuked my system" belong to the h(w)all of f(sh)ame, and not to the decision-making process about puppet features. Please do not take this personally or as offensive, as I am having a rather nice chuckle here, with many analogies springing up on "this is the tool's fault, not mine" subject :)

I would understand this accusation if manifest specified "ensure => present" but puppet would remove the file/package/whatever. But if this removal happens because of some convoluted module/profile/hiera/yaml interaction which user is too lazy/st**id to comprehend, but it resolves to final "ensure => absent", then the problem is not the tools functionality, but users incompetence.

But it is hard to call paying customers incompetent, thus I understand your clutch. However having simple features rejected will make existing users, which reach the borders of puppet's functionality, look towards other solutions, as I am sure you understand.

b.

[Felix Frank]

On 03/19/2015 01:08 AM, Bostjan Skufca wrote:

- both agents request the same environment: one master returns the whole system catalog, the other just bits to manage primary puppet

I don't quite see why the agents would have to use identical environment names. Could you perhaps also approach your problem by making the secondary agent use <environment>-meta or something in that vein?

- on puppet masters server this means duplicate environment data/module repository clones

But does the secondary agent even require a substantial amount of data and modules etc.?

- puppet is a power tool, and irresponsible users that rush into using it, should not stand in the way of progress

Again, that's just a thing you can say. It sounds very impressive, but what does it mean and how does it apply to reality? Irresponsible users or not, we're the ones that get flack for it, that need to support it, that have crying users and get a bad name because "Puppet nuked my system". It will never be their fault, it will always be the fault of the system.

Huh? :)

Users that have systems in production they do not understand, and then slapping puppet on top of it using modules they do not understand, bricking the service (or the system) and yelling "Puppet nuked my system" belong to the h(w)all of f(sh)ame, and not to the decision-making process about puppet features. Please do not take this personally or as offensive, as I am having a rather nice chuckle here, with many analogies springing up on "this is the tool's fault, not mine" subject :)

There is such a thing as UX engineering. You seem to be under the impression that it is about catering to mediocre technicians. That is just not true.

Anyhow, I think everybody has understood that you intend to add this functionality for use by the most advanced users only, but if this includes but 10% of all users, and only an insignificant percentage of those has any use for this whatsoever, then the value of maintaining it upstream appears unimpressive.

Cheers,
Felix

[Bostjan Skufca]

On Thursday, 19 March 2015 01:51:17 UTC+1, Felix Frank wrote:

On 03/19/2015 01:08 AM, Bostjan Skufca wrote:

- both agents request the same environment: one master returns the whole system catalog, the other just bits to manage primary puppet

I don't quite see why the agents would have to use identical environment names. Could you perhaps also approach your problem by making the secondary agent use <environment>-meta or something in that vein?

It could be done, but having single puppet master kind of defies the purpose. If there are multiple masters, having different environment names for same environment is redundant. But I know this is a personal opinion.

- on puppet masters server this means duplicate environment data/module repository clones

But does the secondary agent even require a substantial amount of data and modules etc.?

Not really.

 

- puppet is a power tool, and irresponsible users that rush into using it, should not stand in the way of progress

Again, that's just a thing you can say. It sounds very impressive, but what does it mean and how does it apply to reality? Irresponsible users or not, we're the ones that get flack for it, that need to support it, that have crying users and get a bad name because "Puppet nuked my system". It will never be their fault, it will always be the fault of the system.

Huh? :)

Users that have systems in production they do not understand, and then slapping puppet on top of it using modules they do not understand, bricking the service (or the system) and yelling "Puppet nuked my system" belong to the h(w)all of f(sh)ame, and not to the decision-making process about puppet features. Please do not take this personally or as offensive, as I am having a rather nice chuckle here, with many analogies springing up on "this is the tool's fault, not mine" subject :)

There is such a thing as UX engineering. You seem to be under the impression that it is about catering to mediocre technicians. That is just not true.

UX engineering I support, fully.

 

Anyhow, I think everybody has understood that you intend to add this functionality for use by the most advanced users only, but if this includes but 10% of all users, and only an insignificant percentage of those has any use for this whatsoever, then the value of maintaining it upstream appears unimpressive.

I agree somewhat, as we are not talking about breaking any legacy compatibility, instead we are talking about making things more general. Actually having directory environments as an (default) implementation of EEL would be more in line with where puppet is heading (think ENC which delegates this functionality to the user) than having it as a built-in only option.

Another argument could be made by comparing number of available configuration settings (around 244 ATM) to what I currently use in puppetmaster.conf - 10-15, depending on installation. That does not mean I consider all other config settings unnecessary, in fact it is quite the opposite. Now I know that "feature" is not the same as "configuration setting", but this argument speaks to the puppet's configurability as being a good thing.

Power users are the ones that usually push the limits of things, and if reimplementing a feature in a more general way that does not hinder performance of general customer base, while offering advanced users what they need, and at the same time making maintenance simpler (remember, this would be implemented in a more general way), then I think this is a win-win situation for everybody.

I think my note about suggesting git's way of providing "plumbing and porcelain" interfaces as good approach still stands.

Best regards,

b.

[John Bollinger]

On Wednesday, March 18, 2015 at 7:08:17 PM UTC-5, Bostjan Skufca wrote:

On Monday, 16 March 2015 14:01:58 UTC+1, Daniele Sluijters wrote:

Hi,

It would really help if you elaborated on your use cases in this thread. I read the one message you linked but I'm still not sure about it and I don't feel like reading the whole other thread to get context. If you propose a feature part of that proposal should be a rationale and intended use cases. Preferably with a comparison of how this makes things better compared to the old setup.

I specifically linked to a post (and not the whole thread), it should be more or less context-free.

You did link to a specific message.  It did not adequately describe what you wanted.  At least, I clearly didn't understand what you were after from just that post.

 

I would rather not copy-paste the whole thing here, as next message (not yours specifically) would accuse me of copy-pasting instead of linking:). Aaron Stone provided another use case in this thread, couple of posts ago.

Here is my use case summarized:

- each node has two puppet agents installed, they manage one another

- each of these agents talks to separate a master process

- both agents request the same environment: one master returns the whole system catalog, the other just bits to manage primary puppet

- on puppet masters server this means duplicate environment data/module repository clones

Why?  You have two separate masters managing disjoint aspects of the target machines, via disjoint sets of agents.  Why do they need (or even want) to share *anything*?

 

- this is inefficient as secondary puppet only manages first puppet and thus utilizes one separate module from whole repository

Right.  So what is the advantage in them sharing anything at all?

 

- both puppet masters use the same ENC already

Why?  Each master needs to classify nodes altogether differently than the other.  How does it make sense for them to use the same ENC?  Not that it really matters, I guess.

 

With this feature, I could downsize the management to:

- single environment repository, with environment.conf for primary and environment.conf-secondary for secondary puppet master process

- (please note that I have already patched puppet master so that it supports configurable name for "environment.conf")

Wait, are you trying to get rid of your need for separate `environment.conf` files, or not?  It sounds like you intend to continue with that mod, but if so, then I don't see why it doesn't already address all the issues you describe.

John

[Bostjan Skufca]

On Thursday, 19 March 2015 20:30:08 UTC+1, John Bollinger wrote:

Why?  You have two separate masters managing disjoint aspects of the target machines, via disjoint sets of agents.  Why do they need (or even want) to share *anything*?

Right.  So what is the advantage in them sharing anything at all?

You are right to ask that question. Sharing modules in the same repository instead of having two repos is kind of easier, it means less fiddling around with clones and pulls.

- both puppet masters use the same ENC already

Why?  Each master needs to classify nodes altogether differently than the other.  How does it make sense for them to use the same ENC?  Not that it really matters, I guess.

The truth is you could go about this two ways:

1. two puppet masters, serving content for each node by the same environment name

2. single puppet master, and having two puppet environments for each "environment", one with certain suffix

I prefer first method, as secondary duplicates the effort when deploying updates, as noted before.

With this feature, I could downsize the management to:

- single environment repository, with environment.conf for primary and environment.conf-secondary for secondary puppet master process

- (please note that I have already patched puppet master so that it supports configurable name for "environment.conf")

Wait, are you trying to get rid of your need for separate `environment.conf` files, or not?  It sounds like you intend to continue with that mod, but if so, then I don't see why it doesn't already address all the issues you describe.

Having two environment.conf files is the least of my worries, as this is "set and forget" thing. All I am striving for is replacing two dirs where I need to "git pull" with a single dir, where I do that only once for each environment.

I can achieve what I describe here by doing the following:

- have two masters using same paths for directory environments

- secondary master has environment.conf filename changed to something else, i.e. added suffix -secondary

- in each environment, environment.conf and environment.conf-secondary exist. The second one points puppet to the same module dir, but to different manifest dir.

- secondary manifest is a single catch-all node definition (well, almost) that only includes module for primary puppet's installation

- as this secondary manifest/catalog is so simple, there is little reason to keep it in separate dir/gitrepo - it is better to keep in in the same repo and point puppet to it when needed

Now, this works, but it feels a bit clunky. It feels like it could be done in a more generic way. This is the reason why I came up with the EEL idea. The more I think about it, the more natural it seems for puppet. Maybe it is just me.

Best regards,

b.

[John Bollinger]

On Thursday, March 19, 2015 at 6:19:19 PM UTC-5, Bostjan Skufca wrote:

[...] This is the reason why I came up with the EEL idea. The more I think about it, the more natural it seems for puppet. Maybe it is just me.

I'm afraid responses seem to indicate that it's mostly just you.  Perhaps that's because few people manage different dimensions of the same site via different masters, as you do.

In any event, I think you can achieve your objective of relying on a single repository / branch per environment with stock Puppet, like so:

• Each master has a distinct environment path
• Each environment directory contains an environment.conf file that gives an absolute path to a manifest directory located elsewhere.
  
• The manifest directories referenced by your environment configuration files are arranged however is convenient for you.  Perhaps, for instance, the actual manifest directories would have paths such as /etc/puppet/shared-environment/production/puppet1 and /etc/puppet/shared-environment/production/puppet2, such that you could do a single git pull per environment, as you want. In fact, you could even arrange to be able to do a single git pull for all environments at the same time.
• Other bits that you want to share across masters but not between distinct environments (modules, for instance), could go directly in /etc/puppet/shared-environment/whatever, referenced, again, by absolute paths.

If recording absolute paths in environment.conf files dissatisfies you, then you could use put symlinks in the per-master environment directories to direct relative paths into the shared master environment directory.

Although that may not be ideal, it would accomplish your main objectives, as I understand them.

John

[Bostjan Skufca]

On Friday, 20 March 2015 20:54:40 UTC+1, John Bollinger wrote:

On Thursday, March 19, 2015 at 6:19:19 PM UTC-5, Bostjan Skufca wrote:

[...] This is the reason why I came up with the EEL idea. The more I think about it, the more natural it seems for puppet. Maybe it is just me.

I'm afraid responses seem to indicate that it's mostly just you.  Perhaps that's because few people manage different dimensions of the same site via different masters, as you do.

Could be.

 

In any event, I think you can achieve your objective of relying on a single repository / branch per environment with stock Puppet, like so:

• Each master has a distinct environment path
• Each environment directory contains an environment.conf file that gives an absolute path to a manifest directory located elsewhere.
  
• The manifest directories referenced by your environment configuration files are arranged however is convenient for you.  Perhaps, for instance, the actual manifest directories would have paths such as /etc/puppet/shared-environment/production/puppet1 and /etc/puppet/shared-environment/production/puppet2, such that you could do a single git pull per environment, as you want. In fact, you could even arrange to be able to do a single git pull for all environments at the same time.
• Other bits that you want to share across masters but not between distinct environments (modules, for instance), could go directly in /etc/puppet/shared-environment/whatever, referenced, again, by absolute paths.

If recording absolute paths in environment.conf files dissatisfies you, then you could use put symlinks in the per-master environment directories to direct relative paths into the shared master environment directory.

Although that may not be ideal, it would accomplish your main objectives, as I understand them.

Thanks for all the feedback and suggestions, much appreciated.

b.

[Felix Frank]

On 03/19/2015 12:19 PM, Bostjan Skufca wrote:
>
> I think my note about suggesting git's way of providing "plumbing and
> porcelain" interfaces as good approach still stands.

I'm not so sure, actually.

How exactly would you go about segregating the existing configuration
options into two categories? I don't feel that the interface lends
itself well to that.

Cheers,
Felix

[Trevor Vaughan]

I think this may be an incorrect conceptual tie, but I do think that the plumbing/porcelain idea is both applicable to Puppet and something that Puppet should be advertising as a benefit.

In terms of the conceptual tie, a config file is a config file and is used to manage options. I think that the argument here is:

* Is this feature worth the amount of ongoing testing and maintenance to be relevant to the community in general?

There are, of course, several ways to determine this:

  * Take a poll (which was sort of done here but isn't generally the best place to get a poll, puppet-users would probably be a better place for that)

    ** Hijack: Perhaps there should be a [Feature Proposal] tag in Puppet Users for new feature polls?

  * Implement it as a patch, advertise it, and see how many people start using it. If it's a lot, incorporate it.

  * Implement it in the core, track the amount of upkeep that it takes, and remove it if the Golden Ratio of utility vice upkeep is not met

    ** This one has the downside of not knowing of the utility until you remove it and the Internet punches you in the head.

For the plumbing vs porcelain argument, this is one of the things that drew me to Puppet in the first place! One of the advertised qualities was that the language was written the way an admin thinks. It doesn't *require* you to get into the guts, program Ruby, or anything else you don't want to do. It just lets you configure your systems in a reasonably powerful manner using a relatively simple DSL.

Porcelain:

* The DSL

* Types

* Facts

* Hiera

* ENC (usage)

Plumbing

* The Compiler

* The Server

* Providers

* Facts (here too!)

* Custom Functions

* Hiera Back Ends

* ENC (core)

The porcelain should be this lovely place where the community protects less experienced users from randomly destroying systems and making everyone hate them for trying something new. It should be a place where everything *just works* and each new release is an improvement of efficiency which makes complex tasks straightforward and simple tasks trivial.

Also, the porcelain should promote code understandability and maintainability above all!

The plumbing should be a place where the efficiency happens and where users are allowed to destroy their servers from orbit if they so choose. This is the place where you've decided to ignore the DSL because you understand how things work but you want to be able to provide your work under the porcelain of the given system. Everything here is a leap forward as time progresses and moves at a proper Agile (yeah, I know) pace.

If every user were a power user we would not need this distinction. But, they're not, and some people don't want to deal with the plumbing. They don't want to write code, and they don't want to be a computer scientist. They just want to get things done.

In my opinion, this is one of the areas where Puppet shines and it is something that needs to be actively supported.

Let the systems be safe by default and easy to use but don't stifle the power users lest they be alienated.

After all this, back to the original topic:

* Would I use it? No

* Do I think it would be a great deal of upkeep? No

* Do I think a large portion of users would use it? No

* Do I see hard in incorporating it to see what happens? No

My opinion: Try it and see!

My suggestion is that Bostjan write a patch with clear instructions and announce it to the puppet-users list as something that you'd like to experiment with in terms of utility for the community.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/550DC03A.20608%40Alumni.TU-Berlin.de.

For more options, visit https://groups.google.com/d/optout.

[Bostjan Skufca]

I am not talking about changing configuration options themselves, nor about changing how configuration file looks like (by, for example, providing new sections). I am talking about marking configuration settings as "basic" or "advanced" and then using this "discriminatory value" when writing (or generating) documentation.

b.

[Bostjan Skufca]

On Saturday, 21 March 2015 21:57:02 UTC+1, Trevor Vaughan wrote:

In my opinion, this is one of the areas where Puppet shines and it is something that needs to be actively supported.

Let the systems be safe by default and easy to use but don't stifle the power users lest they be alienated.

After all this, back to the original topic:

* Would I use it? No

* Do I think it would be a great deal of upkeep? No

* Do I think a large portion of users would use it? No

* Do I see hard in incorporating it to see what happens? No

My opinion: Try it and see!

My suggestion is that Bostjan write a patch with clear instructions and announce it to the puppet-users list as something that you'd like to experiment with in terms of utility for the community.

This seems like a sane suggestion.

I will do this shortly.

b.

[Felix Frank]

On 03/23/2015 12:05 PM, Bostjan Skufca wrote:
> I am not talking about changing configuration options themselves, nor
> about changing how configuration file looks like (by, for example,
> providing new sections). I am talking about marking configuration
> settings as "basic" or "advanced" and then using this "discriminatory
> value" when writing (or generating) documentation.

Yes, that is what I was thinking. What I'm saying is, I don't quite see
which existing configuration settings would belong in column A and which
in B (documentation-wise).

But I've got to say, Trevor has put it quite aptly. Will have to put
more thought into this.

Cheers,
Felix