Proposal: External subcommands for puppet

[Andy Parker]

I've had to work on the guts on faces a fair amount and over time there have been a few limitations that they have, which make progress in puppet difficult at times. So after looking over it for a while, and looking at what puppet already has, I'm putting out a proposal that extending puppet with new commands be done differently.

There is a whole doc that I've written up laying out the argument and also with some thought on what it will take to implement it well:

https://docs.google.com/document/d/1ipnIchBsFmISEKlyS_oGXslutnC9m4JJ84zXMgaoqSI/edit?usp=sharing

I'd really like some comments on this. At the moment we are thinking of getting this in a puppet 3.5 release, but that is contingent on some feedback and review of the idea.

--

Andrew Parker

an...@puppetlabs.com

Freenode: zaphod42

Twitter: @aparker42

Software Developer

Join us at PuppetConf 2014, September 23-24 in San Francisco

[Eric Sorenson]

Andy Parker wrote:
> I'd really like some comments on this. At the moment we are thinking of
> getting this in a puppet 3.5 release, but that is contingent on some
> feedback and review of the idea.

FYI all, there hasn't been follow-up on the mailing list but there is a
ton of discussion in the google doc. Please check it out if this is a
topic that interests you.

https://docs.google.com/a/puppetlabs.com/document/d/1ipnIchBsFmISEKlyS_oGXslutnC9m4JJ84zXMgaoqSI/edit

--
Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[Andy Parker]

On Sun, Dec 15, 2013 at 10:10 PM, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

Andy Parker wrote:

I'd really like some comments on this. At the moment we are thinking of
getting this in a puppet 3.5 release, but that is contingent on some
feedback and review of the idea.

FYI all, there hasn't been follow-up on the mailing list but there is a ton of discussion in the google doc. Please check it out if this is a topic that interests you.

https://docs.google.com/a/puppetlabs.com/document/d/1ipnIchBsFmISEKlyS_oGXslutnC9m4JJ84zXMgaoqSI/edit

Thanks for all of the feedback so far. I think most issues have been addressed. The biggest open question is how to deal with sharing configuration information.

Pieter points out that making sure that subcommands have a way of getting a reusable view of the configuration is very important. He also points out that allowing any arbitrary puppet settings that might get added to interfere with existing subcommands that were developed before the setting was added could be a problem.

<quote Pieter>

This seems vague and a little dangerous. Not only does it imply that the same `--vardir` flag could do different things in different subcommands, but it forces subcommand authors to be aware of every 

potential interaction between every core Puppet option (which may be dynamic from one version to the next). In addition, the subcommand may be invoked too late to alter certain properties (e.g. `runmode`, which affects `modulepath`, which may affect subcommand load order).

As an alternative suggestion, is it more reasonable to ask Puppet core to parse out certain "well-known" option names prior to subcommand invocation, document those options well, and handle configuration alteration in Puppet itself? 

</quote>

Jeff pointed out that he'd really like options to be settable from the environment during the discussion about how to make sure that configuration information gets to the subcommands correctly.

<quote Jeff>

I'd really like configuration information, which is basically what command line options are, to be read from the environment. This isn't to say that the environment is the only place, just that it MUST be supported. This will enable subcommands to work well in stateless environments like Heroku, Docker, etc...

This has worked very well for Heroku applications and is specifically called out in documentation on the subject, such as the 12 Factor App http://12factor.net/config

</quote>

I think that some sort of synthesis of those two concerns/suggestions might get to something that would be a) powerful for configuring, b) robust in the face of future changes to puppet and subcomands, and c) understandable to users of the system where command line args and environment variables just "do the right thing".

I'm however having a little trouble with exactly what it might be. Any thoughts?

--
Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/52AE996A.5010902%40puppetlabs.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jeff McCune]

On Mon, Dec 16, 2013 at 11:06 AM, Andy Parker <an...@puppetlabs.com> wrote:

<quote Jeff>

I'd really like configuration information, which is basically what command line options are, to be read from the environment. This isn't to say that the environment is the only place, just that it MUST be supported. This will enable subcommands to work well in stateless environments like Heroku, Docker, etc...

This has worked very well for Heroku applications and is specifically called out in documentation on the subject, such as the 12 Factor App http://12factor.net/config

</quote>

Just a point of clarification, I mean "MUST" in the RFC style of MAY/SHOULD/MUST requirements specification, not in the sense that I'm trying to dictate anything.  So, configuration information would be required to be looked up in the environment in addition to other sources.

 

I think that some sort of synthesis of those two concerns/suggestions might get to something that would be a) powerful for configuring, b) robust in the face of future changes to puppet and subcomands, and c) understandable to users of the system where command line args and environment variables just "do the right thing".

I'm however having a little trouble with exactly what it might be. Any thoughts?

I've had really good luck with a two-pass command line option parser in my own little "toy" sub commands.  The form would look like:

puppet [global options] subcommand [subcommand options]

At the implementation layer, all global options are guarnteed to be fully resolved by the time subcommand options are parsed.  The way both of these are resolved is by walking the precedence list and using the first one that's defined:

1: Value specified as a command line option.

2: Value specified as an environment variable.

3: Value specified in a configuration file.

4: Default value located in the code.

If we require that no subcommand option conflicts the name of a global option, would this give us what we need?

While we have a ton of options and specifying all of them as environment variables might over-flow the environment, in practice only a handful of options after vary from one context to another so I doubt we'd exceed the capacity of the environment.

--

Jeff McCune