Facter config file

[Eric Sorenson]

I've been thinking about a config file for Facter, which has historically not been run-time configurable.

The two problems in front of me that seem applicable are:

* Sometimes, certain facts are just plain bad to collect and users would like to prevent them from even being resolved (see FACT-718, FACT-449, ).

* Some facts are not inherently bad but _are_ expensive and/or change infrequently, so preventing them from being resolved every time would be beneficial (FACT-348)

Are there other problems you're running into in this area that you'd like to see addressed with a "facter.conf"? I'd like to gather all the requirements and start up a little Puppet RFC based on them.

--eric0

[Matthew Gaspar]

A config file for facter is intriguing to me.

Would this solely be used to change behaviors when running facter, or do you see some potential for defining some facts within the config itself?

Also, would you move facter specific configs into this facter.conf instead of being managed from the puppet.conf file (ie: cfacter, factpath, fact_terminus)?

The only problem I sometimes encounter, which may be a usage issue on my part, is when creating custom facts sometimes it'd be nice to just run `facter my_custom_fact` to get the output. If there would be some way to register custom facts so that facter picks them up without having to run puppet or run the ruby code the custom fact is in manually, that would be interesting. If that already exists I haven't found how to do that.

Either way, I think a config where you can either specify which facts to enable or disable would be useful.

[Michael Stahnke]

On Tue, Mar 1, 2016 at 12:25 PM, Matthew Gaspar <gat...@gmail.com> wrote:

A config file for facter is intriguing to me.

Would this solely be used to change behaviors when running facter, or do you see some potential for defining some facts within the config itself?

I'd rather not mix config with content. If we want a facts.txt or something, that's cool, but not configuration.  

Also, would you move facter specific configs into this facter.conf instead of being managed from the puppet.conf file (ie: cfacter, factpath, fact_terminus)?

I would like to see the search path for custom facts set in here. (And get facts.d out of /etc, cause I feel bad about that). 

There might also be something an 'expensive facts' list that we only resolve once a day, or custom TTL settings, or something to improve the performance of gathering facts. I'm just spitballing here though. 

The only problem I sometimes encounter, which may be a usage issue on my part, is when creating custom facts sometimes it'd be nice to just run `facter my_custom_fact` to get the output. If there would be some way to register custom facts so that facter picks them up without having to run puppet or run the ruby code the custom fact is in manually, that would be interesting. If that already exists I haven't found how to do that.

Either way, I think a config where you can either specify which facts to enable or disable would be useful.

 

On Tuesday, March 1, 2016 at 11:52:12 AM UTC-7, Eric Sorenson wrote:

I've been thinking about a config file for Facter, which has historically not been run-time configurable.

The two problems in front of me that seem applicable are:

* Sometimes, certain facts are just plain bad to collect and users would like to prevent them from even being resolved (see FACT-718, FACT-449, ).

* Some facts are not inherently bad but _are_ expensive and/or change infrequently, so preventing them from being resolved every time would be beneficial (FACT-348)

Are there other problems you're running into in this area that you'd like to see addressed with a "facter.conf"? I'd like to gather all the requirements and start up a little Puppet RFC based on them.

--eric0

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/0f42fe6f-1f75-4aff-aa6a-77dd3e17235c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Sorenson]

On Mar 1, 2016, at 12:25 PM, Matthew Gaspar <gat...@gmail.com> wrote:

A config file for facter is intriguing to me.

Would this solely be used to change behaviors when running facter, or do you see some potential for defining some facts within the config itself?

I think it'd be only settings/behavior - if there were fact values, those can go into facts.d as yaml/json/executables today.  (Is there some reason that doesn't work for you?)

Also, would you move facter specific configs into this facter.conf instead of being managed from the puppet.conf file (ie: cfacter, factpath, fact_terminus)?

Hm, some of these are more settings that puppet uses to interact with facter. factpath would certainly move. 

The only problem I sometimes encounter, which may be a usage issue on my part, is when creating custom facts sometimes it'd be nice to just run `facter my_custom_fact` to get the output. If there would be some way to register custom facts so that facter picks them up without having to run puppet or run the ruby code the custom fact is in manually, that would be interesting. If that already exists I haven't found how to do that.

Well, today you can run 'facter -p' to get it to load facts from the pluginsync target directory. Or make these external facts (standalone programs rather than ruby plugins) and they will be loaded on regular facter runs. 

https://docs.puppetlabs.com/facter/3.1/custom_facts.html#external-facts

Does that make sense?

Either way, I think a config where you can either specify which facts to enable or disable would be useful.

Right on, thanks for the reply!

Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

[R.I.Pienaar]

Some individual facts might benefit from configuration. 

 - never consider docker*,and,others for ipaddress fact

  - ec2 facts IP address to hit

  - default gateway device

  - override some paths to required binaries

Etc, tons of these. So some way that we all agree on to ingest config on a per fact basis

--eric0

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/6b93869b-83b3-43dc-8784-bd2cf173e54c%40googlegroups.com.

[Rob Nelson]

On Tuesday, March 1, 2016, Matthew Gaspar <gat...@gmail.com> wrote:

The only problem I sometimes encounter, which may be a usage issue on my part, is when creating custom facts sometimes it'd be nice to just run `facter my_custom_fact` to get the output. If there would be some way to register custom facts so that facter picks them up without having to run puppet or run the ruby code the custom fact is in manually, that would be interesting. If that already exists I haven't found how to do that.

 

This probably isn't in scope for this, but the above is my most highly sought after goal.

 

Either way, I think a config where you can either specify which facts to enable or disable would be useful.

 

We manage firewalls and routers as well as server OSes and it would be nice to flag which facts shouldn't cause errors on every run on those nodes. I would suggest this information should be available locally AND/OR through the master/agent mechanism somehow - please don't require yet another file{} resource for puppet related settings in every node's manifest.

--

Rob Nelson
rnel...@gmail.com

[Kylo Ginsberg]

On Tue, Mar 1, 2016 at 3:13 PM, R.I.Pienaar <r...@devco.net> wrote:

On 1 Mar 2016, at 19:52, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

I've been thinking about a config file for Facter, which has historically not been run-time configurable.

The two problems in front of me that seem applicable are:

* Sometimes, certain facts are just plain bad to collect and users would like to prevent them from even being resolved (see FACT-718, FACT-449, ).

* Some facts are not inherently bad but _are_ expensive and/or change infrequently, so preventing them from being resolved every time would be beneficial (FACT-348)

One question I'm curious to get feedback on is whether such a blacklist (or whitelist?) of facts would be at the top-level-structured-fact basis, or whether there are compelling use case for it to be more fine-grained.

The per-top-level-structured fact basis would have some nice attributes:

* it's simpler (good unless it's too simple)

* given that one of the goals in skipping some facts, that would align pretty nicely with the facter 'resolvers' - whereas to support fine-grained blacklisting of facts might still require *collecting* all the facts, and just blacklisting at the point of return/reporting.

Similar question (but may not be the same answer) for fact ttl's. Also for fact ttl's, I'd think we could provide some useful defaults, e.g. osfamily doesn't change during process lifetime, that sort of thing.

I'd be curious for comments on any of the above.

 

Are there other problems you're running into in this area that you'd like to see addressed with a "facter.conf"? I'd like to gather all the requirements and start up a little Puppet RFC based on them.

Some individual facts might benefit from configuration. 

 - never consider docker*,and,others for ipaddress fact

Ah yes, that makes sense. We've had a few requests for fine-tuning ipaddress fact collection that could possibly be met by a regex along those lines.

  - ec2 facts IP address to hit

Sorry, naive question but I thought the ec2 metadata address was always the same? It's hardwired in facter today.

 

  - default gateway device

  - override some paths to required binaries

What are some of the example use cases for overriding path to binaries? Are there use cases for overriding path to some of the non-binary files that facter processes?

Etc, tons of these. So some way that we all agree on to ingest config on a per fact basis

I like the idea in general, and would love to get more color on the spectrum of use cases, hence the questions above.

Kylo

 

--eric0

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/6b93869b-83b3-43dc-8784-bd2cf173e54c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/3C1D9D77-61CF-49A1-90AB-E7277BB9B636%40devco.net.

For more options, visit https://groups.google.com/d/optout.

--

Kylo Ginsberg | ky...@puppetlabs.com | irc: kylo | twitter: @kylog

[Kylo Ginsberg]

On Tue, Mar 1, 2016 at 3:13 PM, Rob Nelson <rnel...@gmail.com> wrote:

On Tuesday, March 1, 2016, Matthew Gaspar <gat...@gmail.com> wrote:

The only problem I sometimes encounter, which may be a usage issue on my part, is when creating custom facts sometimes it'd be nice to just run `facter my_custom_fact` to get the output. If there would be some way to register custom facts so that facter picks them up without having to run puppet or run the ruby code the custom fact is in manually, that would be interesting. If that already exists I haven't found how to do that.

 

This probably isn't in scope for this, but the above is my most highly sought after goal.

Does 'facter -p my_custom_fact' do the trick here?

 

Either way, I think a config where you can either specify which facts to enable or disable would be useful.

 

We manage firewalls and routers as well as server OSes and it would be nice to flag which facts shouldn't cause errors on every run on those nodes.

Ah yeah, I've heard this a few times as well. But why is it more of a thing on firewalls/routers?

I would suggest this information should be available locally AND/OR through the master/agent mechanism somehow - please don't require yet another file{} resource for puppet related settings in every node's manifest.

Is the concern here about manifest bloat?

 

--

Rob Nelson
rnel...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAC76iT-eEdup-%3DM2jJH-8-_GFj4EywQjX%2BihaS%2BLhgC8_9di_g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Rob Nelson]

On Tuesday, March 1, 2016, Kylo Ginsberg <ky...@puppetlabs.com> wrote:

Does 'facter -p my_custom_fact' do the trick here?

In my case, it's while developing the fact, prior to distribution. Takes some trickery to rig it up for testing.

 

Ah yeah, I've heard this a few times as well. But why is it more of a thing on firewalls/routers?

Lots of the binaries and even paths don't exist, so some of the common facts spew errors; anything custom and you have to deal with the errors in your logs or your two liner becomes an excessively defensive novel.

 

Is the concern here about manifest bloat?

Some. It also separates apply and agent, which may have different desires. On the agent, I may want something different for production vs a feature branch, too, where the fact may actually be in development. If that could be done via environment.conf, say, it could help.

--

Rob Nelson
rnel...@gmail.com

[Henrik Lindberg]

I would very much like to be able to type facts using the puppet type
system. While this could be done in other places, a Facter config would
be very useful to have server side when compiling as the compiler can
then validate against such a configuration without having to do the full
compilation with a full set of values.

This also opens up for the possibility to do real type inference. This
is not possible when facts are in essence untyped.

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

[R.I.Pienaar]

On 2 Mar 2016, at 03:45, Kylo Ginsberg <ky...@puppetlabs.com> wrote:

On Tue, Mar 1, 2016 at 3:13 PM, R.I.Pienaar <r...@devco.net> wrote:

On 1 Mar 2016, at 19:52, Eric Sorenson <eric.s...@puppetlabs.com> wrote:

I've been thinking about a config file for Facter, which has historically not been run-time configurable.

The two problems in front of me that seem applicable are:

* Sometimes, certain facts are just plain bad to collect and users would like to prevent them from even being resolved (see FACT-718, FACT-449, ).

* Some facts are not inherently bad but _are_ expensive and/or change infrequently, so preventing them from being resolved every time would be beneficial (FACT-348)

One question I'm curious to get feedback on is whether such a blacklist (or whitelist?) of facts would be at the top-level-structured-fact basis, or whether there are compelling use case for it to be more fine-grained.

The per-top-level-structured fact basis would have some nice attributes:

* it's simpler (good unless it's too simple)

* given that one of the goals in skipping some facts, that would align pretty nicely with the facter 'resolvers' - whereas to support fine-grained blacklisting of facts might still require *collecting* all the facts, and just blacklisting at the point of return/reporting.

Similar question (but may not be the same answer) for fact ttl's. Also for fact ttl's, I'd think we could provide some useful defaults, e.g. osfamily doesn't change during process lifetime, that sort of thing.

I'd be curious for comments on any of the above.

 

Are there other problems you're running into in this area that you'd like to see addressed with a "facter.conf"? I'd like to gather all the requirements and start up a little Puppet RFC based on them.

Some individual facts might benefit from configuration. 

 - never consider docker*,and,others for ipaddress fact

Ah yes, that makes sense. We've had a few requests for fine-tuning ipaddress fact collection that could possibly be met by a regex along those lines.

  - ec2 facts IP address to hit

Sorry, naive question but I thought the ec2 metadata address was always the same? It's hardwired in facter today.

You get proxies and caches and compatible services for other clouds. Along the same lines I know in the past the fact checked some hard coded MAC address this  could be a good target

 

  - default gateway device

  - override some paths to required binaries

What are some of the example use cases for overriding path to binaries? Are there use cases for overriding path to some of the non-binary files that facter processes?

People who install hand compiled software for example. Not exactly a path but kind of analogous  people who write custom facts that uses internal APIs, these might need pointing to different places in different environments 

 

Etc, tons of these. So some way that we all agree on to ingest config on a per fact basis

I like the idea in general, and would love to get more color on the spectrum of use cases, hence the questions above.

Yeah so my list is more hypothetical than actual I Have These Problems but certainly seen some of these on IRC and the class of feature has many uses

It is though a very difficult feature because you might need to config this at plugin sync time and plugin sync is too dumb to have any kind of selection logic ie. to put different configs on different hosts or template bake configs etc. Whenever I have thought of requesting this feature I always end up needing this part and I think logic in plugin sync would be bad. 

Additionally surfacing the available configs for a fact in a user consumable way that also supports custom facts could be a pain. I have often felt though that facts need to support doc strings that can be queried via the CLI etc, that might help a lot if we go this route - and in general I think would be a hugely helpful thing. 

Kylo

 

--eric0

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/6b93869b-83b3-43dc-8784-bd2cf173e54c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/3C1D9D77-61CF-49A1-90AB-E7277BB9B636%40devco.net.

For more options, visit https://groups.google.com/d/optout.

--

Kylo Ginsberg | ky...@puppetlabs.com | irc: kylo | twitter: @kylog

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALsUZFEZ8arhPooYuv_%2BriofhQnM74aZzG5ZUG0G8cd%3DoscEfA%40mail.gmail.com.

[Trevor Vaughan]

R.I., did I just hear you make a suggestion for writing fact DDL files? Not sure how I feel about that, they're a pain to deal with and confuse users.

The features that I would like to add my vote for:

* Fact TTLs

* Custom fact execution timeouts

* Fact execution timeouts

* Fact parallelization (and settings therein)

* Moving the facts.d location

* Asynchronous fact reporting (I don't want to have to run facter every puppet run)

* System fact ACLs (have the ability to allow other hosts access to facts)

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/C15C5276-E40B-49F0-93EC-FA9087AEE5A3%40devco.net.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

-- This account not approved for unencrypted proprietary information --

[R.I.Pienaar]

On 2 Mar 2016, at 21:16, Trevor Vaughan <tvau...@onyxpoint.com> wrote:

R.I., did I just hear you make a suggestion for writing fact DDL files? Not sure how I feel about that, they're a pain to deal with and confuse users.

Not specifically. Types already have doc strings similar would be enough for this. 

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWw3d9B%2BAfvA5NL8gLp%2BH7-OgbiLDqcw318okz6Nc6nnQ%40mail.gmail.com.

[Eric Sorenson]

On Tue, 1 Mar 2016, Eric Sorenson wrote:

> I've been thinking about a config file for Facter, which has historically
> not been run-time configurable.

A little thread necromancy here, as I finally got back to this and started
working into a real proposal as Puppet RFC 24, Facter Config File. WIP google
doc here:
https://docs.google.com/document/d/1xlj0NYESLHNv5ZO_NJ-1fCssAWzaHsTZIhyaK3uzHQY/edit

I think I summarized everyone's use-case in the "Problems and People" section
but feel free to comment/suggest if I've mischaracterized something. I have a
few more inline questions on the way too.

[Eric Sorenson]

On Wed, 2 Mar 2016, Trevor Vaughan wrote:

> R.I., did I just hear you make a suggestion for writing fact DDL files? Not
> sure how I feel about that, they're a pain to deal with and confuse users.
>
> The features that I would like to add my vote for:
>
> * Fact TTLs
> * Custom fact execution timeouts
> * Fact execution timeouts
> * Fact parallelization (and settings therein)

Hmm, I'd question the ROI of trying to parallelize something that is already
pretty fast and can be made even faster with a working client-side fact cache
that has long TTLs for expensive facts. Are there benefits besides speed you
have in mind here?

> * Moving the facts.d location
> * Asynchronous fact reporting (I don't want to have to run facter every
> puppet run)
> * System fact ACLs (have the ability to allow other hosts access to facts)

Can you explain the problem statement behind this last one a little better?

[Eric Sorenson]

On Wed, 2 Mar 2016, Henrik Lindberg wrote:

> I would very much like to be able to type facts using the puppet type system.
> While this could be done in other places, a Facter config would be very
> useful to have server side when compiling as the compiler can then validate
> against such a configuration without having to do the full compilation with a
> full set of values.
>
> This also opens up for the possibility to do real type inference. This is not
> possible when facts are in essence untyped.

This seems orthogonal to an agent-side facter config; you're talking about a
Type "signature" of what the returned structure of a fact should look like,
right?

[Henrik Lindberg]

On 23/05/16 22:06, Eric Sorenson wrote:
> On Wed, 2 Mar 2016, Henrik Lindberg wrote:
>
>> I would very much like to be able to type facts using the puppet type
>> system. While this could be done in other places, a Facter config
>> would be very useful to have server side when compiling as the
>> compiler can then validate against such a configuration without having
>> to do the full compilation with a full set of values.
>>
>> This also opens up for the possibility to do real type inference. This
>> is not possible when facts are in essence untyped.
>
> This seems orthogonal to an agent-side facter config; you're talking
> about a Type "signature" of what the returned structure of a fact should
> look like, right?
>

Yes like in "the fact 'foo' is an Integer" - preferably expressed in
terms of the Puppet Type System.

- henrik

> Eric Sorenson - eric.s...@puppetlabs.com - freenode #puppet: eric0
> puppet platform // coffee // techno // bicycles
>

[Thomas Müller]

I do see the benefits of a facter config to disable or override facts.

I do see problems that puppet manifests depend on facter facts and puppet configures facter. if the config is not sent with the module facts it requires 2 puppet runs to get the correct config. 

About facter config and override things, maybe the opinionated systemd could lend some inspiration: systemd has 4 directories where it looks up settings:

- /usr/lib/systemd/system/
- /etc/systemd/system
- /etc/systemd/system.d/${unit}.d/*.conf (does not override, but provides the possibility to add settings)
- /run/systemd/system

the last one wins.

maybe something similiar could be done with facter? 

- Thomas

[Michael Smith]

There's some prior art on this question as well: https://projects.puppetlabs.com/issues/11449. It doesn't appear to add anything that wasn't already mentioned here however.

[Eric Sorenson]

Hey all, I completed this write-up, any comments/feedback appreciated.

https://docs.google.com/document/d/1xlj0NYESLHNv5ZO_NJ-1fCssAWzaHsTZIhyaK3uzHQY/edit#

I also put a PR against the puppet-specifications repo showing the filesystem locations in case it's easier for you to visualize this way:

https://github.com/puppetlabs/puppet-specifications/pull/79

Fire away...

--eric0

[Eric Sorenson]

Thanks for the feedback, everyone - I've incorporated as much as I could into the doc and have a PR against the Puppet-RFC repo with the draft content. Please continue to post questions/comments on this thread.

John -- I think the comments on your HOCON question were constructive, but I fear they were not convincing. I looked back through the archives and realized that all of the discussions (which were, as Ryan W noted in the comment, pretty heated) about the introduction of that format happened on internal lists. This was wrong, and I now have a TODO item to more clearly describe the rationale behind and support statements for the use of Hocon around Puppet's ecosystem. 

--eric0