Jira (PUP-5934) Updated fact values should be submitted after each Puppet run

[Reid Vandewiele (JIRA)]

Reid Vandewiele created an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run Issue Type: New Feature Assignee: Unassigned Created: 2016/02/19 2:31 PM Priority: Normal Reporter: Reid Vandewiele Today it is the case that facts are generated prior to a Puppet run, then Puppet makes changes to the system. Because changes are made, it may be the case that the true value of a fact for a system will have changed after the run, but it will not be updated in PuppetDB until the run interval elapses (typically 30 minutes) and the next run begins. It is becoming common for PuppetDB queries to be used to select nodes based on fact values, for grouping and reporting. In addition, modules like Erik Dalén's puppetdbquery allow the master to build catalogs for node A based on facts about node B. The longer the time window during which node B's PuppetDB facts are out sync with the node's real fact values, the longer it takes for the infrastructure as a whole to converge on a correct, consistent configuration. Additionally, the intuitive mental model of reading Puppet reports and fact lists in the console is "Puppet submitted a report at time X, therefore I know that the facts shown are valid as of time X". However, this is not true today as described above. Feature Request In addition to submitting facts at the beginning of each run, Puppet should submit updated fact values at the end of each run along with the report. Add Comment

This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Ping Kylo Ginsberg Change By: Henrik Lindberg Scrum Team: Client Platform

Add Comment

[Kylo Ginsberg (JIRA)]

Kylo Ginsberg commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Hmm, this may warrant a chat to get right. If anything I've thought about decoupling fact submission from agent runs entirely, and ideally making them submitted at different rates depending on their volaitility. In part that idea is inspired by concerns like the one you raise here:

The longer the time window during which node B's PuppetDB facts are out sync with the node's real fact values, the longer it takes for the infrastructure as a whole to converge on a correct, consistent configuration.

In any event, re the solution requested:

Puppet should submit updated fact values at the end of each run along with the report.

The facts at the time the report was submitted seems kind of meaningless. Those fact values weren't used for anything and so don't correspond to anything else, including the report. What you're really asking for is very current facts. I think that's what we should aim for, not piggybacking the facts on the last known thing the agent did.

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

The facts at the time the report was submitted seems kind of meaningless. Those fact values weren't used for anything and so don't correspond to anything else, including the report. What you're really asking for is very current facts. I think that's what we should aim for, not piggybacking the facts on the last known thing the agent did. Yes, I think that's accurate. The suggestion of report submission time was born out of an imagined implementation, and still might make a good first "all facts must be up to date when this event occurs no matter what" milestone. That said, if there were some magical way of just always having up-to-date facts instead, that would totally solve the problem statement.

Add Comment

[Kylo Ginsberg (JIRA)]

Kylo Ginsberg commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Email thread on puppet-users group requesting the same. It dates from 2013, but got a me-too in early 2016: https://groups.google.com/d/msg/puppet-users/z8VijBU3oH4/cxKSti4gVeoJ

Add Comment

[Verne Lindner (JIRA)]

Verne Lindner commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Kylo Ginsberg Are there any near-term plans to address this ticket? I'm wondering if we need to think about labelling in the PE GUI for nodes to indicate the last time their facts were updated, or should we wait for the solution you propose in comments above.

Add Comment

This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)

[Kylo Ginsberg (JIRA)]

Kylo Ginsberg commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Verne Lindner no near-term plans to address this, so a label in the PE GUI with the last update time makes sense to me. Also, /cc Eric Sorenson who's been thinking some about async fact submission and the like.

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele updated an issue

Puppet / PUP-5934

Updated fact values should be submitted after each Puppet run

Change By: Reid Vandewiele

Today it is the case that facts are generated prior to a Puppet run, then Puppet makes changes to the system. Because changes are made, it may be the case that the true value of a fact for a system will have changed after the run, but it will not be updated in PuppetDB until the run interval elapses (typically 30 minutes) and the next run begins.

It is becoming common for PuppetDB queries to be used to select nodes based on fact values, for grouping and reporting. In addition, modules like Erik Dalén's puppetdbquery allow the master to build catalogs for node A based on facts about node B. The longer the time window during which node B's PuppetDB facts are out sync with the node's real fact values, the longer it takes for the infrastructure as a whole to converge on a correct, consistent configuration.

Additionally, the intuitive mental model of reading Puppet reports and fact lists in the console is "Puppet submitted a report at time X, therefore I know that the facts shown are valid as of time X". However, this is not true today as described above.

A common _customer assumed_ use case for facts is being able to query facts in real time to validate information about a server. Because of today's limitations, querying facts does not provide real-time data. At best, the data provided is lagging real-time by up to {{run_interval}} on each node. h2. Feature Request In addition to submitting facts at the beginning of each run, Puppet should submit updated fact values at the end of each run along with the report.

Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Got one idea around this set of functionality... What if the agent ran facter on a regular interval and compared the set of facts against the last run. In the simplest case, if facts differ, initiate a scheduled run, or just send them to the server. (You would obviously always filter out ever changing facts before computing if there is a diff).

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

While real-time is the shining ideal, real-world use cases usually correspond pretty directly to Puppet reports. The impression customers usually have is that when looking at a node detail page, for a node that last reported at timestamp, the facts shown are representative of the enforced state of that node at that time. That is, the fact values at the end of a run. Aiming for real-time as a first iteration seems like a perfect is the enemy of good scenario, and a diminished return.

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Can users who are interested in this run puppet facts upload as a post-run command? I've linked this ticket to another one regarding puppet facts, which will become more relevant as cached catalogs become a common thing.

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson updated an issue

Puppet / PUP-5934

Updated fact values should be submitted after each Puppet run

Change By: Eric Sorenson Team: Agent & Platform

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Eric Sorenson puppet facts upload no longer exists. In past versions of puppet the agent would upload it's facts to the master before compiling a catalog and then those facts were used to compile the catalog. Now you have to submit facts with a request to compile a catalog and there is no way to submit facts outside of the request to compile a catalog ( or so that is my understanding).

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Well, that sucks. [root@cloudline repros]# puppet help facts save   USAGE: puppet facts save [--terminus TERMINUS] [--extra HASH] <key>   API only: create or overwrite an object. As the Faces framework does not currently accept data from STDIN, save actions cannot currently be invoked from the command line.

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Eric Sorenson my understanding is there is no longer an API endpoint on puppetserver to accept facts independent of a catalog request, even if you were inclined to modify the face or agent code to send them.

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Yes, one possible way to achieve this would be to allow agents access to update their own facts and only their own facts directly via the puppetdb API. However, I'm not sure that's desirable in more locked down architectures and it may still make sense to have an API on the master that forwards the facts to PuppetDB. In addition to submitting facts after a puppet run, you should be able to submit facts on whatever interval you prefer.

Add Comment

[Verne Lindner (JIRA)]

Verne Lindner commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Reid Vandewiele If, in the console, we were to report on fact submission time, with report time, do you believe that would provide any value to users?

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Verne Lindner in that it would help clarify to users what was happening, I think yes, there is a small potential value there. However, it's pretty small. The timestamps change all the time, and it's a bit of a stretch for people to read the timestamp and realize that the facts are always lagging the report, and then further connect the facts are older than the report and could have changed. There would be far more value in making the system match people's intuition about how it should work. How it should work is simple. How it does work is very unintuitive.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-5934

Updated fact values should be submitted after each Puppet run

Change By: Henrik Lindberg Labels: triaged

Add Comment

[Verne Lindner (JIRA)]

Verne Lindner commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Henrik Lindberg Just noticed a "triaged" label was applied to this ticket. Does that mean it's heading into a backlog for grooming?

Add Comment

[Verne Lindner (JIRA)]

Verne Lindner commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Henrik Lindberg In reference to PUP-5934, do you have an idea of when that ticket might be groomed? I noticed it going into Triage a while back. For PE Console reporting, this issue impacts accuracy.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

The fact that it was "triaged" and not closed only means that this ticket was considered to be a valid ticket.

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Verne Lindner this came up in discussion on the Professional Services channel today in the context of looking for docs that describe Facter at a high level. Gist of the conversation was our docs do a decent job of describing WHAT Facter is, and HOW Facter works, but don't really touch on WHY Facter is useful. Some suggested "Why do we have Facter" were: Facter allows Puppet manifest writers to create dynamic configurations that adapt to machine-level specifics automatically Facter allows adminstrators to group nodes together by business purpose, application type, SDLC level, or other characteristic facts Facter allows users to create queries and node lists grouped by software licensing status, installed package versions, etc. Today, Facter excels only at that first use case because of this ticket. The second and third use cases are as compelling, if not more, but realizing use cases #2 and #3 is at a real-world disadvantage because the accuracy of information available lags 30 minutes behind reality. This ticket might well be re-titled "Promote Facter from a supporting role into a first-class feature" which fully supports all use cases ascribed to it, and not just use case #1 above.

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Eric Sorenson Are you saying that Puppet will never automatically submit facts after an agent run and it will always be up to the user to configure that themselves if they want to? My suggestion is that this ticket should be implemented on top of PUP-7779. I think Puppet should have the ability to natively submit facts before and after a puppet run and not require the user to setup `puppet facts upload` in a post_run command.

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

+1 to Nick Walker's point. I am strongly in favor of PUP-7779 and think we need it. This ticket most naturally then becomes an improvement request to be built on top of PUP-7779. PUP-7779 is not an alternate solution; it's a prerequisite. If "won't fix" is still correct for this ticket, can you please provide more background information about the decision?

Add Comment

[Jon Pugh (JIRA)]

Jon Pugh commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

+1 to the points made by Nick and Reid. As a customer I find the Puppet Console display of facts as they were before the last run to be extremely un-intuitive and find myself explaining to new team members that the reason they are seeing old values for facts they expected to be updated or created by a puppet run is due to this. A "before and after" view would be ideal - at the moment the "after" view only appears after the next run - some people end up doing a second run just to see the facts change. Make it a parameter on the agent if you want customers to actively acknowledge the behaviour perhaps? We also find it devalues the use of fact queries via PuppetDB significantly though I acknowledge the points made about "very current facts" being the underlying need.

Add Comment

[Michael Smith (JIRA)]

Michael Smith commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Submitting facts as part of the report seems like intuitive behavior to me.

Add Comment

[Jeff Sparrow (JIRA)]

Jeff Sparrow commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

1 have wanted this for 5 years

Add Comment

[William Rodriguez (JIRA)]

William Rodriguez commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

I'd love to see this, especially with PUP-7779. We have many use-cases where having facts updated faster would be a great value add for us. For example, with the package inventory, we don't see packages being updated until the next time puppet runs, which slows down remediation efforts. A simple flag to have the new facts upload face called after execution would be nice and we'd probably set it across the board. I'm pretty sure I've actually commented before to Nick Walker my thoughts on this, well before I knew this ticket even existed.

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Verne Lindner (JIRA)]

Verne Lindner commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

William Rodriguez If a user creates a task in the console to update a package version, then runs a job, then checks their console Packages inventory, will they still see the old package version appear?

Add Comment

[William Rodriguez (JIRA)]

William Rodriguez commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Ok, that was a bad example. I realize now that they actually are updating if updated via package inventory. The situation I was thinking of was a situation where we had updated the puppet-agent package across the fleet via a puppet agent run, using the puppet-agent module not a task, and it took us the length of an agent run to see the changes reflected across the fleet. I had forgotten that the package task actually does update inventory once a remediation action is taken. Sorry for scaring you Verne Lindner

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Verne Lindner that's correct. When a user runs a task in the console to update a package version, the Packages inventory will fail to display the correct package version(s) until after the next Puppet run begins. Interestingly, the Puppet run doesn't have to finish for the new package version to show up in the package inventory—it only has to start. Facts, including package inventory, are updated at the beginning of a Puppet run.

Add Comment

[Verne Lindner (JIRA)]

Verne Lindner commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

William Rodriguez Thanks for the clarification! Ryan Coleman How do you feel about re-opening this ticket as an OPTY? Not seeing updated facts after a run remains a source of confusion for users; as we promote use of tasks in the console, it seems even more important for users to see what they expect to see: create task > run job > see results (and, in some cases, report results), without having to know that an additional run is required if those results include a fact value change.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Now that PUP-7779 is in, I think we could do this in a fairly straight-forward manner by adding a report_facts setting that would cause the Puppet Agent to load and submit a new factset to `/puppet/v3/facts` at the end of the run before the report is sent to `/puppet/v3/report`. This setting should default to false as the additional fact submission would represent a 33% increase in the datasets sent to PuppetDB on each run from facts, catalog, report to facts, catalog, facts, report. For most folks, PuppetDB will handle that just fine — but at large scale it is a big enough increase that has to be enabled deliberately. We could follow the naive implementation above with something more sophisticated that computes the diff between facts loaded at the beginning of the run and those loaded at the end and attaches it to the report. That would be much more efficient as both the amount of data and number of processing operations would be reduced. However, this would require changes to the report format and PuppetDB and thus is a much bigger lift to accomplish.

Add Comment

[Greg Dubicki (JIRA)]

Greg Dubicki commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Big +1 for Charlie Sharpsteen's proposal. I can try to create a PR for this if this idea is accepted.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5934

Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Team: Agent Coremunity

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Greg Dubicki that'd be awesome!

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

PR up with an implementation: https://github.com/puppetlabs/puppet/pull/7666 Targeted Puppet 5.5.z as this feels like something that will be useful to folks on PE 2018.1.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen updated an issue

Puppet / PUP-5934

Updated fact values should be submitted after each Puppet run

Change By: Charlie Sharpsteen Fix Version/s: PUP 6.8.0 Fix Version/s: PUP 6.4.z Fix Version/s: PUP 5.5.z

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Fix Version/s: PUP 6.8.0

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Fix Version/s: PUP 6.4.z Fix Version/s: PUP 5.5.z Fix Version/s: PUP 6.11.0 Fix Version/s: PUP 6.4.5 Fix Version/s: PUP 5.5.18

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Release Notes Summary: Puppet submits facts when requesting a catalog, but if the agent modifies the system while applying the catalog, then the facts in puppetdb won't be refreshed until the agent runs again, which may be 30 minutes (or however runinterval is configured). This feature makes it possible to submit facts again at the end of the agent's run, after the catalog has been applied. To enable this feature, set "resubmit_facts=true" in the agent's puppet.conf. Note doing so will double the load on puppetdb, as each agent will submit facts twice per agent run. By default the feature is disabled. Release Notes: New Feature

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Release Notes Summary: Puppet submits facts when requesting a catalog, but if the agent modifies the system while applying the catalog, then the facts in puppetdb won't be refreshed until the agent runs again, which may be 30 minutes (or however runinterval is configured). This feature makes it possible to submit facts again at the end of the agent's run, after the catalog has been applied. To enable this feature, set "resubmit_facts=true" in the agent's puppet.conf. Note doing so will double the fact submission load on puppetdb, as since each agent will submit facts twice per agent run. By default the feature is disabled.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Sprint: Platform Core KANBAN

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper assigned an issue to Josh Cooper

Puppet / PUP-5934 Updated fact values should be submitted after each Puppet run

Change By: Josh Cooper Assignee: Josh Cooper

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Merged to master in https://github.com/puppetlabs/puppet/commit/4f448a3e3149b9d0ce7fae6d4eac0fb5d6cb0074. This will be backported to 5.5.x in a future PR.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run

Backported to 5.5.x in https://github.com/puppetlabs/puppet/commit/3e6abbc836f69fcd08289c89bb765d02e5519f18. It will be released in 5.5.18, 6.4.5, 6.11.0

Add Comment

[Heston Hoffman (JIRA)]

Heston Hoffman updated an issue

Puppet / PUP-5934

Updated fact values should be submitted after each Puppet run

Change By: Heston Hoffman Labels: resolved-issue-added

Add Comment

[Adam Buxton (JIRA)]

Adam Buxton commented on PUP-5934

Re: Updated fact values should be submitted after each Puppet run Charlie Sharpsteen add it to the quiver of useful `PS made this` https://forge.puppet.com/laura/puppet_agent_settings

Add Comment