Jira (PUP-6708) Agent does not save local copy of last_run_report.yaml if submission to report server fails

[Moses Mendoza (JIRA)]

Moses Mendoza created an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails Issue Type: Bug Affects Versions: PUP 4.6.2 Assignee: Unassigned Created: 2016/09/15 2:40 PM Priority: Normal Reporter: Moses Mendoza On an Ubuntu 16.04 VM running Puppet 1.6.2 agent: The agent does not appear to save a run report if it fails to submit the report to its report server. root@ubuntu16:~# puppet agent -t --server=foo:1 --report_server=bar:2 --http_connect_timeout=3s --http_debug opening connection to foo:1:8140... Warning: Unable to fetch my node definition, but the agent run will continue: Warning: execution expired Info: Retrieving pluginfacts opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': execution expired opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: execution expired Info: Retrieving plugin opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': execution expired opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: execution expired Info: Loading facts opening connection to foo:1:8140... Error: Could not retrieve catalog from remote server: execution expired Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run opening connection to bar:2:8140... Error: Could not send report: execution expired Afterward, no report is saved on the agent: oot@ubuntu16:~# puppet config print reportdir /opt/puppetlabs/puppet/cache/reports root@ubuntu16:~# ls /opt/puppetlabs/puppet/cache/ clientbucket client_data client_yaml facts.d lib preview state root@ubuntu16:~# ls /opt/puppetlabs/puppet/cache/state/ classes.txt graphs last_run_summary.yaml resources.txt state.yaml transactionstore.yaml root@ubuntu16:~# find /opt/puppetlabs/ -name "last_run_*" /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml root@ubuntu16:~# We do, however, appear to still save last_run_summary.yaml. The contents of it after this run were: root@ubuntu16:~# cat /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml --- version: config: puppet: 4.6.2 time: last_run: 1473975435 root@ubuntu16:~# Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Moses Mendoza (JIRA)]

Moses Mendoza updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Moses Mendoza

On an Ubuntu 16.04 VM running Puppet 1.6.2 agent: The agent does not appear to save a run report if it fails to submit the report to its report server.

*Repro* {code}

root@ubuntu16:~# puppet agent -t --server=foo:1 --report_server=bar:2 --http_connect_timeout=3s --http_debug opening connection to foo:1:8140... Warning: Unable to fetch my node definition, but the agent run will continue: Warning: execution expired Info: Retrieving pluginfacts opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': execution expired opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: execution expired Info: Retrieving plugin opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': execution expired opening connection to foo:1:8140... Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: execution expired Info: Loading facts opening connection to foo:1:8140... Error: Could not retrieve catalog from remote server: execution expired Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run opening connection to bar:2:8140... Error: Could not send report: execution expired

{code} # Afterward, no report is saved on the agent: {code}

oot@ubuntu16:~# puppet config print reportdir /opt/puppetlabs/puppet/cache/reports

root@ubuntu16:~# ls /opt/puppetlabs/puppet/cache/ clientbucket  client_data  client_yaml  facts.d  lib  preview  state root@ubuntu16:~# ls /opt/puppetlabs/puppet/cache/state/ classes.txt  graphs  last_run_summary.yaml  resources.txt  state.yaml  transactionstore.yaml root@ubuntu16:~# find /opt/puppetlabs/ -name "last_run_*" /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml root@ubuntu16:~#

{code} We do, however, appear to still save {{last_run_summary.yaml}}. The contents of it after this run were: {code}

root@ubuntu16:~# cat /opt/puppetlabs/puppet/cache/state/last_run_summary.yaml --- version:   config:   puppet: 4.6.2 time:   last_run: 1473975435 root@ubuntu16:~#

{code}

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Story Points: 3

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP  Triage  2016-10-19

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP  2016-10-19  Hold

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2016- 10 11 - 19 Holding Area 02

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2016-11-02  Holding

Add Comment

[Geoff Nichols (JIRA)]

[Geoff Nichols (JIRA)]

[Moses Mendoza (JIRA)]

Moses Mendoza commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails After review on https://github.com/puppetlabs/puppet/pull/5283 it was determined that the initial PR for this was incomplete and needs to be reworked a bit. The original estimate of 3 is probably still accurate.

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP  Holding  2016-11-30

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2016- 11 12 - 30 14

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2016-12- 14 28

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP  2016  2017 - 12 01 - 28 11

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2017-01- 11 25

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2017- 01 02 - 25 08

Add Comment

[Kevin Corcoran (JIRA)]

Kevin Corcoran commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails Moses Mendoza said that he thinks this is the cause of intermittent failures of ruby/puppet/acceptance/tests/reports.failover_master.rb in puppet server CI.

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails

well - I think the root cause of those failures is network issues, but the test ultimately fails because we can't read the report, because the report wasn't written because the report server was unreachable

Add Comment

[Kevin Corcoran (JIRA)]

Kevin Corcoran updated an issue

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Kevin Corcoran Issue Type: Bug CI Blocker

Add Comment

[Kevin Corcoran (JIRA)]

Kevin Corcoran commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails If that was the cause, I would expect other tests to be failing in a similar way, but this is the only test I recall failing recently in that pipeline. But you know better than I do.

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2017-02- 08 22

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2017- 02 03 - 22 08

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: AP 2017-03- 08 22

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: Agent  2017-03-22  Ready for Engineering

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails Moses Mendoza should we perhaps convert this issue to a bug or is it still blocking CI?

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails

yeah just a bug

Add Comment

[Kenn Hussey (JIRA)]

Kenn Hussey updated an issue

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Kenn Hussey Issue Type: CI Blocker Bug

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Geoff Nichols Sprint: Agent Ready for Engineering  1  0

Add Comment

[Maggie Dreyer (JIRA)]

Maggie Dreyer updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Maggie Dreyer Labels: triaged

Add Comment

[Jeremy Barlow (JIRA)]

Jeremy Barlow commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails This one came up again in a recent puppetserver acceptance test run, here: 20:34:20 * master_field should not appear when no master could be conatacted 20:34:20 20:34:20 rus6ehbzrplkj51.delivery.puppetlabs.net (redhat6-32-2) 03:34:20$ puppet agent -t --config /tmp/puppet_conf_test.vM9ww0 --server_list=badmaster:1 --http_connect_timeout=5s --report_server=f9sy80hlxm12sz1.delivery.puppetlabs.net 20:34:20 Warning: Could not select a functional puppet master 20:34:22 Warning: Unable to fetch my node definition, but the agent run will continue: 20:34:27 Warning: execution expired 20:34:27 Info: Retrieving pluginfacts 20:34:27 Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': execution expired 20:34:32 Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: execution expired 20:34:37 Info: Retrieving plugin 20:34:37 Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': execution expired 20:34:42 Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: execution expired 20:34:47 Error: Could not retrieve catalog from remote server: execution expired 20:34:52 Warning: Not using cache on failed catalog 20:34:52 Error: Could not retrieve catalog; skipping run 20:34:52 Error: Could not send report: execution expired 20:34:57 20:34:57 rus6ehbzrplkj51.delivery.puppetlabs.net (redhat6-32-2) executed in 36.96 seconds 20:34:57 Exited: 1 20:34:57 20:34:57 rus6ehbzrplkj51.delivery.puppetlabs.net (redhat6-32-2) 03:34:57$ puppet agent --configprint node_name_value 20:34:57 rus6ehbzrplkj51.delivery.puppetlabs.net 20:34:58 20:34:58 rus6ehbzrplkj51.delivery.puppetlabs.net (redhat6-32-2) executed in 0.85 seconds 20:34:58 20:34:58 f9sy80hlxm12sz1.delivery.puppetlabs.net (redhat6-64-1) 03:34:58$ cat /tmp/report_dir.7PPKSO/rus6ehbzrplkj51.delivery.puppetlabs.net/* 20:34:58 cat: /tmp/report_dir.7PPKSO/rus6ehbzrplkj51.delivery.puppetlabs.net/*: No such file or directory In that case, I didn't see any evidence from the puppetserver.log or puppetserver-access.log file for the target --report-server that it had actually received a report request from the agent – even though the agent appeared to hit the "execution expired" error for the report right after an expiration for the catalog request, both at 20:34:52. The report server was apparently running fine at the time the agent encountered the error, though.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails

A related issue is that puppet doesn't even attempt to send the report (and doesn't cache one locally) if it exhausts the server_list. We should at least store the report locally. There's another ticket about trying to send the report later.

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails

This issue exists the agent uses the indirector to save to the report to the rest terminus and the yaml cache terminus. If the main terminus raises, then the cache terminus' save method is never called https://github.com/puppetlabs/puppet/blob/d567575ba8c5b2c903044b80b0adaab176c8da5d/lib/puppet/indirector/indirection.rb#L316-L319 The agent could decouple those by saving to the main terminus using ignore_cache_terminus: true and saving to the cache using ignore_terminus: true in an ensure block.

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Josh Cooper (Jira)]

Josh Cooper assigned an issue to Josh Cooper

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Josh Cooper Assignee: Josh Cooper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Josh Cooper Fix Version/s: PUP 7.11.0 Fix Version/s: PUP 6.25.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-6708 Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Josh Cooper Sprint: Coremunity Kanban

Add Comment

[Gabriel Nagy (Jira)]

Gabriel Nagy commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails Merged to 6.x in https://github.com/puppetlabs/puppet/commit/6b109de27fe2861e1b21e9e36f708de387c2ac6c

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Josh Cooper Release Notes: Bug Fix Release Notes Summary: The agent will now always save a local copy of its last run report, even if it fails to submit the report to the Puppet server.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-6708

Re: Agent does not save local copy of last_run_report.yaml if submission to report server fails Passed CI in 57fd4e4c4e

Add Comment

[Claire Cadman (Jira)]

Claire Cadman updated an issue

Puppet / PUP-6708

Agent does not save local copy of last_run_report.yaml if submission to report server fails

Change By: Claire Cadman Labels: doc-reviewed

Add Comment