Jira (PUP-11654) Make unchanged events in report optional

[Jonathan Newman (Jira)]

Jonathan Newman created an issue

Puppet / PUP-11654 Make unchanged events in report optional Issue Type: New Feature Assignee: Unassigned Created: 2022/10/13 3:05 PM Priority: Normal Reporter: Jonathan Newman In order to reduce potentially unnecessary network traffic and data that may be unused, Puppet should optionally allow the unchanged resource events to not be serialized when the reports are generated: https://github.com/puppetlabs/puppet/blob/948a4bec67def5244595a6f69976a9a2479f1df1/lib/puppet/transaction/report.rb#L326 Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

[Jonathan Newman (Jira)]

Jonathan Newman commented on PUP-11654

Re: Make unchanged events in report optional Micah Wilson David Piekny Josh Cooper FYI.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11654

Make unchanged events in report optional

Change By: Josh Cooper Fix Version/s: PUP 7.21.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Josh Cooper Team: Phoenix

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11654 Make unchanged events in report optional Change By: Josh Cooper

In order to reduce potentially unnecessary network traffic and data that may be unused, Puppet should optionally allow the unchanged resource statuses and their events to not be serialized when the reports are generated: https://github.com/puppetlabs/puppet/blob/948a4bec67def5244595a6f69976a9a2479f1df1/lib/puppet/transaction/report.rb#L326 *Details* The report contains an array of resource statuses {{Puppet::Resource::Status}} The status contains the result of managing a single resource {{Puppet::Type::<name>}} There may be 0 or more events {{Puppet::Transaction::Event}} per resource status. Each event captures the current and desired values for each property, e.g. if puppet changes the {{owner}} property for the {{file}} type, then the event will contain the old and new values. Here's an example entry from the report in YAML: {code:yaml}   File[/tmp/file]:     title: "/tmp/file"     file: "/etc/puppetlabs/code/environments/production/manifests/site.pp"     line: 2     resource: File[/tmp/file]     resource_type: File     provider_used: posix     containment_path:     - Stage[main]     - Main     - File[/tmp/file]     evaluation_time: 0.002537417     tags:     - file     - class     time: '2022-10-18T22:49:58.301897873+00:00'     failed: false     failed_to_restart: false     changed: true     out_of_sync: true     skipped: false     change_count: 2     out_of_sync_count: 2     events:     - audited: false       property: owner       previous_value: 52       desired_value: 0       historical_value:       message: owner changed 'puppet' to 'root'       name: owner_changed       status: success       time: '2022-10-18T22:49:58.302133825+00:00'       redacted:       corrective_change: false     - audited: false       property: group       previous_value: 52       desired_value: 0       historical_value:       message: group changed 'puppet' to 'root'       name: group_changed       status: success       time: '2022-10-18T22:49:58.303239684+00:00'       redacted:       corrective_change: false     corrective_change: false {code} When the same file is "insync", then the report contains: {code:yaml}   File[/tmp/file]:     title: "/tmp/file"     file: "/etc/puppetlabs/code/environments/production/manifests/site.pp"     line: 2     resource: File[/tmp/file]     resource_type: File     provider_used: posix     containment_path:     - Stage[main]     - Main     - File[/tmp/file]     evaluation_time: 0.000237233     tags:     - file     - class     time: '2022-10-18T22:57:32.999773271+00:00'     failed: false     failed_to_restart: false     changed: false     out_of_sync: false     skipped: false     change_count: 0     out_of_sync_count: 0     events: []     corrective_change: false {code} I believe  {{Puppet::Resource::Status#out_of_sync?}} method can be used to filter which statuses to include/exclude. If it results {{false}}, then no changes were necessary. For this ticket: 1. Create a boolean puppet setting to exclude unchanged resources statuses and their events. For 7.x, it should default to the current behavior of including all statuses. 2. Update the {{Report#to_data_hash}} method to exclude the status object based on "out_of_sync?" One complication is some fields in the report are derived from resource statuses and events. For example, the [{{corrective_change}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L434-L436] and [{{noop_pending}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L220] instance variables are computed in the {{finalize_report}} method. If a report is deserialized using {{Report.from_data_report}} and then re-serialized using {{Report#to_data_hash}}, it's important that the original {{corrective_change}} and {{noop_pending}} data be preserved. I believe it will work that way, but it's something to check. Another complication is the puppetdb report processor determines if a report to be "noop" based on events. https://github.com/puppetlabs/puppetdb/blob/7b7ecbc975e2470a76f3c8a0b4dfb386dd24d54a/puppet/lib/puppet/reports/puppetdb.rb#L46. So we may need to include unchanged events that were explicitly marked as "noop"

Add Comment

[Josh Cooper (Jira)]

I believe the {{Puppet::Resource::Status#out_of_sync?}} method can be used to filter which statuses to include/exclude. If it results {{{}false{}}}, then no changes were necessary.

For this ticket: 1. Create a boolean puppet setting to exclude unchanged resources statuses and their events. For 7.x, it should default to the current behavior of including all statuses. 2. Update the {{Report#to_data_hash}} method to exclude the status object based on "out_of_sync?" One complication is some fields in the report are derived from resource statuses and events. For example, the [{{corrective_change}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L434-L436] and [{{noop_pending}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L220] instance variables are computed in the {{finalize_report}} method. If a report is deserialized using {{Report.from_data_report}} and then re-serialized using {{

{} Report#to_data_hash { }} } , it's important that the original {{corrective_change}} and {{noop_pending}} data be preserved. I believe it will work that way, but it's something to check. Another complication is the puppetdb report processor determines if a report to be "noop" [based on events| [ https://github.com/puppetlabs/puppetdb/blob/7b7ecbc975e2470a76f3c8a0b4dfb386dd24d54a/puppet/lib/puppet/reports/puppetdb.rb#L46] {*}] . {*}  So we may need to include unchanged events that were explicitly marked as "noop".

Add Comment

[Josh Cooper (Jira)]

, it's important that the original {{corrective_change}} and {{noop_pending}} data be preserved. I believe it will work that way, but it's something to check. Another complication is the puppetdb report processor determines if a report to be "noop" [ based on events . |[ https://github.com/puppetlabs/puppetdb/blob/7b7ecbc975e2470a76f3c8a0b4dfb386dd24d54a/puppet/lib/puppet/reports/puppetdb.rb#L46 ]{*}] . {*}  So we may need to include unchanged events that were explicitly marked as "noop" .

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Josh Cooper Labels: needs-validation

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11654 Make unchanged events in report optional Change By: Josh Cooper

I believe the {{Puppet::Resource::Status#out_of_sync?}} method can be used to filter which statuses to include/exclude. If it results returns {{{}false{}}}, then no changes were necessary.

For this ticket: 1. Create a boolean puppet setting to exclude unchanged resources statuses and their events. For 7.x, it should default to the current behavior of including all statuses. 2. Update the {{Report#to_data_hash}} method to exclude the status object based on "out_of_sync?" One complication is some fields in the report are derived from resource statuses and events. For example, the [{{corrective_change}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L434-L436] and [{{noop_pending}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L220] instance variables are computed in the {{finalize_report}} method. If a report is deserialized using {{Report.from_data_report}} and then re-serialized using {{

{} Report#to_data_hash { }} } , it's important that the original {{corrective_change}} and {{noop_pending}} data be preserved. I believe it will work that way, but it's something to check. Another complication is the puppetdb report processor determines if a report to be "noop" [based on events|https://github.com/puppetlabs/puppetdb/blob/7b7ecbc975e2470a76f3c8a0b4dfb386dd24d54a/puppet/lib/puppet/reports/puppetdb.rb#L46]. So we may need to include unchanged events that were explicitly marked as "noop".

Add Comment

[Jonathan Newman (Jira)]

Jonathan Newman updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Jonathan Newman Epic Link: PE-34575

Add Comment

[Morgan Rhodes (Jira)]

Morgan Rhodes updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Morgan Rhodes Story Points: 2

Add Comment

[Morgan Rhodes (Jira)]

Morgan Rhodes updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Morgan Rhodes Sprint: Phoenix 2022-11-09

Add Comment

[Morgan Rhodes (Jira)]

Morgan Rhodes updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Morgan Rhodes Labels: needs-validation

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Josh Cooper

I believe the {{Puppet::Resource::Status#out_of_sync?}} method can be used to filter which statuses to include/exclude. If it returns {{{}false{}}}, then no changes were necessary.

For this ticket: 1. Create a boolean puppet setting to exclude unchanged resources statuses and their events. For 7.x, it should default to the current behavior of including all statuses. 2. Update the {{Report#to_data_hash}} method to exclude the status object based on "out_of_sync?"

One complication is some fields in This should satisfy the report are derived from resource statuses and events happy path . For example, the [{{corrective_change}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L434 A follow up ticket has been created to handle noop - L436] and [{{noop_pending}}|https://github.com/puppetlabs/puppet/blob/eb498be4bf7b4dbb326dbd479d01101f0b226abb/lib/puppet/transaction/report.rb#L220] instance variables are computed ness in the {{finalize_report}} method. If a report is deserialized using {{Report.from_data_report}} and then re PUP - serialized using {{{}Report#to_data_hash{}}}, it's important that the original {{corrective_change}} and {{noop_pending}} data be preserved. I believe it will work that way, but it's something to check. Another complication is the puppetdb report processor determines if a report to be "noop" [based on events|https://github.com/puppetlabs/puppetdb/blob/7b7ecbc975e2470a76f3c8a0b4dfb386dd24d54a/puppet/lib/puppet/reports/puppetdb.rb#L46]. So we may need to include unchanged events that were explicitly marked as "noop". 11661

Add Comment

[Tony Vu (Jira)]

Tony Vu assigned an issue to Tony Vu

Puppet / PUP-11654 Make unchanged events in report optional

Change By: Tony Vu Assignee: Tony Vu

Add Comment