Jira (PUP-10017) Collapse dependency failures into Class level failures

0 views
Skip to first unread message

Adrian Parreiras Horta (JIRA)

unread,
Sep 11, 2019, 12:47:03 PM9/11/19
to puppe...@googlegroups.com
Adrian Parreiras Horta created an issue
 
Puppet / Improvement PUP-10017
Collapse dependency failures into Class level failures
Issue Type: Improvement Improvement
Affects Versions: PUP 6.7.2
Assignee: Unassigned
Components: Catalog Application
Created: 2019/09/11 9:46 AM
Priority: Normal Normal
Reporter: Adrian Parreiras Horta

When a Class has a failed dependency, every individual resource in the class generates a notice level message about the dependency failure and a warning level message about skipping the resource, as well as a skip event in the report.  This can generate a large number of messages and a large report if there are many dependencies, or multiple layers of dependencies.

If possible, all of the skipped resources should be collapsed into the containing class and the messages and skip event should be done at the class level.

It looks like the most recent work on this was done in PUP-3930 and commit 32e2bf2.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Thomas Kishel (JIRA)

unread,
Sep 11, 2019, 1:45:03 PM9/11/19
to puppe...@googlegroups.com
Thomas Kishel commented on Improvement PUP-10017
 
Re: Collapse dependency failures into Class level failures

Given one class that requires another ...

class xxx {
  file { '/impossible/xxx.txt':
    content => 'hello',
  }
}
 
class www {
  require xxx
  file { '/tmp/yyy.txt':
    content => 'hello',
  }
  file { '/tmp/zzz.txt':
    content => 'hello',
  }
}
 
node 'agent.puppetdebug.vlan' {
  include www
}

When one class requires another class ...

Instead of reporting a dependency failure for each resource in the requiring because of a failure for a resource in the required class ...

[root@agent ~]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for agent.puppetdebug.vlan
Info: Applying configuration version '1568222821'
<snip>
Error: /Stage[main]/Xxx/File[/impossible/xxx.txt]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory @ dir_s_mkdir - /impossible/xxx.txt20190911-12082-10konwl.lock (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 2)
Notice: /Stage[main]/Www/File[/tmp/yyy.txt]: Dependency File[/impossible/xxx.txt] has failures: true
Warning: /Stage[main]/Www/File[/tmp/yyy.txt]: Skipping because of failed dependencies
Warning: /Stage[main]/Www/File[/tmp/zzz.txt]: Skipping because of failed dependencies
Notice: Applied catalog in 0.04 seconds

Collapse the duplicate dependency failures for each resource in the requiring class into one dependency failure in the required class ...

[root@agent ~]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for agent.puppetdebug.vlan
Info: Applying configuration version '1568222821'
<snip>
Error: /Stage[main]/Xxx/File[/impossible/xxx.txt]/ensure: change from 'absent' to 'file' failed: Could not set 'file' on ensure: No such file or directory @ dir_s_mkdir - /impossible/xxx.txt20190911-12082-10konwl.lock (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 2)
Notice: /Stage[main]/Www: Dependency Class[/Stage[main]/Xxx] has failures: true
Notice: Applied catalog in 0.04 seconds

This functionality should be configurable, and default to false.

Jarret Lavallee (JIRA)

unread,
Sep 11, 2019, 1:48:02 PM9/11/19
to puppe...@googlegroups.com

Jarret Lavallee (JIRA)

unread,
Sep 12, 2019, 7:41:03 PM9/12/19
to puppe...@googlegroups.com

Jarret Lavallee (JIRA)

unread,
Sep 12, 2019, 7:41:04 PM9/12/19
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
Change By: Jarret Lavallee
CS Priority: Needs Priority Reviewed

Thomas Kishel (JIRA)

unread,
Sep 13, 2019, 12:40:03 PM9/13/19
to puppe...@googlegroups.com
 
Re: Collapse dependency failures into Class level failures

Implementing this should also address the high volume of skips associated with resources using the schedule metaparameter when the run is outside the schedule window.

Thomas Kishel (JIRA)

unread,
Sep 13, 2019, 1:00:16 PM9/13/19
to puppe...@googlegroups.com

A (let's say) suppress_inherited_dependency_failures option would affect the logic here:

https://github.com/puppetlabs/puppet/blob/5.5.x/lib/puppet/transaction.rb#L285
https://github.com/puppetlabs/puppet/blob/5.5.x/lib/puppet/transaction.rb#L398

When suppress_inherited_dependency_failures is true ...

if a resource has a parent
  if the parent is a class
    if the class has a dependency relationship
      if the dependent is a failed resource (or the dependent is a class containing a failed resource)
        mark the class as failed (and mark the resource as failed)
          report (just once) that we are skipping the containing class of the resource          suppress the report that we are skipping the resource

Use case when enabled: given a failure in a resource that is required by a class, a user would see (one) class failure in the report, and could perform a puppet run without suppress_inherited_dependency_failures to identify the resources in the class that were skipped, if necessary. But since the solution to the dependency failure is to resolve the failure in the dependent resource, that would only be valuable for auditing everything that was skipped: the user's attention should be focused on the failed resource (in the example, File['/impossible/xxx.txt']).

Mihai Buzgau (JIRA)

unread,
Sep 16, 2019, 6:23:04 AM9/16/19
to puppe...@googlegroups.com

Mihai Buzgau (JIRA)

unread,
Sep 17, 2019, 4:51:03 AM9/17/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: PR NW - Triage 2019-10-02

Gheorghe Popescu (JIRA)

unread,
Sep 18, 2019, 3:44:03 AM9/18/19
to puppe...@googlegroups.com

Thomas Kishel (JIRA)

unread,
Sep 18, 2019, 6:40:04 PM9/18/19
to puppe...@googlegroups.com
Thomas Kishel updated an issue
Change By: Thomas Kishel
Attachment: with_coalesce.png

Thomas Kishel (JIRA)

unread,
Sep 18, 2019, 6:40:04 PM9/18/19
to puppe...@googlegroups.com
Thomas Kishel updated an issue
When a Class has a failed dependency, every individual resource in the class generates a notice level message about the dependency failure and a warning level message about skipping the resource, as well as a skip event in the report.  This can generate a large number of messages and a large report if there are many dependencies, or multiple layers of dependencies.

At large-node and/or large-code scale, one dependency failure can create an overwhelming number of warnings ... obscuring the root cause and increasing the use of system resources required to process and store the resulting reports.

If possible, all of the skipped resources should be collapsed into the containing class and the messages and skip event should be done at the class level.

It looks like the most recent work on this was done in PUP-3930 and commit 32e2bf2.

Thomas Kishel (JIRA)

unread,
Sep 18, 2019, 6:40:04 PM9/18/19
to puppe...@googlegroups.com
Thomas Kishel updated an issue
Change By: Thomas Kishel
Attachment: with_and_without_coalesce.png

Thomas Kishel (JIRA)

unread,
Sep 18, 2019, 6:40:05 PM9/18/19
to puppe...@googlegroups.com
Thomas Kishel updated an issue
Change By: Thomas Kishel
Attachment: without_coalesce.png

Thomas Kishel (JIRA)

unread,
Sep 18, 2019, 8:01:03 PM9/18/19
to puppe...@googlegroups.com

Thomas Kishel (JIRA)

unread,
Sep 18, 2019, 8:02:03 PM9/18/19
to puppe...@googlegroups.com

For a more extreme example:

class rrr {
  file { '/fail/file_in_rrr.txt':
    content => 'hello',
  }
}
 
class ddd {
  require rrr
  range(1,99).each |$n| {
    file { "/tmp/file_${n}_in_ddd.txt":
      content => 'hello',
    }
  }
}
 
node 'pe-201911-agent.puppetdebug.vlan' {
  file { '/fail/file.txt':
    content => 'hello',
  }
  file { '/tmp/file.txt':
    content => 'hello',
    require => File['/fail/file.txt']
  }
  include ddd
}

Mihai Buzgau (JIRA)

unread,
Sep 25, 2019, 3:52:04 AM9/25/19
to puppe...@googlegroups.com

Mihai Buzgau (JIRA)

unread,
Oct 2, 2019, 4:40:10 AM10/2/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: NW - 2019-10-02 , NW - 2019-10-16

Mihai Buzgau (JIRA)

unread,
Oct 16, 2019, 4:24:09 AM10/16/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: NW - 2019-10-02, NW - 2019-10-16 , NW - 2019-10-30

Mihai Buzgau (JIRA)

unread,
Oct 30, 2019, 5:00:13 AM10/30/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: NW - 2019-10-02, NW - 2019-10-16, NW - 2019-10-30 , NW - 2019-11-13

Mihai Buzgau (JIRA)

unread,
Nov 14, 2019, 5:28:08 AM11/14/19
to puppe...@googlegroups.com
Mihai Buzgau updated an issue
Change By: Mihai Buzgau
Sprint: NW - 2019-10-02, NW - 2019-10-16, NW - 2019-10-30, NW - 2019-11-13 , 2019-11-27

Josh Cooper (JIRA)

unread,
Nov 25, 2019, 4:44:08 PM11/25/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.12.0

Josh Cooper (JIRA)

unread,
Nov 25, 2019, 4:44:08 PM11/25/19
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Fix Version/s: PUP 6.4.5

Josh Cooper (JIRA)

unread,
Nov 25, 2019, 4:45:04 PM11/25/19
to puppe...@googlegroups.com
Josh Cooper commented on Improvement PUP-10017
 
Re: Collapse dependency failures into Class level failures

Passed 6.4.x CI in c08141b365b51bb248e94414e5ca31452371306c, and master in c89b3235935ec99ee5397c7691403361fc69167e

Josh Cooper (JIRA)

unread,
Nov 25, 2019, 4:46:04 PM11/25/19
to puppe...@googlegroups.com

Thomas Kishel (JIRA)

unread,
Nov 25, 2019, 5:38:05 PM11/25/19
to puppe...@googlegroups.com

I've added something ... feel free to improve or replace.

Thomas Kishel (JIRA)

unread,
Nov 25, 2019, 5:38:05 PM11/25/19
to puppe...@googlegroups.com
Thomas Kishel updated an issue
 
Change By: Thomas Kishel
Release Notes Summary: Merge dependency warnings

If a class has a failed dependency, every individual resource in that class generates a `notice` level message about the dependency failure, and a `warning` level message about skipping the resource. At large-node and/or large-code scale, one dependency failure can create an overwhelming number of warnings ... obscuring the root cause and increasing the use of system resources required to process and store the resulting reports.

To collapse all messages caused by one class dependency failure into one message associated with the class, set `merge_dependency_warnings=true`.
Release Notes: New Feature

Maximilian Philipps (JIRA)

unread,
Jan 15, 2020, 3:42:04 AM1/15/20
to puppe...@googlegroups.com
Maximilian Philipps commented on Improvement PUP-10017
 
Re: Collapse dependency failures into Class level failures

From reading the release notes I hoped this would help us. However after renaming the apache package resource in our test environment I still get almost 7000 lines of Skipping because of failed dependencies and just one Skipping resources in class because of failed class dependencies. Is this supposed to only work on class dependencies and not on individual resources? We often tag resources and then build dependencies between them using resource collectors.

Reply all
Reply to author
Forward
0 new messages