Jira (PUP-10532) Puppet fetches metadata for files with "ensure => absent"

0 views
Skip to first unread message

Nate McCurdy (Jira)

unread,
May 22, 2020, 2:48:04 PM5/22/20
to puppe...@googlegroups.com
Nate McCurdy created an issue
 
Puppet / Bug PUP-10532
Puppet fetches metadata for files with "ensure => absent"
Issue Type: Bug Bug
Affects Versions: PUP 6.15.0, PUP 5.5.14
Assignee: Unassigned
Created: 2020/05/22 11:47 AM
Priority: Normal Normal
Reporter: Nate McCurdy
  1. Problem
    Puppet Version: 6.15.0
    Puppet Server Version: 6.11.1
    OS Name/Version: CentOS 7

Puppet makes a HTTP request for /file_metadata/ for files that are set to "ensure => absent" and have source => 'puppet:///modules/....

This defeats the purpose of using static catalogs when your intent is to reduce the amount of file_metadata HTTP traffic. It causes unnecessary traffic and overheard on the Puppet server.

  1. Reproduction
    Given this Puppet code: 

    node 'agent1.vagrant' {
    		 
     
    		 
      file { '/root/one':
    		 
        ensure => file,
    		 
        owner  => 0,
    		 
        group  => 0,
    		 
        mode   => '0644',
    		 
        source => 'puppet:///modules/test/one',
    		 
      }
    		 
     
    		 
      file { '/root/two':
    		 
        ensure => absent,
    		 
        owner  => 0,
    		 
        group  => 0,
    		 
        mode   => '0644',
    		 
        source => 'puppet:///modules/test/two',
    		 
      }
    		 
     
    		 
    }

Running puppet agent -t from agent1.vagrant causes these messages in Puppetserver's logs:

==> /var/log/puppetlabs/puppetserver/puppetserver.log <==
 
2020-05-22T18:44:27.159Z INFO  [qtp9036388-38] [puppetserver] Puppet Inlined resource metadata into static catalog for agent1.vagrant in environment production in 0.01 seconds
 
2020-05-22T18:44:27.160Z INFO  [qtp9036388-38] [puppetserver] Puppet Compiled static catalog for agent1.vagrant in environment production in 0.12 seconds
 
 
 
==> /var/log/puppetlabs/puppetserver/puppetserver-access.log <==
 
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/node/agent1.vagrant?environment=production&configured_environment=production&transaction_uuid=ee221fd5-13a0-4fdd-9413-1b57a98cd690 HTTP/1.1" 200 12572 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 33 - 30
 
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/file_metadatas/pluginfacts?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=md5&source_permissions=use&environment=production HTTP/1.1" 200 240 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 17 - 12
 
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/file_metadatas/plugins?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 244 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 19 - 14
 
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/file_metadatas/locales?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&ignore=%2A.pot&ignore=config.yaml&links=follow&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 244 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 76 - 72
 
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "POST /puppet/v3/catalog/agent1.vagrant?environment=production HTTP/1.1" 200 1864 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 272 25743 268
 
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "GET /puppet/v3/file_metadata/modules/test/two?links=manage&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 258 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 17 - 13
 
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "PUT /puppet/v3/report/agent1.vagrant?environment=production HTTP/1.1" 200 9 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 47 7134 43

  1. Desired Behavior

A file resource with both ensure => absent and source => <valid_puppet_uri> should not result in an agent requesting its metadata.

For example, I expect this resource to not cause a lookup of the file's metadata.

file { '/root/two':
 
  ensure => absent,
 
  owner  => 0,
 
  group  => 0,
 
  mode   => '0644',
 
  source => 'puppet:///modules/test/two',
 
}

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Nate McCurdy (Jira)

unread,
May 22, 2020, 2:50:03 PM5/22/20
to puppe...@googlegroups.com
Nate McCurdy updated an issue
Change By: Nate McCurdy
# h1. Problem
*Puppet Version:* 6.15.0
*Puppet Server Version:* 6.11.1
*OS Name/Version:* CentOS 7


Puppet makes a HTTP request for {{/file_metadata/}} for files that are set to "{{ensure => absent}}" and have {{source => 'puppet:///modules/...}}.

This defeats the purpose of using static catalogs when your intent is to reduce the amount of file_metadata HTTP traffic. It causes unnecessary traffic and overheard on the Puppet server.

# h1. Reproduction
Given this Puppet code:
{code}

node 'agent1.vagrant' {

  file { '/root/one':
    ensure => file,
    owner  => 0,
    group  => 0,
    mode   => '0644',
    source => 'puppet:///modules/test/one',
  }

  file { '/root/two':
    ensure => absent,
    owner  => 0,
    group  => 0,
    mode   => '0644',
    source => 'puppet:///modules/test/two',
  }

}
{code}


Running {{puppet agent -t}} from {{agent1.vagrant}} causes these messages in Puppetserver's logs:

 {noformat}

==> /var/log/puppetlabs/puppetserver/puppetserver.log <==
2020-05-22T18:44:27.159Z INFO  [qtp9036388-38] [puppetserver] Puppet Inlined resource metadata into static catalog for agent1.vagrant in environment production in 0.01 seconds
2020-05-22T18:44:27.160Z INFO  [qtp9036388-38] [puppetserver] Puppet Compiled static catalog for agent1.vagrant in environment production in 0.12 seconds

==> /var/log/puppetlabs/puppetserver/puppetserver-access.log <==
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/node/agent1.vagrant?environment=production&configured_environment=production&transaction_uuid=ee221fd5-13a0-4fdd-9413-1b57a98cd690 HTTP/1.1" 200 12572 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 33 - 30
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/file_metadatas/pluginfacts?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=md5&source_permissions=use&environment=production HTTP/1.1" 200 240 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 17 - 12
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/file_metadatas/plugins?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 244 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 19 - 14
10.20.1.6 - - [22/May/2020:18:44:26 +0000] "GET /puppet/v3/file_metadatas/locales?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&ignore=%2A.pot&ignore=config.yaml&links=follow&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 244 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 76 - 72
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "POST /puppet/v3/catalog/agent1.vagrant?environment=production HTTP/1.1" 200 1864 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 272 25743 268
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "GET /puppet/v3/file_metadata/modules/test/two?links=manage&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 258 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 17 - 13
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "PUT /puppet/v3/report/agent1.vagrant?environment=production HTTP/1.1" 200 9 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 47 7134 43
{noformat}

 # h1. Desired Behavior


A file resource with both {{ensure => absent}} and {{source => <valid_puppet_uri>}} should not result in an agent requesting its metadata.

For example, I expect this resource to not cause a lookup of the file's metadata.

{code}

file { '/root/two':
  ensure => absent,
  owner  => 0,
  group  => 0,
  mode   => '0644',
  source => 'puppet:///modules/test/two',
}
{code}

Josh Cooper (Jira)

unread,
May 22, 2020, 3:37:04 PM5/22/20
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Epic Link: PUP-10533

Nate McCurdy (Jira)

unread,
May 26, 2020, 1:21:04 PM5/26/20
to puppe...@googlegroups.com
Nate McCurdy updated an issue
Change By: Nate McCurdy
h1.  Problem

*Puppet Version:* 6.15.0
*Puppet Server Version:* 6.11.1
*OS Name/Version:* CentOS 7

Puppet makes a HTTP request for {{/file_metadata/}} for files that are set to "{{ensure => absent}}" and have {{source => 'puppet:///modules/...}}.

This defeats the purpose of using static catalogs when your intent is to reduce the amount of file_metadata HTTP traffic. It causes unnecessary traffic and overheard on the Puppet server.
The problem is this line showing the {{GET}} of file_metadata for the "{{two"}} file, which I would expect not to see:
{noformat}
10.20.1.6 - - [22/May/2020:18:44:27 +0000] "GET /puppet/v3/file_metadata/modules/test/two?links=manage&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 258 "-" "Puppet/6.15.0 Ruby/2.5.8-p224 (x86_64-linux)" 17 - 13 {noformat}
h1. Desired Behavior

A file resource with both {{ensure => absent}} and {{source => <valid_puppet_uri>}} should not result in an agent requesting its metadata.

For example, I expect this resource to not cause a lookup of the file's metadata.
{code}
file { '/root/two':
  ensure => absent,
  owner  => 0,
  group  => 0,
  mode   => '0644',
  source => 'puppet:///modules/test/two',
}
{code}

Nate McCurdy (Jira)

unread,
May 27, 2020, 4:48:03 PM5/27/20
to puppe...@googlegroups.com
Nate McCurdy updated an issue
Change By: Nate McCurdy
Method Found: Needs Assessment Customer Feedback

Charlie Sharpsteen (Jira)

unread,
Jul 1, 2020, 12:51:03 PM7/1/20
to puppe...@googlegroups.com
Charlie Sharpsteen updated an issue
Change By: Charlie Sharpsteen
CS Priority: Needs Priority

Jarret Lavallee (Jira)

unread,
Jul 2, 2020, 7:45:04 PM7/2/20
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
Change By: Jarret Lavallee
CS Priority: Needs Priority Reviewed

Jarret Lavallee (Jira)

unread,
Jul 2, 2020, 7:45:04 PM7/2/20
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
Change By: Jarret Lavallee
Team: Coremunity

Josh Cooper (Jira)

unread,
Oct 19, 2022, 11:11:01 PM10/19/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Labels: Easy
This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages