Jira (PUP-11774) Adding lots of hosts on Windows results in "permission denied" errors for some of them

0 views
Skip to first unread message

Dirk Heinrichs (Jira)

unread,
Mar 3, 2023, 2:30:03 AM3/3/23
to puppe...@googlegroups.com
Dirk Heinrichs created an issue
 
Puppet / Bug PUP-11774
Adding lots of hosts on Windows results in "permission denied" errors for some of them
Issue Type: Bug Bug
Affects Versions: PUP 7.21.0
Assignee: Unassigned
Components: Windows
Created: 2023/03/02 11:29 PM
Priority: Major Major
Reporter: Dirk Heinrichs

Puppet Version: 7.21.0
Puppet Server Version: latest
OS Name/Version: Windows Server 2019+

When managing a lot of "hosts" resources on Windows (by means of collecting exported resources), the first ones are applied without problems, but later ones fail with

Error: Puppet::Util::FileType::FileTypeFlat could not write C:/Windows/System32/drivers/etc/hosts: Permission denied @ rb_sysopen - C:/Windows/System32/drivers/etc/hosts

which looks like some kind of race condition to me.

Steps to reproduce:

Apply the following class on a new Windows host when Puppet is already managing at least 20 or 30 other hosts, all of which have the same class applied: 

class hosts {
  # Let each host create a host entry as exported resource
  $nic = $facts['networking']['primary']
  $ip = $facts['networking']['interfaces'][$nic]['ip']
 
  @@host { $facts['networking']['fqdn']:
    comment      => 'Host entry generated by Puppet',
    host_aliases => [$facts['networking']['hostname']],
    ip           => $ip,
    tag          => ['HOST_RECORD'],
  }
 
  # Collect all exported host records and add them to (C:/Windows/System32/drivers)/etc/hosts
  Host <<| tag == 'HOST_RECORD' |>> { ensure => present }
}

Desired Behavior:

All host resources are applied w/o error in the same Puppet run on Windows.

Actual Behavior:

See above.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo

Michael Hashizume (Jira)

unread,
Mar 7, 2023, 4:21:02 PM3/7/23
to puppe...@googlegroups.com

Dirk Heinrichs (Jira)

unread,
May 15, 2023, 2:02:03 AM5/15/23
to puppe...@googlegroups.com
Dirk Heinrichs commented on Bug PUP-11774
 
Re: Adding lots of hosts on Windows results in "permission denied" errors for some of them

Any news here? This is quite annoying as it requires two Puppet runs to completely setup new Windows hosts.

Michael Hashizume (Jira)

unread,
May 16, 2023, 4:16:01 PM5/16/23
to puppe...@googlegroups.com

Hi Dirk Heinrichs , our team has triaged this ticket and will look at this during one of our future work sessions.

Thanks!

Josh Cooper (Jira)

unread,
Jun 13, 2023, 7:22:02 PM6/13/23
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-11774

Dirk Heinrichs we can't reproduce this error. Export and collecting hosts entries with multiple windows agents works as expected for me:

C:\ProgramData\PuppetLabs>puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for veteran-click.delivery.puppetlabs.net
Info: Applying configuration version '1686689540'
Notice: /Stage[main]/Hosts/Host[veteran-click.delivery.puppetlabs.net]/ensure: created
Info: Computing checksum on file C:/Windows/System32/drivers/etc/hosts
Notice: /Stage[main]/Hosts/Host[hotter-maximum.delivery.puppetlabs.net]/ensure: created
Notice: Applied catalog in 0.07 seconds

There is a bug in https://github.com/puppetlabs/puppet/blob/2078f200469675c1e41b9b5b951da32ac47fa326/lib/puppet/util/filetype.rb#L125-L129 as it needs to ensure the Tempfile file descriptor is closed: https://github.com/puppetlabs/puppet/blob/2078f200469675c1e41b9b5b951da32ac47fa326/lib/puppet/util/filetype.rb#L125-L129 but I don't think that would have presented in the way you're seeing (as the lingering file descriptor doesn't prevent ruby from copying the file to C:/Windows/System32/drivers/etc/host).

The other possibility is the call to `FileUtils.cp` is not atomic, so it's possible for two calls to cp to race. But that shouldn't happen as the agent uses a file lock to ensure there is only one instance managing the system at a time.

Ultimately, the filetype should be using Puppet::FileSystem.replace_file to overwrite a file atomically.

Can you provide more steps to reproduce this and the errors you saw when it happens?

This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)
Atlassian logo

Dirk Heinrichs (Jira)

unread,
Jun 14, 2023, 2:29:03 AM6/14/23
to puppe...@googlegroups.com

Josh Cooper, thanks for looking into it.

You wrote "multiple", but your output only shows adding two hosts. As I wrote in the description, you'd need to add at least 20 to 30 host entries in one agent run for the error to show up and then it also doesn't happen all the time. The complete error message is:

Error: Puppet::Util::FileType::FileTypeFlat could not write C:/Windows/System32/drivers/etc/hosts: Permission denied @ rb_sysopen - C:/Windows/System32/drivers/etc/hosts

Error: /Stage[init]/Hosts/Host[XXX]: Could not evaluate: Puppet::Util::FileType::FileTypeFlat could not write C:/Windows/System32/drivers/etc/hosts: Permission denied @ rb_sysopen - C:/Windows/System32/drivers/etc/hosts

and it shows up after multiple tens of entries have already been created w/o problems, and afterwards another bunch of entries is created. Sometimes this happens multiple times during one agent run, sometimes not at all.

"Permission denied" showing up on a Windows host after lots of successful writes to the same file usually indicates a locking problem, rather than a real permission problem. Maybe this can be solved/worked around by simply adding a retry...

Reply all
Reply to author
Forward
0 new messages