Jira (PUP-5808) puppet via sudo from nfs home is not working

8 views
Skip to first unread message

Adam Winberg (JIRA)

unread,
Feb 3, 2016, 11:08:06 AM2/3/16
to puppe...@googlegroups.com
Adam Winberg created an issue
 
Puppet / Bug PUP-5808
puppet via sudo from nfs home is not working
Issue Type: Bug Bug
Affects Versions: PUP 3.8.4
Assignee: Kylo Ginsberg
Components: Types and Providers
Created: 2016/02/03 8:07 AM
Environment:

RHEL7.2, RHEL6.7
puppet-3.8.4-1.el7.noarch, puppet-3.8.4-1.el6.noarch

Priority: Normal Normal
Reporter: Adam Winberg

We use NFS mounted home directories for all our users. If a sysadmin or devop wants to trigger a puppet agent run via sudo, he/she must change dir from the nfs home directory to a local directory, otherwise the puppet run will fail.

This is nothing new, but I'd like to know if it is intended behaviour or if it is something that could be fixed.

So, running an strace on the puppet agent run, I see that puppet executes repeatedly executes a 'chdir' to the cwd. These chdirs are typically performed before a stat operation to get the correct path for a provider (i.e. /usr/bin/mount or /usr/local/sbin/mount). When running via sudo from my homedir, this fails since root has no permissions to enter that directory.

I can work around this by cd'ing to a local dir before running 'sudo puppet', or invoking sudo with the '-i' flag, but to make it failsafe for our devops it would be better if they didnt have to think about that. I would suggest to use the default tempdir of the system (linux=/tmp) instead of the cwd for these chdir operations. But I probably fail to see the complete picture.

Outtake from strace log:

chdir("/home/a001329")                  = -1 EACCES (Permission denied)
write(2, "\33[1;31mError: /Stage[pre]/Core::"..., 137^[[1;31mError: /Stage[pre]/Core::Users::Update_root_pw/Exec[set_grub2_password]: Could not evaluate: Permission denied - /home/a001329^[[0m) = 137
write(2, "\n", 1
)                       = 1
sendto(7, "<27>Feb  3 11:48:42 puppet-agent"..., 161, MSG_NOSIGNAL, NULL, 0) = 161
write(1, "\33[mNotice: /Stage[pre]/Core::Use"..., 131^[[mNotice: /Stage[pre]/Core::Users::Update_root_pw/Exec[grub2-mkconfig]: Dependency Exec[set_grub2_password] has failures: true^[[0m) = 131
write(1, "\n", 1
)                       = 1
sendto(7, "<29>Feb  3 11:48:42 puppet-agent"..., 158, MSG_NOSIGNAL, NULL, 0) = 158
write(2, "\33[1;31mWarning: /Stage[pre]/Core"..., 121^[[1;31mWarning: /Stage[pre]/Core::Users::Update_root_pw/Exec[grub2-mkconfig]: Skipping because of failed dependencies^[[0m) = 121
write(2, "\n", 1
)                       = 1
sendto(7, "<28>Feb  3 11:48:42 puppet-agent"..., 143, MSG_NOSIGNAL, NULL, 0) = 143
stat("/usr/bin/useradd", 0x7ffd573a3f70) = -1 ENOENT (No such file or directory)
stat("/bin/useradd", 0x7ffd573a3f70)    = -1 ENOENT (No such file or directory)
stat("/usr/sbin/useradd", {st_mode=S_IFREG|0750, st_size=114064, ...}) = 0
stat("/usr/sbin/useradd", {st_mode=S_IFREG|0750, st_size=114064, ...}) = 0

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
Atlassian logo

Adam Winberg (JIRA)

unread,
Feb 4, 2016, 1:46:02 AM2/4/16
to puppe...@googlegroups.com
Adam Winberg updated an issue
Change By: Adam Winberg
We use NFS mounted home directories for all our users. If a sysadmin or devop wants to trigger a puppet agent run via sudo, he/she must change dir from the nfs home directory to a local directory, otherwise the puppet run will fail. 

This is nothing new, but I'd like to know if it is intended behaviour or if it is something that could be fixed. 

So, running an strace on the puppet agent run, I see that puppet
 executes
 repeatedly executes a 'chdir' to the cwd. These chdirs are typically performed before a stat operation to get the correct path for a provider (i.e. /usr/bin/mount or /usr/local/sbin/mount). When running via sudo from my homedir, this fails since root has no permissions to enter that directory. 


I can work around this by cd'ing to a local dir before running 'sudo puppet', or invoking sudo with the '-i' flag, but to make it failsafe for our devops it would be better if they didnt have to think about that. I would suggest to use the default tempdir of the system (linux=/tmp) instead of the cwd for these chdir operations. But I probably fail to see the complete picture. 

Outtake from strace log:

{noformat}

chdir("/home/a001329")                  = -1 EACCES (Permission denied)
write(2, "\33[1;31mError: /Stage[pre]/Core::"..., 137^[[1;31mError: /Stage[pre]/Core::Users::Update_root_pw/Exec[set_grub2_password]: Could not evaluate: Permission denied - /home/a001329^[[0m) = 137
write(2, "\n", 1
)                       = 1
sendto(7, "<27>Feb  3 11:48:42 puppet-agent"..., 161, MSG_NOSIGNAL, NULL, 0) = 161
write(1, "\33[mNotice: /Stage[pre]/Core::Use"..., 131^[[mNotice: /Stage[pre]/Core::Users::Update_root_pw/Exec[grub2-mkconfig]: Dependency Exec[set_grub2_password] has failures: true^[[0m) = 131
write(1, "\n", 1
)                       = 1
sendto(7, "<29>Feb  3 11:48:42 puppet-agent"..., 158, MSG_NOSIGNAL, NULL, 0) = 158
write(2, "\33[1;31mWarning: /Stage[pre]/Core"..., 121^[[1;31mWarning: /Stage[pre]/Core::Users::Update_root_pw/Exec[grub2-mkconfig]: Skipping because of failed dependencies^[[0m) = 121
write(2, "\n", 1
)                       = 1
sendto(7, "<28>Feb  3 11:48:42 puppet-agent"..., 143, MSG_NOSIGNAL, NULL, 0) = 143
stat("/usr/bin/useradd", 0x7ffd573a3f70) = -1 ENOENT (No such file or directory)
stat("/bin/useradd", 0x7ffd573a3f70)    = -1 ENOENT (No such file or directory)
stat("/usr/sbin/useradd", {st_mode=S_IFREG|0750, st_size=114064, ...}) = 0
stat("/usr/sbin/useradd", {st_mode=S_IFREG|0750, st_size=114064, ...}) = 0
{noformat}


Adam Winberg (JIRA)

unread,
Feb 4, 2016, 1:47:03 AM2/4/16
to puppe...@googlegroups.com
Adam Winberg updated an issue
We use NFS mounted home directories for all our users. If a sysadmin or devop wants to trigger a puppet agent run via sudo, he/she must change dir from the nfs home directory to a local directory, otherwise the puppet run will fail. 

This is nothing new, but I'd like to know if it is intended behaviour or if it is something that could be fixed. 

So, running an strace on the puppet agent run, I see that puppet repeatedly executes a 'chdir' to the cwd. These chdirs are typically performed before a stat operation to get the correct path for a provider (i.e. /usr/bin/mount or /usr/local/sbin/mount). When running via sudo from my homedir, this fails since root has no permissions to enter that directory. 

I can work around this by cd'ing to a local dir before running 'sudo puppet', or invoking sudo with the '-i' flag, but to make it failsafe for our devops it would be better if they didnt have to think about that. I would suggest to use the default tempdir of the system (linux=/tmp) instead of the cwd for these chdir operations
. But , but  I  probably  might  fail to see the complete picture. 


Outtake from strace log:

{noformat}
chdir("/home/a001329")                  = -1 EACCES (Permission denied)
write(2, "\33[1;31mError: /Stage[pre]/Core::"..., 137^[[1;31mError: /Stage[pre]/Core::Users::Update_root_pw/Exec[set_grub2_password]: Could not evaluate: Permission denied - /home/a001329^[[0m) = 137
write(2, "\n", 1
)                       = 1
sendto(7, "<27>Feb  3 11:48:42 puppet-agent"..., 161, MSG_NOSIGNAL, NULL, 0) = 161
write(1, "\33[mNotice: /Stage[pre]/Core::Use"..., 131^[[mNotice: /Stage[pre]/Core::Users::Update_root_pw/Exec[grub2-mkconfig]: Dependency Exec[set_grub2_password] has failures: true^[[0m) = 131
write(1, "\n", 1
)                       = 1
sendto(7, "<29>Feb  3 11:48:42 puppet-agent"..., 158, MSG_NOSIGNAL, NULL, 0) = 158
write(2, "\33[1;31mWarning: /Stage[pre]/Core"..., 121^[[1;31mWarning: /Stage[pre]/Core::Users::Update_root_pw/Exec[grub2-mkconfig]: Skipping because of failed dependencies^[[0m) = 121
write(2, "\n", 1
)                       = 1
sendto(7, "<28>Feb  3 11:48:42 puppet-agent"..., 143, MSG_NOSIGNAL, NULL, 0) = 143
stat("/usr/bin/useradd", 0x7ffd573a3f70) = -1 ENOENT (No such file or directory)
stat("/bin/useradd", 0x7ffd573a3f70)    = -1 ENOENT (No such file or directory)
stat("/usr/sbin/useradd", {st_mode=S_IFREG|0750, st_size=114064, ...}) = 0
stat("/usr/sbin/useradd", {st_mode=S_IFREG|0750, st_size=114064, ...}) = 0
{noformat}


Adam Winberg (JIRA)

unread,
Aug 31, 2016, 6:26:01 AM8/31/16
to puppe...@googlegroups.com
Adam Winberg commented on Bug PUP-5808
 
Re: puppet via sudo from nfs home is not working

Any comments on this? Seems like we are the only ones affected by this 'bug'...

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

R.I.Pienaar (JIRA)

unread,
Aug 31, 2016, 6:32:02 AM8/31/16
to puppe...@googlegroups.com
R.I.Pienaar commented on Bug PUP-5808

Adam Winberg always better to show the .pp code whenever puppet tells you its failing on some specific .pp code.

Adam Winberg (JIRA)

unread,
Aug 31, 2016, 6:42:02 AM8/31/16
to puppe...@googlegroups.com
Adam Winberg commented on Bug PUP-5808

R.I.Pienaar Not sure what you mean. Puppet is not failing on some specific code, it's failing because I run it via sudo from a directory which root does not have permission to (nfs in this case).

R.I.Pienaar (JIRA)

unread,
Aug 31, 2016, 6:44:01 AM8/31/16
to puppe...@googlegroups.com
R.I.Pienaar commented on Bug PUP-5808

/Stage[pre]/Core::Users::Update_root_pw/Exec[set_grub2_password]: Could not evaluate: Permission denied - /home/a001329

it's failing on Core::Users::Update_root_pw which is some kind of exec. Or are you saying every single resource type of every resource you have in your catalog fails in this manner?

Adam Winberg (JIRA)

unread,
Aug 31, 2016, 7:44:02 AM8/31/16
to puppe...@googlegroups.com
Adam Winberg commented on Bug PUP-5808

ah, ok - yes, it fails repeatedly, not only for this resource. Sorry for the confusion. It fails since puppet tries to do 'chdir(cwd)' (which is not in my exec resource code) and the current working directory is unavailable for the root user.

I should mention that i tried changing the 'cwd' parameter to the exec resource before I discovered that this was not caused by the exec resource in itself and also affected other resources.

It should be easily reproduced by running puppet via sudo from a nfs share (make sure no_root_squash is not set).

R.I.Pienaar (JIRA)

unread,
Aug 31, 2016, 7:52:03 AM8/31/16
to puppe...@googlegroups.com
R.I.Pienaar commented on Bug PUP-5808

Adam Winberg ok, yeah that's what I wanted to determine, thanks. Well I've not seen anyone else complain about this but likewise I am not a Puppet dev so was just curious to see if we can get more info about the scenario maybe it rings bells to others

Henrik Lindberg (JIRA)

unread,
Aug 31, 2016, 8:56:02 AM8/31/16
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
 
Change By: Henrik Lindberg
Scrum Team: Client Platform

Adam Winberg (JIRA)

unread,
Feb 9, 2017, 2:23:02 AM2/9/17
to puppe...@googlegroups.com
Adam Winberg commented on Bug PUP-5808
 
Re: puppet via sudo from nfs home is not working

Since our users keep getting hit by this I've done some more digging. The 'problem' lies in the ruby code for the exec resource -

 puppet/lib/puppet/provider/exec.rb 

In the 'run' method the method 'checkexe' is called to verify existence of the executable. This code fails if you are running puppet via sudo standing in a directory where root has no permissions - for example an nfs mounted home directory. The reason for this, I'm guessing, is that 'checkexe' uses system calls which inherently tries to operate on the cwd. Setting the 'cwd' parameter for the exec resource in my puppet code has no effect, since the 'chdir' to cwd in puppet/lib/puppet/provider/exec.rb happens after the check_exe call. If I do a

 Dir.chdir "/tmp"

before the 'checkexe' call, everything works as expected.

So, to resolve this, which I would consider a bug, I would propose to implement a chdir to a tmp directory on the filesystem before anything else in the run method in puppet/lib/puppet/provider/exec.rb.

Any thoughts on this?

Thomas Mueller (JIRA)

unread,
Feb 9, 2017, 8:42:02 AM2/9/17
to puppe...@googlegroups.com

I've also seen this failure. we allow every user to trigger a puppet run with sudo. if I'm in the nfs-mounted homedir - where root has no access - I also see puppet runs failing.

workaround is to switch cwd to something root-accessible. like cd /; sudo puppet agent -t

Henrik Lindberg (JIRA)

unread,
May 16, 2017, 8:53:02 AM5/16/17
to puppe...@googlegroups.com

Henrik Lindberg (JIRA)

unread,
May 16, 2017, 8:53:03 AM5/16/17
to puppe...@googlegroups.com
Henrik Lindberg assigned an issue to Unassigned
Change By: Henrik Lindberg
Assignee: Kylo Ginsberg

Henrik Lindberg (JIRA)

unread,
May 16, 2017, 8:54:03 AM5/16/17
to puppe...@googlegroups.com

James Ralston (JIRA)

unread,
Sep 4, 2019, 7:11:04 PM9/4/19
to puppe...@googlegroups.com
James Ralston commented on Bug PUP-5808
 
Re: puppet via sudo from nfs home is not working

For posterity: the issue with the exec resource failing due to receiving EACCES when attempting to call chdir() to the cwd was resolved as a (perhaps intentional) side-effect of changes made to how the exec resource is implemented. (In essence, Puppet now calls chdir() in the child process it forks to perform the exec, not in the parent process.)

But the root cause of this issue—that it is never safe to call chdir() except in a child process—is not completely resolved. See PUP-9997 for details.

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

James Ralston (JIRA)

unread,
Sep 4, 2019, 7:21:03 PM9/4/19
to puppe...@googlegroups.com
James Ralston updated an issue
 

From digging, it was PUP-6919 (specifically, commit ebb57c3e) that rewrote the exec resource to have the child process call Dir.chdir().

Puppet 5.5.7 was the first version to contain the fix.

Change By: James Ralston
Fix Version/s: PUP 5.5.7
Reply all
Reply to author
Forward
0 new messages