EXEC resource fails but does not log a failure

[Paul Chernoch]

I have an EXEC command that has an onlyif condition. When I have my puppet service running as the correct user account, all goes well.
When the puppet agent is running under an inadequately privileged account, some operations fail silently.

During a maintenance action, another staff member tried to upgrade the puppet agent.
When that didn't work with our version of enterprise, he reinstalled puppet, but forgot to change the service accounts to be our special puppet user.
That puppet user has access to Team Foundation Server while the default account (NT System) does not.
I expected that puppet enterprise would show error messages in the log and show agent runs as failing.
IT DID NOT.

To diagnose the problem, I started a special shell using "Psexec.exe -i -s cmd.exe". This sysinternals tool allows me to impersonate "nt authority\system".
While running under that account, I verified that my EXEC command and the accompanying "onlyif" command each fail with error code 1.
The EXEC command being run is "TF.EXE" with the "VIEW" option, the Team Foundation Server command line executable.
The onlyif command is a shell call to "ruby.exe" which executes a rub script that also calls TF.EXE, this time with the "HISTORY" option.
The basic idea is that I call TF HISTORY to see if there is a newer file than the one I have extracted. If there is, then I return one value to indicate that EXEC should do its job.
If there is no newer file then I return a code that indicates no changes occurred and EXEC should not perform its action.
If TF.EXE returns an error code in the onlyif command, I decided to tell EXEC that it shoudl try to get the file whether it needs to or not.

Running both TF VIEW and RUBY (which calls TF HISTORY) in the special shell with the wrong user yields return codes of "1".
This should mean the the EXEC failed, but it does not log failure.

What should I do?

Here is a fragment of my puppet code:


  exec { "tf view ${filename} /version:${versionspec}":
    command   => $tfview_cmd,
    path      => $exec_path,
    cwd       => $tf_dir_unix,
    onlyif    => $tfhistory_cmd,
    returns   => ["0"],
    logoutput => true,
    require   => Class['tfview::tfcomponents']
  }

I am running agents on Windows 2008R2.

Paul

[jcbollinger]

On Tuesday, October 7, 2014 11:15:23 AM UTC-5, Paul Chernoch wrote:

I have an EXEC command that has an onlyif condition. When I have my puppet service running as the correct user account, all goes well.
When the puppet agent is running under an inadequately privileged account, some operations fail silently.

During a maintenance action, another staff member tried to upgrade the puppet agent.
When that didn't work with our version of enterprise, he reinstalled puppet, but forgot to change the service accounts to be our special puppet user.
That puppet user has access to Team Foundation Server while the default account (NT System) does not.
I expected that puppet enterprise would show error messages in the log and show agent runs as failing.
IT DID NOT.

To diagnose the problem, I started a special shell using "Psexec.exe -i -s cmd.exe". This sysinternals tool allows me to impersonate "nt authority\system".
While running under that account, I verified that my EXEC command and the accompanying "onlyif" command each fail with error code 1.

It is not a Puppet error nor a reason for resource failure for the 'onlyif' command of an Exec to return a failure code.  Puppet interprets that as a signal that the Exec's command does not need to be run -- that's the whole purpose of 'onlyif'.  If Puppet ran your Exec's 'onlyif' command and it returned a failure code, as you suggest would have happened, then the Exec succeeds.  The onlyif result tells it that the Exec is already in sync.  This is normal behavior.

 

What should I do?

It depends on your intended result.  The 'onlyif', 'unless', and 'creates' parameters of an Exec serve the purpose of determining whether it is already in sync at the beginning of a catalog run.  If it is (because an 'onlyif' fails, an 'unless' succeeds, or the target of 'creates' exists)  then that Exec succeeds without its 'command' being run.  In many cases, that's exactly what's wanted.

On the other hand, if there is a command whose failure must cause the Exec to fail, then that command must be incorporated one way or another into the Exec's 'command' parameter.  Your options for that depend on the node environment (especially the OS); I cannot advise you well for Windows, but one alternative for making an Exec perform compound commands would be to put them all in a batch file and Exec that.  A universal alternative is to put each command in a separate Exec, and declare an appropriate relationship between them, perhaps something like this:

  exec { ${tfhistory_cmd}:

    path      => $exec_path,
    cwd       => $tf_dir_unix,

    returns   => ["0"],
    logoutput => true,
    require   => Class['tfview::tfcomponents']
  }

  exec { "tf view ${filename} /version:${versionspec}":
    command   => $tfview_cmd,
    path      => $exec_path,
    cwd       => $tf_dir_unix,

    returns   => ["0"],
    logoutput => true,

    require   => Exec[$tfhistory_cmd]
  }

That means something closer to what you seem to want.


John

[Brian Morris]

Hello Paul,

I ran into this very issue recently. This happens because Powershell does not properly catch error events. If the PS engine is able to run the script engine while continuing on errors then it will exit with a code of 0. Even if your script utterly failed to run, if the PS engine does not experience an error of its own you get an exit code of 0.

There are ways to turn on better, more sane error trapping in PS, but this will cause you to go deep into MS' proprietary error code system. I get around all of this by just looking at the output of PS commands, such as checking for null output.

[Paul Chernoch]

Thank you for your advice. While researching the problem with a colleague, we discovered the root cause:

TF.EXE HISTORY returns an ERRORLEVEL of 0 (meaning success) in one narrow case when it should not.

Case 1: Running user is Authorized for TFS, no login credentials on command line. Success 0. CORRECT.
Case 2: Running user is Authorized for TFS, good login credentials on command line. Success 0. CORRECT.
Case 3: Running user is Authorized for TFS, bad login credentials on command line. Error 1. CORRECT.
Case 4: Running user is NOT Authorized for TFS, no login credentials on command line. Success 0. INCORRECT.
Case 5: Running user is NOT Authorized for TFS, good login credentials on command line. Success 0. CORRECT.
Case 6: Running user is NOT Authorized for TFS, bad login credentials on command line. Error 1. CORRECT.

So only case 4 produces incorrect results.
Strangely, the TF VIEW command, which actually fetches a file from TFS, handles all the cases properly.

Paul

[Rob Reynolds]

On Wed, Oct 8, 2014 at 2:51 PM, Paul Chernoch <pache...@gmail.com> wrote:

Thank you for your advice. While researching the problem with a colleague, we discovered the root cause:

TF.EXE HISTORY returns an ERRORLEVEL of 0 (meaning success) in one narrow case when it should not.

Case 1: Running user is Authorized for TFS, no login credentials on command line. Success 0. CORRECT.
Case 2: Running user is Authorized for TFS, good login credentials on command line. Success 0. CORRECT.
Case 3: Running user is Authorized for TFS, bad login credentials on command line. Error 1. CORRECT.
Case 4: Running user is NOT Authorized for TFS, no login credentials on command line. Success 0. INCORRECT.
Case 5: Running user is NOT Authorized for TFS, good login credentials on command line. Success 0. CORRECT.
Case 6: Running user is NOT Authorized for TFS, bad login credentials on command line. Error 1. CORRECT.

So only case 4 produces incorrect results.
Strangely, the TF VIEW command, which actually fetches a file from TFS, handles all the cases properly.

Paul

On Tuesday, October 7, 2014 12:15:23 PM UTC-4, Paul Chernoch wrote:

I have an EXEC command that has an onlyif condition. When I have my puppet service running as the correct user account, all goes well.
When the puppet agent is running under an inadequately privileged account, some operations fail silently.

During a maintenance action, another staff member tried to upgrade the puppet agent.
When that didn't work with our version of enterprise, he reinstalled puppet, but forgot to change the service accounts to be our special puppet user.

One thing you can add to your install for the agent that will persist across upgrades is an MSI property you pass to the installer -  PUPPET_AGENT_ACCOUNT_USER="special puppet user" PUPPET_AGENT_ACCOUNT_PASSWORD="special puppet user pass"

See Installing PE Agent Windows[1] for more information. This was introduced in PE 3.2.

[1] https://docs.puppetlabs.com/pe/latest/install_windows.html#msi-properties

 

That puppet user has access to Team Foundation Server while the default account (NT System) does not.
I expected that puppet enterprise would show error messages in the log and show agent runs as failing.
IT DID NOT.

To diagnose the problem, I started a special shell using "Psexec.exe -i -s cmd.exe". This sysinternals tool allows me to impersonate "nt authority\system".
While running under that account, I verified that my EXEC command and the accompanying "onlyif" command each fail with error code 1.
The EXEC command being run is "TF.EXE" with the "VIEW" option, the Team Foundation Server command line executable.
The onlyif command is a shell call to "ruby.exe" which executes a rub script that also calls TF.EXE, this time with the "HISTORY" option.
The basic idea is that I call TF HISTORY to see if there is a newer file than the one I have extracted. If there is, then I return one value to indicate that EXEC should do its job.
If there is no newer file then I return a code that indicates no changes occurred and EXEC should not perform its action.
If TF.EXE returns an error code in the onlyif command, I decided to tell EXEC that it shoudl try to get the file whether it needs to or not.

Running both TF VIEW and RUBY (which calls TF HISTORY) in the special shell with the wrong user yields return codes of "1".
This should mean the the EXEC failed, but it does not log failure.

What should I do?

Here is a fragment of my puppet code:


  exec { "tf view ${filename} /version:${versionspec}":
    command   => $tfview_cmd,
    path      => $exec_path,
    cwd       => $tf_dir_unix,
    onlyif    => $tfhistory_cmd,
    returns   => ["0"],
    logoutput => true,
    require   => Class['tfview::tfcomponents']
  }

I am running agents on Windows 2008R2.

Paul

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/493d455a-eb1a-4af0-80f6-1c742ccb99f4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Rob Reynolds

Developer, Puppet Labs