Puppet Exec resource with complex Environment Variables...

[Sean]

Greetings,

I am working on streamlining some older puppet code, that uses a lot of Exec resources to accomplish it's purposes.  It's not terribly elegant and we're working on design to replace it with code that leverages puppet features.  One of the pieces I'm struggling with is how to set complex environment variables that are available to use in onlyif/unless statements as well as the command itself.  I'm wondering if this just isn't possible?

Here's an example with the aide.  NOTE that we're not in a spot where I can make puppet manage the aide config file, and thus use an aide module and parameters instead of shell variables.  All of these commands run successfully in a bash shell for the various conditions that would apply.  I have other similar scenarios, but aide is one of the more complex ones.

  exec { 'init-aide-database':
    path        => '/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin',
    environment => [ 'DBDIR=$(egrep \'^@@define DBDIR \' /etc/aide.conf|awk \'{print $NF}\')',
      'DBFILE=$(egrep \'^database=file\' /etc/aide.conf|awk -F/ \'{print $NF}\')',
      'DBNEW=$(egrep \'^database_out=file\' /etc/aide.conf|awk -F/ \'{print $NF}\')',
      ],
    command     => '/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}',
    unless      => 'test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}',
    require     => Package['aide'],
    logoutput   => true,
    timeout     => 0,
  }

When executing, puppet always runs the exec because the env var's are empty, so the unless case always fails.  Then we get a scenario where we're running aide --init when it's not needed and the cp command throws an error due to empty variables.

My thought at the moment is that I need to build two shell scripts as file resources.  One to call in the unless test, and another to call in the command if the unless script fails.  Perhaps that's more elegant, but we're hoping to avoid delivering script files to the nodes wherever possible.  I'm hoping the gurus out here might point me in the best direction!

Thank you kindly!

[jcbollinger]

On Friday, December 4, 2015 at 10:27:52 AM UTC-6, Sean wrote:

Greetings,

I am working on streamlining some older puppet code, that uses a lot of Exec resources to accomplish it's purposes.  It's not terribly elegant and we're working on design to replace it with code that leverages puppet features.  One of the pieces I'm struggling with is how to set complex environment variables that are available to use in onlyif/unless statements as well as the command itself.  I'm wondering if this just isn't possible?

Here's an example with the aide.  NOTE that we're not in a spot where I can make puppet manage the aide config file, and thus use an aide module and parameters instead of shell variables.  All of these commands run successfully in a bash shell for the various conditions that would apply.  I have other similar scenarios, but aide is one of the more complex ones.

  exec { 'init-aide-database':
    path        => '/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin',
    environment => [ 'DBDIR=$(egrep \'^@@define DBDIR \' /etc/aide.conf|awk \'{print $NF}\')',
      'DBFILE=$(egrep \'^database=file\' /etc/aide.conf|awk -F/ \'{print $NF}\')',
      'DBNEW=$(egrep \'^database_out=file\' /etc/aide.conf|awk -F/ \'{print $NF}\')',
      ],
    command     => '/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}',
    unless      => 'test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}',
    require     => Package['aide'],
    logoutput   => true,
    timeout     => 0,
  }

When executing, puppet always runs the exec because the env var's are empty, so the unless case always fails.  Then we get a scenario where we're running aide --init when it's not needed and the cp command throws an error due to empty variables.

Are you sure?  I'd be inclined to suppose that the problem was that the environment variables don't have the values you expect them to have.  As far as I am aware, Puppet inserts environment variables into child processes' environments via a mechanism that does not involve evaluating the value expressions via the shell (even when you use the 'shell' provider).

I had thought that the same environment is presented to the 'unless' and 'onlyif' commands as is presented to the main command, but I can't immediately back that up, as the documentation is at best unclear on that point.  If you could establish that it is not, then that might justify filing a ticket.

 

My thought at the moment is that I need to build two shell scripts as file resources.  One to call in the unless test, and another to call in the command if the unless script fails.  Perhaps that's more elegant, but we're hoping to avoid delivering script files to the nodes wherever possible.  I'm hoping the gurus out here might point me in the best direction!

If you must determine the needed values for your environment variables at run time, by executing shell code, then I'm pretty confident that one way or another you will need to build that in to each of the commands you use.  The approach you describe is a reasonable way to do that.

You could also consider whether it would be worth the effort to build a full-fledged custom type.  You would still need to perform some environment mangling, but you could at least avoid managing separate command scripts.


John

[Sean]

John,

Thanks for the reply.  To answer your first question, no I'm not completely sure.  What I can say is that I can run the commands in a shell by hand and the result is what I hope for.  When I run puppet, with this particular class included, puppet hangs for a while with no screen output.  This leads me to believe that the "unless" fails and the "command" executes.  I can verify that by running (in another shell):

[root@desktop ~]# ps -ef|grep aide
root     28085     1  0 14:42 ?        00:00:00 sh -c /usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}
root     28088 28085 20 14:42 ?        00:01:24 /usr/sbin/aide --init

I can say when I run the various piece by hand inside an interactive shell everything comes out as expected:

[root@desktop ~]# DBDIR=$(egrep '^@@define DBDIR ' /etc/aide.conf |awk '{print $NF}')
[root@desktop ~]# echo $DBDIR
/var/lib/aide
[root@desktop ~]# DBFILE=$(egrep '^database=file' /etc/aide.conf |awk -F/ '{print $NF}')
[root@desktop ~]# echo $DBFILE
aide.db.gz
[root@desktop ~]# DBNEW=$(egrep '^database_out=file' /etc/aide.conf |awk -F/ '{print $NF}')
[root@desktop ~]# echo $DBNEW
aide.db.new.gz
[root@desktop ~]# test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}
[root@desktop ~]# echo $?
0
[root@desktop ~]# /usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}
cp: overwrite ‘/var/lib/aide/aide.db.gz’? y

The cp overwrite prompt above comes from the alias cp='cp -i ' in my shell.  I'm not sure if that would be there inside puppet, but if so I'll modify to negate that option.

For testing, I created a separate class for just this one exec.  I modified the unless and command statements as follows:

    command     => 'echo "/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}"',
    unless      => 'echo "test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}" && test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}',

So basically the "unless" echo's what it's doing, then tries to do it.  The "command" just echoes what it would do.  Here's the puppet agent --test --debug output for the class:

Debug: Exec[init-aide-database](provider=posix): Executing check 'echo "test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}" && test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}'
Debug: Executing 'echo "test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}" && test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}'
Debug: /Stage[main]/Testenv/Exec[init-aide-database]/unless: test -f $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database_out=file' /etc/aide.conf|awk -F/ '{print $NF}') && test -f $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database=file' /etc/aide.conf|awk -F/ '{print $NF}')
Debug: /Stage[main]/Testenv/Exec[init-aide-database]/unless: sh: line 0: test: too many arguments
Debug: Exec[init-aide-database](provider=posix): Executing 'echo "/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}"'
Debug: Executing 'echo "/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}"'
Notice: /Stage[main]/Testenv/Exec[init-aide-database]/returns: /usr/sbin/aide --init >/dev/null 2>&1 && cp -p $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database_out=file' /etc/aide.conf|awk -F/ '{print $NF}') $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database=file' /etc/aide.conf|awk -F/ '{print $NF}')
Notice: /Stage[main]/Testenv/Exec[init-aide-database]/returns: executed successfully
Debug: /Stage[main]/Testenv/Exec[init-aide-database]: The container Class[Testenv] will propagate my refresh event
Debug: Class[Testenv]: The container Stage[main] will propagate my refresh event

So it looks like the strings inside the environment variables aren't interpreted by the shell, they're just passed as literal strings.  But notably, the the unless test fails with an error from /usr/bin/test that doesn't surface when running interactively.  I will attempt to run in the shell provider instead of posix, but if that fails, I just re-code everything into shell scripts and have puppet push and execute them.  If we need to make this a bug, I'll be happy to help, but I'm running on EL7 with puppet out of EPEL7, with no intent of going to Puppet 4 anytime soon.

Thanks again for taking a look!

[jcbollinger]

On Monday, December 7, 2015 at 4:51:10 PM UTC-6, Sean wrote:

John,

Thanks for the reply.  To answer your first question, no I'm not completely sure.  What I can say is that I can run the commands in a shell by hand and the result is what I hope for.  When I run puppet, with this particular class included, puppet hangs for a while with no screen output.  This leads me to believe that the "unless" fails and the "command" executes.

I suggest that instead of guessing, you run the agent with --debug output enabled, which indeed you have done.  That should give you a pretty good idea of which commands run and which do not, and at least a somewhat more certain understanding of why.

 

 I can verify that by running (in another shell):

[root@desktop ~]# ps -ef|grep aide
root     28085     1  0 14:42 ?        00:00:00 sh -c /usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}
root     28088 28085 20 14:42 ?        00:01:24 /usr/sbin/aide --init

I can say when I run the various piece by hand inside an interactive shell everything comes out as expected:

[root@desktop ~]# DBDIR=$(egrep '^@@define DBDIR ' /etc/aide.conf |awk '{print $NF}')
[root@desktop ~]# echo $DBDIR
/var/lib/aide
[root@desktop ~]# DBFILE=$(egrep '^database=file' /etc/aide.conf |awk -F/ '{print $NF}')
[root@desktop ~]# echo $DBFILE
aide.db.gz
[root@desktop ~]# DBNEW=$(egrep '^database_out=file' /etc/aide.conf |awk -F/ '{print $NF}')
[root@desktop ~]# echo $DBNEW
aide.db.new.gz
[root@desktop ~]# test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}
[root@desktop ~]# echo $?
0
[root@desktop ~]# /usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}
cp: overwrite ‘/var/lib/aide/aide.db.gz’? y

The most important point that I was trying to convey to you in my previous message was that Puppet does not use a mechanism anything like that to set up the environment for the Exec's commands.  It takes the environment strings you provide, as interpreted by Puppet at catalog-building time, and inserts them directly into the environment.  There will therefore be no command substitution performed, and any variable interpolation will be of Puppet variables, in catalog-building context, not of environment variables in the command's runtime context.

As I said, if you need to determine the values of your environment variables in the manner you present, then you absolutely do need to script that, and have Puppet execute the resulting script.  I also suggested that you might write Ruby code that computes the same values, and build a custom type and provider around that, with which to replace your Exec.  Although cleaner, that would probably require a lot more effort than it's worth.

 

The cp overwrite prompt above comes from the alias cp='cp -i ' in my shell.  I'm not sure if that would be there inside puppet, but if so I'll modify to negate that option.

For testing, I created a separate class for just this one exec.  I modified the unless and command statements as follows:

    command     => 'echo "/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}"',
    unless      => 'echo "test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}" && test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}',

So basically the "unless" echo's what it's doing, then tries to do it.  The "command" just echoes what it would do.  Here's the puppet agent --test --debug output for the class:

Debug: Exec[init-aide-database](provider=posix): Executing check 'echo "test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}" && test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}'
Debug: Executing 'echo "test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}" && test -f ${DBDIR}/${DBNEW} && test -f ${DBDIR}/${DBFILE}'
Debug: /Stage[main]/Testenv/Exec[init-aide-database]/unless: test -f $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database_out=file' /etc/aide.conf|awk -F/ '{print $NF}') && test -f $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database=file' /etc/aide.conf|awk -F/ '{print $NF}')

That last is the 'unless' command and its standard output.  Look carefully at what that's telling you: you set up the output to echo the command, and the result will reflect all shell expansions applicable to the command.  I say again: no further substitutions or expansions will be performed on the command or its arguments.  That's the literal command that Puppet is trying to execute for you.  I'm sure that if you analyze it carefully enough you'll be able to determine why the result is

 

Debug: /Stage[main]/Testenv/Exec[init-aide-database]/unless: sh: line 0: test: too many arguments

This is a failure result, albeit not one of the ones you intended, so Puppet proceeds to execute the main command:

 

Debug: Exec[init-aide-database](provider=posix): Executing 'echo "/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}"'

At this point you should also take note of the "provider=posix".  This reflects the default Exec provider for your platform, which the agent seems to have determined is Unix-y.  It is extremely important to understand that the 'posix' provider executes your command directly, not via a shell.  Thus, redirections, variable expansions, and other shell constructs in the command are not recognized as anything special.  Since this seems not to be what you want, you can instruct Puppet to execute the command via a shell by adding the parameter provider => 'shell' to your Exec declaration.

 

Debug: Executing 'echo "/usr/sbin/aide --init >/dev/null 2>&1 && cp -p ${DBDIR}/${DBNEW} ${DBDIR}/${DBFILE}"'
Notice: /Stage[main]/Testenv/Exec[init-aide-database]/returns: /usr/sbin/aide --init >/dev/null 2>&1 && cp -p $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database_out=file' /etc/aide.conf|awk -F/ '{print $NF}') $(egrep '^@@define DBDIR ' /etc/aide.conf|awk '{print $NF}')/$(egrep '^database=file' /etc/aide.conf|awk -F/ '{print $NF}')
Notice: /Stage[main]/Testenv/Exec[init-aide-database]/returns: executed successfully

And this shows the result: your Exec was executed successfully, producing the given output.  Since it was just an 'echo' command, it is unsurprising that this did not have the desired effect on your system's state.

 

Debug: /Stage[main]/Testenv/Exec[init-aide-database]: The container Class[Testenv] will propagate my refresh event
Debug: Class[Testenv]: The container Stage[main] will propagate my refresh event

So it looks like the strings inside the environment variables aren't interpreted by the shell, they're just passed as literal strings.

Well yes, that's exactly what I said before.

 

 But notably, the the unless test fails with an error from /usr/bin/test that doesn't surface when running interactively.

For exactly the reason you just gave.

 

 I will attempt to run in the shell provider instead of posix,

As I said, your approach to setting up the environment will not work as you want.  Using the 'shell' provider likely will solve some of your problems, but not that one.

 

but if that fails, I just re-code everything into shell scripts and have puppet push and execute them.  If we need to make this a bug,

I don't see any buggy behavior.  Puppet seems to be behaving exactly as I would expect, with the only potentially surprising thing being that even the 'posix' provider runs the 'unless' and 'onlyif' commands via the shell.  I do not account that a bug, however, as (1) it's probably more useful behavior, and (2) it's consistent with the example in the docs.


John