I'm testing a cfengine 2.1.14 -> 2.1.17 upgrade this afternoon and have
hit a snag. It appears that the places in which I use constructs like
control:
fooish = ( ExecResult(/bin/sh -c "/bin/cat file | yada | yada | yada") )
seem to be only executing the first argument (eg: the /bin/cat) instead of
the whole pipeline.
For instance, a cfagent -d2 spins like this waiting for cat to return
(which it never will):
----
HandleFunctionObject(ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' "))
IsBuiltinFunction(ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' "))
IsBuiltinFunction: ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' ")
HandleFunction: ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' ")
FunctionStringToCode(ExecResult)
Appending [/bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' ]
ExpandVarstring(/bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' )
ARG[0] /bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n'
cfpopen(/bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' )
cfengine:: Time out
----
ps -ef during this spin shows that there's only a "/bin/cat" running.
I have several other variables that do similar constructs with find, and
they appear to exhibit the same problem, returning the listing for the
current working directory, instead of the directory I indicated in the
second argument for find.
Did the way ExecResult works change recently, or is this possibly a bug
related to the parameter changes in 2.1.16 (though the Changelog states it
was fixed for 2.1.17a, which I'm not sure is what you get if you grab
ftp://ftp.iu.hio.no/pub/cfengine/cfengine-2.1.17.tar.gz)
Thanks for any clues that could be landed upon my thick head :)
-n
--
-------------------------------------------
nathan hruby <nhr...@uga.edu>
uga enterprise information technology services
production systems support
-------------------------------------------
Nathan,
binning cats is cruel so cfengine doesn't allow you to gloat with "yada
yada" afterwards.
Actually this has to do with the fact that no shell is used to wrap the
command. It could be useful to have a functino which does wrap too. For
now you can do /bin/sh to hush up the cruelty.
Mark
> Nathan,
>
> binning cats is cruel so cfengine doesn't allow you to gloat with "yada
> yada" afterwards.
Umm.. ok :)
> Actually this has to do with the fact that no shell is used to wrap the
> command. It could be useful to have a functino which does wrap too. For
> now you can do /bin/sh to hush up the cruelty.
The exact line that is causing the hang is:
CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n'") )
Is that not correct? It does work in 2.1.14, which is why I'm perplexed.
Thanks!
> binning cats is cruel so cfengine doesn't allow you to gloat with
> "yada yada" afterwards.
>
> Actually this has to do with the fact that no shell is used to wrap
> the command. It could be useful to have a functino which does wrap
> too. For now you can do /bin/sh to hush up the cruelty.
Wow, I so didn't follow that. :-)
nathan r. hruby wrote:
> fooish = ( ExecResult(/bin/sh -c "/bin/cat file | yada | yada | yada") )
Do you use absolute paths for all the yadas? Maybe that is the problem.
Best,
Brendan
--
Senior System Administrator
The University of Chicago
Department of Computer Science
http://www.cs.uchicago.edu/people/brendan
-JM
> _______________________________________________
> Help-cfengine mailing list
> Help-c...@gnu.org
> http://lists.gnu.org/mailman/listinfo/help-> cfengine
>
> CFE only checks that the first command in ExecResult is fully pathed.
Yeah, but what if the environment passed to the shell does not contain a
PATH? Maybe the *shell* can't find the program. Just a thought...
-Jason Martin
> -----Original Message-----
> From: Brendan Strejcek [mailto:bre...@cs.uchicago.edu]
> Sent: Friday, November 11, 2005 8:47 AM
> To: Martin, Jason H
> Cc: Mark Burgess; help-c...@gnu.org
> Subject: Re: 2.1.17 having issue with ExecResult() using pipelines
>
>
> If yada isn't in the normal path then that might be a problem; I don't
> know what PATH will be set to when that happens. Good point.
>
Err.. right: here's the full real command:
CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n'") )
What I'm seeing in 2.1.17 is that everything after "/bin/cat" simply
dissapears and thus no pipeline. eg, run a ps -ef while this is runnning
and the awk and tr are not being executed. The above works in 2.1.14.
I also have several ExecResult() calls that use find and they too exhibit
the same behavior:
MailQueueCount = ( ExecResult(/bin/sh -c "/usr/bin/find /var/spool/mqueue -type f -name 'q*' | wc -l | tr -d ' ' | tr -d '\n' ") )
returns a find for $CWD, not /var/spool/mqueue, and certianly not the
output of "wc -l"
Thanks,
-n
> -Jason Martin
>
>> -----Original Message-----
>> From: Brendan Strejcek [mailto:bre...@cs.uchicago.edu]
>> Sent: Friday, November 11, 2005 8:47 AM
>> To: Martin, Jason H
>> Cc: Mark Burgess; help-c...@gnu.org
>> Subject: Re: 2.1.17 having issue with ExecResult() using pipelines
>>
>>
>> Martin, Jason H wrote:
>>
>>> CFE only checks that the first command in ExecResult is
>> fully pathed.
>>
>> Yeah, but what if the environment passed to the shell does
>> not contain a PATH? Maybe the *shell* can't find the program.
>> Just a thought...
>>
>> Best,
>> Brendan
>>
>> --
>> Senior System Administrator
>> The University of Chicago
>> Department of Computer Science
>> http://www.cs.uchicago.edu/people/brendan
>>
>
>
> _______________________________________________
> Help-cfengine mailing list
> Help-c...@gnu.org
> http://lists.gnu.org/mailman/listinfo/help-cfengine
>
--
> On Fri, 11 Nov 2005, Martin, Jason H wrote:
>
> >If yada isn't in the normal path then that might be a problem; I don't
> >know what PATH will be set to when that happens. Good point.
> >
>
> Err.. right: here's the full real command:
> CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg |
> /bin/awk '{print $1}' | tr -d '\n'") )
>
> What I'm seeing in 2.1.17 is that everything after "/bin/cat" simply
> dissapears and thus no pipeline. eg, run a ps -ef while this is
> runnning and the awk and tr are not being executed. The above works in
> 2.1.14.
It might be interesting to trace the cfagent, so see exactly what it is
doing. You don't really know what snapshot you are getting when you just
run ps at a particular point in time.
Linux: strace -f cfagent -q
Solaris: truss -f cfagent -q # truss output goes to stderr
BSD (and OS X): ktrace -di cfagent -q; ktrace -C; kdump | less; rm ktrace.out
Stick an "env -i" before the cfagent if you don't want your current
environment to influence its behavior.
Also, have you tried cfagent with -d set to various levels?
Best,
Brendan
--
Senior System Administrator
The University of Chicago
Department of Computer Science
http://www.cs.uchicago.edu/people/brendan
http://people.cs.uchicago.edu/~brendan
> nathan r. hruby wrote:
>
>> On Fri, 11 Nov 2005, Martin, Jason H wrote:
>>
>>> If yada isn't in the normal path then that might be a problem; I don't
>>> know what PATH will be set to when that happens. Good point.
>>>
>>
>> Err.. right: here's the full real command:
>> CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg |
>> /bin/awk '{print $1}' | tr -d '\n'") )
>>
>> What I'm seeing in 2.1.17 is that everything after "/bin/cat" simply
>> dissapears and thus no pipeline. eg, run a ps -ef while this is
>> runnning and the awk and tr are not being executed. The above works in
>> 2.1.14.
>
> It might be interesting to trace the cfagent, so see exactly what it is
> doing. You don't really know what snapshot you are getting when you just
> run ps at a particular point in time.
>
> Linux: strace -f cfagent -q
>
> Solaris: truss -f cfagent -q # truss output goes to stderr
>
> BSD (and OS X): ktrace -di cfagent -q; ktrace -C; kdump | less; rm ktrace.out
>
> Stick an "env -i" before the cfagent if you don't want your current
> environment to influence its behavior.
>
strace stops at a read(), which is expected because cfagent is running
"/bin/cat" with no arguments, so it'll never read anything, and spin.
Since it is sitting at a read(), there's plenty time enough for me to
switch to a different terminal and run ps -ef :)
To my less than trained C coding eye, It appears that cfpopen() changed
the way it cleans things up before sending them to execve() and thus,
quoting is being removed and the pipe is being passed as a series of
arguments, instead of one long argument. This is a bit of a shot from a
quick glance at the source between meetings.
strace is attached, note that I had to ^C to finish, so the read() is
unfinished and followed by SIGINT (see line 1766, pid 541)
> Also, have you tried cfagent with -d set to various levels?
>
Yes, the -d2 output was appended to my original message, I'll re-paste
here for completeness:
----
HandleFunctionObject(ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' "))
IsBuiltinFunction(ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' "))
IsBuiltinFunction: ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' ")
HandleFunction: ExecResult(/bin/sh -c "/bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' ")
FunctionStringToCode(ExecResult)
Appending [/bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' ]
ExpandVarstring(/bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' )
ARG[0] /bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n'
cfpopen(/bin/sh -c /bin/cat /proc/loadavg | /bin/awk '{print $1}' | tr -d '\n' )
cfengine:: Time out
----
This looks correct (note the ARG[0]), so that is why I think the bug is in
cfpopen(). Again, it may be that my approach is wrong, and/or 2.1.17 is now
preventing me from taking advantage of a previous bug that I had come to
rely on.
Thanks!
-n
> On Fri, 11 Nov 2005, Martin, Jason H wrote:
>
>> If yada isn't in the normal path then that might be a problem; I don't
>> know what PATH will be set to when that happens. Good point.
>>
>
> Err.. right: here's the full real command:
> CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg |
> /bin/awk '{print $1}' | tr -d '\n'") )
>
> What I'm seeing in 2.1.17 is that everything after "/bin/cat" simply
> dissapears and thus no pipeline. eg, run a ps -ef while this is runnning
> and the awk and tr are not being executed. The above works in 2.1.14.
I've had similar problems (starting in 2.1.16, I think). I've been able
to workaround it by switching the order of quote nesting; in other words:
thisworks= ( ExecResult(/bin/sh -c 'command1 "quotedarg" | command2') )
but
thisnoworks= ( ExecResult(/bin/sh -c "command1 'quotedarg' | command2") )
In cases where I really neeed single quotes on the inside (as you do in
your awk call), I've generally just punted and written little shell
scripts for cfengine to call. I could probably kludge it with special
variables, but it gets nearly unreadable quickly.
> nathan r. hruby wrote:
>
>> On Fri, 11 Nov 2005, Martin, Jason H wrote:
>>
>>> If yada isn't in the normal path then that might be a problem; I don't
>>> know what PATH will be set to when that happens. Good point.
>>>
>>
>> Err.. right: here's the full real command:
>> CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg |
>> /bin/awk '{print $1}' | tr -d '\n'") )
>>
>> What I'm seeing in 2.1.17 is that everything after "/bin/cat" simply
>> dissapears and thus no pipeline. eg, run a ps -ef while this is runnning
>> and the awk and tr are not being executed. The above works in 2.1.14.
>
>
> I've had similar problems (starting in 2.1.16, I think). I've been able to
> workaround it by switching the order of quote nesting; in other words:
>
Yeah, I see that now
http://groups.google.com/group/gnu.cfengine.help/browse_frm/thread/75f340262cc417d7/30b98b9f4d33df5b?q=Quotes+make+a+difference+in+2.1.16&rnum=1#30b98b9f4d33df5b
> thisworks= ( ExecResult(/bin/sh -c 'command1 "quotedarg" | command2') )
>
> but
>
> thisnoworks= ( ExecResult(/bin/sh -c "command1 'quotedarg' | command2") )
>
> In cases where I really neeed single quotes on the inside (as you do in your
> awk call), I've generally just punted and written little shell scripts for
> cfengine to call. I could probably kludge it with special variables, but it
> gets nearly unreadable quickly.
>
Sigh. Neither are good solutions. I have a lot of these little pipelines,
all culled from the little shell scripts I'm trying to get rid of so it
looks like the kludgy special variable it is.. Which doesn't seem to work
either.
Oh hell.
> I have a lot of these little pipelines, all culled from the little
> shell scripts I'm trying to get rid of so it looks like the kludgy
> special variable it is.. Which doesn't seem to work either.
You might be able to replace most of the awk '{print $1}'-type calls
with "cut" invocations:
$ cut -d" " -f1 </proc/loadavg
0.00
ExecShellResult
which will work just like ExecResult but use a shell, so that your pipes
will work now. You can get it from the svn server right now. Snapshot in
a few mins
M
On Fri, 2005-11-11 at 15:24 -0500, nathan r. hruby wrote:
> On Fri, 11 Nov 2005, Chip Seraphine wrote:
>
> > nathan r. hruby wrote:
> >
> >> On Fri, 11 Nov 2005, Martin, Jason H wrote:
> >>
> >>> If yada isn't in the normal path then that might be a problem; I don't
> >>> know what PATH will be set to when that happens. Good point.
> >>>
> >>
> >> Err.. right: here's the full real command:
> >> CurrentLoad = ( ExecResult(/bin/sh -c "/bin/cat /proc/loadavg |
> >> /bin/awk '{print $1}' | tr -d '\n'") )
> >>
> >> What I'm seeing in 2.1.17 is that everything after "/bin/cat" simply
> >> dissapears and thus no pipeline. eg, run a ps -ef while this is runnning
> >> and the awk and tr are not being executed. The above works in 2.1.14.
> >
> >
> > I've had similar problems (starting in 2.1.16, I think). I've been able to
> > workaround it by switching the order of quote nesting; in other words:
> >
>
> Yeah, I see that now
> http://groups.google.com/group/gnu.cfengine.help/browse_frm/thread/75f340262cc417d7/30b98b9f4d33df5b?q=Quotes+make+a+difference+in+2.1.16&rnum=1#30b98b9f4d33df5b
>
> > thisworks= ( ExecResult(/bin/sh -c 'command1 "quotedarg" | command2') )
> >
> > but
> >
> > thisnoworks= ( ExecResult(/bin/sh -c "command1 'quotedarg' | command2") )
> >
> > In cases where I really neeed single quotes on the inside (as you do in your
> > awk call), I've generally just punted and written little shell scripts for
> > cfengine to call. I could probably kludge it with special variables, but it
> > gets nearly unreadable quickly.
> >
>
> Sigh. Neither are good solutions. I have a lot of these little pipelines,
> all culled from the little shell scripts I'm trying to get rid of so it
> looks like the kludgy special variable it is.. Which doesn't seem to work
> either.
>
> Oh hell.
>
> -n
I'm not sure how running
ExecResult(/bin/sh -c ${dblquote}/bin/cat /proc/cpuinfo | grep
Processor${dblquote})
Is any different than
ExecShellresult(/bin/cat /proc/cpuinfo | grep Processor)
...or am I missing something?
Paul Krizak 5900 E. Ben White Blvd. MS 625
Advanced Micro Devices Austin, TX 78741
Linux/Unix Systems Engineering Phone: (512) 602-8775
Microprocessor Solutions Sector
M
On Fri, 2005-11-11 at 15:13 -0600, Paul Krizak wrote:
> What was preventing the pipes from working in ExecResult? I'm a bit
> concerned that it was reportedly working in 2.1.15 (which is the version
> we're on now), and apparently broken in 2.1.16-17...and then fixed with
> a *new* command in 2.1.18cvs?
>
> I'm not sure how running
>
> ExecResult(/bin/sh -c ${dblquote}/bin/cat /proc/cpuinfo | grep
> Processor${dblquote})
>
> Is any different than
>
> ExecShellresult(/bin/cat /proc/cpuinfo | grep Processor)
>
> ...or am I missing something?
This is just a convenience so you don't have to use the /bin/sh
explicitly.
M
classes:
any::
hasbluesmokeloaded = ( ReturnsZero(/bin/sh -c
${dblquote}/sbin/lsmod | /bin/grep -q bluesmoke${dblquote}) )
And it works fine. Will upgrading to 2.1.17 break these class
definitions? Luckily we don't have any shell metacharacters in any
ExecResult statements right now, but I believe I'd be as suprised as
Nathan if I did
control:
any::
bluesmokemod = ( ExecResult(/bin/sh -c ${dblquote}/sbin/lsmod |
/bin/grep bluesmoke_mc${dblquote}) )
and it didn't work, since I'm explicitly spawning a shell to handle the
metacharacters.
Paul Krizak 5900 E. Ben White Blvd. MS 625
Advanced Micro Devices Austin, TX 78741
Linux/Unix Systems Engineering Phone: (512) 602-8775
Microprocessor Solutions Sector
> You might be able to replace most of the awk '{print $1}'-type calls
>
>with "cut" invocations:
>
> $ cut -d" " -f1 </proc/loadavg
> 0.00
>
>
There's various workarounds, but I'm frankly a lot more concerned about
the behavior change with respect to quoting (aka bug) between
2.1.15/2.1.16. :)
Paul Krizak 5900 E. Ben White Blvd. MS 625
Advanced Micro Devices Austin, TX 78741
Linux/Unix Systems Engineering Phone: (512) 602-8775
Microprocessor Solutions Sector
Chip Seraphine wrote:
No. It doesn't throw out anyhing. It simply doesn't understand them
because there is no shell.
In Unix you can execute a process in two ways: directly by asking the
kernel to start a process based on a certain program, or indirctly by
asking it to start a /bin/ssh image and execute the task as a subprogram
of that. The latter has several advantages (ability to use the shell to
parse arguments and use pipes etc) and disadvantages (security issues,
hidden assumptions).
M
If there is a problem I will fix it, but casting uncritical aspersions
just leads to a build up of frustrations in everyone.
M
On Fri, 2005-11-11 at 16:33 -0600, Chip Seraphine wrote:
> Brendan Strejcek wrote:
>
> > You might be able to replace most of the awk '{print $1}'-type calls
> >
> >with "cut" invocations:
> >
> > $ cut -d" " -f1 </proc/loadavg
> > 0.00
> >
> >
>
> There's various workarounds, but I'm frankly a lot more concerned about
> the behavior change with respect to quoting (aka bug) between
> 2.1.15/2.1.16. :)
>
>
I am attempting to use 2.1.17 and the latest. And I am running into
issues with these changes. I am not certain there it happened. I
have been looking through code to trying an figure it out.
>From what I can tell these two function
ExecResult|ReturnZero(/bin/sh -c "what ever your want") have worked
for quite some time. And in most cases I want to use my own shell so
I can do things like >/dev/null etc. (Maybe I can do this with
ExecShellResult result but I don't see any point in adding this since
sh worked great)
As far as the guy with piping problems *shrug* I have piped stuff
before without issue:
cf.portal: orca_running = ( ReturnsZero(${sh} -c '${ps} -aef |
grep -v grep | grep ${navigator_home}/orca/bin/orca >/dev/null 2>&1')
cf.portal: ReturnsZero(${sh} -c '${ps} -auxwww
2>/dev/null | grep -v grep | grep ${navigator_home}/orca/bin/orca
>/dev/null 2>&1') )
What sucks right now is that I have to change the way I am doing
quotes if I want to make use of the latest cfengine binary for what
ever reason. And for why I am not certain? I can't find it in the
code. I can't find when it changed in the code. But the following
use to work and now doesn't:
cf.binaries: getent = ( ExecResult(/bin/sh -c "PATH=${path}
${which} getent 2>/dev/null || echo \'no getent\'") )
Here is a comparison on the functions debugged:
-ExpandVarstring(/bin/sh -c "PATH=${path} ${which} getent 2>/dev/null
|| echo \'no getent\'")
-ExtractInnerVarString(path} ${which} getent 2>/dev/null || echo \'no
getent\'") - syntax verify
+HandleFunction: ExecResult(/bin/sh -c "PATH=${path} ${which} getent
2>/dev/null || echo \'no getent\'")
+FunctionStringToCode(ExecResult)
+ExpandVarstring(/bin/sh -c PATH=${path} ${which} getent 2>/dev/null
|| echo \'no getent\')
+ExtractInnerVarString(path} ${which} getent 2>/dev/null || echo \'no
getent\') - syntax verify
Notice in the second set of output the quote (") is missing around
PATH and getent\'.
So what I am suppose to do is this:
at = ( ExecResult("/bin/sh -c \"PATH=${path} ${which} at
2>/dev/null || echo \'no at\'\"") )
Now what I have is a situation where I am damned if I do and damned if
I don't. If I upgrade the cfengine my code stops working. If I
change the code my current cfengine binaries can't handle it. So I
don't know how to move forward.
Can someone help me find in the code where it changed so I can at
least patch my binaries to remove this?
Why are we breaking the parser so late in the game? Especially with
the start of Cfengine 3. Was this change an accident or what it
purposeful? I did as much reading on this change as I could find.
I would really like to see Cfengine 2 freeze except critical bugs.
> Perfect Order has changed its name to Versatile, which more
> correctly represents the combined capabilities of Versatile
> Mobile Systems and Perfect Order. From the data center to
> users on the move, Versatile turns data into knowledge.
>
> Please visit our new web site at http://www.versatile.com
>
--
Christian Pearce
If you send me a file you are having trouble with I can try to find the
solution. These little fragments with lots of undefined variables are
not helpful.
M