Shell variables in command line usage
Dag Wieers
Being used to tools like ssh or shmux, I was wondering why the examples
for command line usage are more complex wrt. escaping special characters.
[root@moria ansible]# ssh localhost 'echo $TERM'
dumb
I would expect the following to work:
[root@moria ansible]# ansible all -a 'echo $TERM'
127.0.0.1 | success | rc=0 >>
$TERM
But as the example shows, you need double escaping.
[root@moria ansible]# ansible all -a 'echo \$TERM'
127.0.0.1 | success | rc=0 >>
$TERM
[root@moria ansible]# ansible all -a "echo \\$TERM"
127.0.0.1 | success | rc=0 >>
xterm
This makes using the command line functionality much harder than one would
expect in the first place. What's more escaping does not even work with
single quotes, which is what one would use. Having to use double quotes
requires one to have double escaping.
Where is this coming from ? And can this be improved ?
Should I open an issue for this ?
--
-- dag wieers, d...@wieers.com, http://dag.wieers.com/
-- dagit linux solutions, in...@dagit.net, http://dagit.net/
[Any errors in spelling, tact or fact are transmission errors]
Michael DeHaan
Michael DeHaan
Dag Wieers
> Don't open an issue, it's entirely a shell thing.
>
> That example was about echoing the remote's term, not my local term on the remote machine.
Of course. But that's not what is happening in the example:
[root@moria ansible]# echo "$TERM"
xterm
[root@moria ansible]# echo "\$TERM"
$TERM
[root@moria ansible]# echo "\\$TERM"
\xterm
So if you do the example as provided in http://ansible.github.com/examples.html
ansible raleigh -m shell -a "echo \\$TERM"
You are sending "echo \xterm" to the remote bash. Which does:
[root@moria ansible]# echo "echo \\$TERM" | bash
xterm
Now, the whole problem here obviously is that the example makes it very
confusing, if we would use a different variable that is unique (e.g. $$)
you'll see what I mean
[root@moria ansible]# echo "$$"
5334
[root@moria ansible]# echo "\$$"
$$
[root@moria ansible]# echo "\\$$"
\5334
In fact what you would expect to work is this:
[root@moria ansible]# echo 'echo $$' | bash
11531
or
[root@moria ansible]# echo "echo \$$" | bash
11534
instead of
[root@moria ansible]# echo "echo \\$$" | bash
5334
I hope this makes it more clear what I meant. So that's why I am convinced
something is not being processed as it should.
Dag Wieers
> On Fri, 23 Mar 2012, Michael DeHaan wrote:
>
>> Don't open an issue, it's entirely a shell thing.
>>
>> That example was about echoing the remote's term, not my local term on the
>> remote machine.
>
> Of course. But that's not what is happening in the example:
-snip-
> I hope this makes it more clear what I meant. So that's why I am convinced
> something is not being processed as it should.
What I think is happening (and why using single quotes or single escaping
does not work) is that paramiko is escaping the commandline. I wonder if
this is configurable ?
I should really dig into the code now that I know this is a real issue.
Sorry for the noise...
Michael DeHaan
Michael DeHaan
Dag Wieers
> Sorry for the spam -- here's the crux of it. It should not be hard to
> fix at all.
>
> The system tends to treat module_args as an array in places. It's a
> string. There is no reason anything should see it as anything BUT a
> string (the modules like strings), so removing the things that treat
> args as an array is probably the best way to go.
The only piece I can find that interprets the string might be
shlex.split(), however using it as a string still does not work as
expected.
While it does work with a basic paramiko script.
----
#!/usr/bin/python
import sys
import paramiko
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('localhost')
stdin, stdout, stderr = client.exec_command(sys.argv[1])
print stdout.read()
----
[root@moria ansible]# python test.py 'echo $$'
31199
[root@moria ansible]# python test.py 'echo $$'
31218
[root@moria ansible]# python test.py "echo \$$"
31273
> Treat them as a string all the way through the stack.
I cannot find what's causing the behaviour we see in the most simple case
though.
PS I did fix the double quoting where ansible runs logger
seth vidal
I suspect this is due to the args being passed in as a file to the
commands running them. This was done b/c otherwise we would end up with
shell expansion from where you've run them and then shell expansion
again when exec_command ran them.
-sv
Michael DeHaan
module_args=shlex.split(options.module_args),
Dag Wieers
> This in /usr/bin/ansible is what I'm saying should be unnecessary, because strings are tolerated:
>
> module_args=shlex.split(options.module_args),
>
> (The argsfile stuff to transfer the data should not be hurting anything in theory…. the code should not have to split anything at all until the module picks it up on the other side)
>
> This results in the code trying to make the array back into a string and that is not necessary.
I doubt the above is the culprit. The command module is using sys.argv
directly. This was what confused me until Seth hinted this.
However, I noticed that the command module is running Popen in a shell,
and I think that's being a problem (this together potentially with the
shlex.split in the command module).
But what I lack to understand is how Popen integrates with Paramiko. I was
made to believe we were using exec_command() for remote commands, but
that's apparently not the case :-/
seth vidal
Dag Wieers <d...@wieers.com> wrote:
> On Fri, 23 Mar 2012, Michael DeHaan wrote:
>
> > This in /usr/bin/ansible is what I'm saying should be unnecessary,
> > because strings are tolerated:
> >
> > module_args=shlex.split(options.module_args),
> >
> > (The argsfile stuff to transfer the data should not be hurting
> > anything in theory…. the code should not have to split anything at
> > all until the module picks it up on the other side)
> >
> > This results in the code trying to make the array back into a
> > string and that is not necessary.
>
> I doubt the above is the culprit. The command module is using
> sys.argv directly. This was what confused me until Seth hinted this.
>
> However, I noticed that the command module is running Popen in a
> shell, and I think that's being a problem (this together potentially
> with the shlex.split in the command module).
>
> But what I lack to understand is how Popen integrates with Paramiko.
> I was made to believe we were using exec_command() for remote
> commands, but that's apparently not the case :-/
>
So here is what ansible does.
it sends over the module (in this case command) with a set of arguments.
Then it exec_command()s that module with those arguments on the remote
machine.
if you look in your syslog on the remote machine you will see what is
being run.
-sv
Michael DeHaan
Dag Wieers
> On Fri, 23 Mar 2012 15:31:03 +0100 (CET)
> Dag Wieers <d...@wieers.com> wrote:
>
>> But what I lack to understand is how Popen integrates with Paramiko.
>> I was made to believe we were using exec_command() for remote
>> commands, but that's apparently not the case :-/
>
> So here is what ansible does.
Thanks !
> it sends over the module (in this case command) with a set of arguments.
>
> Then it exec_command()s that module with those arguments on the remote
> machine.
>
> if you look in your syslog on the remote machine you will see what is
> being run.
So the example piece of code from command fails even locally:
----
#!/usr/bin/python
import sys
import subprocess
import traceback
try:
cmd = subprocess.Popen(sys.argv[1].split(), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = cmd.communicate()
except (OSError, IOError), e:
print e
sys.exit(1)
except:
print traceback.format_exc()
sys.exit(1)
print out.strip()
----
And you get:
[root@moria ansible]# python test2.py 'echo $$'
$$
This is where it happens.
If you do:
----
#!/usr/bin/python
import sys
import subprocess
import traceback
try:
cmd = subprocess.Popen([ 'echo $$' ], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = cmd.communicate()
except (OSError, IOError), e:
print e
sys.exit(1)
except:
print traceback.format_exc()
sys.exit(1)
print out.strip()
----
Then it magically works. Solution:
- Don't split, but put it in a list
- use shell=True
seth vidal
Dag Wieers <d...@wieers.com> wrote:
so two things:
1. If you specify the module name to ansible as 'shell' I believe it
says shell=True in there and does what you want.
2. the string vs list thing is involved too.
-sv
seth vidal
Michael DeHaan <michael...@gmail.com> wrote:
> Let's take the debugging discussion off list (or use IRC) and just
> include you/me/Seth.
>
> I doubt too many people are interested in it low level details, just
> in making sure that shell vars are happy for those that want to use
> them :)
I think that's exactly what a list is for - so people can find this
discussion later when they search for a question.
archiving for the future!! :)
-sv