Is it intentional to authorize Python escaping in YAML playbooks?

[Nicolas Grilly]

I've noticed that when I use an escape sequence like "\n" in my YAML playbooks, it is interpreted by Python modules like a newline, instead of a two separated characters "\" and "n". Is it intended? It seems redundant because YAML already offers its own escaping mechanism.

I understand the related code is in lib/runner/__init__.py, in Runner._copy_module:

    encoded_args = "\"\"\"%s\"\"\"" % module_args.replace("\"","\\\"")

Maybe it could be replaced by something like this:

    encoded_args = module_args.encode('string_escape')

What is your opinion about this?

Cheers,

Nicolas

[Michael DeHaan]

Can you please show me the example of this in a playbook and how you are observing this behavior (playbook output in verbose mode, etc)?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Michael DeHaan <mic...@ansibleworks.com>
CTO, AnsibleWorks, Inc.
http://www.ansibleworks.com/

[Nicolas Grilly]

On Wednesday, May 22, 2013 3:28:43 PM UTC+2, Michael DeHaan wrote:

Can you please show me the example of this in a playbook and how you are observing this behavior (playbook output in verbose mode, etc)?

Here is an example with this simple playbook:

- hosts: all

  tasks:

    - copy: content=hello\nworld dest=/vagrant/testcopy.txt

I run it using:

$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -i hosts playbook.yml

And get:

TASK: [copy content=hello\nworld dest=/vagrant/testcopy.txt] ****************** 

failed: [127.0.0.1] => {"failed": true}

msg: this module requires key=value arguments (['content=hello', 'world', 'dest=/vagrant/testcopy.txt', 'src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source', 'original_basename=tmpK0KdKX'])

I do not expect it to fail.

I had a look to the module uploaded to the server after replacement of MODULE_ARGS and here is its value:

MODULE_ARGS = """content=hello\nworld dest=/vagrant/testcopy.txt src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source original_basename=tmpK0KdKX"""

I think this is incorrect. "\n" should be escaped like this:

MODULE_ARGS = """content=hello\\nworld dest=/vagrant/testcopy.txt src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source original_basename=tmpK0KdKX"""

or like this (by using a Python raw string prefixed with a "r"):

MODULE_ARGS = r"""content=hello\nworld dest=/vagrant/testcopy.txt src=/home/vagrant/.ansible/tmp/ansible-1369233251.42-198707021399580/source original_basename=tmpK0KdKX"""

Any advice?

Thanks for your help.

Nicolas

[Michael DeHaan]

Ah ok, don't use content in this case without quoting it, and you should be fine.

It's also preferred you use "src" anyway, content was a recent addition. 

Nicolas

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nicolas Grilly]

On Wed, May 22, 2013 at 5:47 PM, Michael DeHaan <mic...@ansibleworks.com> wrote:

Ah ok, don't use content in this case without quoting it, and you should be fine.

It's also preferred you use "src" anyway, content was a recent addition. 

The problem is not specific to module copy with the content arg. I used this just as an example. The problem happens with all Python modules. When args are "injected" in the Python module before being uploaded to the server, they should be escaped but are not.

Here is another example with lineinfile:

- hosts: all

  tasks:

    - copy: content='hello' dest=/vagrant/test.txt

    - lineinfile: dest=/vagrant/test.txt regexp='^hello' line='hello\nworld'

After running this playbook, test.txt should contain :

$ cat test.txt

hello\nworld

Instead, I got:

$ cat test.txt

hello

world

The reason why is because "\n" is not escaped before being "injected" in MODULE_ARGS and thus is interpreted by Python as a newline instead of just two characters "\" and "n".

This behavior is undocumented, and I understand that Ansible tries hard to not be Python specific. For most use cases, one should just need to write YAML playbooks and use the built-in modules. But this small problem makes the playbook implicitly dependent on Python literal string escaping rules.

What would you think of a patch trying to replace (in Runner._copy_file):

    encoded_args = "\"\"\"%s\"\"\"" % module_args.replace("\"","\\\"")

 by something like this:

    encoded_args = repr(module_args)

An example to illustrate the difference:

$ python

>>> module_args = r'regexp="hello\nworld"'

>>> print "\"\"\"%s\"\"\"" % module_args.replace("\"","\\\"")

"""regexp=\"hello\nworld\""""

>>> print repr(module_args)

'regexp="hello\\nworld"'

This is the end of this long message. Sorry!

[Michael DeHaan]

Sure, how about sending me a pull request so we can test it out and make sure it doesn't break anything?

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nicolas Grilly]

On Wed, May 22, 2013 at 8:34 PM, Michael DeHaan <mic...@ansibleworks.com> wrote:

Sure, how about sending me a pull request so we can test it out and make sure it doesn't break anything?

Ok. I can give it a try, but I expect such a change to break some examples of lineinfile that use backslash in regexp. Would you be ok to adapt the examples if it's necessary (basically replacing double backslash by a single backslash), or would it be a show stopper?

[Michael DeHaan]

Backrefs are a pretty new feature (I believe 1.2) in lineinfile, so I'm ok with a patch that also updates them.   Would be good if you can test those too if you can.

Thanks!

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nicolas Grilly]

On Wednesday, May 22, 2013 8:34:59 PM UTC+2, Michael DeHaan wrote:

Sure, how about sending me a pull request so we can test it out and make sure it doesn't break anything?

Done :)

Here is the pull request: https://github.com/ansible/ansible/pull/3005

[lu...@inf.ufrgs.br]

Hi,

I would disagree with Nicolas in that \<something> escaping is not Python specific. Indeed, it seems to be theYAML-way
to escape control characters according to YAML (see: http://yaml.org/spec/current.html#id2517668). It is also the natural way
of regular expressions (http://www.regular-expressions.info/reference.html)

The command bellow should generate line breaks and tabs, and it used to work with previous versions of ansible, both from
command line or from a playbooks.

tasks:
   - copy: content="hello\nworld" dest="/some/file"
   - lineinfile: dest=/vagrant/test.txt regexp='^hello' line='hello\tworld'

Recently, I upgraded ansible to 1.2.2 (pip) and 1.3 (git) and now it is broken. I've tried other forms of escaping (\x09, \u0009), both
with single and double quotes, and none have worked here. The escape sequence is always preserved in its literal form.

Is there another standard way of escaping control characters that I'm missing?
If not possible to revert to the previous (and correct in my PoV) behavior, I would appreciate any hints in how to do proper
do escaping of control chars in string parameters.

Best regards,
Luciano

[Michael DeHaan]

Doesn't seem likely to have changed:

https://github.com/ansible/ansible/commits/devel/library/files/lineinfile

I suspect you were using something earlier than 1.2.2 before maybe, back when it didn't do backrefs.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Luciano Cavalheiro]

Hi,

I think I've misread YAML specifications. Actually it only does expand escape sequences (\c, \x**, \u****) for double quoted scalar blocks.

So, while using double-quotes in the example bellow, actually, from PoV of YAML's parser it is a plain scalar block, because
it does not start with double-quotes.

tasks:
    copy:  content="beforetab\x09after" dest="/some/file"

I've found two solutions for the problem:

1) Convert value to double-quote scalar block (must escape any inner double-quotes in the block or replace by single-quotes):

tasks:
    copy:  "content='beforetab\x09after' dest='/some/file'"

2) Express copy parameters as a dict and use double-quotes for each value that need to use escape sequences.
tasks:
    copy: 
       content: "beforetab\x09after"
       dest: /some/file


However, I'm not sure solution 2 is officially supported or is a hidden feature of ansible. I was not able to find this
information in documentation currently available for playbooks.

BTW, CLI is also working. I forgot expansion of escape sequences does not occur depending on the used unix shell. So in
bash, for instance, one need to Ctrl+V<tab> to insert tab in string, instead of using \t. My mistake.

Best regards,
Luciano

[Michael DeHaan]

Best to just transfer a file and not use "content" IMHO.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Luciano Cavalheiro]

I would agree, except when you need to generate pretty small and somewhat formatted content on-the-fly. Ex.: passing some form of credentials
to remote node...

Anyway, I'm using lineinfile. I just used 'copy' in the example because I thought it would became simpler to explain the issue with YAML.

Thanks for your help,

[]s

[Nicolas Grilly]

Hi,

On Tuesday, July 16, 2013 9:44:03 PM UTC+2, Luciano Cavalheiro wrote:

I would disagree with Nicolas in that \<something> escaping is not Python specific. Indeed, it seems to be theYAML-way
to escape control characters according to YAML (see: http://yaml.org/spec/current.html#id2517668). It is also the natural way
of regular expressions (http://www.regular-expressions.info/reference.html)

A lot of programming languages, including Python and YAML, have in common to use a backslash to start an escape sequence, and yet they have different grammars and interpret what follows the backslash in different ways. This is why I made a distinction between Python escape sequences and YAML escape sequences: they both start with a backslash, still they do not share the exact same syntax. 

YAML escape sequences must be used in playbooks, instead of Python escape sequences, because Ansible modules are language agnostic as much as possible.

When you use a YAML escape sequence in your playbook, Ansible converts it to Unicode in memory, and converts it back to a Python escape sequence injected in the module transmitted to the server.

Best regards,

Nicolas