replace module regexp with newline character

[John Harmon]

I need to use replace to find a regexp that includes \n; however, I keep getting an error.  How can I properly do this (don't even know if I am using back reference correct either, but if I can get past this error I think I can figure that part out)?

Thanks in advance!

---
# DNS Update For Ubuntu 18
- name: Update DNS servers
  replace:
    path: /etc/netplan/50-cloud-init.yaml
    regexp: ".*nameservers.*$\n.*addresses:(.*$)"
    replace: "\1 {{ dns1 }}, {{ dns 2}}"
  notify: "{{ handler_name }}"

Error:

TASK [dns_update : include_tasks] ********************************************************************************************************************************************************************************************************
fatal: [ansible-ub18]: FAILED! => {"reason": "Syntax Error while loading YAML.\n  found unknown escape character\n\nThe error appears to have been in '/etc/ansible/role/dns_update/tasks/Ubuntu18.yml': line 7, column 15, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    regexp: \".*nameservers.*$[\\n].*addresses:(.*$)\"\n    replace: \"\\1 {{ dns1 }}, {{ dns 2}}\"\n              ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"}

[John Harmon]

Sorry, last error message is slightly innaccurate.... I was testing a change in that one (difference is brackets [ ]):

TASK [dns_update : include_tasks] ********************************************************************************************************************************************************************************************************
fatal: [ansible-ub18]: FAILED! => {"reason": "Syntax Error while loading YAML.\n  found unknown escape character\n\nThe error appears to have been in '/etc/ansible/role/dns_update/tasks/Ubuntu18.yml': line 7, column 15, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    regexp: \".*nameservers.*$\\n.*addresses:(.*$)\"\n    replace: \"\\1 {{ dns1 }}, {{ dns 2}}\"\n              ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"}

[Matt Martz]

I'd recommend any time you need to use escape characters, that you use single quotes around the value, instead of double quotes.  It will resolve many of the issues with how pyyaml does it's parsing

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/85332d11-9309-4182-8d94-c47a1c1cc6a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Martz
@sivel
sivel.net

[John Harmon]

I have tried single quotes too around the regexp, but get a similar issue

TASK [dns_update : include_tasks] ********************************************************************************************************************************************************************************************************
fatal: [ansible-ub18]: FAILED! => {"reason": "Syntax Error while loading YAML.\n  found unknown escape character\n\nThe error appears to have been in '/etc/ansible/role/dns_update/tasks/Ubuntu18.yml': line 7, column 15, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    regexp: '.*nameservers.*$\\n.*addresses:(.*$)'\n    replace: \"\\1 {{ dns1 }}, {{ dns 2}}\"\n              ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"}

[Kai Stian Olstad]

If you add this sed to correct the formatting somewhat
ansible-playbook playbbok.yml | sed 's/\\n/\n/g'

you'll see that is pointing to your double quote in the replace line.


--
Kai Stian Olstad

[John Harmon]

Thx all.  Those helped.  I now get it to replace, but it replaces my string and not just my backref.   Not sure what I am doing wrong.  I think that is just the default behavior of replace.  Can't use quantifiers with negative look aheads/behinds.

[Kai Stian Olstad]

On 21.03.2019 20:51, John Harmon wrote:
>
> Thx all. Those helped. I now get it to replace, but it replaces my string
> and not just my backref. Not sure what I am doing wrong. I think that is
> just the default behavior of replace. Can't use quantifiers with negative
> look aheads/behinds.

It will be easier to help if you could show what do have in the file and what is should be after replace.


--
Kai Stian Olstad

[John Harmon]

I didn't post it so as to not overburden everyone with my issues; however, if it isn't a big issue I could use some help as I can't seem to get it working.  I am trying to replace the DNS servers, located in the second addresses line toward the bottom.

network:
    ethernets:
        enp0s3:
            addresses: [10.10.50.0/24]
            gateway4: 10.10.50.5
            dhcp4: no
            nameservers:
                    addresses: [10.1.2.3, 10.1.2.4]
    version: 2

[Kai Stian Olstad]

On 22.03.2019 16:11, John Harmon wrote:
>
>> It will be easier to help if you could show what do have in the file and
>> what is should be after replace.
>>
>>
>> --
>> Kai Stian Olstad
>>
>
> I didn't post it so as to not overburden everyone with my issues; however,

Regexp is not easy without having the actual text it's going to parse, especially when you don't know if your regexp is working.

> if it isn't a big issue I could use some help

Never is with all the relevant information.

> as I can't seem to get it
> working. I am trying to replace the DNS servers, located in the second
> addresses line toward the bottom.
>
> network:
> ethernets:
> enp0s3:
> addresses: [10.10.50.0/24]
> gateway4: 10.10.50.5
> dhcp4: no
> nameservers:
> addresses: [10.1.2.3, 10.1.2.4]
> version: 2

This made it a lot easier and only took a minute to solve(hopefully since it's not tested).
Try this :

regexp: '(.*nameservers.*$\n.*addresses:).*$'
replace: '\1 [{{ dns1 }}, {{ dns2 }}]'

--
Kai Stian Olstad

[John Harmon]

Thx Kai.

I tried it, and the regexp test shows that the regexp is good, but it doesn't replace the line.  It just says ok, and continues on.  Here is my slightly modified variant also doing the same thing (or to be more specific, not doing anything):

    regexp: 'nameservers:\n.*addresses:.*\[(.*)\]'
    line: '\1 {{ dns1 }}, {{ dns2 }}'

Result:

TASK [dns_update : Update DNS servers] ***************************************************************************************************************************************************************************************************
ok: [ansible-ub18]

File remains the same

[Kai Stian Olstad]

On 22.03.2019 17:47, John Harmon wrote:
>
> Thx Kai.
>
> I tried it, and the regexp test shows that the regexp is good, but it
> doesn't replace the line. It just says ok, and continues on. Here is my
> slightly modified variant also doing the same thing (or to be more
> specific, not doing anything):
>
> regexp: 'nameservers:\n.*addresses:.*\[(.*)\]'
> line: '\1 {{ dns1 }}, {{ dns2 }}'

Replace doesn't have line: to my knowledge.

> Result:
>
> TASK [dns_update : Update DNS servers]
> ***************************************************************************************************************************************************************************************************
> ok: [ansible-ub18]
>
>
> File remains the same

So, to check I created a little test with my regexp to see if it work, and it does.

$ more dns.yaml

network:
ethernets:
enp0s3:
addresses: [10.10.50.0/24]
gateway4: 10.10.50.5
dhcp4: no
nameservers:
addresses: [10.1.2.3, 10.1.2.4]
version: 2

$ cat test.yml
---
- hosts: localhost
vars:
dns1: 1.1.1.1
dns2: 2.2.2.2
gather_facts: no
tasks:
- replace:
path: dns.yaml

regexp: '(.*nameservers.*$\n.*addresses:).*$'
replace: '\1 [{{ dns1 }}, {{ dns2 }}]'

$ ansible-playbook test.yml

TASK [replace] *************************************************************
--- before: dns.yaml
+++ after: dns.yaml
@@ -5,5 +5,5 @@

gateway4: 10.10.50.5
dhcp4: no
nameservers:

- addresses: [10.1.2.3, 10.1.2.4]
+ addresses: [1.1.1.1, 2.2.2.2]
version: 2

changed: [localhost] => {
"changed": true
}

MSG:

1 replacements made


PLAY RECAP *****************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=0


$ cat dns.yaml

network:
ethernets:
enp0s3:
addresses: [10.10.50.0/24]
gateway4: 10.10.50.5
dhcp4: no
nameservers:

addresses: [1.1.1.1, 2.2.2.2]
version: 2


So it does work in my test.

--
Kai Stian Olstad

[John Harmon]

You are right.... I switched to lineinfile sometime between yesterday and today.  I switched back to replace due to your success (and I prefer to use it as I believe it will replace all found instances).

Seems to be working now, however, one follow-up question regarding backrefs.  I thought \1 replaced the content of group1?  Is that incorrect?  The content that was replaced doesn't  fall in a group (sorry, I lack experience with backrefs).

Here is my regex tester.  You can see the "full match" is replaced, and not the "Group 1": https://regex101.com/r/jfGg83/4

[John Harmon]

Oh, I think I misunderstood (as I suspected) how backrefs work with substitution.  I thought what was referenced was being replaced, when in reality it seems to just be putting back what we want to keep and just replacing the rest.

[Kai Stian Olstad]

Yes, all the things matching the regexp: line will be replaced with the content of the replace: line.
Since nameservers: and address: shouldn't be discard/replace, the parenthesis around them make it so that we can reference them as \1 in the replace: line.


--
Kai Stian Olstad