lineinfile problem

[mark....@dsv.com]

I have a problem where a line that contains a certain space delimited string must exist in a file.  The string can be anywhere in the line: start, middle, or end.  If a line that contains that string does not exist, then a line with that string needs to be inserted into the file.  I have been trying to use the lineinfile module to solve this, but so far I have been unsuccessful.  Below is the code I have tried:

- lineinfile:

dest: "/path/to/myfile"
regexp: '^(.*)(\s*this_string_must_exist)(\s+.*)$'
line: '\g<1> this_string_must_exist\g<3>'
backrefs: true
state: present

I have tried many variations of the regular expression and line, but nothing I have tried totally achieves what I am looking for.  I have verified that the regular expression can locate a line with the string regardless of where the string is within the line, and if such a line is located it is not modified.  But I have not been successful in getting a new line inserted if it does not already exist.
Does anyone have any suggestions?
-Mark

[Kai Stian Olstad]

The reason it doesn't work is explained in the documentation.

https://docs.ansible.com/ansible/lineinfile_module.html#options

In the comments about the backrefs:
"... and if the regexp doesn't match anywhere in the file, the file will
be left unchanged."

So you'll need to find a way to do it without backrefs: yes if you are
going to use lineinfile.

--
Kai Stian Olstad

[mark....@dsv.com]

    Yes, I finally spotted that immediately after I posted my initial query (when all else fails, read the documentation).  Now I am trying to use lineinfile (or maybe grep) to look for the string and register the results, then use a second lineinfile to add a line if the results of the first lineinfile indicates that it is missing.  This is a bit complicated because I have several strings that I need to look for and I want to use 'with_items' to store them.
    -Mark

[Jinesh Choksi]

Hi Mark,

I found the following https://relativkreativ.at/articles/how-to-use-ansibles-lineinfile-module-in-a-bulletproof-way very useful when I needed to find a reliable way to edit lines in a configuration file with Ansible tasks.

See if it helps you understand how to use lineinfile in a bulletproof way.

Regards,

Jinesh 

[mark....@dsv.com]

    Your article had interesting information, but it doesn't look like it will help.  I tried using negative look-ahead, but got nowhere with it.
    Now I am trying to use egrep to scan the file, and then use lineinfile to act upon the results.  I put together the below snippet for testing:

  - shell: chdir=~/tmp egrep "^.*\s*{{item}}\s+.*$" mytestfile
    register: result
    changed_when: false
    failed_when: false
    with_items:
        - "string2"
        - "string6"
        - "string7"

  - debug: var={{item}}
    with_items:
        - result

    This returns the following:

TASK [debug] *******************************************************************
ok: [localhost] => (item=result) => {
    "item": "result",
    "result": {
        "changed": false,
        "msg": "All items completed",
        "results": [
            {
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": false,
                "cmd": "egrep \"^.*\\s*string2\\s+.*$\" mytestfile",
                "delta": "0:00:00.003006",
                "end": "2017-01-26 22:33:27.722164",
                "failed": false,
                "failed_when_result": false,
                "invocation": {
                    "module_args": {
                        "_raw_params": "egrep \"^.*\\s*string2\\s+.*$\" mytestfile",
                        "_uses_shell": true,
                        "chdir": "/home/pdxmft/tmp",
                        "creates": null,
                        "executable": null,
                        "removes": null,
                        "warn": true
                    },
                    "module_name": "command"
                },
                "item": "string2",
                "rc": 0,
                "start": "2017-01-26 22:33:27.719158",
                "stderr": "",
                "stdout": "string1 string2 string3",
                "stdout_lines": [
                    "string1 string2 string3"
                ],
                "warnings": []
            },
            {
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": false,
                "cmd": "egrep \"^.*\\s*string6\\s+.*$\" mytestfile",
                "delta": "0:00:00.003093",
                "end": "2017-01-26 22:33:27.837867",
                "failed": false,
                "failed_when_result": false,
                "invocation": {
                    "module_args": {
                        "_raw_params": "egrep \"^.*\\s*string6\\s+.*$\" mytestfile",
                        "_uses_shell": true,
                        "chdir": "/home/pdxmft/tmp",
                        "creates": null,
                        "executable": null,
                        "removes": null,
                        "warn": true
                    },
                    "module_name": "command"
                },
                "item": "string6",
                "rc": 1,
                "start": "2017-01-26 22:33:27.834774",
                "stderr": "",
                "stdout": "",
                "stdout_lines": [],
                "warnings": []
            },
            {
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": false,
                "cmd": "egrep \"^.*\\s*string7\\s+.*$\" mytestfile",
                "delta": "0:00:00.002986",
                "end": "2017-01-26 22:33:27.951455",
                "failed": false,
                "failed_when_result": false,
                "invocation": {
                    "module_args": {
                        "_raw_params": "egrep \"^.*\\s*string7\\s+.*$\" mytestfile",
                        "_uses_shell": true,
                        "chdir": "/home/pdxmft/tmp",
                        "creates": null,
                        "executable": null,
                        "removes": null,
                        "warn": true
                    },
                    "module_name": "command"
                },
                "item": "string7",
                "rc": 0,
                "start": "2017-01-26 22:33:27.948469",
                "stderr": "",
                "stdout": "string7 string8 string9",
                "stdout_lines": [
                    "string7 string8 string9"
                ],
                "warnings": []
            }
        ]
    }
}

    What I would like to do now is walk through the results[] list and extract the "rc" and "item" values, something along the line of this:

  - debug: var={{item.rc}}
    with_items:
        - result.results[*]

    But of course that does not work.  So how can I walk through a list in this manner?  I looked at 'with_subelements' but was unable to put something together there.
    -Mark
 

[Johannes Kastl]

On 27.01.17 00:01 mark....@dsv.com wrote:

> What I would like to do now is walk through the results[] list and
> extract the "rc" and "item" values, something along the line of
> this:
>
> - debug: var={{item.rc}}
> with_items:
> - result.results[*]
>
> But of course that does not work. So how can I walk through a list
> in this manner?

First guess: Drop the "[*]". Ansible knows this is a list and should
use each element as "item".

Johannes

[Jinesh Choksi]

File: example_file.txt

string1 string2 string3

string4 string5 string0

string7 string8 string9

File: run.sh

#!/usr/bin/env bash

ansible-playbook -vvv -i 'localhost,' -c local test.yml

File: test.yml

- hosts:

    - localhost

  tasks:

    - name: "Slurp the file we wish to search"

      slurp:

        src: ./example_file.txt

      register: slurped_file

    - name: "To avoid base64 decoding multiple times, we do it once"

      set_fact: plain_file="{{ slurped_file['content'] | b64decode }}"

#    - name: "You can use this task to test your regex"

#      debug: msg="{{ plain_file | regex_search('string7') is none}}"

    - name: "Insert the wanted line into the file as it doesn't exist"

      lineinfile:

        dest: ./example_file.txt

        line: "something {{ item }} somethingelse"

        state: present

      when: plain_file | regex_search(item) is none

      with_items:

        - string6

        - string1

        - string8

        - foobar

Example output:

$  ./run.sh

Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: test.yml *************************************************************

1 plays in test.yml

PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************

Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: algomi-deploy

<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950 `" && echo ansible-tmp-1485515755.89-254326766505950="` echo ~/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950 `" ) && sleep 0'

<localhost> PUT /tmp/tmpP7Zu66 TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/setup.py

<localhost> EXEC /bin/sh -c 'chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/setup.py && sleep 0'

<localhost> EXEC /bin/sh -c '/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/setup.py; rm -rf "/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/" > /dev/null 2>&1 && sleep 0'

ok: [localhost]

TASK [Slurp the file we wish to search] ****************************************

task path: /opt/home/algomi-deploy/test.yml:4

Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/network/basics/slurp.py

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: algomi-deploy

<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445 `" && echo ansible-tmp-1485515756.18-190467128946445="` echo ~/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445 `" ) && sleep 0'

<localhost> PUT /tmp/tmpUjpiaz TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/slurp.py

<localhost> EXEC /bin/sh -c 'chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/slurp.py && sleep 0'

<localhost> EXEC /bin/sh -c '/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/slurp.py; rm -rf "/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/" > /dev/null 2>&1 && sleep 0'

ok: [localhost] => {

    "changed": false,

    "content": "c3RyaW5nMSBzdHJpbmcyIHN0cmluZzMKc3RyaW5nNCBzdHJpbmc1IHN0cmluZzAKc3RyaW5nNyBzdHJpbmc4IHN0cmluZzkK",

    "encoding": "base64",

    "invocation": {

        "module_args": {

            "src": "./example_file.txt"

        },

        "module_name": "slurp"

    },

    "source": "./example_file.txt"

}

TASK [To avoid base64 decoding multiple times, we do it once] ******************

task path: /opt/home/algomi-deploy/test.yml:10

ok: [localhost] => {

    "ansible_facts": {

        "plain_file": "string1 string2 string3\nstring4 string5 string0\nstring7 string8 string9\n"

    },

    "changed": false,

    "invocation": {

        "module_args": {

            "plain_file": "string1 string2 string3\nstring4 string5 string0\nstring7 string8 string9\n"

        },

        "module_name": "set_fact"

    }

}

TASK [Insert the wanted line into the file as it doesn't exist] ****************

task path: /opt/home/algomi-deploy/test.yml:17

Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/files/lineinfile.py

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: algomi-deploy

<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270 `" && echo ansible-tmp-1485515756.42-101048875702270="` echo ~/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270 `" ) && sleep 0'

<localhost> PUT /tmp/tmpPv_qtT TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/lineinfile.py

<localhost> EXEC /bin/sh -c 'chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/lineinfile.py && sleep 0'

<localhost> EXEC /bin/sh -c '/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/lineinfile.py; rm -rf "/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/" > /dev/null 2>&1 && sleep 0'

changed: [localhost] => (item=string6) => {

    "backup": "",

    "changed": true,

    "diff": [

        {

            "after": "",

            "after_header": "./example_file.txt (content)",

            "before": "",

            "before_header": "./example_file.txt (content)"

        },

        {

            "after_header": "./example_file.txt (file attributes)",

            "before_header": "./example_file.txt (file attributes)"

        }

    ],

    "invocation": {

        "module_args": {

            "backrefs": false,

            "backup": false,

            "content": null,

            "create": false,

            "delimiter": null,

            "dest": "./example_file.txt",

            "directory_mode": null,

            "follow": false,

            "force": null,

            "group": null,

            "insertafter": null,

            "insertbefore": null,

            "line": "something string6 somethingelse",

            "mode": null,

            "owner": null,

            "regexp": null,

            "remote_src": null,

            "selevel": null,

            "serole": null,

            "setype": null,

            "seuser": null,

            "src": null,

            "state": "present",

            "unsafe_writes": null,

            "validate": null

        },

        "module_name": "lineinfile"

    },

    "item": "string6",

    "msg": "line added"

}

skipping: [localhost] => (item=string1)  => {

    "changed": false,

    "item": "string1",

    "skip_reason": "Conditional check failed",

    "skipped": true

}

skipping: [localhost] => (item=string8)  => {

    "changed": false,

    "item": "string8",

    "skip_reason": "Conditional check failed",

    "skipped": true

}

Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/files/lineinfile.py

<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938 `" && echo ansible-tmp-1485515756.52-54649606538938="` echo ~/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938 `" ) && sleep 0'

<localhost> PUT /tmp/tmpAvxVae TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/lineinfile.py

<localhost> EXEC /bin/sh -c 'chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/lineinfile.py && sleep 0'

<localhost> EXEC /bin/sh -c '/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/lineinfile.py; rm -rf "/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/" > /dev/null 2>&1 && sleep 0'

changed: [localhost] => (item=foobar) => {

    "backup": "",

    "changed": true,

    "diff": [

        {

            "after": "",

            "after_header": "./example_file.txt (content)",

            "before": "",

            "before_header": "./example_file.txt (content)"

        },

        {

            "after_header": "./example_file.txt (file attributes)",

            "before_header": "./example_file.txt (file attributes)"

        }

    ],

    "invocation": {

        "module_args": {

            "backrefs": false,

            "backup": false,

            "content": null,

            "create": false,

            "delimiter": null,

            "dest": "./example_file.txt",

            "directory_mode": null,

            "follow": false,

            "force": null,

            "group": null,

            "insertafter": null,

            "insertbefore": null,

            "line": "something foobar somethingelse",

            "mode": null,

            "owner": null,

            "regexp": null,

            "remote_src": null,

            "selevel": null,

            "serole": null,

            "setype": null,

            "seuser": null,

            "src": null,

            "state": "present",

            "unsafe_writes": null,

            "validate": null

        },

        "module_name": "lineinfile"

    },

    "item": "foobar",

    "msg": "line added"

}

PLAY RECAP *********************************************************************

localhost                  : ok=4    changed=1    unreachable=0    failed=0

Final contents of the example_file.txt file:

string1 string2 string3

string4 string5 string0

string7 string8 string9

something string6 somethingelse

something foobar somethingelse

- The above slurps a file from the "remote" node.

- It converts the base64 encoded contents of the file into plain text so that we can regex search it

- It uses lineinfile to insert lines using with_items when the regex_search of the plain test file doesn't match an item

The downsides of the above play are that the slurping a large file may not be possible for you due to memory restrictions. Also, you weren't clear on what should be inserted into the file if the sought for string was not found and so the solution may not be complete.

Note: the regex_search filter is something new (i.e. not in the docs yet). I didn't know about it until I came across this PR https://github.com/ansible/ansible/pull/14696

I ran the above on ansible 2.2.1.0.

[mark....@dsv.com]

    I was not aware of the slurp module prior to this.  That combined with the regex_search filter certainly opens many avenues for searching files.  I can see where this could be problematic for large files, but in this case that is not an issue.  I am running Ansible version 2.2.0.0 and that appears to include the regex_search filter.  This solution works for me, so I will go with it.
    I looked around for more information about the regex_search filter, but found very little.  It is stored in the ansible/plugins/filter/core.py file if you want to look at it and see what other undocumented filters are available.
    -Mark