salt.states.file.replace (multiline) regular expression

[Aubrey Falconer]

Greetings all! Rather new to Salt, please forgive the simplistic nature of this inquiry.

Objective:
Change "enabled=0" to "enabled=1" in a text file using a regular expression.

Reference:
Relevant Docs: http://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html
Sample Regex: http://regexr.com/3a5v3
Sample File: http://hastebin.com/edinuxesaq.sm
Sample Template: http://hastebin.com/xubovijevo.sm

Questions:

1) What's the best way to test a salt state under development?
I initially attempted to test various replace approaches using cli invocation:
salt-call -l all file.replace path="/etc/yum.repos.d/remi.repo" pattern="/(\[remi-php55\][\s\S]*?enabled)=([0-9])/" repl="$1=1" bufsize=file
This, and every variation of it I could think of, failed with nonexplanatory errors.
I'm now using a template for testing, invoked using sudo salt-call -l all state.template '/srv/salt/php-55/init.sls'. This works, but I had to manually comment out several dependency directives so it could run with the limited scope. Is the best way to test new templates, or am I missing something?

2) Why doesn't cli invocation work?
Here's a sample pulled from the official documentation: salt '*' file.replace /etc/httpd/httpd.conf pattern='LogLevel warn' repl='LogLevel info'

Here's the simplest possible testcase I could reduce things to - each of the following commands fails with errors such as "Passed invalid arguments: first argument must be string or compiled pattern". What's the right way to do masterless cli invocation?

echo '12342' > /tmp/test.txt
salt-call -l all file.replace path="/etc/yum.repos.d/remi.repo" pattern="/(\[remi-php55\][\s\S]*?enabled)=([0-9])/" repl="$1=1" bufsize=file
salt-call file.replace /tmp/test.txt pattern='42' repl='43'
salt-call file.replace /tmp/test.txt pattern='/42/' repl='43'
salt-call file.replace /tmp/test.txt pattern=/42/ repl=43
salt-call file.replace /tmp/test.txt '42' '43'
salt-call file.replace /tmp/test.txt 42 43

3) Any idea why this regex finds no matches?

(sample file) cat /tmp/search.txt

asdf42

asdf43

asdf44

asdf45

(sample template)

/tmp/search.txt:

  file.replace:

    - pattern: |

        42([\s\S]*)44

    - repl: xxx

    - bufsize: file

(sample output)

          ID: /tmp/search.txt

    Function: file.replace

      Result: True

     Comment: No changes were made

4) I feel the sample template referenced above is failing for the same reason that the simple 42 regex is failing - multiline expressions don't seem to work even when bufsize is set to file. Please let me know if there's something simple being overlooked here.

[Aubrey Falconer]

Anyone have a functional sample of file.replace cli invocation, or a working multiline regex?

Many thanks!

[Stephen Spencer]

The file.replace state function is a thin wrapper around the execution module function, so I think this example will work for either (the example just happens to be defined in a state file)

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You know, I used to think it was awful that life was so unfair. Then I thought, wouldn't it be much worse if life were fair, and all the terrible things that happen to us come because we actually deserve them? So, now I take great comfort in the general hostility and unfairness of the universe.

[Stephen Spencer]

and that example would be...

{% set outer = loop.index -%}

{% for k, v in pillar.sysconfig.pg.items() %}

pg_conf_{{k}}_{{outer}}:

  file.replace:•

    - name: /var/lib/pgsql/{{pgv_dot}}/data/postgresql.conf

    - pattern: '#?(\s*{{k}}\s*=)\s*.*(#\s.*)?'

    - repl: \1 '{{v}}'

    - show_changes: True

    - watch_in:

      - service: postgres_service_{{outer}}

{% endfor -%}

Here be the tricksies to it:

. The pattern must always be (single) quoted.  Otherwise the YAML parser will soil its trousers.  

. The replacement value (probably) doesn't always have to be ; however, if you are a clever regexian and like to use re.sub group replacement, you have to express it as shown above.  The reference should never be quoted.  I haven't figured out why; it Just Doesn't Work. In the example the '{{v}}' is quoted because of the on/off values provided from the pillar (what the outer loop is iterating over) that otherwise would be transposed to True/False because jinja is insane.

-S

[Stephen Spencer]

Oh, and the python backref syntax is sed-ish (\1) rather than perl-ish ($1).

[Aubrey Falconer]

Thx, useful tips! Your single line regex example is working great on a pgsql.conf file I pointed it to, but the multiline variant I just tested is still not functioning properly.

This works: (replaces a value on one line)

/tmp/pgsql.conf:
  file.replace:
    - pattern: '#?(\s*autovacuum\s*=)\s*.*(#\s.*)?'
    - repl: \1 autovacuuuum
    - show_changes: True
    - bufsize: file

 

This doesn't: (attempts to match two lines, finds nothing)

/tmp/pgsql.conf:
  file.replace:
    - pattern: '(#autovacuum_max_workers.*)\n+?.*(#.*)'
    - repl: \1 autovacuuuum
    - show_changes: True
    - bufsize: file

Here's the multiline variant working perfectly in a web-based regex engine: http://regexr.com/3a7ou

Further ideas?

[Stephen Spencer]

Ah.  That's right!  Multiline.  Ok, so you're close:

 Add flags

[Aubrey Falconer]

Good point. Finally working!

/tmp/pgsql.conf:
  file.replace:
    - pattern: '#?(\s*autovacuum\s*=)\s*.*(#\s.*)?'

    - flags: ['DOTALL', 'MULTILINE']

    - repl: \1 autovacuuuum
    - show_changes: True
    - bufsize: file

[Stephen Spencer]

Oh, right!  Multiline!

You're very close then.  Then argument you want to use is "flags" and needs to be a list of one or more of the following:

IGNORECASE

LOCALE (make \w \W \b \B locale dependent)

MULTILINE (dingdingding)

DOTALL ('.' matches any character including newline)

VERBOSE (ignore whitespace and comments)

UNICODE (\w \W \b \B locale dependent)

So that would be something along the lines of:

pg_conf_{{k}}_{{outer}}:

  file.replace:•

    - name: /var/lib/pgsql/{{pgv_dot}}/data/postgresql.conf

    - pattern: '#?(\s*{{k}}\s*=)\s*.*(#\s.*)?'

    - repl: \1 '{{v}}'

    - show_changes: True

    - flags:

      - [this space intentionally left blank]

        - MULTILINE

        - IGNORECASE # inserted for further structural demonstration

    - watch_in:

      - service: postgres_service_{{outer}}

[Aubrey Falconer]

Thx for your suggestions, can't believe this is still stumping me. Two issues:

1) CLI invocation does not work, period. Example:

cat /tmp/search.txt
asdf42
asdf43
asdf44
asdf4 
 

salt-call --local file.replace /tmp/search.txt pattern='42' repl='what' flags='[DOTALL, MULTILINE]'

Passed invalid arguments: first argument must be string or compiled pattern

 

 This is exactly what the documentation ( http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.file.html#salt.modules.file.replace ) describes doing. Anyone know what gives?

2) State invocation does not work reliably, and I can't figure out how to debug it.

@Stephen Spencer's sample functions as it should, but dozens of my (incredibly simple) testcase variants fail to match anything. Example:

cat /tmp/search.txt
asdf42
asdf43
asdf44
asdf4 

echo /tmp/search.txt > /tmp/search2.txt

/tmp/search.txt:
  file.replace:
    - pattern: |
        42([\s\S]*)44
    - repl: xxx

    - show_changes: True
    - bufsize: file

    - flags: ['DOTALL', 'MULTILINE']

/tmp/search2.txt:
  file.replace:
    - pattern: '(42)(.*)(44)'
    - repl: foo

    - flags: ['DOTALL', 'MULTILINE']

    - show_changes: True
    - bufsize: file

local:
----------

          ID: /tmp/search.txt
    Function: file.replace
      Result: True
     Comment: No changes were made

     Started: 23:49:02.407918
    Duration: 12.329 ms
     Changes:
----------
          ID: /tmp/search2.txt

    Function: file.replace
      Result: True
     Comment: No changes were made

     Started: 23:49:02.420545
    Duration: 1.734 ms
     Changes:
----------

Same expressions in a Python regex tester such as http://pythex.org match correctly. How do I zero in on what is going wrong?

[David Ward]

Did we get this working?

It seems like it might never work:

https://github.com/saltstack/salt/issues/7999

[Seth House]

No one has tackled that one yet. The fix is straightforward and
detailed in the issue comments: remove fileinput and read the full
file into memory. The multiline flag will then work as intended. If
someone is willing to put in the time I'd be happy to work with
him/her on it.

[Loren Gordon]

Actually, fileinput is out already, but the file is still read line by line (so multi-line regex doesn't work currently).

-Loren

[Colton Myers]

Loren got this today! https://github.com/saltstack/salt/pull/28174

--
Colton Myers
Core Engineer, SaltStack
@basepi on Twitter/Github/IRC