Multiline Regex Capture matching multiple values

[Chris Gadd]

Hi,

The documentation for the Multiline Regex Capture log filter suggests that multiple matches are possible:

When multiple values match, then $data.key will equal all of the matched values, separated by a newline.

I've tried a few different variations but can't get this to work. Should it find multiple values in a single step? Or does it mean when used as a global filter across many steps, or across many nodes? Or do you need to define more than 2 capture groups, one for each match?

Would love to see an example, eg capturing "hello" and "hej" from this data:

message1 hello
message2 hej

Sample job (which doesn't capture anything):

- defaultTab: output

  description: ''

  executionEnabled: true

  id: 697d5792-8fd0-40e8-b6ab-6a9c0af16a80

  loglevel: INFO

  name: regex tester

  nodeFilterEditable: false

  plugins:

    ExecutionLifecycle: null

  scheduleEnabled: true

  sequence:

    commands:

    - configuration:

        command: echo -e "message1 hello\nmessage2 hej"

      description: create output

      nodeStep: true

      plugins:

        LogFilter:

        - config:

            hideOutput: 'false'

            logData: 'true'

            name: myvar

            regex: \w+ (\w+)

          type: key-value-data-multilines

      type: localexec

    keepgoing: false

    strategy: node-first

  uuid: 697d5792-8fd0-40e8-b6ab-6a9c0af16a80

thanks,

chris

[rac...@rundeck.com]

Ho Chris,

Testing your job, I see that the regex always interprets your command as a one-line print, I tested using “normal” regex log output filter, and works as wanted using this regex definition: (.+)\s(.+), please check this job definition.

- defaultTab: output
  description: ''
  executionEnabled: true
  id: 697d5792-8fd0-40e8-b6ab-6a9c0af16a80
  loglevel: INFO
  name: regex tester
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - configuration:
        command: echo -e 'message1 hello\nmessage2 world'
      description: create output
      nodeStep: true
      plugins:
        LogFilter:
        - config:
            invalidKeyPattern: \s|\$|\{|\}|\\
            logData: 'true'
            regex: (.+)\s(.+)
          type: key-value-data
      type: localexec
    keepgoing: false
    strategy: node-first
  uuid: 697d5792-8fd0-40e8-b6ab-6a9c0af16a80

And here the result.

Hope it helps!

​

[Gadd, Chris, Vodafone NZ]

Yeah, I understand that works, but in that example you have two keys and two values, whereas the documentation talks about one key with multiple values. A better example would be trying to capture both ‘hello’ and ‘world’ with this text:

message hello

message world

 

Still, it’s the multiline regex capture I’m interested in, and that’s the only one that mentions this capability of capturing multiple values. It sounds like it might be useful for some output like this, where you want to capture all of the values (and don’t care about the keys):

key1

value1

key2

value2

 

The docs highlight it as a tip, so it seems like it should work. Must be something I’m missing here.

thanks,

chris

 

 

C2 General

From: rundeck...@googlegroups.com <rundeck...@googlegroups.com> On Behalf Of rac...@rundeck.com
Sent: Saturday, 20 March 2021 4:08 am
To: rundeck-discuss <rundeck...@googlegroups.com>
Subject: [rundeck] Re: Multiline Regex Capture matching multiple values

 

CYBER SECURITY WARNING: This email is from an external source - be careful of attachments and links. Please follow the Cyber Code and report suspicious emails.

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/jL5m5uS1-Eg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/c7402514-2510-4f9a-831d-39b1d3e5bc6dn%40googlegroups.com.

[rac...@rundeck.com]

Hi Greg,

Check this example of multiple values with the same key. Here you can see more job definition examples.

Hope it helps!

[Gadd, Chris, Vodafone NZ]

Isn’t your example a single value? The capture group (.*) matches once on a block of text, which happens to include multiple lines. My read of this:

When multiple values match, then $data.key will equal all of the matched values, separated by a newline.

was that a capture group could match more than one block of text (a value). If it’s as you suggest, then that’s really just saying that a match can include multiple lines, which doesn’t seem like much of a tip when that’d be a key reason for using the multiline regex capture logfilter instead of the key-value data logfilter.

If it is just a difference in terminology then all fine, but it would be really good if it could capture multiple blocks of text!

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/ef075ebe-a89e-4f01-9e10-8da8b705a6d4n%40googlegroups.com.

[Royke K]

Apologize to resurect the old thread

Maybe to clarify / re-questioned the issue . here is my problem that may look similar to the very first question

I have data like this from list of directory , generated from nfsen -A  command

Mail_Service/RADIUS
Mail_Service/Rackspace

Mail_Service/MX_SMTP
National_Peering/HyperNet
National_Peering/IXP
Online_Games/Blizzard
Online_Games/MOBA
International_Peering/TELIA
International_Peering/twitch

with key-value-data regex : ^(.+?)\/(.*)

and : ^(.+?)=(.+?)$

and other combination tested in regex101.

complete definition below :

<joblist>
  <job>
    <defaultTab>nodes</defaultTab>
    <description>List Profiles</description>
    <dispatch>
      <excludePrecedence>true</excludePrecedence>
      <keepgoing>false</keepgoing>
      <rankOrder>ascending</rankOrder>
      <successOnEmptyNodeFilter>false</successOnEmptyNodeFilter>
      <threadcount>1</threadcount>
    </dispatch>
    <executionEnabled>true</executionEnabled>
    <group>Management</group>
    <id>2aa0d78e-bf78-4495-a6c3-caa0c57cf0d4</id>
    <loglevel>INFO</loglevel>
    <name>List Profiles</name>
    <nodeFilterEditable>false</nodeFilterEditable>
    <nodefilters>
      <filter>name:nfsen</filter>
    </nodefilters>
    <nodesSelectedByDefault>true</nodesSelectedByDefault>
    <plugins />
    <scheduleEnabled>true</scheduleEnabled>
    <sequence keepgoing='false' strategy='node-first'>
      <command>
        <description>List All Profiles</description>
        <exec>/usr/local/nfsen/bin/nfsen -A</exec>
        <plugins>
          <LogFilter type='key-value-data'>
            <config>
              <invalidKeyPattern>\s|\$|\{|\}|\\</invalidKeyPattern>
              <logData>true</logData>
              <name>profile</name>
              <regex>^(.+?)\/(.*)</regex>
            </config>
          </LogFilter>
        </plugins>
      </command>
    </sequence>
    <uuid>2aa0d78e-bf78-4495-a6c3-caa0c57cf0d4</uuid>
  </job>
</joblist>

Instead of below as I expected to be :

I've got this :

I've read and follow all the examples from : https://github.com/rundeck-plugins/multiline-regex-datacapture-filter

Maybe I'm not quite follow the examples or not , I don't know. Need your help or further explanation.

Thank you.

Regards