Global Log Filters

[cedthes]

Hi there,

I haven't been able to find anything usefull for now about my issue so i'm asking here.(sorry if i've missed an older post about it)

We are currently working on moving our rundeck plateform from a 2.9.2 version to 3.1.0 and some of our jobs aren't working as intended.

We have jobs using global log filters to get a log path and then we execute cat ${data.key} later to show the complete log on screen.

On older rundeck :

Global log filter :

 - Key Value Data : ^.*(est)\s*:\s*(.+)$

Step 1 : job execution with a result showing a last line with "est : /pathtolog" in it.

Step 2 : remote command : echo ${data.est}

Step 2 is showing the complete log as intended

The filter was working well and we had a data called "est"  with "/pathtolog" as value.

But now in newer version data is empty and cat is crashing (poor cat !)

So right now i've been testing a lot of stuff but i'm mainly trying to find out what have changed between old and new rundeck.

I hope i've made myself clear, don't hesitate to correct/ask more details if needed.

Thx again in advance for any help !

[Reiner Acuña]

Hi Cedthes,

I tested your regex under 3.1.0 and works like a charm, test this Job definition:

<joblist>

  <job>

    <defaultTab>nodes</defaultTab>

    <description></description>

    <executionEnabled>true</executionEnabled>

    <id>c192276c-5cac-4433-a579-78f520919ecf</id>

    <loglevel>INFO</loglevel>

    <name>JobREGEX</name>

    <nodeFilterEditable>false</nodeFilterEditable>

    <scheduleEnabled>true</scheduleEnabled>

    <sequence keepgoing='false' strategy='node-first'>

      <command>

        <plugins>

          <LogFilter type='key-value-data'>

            <config>

              <logData>true</logData>

              <regex>^.*(est)\s*:\s*(.+)$</regex>

            </config>

          </LogFilter>

        </plugins>

        <script><![CDATA[echo "est : /home/mordor/sauron.txt"]]></script>

        <scriptargs />

      </command>

      <command>

        <exec>echo ${data.est}</exec>

      </command>

    </sequence>

    <uuid>c192276c-5cac-4433-a579-78f520919ecf</uuid>

  </job>

</joblist>

Best regards!

[cedthes]

Thx for testing !

I'm still puzzled, now at least i know it's not a regex problem but something else, maybe linked to the way our jobs works.

Your job seems to work, the only difference i've seen is the fact your are using filter on step and we're using a global log filter (on the whole job instead of a single step)

Since it's another job called inside this one we can't add a log filter on it...

I'm going to do some additional testing...

Thx anyway, at least i've got a working exemple !

[cedthes]

I did some tests, using global log filter on your job is working...

Alright... i'm gonna transform your yob into one of ours just to see when it's not gonna work anymore.

[cedthes]

So it's the fact i'm using another subjob in the main job that causes the issue.

If i'm only using scripts or commands it's working as intended, if i'm calling another job it's not...

That's a little bit of a problem for us...

[cedthes]

I found a workaround based on an answer in issue :

https://github.com/rundeck/rundeck/issues/3038

With our older rundeck we were calling another job and then applying log filters and using the result later on.

Now if on the sub-job i'm applying the global log filter directly on it and then export the result as a global variable.

I can then use it with ${export.MyVariable} in any jobs calling the first one.

So it works but i'll have to do some change on all our jobs (+2000)

regards,

CedThes.

[cedthes]

So... right now i'll have to rewrite a lot of our jobs and scripts because i've a "framework" job with a +600 line scripts called sometimes multiple time in the same job but with different arguments and now if i'm using the export method it will try to do a new export of all data at each step resulting in errors and fake failed jobs...

I'm considering giving my no-go for the update... 

[Reiner Acuña]

Hi Cedthes,

Thanks for your feedback.

I tested on 2.9.X, 2.10.X, 2.11.X, 3.0.X and 3.1.0 and you're right, this way of getting data values was replaced by the global variable step after 2.9.X. Maybe is a good idea to get back this functionality to update all old instances versions with a lot of jobs without edit any job one by one.

I opened a GitHub issue to follow this:

https://github.com/rundeck/rundeck/issues/5266

Hope it helps!

[cedthes]

Hi 

Many thanks for the issue ! Right now i've been re-working our jobs and had to change a lot of things (we are now left with ~500 jobs instead of 2500+) but that would totally help to have that feature back.

I'm using a framework of a sort in a job called multiple time with different result in the same parent job and right now i can't be using it like that anymore since it tries at every run to do a global variable export. On older version i was using the parent job for data collection but with the new one it's the child, and if it's used multiple time in the same job it's just not work at all (if the filter don't find every single time everything it wanted it just create an error and the job is marked as a failure even if it's working.)

Anyway, again thx a lot !