Jq JSON Log Filter

[Test Kumar]

Hello Everyone,

I am trying to install Jq json log filter but it gives me error while uploading plugin from GUI.

Below is the JAR file which I am uploading for installing plugin

 JAR file URL - https://repo1.maven.org/maven2/net/thisptr/jackson-jq-cli/1.0.0-preview.20210928/jackson-jq-cli-1.0.0-preview.20210928.jar

Thanks & Regards,

Shiva Kesarwani

[rac...@rundeck.com]

Hi Shiva,

It seems an invalid plugin, researching a bit I found that isn't a Rundeck plugin. Take a look.

But if you are looking for the jq output filter, this is the right place.

Hope it helps!

[Test Kumar]

Hi Team,

Thanks for the help. It works for me

[Test Kumar]

I am not getting data in the next step when I am using jq json filter.
Below is the YAML for the same
- defaultTab: nodes
  description: Flow for ADMS
  executionEnabled: true
  id: 32270c21-2e91-4daf-87ef-7dba641c3189
  loglevel: INFO
  name: Flow for ADMS
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - configuration:
        authentication: Basic
        checkResponseCode: 'false'
        headers: |-
          {
            Content-Type: 'application/json',
            Accept: 'application/json'
          }
        method: GET
        password: test16kumar
        printResponse: 'true'
        printResponseToFile: 'false'
        proxySettings: 'false'
        remoteUrl: https://service-now.com/api/now/table/incident?sysparm_limit=1&active=true&category=inquiry
        sslVerify: 'true'
        timeout: '30000'
        username: test16kumar
      description: Get Data From Service Now
      nodeStep: false
      plugins:
        LogFilter:
        - config:
            filter: .
            logData: 'true'
            prefix: data
          type: json-mapper
      type: edu.ohio.ais.rundeck.HttpWorkflowStepPlugin
    - description: Print Value of previous step
      script: |-
        #!/bin/bash
        echo ${data}
    keepgoing: false
    strategy: node-first
  uuid: 32270c21-2e91-4daf-87ef-7dba641c3189

[rac...@rundeck.com]

Hi!

To call data/options in a script step, use this way: @data.mydata@ instead of ${data}, that’s doesn’t work even on command steps, in command steps the correct way to call a data/options is ${data.mydata}. Take a look.

I made a working example using the JSONTest website to retrieve the JSON content and the HTTP Step plugin:

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 8c131821-ff27-4c36-8d3b-dd932d35ad7b
  loglevel: INFO
  name: JQExample
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - description: Just a friendly message :)
      exec: echo "Starting..."
    - configuration:
        authentication: None
        checkResponseCode: 'false'
        method: GET
        printResponse: 'true'
        printResponseToFile: 'false'
        proxySettings: 'false'
        remoteUrl: http://echo.jsontest.com/key/value/one/two
        sslVerify: 'false'
        timeout: '30000'
      description: Get json data and jq filtering
      nodeStep: true
      plugins:
        LogFilter:
        - config:
            filter: .
            logData: 'true'
            prefix: result
          type: json-mapper
      type: edu.ohio.ais.rundeck.HttpWorkflowNodeStepPlugin
    - description: Command step way
      exec: 'echo "result is: ${data.one} and ${data.key}"'
    - description: Inline script way
      fileExtension: .sh
      interpreterArgsQuoted: false
      script: 'echo "result is: @data.one@ and @data.key@"'
      scriptInterpreter: /bin/bash
    keepgoing: false
    strategy: node-first
  uuid: 8c131821-ff27-4c36-8d3b-dd932d35ad7b

Check the result here.

Hope it helps!

​

[Test Kumar]

Hello,
If the result is array of object like below 
[{name: "Test",  surname: "kumar"}, { name: "best", surname: "yadav" }]

Then how can it works, can you please help me

[rac...@rundeck.com]

Hi,

That is a good scenario, and testing carefully the JQ JSON filter plugin, it seems that in every array iteration the plugin always gets the last entry because the prefix name is always the same. You can open a new enhancement request here.

For that reason, we need a way to iterate the array and put the values with different keys to generate the data values

I created this script that needs jq.

# here you can get the json content using curl.
my_array=( $(echo "[ { \"name\": \"Test\",  \"surname\": \"kumar\" }, { \"name\": \"best\", \"surname\": \"yadav\" } ]" | jq -r .[].surname) )

# prints each element as a key/value
for key in ${!my_array[@]}; do
    echo "mykey_${key}="${my_array[${key}]}
done

I printed the JSON file directly. You can get the JSON content using curl, then proceed to save the elements in the array, and finally generate the key/value elements.

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 8791e56d-b82e-4a80-8efd-64f2856dc299
  loglevel: INFO
  name: JQExampleArray
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - description: Just a friendly message :)
      exec: echo "Starting..."
    - description: Get the JSON data and generate the key/value data
      fileExtension: .sh
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            invalidKeyPattern: \s|\$|\{|\}|\\
            logData: 'true'
            regex: (.*)=(.*)
          type: key-value-data
      script: |+
        # here you can get the json content using curl.
        my_array=( $(echo "[ { \"name\": \"Test\",  \"surname\": \"kumar\" }, { \"name\": \"best\", \"surname\": \"yadav\" } ]" | jq -r .[].surname) )

        # prints each element as a key/value
        for key in ${!my_array[@]}; do
            echo "mykey_${key}="${my_array[${key}]}
        done

      scriptInterpreter: /bin/bash
    - description: print the data variable on command step
      exec: echo ${data.mykey_0}
    - description: print the data variable on inline-script step
      fileExtension: .sh
      interpreterArgsQuoted: false
      script: echo @data.mykey_1@
      scriptInterpreter: /bin/bash
    keepgoing: false
    strategy: node-first
  uuid: 8791e56d-b82e-4a80-8efd-64f2856dc299

Check the final result.

Of course, that script is improvable, exists many ways to do the same there :-)

Hope it helps!

​

[Test Kumar]

Yes, the last one helps but there is another challenge when I am accessing the log filter value in the command & script step then it is accessible.
But when I am accessing it in the HTTP step it does not give any response.
Can you please help me out with that

[rac...@rundeck.com]

Hi Kumar,

In my example, the JSON data could be retrieved using cURL (I posted the data directly only for testing).

Now, if you are using the HTTP Step plugin for some reason, make sure to mark the "Print Response" textbox.

Anyway, a good approach could be to use "print response to file" using the HTTP Step plugin  (as a first step) and then process the file in the inline script like my example (as a second step).

Regards.

[Test Kumar]

Hi Team,
My case is different below I am explaining my problem
First step --> This is my HTTP step where I hit one api, print response and log filter that response.
Second step --> This is my script step where I use @data.result@ to access the key-value pair, then process this value to use it as input in third step and here also I use log filter

Third Step --> Again this is my HTTP step  but I am not able to access @data.result@ of second step log

[rac...@rundeck.com]

Hi Kumar,

@data.result@ is only for inline script steps, if you wanna use the data on the command step, or use a plugin like the HTTP Step please use the ${data.result}.

More info here.

Greetings.

​

[Test Kumar]

Hello Team,

Thanks for the help.
I have to iterate http step over an array is it possible in rundeck??

[rac...@rundeck.com]

Hi Kumar,

You can save the JSON content in a file using the "print response to file" HTTP step option (as I said here) and then apply the script in the next step (my example prints the file directly using echo but you can get the JSON content from a file, e.g. using cat tool).

Regards.