Consolidate Logs to CSV ?

[Cp Divers]

Hello guys,

I'm brand new to rundeck.

I have a rundeck job gathering server information such as server name, OS, IP, etc ..

This job runs on 20 nodes, and everything is fine.

Now I'd like to consolidate all those 20 logs into 1, so I can easily see my result into 1 single file.

How can I achieve that ?

Later on, my goal is to works with keyword=value, so I can generate a CSV file, with all my nodes (name,OS,IP, etc..)

Thanks for your help.

[rac...@rundeck.com]

Hi,

You can use the job reference step to manage this. I leave the examples in YAML format to import, adapt, and test in your instance.

The first child job collects the logs on each target node and put the content on a shared file (at a shared location):

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 626e91dd-c285-4f53-989c-5752d598e105
  loglevel: INFO
  name: Collector
  nodeFilterEditable: false
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: false
      rankOrder: ascending
      successOnEmptyNodeFilter: false
      threadcount: '1'
    filter: node.*
  nodesSelectedByDefault: true
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - fileExtension: .sh
      interpreterArgsQuoted: false
      script: echo $(uname -a) >> /path/to/shared/file.txt
      scriptInterpreter: /bin/bash
    keepgoing: false
    strategy: sequential
  uuid: 626e91dd-c285-4f53-989c-5752d598e105

The second child job (dispatching to a shared file machine) takes the shared file and creates the CSV file using python 3 code (in an inline script using the python 3 interpreter).

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 051ea7f8-bf8e-47e5-bbb8-c797dfa91fb4
  loglevel: INFO
  name: CSVJob
  nodeFilterEditable: false
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: false
      rankOrder: ascending
      successOnEmptyNodeFilter: false
      threadcount: '1'
    filter: shared_node
  nodesSelectedByDefault: true
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - fileExtension: .py
      interpreterArgsQuoted: false
      script: |-
        import csv

        with open('file.txt', 'r') as in_file:
            stripped = (line.strip() for line in in_file)
            lines = (line.split(",") for line in stripped if line)
            with open('log.csv', 'w') as out_file:
                writer = csv.writer(out_file)
                writer.writerow(('command', 'intro'))
                writer.writerows(lines)
      scriptInterpreter: python3
    - description: cleaner step
      exec: rm file.txt
    - exec: echo "done"
    keepgoing: false
    strategy: node-first
  uuid: 051ea7f8-bf8e-47e5-bbb8-c797dfa91fb4

And the parent job which calls the first and the second jobs sequentially:

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: b46413ed-3e47-4b3d-bc45-99bf0ab83464
  loglevel: INFO
  name: Parent
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - jobref:
        group: ''
        name: Collector
        nodeStep: 'true'
        uuid: 626e91dd-c285-4f53-989c-5752d598e105
    - jobref:
        group: ''
        name: CSVJob
        nodeStep: 'true'
        uuid: 051ea7f8-bf8e-47e5-bbb8-c797dfa91fb4
    keepgoing: false
    strategy: sequential
  uuid: b46413ed-3e47-4b3d-bc45-99bf0ab83464

Here you can see another basic example related to job reference step.

Hope it helps!

​

[Cp Divers]

Thank you very much for your help, I have not try yet

Anyhow it seems to me that this should a basic feature in rundeck.

[Cp Divers]

OK coming back to the solution you did provide me. I look at it, and I have no doubt it could work. I do not undertstand why I should send the result of the collector to a shared folder from the node I query.

First of all not all nodes have access to a shared location to send the result of the inventory. But more importantly, rundeck gets the logs of the execution, therefore rundeck has all the data ! It can even treat those logs, since it can process Key Value Data.

I would hope there is a way for rundeck to query and/or consolidate its own log.

so let me try to rephrase my question.

I have a job querying some data, and returning this data in the rundeck logs. 

I can even use Key Value DATA, so rundeck returns some clean data.

I just would like to consolidate all this data, from the 20 nodes logs that I have, in 1 single files.

There has to be a way, right ?

 

[rac...@rundeck.com]

You’re right!

A good way to do that is to use the save-to-file plugin in the “collector job”, this plugin saves all logs in the rundeck server (without shared directories, etc).

The first step is to install the plugin: Go to Gear Icon > Plugins > Find Plugins, type “save” in “Search for” textbox and press the Enter key, then, you will see the plugin listed, click on the “Install” button, no restart is required.

The strategy is similar:

On the collector job (which is dispatched to all remote nodes) you can add a new Log Filter (edit the job, go to the step which capture/generate the data, click on the tiny gear icon and select “Save to File”, define a local Rundeck server filepath and click on the “Append to File” checkbox, then save the job).

The parent job calls the collector job as a first step, the second step takes the file generated by the save-to-file plugin (from the collector job) and uses it as you want (using python code, bash, etc…).

Let me share with you the job definition examples:

Parent job:

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: d1b73764-9311-4ad7-868d-88ee7611afde
  loglevel: INFO
  name: ParentJob
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - jobref:
        group: ''
        name: Collector
        nodeStep: 'true'
        uuid: d07aa463-6e42-4917-922a-b22423b874da
    - fileExtension: .sh
      interpreterArgsQuoted: false
      script: |-
        cat /home/rundeck/mylog.txt
        rm /home/rundeck/mylog.txt
      scriptInterpreter: /bin/bash
    keepgoing: false
    strategy: node-first
  uuid: d1b73764-9311-4ad7-868d-88ee7611afde

Collector job:

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: d07aa463-6e42-4917-922a-b22423b874da
  loglevel: INFO
  name: Collector
  nodeFilterEditable: false
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: false
      rankOrder: ascending
      successOnEmptyNodeFilter: false
      threadcount: '1'
    filter: node.*
  nodesSelectedByDefault: true
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - exec: echo "hi"
      plugins:
        LogFilter:
        - config:
            appendToFile: 'true'
            fileDestination: /home/rundeck/mylog.txt
          type: SaveToFile
    keepgoing: false
    strategy: node-first
  uuid: d07aa463-6e42-4917-922a-b22423b874da

This is more accurate to your use case :-)

Hope it helps!

PD: Post fixed again due some problems with the message format.

​

[Personne]

Going to try this save to file plugin for sure.

In the meantime, here is my job,
I'm just trying to use rundeck to collect information for each node we have. In other words, just doing an inventory.

Remember, this is my very first rundeck job ever !

I just do not know how to achieve my goal the best way with rundeck. and maybe more importantly how to present the data to rundeck. 

As you will see I'm using key pair values for now, but I wonder if presenting the result as JSON or YAML to rundeck would be better.

Any idea ?

- defaultTab: nodes

  description: |

    Test PowerShell on remote nodes

    Verifies Rundeck's ability to run PowerShell on remote Windows nodes within this project.

  executionEnabled: true

  group: devops/utils

  id: d05e7d97-ecda-4c26-81b9-6458fff73da5

  loglevel: INFO

  name: Test Windows Connection Data

  nodeFilterEditable: false

  nodefilters:

    dispatch:

      excludePrecedence: true

      keepgoing: true

      rankOrder: ascending

      successOnEmptyNodeFilter: false

      threadcount: '10'

    filter: 'osFamily: windows '

  nodesSelectedByDefault: true

  plugins:

    ExecutionLifecycle: null

  scheduleEnabled: true

  sequence:

    commands:

    - fileExtension: ps1

      interpreterArgsQuoted: false

      plugins:

        LogFilter: []

      script: "#######################################\n#\n# Rundeck test for Windows\n\

        #\n#######################################\n$osLabel = (Get-ItemProperty -Path\

        \ 'Registry::HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion'\

        \ ProductName).ProductName\n$major = $PSVersionTable.PSVersion.Major\n$minor\

        \ = $PSVersionTable.PSVersion.Minor\n\nwrite-host \"RUNDECK:DATA:hostname=$($env:computerName)\"\

        \nwrite-host \"RUNDECK:DATA:FQDN=$(([System.Net.Dns]::GetHostByName(($env:computerName))).Hostname)\"\

        \nwrite-host \"RUNDECK:DATA:OS=$($osLabel)\"\nwrite-host \"RUNDECK:DATA:PSMajor=$($major)\"\

        \nwrite-host \"RUNDECK:DATA:PSMinor=$($minor)\"\nwrite-host \"RUNDECK:DATA:NTPSource=$($(w32tm\

        \ /query /source))\"\n\ntry { $dns=get-dnsserver -ea silentlycontinue }\n\

        catch {$dns=$false}\n\nif ($dns) {\n    write-host \"RUNDECK:DATA:DNSServer=TRUE\"\

        \n    $fwrd = (Get-DnsServerForwarder -ea silentlycontinue).ipaddress.ipaddresstostring\

        \ -join \" / \"\n    write-host \"RUNDECK:DATA:DNSForwarder=$fwrd\"\n    $cfwrd\

        \ = @(Get-DnsServerZone | ? {$_.zonetype -eq 'Forwarder'} | sort zonename)\n\

        \    #Write-host \"Conditional Forwarders:\"\n    if ($cfwrd) {\n        #\

        \ Write-host \"- Count: $($cfwrd.count)\"  \n        $cfi=1\n        foreach\

        \ ($c in $cfwrd) {\n            \"RUNDECK:DATA:DNSForwarderZone{0}- {1} [{2}]\"\

        \ -f $cfi, $c.zonename, ($c.masterservers.IPAddressToString -join \" / \"\

        )\n        }\n\n    } else {\n\n    }\n} else { write-host \"RUNDECK:DATA:DNSServer=FALSE\"\

        }\n\n"

      scriptInterpreter: powershell.exe

    - configuration:

        debugOnly: 'false'

      description: consolidate

      nodeStep: false

      type: log-data-step

    keepgoing: false

    pluginConfig:

      LogFilter:

      - config:

          invalidKeyPattern: \s|\$|\{|\}|\\

          logData: 'true'

          name: XXXXXXX

          regex: ^RUNDECK:DATA:\s*([^\s]+?)\s*=\s*(.+)$

        type: key-value-data

    strategy: node-first

  uuid: d05e7d97-ecda-4c26-81b9-6458fff73da5

--
You received this message because you are subscribed to a topic in the Google Groups "rundeck-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rundeck-discuss/iBOqXfPrJvI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/76000a04-0be8-495e-a0d6-cb37368c3635n%40googlegroups.com.

[rac...@rundeck.com]

Hi!

Playing with your job definition, I simplified the script to print a key=value data (captured by a new regex), also, I added a step to capture the data values from the first script, based on this.

The output is a YAML-based output. Please take a look:

- defaultTab: nodes
  description: |
    Test PowerShell on remote nodes

    Verifies Rundeck's ability to run PowerShell on remote Windows nodes within this project.
  executionEnabled: true
  group: devops/utils
  id: d05e7d97-ecda-4c26-81b9-6458fff73da5
  loglevel: INFO
  name: Test Windows Connection Data
  nodeFilterEditable: false
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: true
      rankOrder: ascending
      successOnEmptyNodeFilter: false
      threadcount: '10'
    filter: 'osFamily: windows '
  nodesSelectedByDefault: true
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - fileExtension: ps1
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            invalidKeyPattern: \s|\$|\{|\}|\\
            logData: 'true'
            regex: ^(.*)\s*=\s*(.+)$
          type: key-value-data
      script: "#######################################\n#\n# Rundeck test for Windows\n\
        #\n#######################################\n$osLabel = (Get-ItemProperty -Path\
        \ 'Registry::HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion'\
        \ ProductName).ProductName\n$major = $PSVersionTable.PSVersion.Major\n$minor\
        \ = $PSVersionTable.PSVersion.Minor\n\nwrite-host \"hostname=$($env:computerName)\"\
        \nwrite-host \"FQDN=$(([System.Net.Dns]::GetHostByName(($env:computerName))).Hostname)\"\
        \nwrite-host \"OS=$($osLabel)\"\nwrite-host \"PSMajor=$($major)\"\nwrite-host\
        \ \"PSMinor=$($minor)\"\nwrite-host \"NTPSource=$($(w32tm /query /source))\"\
        \n\ntry { $dns=get-dnsserver -ea silentlycontinue }\ncatch {$dns=$false}\n\
        \nif ($dns) {\n    write-host \"DNSServer=TRUE\"\n    $fwrd = (Get-DnsServerForwarder\
        \ -ea silentlycontinue).ipaddress.ipaddresstostring -join \" / \"\n    write-host\
        \ \"DNSForwarder=$fwrd\"\n    $cfwrd = @(Get-DnsServerZone | ? {$_.zonetype\
        \ -eq 'Forwarder'} | sort zonename)\n    #Write-host \"Conditional Forwarders:\"\
        \n    if ($cfwrd) {\n        # Write-host \"- Count: $($cfwrd.count)\"  \n\
        \        $cfi=1\n        foreach ($c in $cfwrd) {\n            \"DNSForwarderZone{0}-\
        \ {1} [{2}]\" -f $cfi, $c.zonename, ($c.masterservers.IPAddressToString -join\
        \ \" / \")\n        }\n\n    } else {\n\n    }\n} else { write-host \"DNSServer=FALSE\"\
        }\n\n"
      scriptInterpreter: powershell.exe
    - fileExtension: .ps1
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            appendToFile: 'true'
            fileDestination: /home/m68k/Downloads/mylog.yaml
          type: SaveToFile
      script: |-
        Write-host "@node.name@:"
        Write-host "  hostname: @data.hostname@"
        Write-host "  fqdn: @data.FQDN@"
        Write-host "  operating_system: @data.OS@"
        Write-host "  psmajor: @data.PSMajor@"
        Write-host "  psminor: @data.PSMinor@"
        Write-host "  NTPSource: @data.NTPSource@"
        Write-host "  DNSServer: @data.DNSServer@"
      scriptInterpreter: powershell.exe
    keepgoing: false
    strategy: node-first
  uuid: d05e7d97-ecda-4c26-81b9-6458fff73da5

File content:

windows:
   hostname: WIN-R7I3P6RNA10
   fqdn: WIN-R7I3P6RNA10
   operating_system: Windows Server 2019 Datacenter Evaluation
   psmajor: 5
   psminor: 1
   NTPSource: time.windows.com,0x8 
   DNSServer: FALSE

I think that is a good way to reach your goal :-)

Hope it helps!

​

[Cp Divers]

Thank you that gives me some idea, I had no idea about this format ' @data.hostname@'

I does give me some options for sure.

Going to have some fun

[rac...@rundeck.com]

Good!

The data variables are generated using the log data pattern in the job/steps, check this, here you can see an example, and here an awesome explanation (it's based on Rundeck 2.11 but is the same principle).

Regards!

[Cp Divers]

Yep, I have already read all of this, but as I said I'm a rundeck newbie, and surely missing some concept.

But maybe my question was not clear, so let me try again

1) a job can run on multiple nodes, and then create a separate log for each node.

2) I can use key=value regex within that log, to create variables that I can use and print it that same log 

Question: is the only way to return data from a job is by the log ? I understand I could drop a file in a share folder, or even my script send an email for each node, etc ... but that is not the point.

Question: is there a way to use the key pair process in 1 log, and use them in another step ?

Another way to maybe ask my question would be

I have a job, wich runs on 3 nodes:

- log1 return "5"

- log2 return "12"

- log3 return "8"

How can I generate a log, or an email with the result of 25 ?

---------------------

Going back to merging log files:

I'm trying to take all those logs, process the data of each log, and output the result in 1 log.

If I write in my log a csv format, where my key pairs are formatted as a CSV, it will work.

the content of log1 will be "server1","ip1","time1"

the content of log2 will be "server2","ip2","time2"

etc ....

How can I easily with rundeck take those outputs and merge them in 1 file ? and get

"server1","ip1","time1"

"server2","ip2","time2"

Now what about if the output in my log is a JSON format.

I would have an output:

log1:

{

    "hostname":  "server1",

    "FQDN":  "server1.test.local"

 }

log2:

{

    "hostname":  "server2",

    "FQDN":  "server2.test.local"

}

If I concatenate these file, the JSON format will not be valid, I would get

{

    "hostname":  "server1",

    "FQDN":  "server1.test.local"

}

{

    "hostname":  "server2",

    "FQDN":  "server2.test.local"

}

instead of a valid format of

[

{

    "hostname":  "server1",

    "FQDN":  "server1.test.local"

},

{

    "hostname":  "server2",

    "FQDN":  "server2.test.local"

}

]

Is it possible to achieve this in rundeck ? if so, how ?

TIA

[rac...@rundeck.com]

Hi,

Yeah, that’s is the main idea of this and this, you can combine it in the following way:

1) The child job that collects data from all remote nodes to create formatted info using key/value data. With the save-to-file plugin (attached on the second step, all data is “saved” on a Rundeck server local file (not at a shared directory server).

2) A parent job, this job calls the Child job using job reference step, and then you can add an extra inline-script step (on bash/python/PowerShell) that takes the file generated by the first job reference step and process it as you want (in this example I used python3 with the libyaml to get the values from the YAML file).

So, the basic example:

a) Child job (AKA “The Collector”) It’s configured to execute against all windows remote nodes and collect the information from them, all information is added to a rundeck server local file using the save-to-file plugin (not using a shared directory way). In this job, the first step generates key/value data which is captured in the second step, in my last post you can see some basic examples.

- defaultTab: nodes
  description: |
    Test PowerShell on remote nodes

    Verifies Rundeck's ability to run PowerShell on remote Windows nodes within this project.
  executionEnabled: true
  group: devops/utils
  id: d05e7d97-ecda-4c26-81b9-6458fff73da5
  loglevel: INFO
  name: Test Windows Connection Data
  nodeFilterEditable: false
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: true
      rankOrder: ascending
      successOnEmptyNodeFilter: false
      threadcount: '10'
    filter: 'osFamily: windows '
  nodesSelectedByDefault: true
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - fileExtension: ps1
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            invalidKeyPattern: \s|\$|\{|\}|\\
            logData: 'true'
            regex: ^(.*)\s*=\s*(.+)$
          type: key-value-data
      script: "#######################################\n#\n# Rundeck test for Windows\n\
        #\n#######################################\n$osLabel = (Get-ItemProperty -Path\
        \ 'Registry::HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows NT\\CurrentVersion'\
        \ ProductName).ProductName\n$major = $PSVersionTable.PSVersion.Major\n$minor\
        \ = $PSVersionTable.PSVersion.Minor\n\nwrite-host \"hostname=$($env:computerName)\"\
        \nwrite-host \"FQDN=$(([System.Net.Dns]::GetHostByName(($env:computerName))).Hostname)\"\
        \nwrite-host \"OS=$($osLabel)\"\nwrite-host \"PSMajor=$($major)\"\nwrite-host\
        \ \"PSMinor=$($minor)\"\nwrite-host \"NTPSource=$($(w32tm /query /source))\"\
        \n\ntry { $dns=get-dnsserver -ea silentlycontinue }\ncatch {$dns=$false}\n\
        \nif ($dns) {\n    write-host \"DNSServer=TRUE\"\n    $fwrd = (Get-DnsServerForwarder\
        \ -ea silentlycontinue).ipaddress.ipaddresstostring -join \" / \"\n    write-host\
        \ \"DNSForwarder=$fwrd\"\n    $cfwrd = @(Get-DnsServerZone | ? {$_.zonetype\
        \ -eq 'Forwarder'} | sort zonename)\n    #Write-host \"Conditional Forwarders:\"\
        \n    if ($cfwrd) {\n        # Write-host \"- Count: $($cfwrd.count)\"  \n\
        \        $cfi=1\n        foreach ($c in $cfwrd) {\n            \"DNSForwarderZone{0}-\
        \ {1} [{2}]\" -f $cfi, $c.zonename, ($c.masterservers.IPAddressToString -join\
        \ \" / \")\n        }\n\n    } else {\n\n    }\n} else { write-host \"DNSServer=FALSE\"\
        }\n\n"
      scriptInterpreter: powershell.exe
    - fileExtension: .ps1
      interpreterArgsQuoted: false
      plugins:
        LogFilter:
        - config:
            appendToFile: 'true'
            fileDestination: myfile
          type: SaveToFile
      script: |-
        Write-host "@node.name@:"
        Write-host "  hostname: @data.hostname@"
        Write-host "  fqdn: @data.FQDN@"
        Write-host "  operating_system: @data.OS@"
        Write-host "  psmajor: @data.PSMajor@"
        Write-host "  psminor: @data.PSMinor@"
        Write-host "  NTPSource: @data.NTPSource@"
        Write-host "  DNSServer: @data.DNSServer@"
      scriptInterpreter: powershell.exe
    keepgoing: false
    strategy: node-first
  uuid: d05e7d97-ecda-4c26-81b9-6458fff73da5

b) Parent job (A sequential workflow with two steps: The first one calls the child job and the second one is an inline-script step that takes the file and process it, in this example I’m using python 3 with the pyyaml library to print the values, you can use PowerShell or bash if you like).

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 41c0eee1-517c-4e3a-9ce9-06100da9ff66
  loglevel: INFO
  name: Parent
  nodeFilterEditable: false
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - jobref:
        group: ''
        name: Child
        nodeStep: 'true'
        uuid: d05e7d97-ecda-4c26-81b9-6458fff73da5
    - fileExtension: .py
      interpreterArgsQuoted: false
      script: |-
        # example using pyyaml :-)

        import yaml

        with open(r'myfile') as file:
            the_list = yaml.load(file, Loader=yaml.FullLoader)

            print(the_list)
      scriptInterpreter: python3
    keepgoing: false
    strategy: sequential
  uuid: 41c0eee1-517c-4e3a-9ce9-06100da9ff66

Hope it helps!

​