AWX 22.0.0 External Logging Settings to ELK Stack

[Jerome Arellano]

Hello AWX Project Team, To whom it may concern,

I would like to seek your expertise on how to setup the external logging settings of Ansible AWX.

AWX Environment:
OS: CentOS Stream 8
K3S:
Client Version: v1.26.3+k3s1
Kustomize Version: v4.5.7
Server Version: v1.26.3+k3s1
AWX-Operator: 2.0.0
Static IP: 192.168.XXX.11

ELK Enviroment:
OS: CentOS Stream 8
ELK Version: 7.17.9
Static IP: 192.168.XXX.12

I configured the ELK Stack as per usual below are my configs.

Kibana config: /etc/kibana/kibana.yml
server.host: "0.0.0.0"
server.publicBaseUrl: "http://kibana.local.com"

Elasticsearch config: /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Logstash config: /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash

Logstash Pipeline config: /etc/logstash/conf.d/awx_pipeline.conf
input {
  http {
port => 5055
codec => "json"
  }
}

filter {
  mutate {
remove_field => [ "[headers]" ]
  }
}

output {
  elasticsearch {
hosts => ["http://localhost:9200"]
index => "awx-logs-%{+YYYY.MM.dd}"
manage_template => false
  }
}

Then next, i configure AWX Settings > Logging Settings to the following values:
Logging Aggregator: 192.168.XXX.12
Logging Aggregator Port: 5055
Logging Aggregator Type: logstash
Logging Aggregator Protocol: HTTPS/HTTP
Logging Aggregator Level Threshold: INFO

With this configuration, I wasn't able to collect any logs or output from AWX but i did a curl test if the http://192.168.XXX.12:5055 wasn't reachable but it posted and Kibana was able to display the logs under index awx-logs-*.

curl -XPOST -H "Content-Type: application/json" http://192.168.XXX.12:5055/ -d '{"message": "Hello, World!"}'
curl -XPOST -H "Content-Type: application/json" http://192.168.XXX.12:5055/ -d '{"message": "Hello, World2!"}'

using this test, I was able to conclude that my ELK Stack is working.
Am i missing something on logging settings or a configuration on AWX side? 
Thank you in advance for your response.

[rakesh boinapally]

I also had the similar issue in the past.Glad that it was just not me.

Some thing is off with this log forwarding from the AWX to ELK Stack.

Thanks,

Rakesh 

[Jerome Arellano]

Hello Rakesh,

I figured it out.
Using the boilerplate and configuration of the ELK stack above. I've made an adjustment on AWX GUI. See below.
I configure AWX Settings > Logging Settings to the following values:
Logging Aggregator: http://192.168.XXX.12:5055
Logging Aggregator Port: null

Logging Aggregator Type: logstash
Logging Aggregator Protocol: HTTPS/HTTP
Logging Aggregator Level Threshold: INFO

Loggers Sending Data to Log Aggregator Form:
[
  "job_events"
]

logstash was able to collect the data from the execution of every jobs/templates on my AWX. go try it. cheers mate!

Jerome