Parse AWS Json logs

[Obay Abadi]

Hi Community,

Thank you for your efforts on this project,

I'm Integrating my Wazuh  v4.2.7 with AWS S3 bucket to get logs of my internet-facing apps, and it's running fine but the alerts e.g (AWS ALB alert.) comes without decoding the important fields, so you'll have to open the json file every time you do investigation.

I did a local_decoder.xml see below

<decoder name="obey_json">
  <parent>json</parent>
  <regex>"source":"(\w+)",clientIp":"(\d+.\d+.\d+.\d+)","user-agent":"(\S+)","httpMethod":"(\w+)","host":"(\S+)","uri":"(\S+)","action":"(\w+)"</regex>
  <order>source,clientIp,user-agent,httpMethod,host,uri,action</order>
</decoder>

But it corrupts the parent Json decoder and all alerts will be gone, So i worked on the parent Json decoder and it looked like this below 

 

<decoder name="json">
  <prematch>^{\s*"</prematch>
  <regex>"source":"(\w+)",clientIp":"(\d+.\d+.\d+.\d+)","user-agent":"(\S+)","httpMethod":"(\w+)","host":"(\S+)","uri":"(\S+)","action":"(\w+)"</regex>
  <order>source,clientIp,user-agent,httpMethod,host,uri,action</order>
</decoder>

then and i tried the logtest script to check but it's give me this output below 

**Phase 2: Completed decoding.
    name: 'json

can i get some help please on this

Thanks and regards,

Obay

[Obay Abadi]

This is an example of a log

{
  "agent": {
    "name": "Wazuh-Manager",
    "id": "000"
  },
  "manager": {
    "name": "Wazuh-Manager"
  },
  "data": {
    "integration": "aws",
    "aws": {
      "received_bytes": "264",
      "request": "GET http://localhost.com:80/ HTTP/1.1",
      "target_status_code_list": "-",
      "target_port_list": "-",
      "target_processing_time": "-1",
      "log_info": {
        "s3bucket": "loadbalancer-logs",
        "log_file": "20221017T0920Z_1nb5j2ss.log.gz"
      },
      "ssl_cipher": "-",
      "source": "alb",
      "type": "http",
      "sent_bytes": "334",
      "client_port": "100.0.0.22:58295",
      "target_port": "-",
      "domain_name": "-",
      "error_reason": "-",
      "classification_reason": "-",
      "elb": "app/web-new/9c8f5efa20aafd05",
      "user_agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
      "ssl_protocol": "-",
      "target_group_arn": "-",
      "trace_id": "Root=1-634d1d67-5c01f240315f2",
      "matched_rule_priority": "0",
      "elb_status_code": "301",
      "classification": "-",
      "request_processing_time": "-1",
      "chosen_cert_arn": "-",
      "response_processing_time": "-1",
      "target_status_code": "-",
      "request_creation_time": "2022-10-17T09:16:23.972000Z",
      "time": "2022-10-17T09:16:23.981719Z",
      "redirect_url": "https://localhost.com:443/",
      "action_executed": "waf,redirect"
    }
  },
  "rule": {
    "firedtimes": 220,
    "mail": false,
    "level": 3,
    "description": "AWS ALB alert.",
    "groups": [
      "amazon",
      "aws",
      "aws_alb"
    ],
    "id": "80325"
  },
  "decoder": {
    "name": "json"
  },
  "input": {
    "type": "log"
  },
  "@timestamp": "2022-10-17T09:20:35.047Z",
  "location": "Wazuh-AWS",
  "id": "168435.123133",
  "timestamp": "2022-10-17T09:20:35.047+0000",
  "_id": "7-I-5YMBu8peyJ2DeL"
}

[Julián Morales]

Hi Obay,

When a decoder has children, automatically the regex and order options of this parent decoder are ignored. This is why when you create a JSON child decoder, JSON events are no longer processed.
But this should not be a limitation, you can use the technique of sibling decoders (doc here), creating multiple children of the JSON decoder. This could decode your logs with regex+json plugin decoder.

First of all, you must restore the original json decoder:
/var/ossec/ruleset/decoders/0006-json_decoders.xml

<decoder name="json">
  <prematch>^{\s*"</prematch>

  <plugin_decoder>JSON_Decoder</plugin_decoder>
</decoder>

Now, for example, suppose the following log:

The number 88 is inside a string, the json decoder will not decode it in a separate field, but we can use the above technique and create the following custom decoders:

/var/ossec/etc/decoders/local_decoder.xml:

<decoder name="json-child">
    <parent>json</parent>
    <regex>important number is: (\d+)</regex>
    <order>important_number</order>
</decoder>

<decoder name="json-child">
    <parent>json</parent>
    <plugin_decoder>JSON_Decoder</plugin_decoder>
</decoder>




Finally in logtest, we can see how it extracts the number inside the value of a json:


╰─# /var/ossec/bin/wazuh-logtest  
Starting wazuh-logtest v4.5.0
Type one log per line

{"test_field" : "value. important number is: 88"}

**Phase 1: Completed pre-decoding.
        full event: '{"test_field" : "value. important number is: 88"}'

**Phase 2: Completed decoding.
        name: 'json'
        important_number: '88'
        test_field: 'value. important number is: 88'


I hope you find this useful

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9580a631-a655-44ba-95f8-5bede9bb36e7n%40googlegroups.com.