Custom Decoder

[Marc Bonoan]

Hello,

I am pretty new to this and trying to learn as much as I can. I am trying to build a custom decoder for some logs that I am trying properly display and report on.

Here is a sample log

{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}

I would like it to be broken down into something like this. I read somewhere that I would need to use regex. I am trying to understand it but currently struggling.

{
"timestamp":"2016-05-02T17:46:48.515262+0000",
"level":"error",
"label":"EXPRESS",
"environment":"development",
"message": "172.105.161.246 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"",
"meta": {
"environment":"development",
"source_ip":"172.105.161.246",
"request_url":"GET / HTTP/1.1",
"response_code":"418",
"response_size":"14",
"referrer":"-",
"user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
}
}

Whats the best way to go about this? I would appreciate any help.

Thank you

[Cesar Moreno]

Hello Marc,

Thanks for posting on the Wazuh mailing list. Hope everything is very well.

The following guide can help you to understand and create your custom decoders and rules:

https://documentation.wazuh.com/current/user-manual/ruleset/custom.html

In this case, this is in json format, so the json decoder will show you the needed information depending on the log (key: value).

To test this, you can use the ossec-logtest script in the /var/ossec/bin/ directory. The result by default:

---

log: '{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}'

**Phase 2: Completed decoding.

       decoder: 'json'

       level: 'error'

       label: 'EXPRESS'

       message: '170.145.61.26 - - [19/May/2021:19:44:41 +0000] "GET / HTTP/1.1" 418 14 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"'

       environment: 'development'

       meta.environment: 'development'

---

I've created the following decoder as a json's child in the /var/ossec/etc/decoders/local_decoder.xml file:

---

<!--

{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}

-->

<decoder name="express-custom">

  <parent>json</parent>

  <regex>"message":"(\d+.\d+.\d+.\d+) \S+ \S+ [(\d+/\w+/\d+:\S+) \S+] \\"(\S+ / \S+)\\" (\S+) (\S+) \\"(\S+)\\" \\"(\.+)\\"</regex>

  <order>source_ip,timestamp,request_type,response_code,response_type,referrer,user_agent</order>

</decoder>

---

With the following result that looks something like you need:

Note that I've not used the day, month, year, and time separately since it's not possible to format in the decoder the time format as you need. But you can use the Wazuh template in Filebeat or Logstash to use a known date format by ElasticSearch and get it as date to Elasticsearch. For example using the entire date as timestamp (keyword): [(\d+/\w+/\d+:\S+) \S+] 

---

**Phase 1: Completed pre-decoding.

       full event: '{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}'

       timestamp: '(null)'

       hostname: 'manager4-1'

       program_name: '(null)'

       log: '{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}'

**Phase 2: Completed decoding.

       decoder: 'json'

       source_ip: '170.145.61.26'

       timestamp: '19/May/2021:19:44:41'

       request_type: 'GET / HTTP/1.1'

       response_code: '418'

       response_type: '14'

       referrer: '-'

       user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'

---

Since Phase 3 is a level 2 rule, it won't create an alert, so you'll need something like this in your /var/ossec/etc/rules/local_rules.xml. I've created the following example to create alerts when the json decoder finds the label:EXPRESS as the parent, then the child as level 10 when it finds the level:error.

---

<!--

{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}

-->

<group name="express-group">

  <rule id="102000" level="0">

    <decoded_as>json</decoded_as>

    <field name="label">^EXPRESS$</field>

    <description>Express rules group</description>

  </rule>

  <rule id="102001" level="10">

    <if_sid>102000</if_sid>

    <field name="level">^error$</field>

    <description>Error: Environment: $(environment) - Level: $(level) - IP: $(source_ip)</description>

    <!--options>no_full_log</options-->

  </rule>

</group>

---

And the final output is the following:

---

**Phase 1: Completed pre-decoding.

       full event: '{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}'

       timestamp: '(null)'

       hostname: 'manager4-1'

       program_name: '(null)'

       log: '{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}'

**Phase 2: Completed decoding.

       decoder: 'json'

       level: 'error'

       label: 'EXPRESS'

       source_ip: '170.145.61.26'

       timestamp: '19/May/2021:19:44:41'

       request_type: 'GET / HTTP/1.1'

       response_code: '418'

       response_type: '14'

       referrer: '-'

       user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'

       environment: 'development'

**Phase 3: Completed filtering (rules).

       Rule id: '102001'

       Level: '10'

       Description: 'Error: Environment: development - Level: error - IP: 170.145.61.26'

**Alert to be generated.

---

Hope this helps you to understand how it works.

Any questions, please let me know, I'm glad to help.

Kind regards,

Cesar Moreno.

[Cesar Moreno]

Hello again Marc,

I forgot to tell you how I got the information outside the message information, please find the following decoder:

<!--

{"level":"error","label":"EXPRESS","message":"170.145.61.26 - - [19/May/2021:19:44:41 +0000] \"GET / HTTP/1.1\" 418 14 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36\"","environment":"development","meta":{"environment":"development"}}

-->

<decoder name="express-custom">

  <parent>json</parent>

  <regex>"level":"(\S+)","label":"(\S+)","message":"(\d+.\d+.\d+.\d+) \S+ \S+ [(\d+/\w+/\d+:\S+) \S+] \\"(\S+ / \S+)\\" (\S+) (\S+) \\"(\S+)\\" \\"(\.+)\\"","environment":"(\S+)","meta":{\.*}</regex>

  <order>level,label,source_ip,timestamp,request_type,response_code,response_type,referrer,user_agent,environment,meta</order>

</decoder>

Hope this helps, I look forward to your feedback.

Kind regards,

Cesar Moreno.

[Marc Bonoan]

Thank you. I will try this shortly and let you know.

Ill start learning regex as it seems like it will keep coming up in the future.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/35a3a608-08d0-4ad3-afa8-392fcfca23ben%40googlegroups.com.

[Cesar Moreno]

Ok Marc, 

As additional information, please check the following guides:

Decoders Syntax:

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/decoders.html

Also, please find the Regex syntax used by the tags:

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/regex.html

Hope this helps.

Kind regards,

Cesar Moreno.

[marc aron bonoan]

I tweaked the regex a bit and it now works on all the logs and in logtest.

My problem now is that its not showing up in kibana even tho I see in ossec.log that its seeing the log entry. I also tested the same exact log in logtest and it was fine

[Cesar Moreno]

Hello Aron,

You can check the logs/archives/archives.json after you enable it in ossec.conf (<logall_json>yes</logall_json>) and restart the service.

If you receive those logs here, means you get the logs on the manager. The archives.log will show you if exist a rule that matches and what level it is.

In this case, you'll probably need to correct the rule that gets the decoded event and parse it as an alert. The  logtest will let you know what's the rule if exists and what is its level (should be level 3 or higher).

Hope this helps. Any questions, please let me know, I'm glad to help.

Kind regards,
Cesar Moreno.

[Marc Bonoan]

Hi Cesar,

I attached a screenshot of the ossec log test. It is being broken up properly. And I got this log entry from tailing ossec.log

It is currently rule level 3 so it should be displayed in kibana. I had them coming into Kibana perfectly fine before I put in the custom decoder and it all stopped coming in after implementing the custom decoder. I have also looked at the archive logs, it contains the same thing I saw from ossec.log which is this. I also confirmed that these logs works in ossec log test and has rule level 3

2021 May 27 21:41:15 ip-172-31-15-155->aws-s3 DEBUG: +++ Sending events to Analysd...

2021 May 27 21:41:15 ip-172-31-15-155->aws-s3 DEBUG: The message is "{"level":"debug","label":"EXPRESS","message":"172.31.4.74 - - [27/May/2021:07:24:36 +0000] \"GET /api/ping HTTP/1.1\" 200 4 \"-\" \"ELB-HealthChecker/2.0\"","environment":"development","meta":{"environment":"development"}}"

[Cesar Moreno]

Hi Marc,

Unfortunately, I could verify that creating a json child, could cause some issues with other logs that also use json decoder, so you'll need to tune it for others too. I just wanted to help you to find an alternative solution for this case.

By default, the json decoder takes the information in the json log and parses it depending on the pair {key:value}. If you have the chance to correct it on the origin, this would be great to have the information correctly decoded by the json decoder. In addition, if you have the chance to change the log format as Syslog, or send the "message" separately, for example, you'll be able to create your custom decoders and rules.

Hope this helps, any questions, please let me know. I'm glad to help.

Kind regards,

Cesar Moreno.