Wazuh Double Decoding problem

[Dhiraj Ambigapathi]

So I'm trying to pull logs from a reverse proxy docker.
Logs as follows

{"log":"192.168.0.6 - - [21/Jul/2023:14:15:05 +0000] \"GET /assets/abc.css HTTP/1.0\" 200 2003 \"https://random url\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0\"\n","stream":"stdout","time":"2023-07-21T14:15:05.793190376Z"}
{"log":"192.168.0.6 - - [21/Jul/2023:14:15:05 +0000] \"GET /assets/sbc.js HTTP/1.0\" 200 1729077 \"https://random-url\" \"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0\"\n","stream":"stdout","time":"2023-07-21T14:15:05.793755065Z"}
{"log":"192.168.0.6 - - [21/Jul/2023:14:15:16 +0000] \"GET /abc HTTP/1.0\" 200 461 \"-\" \"abc\"\n","stream":"stdout","time":"2023-07-21T14:15:16.201156631Z"}
{"log":"192.168.0.6 - - [21/Jul/2023:14:15:46 +0000] \"GET /abc HTTP/1.0\" 200 461 \"-\" \"abc\"\n","stream":"stdout","time":"2023-07-21T14:15:46.277675939Z"}

Wazuh parses JSON format, but couldn't parse the syslog format.

I tried parent-child parsers but it didn't work.

[Eric Franco Fahnle]

Hi Dhiraj! Hope you're doing great.

I didn't fully understand what you're trying to do and what is the problem that you're facing. Could you please provide a more detailed explanation of what you're trying to accomplish?

Thanks,

Eric

[Dhiraj Ambigapathi]

Hi Eric,

You can see in screenshot, I'm using wazuh-logtest. The output should show fields like log, ip, url etc. But it shows pre-decoding and stops. I want to extract field from the log.

[Eric Franco Fahnle]

Could you please share the configuration for the decoder and rules you're using?

[Dhiraj Ambigapathi]

Hi Eric,

Sorry for the late reply.
I'm using following decoder
<decoder name="ax">
    <prematch type="pcre2">^\{"log":</prematch>
    <regex offset="after_prematch" type="pcre2">{"log":"(\d+.\d+.\d+.\d+).-.-.\[(\d+\/\w+\/\w+):(\d+:\d+:\d+)\s.....\]\s."(GET|POST)\s(\s+|\S+)\s(HTTP\/\d+..)\\"\s(\d+)\s(\d+)\s........([a-zA-Z0-9. \/\(;_\),]+)\s([a-zA-Z0-9. \/\\]+)[\s+|\S+]+</regex>
    <order>attackerip,date,time,HTTP_Method,URI,HTTP_Protocol,HTTP_Status,Response_Length,Browser</order>
</decoder>

for Log
{"log":"135.125.246.110 - - [28/Jul/2023:02:49:59 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.
0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-28T02:49:59.322665998Z"}

It shows proper mapping on regex101 pcre2 but not working for Wazuh-logtest

[Eric Franco Fahnle]

Hi Dhiraj, I haven't had the opportunity to test it yet, but I'll come back with an answer as soon as I can.

Regards!

Eric

[Dhiraj Ambigapathi]

Hi Eric,
Any update on this

[Eric Franco Fahnle]

Hi Dhiraj, I've trying to get this to work, but unfortunately it's getting a little tricky.

The problem I'm seeing is the logs in the format you have shared "look like" a json (because of the curly braces) but it's not actually a json, as it has not a valid json format. After testing your decoder, which matches on a pure-regex testing tool, wazuh-logtest fails to identify it and classifies it as json, therefore not working as expected. This is my result using wazuh-logtest:

:/# /var/ossec/bin/wazuh-logtest

{"log":"135.125.246.110 - - [28/Jul/2023:02:49:59 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-28T02:49:59.322665998Z"}

**Phase 1: Completed pre-decoding.
full event: '{"log":"135.125.246.110 - - [28/Jul/2023:02:49:59 +0000] \"GET /.env HTTP/1.1\" 403 548 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\" \"-\"\n","stream":"stdout","time":"2023-07-28T02:49:59.322665998Z"}'

**Phase 2: Completed decoding.
name: 'json'
log: '135.125.246.110 - - [28/Jul/2023:02:49:59 +0000] "GET /.env HTTP/1.1" 403 548 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"
'
stream: 'stdout'
time: '2023-07-28T02:49:59.322665998Z'

Do you think you could change the logging format in the reverse proxy you're using? Without the curly braces "{}" it'll probably be easier.

Also, please take a look at this other Google Groups issue: https://groups.google.com/g/wazuh/c/QJZqBuuph8A

Regards,

Eric

[Dhiraj Ambigapathi]

So I tried out_format option from  here, but I can't figure out what the log looks like after formatting. This should break the JSON problem but don't know what the log looks like after agent sends to manager.

[Dhiraj Ambigapathi]

So now the logs look like following
2023 Aug 16 11:33:04 (ubuntu) any->/var/log/auth.log Aug 16 11:33:04 ubuntu sshd[20509]: Connection closed by authenticating user root 117.158.62.70 p
ort 41406 [preauth]
It breaks JSON but still couldn't parse.

[Eric Franco Fahnle]

Hi Dhiraj! 

I still believe the log looks a little messy, with that double timestamp for example. If I test it without all the first part, the example you provided gets easily decoded:

:/# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.4.3
Type one log per line

Aug 16 11:33:04 ubuntu sshd[20509]: Connection closed by authenticating user root 117.158.62.70 port 41406 [preauth]

**Phase 1: Completed pre-decoding.
full event: 'Aug 16 11:33:04 ubuntu sshd[20509]: Connection closed by authenticating user root 117.158.62.70 port 41406 [preauth]'
timestamp: 'Aug 16 11:33:04'
hostname: 'ubuntu'
program_name: 'sshd'

**Phase 2: Completed decoding.
name: 'sshd'

**Phase 3: Completed filtering (rules).
id: '5722'
level: '0'
description: 'sshd: ssh connection closed.'
groups: '['syslog', 'sshd']'
firedtimes: '1'
gdpr: '['IV_32.2']'
hipaa: '['164.312.b']'
mail: 'False'
nist_800_53: '['AU.14', 'AC.7']'
pci_dss: '['10.2.5']'
tsc: '['CC6.8', 'CC7.2', 'CC7.3']'

Without the first timestamp, this decoder config parsed the beginning:

<decoder name="example">
  <prematch type="pcre2">^\(\w+\) (.*)\-\>(.*) </prematch>
</decoder>

Here the test:

:/# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.4.3
Type one log per line

(ubuntu) any->/var/log/auth.log

**Phase 1: Completed pre-decoding.
full event: '(ubuntu) any->/var/log/auth.log'

**Phase 2: Completed decoding.
name: 'example'

Hope this helps.

Regards,

Eric

[Dhiraj Ambigapathi]

Hi Eric,
I can't change the log format as it is created by Docker. I'll need to check for parsing properly so it can extract relevant information.

[Eric Franco Fahnle]

Hi Dhiraj! Hope you're doing great. 

I see that when we started, the logs corresponded to a reverse proxy HTTP logs (GET, POST...) but the last example shows sshd in syslog format. Did your requirements change?

Also, would you mind telling me which reverse proxy are you using and its version? Maybe we can see if we can adjust the logging format.

Thanks!

Eric

[Dhiraj Ambigapathi]

Hi Eric,
After spending days on this, I referred few blogs on use cases and with help of wazuh support came up with the following decoder.
<decoder name="web-sibling">
  <parent>json</parent>
  <use_own_name>true</use_own_name>
  <regex>client_ip\\":\\"(\S+)\\"</regex>
  <order>client_ip</order>
 </decoder>

<decoder name="web-sibling">
  <parent>json</parent>
  <use_own_name>true</use_own_name>
  <regex>server_id\\":\\"(\S+)\\"</regex>
  <order>server_id</order>
 </decoder>

It works and I'm getting logs properly now for Web Access and Reverse Proxy Docker. I don't need any <out_format> for this to work, just decoders and rules needs to be more specific to avoid conflict in future.