Decoder issue

[MajorFudge]

Hello team,

I'm encountered some problems with custom decoders and rules for syslog traffic.

1. Logs received by Fluentd for parsing and normalizing

2. Fluentd sends logs to the Wazuh Manager

3. There are custom rules and decoders implemented

The problem is that alerts do not generated in /alerts.json file.

Wazuh version:

/var/ossec/bin/wazuh-control -j info
{"error":0,"data":[{"WAZUH_VERSION":"v4.4.0"},{"WAZUH_REVISION":"40405"},{"WAZUH_TYPE":"server"}]}

### Log sample
'aN..T).....<13>May 24 01:00:09 wfluentd01 checkpoint: {"timestamp":"2023-05-24T01:00:08+03:00","host":"10.196.8.12","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1684879208","source.ip":"10.197.1.220","destination.ip":"10.196.1.120","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"993","source.port":"40108","device.product.family":"Network"}'

### Custom decoder
<decoder name="checkpoint_json_extractor">
    <prematch>checkpoint: </prematch>
    <plugin_decoder offset="after_prematch">JSON_Decoder</plugin_decoder>
</decoder>

### Custom rule
<group name="fluentd parser rules">
  <rule id="100001" level="5">
    <decoded_as>checkpoint_json_extractor</decoded_as>    
    <match>checkpoint</match>
    <description>checkpoint_traffic_logs.</description>
  </rule>
</group>

### Logtest via  /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.4.0
Type one log per line


**Phase 1: Completed pre-decoding.
full event: 'aN..T).....<13>May 24 01:00:09 wfluentd01 checkpoint: {"timestamp":"2023-05-24T01:00:08+03:00","host":"10.196.8.12","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1684879208","source.ip":"10.97.10.22","destination.ip":"10.196.1.120","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"993","source.port":"40108","device.product.family":"Network"}'

**Phase 2: Completed decoding.
name: 'checkpoint_json_extractor'
destination.ip: '10.196.1.120'
destination.port: '993'
destination.proto: '6'
device.layer_name: 'Network'
device.layer_uuid: '8a994dd3-993e-4c0c-92a1-a8630b153f4c'
device.match_id: '18'
device.product.family: 'Network'
device.product.name: 'VPN-1 & FireWall-1'
device.row.end: '0'
device.row.start: '0'
device.rule_name: 'TEMP'
device.rule_uid: 'd0cc4a57-21e7-4294-b489-ec2ab7994016'
device.up_match.input.start: 'TABLE_START'
device.up_match_table.end: 'TABLE_END'
filed1: '-'
host: '10.196.8.12'
log.delay: '1684879208'
log.id: '1.3.6.1.4.1.2620'
log.type: 'Log'
source.ip: '10.197.1.220'
source.port: '40108'
syslogtag: 'CP-GW'
timestamp: '2023-05-24T01:00:08+03:00'

**Phase 3: Completed filtering (rules).
id: '100001'
level: '5'
description: 'checkpoint_traffic_logs.'
groups: '['fluentd parser rules']'
firedtimes: '1'
mail: 'False'
**Alert to be generated.


### Events in archives.json after logall_json
sudo tail -n 20 /var/ossec/logs/archives/archives.json | grep checkpoint

{"timestamp":"2023-05-24T21:51:24.481+0300","agent":{"id":"000","name":"wazuhserver01"},"manager":{"name":"wazuhserver01"},"id":"1684954284.498855","cluster":{"name":"wazuh","node":"wazuhserver01"},"full_log":"May 24 21:51:24 wfluentd01 checkpoint: {\"timestamp\":\"2023-05-24T21:51:24+03:00\",\"host\":\"10.196.8.12\",\"syslogtag\":\"CP-GW\",\"filed1\":\"-\",\"log.type\":\"Log\",\"log.id\":\"1.3.6.1.4.1.2620\",\"log.delay\":\"1684954284\",\"destination.service\":\"http\",\"source.ip\":\"10.197.1.220\",\"destination.ip\":\"10.196.12.22\",\"destination.proto\":\"6\",\"device.up_match.input.start\":\"TABLE_START\",\"device.row.start\":\"0\",\"device.match_id\":\"18\",\"device.layer_uuid\":\"8a994dd3-993e-4c0c-92a1-a8630b153f4c\",\"device.layer_name\":\"Network\",\"device.rule_uid\":\"d0cc4a57-21e7-4294-b489-ec2ab7994016\",\"device.rule_name\":\"TEMP\",\"device.row.end\":\"0\",\"device.up_match_table.end\":\"TABLE_END\",\"device.product.name\":\"VPN-1 & FireWall-1\",\"destination.port\":\"80\",\"source.port\":\"59212\",\"device.product.family\":\"Network\"}","predecoder":{"program_name":"checkpoint","timestamp":"May 24 21:51:24","hostname":"wfluentd01"},"decoder":{},"location":"10.197.8.12"}

{"timestamp":"2023-05-24T21:51:24.481+0300","agent":{"id":"000","name":"wazuhserver01"},"manager":{"name":"wazuhserver01"},"id":"1684954284.498855","cluster":{"name":"wazuh","node":"wazuhserver01"},"full_log":"May 24 21:51:24 wfluentd01 checkpoint: {\"timestamp\":\"2023-05-24T21:51:24+03:00\",\"host\":\"10.196.8.12\",\"syslogtag\":\"CP-GW\",\"filed1\":\"-\",\"log.type\":\"Log\",\"log.id\":\"1.3.6.1.4.1.2620\",\"log.delay\":\"1684954284\",\"destination.service\":\"Modbus\",\"source.ip\":\"10.96.44.111\",\"destination.ip\":\"10.199.8.13\",\"destination.proto\":\"6\",\"device.up_match.input.start\":\"TABLE_START\",\"device.row.start\":\"0\",\"device.match_id\":\"18\",\"device.layer_uuid\":\"8a994dd3-993e-4c0c-92a1-a8630b153f4c\",\"device.layer_name\":\"Network\",\"device.rule_uid\":\"d0cc4a57-21e7-4294-b489-ec2ab7994016\",\"device.rule_name\":\"TEMP\",\"device.row.end\":\"0\",\"device.up_match_table.end\":\"TABLE_END\",\"device.product.name\":\"VPN-1 & FireWall-1\",\"destination.port\":\"502\",\"source.port\":\"51490\",\"device.product.family\":\"Network\"}","predecoder":{"program_name":"checkpoint","timestamp":"May 24 21:51:24","hostname":"wfluentd01"},"decoder":{},"location":"10.197.8.12"}

It contains some additional symbols "\" as a separator.

I tried to set up a rsyslog server with debug mode on wazuh manager to see how my normalized log looks like:
Rsyslog debug output:
Debug line with all properties:
FROMHOST: '10.197.7.180', fromhost-ip: '10.197.7.180', HOSTNAME: 'wfluentd01', PRI: 13,
syslogtag 'checkpoint:', programname: 'checkpoint', APP-NAME: 'checkpoint', PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 27 12:30:40', STRUCTURED-DATA: '-',
msg: ' {"timestamp":"2023-05-27T12:30:39+03:00","host":"10.196.5.120","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1685179839","destination.service":"https","source.ip":"10.197.1.210","destination.ip":"10.196.1.110","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"443","source.port":"51796","device.product.family":"Network"}'
escaped msg: ' {"timestamp":"2023-05-27T12:30:39+03:00","host":"10.196.5.120","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1685179839","destination.service":"https","source.ip":"10.197.1.210","destination.ip":"10.196.1.110","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"443","source.port":"51796","device.product.family":"Network"}'
inputname: imudp rawmsg: '<13>May 27 12:30:40 wfluentd01 checkpoint: {"timestamp":"2023-05-27T12:30:39+03:00","host":"10.196.5.120","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1685179839","destination.service":"https","source.ip":"10.197.1.210","destination.ip":"10.196.1.110","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"443","source.port":"51796","device.product.family":"Network"}'

It looks ok.wserver01

Also I see that syslog is ok in the tcpdump.
### TCPDUMP OUTPUT
~$ sudo tcpdump port 10514 -AA
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:43:53.387534 IP 10.197.7.180.39090 > wserver01.10514: UDP, length 776
.PV..s.PV.....E..$D.@.@.B.
aN.
aN...)....c<13>May 27 12:43:53 wfluentd01 checkpoint: {"timestamp":"2023-05-27T12:43:48+03:00","host":"10.196.5.120","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1685180628","destination.service":"https","source.ip":"10.97.10.22","destination.ip":"10.96.10.12","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"443","source.port":"55958","device.product.family":"Network"}
12:43:53.387564 IP 10.197.7.180.39090 > wserver01.10514: UDP, length 776
.PV..s.PV.....E..$D.@.@.B.
aN.

Looking forward to your reply.

Best regards,

Kirill

[Damian Nicastro]

Hello Krill:

I hope you are fine

I have tested your rule and decoder creating a file with the log sample you have sent and the rule is triggered in the alerts.json file:

{"timestamp":"2023-05-29T11:44:37.225+0000","rule":{"level":5,"description":"checkpoint_traffic_logs.","id":"100101","firedtimes":1,"mail":false,"groups":["fluentd parser rules"]},"agent":{"id":"000","name":"wazuh-server"},"manager":{"name":"wazuh-server"},"id":"1685360677.4494573","full_log":"'aN..T).....<13>May 24 01:00:09 wfluentd01 checkpoint: {\"timestamp\":\"2023-05-24T01:00:08+03:00\",\"host\":\"10.196.8.12\",\"syslogtag\":\"CP-GW\",\"filed1\":\"-\",\"log.type\":\"Log\",\"log.id\":\"1.3.6.1.4.1.2620\",\"log.delay\":\"1684879208\",\"source.ip\":\"10.197.1.220\",\"destination.ip\":\"10.196.1.120\",\"destination.proto\":\"6\",\"device.up_match.input.start\":\"TABLE_START\",\"device.row.start\":\"0\",\"device.match_id\":\"18\",\"device.layer_uuid\":\"8a994dd3-993e-4c0c-92a1-a8630b153f4c\",\"device.layer_name\":\"Network\",\"device.rule_uid\":\"d0cc4a57-21e7-4294-b489-ec2ab7994016\",\"device.rule_name\":\"TEMP\",\"device.row.end\":\"0\",\"device.up_match_table.end\":\"TABLE_END\",\"device.product.name\":\"VPN-1 & FireWall-1\",\"destination.port\":\"993\",\"source.port\":\"40108\",\"device.product.family\":\"Network\"}'","decoder":{"name":"checkpoint_json_extractor"},"data":{"timestamp":"2023-05-24T01:00:08+03:00","host":"10.196.8.12","syslogtag":"CP-GW","filed1":"-","log":{"type":"Log","id":"1.3.6.1.4.1.2620","delay":"1684879208"},"source":{"ip":"10.197.1.220","port":"40108"},"destination":{"ip":"10.196.1.120","proto":"6","port":"993"},"device":{"up_match":{"input":{"start":"TABLE_START"}},"row":{"start":"0","end":"0"},"match_id":"18","layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","layer_name":"Network","rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","rule_name":"TEMP","up_match_table":{"end":"TABLE_END"},"product":{"name":"VPN-1 & FireWall-1","family":"Network"}}},"location":"/var/log/fluentd.log"}

Please, notice that the beginning of the "full_log" field is different than in your archives.json. It seems that you are not receiving the same sample of log and that why the rule is not matching. You might to check exactly how the log is coming and modify the decoder accordingly

I hope this helps.

Thanks

[MajorFudge]

I think that problem in the decoder.

This event is a syslog traffic. Based on wazuh documentation I can't use "prematch" to match syslog header.

(https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/decoders.html)

So this line in the decoder simply not working: <prematch>checkpoint: </prematch>.

Maybe I need to use regexp to match the begining of the line and <plugin_decoder offset="after_regexp">JSON_Decoder</plugin_decoder>

I collected a tcpdump to see the input event. It contains some sysmbols and a syslog priority. Wazuh cuts these symbols and a priority and put event to the archives without it.

### TCPDUMP OUTPUT
~$ tcpdump port 10514 -AA

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:43:53.387534 IP 10.197.7.180.39090 > wserver01.10514: UDP, length 776
.PV..s.PV.....E..$D.@.@.B.
aN.
aN...)....c<13>May 27 12:43:53 wfluentd01 checkpoint: {"timestamp":"2023-05-27T12:43:48+03:00","host":"10.196.5.120","syslogtag":"CP-GW","filed1":"-","log.type":"Log","log.id":"1.3.6.1.4.1.2620","log.delay":"1685180628","destination.service":"https","source.ip":"10.97.10.22","destination.ip":"10.96.10.12","destination.proto":"6","device.up_match.input.start":"TABLE_START","device.row.start":"0","device.match_id":"18","device.layer_uuid":"8a994dd3-993e-4c0c-92a1-a8630b153f4c","device.layer_name":"Network","device.rule_uid":"d0cc4a57-21e7-4294-b489-ec2ab7994016","device.rule_name":"TEMP","device.row.end":"0","device.up_match_table.end":"TABLE_END","device.product.name":"VPN-1 & FireWall-1","destination.port":"443","source.port":"55958","device.product.family":"Network"}
12:43:53.387564 IP 10.197.7.180.39090 > wserver01.10514: UDP, length 776
.PV..s.

понедельник, 29 мая 2023 г. в 16:53:49 UTC+2, Damian Nicastro:

[MajorFudge]

I tried something like that:

<decoder name="checkpoint">
    <program_name>checkpoint</program_name>
</decoder>

<decoder name="checkpoint_child">
    <parent>checkpoint</parent>
    <regex type="pcre2">(wfluentd01\-srv\-stage)\s(checkpoint\:)\s</regex>
    <order>srcuser,system_name</order>
    <plugin_decoder offset="after_regex">JSON_Decoder</plugin_decoder>
</decoder>

But I receive an error:

{"timestamp":"2023/05/29 18:36:57","tag":"wazuh-analysisd","level":"error","description":"(2111): Additional data to plugin decoder: 'checkpoint_child'."}
{"timestamp":"2023/05/29 18:36:57","tag":"wazuh-analysisd","level":"critical","description":"(1202): Configuration error at 'etc/decoders/local_decoder.xml'."}

понедельник, 29 мая 2023 г. в 17:30:19 UTC+2, MajorFudge:

[MajorFudge]

Yeah, i finally made it!

I changed the decoder:

<decoder name="checkpoint">
    <program_name>checkpoint</program_name>

    <plugin_decoder>JSON_Decoder</plugin_decoder>
</decoder>

I guees in case of syslog "plugin_decoder" works with msg field, so you can ignore syslog headers.

понедельник, 29 мая 2023 г. в 17:39:33 UTC+2, MajorFudge:

[Damian Nicastro]

Hello MajorFude:

I am glad you could solve it. 

Obviously, there are several ways to write the decoders. The <program_name> ta takes advantage of the pre-decoded "program_name" field to select the decoder that will match the log.

Thanks