Help for creating a custom decoder for pfBlockerNG logs

[mauro....@cmcc.it]

Dear Users,

I recently started collecting external logs provided by pfBlockerNG package (available and installed in pfSense) in Wazuh.

Unfortunately, I'm having some problem processing these logs using decoders/rules.

At this moment:

- logs are correctly received and saved by Wazuh syslog server in /var/log/pfsense/pfsense_pfB.log;

- ossec.conf file has been updated as follow

  <localfile>

    <log_format>syslog</log_format>

    <location>/var/log/pfsense/pfsense_pfB.log</location>

  </localfile>

Unfortunately, it seems that the already available (standard) pfSense decoder/rules are not able to process the log lines similar to the following ones:

2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+

TEST n.1:

/var/ossec/bin/wazuh-logtest-legacy

2023/06/07 13:24:58 wazuh-testrule: INFO: Started (pid: 12315).

Since Wazuh v4.1.0 this binary is deprecated. Use wazuh-logtest instead

wazuh-testrule: Type one log per line.

2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+

**Phase 1: Completed pre-decoding.

       full event: '2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/2'

       timestamp: '2023-06-07T14:30:58+02:00'

       hostname: 'pfSense_LAN'

       program_name: '(null)'

       log: '- Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/2'

**Phase 2: Completed decoding.

       No decoder matched.

TEST n. 2

/var/ossec/bin/wazuh-logtest

Starting wazuh-logtest v4.4.1

Type one log per line

2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+

**Phase 1: Completed pre-decoding.

full event: '2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/2'

timestamp: '2023-06-07T14:30:58+02:00'

**Phase 2: Completed decoding.

No decoder matched.

Could you please help me to extract relevant fields from the logs using regex?

Watching the log line mentioned above, the relevant fields are:
Event time: Jun 7 14:30:58
interface name: em0
interface label: DMZ
action: block
Protocol: UDP
SRC IP: x.x.x.x
DST IP: y.y.y.y
RU: geo location 
pfB_top_v4: blacklist name

I tried to do ti by myself, but I'm very newbie to solve this problem :(

Thank you in advance,
Mauro

[John Soliani]

Hello Mauro,

Thank you for posting in our community!

Allow me to clarify the 3 phases in the logtest tool.

• **Phase 1: the pre-decoder stage, it’s an automated process that we can not manage, it is the first step the engine does to understand the log.
• **Phase 2: the decoder stage, this is the one we can have full control to decode the log accordingly.
• **Phase 3: the rules stage, we also have full control over this stage, the rules/conditions the log should match to trigger an alert.

As we can see in your briefing, in both tests, the 2nd phase showed the No decoder matched. message, and without decoders, we won’t be able to trigger alerts right?

While you take a look at our documentation here:

• Wazuh: Ruleset

Especially these links:

Custom rules and decoders

Adding new decoders and rules
Changing an existing rule
Changing an existing decoder

Dynamic fields

Traditional decoders
Dynamic decoders

Ruleset XML syntax

Decoders Syntax
Rules Syntax
Regular Expression Syntax
Perl-compatible Regular Expressions
Sibling Decoders

Testing decoders and rules

Configuration
Using the Wazuh dashboard and the command line tool
Using the Wazuh API

I will prepare a decoder for this log format you attached to the message.

2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+

I’ll keep you posted!

​

[mauro....@cmcc.it]

Hello John,

many many thanks for your detailed reply and very professional support.

I really appreciate it.

I'm looking forward to analyse the decoder you will prepare.

Meanwhile, I'm starting reading the links you provided.

Thanks again,

Mauro

[John Soliani]

Hey Mauro,

I’m back! Firstly, allow me to mention that the logtest-legacy tool shows you how to deal with this log, you need to decode this part of the log:
log: - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+

Here’s how the decoder would go:

In /var/ossec/decoders/local_decoder.xml add these lines:

<decoder name="pf"> <prematch>^- \w\w\w \d+ \d\d:\d\d:\d\d,\d+,</prematch> </decoder> <decoder name="pf_child"> <parent>pf</parent> <regex>^- (\w\w\w \d+ \d\d:\d\d:\d\d),(\d+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),(\S+),</regex> <order>evt_timestamp,evt_code,iface_name,iface_label,action,value4,value5,protocol,srcip,dstip,value9,value10,value11,value12,value13,value14,value15,value16,value17,value18</order> </decoder>

Then, we’ll need to modify the stock pfsense ruleID 87700 by adding these lines at the end of the file /var/ossec/rules/local_rules.xml:

<group name="pfsense,"> <rule id="87700" level="0" overwrite="yes"> <decoded_as>pf</decoded_as> <description>pfSense firewall rules grouped.</description> </rule> </group>

Now, let’s see how the wazuh engine reacts to the log you provided:

• echo '2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+' | /var/ossec/bin/wazuh-logtest

**Phase 1: Completed pre-decoding. full event: '2023-06-07T14:30:58+02:00 pfSense_LAN - Jun 7 14:30:58,1770023929,em0,DMZ,block,4,17,UDP,x.x.x.x,y.y.y.y,64974,53,out,RU,pfB_Top_v4,193.232.128.0/20,RU_v4,a.dns.ripn.net,wan,null,+' timestamp: '2023-06-07T14:30:58+02:00' **Phase 2: Completed decoding. name: 'pf' action: 'block' dstip: 'y.y.y.y' evt_code: '1770023929' evt_timestamp: 'Jun 7 14:30:58' iface_label: 'DMZ' iface_name: 'em0' protocol: 'UDP' srcip: 'x.x.x.x' value10: '53' value11: 'out' value12: 'RU' value13: 'pfB_Top_v4' value14: '193.232.128.0/20' value15: 'RU_v4' value16: 'a.dns.ripn.net' value17: 'wan' value18: 'null' value4: '4' value5: '17' value9: '64974' **Phase 3: Completed filtering (rules). id: '87701' level: '5' description: 'pfSense firewall drop event.' groups: '['pfsense', 'firewall_block']' firedtimes: '1' gpg13: '['4.12']' hipaa: '['164.312.a.1']' mail: 'False' nist_800_53: '['SC.7']' pci_dss: '['1.4']' tsc: '['CC6.7', 'CC6.8']' **Alert to be generated.

Hope this helps!
John.-

​

[Mauro Tridici]

OMG, John! it’s incredible! 

You are so fast :)

I took longer to copy the code than you to create it.

Many thanks again.

It works like a charm. So, now it’s my turn: I need to understand you code.

Have a great day.

Kind Regads,

Mauro

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/PYOjFtZ6K9o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/def48e6b-22ac-400a-bdfc-ea07ce593babn%40googlegroups.com.

[Mauro Tridici]

Hi again, John.

Last question, I promise :)

Taking a look at the Wazuh “Security Alerts” page and clicking on “JSON”, I see that all the relevant data are ok:

"data": { "value5": "17", "srcip": “x.x.x.x", "iface_name": "em0", "evt_timestamp": "Jun 7 18:07:40", "value4": "4", "value9": "42061", "protocol": "UDP", "value18": "null", "evt_code": "1770023929", "value15": "RU_v4", "action": "block", "value14": "194.190.120.0/21", "value17": "wan", "dstip": “y.y.y.y", "value16": "d.dns.ripn.net", "value11": "out", "value10": "53", "value13": "pfB_Top_v4", "iface_label": "DMZ", "value12": "RU"

but the geolocation info are always the same and they are not actually related to the srcip:

"@timestamp": "2023-06-07T16:06:32.424Z", "location": "/var/log/pfsense/pfsense_pfB.log", "GeoLocation": { "city_name": "Milan", "country_name": "Italy", "region_name": "Milan", "location": { "lon": 9.1889, "lat": 45.4707 } },

Is there a way to fix/update the real geolocation related to the SRCIP?

Thank you in advance,

Mauro

[John Soliani]

Hey Mauro,

My last message was deleted, here’s the answer again.
The data.srcip is working correctly for the GeoLocation, I tested using my IP, and the alert showed me my country, then I used the IP 193.232.128.10 and the alert showed me this:

"GeoLocation": { "country_name": "Russia", "location": { "lon": 37.6068, "lat": 55.7386 } }

Which is also correct.
You should check that pfsense documentation and make sure the order of the fields in the log is correct.

Regards,
John.-

​

[Mauro Tridici]

Hello John,

Thank you for your patience and for you reply.

I checked the pfsense documentation, the order seems to be correct.

I tried to heck the srcip in https://www.iplocation.net/ip-lookup site and … surprise!

The same IP address seems to be detected in different italian areas by the different geolocators shown in the web page mentioned above. The wrong one is Milan 

Geolocation data from DB-IP gives the wrong answer (Milan)

Geolocation data from IP2Location gives the right answer (because I know where is located the srcip public iP :-) )

Question: is there a way to change the geolocator used by Wazuh?

Thank you,

Mauro

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/PYOjFtZ6K9o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0d914070-3b0f-4da4-b8d5-1e82a6298dd4n%40googlegroups.com.

[John Soliani]

Hello Mauro,

 After doing some research, I found that the GeoLocation database of `OpenSearch` is outdated and you can update it manually to improve the detection. Here's how to do this:

Please see the steps below:

• The first thing will be to download the updated version of the databases used by the wazuh-indexer to enrich alerts with GeoIP information, you can sign-up here for a free account:

• Once you have your new account, sign in and you will be able to download the databases (see screenshot).

• Stop the wazuh-indexer service by running this command: systemctl stop wazuh-indexer

• Rename or move these files from the following directory: /usr/share/wazuh-indexer/modules/ingest-geoip/

• Extract the downloaded zip files and copy the three .mmdb files to the directory /usr/share/wazuh-indexer/modules/ingest-geoip/
  

• Change the ownership and permissions of the files with:  chown wazuh-indexer:wazuh-indexer /usr/share/wazuh-indexer/modules/ingest-geoip/GeoLite2-*  and  chmod 640 /usr/share/wazuh-indexer/modules/ingest-geoip/GeoLite2-*
  

• Start the wazuh-indexer service with systemctl start wazuh-indexer
  

GeoIP information should be correct now. See some before and after screenshots for some test events.

Please let me know if this helped!
John.-

[Mauro Tridici]

Hello John,

many many thanks for such detailed instructions and professional support.

I really appreciated it.

Thanks to you help, I was able to fix the issue.

I will save these important instructions in my notes and I will apply them on a regular basis.

Have a great week-end.

All the best,

Mauro

On 16 Jun 2023, at 22:13, 'John Soliani' via Wazuh mailing list <wa...@googlegroups.com> wrote:

Hello Mauro,

 After doing some research, I found that the GeoLocation database of `OpenSearch` is outdated and you can update it manually to improve the detection. Here's how to do this:

Please see the steps below:

• The first thing will be to download the updated version of the databases used by the wazuh-indexer to enrich alerts with GeoIP information, you can sign-up here for a free account:
• Once you have your new account, sign in and you will be able to download the databases (see screenshot).
• Stop the wazuh-indexer service by running this command: systemctl stop wazuh-indexer
• Rename or move these files from the following directory: /usr/share/wazuh-indexer/modules/ingest-geoip/
• Extract the downloaded zip files and copy the three .mmdb files to the directory /usr/share/wazuh-indexer/modules/ingest-geoip/
  
• Change the ownership and permissions of the files with:  chown wazuh-indexer:wazuh-indexer /usr/share/wazuh-indexer/modules/ingest-geoip/GeoLite2-*  and  chmod 640 /usr/share/wazuh-indexer/modules/ingest-geoip/GeoLite2-*
  
• Start the wazuh-indexer service with systemctl start wazuh-indexer
  

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bf606485-c83f-4ef3-979d-a7846eab91b2n%40googlegroups.com.
<2023-06-16_17-03.jpg>

[wal Bz]

dear users
i need some help here !
could you help me to parse this dnsbl.log in wazuh :

Sep 28 15:01:53 pfSense - DNSBL-Full,Sep 28 14:55:43,canabis-graine.com,192.168.210.101,-|GET / HTTP/1.1|Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0,DNSBL,DNSBL_UT1,canabis-graine.com,UT1_drogue,+

i want to know the timestamp, src ip, malicious domain (here is canabis-graine.com)...
NB: this line from dnsbl.log (from pfblockerng installed on pfsense) recieved by wazuh in /var/log/syslog with syslog-ng

      ossec.conf file :

             <localfile>

                    <log_format>syslog</log_format>

                    <location>/var/log/syslog</location>

             </localfile>

[wal Bz]

# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.5.2

Type one log per line

Sep 28 15:01:53 pfSense - DNSBL-Full,Sep 28 14:55:43,canabis-graine.com,192.168.210.101,-|GET / HTTP/1.1|Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0,DNSBL,DNSBL_UT1,canabis-graine.com,UT1_drogue,+

**Phase 1: Completed pre-decoding.
full event: 'Sep 28 15:01:53 pfSense - DNSBL-Full,Sep 28 14:55:43,canabis-graine.com,192.168.210.101,-|GET / HTTP/1.1|Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0,DNSBL,DNSBL_UT1,canabis-graine.com,UT1_drogue,+'
timestamp: 'Sep 28 15:01:53'
hostname: 'pfSense'

**Phase 2: Completed decoding.
No decoder matched.