Fortigate decoder not working properly

1,288 views

Skip to first unread message

fnuzon

unread,

Jun 21, 2018, 4:55:23 AM6/21/18

to Wazuh mailing list

I have a problem with fortigate decoder. Fortigate syntax was different to decoder and i added double quotes to decoder

<prematch>date=\S+ time=\.+ devname="(\S+)" devid="(FG\w+)" logid="(\d+)" </prematch>

<type>syslog</type>

</decoder>

and now it recognizes that its fortigate, but now the logs stopped coming to kibana even though all the packets from firewall still keeps coming to archive.log but they dont appear to be in alerts.json so i came to conclusion that it still gets stuck on decoder or rule. How do i proceed from this. Only modification to decoder was the double quotes.

Log for logtest:

2018 Jun 20 09:23:35 TEST->127.0.0.1 date=2018-06-20 time=12:23:34 devname="TEST" devid="FGT5AA92849234" logid="111111111" type="utm" subtype="webfilter" eventtype="urlfilter" level="information" vd="root" eventtime=22222222 urlfilteridx=0 policyid=10 sessionid=333333333 srcip=4.4.4.4 srcport=44 srcintf="port4" srcintfrole="undefined" dstip=5.5.5.5 dstport=55 dstintf="port5" dstintfrole="undefined" proto=2 service="HTTP" hostname="google.com" profile="TEST" action="passthrough" reqtype="direct" url="/" sentbyte=202 rcvdbyte=0 direction="outgoing" msg="URL was exempted because it is in the URL filter list"

/var/ossec/bin/ossec-logtest -v

2018/06/21 07:27:33 ossec-testrule: INFO: Started (pid: 26621).

ossec-testrule: Type one log per line.

**Phase 1: Completed pre-decoding.

full event: '2018 Jun 20 09:23:35 TEST->127.0.0.1 date=2018-06-20 time=12:23:34 devname="TEST" devid="FGT5AA92849234" logid="111111111" type="utm" subtype="webfilter" eventtype="urlfilter" level="information" vd="root" eventtime=22222222 urlfilteridx=0 policyid=10 sessionid=333333333 srcip=4.4.4.4 srcport=44 srcintf="port4" srcintfrole="undefined" dstip=5.5.5.5 dstport=55 dstintf="port5" dstintfrole="undefined" proto=2 service="HTTP" hostname="google.com" profile="TEST" action="passthrough" reqtype="direct" url="/" sentbyte=202 rcvdbyte=0 direction="outgoing" msg="URL was exempted because it is in the URL filter list"'

timestamp: '2018 Jun 20 09:23:35'

hostname: 'AZLX03'

program_name: '(null)'

log: 'TEST->127.0.0.1 date=2018-06-20 time=12:23:34 devname="TEST" devid="FGT5AA92849234" logid="111111111" type="utm" subtype="webfilter" eventtype="urlfilter" level="information" vd="root" eventtime=22222222 urlfilteridx=0 policyid=10 sessionid=333333333 srcip=4.4.4.4 srcport=44 srcintf="port4" srcintfrole="undefined" dstip=5.5.5.5 dstport=55 dstintf="port5" dstintfrole="undefined" proto=2 service="HTTP" hostname="google.com" profile="TEST" action="passthrough" reqtype="direct" url="/" sentbyte=202 rcvdbyte=0 direction="outgoing" msg="URL was exempted because it is in the URL filter list"'

**Phase 2: Completed decoding.

decoder: 'fortigate-firewall-v5'

**Rule debugging:

Trying rule: 1 - Generic template for all syslog rules.

*Rule 1 matched.

*Trying child rules.

Trying rule: 600 - Active Response Messages Grouped

Trying rule: 200 - Grouping of wazuh rules.

Trying rule: 2100 - NFS rules grouped.

Trying rule: 2507 - OpenLDAP group.

Trying rule: 2550 - rshd messages grouped.

Trying rule: 2701 - Ignoring procmail messages.

Trying rule: 2800 - Pre-match rule for smartd.

Trying rule: 5100 - Pre-match rule for kernel messages

Trying rule: 5200 - Ignoring hpiod for producing useless logs.

Trying rule: 2830 - Crontab rule group.

Trying rule: 5300 - Initial grouping for su messages.

Trying rule: 5905 - useradd failed.

Trying rule: 5400 - Initial group for sudo messages

Trying rule: 9100 - PPTPD messages grouped

Trying rule: 9200 - Squid syslog messages grouped

Trying rule: 2900 - Dpkg (Debian Package) log.

Trying rule: 2930 - Yum logs.

Trying rule: 2931 - Yum logs.

Trying rule: 2940 - NetworkManager grouping.

Trying rule: 2943 - nouveau driver grouping

Trying rule: 3100 - Grouping of the sendmail rules.

Trying rule: 3190 - Grouping of the smf-sav sendmail milter rules.

Trying rule: 3300 - Grouping of the postfix reject rules.

Trying rule: 3320 - Grouping of the postfix rules.

Trying rule: 3390 - Grouping of the clamsmtpd rules.

Trying rule: 3395 - Grouping of the postfix warning rules.

Trying rule: 3500 - Grouping for the spamd rules

Trying rule: 3600 - Grouping of the imapd rules.

Trying rule: 3700 - Grouping of mailscanner rules.

Trying rule: 3800 - Grouping of Exchange rules.

Trying rule: 3900 - Grouping for the courier rules.

Trying rule: 4300 - Grouping of PIX rules

Trying rule: 4500 - Grouping for the Netscreen Firewall rules

Trying rule: 4700 - Grouping of Cisco IOS rules.

Trying rule: 4800 - SonicWall messages grouped.

Trying rule: 5500 - Grouping of the pam_unix rules.

Trying rule: 5556 - unix_chkpwd grouping.

Trying rule: 5600 - Grouping for the telnetd rules

Trying rule: 5700 - SSHD messages grouped.

Trying rule: 5757 - Bad DNS mapping.

Trying rule: 6100 - Solaris BSM Auditing messages grouped.

Trying rule: 6200 - Asterisk messages grouped.

Trying rule: 6300 - Grouping for the MS-DHCP ipv4 rules.

Trying rule: 6350 - Grouping for the MS-DHCP ipv6 rules.

Trying rule: 7200 - Arpwatch messages grouped.

Trying rule: 7300 - Grouping of Symantec AV rules.

Trying rule: 7400 - Grouping of Symantec Web Security rules.

Trying rule: 7600 - Grouping of Trend OSCE rules.

Trying rule: 9300 - Grouping for the Horde imp rules.

Trying rule: 9400 - Roundcube messages grouped.

Trying rule: 9500 - Wordpress messages grouped.

Trying rule: 9600 - cimserver messages grouped.

Trying rule: 9700 - Dovecot Messages Grouped.

Trying rule: 9770 - dovecot-info grouping.

Trying rule: 9800 - Grouping for the vm-pop3d rules.

Trying rule: 9900 - Grouping for the vpopmail rules.

Trying rule: 11100 - Grouping for the ftpd rules.

Trying rule: 11200 - Grouping for the proftpd rules.

Trying rule: 11300 - Grouping for the pure-ftpd rules.

Trying rule: 11310 - Rule grouping for pure ftpd transfers.

Trying rule: 11400 - Grouping for the vsftpd rules.

Trying rule: 11500 - Grouping for the Microsoft ftp rules.

Trying rule: 12100 - Grouping of the named rules

Trying rule: 13100 - Grouping for the smbd rules.

Trying rule: 13106 -

Trying rule: 14100 - Grouping of racoon rules.

Trying rule: 14200 - Grouping of Cisco VPN concentrator rules

Trying rule: 19100 - VMWare messages grouped.

Trying rule: 19101 - VMWare ESX syslog messages grouped.

Trying rule: 30100 - Apache messages grouped.

Trying rule: 31200 - Grouping of Zeus rules.

Trying rule: 31300 - Nginx messages grouped.

Trying rule: 31404 - PHP Warning message.

Trying rule: 31405 - PHP Fatal error.

Trying rule: 31406 - PHP Parse error.

Trying rule: 40700 - Systemd rules

Trying rule: 40900 - firewalld grouping

Trying rule: 50100 - MySQL messages grouped.

Trying rule: 50500 - PostgreSQL messages grouped.

Trying rule: 51000 - Grouping for dropbear rules.

Trying rule: 51500 - Grouping of bsd_kernel alerts

Trying rule: 51521 - Grouping for groupdel rules.

Trying rule: 51523 - No core dumps.

Trying rule: 51525 - ftp-proxy cannot connect to a server.

Trying rule: 51526 - Hard drive is dying.

Trying rule: 51527 - CARP master to backup.

Trying rule: 51528 - Duplicate IPv6 address.

Trying rule: 51529 - Could not load a firmware.

Trying rule: 51530 - hotplugd could not open a file.

Trying rule: 51532 - Bad ntp peer.

Trying rule: 51550 - doas grouping

Trying rule: 52500 - Clamd messages grouped.

Trying rule: 52501 - ClamAV: database update

Trying rule: 53500 - OpenSMTPd grouping.

Trying rule: 500000 - Unbound grouping.

Trying rule: 80000 - Puppet Master messages grouped.

Trying rule: 80001 - Puppet Agent messages grouped.

Trying rule: 80100 - Netscaler messages grouped.

Trying rule: 80200 - Amazon alerts.

Trying rule: 80500 - Serv-u messages grouped.

Trying rule: 80700 - Audit: messages grouped.

Trying rule: 81100 - USB messages grouped.

Trying rule: 81300 - Redis messages grouped.

Trying rule: 81400 - OpenSCAP messages grouped.

Trying rule: 81600 - Fortigate v3 messages grouped.

Trying rule: 81601 - Fortigate v4 messages grouped.

Trying rule: 81602 - Fortigate v5 messages grouped.

*Rule 81602 matched.

*Trying child rules.

Trying rule: 81603 - Fortigate messages grouped.

*Rule 81603 matched.

*Trying child rules.

Trying rule: 81620 - Fortigate: URL Blocked by Firewall.

Trying rule: 81628 - Fortigate Attack Detected

Trying rule: 81608 - Fortigate: Configuration changed.

Trying rule: 81604 - Fortigate: IP Sec DPD Failed.

Trying rule: 81606 - Fortigate: Login failed.

Trying rule: 81610 - Fortigate: Default tunneling setting. Could be IPS.

Trying rule: 81614 - Fortigate: SSL VPN User failed login attempt

Trying rule: 81616 - Fortigate: User logout successful

Trying rule: 81612 - Fortigate: Firewall configuration changes

Trying rule: 81622 - Fortigate: VPN User connected.

Trying rule: 81624 - Fortigate: VPN User disconnected.

Trying rule: 81626 - Fortigate: User successfully logged into firewall interface.

Trying rule: 81629 - Fortigate Attack Dropped

Trying rule: 81618 - Fortigate: Traffic to be aware of.

**Phase 3: Completed filtering (rules).

Rule id: '81603'

Level: '0'

Description: 'Fortigate messages grouped.'

miguel...@wazuh.com

unread,

Jul 2, 2018, 10:15:05 AM7/2/18

to Wazuh mailing list

Hi fnuzon. I am Miguel Ángel from Wazuh support team.

It seems that there are some differences between your FortiGate events and those we got.

Just to be sure. What version of FortiOS (FortiGate) are you using?

To answer to your doubts, as you said, alerts.log and alerts.json contains just the alerts generated by Wazuh and Kibana receive then through Logstash and Elasticsearch. So yes it's probably something wrong with the decoders and rules.

I will try to help you with your new decoders!

First of all I recommend you to copy the original Wazuh decoders and rules from the ruleset to your custom rules and decoders directory.

By default etc/decoders and etc/rules directories are used for custom rules and decoders so you can work with these directories:

 cp /var/ossec/ruleset/decoders/0100-fortigate_decoders.xml /var/ossec/etc/decoders/fortigate_decoders-0100.xml
 cp /var/ossec/ruleset/rules/0390-fortigate_rules.xml /var/ossec/etc/rules/fortigate_rules-0390.xml

Try to name them differently from the originals to avoid some duplicate issues.

Also, check your ossec.conf to exclude the original rules and decoders files to also avoid duplicates.
You should have something like this in your ruleset section in your ossec.conf file:

<ruleset>
    <!-- Default ruleset -->
    <decoder_dir>ruleset/decoders</decoder_dir>
    <decoder_exclude>0100-fortigate_decoders.xml</decoder_exclude>
    <rule_dir>ruleset/rules</rule_dir>
    <rule_exclude>0215-policy_rules.xml</rule_exclude>
    <rule_exclude>0390-fortigate_rules.xml</rule_exclude>
    <list>etc/lists/audit-keys</list>


    <!-- User-defined ruleset -->
    <decoder_dir>etc/decoders</decoder_dir>
    <rule_dir>etc/rules</rule_dir>
  </ruleset>

Now you are ready to work with your custom rules and decoders:

First, we have the parent decoder for FortiGate v5 which catch the first part of the event:

<!-- FortiOS 5.0 via syslog -->
<decoder name="fortigate-firewall-v5">
    <prematch>date=\S+ time=\.+ devname=\S+ devid=FG\w+ logid=\d+ </prematch>
    <type>syslog</type>
</decoder>

In order to work with your event we can modify it just like you did:

<!-- FortiOS 5.0 via syslog -->
<decoder name="fortigate-firewall-v5">
    <prematch>date=\S+ time=\.+ devname="\S+" devid="FG\w+" logid="\d+" </prematch>
    <type>syslog</type>
</decoder>

Then reading your test event we can see that it's an event with type=utm and subtype=webfilter so let's modify the decoder that catches these type of event:

This is the Wazuh original:

<decoder name="fortigate-firewall-v5-utm-webfilter">
    <parent>fortigate-firewall-v5</parent>
    <prematch offset="after_parent">type=utm subtype=webfilter</prematch>
    <regex offset="after_parent">level=(\S+) \.+ user="(\.*)" srcip=(\S+) srcport=(\d+) \.+ dstip=(\S+) dstport=(\d+) \.+ hostname="(\.+)" \.+ action=(\w+)</regex>
    <order>status,srcuser,srcip,srcport,dstip,dstport,url,action</order>
</decoder>

And this is an example that works with your event:

<decoder name="fortigate-firewall-v5-utm-webfilter">
    <parent>fortigate-firewall-v5</parent>
    <prematch offset="after_parent">type="utm" subtype="webfilter"</prematch>
    <regex offset="after_parent">level="(\w+)" \.+ srcip=(\S+) srcport=(\d+) \.+ dstip=(\S+) dstport=(\d+) \.+ hostname="(\.+)" \.+ action="(\w+)"</regex>
    <order>status,srcip,srcport,dstip,dstport,url,action</order>
</decoder>

You have to be very literal using regular expressions (Regex). See the changes with the double quotes character.
There is no user field in your event so I removed it.

If you want more info about Regex take a look here: https://documentation.wazuh.com/3.x/user-manual/ruleset/ruleset-xml-syntax/regex.html

Hope I helped you with these answers. Don't hesitate to ask me anything else!

Kind regards,
Miguel Ángel

Louis Bernardo

unread,

Aug 1, 2018, 11:17:43 AM8/1/18

to Wazuh mailing list

Hi Miguel,

I am also busy working on rules for Fortigate. I see that there are significant differences between the original and the new Fortios logs, it seems they have jumbled the fields and added "" to certain values while removing it from others. Our version of Fortios is 5.6.4. I have attached the log reference manual for that specific one, there is a ton of information in there.

I have switched my syslog output to csv as cef seemed problematic with consistency (and detail), sample included below. I have tried to adjust the rules accordingly but it isn't working so well. I have been considering build an entirely new ruleset.

<185>date=2018-07-31,time=15:35:19,devname="XXXX-XX-XX-XX-XX",devid="FGXXXXXXXXXXXXXX",logid="0419016384",type="utm",subtype="ips",eventtype="signature",level="alert",vd="XXXX-XX-XX",eventtime=1533044119,severity="info",srcip=10.10.10.10,srccountry="United States",dstip=11.11.11.11,srcintf="INT2",srcintfrole="undefined",dstintf="port20",dstintfrole="undefined",sessionid=1350677726,action="detected",proto=6,service="SSL",policyid=516,attack="SSL.Anonymous.Ciphers.Negotiation",srcport=17864,dstport=7971,direction="outgoing",attackid=43544,profile="default",ref="http://www.fortinet.com/ids/VID43544",incidentserialno=680172673,msg="applications3: SSL.Anonymous.Ciphers.Negotiation,"

Considering the volume do you think it would be possible for a collaboration to update the rules for Fortios 5.6.4? Apparently the big change happened in 5.3, so new rules should cover 5.3 and up.

Cheers,

Louis

FortiOS-5.6.4-Log-Reference.pdf

Reply all

Reply to author

Forward

0 new messages