exim decoder/rules not fired

[Frank Soyer]

Hi,
I made an upgrade from 2.8.3 to 2.9.4, for handling exim logs/rules. But this decoder or rules doesn't seems to be tested. Here is a debug session :

# bin/ossec-logtest -v
2018/06/25 11:32:41 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2018/06/25 11:32:41 ossec-testrule: INFO: Started (pid: 15189).
ossec-testrule: Type one log per line.

2017-01-23 03:44:14 dovecot_login authenticator failed for (hydra) [10.101.1.18]:35686: 535 Incorrect authentication data (set_id=user)

**Phase 1: Completed pre-decoding.
       full event: '2017-01-23 03:44:14 dovecot_login authenticator failed for (hydra) [10.101.1.18]:35686: 535 Incorrect authentication data (set_id=user)'
       hostname: 'logcollector'
       program_name: '(null)'
       log: '2017-01-23 03:44:14 dovecot_login authenticator failed for (hydra) [10.101.1.18]:35686: 535 Incorrect authentication data (set_id=user)'

**Phase 2: Completed decoding.
       decoder: 'windows-date-format'
       srcip: '10.101.1.18'
       dstuser: 'user'

**Rule debugging:
    Trying rule: 1 - Generic template for all syslog rules.
       *Rule 1 matched.
       *Trying child rules.
    Trying rule: 5500 - Grouping of the pam_unix rules.
    Trying rule: 5556 - unix_chkpwd grouping.
    Trying rule: 5700 - SSHD messages grouped.
    Trying rule: 5757 - Bad DNS mapping.
    Trying rule: 5600 - Grouping for the telnetd rules
    Trying rule: 2100 - NFS rules grouped.
    Trying rule: 2507 - OpenLDAP group.
    Trying rule: 2550 - rshd messages grouped.
    Trying rule: 2701 - Ignoring procmail messages.
    Trying rule: 2800 - Pre-match rule for smartd.
    Trying rule: 5100 - Pre-match rule for kernel messages
    Trying rule: 5200 - Ignoring hpiod for producing useless logs.
    Trying rule: 2830 - Crontab rule group.
    Trying rule: 5300 - Initial grouping for su messages.
    Trying rule: 5905 - useradd failed.
    Trying rule: 5400 - Initial group for sudo messages
    Trying rule: 9100 - PPTPD messages grouped
    Trying rule: 9200 - Squid syslog messages grouped
    Trying rule: 2900 - Dpkg (Debian Package) log.
       *Rule 2900 matched.
       *Trying child rules.
    Trying rule: 2902 - New dpkg (Debian Package) installed.
    Trying rule: 2903 - Dpkg (Debian Package) removed.
    Trying rule: 2901 - New dpkg (Debian Package) requested to install.

**Phase 3: Completed filtering (rules).
       Rule id: '2900'
       Level: '0'
       Description: 'Dpkg (Debian Package) log.'

as you can see, the exim rules are never tested... The line is the one given as example in exim_rule.xml.

The etc/decoder.xml contains exim decoders, and it is loaded, like exim_rules.xml

2018/06/25 11:32:30 ossec-testrule: INFO: Reading decoder file etc/decoder.xml.
2018/06/25 11:32:31 ossec-testrule: INFO: Started (pid: 15125).
2018/06/25 11:32:31 ossec-execd: INFO: Started (pid: 15148).
2018/06/25 11:32:31 ossec-analysisd: INFO: Reading decoder file etc/decoder.xml.
...
2018/06/25 11:32:31 ossec-analysisd: INFO: Reading rules file: 'exim_rules.xml'
...

How can I solve this ?

Thanks
Frank

[dan (ddp)]

[snip]

>> Trying rule: 2900 - Dpkg (Debian Package) log.
>> *Rule 2900 matched.
>> *Trying child rules.
>> Trying rule: 2902 - New dpkg (Debian Package) installed.
>> Trying rule: 2903 - Dpkg (Debian Package) removed.
>> Trying rule: 2901 - New dpkg (Debian Package) requested to install.
>>
>> **Phase 3: Completed filtering (rules).
>> Rule id: '2900'
>> Level: '0'
>> Description: 'Dpkg (Debian Package) log.'
>

This is what I'm getting with master:
2018/06/27 08:41:56 ossec-testrule: INFO: Reading local decoder file.

ossec-testrule: Type one log per line.

**Phase 1: Completed pre-decoding.
full event: '2017-01-23 03:44:14 dovecot_login authenticator
failed for (hydra) [10.101.1.18]:35686: 535 Incorrect authentica
tion data (set_id=user)'

hostname: 'rossak'

program_name: '(null)'
log: '2017-01-23 03:44:14 dovecot_login authenticator failed
for (hydra) [10.101.1.18]:35686: 535 Incorrect authentication da
ta (set_id=user)'

**Phase 2: Completed decoding.
decoder: 'windows-date-format'
srcip: '10.101.1.18'
dstuser: 'user'

**Phase 3: Completed filtering (rules).

Rule id: '13006'
Level: '5'
Description: 'Exim Auth failed'
**Alert to be generated.

It doesn't look like the exim tests are in the 2.9 branch.
Moving the exim decoders above the debian dpkg decoders might help,
but I can't really test it.

[FSoyer]

Hi Dan,

thank you for this highlight. Not sure how we can "move the exim decoders above the debian dpkg decoders" as... there is no dpkg decoders : it is rules, in syslog_rules.xml. So alphabetically probably loaded after "exim_rules.xml".

Just to understand, what means "Trying child rules"  in phase 3, is this relative to  "<group name="syslog,xxxxx,"> ? (If yes, I can confirm that the exim rules are not "affiliate" to syslog in 2.9, but I tried to add it to the group_name exim without success). It seemed to me like a configuration problem, so I do not see what is the link with ossec version ?

Last question : have you/can you launch the logtest with "-v" to see the rules checked on your side ? 

Thanks

[dan (ddp)]

On Thu, Jun 28, 2018 at 4:05 AM, FSoyer <frank...@gmail.com> wrote:
> Hi Dan,
> thank you for this highlight. Not sure how we can "move the exim decoders
> above the debian dpkg decoders" as... there is no dpkg decoders : it is

I was just confused. I thought there were dpkg decoders, but apparently not.

> rules, in syslog_rules.xml. So alphabetically probably loaded after
> "exim_rules.xml".
> Just to understand, what means "Trying child rules" in phase 3, is this

For reference:

Trying rule: 2900 - Dpkg (Debian Package) log.
*Rule 2900 matched.
*Trying child rules.

So rule 2900 matched your log message. So any rule that has
'<parent>2900</parent>' in it is a child rule of 2900.

> relative to "<group name="syslog,xxxxx,"> ? (If yes, I can confirm that the
> exim rules are not "affiliate" to syslog in 2.9, but I tried to add it to
> the group_name exim without success). It seemed to me like a configuration
> problem, so I do not see what is the link with ossec version ?
>

The version matters because the master branch gets the most work. I
don't keep a 2.9 system around, so I can't determine what's going on
there exactly.

> Last question : have you/can you launch the logtest with "-v" to see the
> rules checked on your side ?
>

2018/06/28 06:31:39 ossec-testrule: INFO: Reading local decoder file.
2018/06/28 06:31:40 ossec-testrule: INFO: Started (pid: 4225).

ossec-testrule: Type one log per line.



**Phase 1: Completed pre-decoding.
full event: '2017-01-23 03:44:14 dovecot_login authenticator
failed for (hydra) [10.101.1.18]:35686: 535 Incorrect authentication
data (set_id=user)'

hostname: 'alpine0'

program_name: '(null)'
log: '2017-01-23 03:44:14 dovecot_login authenticator failed
for (hydra) [10.101.1.18]:35686: 535 Incorrect authentication data
(set_id=user)'

**Phase 2: Completed decoding.
decoder: 'windows-date-format'
srcip: '10.101.1.18'
dstuser: 'user'

**Rule debugging:
Trying rule: 1 - Generic template for all syslog rules.
*Rule 1 matched.
*Trying child rules.

Trying rule: 5500 - Grouping of the pam_unix rules.
Trying rule: 5556 - unix_chkpwd grouping.
Trying rule: 5700 - SSHD messages grouped.
Trying rule: 5757 - Bad DNS mapping.
Trying rule: 5600 - Grouping for the telnetd rules
Trying rule: 2100 - NFS rules grouped.
Trying rule: 2507 - OpenLDAP group.
Trying rule: 2550 - rshd messages grouped.
Trying rule: 2701 - Ignoring procmail messages.
Trying rule: 2800 - Pre-match rule for smartd.
Trying rule: 5100 - Pre-match rule for kernel messages
Trying rule: 5200 - Ignoring hpiod for producing useless logs.
Trying rule: 2830 - Crontab rule group.
Trying rule: 5300 - Initial grouping for su messages.
Trying rule: 5905 - useradd failed.
Trying rule: 5400 - Initial group for sudo messages
Trying rule: 9100 - PPTPD messages grouped
Trying rule: 9200 - Squid syslog messages grouped

Trying rule: 2900 - Dpkg (Debian Package) log.

Trying rule: 2930 - Yum logs.
Trying rule: 2931 - Yum logs.
Trying rule: 2940 - NetworkManager grouping.
Trying rule: 2943 - nouveau driver grouping
Trying rule: 7200 - Grouping of the arpwatch rules.
Trying rule: 7300 - Grouping of Symantec AV rules.
Trying rule: 7400 - Grouping of Symantec Web Security rules.
Trying rule: 4300 - Grouping of PIX rules
Trying rule: 12100 - Grouping of the named rules
Trying rule: 13100 - Grouping for the smbd rules.
Trying rule: 13106 - (null)
Trying rule: 11400 - Grouping for the vsftpd rules.
Trying rule: 11300 - Grouping for the pure-ftpd rules.
Trying rule: 11310 - Rule grouping for pure ftpd transfers.
Trying rule: 11200 - Grouping for the proftpd rules.
Trying rule: 11500 - Grouping for the Microsoft ftp rules.
Trying rule: 11100 - Grouping for the ftpd rules.
Trying rule: 9300 - Grouping for the Horde imp rules.
Trying rule: 9400 - Roundcube messages grouped.
Trying rule: 9500 - Wordpress messages grouped.
Trying rule: 9600 - cimserver messages grouped.
Trying rule: 9900 - Grouping for the vpopmail rules.
Trying rule: 9800 - Grouping for the vm-pop3d rules.
Trying rule: 3900 - Grouping for the courier rules.
Trying rule: 30100 - Apache messages grouped.
Trying rule: 31300 - Nginx messages grouped.
Trying rule: 31404 - PHP Warning message.
Trying rule: 31405 - PHP Fatal error.
Trying rule: 31406 - PHP Parse error.
Trying rule: 50100 - MySQL messages grouped.
Trying rule: 50500 - PostgreSQL messages grouped.
Trying rule: 4700 - Grouping of Cisco IOS rules.
Trying rule: 4500 - Grouping for the Netscreen Firewall rules
Trying rule: 4800 - SonicWall messages grouped.
Trying rule: 3300 - Grouping of the postfix reject rules.
Trying rule: 3320 - Grouping of the postfix rules.
Trying rule: 3390 - Grouping of the clamsmtpd rules.
Trying rule: 3100 - Grouping of the sendmail rules.
Trying rule: 3190 - Grouping of the smf-sav sendmail milter rules.
Trying rule: 3600 - Grouping of the imapd rules.
Trying rule: 3700 - Grouping of mailscanner rules.
Trying rule: 9700 - Dovecot Messages Grouped.
Trying rule: 9770 - dovecot-info grouping.
Trying rule: 3800 - Grouping of Exchange rules.
Trying rule: 14100 - Grouping of racoon rules.
Trying rule: 14200 - Grouping of Cisco VPN concentrator rules
Trying rule: 3500 - Grouping for the spamd rules
Trying rule: 7600 - Grouping of Trend OSCE rules.
Trying rule: 31200 - Grouping of Zeus rules.
Trying rule: 6100 - Solaris BSM Auditing messages grouped.
Trying rule: 19100 - VMWare messages grouped.
Trying rule: 19101 - VMWare ESX syslog messages grouped.
Trying rule: 6300 - Grouping for the MS-DHCP rules.
Trying rule: 6350 - Grouping for the MS-DHCP rules.
Trying rule: 6200 - Asterisk messages grouped.
Trying rule: 600 - Active Response Messages Grouped
Trying rule: 51500 - Grouping of bsd_kernel alerts
Trying rule: 51521 - Grouping for groupdel rules.
Trying rule: 51523 - No core dumps.
Trying rule: 51525 - ftp-proxy cannot connect to a server.
Trying rule: 51526 - Hard drive is dying.
Trying rule: 51527 - CARP master to backup.
Trying rule: 51528 - Duplicate IPv6 address.
Trying rule: 51529 - Could not load a firmware.
Trying rule: 51530 - hotplugd could not open a file.
Trying rule: 51532 - Bad ntp peer.
Trying rule: 51550 - doas grouping
Trying rule: 52500 - Grouping of the clamd rules.
Trying rule: 52501 - ClamAV database update
Trying rule: 51000 - Grouping for dropbear rules.
Trying rule: 53500 - OpenSMTPd grouping.
Trying rule: 13000 - Exim SMTP Messages Grouped.
Trying rule: 13001 - dovecot messages grouped.
*Rule 13001 matched.
*Trying child rules.
Trying rule: 13006 - Exim Auth failed
*Rule 13006 matched.
*Trying child rules.
Trying rule: 13007 - Exim brute force attack (multiple auth failures).

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "ossec-list" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ossec-list+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[FSoyer]

Well, sorry if I have trouble understanding the process, but I found only "<parent>" tag in decoder.xml, with decoder parent names (not ids), not in rules files. So when I search child rules for id 2900 with parent tag, I find nothing.

Another thing about versions, what do you mean by "master" ? When I look at https://github.com/ossec/ossec-hids/releases, I see that the latest is 2.9.4 (the one I've installed. Do you mean beta versions ?

Frank

[dan (ddp)]

On Thu, Jun 28, 2018 at 9:13 AM, FSoyer <frank...@gmail.com> wrote:
> Well, sorry if I have trouble understanding the process, but I found only
> "<parent>" tag in decoder.xml, with decoder parent names (not ids), not in
> rules files. So when I search child rules for id 2900 with parent tag, I
> find nothing.
>

Sorry again, my fault. I meant '<if_sid>2900</if_sid>'. I'm trying to
do too many things at once.

> Another thing about versions, what do you mean by "master" ? When I look at
> https://github.com/ossec/ossec-hids/releases, I see that the latest is 2.9.4
> (the one I've installed. Do you mean beta versions ?
>

I mean the git master, or head. I usually use the latest code (or a
branch of it if I'm working on something) instead of relases.
So if you do a checkout of https://github.com/ossec/ossec-hids.git,
you'll get the master branch.
If you wanted what ended up in the 2.9.4 release, you'd probably have
to do something like `git checkout 2.9.4` inside of your checked out
repository.

[FSoyer]

Le jeudi 28 juin 2018 16:02:08 UTC+2, dan (ddpbsd) a écrit :

Sorry again, my fault. I meant '<if_sid>2900</if_sid>'. I'm trying to
do too many things at once.

 No problem : me too ^^

I mean the git master, or head. I usually use the latest code (or a
branch of it if I'm working on something) instead of relases.
So if you do a checkout of https://github.com/ossec/ossec-hids.git,
you'll get the master branch.
If you wanted what ended up in the 2.9.4 release, you'd probably have
to do something like `git checkout 2.9.4` inside of your checked out
repository.

OK ! Now I see. Reading the decoder.xml on github, I see this :

<rule id="2900" level="0">
	<decoded_as>windows-date-format</decoded_as>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d startup |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d status |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d remove |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d configure |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d install |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d purge |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d trigproc |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d conffile |</regex>
	<regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d upgrade </regex>
	<description>Dpkg (Debian Package) log.</description>
	</rule>

 when mine was just :

  <rule id="2900" level="0">

    <decoded_as>windows-date-format</decoded_as>

    <regex>^\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d \w+ </regex>

    <description>Dpkg (Debian Package) log.</description>

  </rule>

Not a surprise that it decoded all messages as id 2900 !

Replacing just this block solves the problem.

I think that I'll first replace all the decoder.xml file by the one on github, to avoid reinstalling. If it don't work I'll do a complete checkout.

Thanks !

[FSoyer]

Oops ! Me too as I said :)

replace "decoders.xml" by "syslog_rules.xml" !