Multi-line decoder for cisco ironport logs

[james morreau]

Hello,

I’m collecting cisco ironport logs from syslog, so all the logs are separated in one-line format. I found out that decoders have support to pcre syntax and with that I can create a decoder that support multiples lines and now I don’t have the knowledge on how to start one.

My objective is to create a decoder that can match all the lines in this example log bellow and create a single event, so I can parse the others camps later.

I noticed that all the logs start with this pattern:

Feb 23 21:00:00 mail_logs: Info: Start MID 15087034 ICID 25529386

​

 And ends with this one:

Feb 23 21:00:23 mail_logs: Info: Message finished MID 15087034 done

​

But i can't find out how to use that to create a decoder that can match all the multiple lines.

Example log:

Feb 23 21:00:00 mail_logs: Info: Start MID 15087034 ICID 25529386
Feb 23 21:00:00 mail_logs: Info: MID 15087034 ICID 25529386 From: <nor...@email.teams.microsoft.com>
Feb 23 21:00:00 mail_logs: Info: MID 15087034 ICID 25529386 RID 0 To: <f...@mail.com>
Feb 23 21:00:00 mail_logs: Info: MID 15087034 SPF: mailfrom identity nor...@email.teams.microsoft.com Pass (v=spf1)
Feb 23 21:00:00 mail_logs: Info: MID 15087034 DKIM: pass signature verified (d=email.teams.microsoft.com s=selector1 i=@email.teams.microsoft.com)
Feb 23 21:00:00 mail_logs: Info: MID 15087034 DMARC: Message from domain email.teams.microsoft.com, DMARC pass (SPF aligned True, DKIM aligned True)
Feb 23 21:00:00 mail_logs: Info: MID 15087034 Message-ID '<c9ede-32-457f-236.e-12.protection.outlook.com>'
Feb 23 21:00:00 mail_logs: Info: MID 15087034 DMARC: Verification passed
Feb 23 21:00:00 mail_logs: Info: MID 15087034 Subject "=?f-8?B?J5W52aW9YSBtdlbQ==?="
Feb 23 21:00:00 mail_logs: Info: MID 15087034 SDR: Domains for which SDR is requested: reverse DNS host: mail-bn8na.outbound.protection.outlook.com, helo: NA11-B-.outbound.protection.outlook.com, env-from: email.teams.microsoft.com, header-from: email.teams.microsoft.com, reply-to: Not Present
Feb 23 21:00:01 mail_logs: Info: MID 15087034 SDR: Consolidated Sender Reputation: Neutral, Threat Category: N/A. Youngest Domain Age: 27 years 11 months 1 day for domain: mail-bn8nam11on2073.outbound.protection.outlook.com
Feb 23 21:00:01 mail_logs: Info: MID 15087034 SDR: Tracker Header : +MTP565w0JszgXb6ryJRwetmrK6Oj9utMWg+/ifJz8v7B6YQWc5uXFEDilP1vm9DdxSxkItjv+AiT8zR5/HXKqltmhmh6e2JWobc4+k5E1piF13
Feb 23 21:00:01 mail_logs: Info: MID 15087034 ready 72274 bytes from <nor...@email.teams.microsoft.com>
Feb 23 21:00:01 mail_logs: Info: MID 15087034 matched all recipients for per-recipient policy DEFAULT in the inbound table
Feb 23 21:00:01 mail_logs: Info: MID 15087034 interim verdict using engine: CASE spam negative
Feb 23 21:00:01 mail_logs: Info: MID 15087034 using engine: CASE spam negative
Feb 23 21:00:01 mail_logs: Info: MID 15087034 interim AV verdict using McAfee CLEAN
Feb 23 21:00:01 mail_logs: Info: MID 15087034 interim AV verdict using Sophos CLEAN
Feb 23 21:00:01 mail_logs: Info: MID 15087034 antivirus negative
Feb 23 21:00:02 mail_logs: Info: MID 15087034 AMP file reputation verdict : UNKNOWN
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://schema.org/SignedAdaptiveCard has reputation 4.9 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://schema.org/extensions has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://go.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://www.w3.org/1999/xhtml has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/4 has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 Outbreak Filters: verdict negative
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 Custom Log Entry: <===> URL FOUND <===>
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/0 has reputation 9.2 matched Condition: URL Reputation Rule
Feb 23 21:00:02 mail_logs: Info: MID 15087034 queued for delivery
Feb 23 21:00:02 mail_logs: Info: Delivery start DCID 6108799 MID 15087034 to RID [0]
Feb 23 21:00:23 mail_logs: Info: Message done DCID 6108799 MID 15087034 to RID [0] [('from', '"=?=f=W4gQnJpdG8gRnJ="\r\n <nor...@email.teams.microsoft.com>'), ('to', 'f...@mail.com')]
Feb 23 21:00:23 mail_logs: Info: MID 15087034 RID [0] Response '2.6.0 <c9ee-68f-be2FT036.eop-nam12.ption.outlook.com> [InternalId=8] Queued mail for delivery'
Feb 23 21:00:23 mail_logs: Info: Message finished MID 15087034 done

​

​

[Selu López]

Hello James,

Looking at these logs, I wonder if for this use case it is really necessary to use a multi-line decoder. Each event has its own timestamp, source, etc. This is a difference compared to logs like Windows, where each line individually does not make sense (as you can see below) and therefore multi-line decoder is used.
```
<Event name="RulesFileInfo" time="07/13/2021 22:52:02" utc="07/14/2021 05:52:02">
  <EventProperty name="RulesFilePath" />
  <EventProperty name="RulesFileHash" />
  <EventProperty name="TotalRulesCount">1</EventProperty>
  <EventProperty name="ImplicitRulesCount">1</EventProperty>
</Event>
```

Check if it might make sense to make rules for certain logs individually. However, if you need to bundle it all together and treat it as a single event, the following localfile block should work for you:
```
  <localfile>
    <location>/var/log/mail.log</location>
    <log_format>multi-line-regex</log_format>
    <multiline_regex match="end" timeout="100" replace="wspace">\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}\smail_logs: Info: Message finished MID \d* done</multiline_regex>
  </localfile>
```

That will cause the logs not to be considered a complete event until the last line is read (Feb 23 21:00:23 mail_logs: Info: Message finished MID 15087034 done). Remember to adjust some parameters like timeout (which sets the max waiting time in seconds to receive a new line) so it suits your logs.

After adding that to your ossec.conf, you should receive an event like the one below the next time those logs are generated. From there you can create custom decoders and rules.
```
2022 Feb 25 09:25:08 wazuh-master->/var/log/mail.log Feb 25 09:25:08 wazuh-master mail_logs:Feb 23 21:00:00 mail_logs: Info: Start MID 15087034 ICID 25529386Feb 23 21:00:00 mail_logs: Info: MID 15087034 ICID 25529386 From: <nor...@email.teams.microsoft.com>Feb 23 21:00:00 mail_logs: Info: MID 15087034 ICID 25529386 RID 0 To: <f...@mail.com>Feb 23 21:00:00 mail_logs: Info: MID 15087034 SPF: mailfrom identity nor...@email.teams.microsoft.com Pass (v=spf1)Feb 23 21:00:00 mail_logs: Info: MID 15087034 DKIM: pass signature verified (d=email.teams.microsoft.com s=selector1 i=@email.teams.microsoft.com)Feb 23 21:00:00 mail_logs: Info: MID 15087034 DMARC: Message from domain email.teams.microsoft.com, DMARC pass (SPF aligned True, DKIM aligned True)Feb 23 21:00:00 mail_logs: Info: MID 15087034 Message-ID '<c9ede-32-457f-236.e-12.protection.outlook.com>'Feb 23 21:00:00 mail_logs: Info: MID 15087034 DMARC: Verification passedFeb 23 21:00:00 mail_logs: Info: MID 15087034 Subject "=?f-8?B?J5W52aW9YSBtdlbQ==?="Feb 23 21:00:00 mail_logs: Info: MID 15087034 SDR: Domains for which SDR is requested: reverse DNS host: mail-bn8na.outbound.protection.outlook.com, helo: NA11-B-.outbound.protection.outlook.com, env-from: email.teams.microsoft.com, header-from: email.teams.microsoft.com, reply-to: Not PresentFeb 23 21:00:01 mail_logs: Info: MID 15087034 SDR: Consolidated Sender Reputation: Neutral, Threat Category: N/A. Youngest Domain Age: 27 years 11 months 1 day for domain: mail-bn8nam11on2073.outbound.protection.outlook.comFeb 23 21:00:01 mail_logs: Info: MID 15087034 SDR: Tracker Header : +MTP565w0JszgXb6ryJRwetmrK6Oj9utMWg+/ifJz8v7B6YQWc5uXFEDilP1vm9DdxSxkItjv+AiT8zR5/HXKqltmhmh6e2JWobc4+k5E1piF13Feb 23 21:00:01 mail_logs: Info: MID 15087034 ready 72274 bytes from <nor...@email.teams.microsoft.com>Feb 23 21:00:01 mail_logs: Info: MID 15087034 matched all recipients for per-recipient policy DEFAULT in the inbound tableFeb 23 21:00:01 mail_logs: Info: MID 15087034 interim verdict using engine: CASE spam negativeFeb 23 21:00:01 mail_logs: Info: MID 15087034 using engine: CASE spam negativeFeb 23 21:00:01 mail_logs: Info: MID 15087034 interim AV verdict using McAfee CLEANFeb 23 21:00:01 mail_logs: Info: MID 15087034 interim AV verdict using Sophos CLEANFeb 23 21:00:01 mail_logs: Info: MID 15087034 antivirus negativeFeb 23 21:00:02 mail_logs: Info: MID 15087034 AMP file reputation verdict : UNKNOWNFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://schema.org/SignedAdaptiveCard has reputation 4.9 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://schema.org/extensions has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://go.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL http://www.w3.org/1999/xhtml has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/4 has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 Outbreak Filters: verdict negativeFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 Custom Log Entry: <===> URL FOUND <===>Feb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/ has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 URL https://urlshortener.teams.microsoft.com/0 has reputation 9.2 matched Condition: URL Reputation RuleFeb 23 21:00:02 mail_logs: Info: MID 15087034 queued for deliveryFeb 23 21:00:02 mail_logs: Info: Delivery start DCID 6108799 MID 15087034 to RID [0]Feb 23 21:00:23 mail_logs: Info: Message done DCID 6108799 MID 15087034 to RID [0] [('from', '"=?=f=W4gQnJpdG8gRnJ=" <nor...@email.teams.microsoft.com>'), ('to', 'f...@mail.com')]Feb 23 21:00:23 mail_logs: Info: MID 15087034 RID [0] Response '2.6.0 <c9ee-68f-be2FT036.eop-nam12.ption.outlook.com> [InternalId=8] Queued mail for delivery'Feb 23 21:00:23 mail_logs: Info: Message finished MID 15087034 done
```

Regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0eadf173-999b-4df7-be64-00f07f47f270n%40googlegroups.com.

[james morreau]

Hello Jose, I think I understand now, thank you very much for your help and attention.

Best Regards.