AWS SES Decoder and Rule

[Defender]

Hey team! Thank you for your hard work!
I configure getting logs from AWS mail server - SES.
Please help me to build a decoder and a rule. I am a newbie in Wazuh. Tried to understand the documentation, but it doesn't work.

Best Regards.

[Defender]

It's beginning to work. But Order not extract. What am I doing wrong?

<decoder name="SES_json">
    <prematch>"Message" : "</prematch>
    <plugin_decoder offset="after_prematch">JSON_Decoder</plugin_decoder>
</decoder>

<decoder name="SES_json_child">
    <parent>SES_json</parent>
    <regex>"Message" : "{\\"notificationType\\":\\"Delivery\\",\\"mail\\":{\\"timestamp\\":\\"(\d+\d+\d+\d+-\d+-\d+.\d+:\d+:\d+.....)</regex>
    <order>date</order>
</decoder>

<group name="SES">
<rule id="100302" level="5">
   <decoded_as>SES_json</decoded_as>
   <description>SES Delivery</description>
</rule>
</group>

вторник, 30 августа 2022 г. в 20:55:29 UTC+3, Defender:

[Jonathan Martín Valera]

Hi,

I will try to help you with your use case :D

I have seen that you have shared a file with an example log, but I have doubts about if the format is specifically that, because in your test it says that it decodes as JSON, but notice that the log you have shared has like 3 parts: Date, Subject and Text; and each of them is a JSON but the general log is not. Is the log multi-line, or is the original log written on a single line (no line breaks)? The general log begins with { and ends with }?

Can you specify more information about this (answer my questions and anything else you can provide for clarification) and pass an exact example of the log you want to parse? It is necessary to create the decoder correctly.

Best regards.

​

[Defender]

Hi Jonathan. Thanks for your response!
Log is multi-line.
Yes, The general log begins with { and ends with }.
The logs come to my email and then I use a python script to take them to a file. Like the one I attached.
In a private message I sent you a screenshot of how the log comes to wazuh (tail -f /var/ossec/logs/archives/archives.log)

среда, 31 августа 2022 г. в 11:54:32 UTC+3, Jonathan Martín Valera:

[Jonathan Martín Valera]

Hi,

Note that it is sending you an event for each line of your log file, that is to say, instead of generating an event with all the JSON, it is generating an event for each line (\n). This happens because you have not configured to process the log in multiline mode (see https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#multiline-regex).

I have been testing with your log example, and the configuration that I would apply to process your log in multiline format is the following:

<localfile>
  <log_format>multi-line-regex</log_format>
  <location>/home/vagrant/test.log</location>
  <multiline_regex replace="wspace" match="end">Text: \{\.*\}</multiline_regex>
</localfile>

Note: Update the location field with the path to your log file.

Regarding the decoder and rule, I share with you an example that decodes the timestamp field of mail as date. Add the following decoder to /var/ossec/etc/decoders/local_decoder.xml.

<decoder name="SES_json">
   <prematch>Date:\.*Subject:{\.*}\.*Text:{\.*}</prematch>
   <regex>"Message"\s*:\s*"{\.*mail\\":{\\"timestamp\\":\\"(\S+)\\"</regex>
  <order>date</order>
</decoder>

Regarding the rule, I see that you do not use the decoded field at all, but you want it to generate an alert when such an event occurs and is decoded with the SES_json decoder. So, add the following in /var/ossec/etc/rules/local_rules.xml in case you don’t have it there:

<group name="SES">
  <rule id="100302" level="5">
    <decoded_as>SES_json</decoded_as>
     <description>SES Delivery</description>
  </rule>
</group>

From now on, all of these events will generate alerts.

Try it and let us know the results.

Best regards.

​

[Defender]

Hi Jonathan. Thanks for your response!

I did everything according to your instructions.
Then I paste the log I sent to /var/ossec/bin# ./wazuh-logtest
The result is No decoder matched.

среда, 31 августа 2022 г. в 13:33:18 UTC+3, Jonathan Martín Valera:

[Jonathan Martín Valera]

Hi,

It is normal that it does not work in the wazuh-logtest tool because your log is multiline. Note that we have applied the multi-line configuration for log analysis in the file itself. If what you want to do is to test the log in the wazuh-logtest tool, then you have to convert your multiline log to single-line format. For this you can replace all the line break characters (\n) by blanks (\s). Once this is done, if you use the decoders and rules I mentioned before, you should get the following output:

# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.3.6
Type one log per line

Date:2022-08-31 13:30:48+00:00 Subject:{   "Type" : "Notification",   "MessageId" : "9b2ace03-b65d-5a0e-83ab-958ba51518e6",   "TopicArn" : "arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery",   "Message" : "{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2022-08-30T13:30:44.244Z\",\"source\":\"in...@mydomain.com\",\"sourceArn\":\"arn:aws:ses:us-west-2:111111111111:identity/mydomain.com\",\"sourceIp\":\"11.11.11.11\",\"callerIdentity\":\"ses-smtp-user.20220504-132337\",\"sendingAccountId\":\"111111111111\",\"messageId\":\"01010182eef27894-c114020d-a176-48e1-bf3c-ad4b61fc2627-000000\",\"destination\":[\"user.f...@domainuser.ru\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"Received\",\"value\":\"from pc44 (pc44.domain.co [11.11.11.11]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-d-9BJVL0VRI) id i7X2jzm5D3MjIgJY2CjU for user.f...@domainuser.ru; Tue, 30 Aug 2022 13:30:44 +0000 (UTC)\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"From\",\"value\":\"\\\"Domain Notifications\\\" <in...@mydomain.com>\"},{\"name\":\"To\",\"value\":\"\\\"User Feemale\\\" <user.f...@domainuser.ru>\"},{\"name\":\"Date\",\"value\":\"30 Aug 2022 13:30:44 +0000\"},{\"name\":\"Subject\",\"value\":\"Domain: Translation status has changed: ct-mobile-os-comparison\"},{\"name\":\"Content-Type\",\"value\":\"text/html; charset=utf-8\"},{\"name\":\"Content-Transfer-Encoding\",\"value\":\"base64\"}],\"commonHeaders\":{\"from\":[\"Domain Notifications <in...@mydomain.com>\"],\"date\":\"30 Aug 2022 13:30:44 +0000\",\"to\":[\"User Feemale <user.f...@domainuser.ru>\"],\"subject\":\"Domain: Translation status has changed: ct-mobile-os-comparison\"}},\"delivery\":{\"timestamp\":\"2022-08-30T13:30:48.630Z\",\"processingTimeMillis\":4386,\"recipients\":[\"user.f...@domainuser.ru\"],\"smtpResponse\":\"250 2.6.0 <01010182eef27894-c114020d-a176...@us-west-2.amazonses.com> [InternalId=114911850014996, Hostname=pcnameuser.domain.prod.outlook.com] 19027 bytes in 0.110, 168.793 KB/sec Queued mail for delivery\",\"remoteMtaIp\":\"104.47.57.138\",\"reportingMTA\":\"a27-30.smtp-out.us-west-2.amazonses.com\"}}",   "Timestamp" : "2022-08-30T13:30:48.701Z",   "SignatureVersion" : "1",   "Signature" : "rgnFUW9w6dsdU3FPD5YDA7f1fibusZ+KO3LH0I/l6a0//tkmPoWlohJuCqMPMYPkCREuP7Yuyz2agDyNbBa+PE2J6NTOHVRzR7IIuEgumN63TKwCUJbs6FfMDoGwm1uBpt6Tbkyg8k41VPJ8ddSrBYijwKacNgknxAZd+NSbGQwAxmu/wMBW2yA+5I1bz9upuj/LxMVVSPLdhw/QxgH5lmdRR+XAXyJJlPvGtFnSj6FXJC9VmBKNw8aSIqKvEfapHQjNY3KHEzliYe7QxRix6c6k5+zbP5G50X6i7mXQ92sUlL0YJtSZr0n1wI96qY9th6KyCTWUp5Shidri/txDog==",   "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",   "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery:97c76d01-0b9f-4fa4-a437-f513b1960288" } Text:{   "Type" : "Notification",   "MessageId" : "9b2ace03-b65d-5a0e-83ab-958ba51518e6",   "TopicArn" : "arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery",   "Message" : "{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2022-08-30T13:30:44.244Z\",\"source\":\"in...@mydomain.com\",\"sourceArn\":\"arn:aws:ses:us-west-2:111111111111:identity/mydomain.com\",\"sourceIp\":\"11.11.11.11\",\"callerIdentity\":\"ses-smtp-user.20220504-132337\",\"sendingAccountId\":\"111111111111\",\"messageId\":\"01010182eef27894-c114020d-a176-48e1-bf3c-ad4b61fc2627-000000\",\"destination\":[\"user.f...@domainuser.ru\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"Received\",\"value\":\"from pc44 (pc44.domain.co [11.11.11.11]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-d-9BJVL0VRI) id i7X2jzm5D3MjIgJY2CjU for user.f...@domainuser.ru; Tue, 30 Aug 2022 13:30:44 +0000 (UTC)\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"From\",\"value\":\"\\\"Domain Notifications\\\" <in...@mydomain.com>\"},{\"name\":\"To\",\"value\":\"\\\"User Feemale\\\" <user.f...@domainuser.ru>\"},{\"name\":\"Date\",\"value\":\"30 Aug 2022 13:30:44 +0000\"},{\"name\":\"Subject\",\"value\":\"Domain: Translation status has changed: ct-mobile-os-comparison\"},{\"name\":\"Content-Type\",\"value\":\"text/html; charset=utf-8\"},{\"name\":\"Content-Transfer-Encoding\",\"value\":\"base64\"}],\"commonHeaders\":{\"from\":[\"Domain Notifications <in...@mydomain.com>\"],\"date\":\"30 Aug 2022 13:30:44 +0000\",\"to\":[\"User Feemale <user.f...@domainuser.ru>\"],\"subject\":\"Domain: Translation status has changed: ct-mobile-os-comparison\"}},\"delivery\":{\"timestamp\":\"2022-08-30T13:30:48.630Z\",\"processingTimeMillis\":4386,\"recipients\":[\"user.f...@domainuser.ru\"],\"smtpResponse\":\"250 2.6.0 <01010182eef27894-c114020d-a176...@us-west-2.amazonses.com> [InternalId=114911850014996, Hostname=pcnameuser.domain.prod.outlook.com] 19027 bytes in 0.110, 168.793 KB/sec Queued mail for delivery\",\"remoteMtaIp\":\"104.47.57.138\",\"reportingMTA\":\"a27-30.smtp-out.us-west-2.amazonses.com\"}}",   "Timestamp" : "2022-08-30T13:30:48.701Z",   "SignatureVersion" : "1",   "Signature" : "rgnFUW9w6dsdU3FPD5YDA7f1fibusZ+KO3LH0I/l6a0//tkmPoWlohJuCqMPMYPkCREuP7Yuyz2agDyNbBa+PE2J6NTOHVRzR7IIuEgumN63TKwCUJbs6FfMDoGwm1uBpt6Tbkyg8k41VPJ8ddSrBYijwKacNgknxAZd+NSbGQwAxmu/wMBW2yA+5I1bz9upuj/LxMVVSPLdhw/QxgH5lmdRR+XAXyJJlPvGtFnSj6FXJC9VmBKNw8aSIqKvEfapHQjNY3KHEzliYe7QxRix6c6k5+zbP5G50X6i7mXQ92sUlL0YJtSZr0n1wI96qY9th6KyCTWUp5Shidri/txDog==",   "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",   "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery:97c76d01-0b9f-4fa4-a437-f513b1960288" }

**Phase 1: Completed pre-decoding.
    full event: 'Date:2022-08-31 13:30:48+00:00 Subject:{   "Type" : "Notification",   "MessageId" : "9b2ace03-b65d-5a0e-83ab-958ba51518e6",   "TopicArn" : "arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery",   "Message" : "{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2022-08-30T13:30:44.244Z\",\"source\":\"in...@mydomain.com\",\"sourceArn\":\"arn:aws:ses:us-west-2:111111111111:identity/mydomain.com\",\"sourceIp\":\"11.11.11.11\",\"callerIdentity\":\"ses-smtp-user.20220504-132337\",\"sendingAccountId\":\"111111111111\",\"messageId\":\"01010182eef27894-c114020d-a176-48e1-bf3c-ad4b61fc2627-000000\",\"destination\":[\"user.f...@domainuser.ru\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"Received\",\"value\":\"from pc44 (pc44.domain.co [11.11.11.11]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-d-9BJVL0VRI) id i7X2jzm5D3MjIgJY2CjU for user.f...@domainuser.ru; Tue, 30 Aug 2022 13:30:44 +0000 (UTC)\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"From\",\"value\":\"\\\"Domain Notifications\\\" <in...@mydomain.com>\"},{\"name\":\"To\",\"value\":\"\\\"User Feemale\\\" <user.f...@domainuser.ru>\"},{\"name\":\"Date\",\"value\":\"30 Aug 2022 13:30:44 +0000\"},{\"name\":\"Subject\",\"value\":\"Domain: Translation status has changed: ct-mobile-os-comparison\"},{\"name\":\"Content-Type\",\"value\":\"text/html; charset=utf-8\"},{\"name\":\"Content-Transfer-Encoding\",\"value\":\"base64\"}],\"commonHeaders\":{\"from\":[\"Domain Notifications <in...@mydomain.com>\"],\"date\":\"30 Aug 2022 13:30:44 +0000\",\"to\":[\"User Feemale <user.f...@domainuser.ru>\"],\"subject\":\"Domain: Translation status has changed: ct-mobile-os-comparison\"}},\"delivery\":{\"timestamp\":\"2022-08-30T13:30:48.630Z\",\"processingTimeMillis\":4386,\"recipients\":[\"user.f...@domainuser.ru\"],\"smtpResponse\":\"250 2.6.0 <01010182eef27894-c114020d-a176...@us-west-2.amazonses.com> [InternalId=114911850014996, Hostname=pcnameuser.domain.prod.outlook.com] 19027 bytes in 0.110, 168.793 KB/sec Queued mail for delivery\",\"remoteMtaIp\":\"104.47.57.138\",\"reportingMTA\":\"a27-30.smtp-out.us-west-2.amazonses.com\"}}",   "Timestamp" : "2022-08-30T13:30:48.701Z",   "SignatureVersion" : "1",   "Signature" : "rgnFUW9w6dsdU3FPD5YDA7f1fibusZ+KO3LH0I/l6a0//tkmPoWlohJuCqMPMYPkCREuP7Yuyz2agDyNbBa+PE2J6NTOHVRzR7IIuEgumN63TKwCUJbs6FfMDoGwm1uBpt6Tbkyg8k41VPJ8ddSrBYijwKacNgknxAZd+NSbGQwAxmu/wMBW2yA+5I1bz9upuj/LxMVVSPLdhw/QxgH5lmdRR+XAXyJJlPvGtFnSj6FXJC9VmBKNw8aSIqKvEfapHQjNY3KHEzliYe7QxRix6c6k5+zbP5G50X6i7mXQ92sUlL0YJtSZr0n1wI96qY9th6KyCTWUp5Shidri/txDog==",   "SigningCertURL" : "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem",   "UnsubscribeURL" : "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery:97c76d01-0b9f-4fa4-a437-f513b1960288" } Text:{   "Type" : "Notification",   "MessageId" : "9b2ace03-b65d-5a0e-83ab-958ba51518e6",   "TopicArn" : "arn:aws:sns:us-west-2:111111111111:Domain-com-Delivery",   "Message" : "{\"notificationType\":\"Delivery\",\"mail\":{\"timestamp\":\"2022-08-30T13:30:44.244Z\",\"source\":\"in...@mydomain.com\",\"sourceArn\":\"arn:aws:ses:us-west-2:111111111111:identity/mydomain.com\",\"sourceIp\":\"11.11.11.11\",\"callerIdentity\":\"ses-smtp-user.20220504-132337\",\"sendingAccountId\":\"111111111111\",\"messageId\":\"01010182eef27894-c114020d-a176-48e1-bf3c-ad4b61fc2627-000000\",\"destination\":[\"user.f...@domainuser.ru\"],\"headersTruncated\":false,\"headers\":[{\"name\":\"Received\",\"value\":\"from pc44 (pc44.domain.co [11.11.11.11]) by email-smtp.amazonaws.com with SMTP (SimpleEmailService-d-9BJVL0VRI) id i7X2jzm5D3MjIgJY2CjU for user.f...@domainuser.ru; Tue, 30 Aug 2022 13:30:44 +0000 (UTC)\"},{\"name\":\"MIME-Version\",\"value\":\"1.0\"},{\"name\":\"From\",\"value\":\"\\\"Domain Notifications\\\" <in...@mydomain.com>\"},{\"name\":\"To\",\"value\":\"\\\"User Feemale\\\" <user.f...@domainuser.ru>\"},{\"name\":\"Date\",\"value\":\"30 Aug 2022 13:3'

**Phase 2: Completed decoding.
    name: 'SES_json'
    date: '2022-08-30T13:30:44.244Z'

**Phase 3: Completed filtering (rules).
    id: '100302'
    level: '5'
    description: 'SES Delivery'
    groups: '['SES']'
    firedtimes: '1'
    mail: 'False'
**Alert to be generated.

​

[Defender]

Yes Jonathan, it really works. That's really fantastic. Thank you.

среда, 31 августа 2022 г. в 17:05:23 UTC+3, Jonathan Martín Valera: