Problem - Wazuh downloads AWS WAF logs everyday starting from same date.

[Bhavesh Bhanushali]

Hi,

My apology for writing longer email. Intent is to explain problem properly.

We are working on POC to use Wazuh as a SIEM for AWS services. We have started with AWS WAF v2 and could see alert/event information in the wazuh application.

We are happy with the information displayed but we have noticed one issue when we configure wazuh to fetch logs after specific date. 

As per wazuh documentation, the wazuh is configured to read logs from S3 bucket where structure of the S3 path is "s3-bucketname/year/numeric-month/day/hour/filename" e.g "waf-logs-bucket/2021/04/14/01/waf-logs-bucket-1-2021-04-14-01-00-55-dc8b76c5-ba59-45f4-a529-968c7da09c0b".

Here is the config in ossec.conf

 <wodle name="aws-s3">

  <disabled>no</disabled>

  <interval>120m</interval>

  <run_on_start>yes</run_on_start>

  <skip_on_error>yes</skip_on_error>

  <bucket type="waf">

    <name>waf-logs-bucket</name>

    <path></path>

        <only_logs_after>2021-APR-14</only_logs_after>

    <aws_profile>my-prod-profile</aws_profile>

  </bucket>

</wodle>

Problem - Wazuh downloads logs everyday starting from same date of 14 April 2021. So logs of 22 April also contains events of 14 April.

We request someone to help /share some direction so that we can fix it and use wazuh for other AWS services.

Thanks in Advance!

Regards,

Bhavesh Bhanushali

[Cesar Moreno]

Hello  Bhavesh,

Thank you for posting in the Wazuh community.

If you remove the only_logs_after configuration, do you get the logs as expected with logs after the date you've set (2021-APR-14)?

As you can find in this Wafuh for AWS WAF guide, in step 7 (attached) the destination to send to an S3 Bucket, Select the S3 Buchet created previously and put it in the <name>, and finally add the path you've chosen as prefix then put it between the 

<path></path> tags.

 <wodle name="aws-s3">

  <disabled>no</disabled>

  <interval>120m</interval>

  <run_on_start>yes</run_on_start>

  <skip_on_error>yes</skip_on_error>

  <bucket type="waf">

    <name>waf-logs-bucket</name> <!-- PUT HERE THE S3 BUCKET CHOSEN IN STEP 7 --> 

    <path></path> <!-- PUT HERE THE PREFIX CHOSEN IN STEP 7 -->

        <only_logs_after>2021-APR-14</only_logs_after>

    <aws_profile>my-prod-profile</aws_profile>

  </bucket>

</wodle>

If you are getting logs as expected, but you want to avoid downloading the old logs from the bucket, you can consider this AWS link to manage the s3 bucket lifecycle to change the logs retention policy.

The wazuh app will take all those logs and merge the logs that don't exist in the ElasticSearch.

On the other hand, in the following guide, you can find the different examples to integrate the different AWS services and buckets supported by Wazuh:

• https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-s3.html#example-of-configuration

In the following blog, you have a complete example to configure Wazuh to ingest logs from AWS CloudTrail S3 Bucket:

• https://wazuh.com/blog/integrating-aws-cloudtrail-in-wazuh/
  

Additionally, to get the debug information in the /var/ossec/etc/local_internal_options.conf file, you have to add this to get the detailed level, after troubleshooting the connections to AWS buckets, you should remove it or change to 0: wazuh_modules.debug=2

You'll be able to find those logs on /var/ossec/logs/ossec.log file. If you get something like follows, you can execute the script /var/ossec/wodles/aws/aws-s3 manually and add --debug 2 to find out what is happening on the execution:

• 2021/04/22 23:34:08 wazuh-modulesd:aws-s3[21386] wm_aws.c:382 at wm_aws_run_s3(): DEBUG: Launching S3 Command: /var/ossec/wodles/aws/aws-s3 --bucket waf-logs-bucket --access_key AKSSSSSEEECCCRREEETT2K --secret_key  AKSSSSSEEECCCRREEETTKKKEEEEYYY2K  --only_logs_after 2021-APR-14 --type waf --debug 2 --skip_on_error  

Hope this answers your question. Anything you need, please let me know, I'll follow up on this to help you.

Kind regards,

Cesar Moreno.

[Bhavesh Bhanushali]

Hi Cesar,

Thanks for your detailed explanation and quick response.

I will surely look into some of the additional reference points and suggestion provided by you.

We have also noticed that logs download from AWS WAF are of only 2 days (14 April 2021 and 15 April 2021) only and it starts with 14 April everyday but hour is different. 

If we want to delete all alerts and events and retry again than could you please suggest steps or share path of the documentation.

Thanks in Advance!

With Regards

Bhavesh R Bhanushali 

[Cesar Moreno]

Hello  Bhavesh,

Are there AWS WAF logs in the cloud available for download after the 15th, but the AWS Wodle is unable to get them?

I believe that it's not necessary to remove those old logs. Anyway, I'd verify if the log time is the correct one and matches the information in the AWS WAF.

To achieve that, you can share with me if you want, some logs with obfuscated info on AWS WAF and some logs from archives.json if you have enabled the logall_json in the ossec.conf. I'd like to verify the decoders and rules that we have created for those updated AWS Waf logs.

Hope this helps, I look forward to your reply if you need to verify some logs that are not being decoded and alerted on the Wazuh app.

Kind regards,

Cesar Moreno

[Bhavesh Bhanushali]

Hello Cesar,

Thanks for looking into my issue.

Here are the attachments that you are looking for. We have not enabled logall_json so no archives available. 

The filename "aws-waf-logs-structure*" shows S3 folders available in 14 Date folder. These folders also contains objects. Filename "aws-waf-logs-structure-2-22-date-exploded.png" shows objects present in 22 April folder. These folders stores WAF logs in AWS S3. This attachment are added to confirm that logs exists for all dates.

The other files attached are screenshots of security events in Wazuh application.

• security-events-1.png - This file shows count of alerts and bar chart to indicates dates on which alerts were downloaded from S3. Notice the first 2 records of date "April 22, 2021 @ 12:14:09.853" and "April 22, 2021 @ 12:14:09.850"
• security-events-2.png - This file shows 1st record " April 22, 2021 @ 12:14:09.853" exploded JSON tab. Observe the log_file name "2021/04/15/09/aws-waf-logs-#####-####-1-2021-04-15-09-51-24-1e51c2d3-9ac6-4c12-a1a0-4f7aac39b54c"
  
• security-events-3.png - This file shows 2nd record " April 22, 2021 @ 12:14:09.850" exploded JSON tab. Observe the log_file name "2021/04/15/09/aws-waf-logs-#####-####-1-2021-04-15-09-51-24-1e51c2d3-9ac6-4c12-a1a0-4f7aac39b54c" This is the same file in above bullet point.
  

Note that 22nd April records are showing logs of 15th April 2021. This is the problem we are trying to explain. Ideally, 22nd April should download logs of 22nd April from S3 buckets.

We have added 2 log files (16 April and 22 April) as samples.

Hope this helps.

Thanks in Advance

Bhavesh R Bhanushali

[Cesar Moreno]

Hello  Bhavesh,

Thank you very much for this detailed information, hope you are doing very well.

Those logs for 14th and 15th, are not being duplicated in the cloud with different timestamp, are them? Wazuh app will show different alerts for April 22, 2021 @ 12:14:09.850 and April 22, 2021 @ 12:14:09.853 since these are different.

If the WAF is creating new log files in the AWS S3  bucket on the , the Wazuh app will recognize it as an unparsed log and will parse it since it doesn't exist with the script execution timestamp, and it's working as expected. But, if you are getting everyday same logs with the same datetime in Wazuh, but in the cloud you have only one log with the same datetime, I'll have to double-check the script. Anyway, if this is the case, and with the apropiated permissions on the bucket, you can use in the script the option to remove (<remove_from_bucket>yes</remove_from_bucket>) the logs already parsed if you wont use them anymore.

Are you currently using the WAF monitoring with Wazuh app and is it working as expected but getting old logs being reparsed again only when you add the option only_logs_after? I'd recommend you to remove the skip_on_error since it's yes by default. I'd also advise you to check the clod WAF logs in the bucket and confirm those logs are not being re-writed on the bucket currently.

Hope this helps. Any questions, please let me know.

Kind regards,

Cesar Moreno.

[Bhavesh Bhanushali]

Hello Cesar,

Thanks for your prompt response.

We have confirmed and verified that in AWS S3, WAF logs are stored daily with new date filename, the contents of the logs are for the date on which new file is created. for example if today date is 27 April 2021 than folder path created on AWS S3 is "bucketName/2021/04/27/hr/aws-waf-logs-proj-1-2021-04-27-06-01-13-21ac7d0c-3f60-4c36-8833-9f271eddfa5f" This file contains entries with timestamp as "1619503258514" which corresponds to 27 April 2021. We have confirmed that logs contains new entries as per the current date.

Due to policy requirement, we cannot remove the logs after SIEM reads it. We will try with the change related to error tag.

Just to add we have disabled logs aggregation in wazuh after 22 April 2021 till the time we understand the behaviour.

Hope this helps!

Thanks,

Bhavesh R Bhanushali

[Cesar Moreno]

Hello  Bhavesh,

Hope you are well today.

This AWS behavior is expected and Wazuh will collect those logs depending on the schedule and will parse only the logs that still don't exist in Elasticsearch with the timestamp of the script execution, that data shouldn't be duplicated. Anyway, I want to confirm also, you have this configured only in one of the managers if you have a cluster in order to avoid duplicated data due to duplicated connections to download the logs.

Any questions, please let me know, I'm glad to help you!

Kind regards,

Cesar Moreno.