Wazuh not finding GuardDuty logs in S3 bucket

[Jan Zioło]

Dears

I've been struggling for a while with integrating Wazuh 4.11.1 with AWS GuardDuty.

I have GuardDuty logging files into S3 bucket. All IAM permissions work fine and I already have logs from CloudTrail ingested correctly.

No GuardyDuty events appear in Wazuh however.

I added another bucket of GuardDuty type to s3 wodle like so

  <wodle name="aws-s3">
    <disabled>no</disabled>
    <interval>1m</interval>
    <run_on_start>yes</run_on_start>
    <skip_on_error>yes</skip_on_error>
    <bucket type="cloudtrail">
      <name>my-centralized-log-archive</name>
      <only_logs_after>2025-MAR-28</only_logs_after>
      <path_suffix>my_org_id</path_suffix>
    </bucket>
    <bucket type="guardduty">
      <name>my-centralized-log-archive</name>
      <only_logs_after>2025-MAR-28</only_logs_after>
    </bucket>
  </wodle>

There are some logs in .jsonl format inside the bucket, I also made sure with Ruleset test that Guard Duty events get parsed correctly. I tried different only_logs_after and even pointing directly do month and day folder in a bucket - but Wazuh does not seem to see any logs inside S3.

  /var/ossec/wodles/aws/aws-s3 -b my-centralized-log-archive -t guardduty -d2
DEBUG: +++ Debug mode on - Level: 2
DEBUG: Generating default configuration for retries: mode standard - max_attempts 10
DEBUG: +++ Working on REDACTED - eu-west-2
DEBUG: +++ Marker: AWSLogs/REDACTED/GuardDuty/eu-west-2/2025/03/31/380c6d62-da86-4e10-9751-cb1aea9ae18d.jsonl.gz
DEBUG: +++ No logs to process in bucket: REDACTED/eu-west-2
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on REDACTED - eu-west-2
DEBUG: +++ Marker: AWSLogs/REDACTED/GuardDuty/eu-west-2/2025/04/01
DEBUG: +++ No logs to process in bucket: REDACTED/eu-west-2
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on REDACTED - eu-west-2
DEBUG: +++ Marker: AWSLogs/REDACTED/GuardDuty/eu-west-2/2025/04/01
DEBUG: +++ No logs to process in bucket: REDACTED/eu-west-2
DEBUG: +++ DB Maintenance

Any idea what may be the reason here and how to proceed? I really went through lot of searching with no results. What am I missing?

The Software House spółka z ograniczoną odpowiedzialnością with registered office in Gliwice, ul. Dolnych Wałów 8, 44-100, entered into the register of entrepreneurs conducted by the District Court in Gliwice, 10th Commercial Division of the National Court Register under the KRS number 0000435101, NIP: 527-268-05-43, REGON: 146211123

The controller of personal data is The Software House sp. z o.o. with its registered seat in Gliwice ul. Dolnych Wałów 8. Personal data is processed for the purpose of handling correspondence with you. As we process your personal data, you have the right to request: access to your personal data, their rectification, erasure or restriction of processing, to object to the processing of your data, to lodge a complaint with the President of Personal Data Protection Office (Prezes Urzędu Ochrony Danych Osobowych), should you consider that the processing of your personal data violates the law. For more information on data processing, see our Privacy Policy. The rules for reporting violations of the law can be found in the Internal Reporting Procedure.

[José Luis Cosentino]

Hello,

It seems you already have the credentials configuration. And the proper access to the account. But Since Wazuh is saying that you don't have logs. Could be related with the bucket log path structure:
<WAZUH_AWS_BUCKET>/<prefix>/<year>/<month>/<day>/<hh>
Here is the full table of log paths:
https://documentation.wazuh.com/current/cloud-security/amazon/services/supported-services/index.html#:~:text=bucket-,guardduty,-%3CWAZUH_AWS_BUCKET%3E/%3Cprefix%3E/%3Cyear

- Verify if you have the proper permission to read logs in your bucket:  s3:GetObject, s3:ListBucket
https://documentation.wazuh.com/current/cloud-security/amazon/services/supported-services/guardduty.html#aws-configuration

- Run the aws-s3 wodle command manually with debugging enabled to confirm if files are being skipped or processed incorrectly:
/var/ossec/wodles/aws/aws-s3 -b <bucket_name> -t guardduty --debug 2

Let me know your outputs, please.

[Jan Zioło]

Hi Jose, thank you very much for addressing my issue.

I tried pointing different paths in ossec.conf, for no effect. Here is a summary of my configuration:

S3 bucket file structure:

AWSLogs/<account_id>/GuardDuty/<region>/YYYY/MM/DD/

ossec.conf file, s3 wodle:

  <wodle name="aws-s3">
    <disabled>no</disabled>
    <interval>1m</interval>
    <run_on_start>yes</run_on_start>
    <skip_on_error>yes</skip_on_error>
    <bucket type="cloudtrail">

      <name><my-bucket-name></name>
      <only_logs_after>2025-MAR-28</only_logs_after>
      <path_suffix><org_id></path_suffix>
    </bucket>
    <bucket type="guardduty">
      <name><my-bucket-name></name>
      <!--<path>AWSLogs/<account_id>/GuardDuty/</path>-->
      <only_logs_after>2025-MAR-28</only_logs_after>
      <regions>us-east-1,us-east-2,us-west-1,us-west-2,ap-east-1,ap-south-1,ap-northeast-3,ap-northeast-2,ap-southeast-1,ap-southeast-2,ap-northeast-1,ca-central-1,eu-central-1,eu-west-1,eu-west-2,eu-west-3,eu-north-1,sa-east-1</regions>
    </bucket>
  </wodle>

I tried with path AWSLogs/<account_id>/GuardDuty/, AWSLogs/*/GuardDuty/, and with no path at all. Also tried with regions and without them.

Here is Wazuh log output just after the manager restart:

tail /var/ossec/logs/ossec.log | grep aws
2025/04/02 10:23:25 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.
2025/04/02 10:23:25 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: <my-bucket-name>, Path suffix: <org_id>, Type: cloudtrail)
2025/04/02 10:23:26 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: <my-bucket-name>, Type: guardduty)
2025/04/02 10:23:28 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.
2025/04/02 10:24:25 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.
2025/04/02 10:24:25 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: <my-bucket-name>, Path suffix: <org_id>, Type: cloudtrail)
2025/04/02 10:24:26 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: <my-bucket-name>, Type: guardduty)
2025/04/02 10:24:27 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

And manual wodle run output:

/var/ossec/wodles/aws/aws-s3 --bucket <my-bucket-name> --type guardduty --debug 4
DEBUG: +++ Debug mode on - Level: 4

DEBUG: Generating default configuration for retries: mode standard - max_attempts 10

DEBUG: +++ Working on <account_id> - eu-west-2
DEBUG: +++ Marker: AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/04/02/db9ce5f5-b1a7-4646-ab7a-7da8a80ffb41.jsonl.gz
DEBUG: +++ No logs to process in bucket: <account_id>/eu-west-2
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on <other_account_id> - eu-west-2
DEBUG: +++ Marker: AWSLogs/<other_account_id>/GuardDuty/eu-west-2/2025/04/02
DEBUG: +++ No logs to process in bucket: <other_account_id>/eu-west-2
DEBUG: +++ DB Maintenance
DEBUG: +++ Working on <third_account_id> - eu-west-2
DEBUG: +++ Marker: AWSLogs/<third_account_id>/GuardDuty/eu-west-2/2025/04/02
DEBUG: +++ No logs to process in bucket: <third_account_id>/eu-west-2
DEBUG: +++ DB Maintenance

I also got inside s3_cloudtrail.db (the only .db file present) and output from SQL query, guardduty table:

sqlite> SELECT * FROM guardduty;
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/03/29/32279e7b-ad2c-41bd-b1b1-41219347c80c.jsonl.gz|2025-04-02 10:11:18|20250329
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/03/29/b15e4b92-ca6d-404a-8529-94f4c63b5ebb.jsonl.gz|2025-04-02 10:11:18|20250329
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/03/29/ee807a71-b78b-4689-afdb-5bc04b122bbb.jsonl.gz|2025-04-02 10:11:18|20250329
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/03/31/380c6d62-da86-4e10-9751-cb1aea9ae18d.jsonl.gz|2025-04-02 10:11:18|20250331
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/04/01/79e9cce1-5b25-48bb-922d-ce387db1a314.jsonl.gz|2025-04-02 10:11:18|20250401
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/04/02/9b1fdff6-2d36-4843-8683-686e48fc785a.jsonl.gz|2025-04-02 10:11:18|20250402
<my-bucket-name>/|<account_id>|AWSLogs/<account_id>/GuardDuty/eu-west-2/2025/04/02/db9ce5f5-b1a7-4646-ab7a-7da8a80ffb41.jsonl.gz|2025-04-02 10:11:18|20250402

So definitely GuardDuty logs are in the bucket and Wazuh does see them. Just none of it appears in Wazuh dashboard, Cloud Security -> AWS. The only source is cloudtrail.

I also wonder, why there is no other .db file than cloudtrail s3_cloudtrail.db? Is it expected, should there not be separate one for guard duty?

[José Luis Cosentino]

Even when you show that wazuh is not finding the logs. You are mentioning logs in the DB as well. Can you please verify if those are located in Archives?

In this instance, you can enable Archives to see whether any of the events Wazuh is receiving have anything to do with AWS. If so, it has been shown that Wazuh is ingesting the events; however, you will need to develop more "decoders or rules" to cause those events to be displayed in the dashboard as "alerts.”

For this practice, you need to go to
☰ > Server management > Settings

Open image-20241121-182702.png

And edit the server configuration, clicking on Edit Configuration:

Open image-20241121-182928.png

Then you need to change the <logall_json></logall_json> From no to yes

<ossec_config> <global> <jsonout_output>yes</jsonout_output> <alerts_log>yes</alerts_log> <logall>no</logall> <logall_json>yes</logall_json> <------------------------------ Here <email_notification>no</email_notification> <smtp_server>smtp.example.wazuh.com</smtp_server> <email_from>wa...@example.wazuh.com</email_from> <email_to>reci...@example.wazuh.com</email_to> <email_maxperhour>12</email_maxperhour> <email_log_source>alerts.log</email_log_source> <agents_disconnection_time>10m</agents_disconnection_time> <agents_disconnection_alert_time>0</agents_disconnection_alert_time> <update_check>yes</update_check> </global>

Save the changes and restart the manager:

Open image-20241121-183415.png

Now you were ready to see all the events coming to the Wazuh-Manager, and you can validate if those AWS events are there or not.

To do so, you just need to grep any AWSlogs familiar keyword in the archives.json file like this:
grep -i ABCDEF1234567890 /var/ossec/logs/archives/archives.json Please let me know your outputs. Regards!

[Jan Zioło]

I changed logall_json to yes and checked the archives and no logs with source: guardduty are there - only source: cloudtrail.

So the logs are not ingested into Wazuh after all. Do you have any suggestions why, and what else could I try to solve this?

[Jan Zioło]

Ok, so I went again through manual attempts to launch this wodle. I got into the .db file and erased everything in guardduty table. The I ran the ams wodle again with guarduty parameter and this time it started ingesting data into Wazuh as expected...

Not sure why as I also tried with cleaning this db before, but looks like the problem is solved.

Regards

Jan