AWS S3 ingestion fails to pick up logs

[Louis Hather]

Hi,

I'm running Wazuh 3.13.1 (Kibana 7.8.1) and have configured it to ingest logs from AWS GuardDuty, CloudTrail, and Cisco Umbrella logs from S3. I'm able to ingest logs from GuardDuty, but CloudTrail and Cisco Umbrella aren't working. There doesn't seem to be much in the way of errors:

2021/01/14 12:05:03 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.

2021/01/14 12:05:03 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: redacted, Type: cloudtrail, Account ID: redacted)

2021/01/14 12:05:08 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: redacted Type: cloudtrail, Account ID: redacted)

2021/01/14 12:05:12 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: redacted, Type: cloudtrail, Account ID: redacted)

2021/01/14 12:05:16 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: redacted, Type: guardduty, Account ID: redacted)

2021/01/14 12:05:18 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: redacted, Path: dnslogs, Type: cisco_umbrella)

2021/01/14 12:05:19 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

Running the aws-s3 wodle manually shows it's able to get logs, though no events are ingested aside from a event with ID 11 level 4, with the full_log field:

The average number of logs between 15:00 and 16:00 is 40091. We reached 100229.

The event seems to have some information taken from CloudTrail. The same is true for Cisco Umbrella. Interestingly event 11 isn't logged when the wodle is run on a schedule, only manually. 

[carlos...@wazuh.com]

Hello Louis,

Let me try to help you with this issue.

Could you please share with us your current AWS module configuration, so I can review it an ensure everything is ok? You can find it on Kibana and also in the "{WAZUH_PATH}/etc/ossec.conf" file for the node you have this configured. Make sure you remove any sensitive data, if present, as you did previously. Also, please check your alert level value. Any alert raised with a level below that value will be ignored, so that could be the case too. You can find it the "alerts" section of your ossec.conf configuration.

Additionally, I suggest you to enable debug mode for modulesd and try to run the module again, manually and scheduled, to see if there is any error or warning message. You can easily enable it by adding the following line to the "{WAZUH_PATH}/etc/local_internal_options.conf" file:

wazuh_modules.debug=2

After that, restart the wazuh service and look for any AWS related messages in the "{WAZUH_PATH}/logs/ossec.log". You can do so by running the following command for a few minutes:

tail -f {WAZUH_PATH}/logs/ossec.log | grep aws

Don't forget to replace "{WAZUH_PATH}" with the path you have Wazuh installed on. By default, it is "/var/ossec/".

With this information I may able to see whats happening with your logs. Take into account that it is possible that you are ingesting the logs but they don't trigger any alert simply because they shouldn't.

Regards.

[Louis Hather]

Thank you for the response!

The AWS module configuration is as follows:

<ossec_config>

  <wodle name="aws-s3">

    <disabled>no</disabled>

    <remove_from_bucket>no</remove_from_bucket>

    <interval>30m</interval>

    <run_on_start>yes</run_on_start>

    <skip_on_error>no</skip_on_error>

    <bucket type="cloudtrail">

      <name>REDACTED</name>

      <only_logs_after>2021-JAN-14</only_logs_after>

      <iam_role_arn>REDACTED</iam_role_arn>

    </bucket>

    <bucket type="guardduty">

      <name>REDACTED</name>

      <only_logs_after>2021-JAN-14</only_logs_after>

      <iam_role_arn>REDACTED</iam_role_arn>

      <path>firehose/</path>

    </bucket>

  </wodle>

</ossec_config>

The coordinator sits in one AWS account, while the IAM role and S3 buckets configured are in a separate account. The role applied to the coordinator allows it to assume said role in the other account. I know this works, as running the aws-s3 command (copied from ossec.log with wazuh_modules.debug=2) will display data from these buckets in stdout. We can also see that the roles have been used to access the buckets via IAM access advisor. 

DEBUG: +++ Working on REDACTED - REDACTED

DEBUG: +++ Marker: AWSLogs/REDACTED/CloudTrail/REDACTED/2021/01/14/REDACTED_CloudTrail_REDACTED_20210114T0025Z_37F25NhZ2zjVmk90.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/REDACTED/CloudTrail/REDACTED/2021/01/14/REDACTED_CloudTrail_REDACTED_20210114T0100Z_g4OvrL9efPcOrGDl.json.gz

I can make it reprocess files, and new files are being picked up, and there are no error messages that I can see.

Entries from Cloudtrail can also be found in ossec/logs/archives.log. Lowering the alert level to 0 didn't seem to change anything. 

[carlos...@wazuh.com]

Its seems the integration with CloudTrail is working just fine.

I guess when you say that no events are being ingested you are checking it in the Wazuh APP (kibana). If that's the case, please ensure you are looking for the alerts in the right place. For example, AWS CloudTrail logs should appear in the "Security Information Management" > "Amazon AWS" section:

By default that section is hide. Here is how you can enable it in the latest version of Kibana:

And here is how to enable it for older Kibana releases:

Please check if there are any CloudTrail alert there. If not, please check if any of your CloudTrail logs should be triggering any alert.

In case this does not solve your issue please share with us one of those Cloudtrail entries that can be found in ossec/logs/archives.log. 

Regards.

[Louis Hather]

Hi,

The Amazon AWS module is enabled - I can use it to see events from GuardDuty but nothing from CloudTrail.

Here's an entry from the archive logs - there's tons of events there but I don't think they're being ingested. 

{
"integration": "aws",
"aws": {
"log_info": {
"aws_account_alias": "",
"log_file": "AWSLogs/REDACTED..",
"s3bucket": "REDACTED"
},
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "REDACTED",
"arn": "REDACTED",
"accountId": "REDACTED",
"accessKeyId": "REDACTED",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "REDACTED",
"arn": "REDACTED",
"accountId": "REDACTED",
"userName": "REDACTED"
},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "REDACTED"
}
}
},
"eventTime": "REDACTED",
"eventSource": "lambda.amazonaws.com",
"eventName": "REDACTED",
"awsRegion": "REDACTED",
"sourceIPAddress": "REDACTED",
"userAgent": "REDACTED",
"requestID": "REDACTED",
"eventID": "REDACTED",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"eventCategory": "Management",
"recipientAccountId": "REDACTED",
"source": "cloudtrail",
"aws_account_id": "REDACTED",
"source_ip_address": "REDACTED"
}
}

[Louis Hather]

Hi, I think the issue is the logs don't match any rules, though I believe what we're ingesting should match rule 80202.

I used ossec-logtest to see what happens to a sample cloudtrail event. 

**Phase 1: Completed pre-decoding.

       full event: '{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/.... "eventSource": "s3.amazonaws.com", "eventName": "DeleteBucket", "awsRegion": "... "source": "cloudtrail", "...}}'

       timestamp: '(null)'

       hostname: '...'

       program_name: '(null)'

       log: '{"integration": "aws", "aws": {"log_info": {"aws_account_alias": "", "log_file": "AWSLogs/.... "eventSource": "s3.amazonaws.com", "eventName": "DeleteBucket", "awsRegion": "... "source": "cloudtrail", "...}}'

**Phase 2: Completed decoding.

       decoder: 'json'

       integration: 'aws'

      ..

       aws.eventName: 'DeleteBucket'

       ..

       aws.source: 'cloudtrail'

       ..

**Phase 3: Completed filtering (rules).

       Rule id: '80200'

       Level: '0'

       Description: 'AWS alert.'

I've removed the bulk of the event. The event name, DeleteBucket, is listed in lists/amazon/aws-eventnames so I think that should've trigged 80202.