KeyError: 'CommonPrefixes': Integration with AWS config

[GopiV]

Note: I've already opened issue on git but then recommendation is to POST issues on google maillist. https://github.com/wazuh/wazuh/issues/4166

Version: 3.9.2 I'm getting below exception while trying to integrate aws config with Wazuh. Unknown error: 'CommonPrefixes' Traceback (most recent call last): File "/var/ossec/wodles/aws/aws-s3.py", line 2272, in main(sys.argv[1:]) File "/var/ossec/wodles/aws/aws-s3.py", line 2239, in main bucket.iter_bucket(options.aws_account_id, options.regions) File "/var/ossec/wodles/aws/aws-s3.py", line 710, in iter_bucket self.iter_regions_and_accounts(account_id, regions) File "/var/ossec/wodles/aws/aws-s3.py", line 993, in iter_regions_and_accounts account_id = self.find_account_ids() File "/var/ossec/wodles/aws/aws-s3.py", line 862, in find_account_ids Delimiter='/')['CommonPrefixes'] KeyError: 'CommonPrefixes' But there is no key name called 'CommonPrefixes' in the output of list_objects_v2 for config s3 bucket just like guardduty bucket while it is present for cloudtrail. Anyway looks like for guardduty wazuh is not trying to get find_account_ids which makes sense but for config it is done which is not necessary and not being handled. I've event tried your latest aws-s3.py which is throwing same error for config. Bucket: wazuh-config-xxx Prefix: config/AWSLogs/ list_objects_v2: {'ResponseMetadata': {'RequestId': 'xxx', 'HostId': 'xxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'xxx', 'x-amz-request-id': 'xxx', 'date': 'Wed, 30 Oct 2019 00:12:11 GMT', 'x-amz-bucket-region': 'us-west-2', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Name': 'wazuh-config-xxx', 'Prefix': 'config/AWSLogs/', 'Delimiter': '/', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 0} s3.list_objects_v2(Bucket='wazuh-guardduty-xxx', Prefix='guardduty/AWSLogs/', Delimiter='/') {'ResponseMetadata': {'RequestId': 'xxx', 'HostId': 'xxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'xxx', 'x-amz-request-id': 'xxx', 'date': 'Wed, 30 Oct 2019 00:14:04 GMT', 'x-amz-bucket-region': 'us-west-2', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 1}, 'IsTruncated': False, 'Name': 'wazuh-guardduty-xxx', 'Prefix': 'guardduty/AWSLogs/', 'Delimiter': '/', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 0} s3.list_objects_v2(Bucket='wazuh-cloudtrail-xxx', Prefix='cloudtrail/AWSLogs/', Delimiter='/') {'ResponseMetadata': {'RequestId': 'xxx', 'HostId': 'xxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'xxx', 'x-amz-request-id': 'xxx', 'date': 'Wed, 30 Oct 2019 00:15:05 GMT', 'x-amz-bucket-region': 'us-west-2', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 1}, 'IsTruncated': False, 'Name': 'wazuh-cloudtrail-xxx', 'Prefix': 'cloudtrail/AWSLogs/', 'Delimiter': '/', 'MaxKeys': 1000, 'CommonPrefixes': [{'Prefix': 'cloudtrail/AWSLogs/xxxx/'}], 'EncodingType': 'url', 'KeyCount': 1} I've also used your latest aws-s3.py file yesterday. In the new file wazuh is handling exception from list_objects_v2 as "ERROR: Invalid type of bucket. The bucket was set up as 'config' type and this bucket does not contain log files from this type. Try with other type: {'guardduty', 'cloudtrail', 'custom', 'vpcflow'}". But I think the error handling is not done properly and the new version is actual masking the real exception. The real underlying exception is still KeyError: 'CommonPrefixes' because the output of self.client.list_objects_v2(Bucket=self.bucket, Prefix=self.get_base_prefix(), Delimiter='/') doesn't contain key with name "CommonPrefixes" when we are using bucket type of config just like bucket type gurardduty. The logs path for config doesn't contain account id. This is happening when find_account_ids method is being called. /var/ossec/wodles/aws/aws-s3 --bucket wazuh-config-xxx --trail_prefix config --type config --debug 3 +++ sys args: ['--bucket', 'wazuh-config-xxx', '--trail_prefix', 'config', '--type', 'config', '--debug', '3'] DEBUG: +++ Debug mode on - Level: 3 DEBUG custom: dbpath: /var/ossec/wodles/aws/s3_cloudtrail.db DEBUG: +++ In iter_bucket {'ResponseMetadata': {'RequestId': 'xxx', 'HostId': 'xxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'xxx', 'x-amz-request-id': 'xxx', 'date': 'Tue, 29 Oct 2019 23:37:36 GMT', 'x-amz-bucket-region': 'us-west-2', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 0}, 'IsTruncated': False, 'Name': 'wazuh-config-xxx', 'Prefix': 'config/AWSLogs/', 'Delimiter': '/', 'MaxKeys': 1000, 'EncodingType': 'url', 'KeyCount': 0} ERROR: Invalid type of bucket. The bucket was set up as 'config' type and this bucket does not contain log files from this type. Try with other type: {'guardduty', 'cloudtrail', 'custom', 'vpcflow'} Also log files are present in config s3 bucket. s3 = boto3.client('s3') s3.list_objects_v2(Bucket='wazuh-config-xxx') {'ResponseMetadata': {'RequestId': 'xxx', 'HostId': 'xxx', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'xxx', 'x-amz-request-id': 'xxx', 'date': 'Wed, 30 Oct 2019 16:00:53 GMT', 'x-amz-bucket-region': 'us-west-2', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3'}, 'RetryAttempts': 1}, 'IsTruncated': False, 'Contents': [{'Key': 'config/2019/10/29/21/xxx-1-2019-10-29-21-55-39-257e84e7-a1f2-4a2f-9f27-fe048d8d2ac8', 'LastModified': datetime.datetime(2019, 10, 29, 22, 0, 41, tzinfo=tzutc()), 'ETag': '"a708df98bbb6caf30d4bc45198e7cb36"', 'Size': 145907, 'StorageClass': 'STANDARD'}, {'Key': 'config/2019/10/29/22/xxx-1-2019-10-29-22-00-40-a485c626-2b31-4ae1-bcef-a6e72333ec31', 'LastModified': datetime.datetime(2019, 10, 29, 22, 5, 43, tzinfo=tzutc()), 'ETag': '"3de7fbfa0aa56354ed6cbbf7d1b0aea7"', 'Size': 289643,.............. woodle config:   <wodle name="aws-s3">     <disabled>no</disabled>     <interval>10m</interval>     <run_on_start>yes</run_on_start>     <skip_on_error>yes</skip_on_error>     <bucket type="guardduty">       <name>wazuh-guardduty-xxx</name>       <path>guardduty</path>     </bucket>     <bucket type="cloudtrail">       <name>wazuh-cloudtrail-xxx</name>       <path>cloudtrail</path>       <aws_account_id>xxx</aws_account_id>       <regions>us-west-2</regions>       <only_logs_after>2019-OCT-29</only_logs_after>     </bucket>     <bucket type="config">       <name>wazuh-config-xxx</name>       <path>config/</path>     </bucket>   </wodle>

[Demetrio Ruiz]

Hi GopiV,

The expected path to the AWS Config logs by the wodle is the next: <bucket_name>/<prefix>/AWSLogs/<account_id>/Config/<region>/<year>/<month>/<day>. For GuardDuty, the expected path is <bucket_name>/<prefix>/<year>/<month>/<day>/<hh>.

The wodle isn't flexible with this, and this should be improved in order to avoid error as yours.

In our test environment, we have logs of these services which have these paths. If you haven't got a subdirectory <account_id>, we should give support for it (we need to provide a more flexible way of getting logs in all services). I think that removing these lines https://github.com/wazuh/wazuh/blob/3.10/wodles/aws/aws_s3.py#L996-L999, you should get logs from Config service. Below there is the code that I suggest to you for replacing in your wodle and getting Config logs:

    def iter_regions_and_accounts(self, account_id, regions):
        # fix for getting logs when no account_id in paths
        if not regions:
            regions = self.find_regions(aws_account_id)
        for aws_region in regions:
            if self.old_version:
                self.migrate(aws_account_id=aws_account_id, aws_region=aws_region)
            debug("+++ Working on {} - {}".format(aws_account_id, aws_region), 1)
            # for processing logs day by day
            date_list = self.get_date_list(aws_account_id, aws_region)
            for date in date_list:
                self.iter_files_in_bucket(aws_account_id, aws_region, date)
            self.db_maintenance(aws_account_id=aws_account_id, aws_region=aws_region)

About handling the exception Unknown error: 'CommonPrefixes', I agree with you and we will try to improve it in next releases. When we handle this exception, we are supposing that the paths to the logs are fixed, but in your situation this is different, and this supposition is wrong.

Thank you very much for helping us to improve the AWS integration. I hope that after applying this change you can process Config logs.

Best regards,

Demetrio.

[GopiV]

Demetrio, 

I found the reason for all this confusion. Looks like the documentation needs to be updated.

https://documentation.wazuh.com/3.10/amazon/services/config.html

https://documentation.wazuh.com/3.9/amazon/services/config.html

Documentation says to use Cloudwatch to send Config events to firehose. firehose sends logs to S3. In this flow AWS is creating this directory structure under prefix: YYYY/MM/DD/HH

S3 bucket that we use for config to store configuration history and configuration snapshot files has this directory structure under prefix: /AWSLogs/<ACCOUNT_NUMBER>/Config/<REGION>/YYYY/MM/DD/ConfigHistory.

aws-s3.py code is following S3 bucket format of config which documentation provides totally different instructions. Again AWS should follow standard directory structure format for all their logs and the same time as you said should be able to handle any directory structure to find the logs which make it more resilient to dir structure changes from AWS.

[Demetrio Ruiz]

Hi Gopi V,

It seems that our documentation is confusing. I added a screenshot in step 3 of AWS Config documentation to avoid it:

In this screenshot, you can see that the AWS Config logs are stored in the expected directory structure.

Best regards,

Demetrio.