AWS S3 Bucket Error

[Utkarsh Bhargava]

Hello Community,

I am trying to read S3 bucket and facing the following error :

2021/03/08 18:04:39 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs/CloudTrail/AWSLogs/165328851411/CloudTrail/ap-southeast-2/2021/03/, Type: cloudtrail)

2021/03/08 18:04:40 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Returned exit code 12

2021/03/08 18:04:40 wazuh-modulesd:aws-s3: WARNING: Bucket:  -  Parameter validation failed:

Invalid bucket name "clrplogs/CloudTrail/AWSLogs/165328851411/CloudTrail/ap-southeast-2/2021/03/": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-]{1,63}$"

Can anyone help me how I can fix it.

regards

[Jose Miguel Hernandez Garcia]

Hi!

Could you share your aws-s3 wodle configuration from your ossec.conf ?

Looks like you set a wrong bucket name. Keep in mind that if you want to add an specific path from inside your bucket, you need to add the <path> option to your bucket section. I.E:

<wodle name="aws-s3">

    <disabled>no</disabled>

    <remove_from_bucket>no</remove_from_bucket>

    <interval>10m</interval>

    <run_on_start>no</run_on_start>

    <skip_on_error>no</skip_on_error>

    <bucket type="cloudtrail">

        <name>s3-dev-bucket</name>

        <access_key>insert_access_key</access_key>

        <secret_key>insert_secret_key</secret_key>

        <only_logs_after>2018-JUN-01</only_logs_after>

        <regions>us-east-1,us-west-1,eu-central-1</regions>

        <path>/dev1/</path>

        <aws_account_id>123456789012</aws_account_id>

        <aws_account_alias>dev1-account</aws_account_alias>

    </bucket>

</wodle>

Hope that this helps!

Best regards,

Josemi.

[Utkarsh Bhargava]

Hi Jose,

Thank you for your quick response. I have just made changes into my configuration file and now it's giving me error 14 ( Empty Bucket) Allthough my bucket is full of logs.

Here's the configuration :

<wodle name="aws-s3">

    <disabled>no</disabled>

    <remove_from_bucket>no</remove_from_bucket>

    <interval>10m</interval>

    <run_on_start>yes</run_on_start>

    <skip_on_error>no</skip_on_error>

    <bucket type="cloudtrail">

        <name>clrplogs</name>

        <access_key>aws-key</access_key>

        <secret_key>aws-secret-key</secret_key>

        <only_logs_after>2021-MAR-01</only_logs_after>

        <regions>ap-southeast-1</regions>

        <path>clrplogs/CloudTrail/AWSLogs/165328851411/CloudTrail/ap-southeast-2/03/01/</path>>

        <aws_account_id>aws-id-number</aws_account_id>

        <aws_account_alias>ismac-role</aws_account_alias

    </bucket>

</wodle>

Here's the error log :

2021/03/09 13:52:24 wazuh-modulesd:aws-s3: WARNING: Bucket: 165328851411(ismac-role)  -  Returned exit code 14

2021/03/09 13:52:24 wazuh-modulesd:aws-s3: WARNING: Bucket: 165328851411(ismac-role)  -  No files were found in 'clrplogs/clrplogs/CloudTrail/AWSLogs/165328851411/CloudTrail/ap-southeast-2/03/01/'. No logs will be processed.

I don't know why it's unable to read the logs out of my bucket.

regards

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5c7646dd-77ec-4517-9de2-3b027e780a6bn%40googlegroups.com.

[Jose Miguel Hernandez Garcia]

Hi!

Sorry for the late reply.

Keep in mind that it's not necessary to add the bucket name in the <path></path> attribute:

<path>clrplogs/CloudTrail/AWSLogs/165328851411/CloudTrail/ap-southeast-2/03/01/</path>

should be:

<path>/CloudTrail/AWSLogs/165328851411/CloudTrail/ap-southeast-2/03/01/</path>

Also, what permissions does the AWS role/user that are you adding to the configuration has? Are you able to list the content of your bucket from within the manager server by using the AWS CLI? 

You could do this by executing the following command:

aws s3 ls s3://your_bucket

This could help you check if you are facing any permission issues.

Hope that this helps!

Best regards,

Josemi.

[Utkarsh Bhargava]

Hi Jose,

Sorry for this delayed response.

I tried accessing S3 buckets using AWS CLI and it worked for me but still I am unable to read the S3 bucket. I am not receiving any error messages on wazuh logs.

When I am looking into elasticsearch I only have 2 logs in past 30 minutes that too from CloudTrail.

My S3 bucket is full of logs.

For your consideration here's my AWS Module configuration:

<wodle name="aws-s3">

    <disabled>no</disabled>

    <remove_from_bucket>no</remove_from_bucket>

    <interval>10m</interval>

    <run_on_start>yes</run_on_start>

    <skip_on_error>no</skip_on_error>

    <bucket type="waf">

        <name>clrplogs</name>

        <path>2021</path>

        <access_key>aws access key</access_key>

        <secret_key>aws secret key</secret_key>

        <only_logs_after>2021-MAR-01</only_logs_after>

        <regions>ap-southeast-1</regions>

        <aws_account_id>165528851518</aws_account_id>

        <aws_account_alias>admin-role</aws_account_alias>

        <iam_role_arn>arn:aws:iam::165528851518:role/admin-role</iam_role_arn>

    </bucket>

    <bucket type="cloudtrail">

        <name>clrplogs</name>

        <path>CloudTrail</path>

        <access_key>aws access key</access_key>

        <secret_key>aws secret key</secret_key>

        <only_logs_after>2021-MAR-01</only_logs_after>

        <regions>ap-southeast-1</regions>

        <aws_account_id>165528851518</aws_account_id>

        <aws_account_alias>admin-role</aws_account_alias>

        <iam_role_arn>arn:aws:iam::165528851518:role/admin-role</iam_role_arn>

    </bucket>

<bucket type="vpcflow">

        <name>clrplogs</name>

        <path>Flowlogs</path>

        <access_key>aws access key</access_key>

        <secret_key>aws secret key</secret_key>

        <only_logs_after>2021-MAR-01</only_logs_after>

        <regions>ap-southeast-1</regions>

        <aws_account_id>165528851518</aws_account_id>

        <aws_account_alias>admin-role</aws_account_alias>

        <iam_role_arn>arn:aws:iam::165528851518:role/admin-role</iam_role_arn>

    </bucket>

    <bucket type="custom">

        <name>clrplogs</name>

        <path>RDSLogs</path>

        <access_key>aws access key</access_key>

        <secret_key>aws secret key</secret_key>

        <only_logs_after>2021-MAR-01</only_logs_after>

        <regions>ap-southeast-1</regions>

        <aws_account_id>165528851518</aws_account_id>

        <aws_account_alias>admin-role</aws_account_alias>

        <iam_role_arn>arn:aws:iam::165528851518:role/admin-role</iam_role_arn>

    </bucket>

<bucket type="custom">

        <name>clrplogs</name>

        <path>CLRP_EnvtLogs</path>

        <access_key>aws access key</access_key>

        <secret_key>aws secret key</secret_key>

        <only_logs_after>2021-MAR-01</only_logs_after>

        <regions>ap-southeast-1</regions>

        <aws_account_id>165528851518</aws_account_id>

        <aws_account_alias>admin-role</aws_account_alias>

        <iam_role_arn>arn:aws:iam::165528851518:role/admin-role</iam_role_arn>

    </bucket>

</wodle>

Please let me know if there's any issues with this configuration.

regards

Utkarsh

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c902c093-9406-4302-98b1-eaf6140d140en%40googlegroups.com.

[Utkarsh Bhargava]

Here are the wazuh logs :

2021/03/15 07:29:23 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: CloudTrail, Type: cloudtrail, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:29:27 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: Flowlogs, Type: vpcflow, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:29:30 wazuh-modulesd:aws-s3: WARNING: Bucket: 165328851411(ismac-role)  -  Returned exit code 12

2021/03/15 07:29:30 wazuh-modulesd:aws-s3: WARNING: Bucket: 165328851411(ismac-role)  -  An error occurred (UnauthorizedOperation) when calling the DescribeFlowLogs operation: You are not authorized to perform this operation.

2021/03/15 07:29:30 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: RDSLogs, Type: custom, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:29:33 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: CLRP_EnvtLogs, Type: custom, Account ID: 565358651418, Account admin: ismac-role)

2021/03/15 07:29:41 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

2021/03/15 07:39:20 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.

2021/03/15 07:39:20 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: 2021, Type: waf, Account ID: 165328851411, Account Alias: admin-role)

2021/03/15 07:39:23 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: CloudTrail, Type: cloudtrail, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:39:28 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: Flowlogs, Type: vpcflow, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:39:31 wazuh-modulesd:aws-s3: WARNING: Bucket: 165328851411(ismac-role)  -  Returned exit code 12

2021/03/15 07:39:31 wazuh-modulesd:aws-s3: WARNING: Bucket: 165328851411(ismac-role)  -  An error occurred (UnauthorizedOperation) when calling the DescribeFlowLogs operation: You are not authorized to perform this operation.

2021/03/15 07:39:31 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: RDSLogs, Type: custom, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:39:34 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analysis: (Bucket: clrplogs, Path: CLRP_EnvtLogs, Type: custom, Account ID: 565358651418, Account Alias: admin-role)

2021/03/15 07:40:00 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

thank you

[Jose Miguel Hernandez Garcia]

Hi!

By checking these logs, looks like the role you are using to obtain the S3 logs does not have the required permission. Did you double checked this role in IAM to see if the role has all the correct permissions?

Related to your configuration:

Keep in mind that if you are using an IAM Role it's not necessary to use <access_key> and <secret_key> credentials and vice-versa. 

 Also, as the aws-s3 wodle documentation says (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-s3.html#bucket-type), <aws_account_id> only works with CloudTrail buckets.

Hope that this helps,

Best regards,

Josemi.

[Jose Miguel Hernandez Garcia]

Hi!

Did you manage to solve your problem? Do you need more help with it?

[Utkarsh Bhargava]

Hi Jose,

Thank you for your email. I will look into it and get back to you.

As of now I am unable to read the logs.

regards

On Mar 15 2021, at 6:05 pm, Jose Miguel Hernandez Garcia <jm.her...@wazuh.com> wrote:

Hi!

By checking these logs, looks like the role you are using to obtain the S3 logs does not have the required permission. Did you double checked this role in IAM to see if the role has all the correct permissions?

Related to your configuration:

Keep in mind that if you are using an IAM Role it's not necessary to use <access_key> and <secret_key> credentials and vice-versa. 

 Also, as the aws-s3 wodle documentation says (https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-s3.html#bucket-type), <aws_account_id> only works with CloudTrail buckets.

Hope that this helps,

Best regards,

Josemi.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/36c80374-3561-4aa9-9f50-328303b39fe2n%40googlegroups.com.