Cloudtrail - bucket content not deleted

[Mihailo Vicanović]

Hi everyone,

I'm having this awkward situation where AWS module is not deleting the contents from S3 buckets which for some reason results in alerts being repeated multiple times. 

Here are the configurations from ossec.conf I tried but with same result:

  <wodle name="aws-s3">
    <disabled>no</disabled>
    <interval>10m</interval>
    <run_on_start>no</run_on_start>
    <skip_on_error>no</skip_on_error>
    <bucket type="cloudtrail">
      <name>my-wazuh-cloudtrail</name>
      <access_key>SECRET</access_key>
      <secret_key>SECRET</secret_key>
      <remove_from_bucket>yes</remove_from_bucket>
    </bucket>
  </wodle>




  <wodle name="aws-s3">
    <disabled>no</disabled>
    <interval>10m</interval>
    <run_on_start>no</run_on_start>
    <skip_on_error>no</skip_on_error>
    <remove_from_bucket>yes</remove_from_bucket>
    <bucket type="cloudtrail">
      <name>my-wazuh-cloudtrail</name>
      <access_key>SECRET</access_key>
      <secret_key>SECRET</secret_key>
    </bucket>
  </wodle>

When using AWS cli from the same server I am able to remove bucket contents. What is more confusing there are no errors in logs:

2018/10/09 09:17:19 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.

2018/10/09 09:17:19 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analisys: sbg-wazuh-cloudtrail

2018/10/09 09:17:24 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

I am being wrong or is there a glitch?

[Jeremy Phillips]

Hi Mihailo,

If you have more than 1000 logs in the bucket, it's possible you could be running into this issue, but I would think that remove_from_bucket would actually resolve this, as it would keep the number of logs below 1000.

I would suggest follow the steps from the troubleshooting guide - https://documentation.wazuh.com/current/amazon/troubleshooting.html - for debugging of the configuration (I would suggest debug level 2 for this situation) and post the debug output from the log to the mailing list.  You should see "++ Found new log: myLogFile" followed by  "+++ Remove file from S3 Bucket: myLogFile".

Also, please confirm you only have the wodle configuration once in the ossec.conf.  The email implies you have it twice???

Thanks,

Jeremy

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/68b3e42c-efb8-4602-aa0d-36e1662bc54b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mihailo Vicanović]

Hi Jeremy,

Thank you for quick response.

First, I can confirm that I have only one wodle configuration. Sorry for the misrepresentation. Those are two versions of wodle config I used.

So, I turned debugging on and here are the results:

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: DEBUG: Found a bucket tag

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: DEBUG: Creating first bucket structure

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: DEBUG: Loop thru child nodes

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: DEBUG: Parse child node: name

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: DEBUG: Parse child node: access_key

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: DEBUG: Parse child node: secret_key

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: INFO: Module AWS started

2018/10/10 09:48:02 wazuh-modulesd:aws-s3: INFO: Waiting interval to start fetching.

2018/10/10 09:57:19 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.

2018/10/10 09:57:19 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analisys: my-cloud-wazuh-cloudtrail

2018/10/10 09:57:19 wazuh-modulesd:aws-s3: DEBUG: Create argument list

2018/10/10 09:57:19 wazuh-modulesd:aws-s3: DEBUG: Launching S3 Command: /var/ossec/wodles/aws/aws-s3 --bucket my-cloud-wazuh-cloudtrail --access_key SECRET --secret_key SECRET --type cloudtrail --debug 2

2018/10/10 09:57:21 wazuh-modulesd:aws-s3: DEBUG: Bucket:  -  OUTPUT: DEBUG: Args: ['/var/ossec/wodles/aws/aws-s3', '--bucket', 'my-cloud-wazuh-cloudtrail', '--access_key', 'SECRET', '--secret_key', 'SECRET', '--type', 'cloudtrail', '--debug', '2']

2018/10/10 09:57:28 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

2018/10/10 10:07:19 wazuh-modulesd:aws-s3: INFO: Starting fetching of logs.

2018/10/10 10:07:19 wazuh-modulesd:aws-s3: INFO: Executing Bucket Analisys: my-cloud-wazuh-cloudtrail

2018/10/10 10:07:19 wazuh-modulesd:aws-s3: DEBUG: Create argument list

2018/10/10 10:07:19 wazuh-modulesd:aws-s3: DEBUG: Launching S3 Command: /var/ossec/wodles/aws/aws-s3 --bucket my-cloud-wazuh-cloudtrail --access_key SECRET --secret_key SECRET --type cloudtrail --debug 2

2018/10/10 10:07:21 wazuh-modulesd:aws-s3: DEBUG: Bucket:  -  OUTPUT: DEBUG: Args: ['/var/ossec/wodles/aws/aws-s3', '--bucket', 'my-cloud-wazuh-cloudtrail', '--access_key', 'SECRET', '--secret_key', 'SECRET', '--type', 'cloudtrail', '--debug', '2']

2018/10/10 10:07:29 wazuh-modulesd:aws-s3: INFO: Fetching logs finished.

As you may see, it looks like argument referring to deletion or removal of logs is missing from the list. 

Here is the other part of the logs (sanitized):

DEBUG: +++ Working on ACCOUNT_ID - us-west-1

DEBUG: +++ Marker: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0005Z_SXB3Qj4faWzx0rHY.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0035Z_en2FNSb8QlVQgXvm.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0105Z_meAZIXf9FMn1cX1P.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0135Z_KXswyBPmMlhpw9bl.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0205Z_woto1LZDlqIln9Ri.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0235Z_zTioCj2LQfadWCQj.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0305Z_vB5j3ASy2yKR74Xi.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0335Z_6lZhh4fQsTd5kHBn.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0405Z_2F4QTQW2HFHPJfzh.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0435Z_GAEMYZSGYJFO5ouy.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0505Z_rzkli0DpPhHqC0K6.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0535Z_CXOtbbCzY3jEl4ro.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0605Z_Vb3nyMWgTFVlG90J.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0635Z_qEF8D6bvCwn3ekbB.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0705Z_fua8sHC5iYpLJSjx.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0720Z_3zch5ZCRZ2af2PqM.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0720Z_OhVhZ6dYUwKIVv5D.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0735Z_CmS4NUDviwAymCRL.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0805Z_65AXfLHaOVJlBJko.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0835Z_09aW368IRgjO60zJ.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0905Z_VBqZx7HXWkekMcjQ.json.gz

DEBUG: ++ Skipping previously processed file: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T0935Z_C0CEC4Al4uqJXzxe.json.gz

DEBUG: ++ Found new log: AWSLogs/ACCOUNT_ID/CloudTrail/us-west-1/2018/10/10/ACCOUNT_ID_CloudTrail_us-west-1_20181010T1005Z_w73Ao5azNhuJO2yX.json.gz

It looks like deletion/removal wasn't even attempted...

And I have configured number of events to be 1500. 

[Jeremy Phillips]

Hi Mihailo,

I did some digging thru the code and testing of various configs.  In short, the remove_from_bucket functionality doesn't appear to be functioning as expected.  I've opened an issue on this - https://github.com/wazuh/wazuh/issues/1618

In the short term, to get the expected output you want, please try adding the remove_from_bucket option to the bucket with a value of no.  That should produce the result you are looking for...

Thanks,

Jeremy

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/3eca3d5d-668f-4085-a3d4-3a5a6cce7e5f%40googlegroups.com.

[Mihailo Vicanović]

Thanks Jeremy. 

I have followed your instructions and with remove_from_bucket option to the bucket with a value set to no it works as expected.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.