Anti-flooding mechanism doesn't work

[mauro....@cmcc.it]

Dear All,

one of our Wazuh agents is under flood attack and a lot of break attempts are detected in the httpd.log file.
So, this agent is sending a huge load of logs in the same seconds with the same timestamp to the manager.
Wazuh Manager reacts sending, also at the same time, a great number of active-response tasks that often cause a dangerous hang of the OS.

If I execute "ps -ef|grep firewall-cmd" command, I can see a lot of firewall-cmd pending processes on the agent.

In order to block this behavior, I changed the max_eps value for this specific agent to 5:

<client_buffer>
    <!-- Agent buffer options -->
    <disabled>no</disabled>
    <queue_size>5000</queue_size>
    <events_per_second>5</events_per_second>
</client_buffer>

and I enforced agent.min_eps value running the following command: echo "agent.min_eps=10" >> /var/ossec/etc/local_internal_options.conf + wazuh-agent daemon restart

Unfortunately, nothing changed: wazuh agent is still sending a great number of events to Wazuh Manager.

Is there something I can do?
Could you please help me to solve this issue?

Thanks in advance,
Mauro

[Santiago Belluzzo]

Hey!

It seems that there's an error in your configuration. The agent.min_eps is the internal config for the lowest value that can be set on the events_per_second of that agent

You are setting the agent.min_eps to 10 but then trying to set the events_per_second to a value lower than that (5). Although this should be setting the events_per_second to 10 which is a low value but not exactly what you expect.

You can confirm that this is happening and being set by restarting the agent and searching in the {wazuh-Install-folder}/logs/ossec.log for the string:

    Client buffer throughput too low: set to 10 eps

If you actually want to set he events_per_second to 5 you'll need to set the agent.min_eps = 5 too

In case it helps here are our docs about the log flooding mechanisms 

https://documentation.wazuh.com/current/user-manual/capabilities/antiflooding.html

as well as a 'Lab' on how to set up and test an example configuration

https://documentation.wazuh.com/current/learning-wazuh/survive-flood.html

Let me know if this helps.

Santi

[Mauro Tridici]

Hello Santiago,

thank you for your help.

I just fixed my configuration and I did the check you suggested: in the agent logs, I can’t see any output or warning related to the client buffer settings.

It seems that the agent didn’t accept the change. So, I reverted to the standard client buffer configuration (that is max_eps=500 and "agent.min_eps=5" line removed from the agent local config)

Anyway, I’m still noticing that, when the agent is under the flood attack, rules n. 31123 and 31163 are fired a lot of times in a few seconds (please, take a look at the screenshot below).

The most part of the fired rules are “translated” in “active-reponse” actions by the Wazuh Manager and, for this reason, I see a lot of pending firewall-drop scripts in the agent.

Is there an alternative way to reduce the impact of a flood attack on the number of times that the rules are fired?

Many thanks in advance,

Mauro

On 24 Feb 2022, at 21:49, Santiago Belluzzo <santiago...@wazuh.com> wrote:

Hey!

It seems that there's an error in your configuration. The agent.min_eps is the internal config for the lowest value that can be set on the events_per_second of that agent

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/WKCOk5-xfDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/aca42597-ca9b-4dc8-9993-722a16321a2cn%40googlegroups.com.
<Screenshot_20220224_174046.png>

[Santiago Belluzzo]

Sure!

You can modify the 'spammy' rules (31123 and 31163 in your case) and add a 'ignore' time setting (or alternatively a  'Frequency' and 'Timeframe' setting) to them so they only generate and send alerts every X seconds

Here are some steps on how to modify and replace stock rules https://documentation.wazuh.com/current/learning-wazuh/replace-stock-rule.html that you can follow as an example, adapting it to your case.

You can see the documentation on those settings here https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html?highlight=ignore

Let me know if this helps,

Santi

[Mauro Tridici]

Great! Many thanks, Santiago! It seems it works :)

In my local_rule file I added these lines:

<group name="web,accesslog,">

  <rule id="31163" level="10" frequency="14" timeframe="120" overwrite="yes">

    <if_matched_sid>31123</if_matched_sid>

    <same_source_ip />

    <ignore>60</ignore>

    <description>Multiple web server 503 error code (Service unavailable).</description>

    <group>web_scan,recon,pci_dss_6.5,pci_dss_11.4,pci_dss_10.6.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_SA.11,nist_800_53_SI.4,nist_800_53_AU.6,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>

  </rule>

</group>

Last question, I promise :) 

Is there a way to apply the “new” rule only for a particular and specific agent and leave the default one for the other agents?

If it is not possible, no problem. I will use the new rule for all the agents.

Thank you again for your help and patience.

Have a great weekend.

Mauro

On 25 Feb 2022, at 22:10, Santiago Belluzzo <santiago...@wazuh.com> wrote:

Sure!

You can modify the 'spammy' rules (31123 and 31163 in your case) and add a 'ignore' time setting (or alternatively a  'Frequency' and 'Timeframe' setting) to them so they only generate and send alerts every X seconds

Here are some steps on how to modify and replace stock rules https://documentation.wazuh.com/current/learning-wazuh/replace-stock-rule.html that you can follow as an example, adapting it to your case.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/edcea622-9b9e-4bbe-aeb9-38205585e4b4n%40googlegroups.com.
<ignore.png>

[Mauro Tridici]

Hello Santiago,

Sorry if I’m disturbing you again, but I just noticed that “ignore” instruction (maybe) is not working as expected.

When rule #31163 is fired, it should be “ignored” for 180 seconds, right?

If my understanding is correct, I shouldn’t see the same rule fired in a few seconds.

Unfortunately, in Wazuh GUI I see a this unexpected behavior:

This is the rule #31163 defined in local_rules.xml file

<group name="web,accesslog,">

  <rule id="31163" level="10" frequency="14" timeframe="120" overwrite="yes">

    <if_matched_sid>31123</if_matched_sid>

    <same_source_ip />

    <ignore>180</ignore>

    <description>Multiple web server 503 error code (Service unavailable).</description>

    <group>web_scan,recon,pci_dss_6.5,pci_dss_11.4,pci_dss_10.6.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_SA.11,nist_800_53_SI.4,nist_800_53_AU.6,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>

  </rule>

</group>

Could you please help me to understand how to solve this issue?

Thank you in advance,

Mauro

On 26 Feb 2022, at 22:02, Mauro Tridici <mauro....@cmcc.it> wrote:

Great! Many thanks, Santiago! It seems it works :)

In my local_rule file I added these lines:

<group name="web,accesslog,">

  <rule id="31163" level="10" frequency="14" timeframe="120" overwrite="yes">

    <if_matched_sid>31123</if_matched_sid>

    <same_source_ip />

    <ignore>180</ignore>

[Santiago Belluzzo]

Hey!

Sorry for the late response. I checked the rule and it seems that the ignore is not being set correctly. The rule should look like

-----------------------

  <rule id="31163" level="10" ignore="180" frequency="14" timeframe="120" overwrite="yes">

    <if_matched_sid>31123</if_matched_sid>

    <same_source_ip />

    <description>Multiple web server 503 error code (Service unavailable).</description>

    <group>web_scan,recon,pci_dss_6.5,pci_dss_11.4,pci_dss_10.6.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_SA.11,nist_800_53_SI.4,nist_800_53_AU.6,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>

  </rule>

--------------------------------

This means that the rule 31163 will only trigger an alert if its matched 14 times in 120 seconds and can only trigger the alert once every 180 seconds. I would suggest removing the frequency and timeframe and only set the Ignore time first and see if that suffices to solve your issue.

Try this configuration and let me know how it goes,

Santi

[Mauro Tridici]

Hello Santiago,

many thanks for you support and patience.

I just made the correction you suggested. I will check the behavior of Wazuh during the next days and I will let you know.

I really appreciated your help.

Kind Regards,

Mauro

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/defc68ee-1988-43d7-b983-8a94935f4864n%40googlegroups.com.