Correlation ruleset syntax

[TUKARAM GAONKAR]

Hi Team ,

I have wrriten below rule . When i want to genrate alert if Ip connected from Blacklisted IP followed by authentication successfull.

But correlation rule is not working . Please help me resolved the isue .

Rule ::::::::::::::::

<group name="syslog,attacks,”>

<rule id="100100" level="12">

    <if_group>web|attack|attacks</if_group>

    <list field="srcip" lookup="address_match_key">etc/lists/blacklist_alienvault_windows_test</list>

    <description>IP address found in AlienVault reputation database.</description>

</rule>

 </group>

Correlation Rule ::::::::::

<group name="syslog,attacks,”>

   <rule id="900805" level="12" timeframe= "500"> 

   <if_matched_sid>100100</if_matched_sid>

   <if_sid>5715</if_sid>

   <same_srcip />

   <description>Authentication success from Blacklisted IP</description>

   <mitre> <id>T1531</id> </mitre>        

 <group>pci_dss_10.2.7,pci_dss_10.2.5,pci_dss_8.1.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,hipaa_164.312.a.2.I,hipaa_164.312.a.2.II,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AC.2,nist_800_53_IA.4,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group> 

 </rule>

 </group>

 

Log1

178.30.160.203 - - [24/Jun/2022:18:16:35 +0530] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko”

Log2

Jun 24 18:09:27 test sshd[2066210]: Accepted password for helpdeskmum from 178.30.160.203 port 20987 ssh2

[TUKARAM GAONKAR]

Hi Team ,

Please help

[Julián Morales]

Hi TUKARAM GAONKAR!,

Your rules are fine, you just forgot to add the definition of frequency, so your rule should look like this:

<group name="syslog,attacks,">
   <rule id="900805" level="12" timeframe= "500" frequency="2">

   <if_matched_sid>100100</if_matched_sid>
   <if_sid>5715</if_sid>
   <same_srcip />
   <description>Authentication success from Blacklisted IP</description>
   <mitre> <id>T1531</id> </mitre>        
 <group>pci_dss_10.2.7,pci_dss_10.2.5,pci_dss_8.1.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,hipaa_164.312.a.2.I,hipaa_164.312.a.2.II,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AC.2,nist_800_53_IA.4,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
 </rule>
 </group>

I have tested it in Wazuh-Logtest and it works great:

╰─# /var/ossec/bin/wazuh-logtest                                              130 ↵

Starting wazuh-logtest v4.3.5
Type one log per line

178.30.160.203 - - [24/Jun/2022:18:16:35 +0530] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko”

**Phase 1: Completed pre-decoding.
        full event: '178.30.160.203 - - [24/Jun/2022:18:16:35 +0530] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko”'

**Phase 2: Completed decoding.
        name: 'web-accesslog'
        id: '304'
        protocol: 'GET'
        srcip: '178.30.160.203'
        url: '/'

**Phase 3: Completed filtering (rules).
        id: '100100'
        level: '12'
        description: 'IP address found in AlienVault reputation database.'
        groups: '['syslog', 'attacks']'
        firedtimes: '1'
        mail: 'True'
**Alert to be generated.

Jun 24 18:09:27 test sshd[2066210]: Accepted password for helpdeskmum from 178.30.160.203 port 20987 ssh2

**Phase 1: Completed pre-decoding.
        full event: 'Jun 24 18:09:27 test sshd[2066210]: Accepted password for helpdeskmum from 178.30.160.203 port 20987 ssh2'
        timestamp: 'Jun 24 18:09:27'
        hostname: 'test'
        program_name: 'sshd'

**Phase 2: Completed decoding.
        name: 'sshd'
        parent: 'sshd'
        dstuser: 'helpdeskmum'
        srcip: '178.30.160.203'
        srcport: '20987'

**Phase 3: Completed filtering (rules).
        id: '900805'
        level: '12'
        description: 'Authentication success from Blacklisted IP'
        groups: '['syslog', 'attacks']'
        firedtimes: '1'
        frequency: '2'
        gdpr: '['IV_35.7.d', 'IV_32.2']'
        gpg13: '['4.13']'
        hipaa: '['164.312.b', '164.312.a.2.I', '164.312.a.2.II']'
        mail: 'True'
        mitre.id: '['T1531']'
        mitre.tactic: '['Impact']'
        mitre.technique: '['Account Access Removal']'
        nist_800_53: '['AU.14', 'AC.7', 'AC.2', 'IA.4']'
        pci_dss: '['10.2.7', '10.2.5', '8.1.2']'
        tsc: '['CC6.8', 'CC7.2', 'CC7.3']'
**Alert to be generated.

Regards,
Julian

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAKK7VjogCwu%3DDawRQ7KaJ%2BRbqzNRgpd4APux6ZuJgE6zO7Gkjg%40mail.gmail.com.

[TUKARAM GAONKAR]

Hi Julián,

Thanks for your reply. But I don't understand why you have used frequency=2 in this rule.

[TUKARAM GAONKAR]

Hi Julian,

I have tested from logtest.But it is not working for me . When I am copying 2nd logs after 1st logs it just showing rule id 5715.

[Julián Morales]

Hi TUKARAM GAONKAR,

There is a bug here. The combination of the options if_matched_sid, if_sid and same_XXX do not generate a consistent behavior. Today I am going to investigate this and create the issue.
What version of wazuh are you using?

[TUKARAM GAONKAR]

Hi Julián,

Is there any alternative or workaround for this. I am using wazuh 4.3.1 

On Tue, 28 Jun, 2022, 7:46 pm TUKARAM GAONKAR, <tukaramga...@gmail.com> wrote:

Hi Julian,

I am using wazuh version 4.3.1.

[Julián Morales]

Hi TUKARAM GAONKAR!,

I can't replicate the behavior of your environment, I think we have some different configuration. Could you confirm me that you have the list added in the ossec.conf, it should look like this:

  <ruleset>
    <!-- Default ruleset -->
      ....
    <!-- User-defined ruleset -->
    <decoder_dir>etc/decoders</decoder_dir>
    <rule_dir>etc/rules</rule_dir>
    ....
    <list>etc/lists/blacklist_alienvault_windows_test</list>
  </ruleset>

Could you please share with me the Wazuh-Logtest output of the following log:
```

178.30.160.203 - - [24/Jun/2022:18:16:35 +0530] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko”

```

Regards,
Julian

[TUKARAM GAONKAR]

HI Julián,

Please find attached doc of ossec.conf and logtest output.

Everything is present.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/08356b30-01d1-41a7-8eb3-97a02fa646acn%40googlegroups.com.

[TUKARAM GAONKAR]

Hi Julián,

I have created another rule with the same if_matched_sid, if_sid and same_XXX triggering on logtest but it won't trigger on live when I am pushed and create a live scenario. 

[TUKARAM GAONKAR]

Hi Julián,

Any update

[Julián Morales]

Hi TUKARAM GAONKAR,

I can't replicate your problem, I installed the same version of wazuh you tell me. then I added the custom rules in /var/ossec/etc/local_rules.xml:

[root@wazuh-server ~]cat  /var/ossec/etc/rules/local_rules.xml
<!-- Local rules -->

<!-- Modify it at your will. -->
<!-- Copyright (C) 2015, Wazuh Inc. -->

<!-- Example -->
<group name="local,syslog,sshd,">

  <!--
  Dec 10 01:02:02 host sshd[1234]: Failed none for root from 1.1.1.1 port 1066 ssh2
  -->
  <rule id="100001" level="5">
    <if_sid>5716</if_sid>
    <srcip>1.1.1.1</srcip>
    <description>sshd: authentication failed from IP 1.1.1.1.</description>
    <group>authentication_failed,pci_dss_10.2.4,pci_dss_10.2.5,</group>
  </rule>

</group>

<group name="syslog,attacks,">
<rule id="100100" level="12">
    <if_group>web|attack|attacks</if_group>
    <list field="srcip" lookup="address_match_key">etc/lists/blacklist_alienvault_windows_test</list>
    <description>IP address found in AlienVault reputation database.</description>
</rule>
</group>

<group name="syslog,attacks,">
   <rule id="900805" level="12" timeframe= "500" frequency="2">

   <if_matched_sid>100100</if_matched_sid>
   <if_sid>5715</if_sid>
   <same_srcip />
   <description>Authentication success from Blacklisted IP</description>
   <mitre> <id>T1531</id> </mitre>        
 <group>pci_dss_10.2.7,pci_dss_10.2.5,pci_dss_8.1.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,hipaa_164.312.a.2.I,hipaa_164.312.a.2.II,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AC.2,nist_800_53_IA.4,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
 </rule>
 </group>

Then I created the list file, adding the ip of the 100100 rule:

[root@wazuh-server ~]# cat /var/ossec/etc/lists/blacklist_alienvault_windows_test
178.30.160.203:

I have also added it to ossec.conf:

  <ruleset>
    <!-- Default ruleset -->

    <decoder_dir>ruleset/decoders</decoder_dir>
    <rule_dir>ruleset/rules</rule_dir>
    <rule_exclude>0215-policy_rules.xml</rule_exclude>
    <list>etc/lists/audit-keys</list>
    <list>etc/lists/amazon/aws-eventnames</list>
    <list>etc/lists/security-eventchannel</list>
    <list>etc/lists/blacklist_alienvault_windows_test</list>

    <!-- User-defined ruleset -->
    <decoder_dir>etc/decoders</decoder_dir>
    <rule_dir>etc/rules</rule_dir>

  </ruleset>

Finally, when tested with Wazuh-Logtest:

[root@wazuh-server ~]# /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.3

Type one log per line

178.30.160.203 - - [24/Jun/2022:18:16:35 +0530] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko”

**Phase 1: Completed pre-decoding.
full event: '178.30.160.203 - - [24/Jun/2022:18:16:35 +0530] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko”'

**Phase 2: Completed decoding.
name: 'web-accesslog'
id: '304'
protocol: 'GET'
srcip: '178.30.160.203'
url: '/'

**Phase 3: Completed filtering (rules).
id: '100100'
level: '12'
description: 'IP address found in AlienVault reputation database.'
groups: '['syslog', 'attacks']'
firedtimes: '1'
mail: 'True'
**Alert to be generated.

Jun 24 18:09:27 test sshd[2066210]: Accepted password for helpdeskmum from 178.30.160.203 port 20987 ssh2

**Phase 1: Completed pre-decoding.

**Alert to be generated.

Could you share with me the custom rules file and also the logtest output after injecting both logs?

[TUKARAM GAONKAR]

Hi Julian,

I found that rule working perfectly in wazuh-logtest . But after pushing it in the live environment it started showing inconsistent behaviour.

I found this same inconsistent behaviour for all my rules where I used if_sid and If_matched_sid with same_XXX.

Please guide me . 

[Julián Morales]

Good to know that this works for you in logtest. Could you tell me more about this inconsistent behavior?
I do not understand the problem in the document you share with me. What is the log sequence that triggers these rules? How should it behave?

[TUKARAM GAONKAR]

HI Julián ,

I have saved this ruleset and restarted wazuh manager . This ruleset is working properly in wazuh-logtest . But it is not working in a live environment . 

As per doc in last image i have shown one log sequence in my environment . Where rule 100100 works perfectly but after this  5503 rules come.

As per my correlation rule , rule id 900805 should be triggered . But it is not triggered . 

This behaviour is observed for other rules as well .Where i used if_match_sid and If_sid syntax . Those rules work perfectly in the wazuh-logtest but not in the actual environment.

Sometimes it triggers 1 or 2 times but is not consistent .

[Julián Morales]

In order to analyze this, could you share with me some alerts in json format that are not coincidental? This will help me to check the times between alerts, and check it against the timeframe. On the other hand, when you use if_match_sid and If_sid syntax, you also use CDB lists?
Remember to review the alerts and change/replace data that may be confidential (IPs, API Key, etc), for `---` or any other character.

Regards

[TUKARAM GAONKAR]

When I am using if_match_sid and If_sid  then I am not using a CDB list . I use cbd list either for if_sid or for if_matched_sid syntax . I will share json data one to one on your mail id .

[Julián Morales]

 this:Hi TUKARAM GAONKAR

Maybe I don't have your custom rules updated, but let's remember that a log can only trigger one alert, and an alert can only be triggered by one log. i.e. if one log matches rule 100100 and the other log matches alert 5503, but by correlation you want to trigger rule 900805, then rule 5503 must be a child of 100100.
Let me see if I understand correctly, you have the log starting with ``date=2022-07-06 time=19:19:11 devname=``` triggering the rule 100100 and the log starting with ``Jul 6 19:19:14 stlmisptest sshd``` triggering the rule 5503. And you want the second log to trigger by correlation the 900805. So then from the rules you shared with me earlier, rule 900805 should look something like this:

group name="syslog,attacks,">

<rule id="900805" level="12" timeframe= "500" frequency="2">

<if_matched_sid>100100</if_matched_sid>

<if_sid>5715, 5503</if_sid>

<same_srcip />

<description>Authentication success from Blacklisted IP</description>

<mitre> <id>T1531</id> </mitre>

<group>pci_dss_10.2.7,pci_dss_10.2.5,pci_dss_8.1.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,hipaa_164.312.a.2.I,hipaa_164.312.a.2.II,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AC.2,nist_800_53_IA.4,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>

</rule>

</group>

After adding the above rule, I have tested this in logtest and it works (triggering the 900805 rule).
If I understand correctly, you have the same behavior as I tell you in logtest. But after restarting the manager this behavior does not always work in the manager, am I right?

[TUKARAM GAONKAR]

Yes, you are correct. 

[TUKARAM GAONKAR]

Hi Julian,

Any update

[Julián Morales]

Hi TUKARAM GAONKAR,

I have tried to replicate this behavior, but without success.
Do you have a cluster and/or a load balancer
Could you share with me privately (as there may be sensitive information), a fragment of the alert file(/var/ossec/log/alerts/alerts.json) where you see that the behavior is not as expected?

Regards

[TUKARAM GAONKAR]

I have elk cluster with standalone wazuh manager . So what exactly you need in cluster.

[Julián Morales]

If you have a standalone wazuh manager, then you should not have any correlation problems.
At this point I would need just a fragment of the alert.json where I could see the erratic behavior, this will help me to see the order in which the events were processed.

[TUKARAM GAONKAR]

Hi Julian ,

I have sent alert.jason file in separate email.

On Tue, 12 Jul 2022 at 16:05, TUKARAM GAONKAR <tukaramga...@gmail.com> wrote:

Hi Julián,

Please find attached file

[Julián Morales]

Thanks TUKARAM GAONKAR!, I will analyze them and get back to you when I have an answer.

[Julián Morales]

Hi TUKARAM GAONKAR,

After analyzing the logs I did not find any failure in the expected behavior of wazuh.
I have also noticed that the logs of the rules 5503 are coming from the agent with id 24, while the rule 100100 is from the manager.
Note that if you want to correlate events from different agents or agent and amanger with frequency and timeframe, you have to enable global_frequency:
https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html#global-frequency
global_frequency specifies that the events of all agents will be contemplated when using the frequency and timeframe options. By default, only the events generated by the same agent will be taken into account to increase the frequency counter for a rule.

Regards,

Julian

[TUKARAM GAONKAR]

HI Julian ,

I have made changes as mentioned by you . This time rule is triggering  but it shows lots of inconsistency. Sometimes it triggered sometime not.

<group name="Auth_attack,">
   <rule id="900310" level="11" timeframe= "500" frequency="2">
   <if_matched_sid>100100</if_matched_sid>

   <same_srcip />
   <description>Authentication success from Blacklisted IP</description>
   <mitre> <id>T1531</id> </mitre>        
   <group>pci_dss_10.2.7,pci_dss_10.2.5,pci_dss_8.1.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,hipaa_164.312.a.2.I,hipaa_164.312.a.2.II,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AC.2,nist_800_53_IA.4,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
 </rule>

   <rule id="900810" level="12" timeframe= "120" >
   <if_matched_sid>31168</if_matched_sid>
   <if_sid>5503</if_sid>
   <same_srcip />
   <global_frequency />
   <description>Shellshock attack followed by authentication failure</description>

   <mitre> <id>T1531</id> </mitre>        
   <group>pci_dss_10.2.7,pci_dss_10.2.5,pci_dss_8.1.2,gpg13_4.13,gdpr_IV_35.7.d,gdpr_IV_32.2,hipaa_164.312.b,hipaa_164.312.a.2.I,hipaa_164.312.a.2.II,nist_800_53_AU.14,nist_800_53_AC.7,nist_800_53_AC.2,nist_800_53_IA.4,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,</group>
 </rule>
 </group>

I will share Json logs privately to you.

[TUKARAM GAONKAR]

Hi Julian ,

Please check and let me know if you have any questions.

On Tue, 26 Jul 2022 at 17:58, TUKARAM GAONKAR <tukaramga...@gmail.com> wrote:

Hi Julián,

I have recorded a video for your better understanding .

Please let me know if you have any questions.

On Mon, 18 Jul 2022 at 10:19, Julián Morales <julian....@wazuh.com> wrote:

HI TUKARAM,

Could you help me to understand the inconsistencies? I see that the behavior is as expected in the alerts.
The 57XX rules and the 100100 rule seem to be generated correctly. As for rule 900810, I don't know because the alerts of 31168 are not present.

Regards,

Julian

On Fri, Jul 15, 2022 at 4:50 AM TUKARAM GAONKAR <tukaramga...@gmail.com> wrote:

HI Julian ,

Here you see lots of inconsistency in behaviour for this rule.

[TUKARAM GAONKAR]

Hi Julian,

Please help..

[Julián Morales]

Hi TUKARAM, I need time to analyze it (the video), I will come with an answer as soon as possible.

Regards.

[TUKARAM GAONKAR]

Ok, no problem. 

[Julián Morales]

Sorry for the late reply TUKARAM.

I had to create a whole new environment to try to replicate this.
It is a bug, I have seen the video and it is not the expected behavior. Unfortunately I have not been able to replicate it in my environment with custom rules.
Could you share with me (privately) all your custom rules involved and the alert file with the alerts generated in your demo environment? This will help me to try to replicate the bug and create the issue on github.

Regards,

Julian

[TUKARAM GAONKAR]

Hi Julián,

I have shared a custom rule to you in separate mail.

[Julián Morales]

Thanks, I have received it.
I will try again to reproduce the bug with the data you share with me and leave any information in this thread.
Thanks!

[Julián Morales]

Hi TUKARAM,

If it is not too much trouble, could you please share with me the alerts.json that was generated in the video you shared with me? this will help me to refine your rules and test exactly the same case. thanks.

Regards

[TUKARAM GAONKAR]

HI Julian ,

I have shared alert.json output on private mail to you.

[Julián Morales]

Hi TUKARAM,

I have corroborated the bug, created the issue, simplified and removed any sensitive data. If you wish you could see the issue here:

Sometimes the correlation of rules with global frequency fails. #14526

Sorry for the delay, there were a lot of rules and logs to review. Any news regarding this will be added to the issue.
Thank you very much for reporting this and taking the time to share it, it is important for us to keep improving.

Regards,
Julian