wazuh Agent operational Alerts

[John Carry]

Dear Team,

We are trying to prioritize wazuh agent few alerts to level 12 category, but unfortunately we are observing just wazuh agent start and stop alerts are triggered  but no alert triggered for when wazuh agent is removed or a new agent is created.

Please let me know why we are not getting alert triggered for below rules:

1) We uninstalled wazuh agent and it should trigger below rule but unfortunetaly it didn't:

  <rule id="505" level="3">
    <if_sid>500</if_sid>
    <match>Agent removed</match>
    <description>Ossec agent removed.</description>
    <mitre>
      <id>T1562.001</id>
    </mitre>
    <group>pci_dss_10.6.1,pci_dss_10.2.6,gpg13_10.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.6,nist_800_53_AU.14,nist_800_53_AU.5,tsc_CC7.2,tsc_CC7.3,tsc_CC6.8,</group>
  </rule>

2)  We installed wazuh agent and it should trigger below rule but unfortunately it didn't:

  <rule id="501" level="3">
    <if_sid>500</if_sid>
    <if_fts />
    <match>Agent started</match>
    <description>New ossec agent connected.</description>
    <group>pci_dss_10.6.1,gpg13_10.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.6,tsc_CC7.2,tsc_CC7.3,</group>
  </rule>

[Christian Borla]

Hi John!
I hope you are doing fine!
These rules are from 0015-ossec_rules.xml file, those rules looks good.
I would like to know how do you add or removed an agent? which method do you use?

One of the points to take into account is the log_alert_level, in  /var/ossec/etc/ossec.conf, if the alert level does not exceed this value, no alert will be generated.
Did you check it?

  <alerts>
    <log_alert_level>1</log_alert_level>
    <email_alert_level>12</email_alert_level>
  </alerts>

Did you try increasing the level of rules 501 and 505?
I will try to reprodecie the casa en come back as soon as possible.
Regards.

[John Carry]

Let me answer your Queries one by one:

ossec.conf: <log_alert_level>

Method to install and uninstall the wazuh agent:

I simply follow below offical docu to install and uninstall wazuh agent.

https://documentation.wazuh.com/4.3/installation-guide/wazuh-agent/wazuh-agent-package-linux.html

Rules File:

Indeed the mentioned rules are from same file you mentioned:

Please help me out as this is weird unexpected issue.

[Christian Borla]

Hi!
Yes! I found something here, regarding rule 501 looks like the <if_fts> option it's not working.
I found following alert in my environment.

{"timestamp":"2023-05-10T08:15:36.688-0300","rule":{"level":3,"description":"Ossec agent started.","id":"503","firedtimes":1,"mail":false,"groups":["ossec"],"pci_dss":["10.6.1","10.2.6"],"gpg13":["10.1"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14","AU.5"],"tsc":["CC7.2","CC7.3","CC6.8"]},"agent":{"id":"001","name":"DESKTOP","ip":"2803:9800:9882:1111:5E87:5DE6:1111:BBBB"},"manager":{"name":"VBox"},"id":"16837173.33015","full_log":"ossec: Agent started: 'DESKTOP-U8OHD3A->any'.","decoder":{"parent":"ossec","name":"ossec"},"data":{"extra_data":"DESKTOP->any"},"location":"wazuh-agent"}

Where the log that trigger it is 

ossec: Agent started: 'DESKTOP-U8OHD3A->any

Testing it by wazuh-logtest it's working, firts time triggers rule 501 and second test trigger rule 503.

/var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.4.0
Type one log per line

ossec: Agent started: 'DESKTOP-U8OHD3A->any

**Phase 1: Completed pre-decoding.
full event: 'ossec: Agent started: 'DESKTOP-U8OHD3A->any'

**Phase 2: Completed decoding.
name: 'ossec'
parent: 'ossec'

**Phase 3: Completed filtering (rules).
id: '501'
level: '3'
description: 'New ossec agent connected.'
groups: '['ossec']'
firedtimes: '1'
gdpr: '['IV_35.7.d']'
gpg13: '['10.1']'
hipaa: '['164.312.b']'
mail: 'False'
nist_800_53: '['AU.6']'
pci_dss: '['10.6.1']'
tsc: '['CC7.2', 'CC7.3']'
**Alert to be generated.                   <----------------------------- Alert

ossec: Agent started: 'DESKTOP-U8OHD3A->any

**Phase 1: Completed pre-decoding.
full event: 'ossec: Agent started: 'DESKTOP-U8OHD3A->any'

**Phase 2: Completed decoding.
name: 'ossec'
parent: 'ossec'

**Phase 3: Completed filtering (rules).
id: '503'
level: '3'
description: 'Ossec agent started.'
groups: '['ossec']'
firedtimes: '1'
gdpr: '['IV_35.7.d']'
gpg13: '['10.1']'
hipaa: '['164.312.b']'
mail: 'False'
nist_800_53: '['AU.6', 'AU.14', 'AU.5']'
pci_dss: '['10.6.1', '10.2.6']'
tsc: '['CC7.2', 'CC7.3', 'CC6.8']'
**Alert to be generated.                          <----------------------------- Alert


Rules 501 and 503 have the same match, they only differ in the <if_fts /> condition. So it should triggers 501 at first time.

  <rule id="501" level="3">
    <if_sid>500</if_sid>
    <if_fts />

    <options>alert_by_email</options>

    <match>Agent started</match>
    <description>New ossec agent connected.</description>
    <group>pci_dss_10.6.1,gpg13_10.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.6,tsc_CC7.2,tsc_CC7.3,</group>
  </rule>

  <rule id="503" level="3">
    <if_sid>500</if_sid>
    <options>alert_by_email</options>
    <match>Agent started</match>
    <description>Ossec agent started.</description>
    <group>pci_dss_10.6.1,pci_dss_10.2.6,gpg13_10.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.6,nist_800_53_AU.14,nist_800_53_AU.5,tsc_CC7.2,tsc_CC7.3,tsc_CC6.8,</group>
  </rule>

I will try to look for more information about the rule 505, In another hand, I thik if you unistall the agent without removing it from the manager, only stop alert will trigger, I will try to found a way but I think rule 505 is related this.

Regards.

[Christian Borla]

Hi John.

I found how it works, the events that triggers mentioned rules are generated by monitord, monitord is a Wazuh module in charge of check different functionalities, one of them is Agent status, following are the possible messages.

/* OSSEC alert messages */
"ossec: Manager started."
"ossec: Agent started: '%s->%s'."
"ossec: Agent stopped: '%s->%s'."
"ossec: Agent disconnected: '%s'."
"ossec: Agent removed: '%s'."

So rules from 500 to 505 were triggered only if one of these events fires.
link monitord configuration. 

To trigger the rule 505 Ossec agent removed, you should set the monitord.delete_old_agents option,  which represents the minutes before deleteing an disconnected agent.
Let me knof if this helps.
Regards.

[John Carry]

Let me question and share concerns in parallel to your message:

Please find my answers in Italic Red :

Your Last Message:

Hi!
Yes! I found something here, regarding rule 501 looks like the <if_fts> option it's not working.

1) Customer: So should I remove the <if_fts> part from rule 501 ?

I found following alert in my environment.

{"timestamp":"2023-05-10T08:15:36.688-0300","rule":{"level":3,"description":"Ossec agent started.","id":"503","firedtimes":1,"mail":false,"groups":["ossec"],"pci_dss":["10.6.1","10.2.6"],"gpg13":["10.1"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14","AU.5"],"tsc":["CC7.2","CC7.3","CC6.8"]},"agent":{"id":"001","name":"DESKTOP","ip":"2803:9800:9882:1111:5E87:5DE6:1111:BBBB"},"manager":{"name":"VBox"},"id":"16837173.33015","full_log":"ossec: Agent started: 'DESKTOP-U8OHD3A->any'.","decoder":{"parent":"ossec","name":"ossec"},"data":{"extra_data":"DESKTOP->any"},"location":"wazuh-agent"}

2) Customer:   Please let me know where I can find the agent operational logs like the one shared above i-e Agent disconnected, started, stopped or removed... As I am unable to find the logs (Against which rule 501 -505 will match) , just to bring into your knowledge I can't find the operational  i-e Agent disconnected, started, stopped or removed logs inside the ossec.log file at the monitored server (Where wazuh agent is installed), so please let me know where I can find such logs? are they logged on the wazuh server end ? if yes then where ?

3) Customer: If rule 501 and 503 are the same then what's the purpose to create two rules for same behavior ? like in both cases  it will match "Agent started" which means it will trigger everytime an existing agent 's service is started for any reason.. that means these rules will  not distinguish between the existing agent or when a new agent in installed...

  <rule id="501" level="3">
    <if_sid>500</if_sid>
    <if_fts />

    <options>alert_by_email</options>

    <match>Agent started</match>
    <description>New ossec agent connected.</description>
    <group>pci_dss_10.6.1,gpg13_10.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.6,tsc_CC7.2,tsc_CC7.3,</group>
  </rule>

  <rule id="503" level="3">
    <if_sid>500</if_sid>
    <options>alert_by_email</options>
    <match>Agent started</match>
    <description>Ossec agent started.</description>
    <group>pci_dss_10.6.1,pci_dss_10.2.6,gpg13_10.1,gdpr_IV_35.7.d,hipaa_164.312.b,nist_800_53_AU.6,nist_800_53_AU.14,nist_800_53_AU.5,tsc_CC7.2,tsc_CC7.3,tsc_CC6.8,</group>
  </rule>

I found how it works, the events that triggers mentioned rules are generated by monitord, monitord is a Wazuh module in charge of check different functionalities, one of them is Agent status, following are the possible messages.

/* OSSEC alert messages */
"ossec: Manager started."
"ossec: Agent started: '%s->%s'."
"ossec: Agent stopped: '%s->%s'."
"ossec: Agent disconnected: '%s'."
"ossec: Agent removed: '%s'."

4) Customer:   Please let me know where I can find the agent operational logs like the one shared above.

So rules from 500 to 505 were triggered only if one of these events fires.
link monitord configuration. 

To trigger the rule 505 Ossec agent removed, you should set the monitord.delete_old_agents option,  which represents the minutes before deleteing an disconnected agent.

 5) Customer: Could you please elaborate and explain in more detail about the monitord.delete_old_agents because when I enabled it with the value 5 and then proceed to uninstalled the agent on the server it trigger no rule for rule it 505 instead it removed all the disconnected agents from wazuh server agent dashboard (Alarming and create panic at our end) ..  our objective here is just to trigger an alert when ever wazuh agent is uninstalled ore removed from the user or the monitored system end (Where agent is installed).

Please note that after setting the monitord.delete_old_agents to 5 resulted the Disconnected agent count from 50 to zero , please ignore the latest count if 1 as it is the current status of my testing machine.

Please help us out.

Please respond to our 5 questions and concerns in-order.

[Christian Borla]

Hello John
I am sorry to hear that you were running these tests in a productive environment, and did not expect the deletion of disconnected agents. Maybe I misunderstood the initial message but the initial query was how to generate such alerts, how to make such rules trigger. Sorry if I wasn't clear, but the same documentation I passed you indicates that the monitord module deletes the agent if that functionality is configured. docu
I will now try to be as clear as possible in answering each query.

1) Customer: So should I remove the <if_fts> part from rule 501 ?

No, becuse is the unique difference between rule 501 and 503, the idea is 501 (New ossec agent connected) fires first time that monitord reports "Agent started", following messages should trigger rule 503 (Agent started), ever talking about same agent that, rule link, decoder link

2) Customer:   Please let me know where I can find the agent operational logs like the one shared above i-e Agent disconnected, started, stopped or removed... As I am unable to find the logs (Against which rule 501 -505 will match) , just to bring into your knowledge I can't find the operational  i-e Agent disconnected, started, stopped or removed logs inside the ossec.log file at the monitored server (Where wazuh agent is installed), so please let me know where I can find such logs? are they logged on the wazuh server end ? if yes then where ?

I think you are talking about following event

{"timestamp":"2023-05-10T08:15:36.688-0300","rule":{"level":3,"description":"Ossec agent started.","id":"503","firedtimes":1,"mail":false,"groups":["ossec"],"pci_dss":["10.6.1","10.2.6"],"gpg13":["10.1"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14","AU.5"],"tsc":["CC7.2","CC7.3","CC6.8"]},"agent":{"id":"001","name":"DESKTOP","ip":"2803:9800:9882:1111:5E87:5DE6:1111:BBBB"},"manager":{"name":"VBox"},"id":"16837173.33015","full_log":"ossec: Agent started: 'DESKTOP-U8OHD3A->any'.","decoder":{"parent":"ossec","name":"ossec"},"data":{"extra_data":"DESKTOP->any"},"location":"wazuh-agent"}

That is an alert, I found it in /var/ossec/logs/alert/alert.json in the manager side, that file is enabled by default and contains every alert in the system, is the result of procesing a log and match with a rule, it includes the name and id

"Ossec agent started.","id":"503

If you want see the logs in manager side, look for them in /var/ossec/logs/ossec.log also in manager side. there are every Wazuh internal log, you can increse the verbosity of that log enabling it in /var/ossec/etc/local_internal_options.conf file.

How it works, rules 501 to 505 were created to be used with monitord, monitord informs (into /var/ossec/logs/ossec.log) when an agent has just connected, or it's disconnect from the manager, also it has te capability to delete old agents using monitord.delete_old_agents option, the possible messages from remoted are internal messages, defined in the code:

/* OSSEC alert messages */
"ossec: Manager started."
"ossec: Agent started: '%s->%s'."
"ossec: Agent stopped: '%s->%s'."
"ossec: Agent disconnected: '%s'."
"ossec: Agent removed: '%s'."

for example from my alert shared above, the message that triggers my rule was the section in the full_log,  "ossec: Agent started: 'DESKTOP-U8OHD3A->any'" how my agent was already created it fires rule number 503 instead 501. 
link to code messages.

3) Customer: If rule 501 and 503 are the same then what's the purpose to create two rules for same behavior ? like in both cases  it will match "Agent started" which means it will trigger everytime an existing agent 's service is started for any reason.. that means these rules will  not distinguish between the existing agent or when a new agent in installed...

Same answer as first question.

4) Customer:   Please let me know where I can find the agent operational logs like the one shared above.

You can find them in  /var/ossec/logs/ossec.log.
Are difined in link to code messages.

5) Customer: Could you please elaborate and explain in more detail about the monitord.delete_old_agents because when I enabled it with the value 5 and then proceed to uninstalled the agent on the server it trigger no rule for rule it 505 instead it removed all the disconnected agents from wazuh server agent dashboard (Alarming and create panic at our end) ..  our objective here is just to trigger an alert when ever wazuh agent is uninstalled ore removed from the user or the monitored system end (Where agent is installed).

The monitord.delete_old_agents option represents the number of minutes before deleting a disconnected agent from the manager list, agents are kept installed on each node, they are just removed from the manager list, if an agent tries to connect to the manager again, and the autoenrollment functionality is enabled, it will do so. link

Regarding our objective here is just to trigger an alert when ever wazuh agent is uninstalled ore removed from the user or the monitored system end (Where agent is installed).
I think it's not possible know if anget is unistalled or removed from the user, only if is appear as a disconnected in the system, agent send the SHUTDOWN message or stop, when it is stopped, but if it is then uninstalled, there is no agent to collect that information from the node log and send it to the manager. 

Normal communication between agent an manager are, agent send a start message, then functional messages with collected events or keepalives, to know that is alive, and if agent has to stops send an stop message or shutdown (there you can see the agent as discconected in the dashboard), in case the machine shuts down without properly closing the agent, the shutdow message does not arrive, the dashboard will show it as connected for 10 minutes, after that time without receiving any message from that agent it will be switched to disconnected. 

In conclusion, rules from 501 to 505 are triggered by monitord messages. If you think we should improve this feature, you could always open an issue with your feature request in https://github.com/wazuh/wazuh and we will evaluate with the team how and when to implement it.
I hope this information is useful to you and your team, sorry again if I wasn't clear.
Regards.

[John Carry]

Thanks Borla for your detailed response, further could you please confirm weather these alert types i-e agent start, stop and disconnected based alerts are same for both windows and linux OS? Means will these same rules from 500-505 work for cross-OS platform?

[Christian Borla]

Hi John!
Yes, that is correct, these alerts are generated by the Wazuh Manager, from internal messages from it, it does not depend on the agent's operating system.
Regards.