Wazuh logging too few rules 60106 - Logon audit_success

[Cézar]

Hi guys,

I have been facing a problem in the few days that I am not sure how to solve, and would like a insight from everyone what could be.

A certain day in the past month wazuh decreased A LOT the occurrence of this rule. I used to have a thousand of these messages a day, now I have 30 in the past month.

I am not sure what broke internally in wazuh or if this is an agent configuration issue, or maybe even an infrastructure problem. Any leads how I can start debugging the cause?

What I remember I did during that time was to delete security-auditlog from the past few months at elasticsearch. But i think this has nothing to do with it.

Any help is welcome,

Yours faithfully,

Cézar

[Jesus Linares]

Hi Cézar,

Here are a few steps that could help to find the root cause of the decreased volume of alerts:

1. Review your ruleset

• Did you create new rules that could be matching the events that were previously matched by rule 60106?
• Did you update Wazuh in the past month? Maybe the ruleset changed in a new version. What version do you run?

2. Review your indexer

OK, assuming that the ruleset is OK, maybe the alert is generated but it is not indexed. This is usually an error with the indexer template:

• Review the logs of your alerts.json forwarder to the indexer: Filebeat/Logstash
• Review the logs of your indexer (Elasticsearch/Opendistro/OpenSearch/Wazuh-indexer)
• Check directly the alerts.json. If there are more alerts with id 60106 in this file than in your indexer, there is an indexing error

3. Review your agents

Rule 60106 is for Windows logon success and it is catching Windows events: 528, 540, 673, 4624, 4769. Review your agents:

• Is it possible that you changed the agent configuration and you are not collecting those event IDs? Review the ossec.conf of your agent and the agent.conf of your remote configuration in the manager.
• Also, you can enable "archives.json" in the manager and review all the events that you are receiving. Be careful if it is a production environment, it can fill the hard drive in minutes.

4. Review if the system is flooded

• Your agent can be flooded and discarding events. Review the ossec.log of your agent. Check out the documentation: https://documentation.wazuh.com/current/user-manual/capabilities/antiflooding.html?highlight=flooding
• The manager can be dropping events. Check out 2 metrics (https://documentation.wazuh.com/current/user-manual/reference/statistics-files/index.html):
  • remoted.discarded_count
  • analysisd.events_dropped

5. Review your Windows configuration

If your ruleset is OK, you don't have any indexing errors, your agents are configured to read the proper Windows events IDs and the system is not flooded, you should review your configuration. Are you generating these logs in your Windows servers?

Also, it is obvious but, do you have the same number of active agents/servers?

Happy troubleshooting!

Regards.

[Cézar]

Thank you very much for your answer!

I will go by parts what I have checked with your information.

1. Review your ruleset

I did not write new rules that use these windows message as a base. I have some custom rules for our switches though, and they are working fine.

What I did in the past few days was update wazuh from 4.2 to 4.3. However I have noticed that this problem started before this update.

2. Review your indexer

This might be related but I am not so sure. In the past month I had a problem with elasticsearch shards, it has gone past the limit of 1000 shards, so I had to delete some older index, and I am not sure if I did something to rebuild the indexes that might have break something internally.

I have checked alerts.json and there are no alerts of this month regarding this rule, what I did was:

grep '"id":"60106"' alerts.json

And there was no alert.

3. Review your agents

I have some custom agent.conf files for groups of agents, I have a question here. This is my agent.conf file for the default group:

  <agent_config>
    <!-- Shared agent configuration here -->
          DELETE /agents/001/group/default
    <localfile>
      <location>Security</location>
      <log_format>eventchannel</log_format>
      <query>Event/System[EventID!=4673|4674]</query>
    </localfile>
  </agent_config>


Is this fine? I mean, can I write only the <localfile> ? Or do I have to replicate every config for the agent, for example I have to write <syscheck>, <sca>, <rootcheck>, etc.

I have also turned on logall and checked the archives.json with:

grep '"id":"60106"' archives.json

 

And it did not show any results as well.

4. Review if the system is flooded

I have checked both files and they show:

dicarded_count='0'

events_dropped='0'

5. Review your Windows configuration

I have checked the windows event messaged at the windows servers and the event is being generated in there

Also, it is obvious but, do you have the same number of active agents/servers?

If by this you mean the agent is connected to the manager, it does show in the manager the agent I am debugging. I am looking at a production environment, so there is hundred of agents connected to the manager, and no logon messaged whatsoever.

Again, I would like to thank you for your patience and help!

Regards,

Cézar

[Jesus Linares]

Hi Cézar,

> Is this fine? I mean, can I write only the <localfile> ? Or do I have to replicate every config for the agent, for example I have to write <syscheck>, <sca>, <rootcheck>, etc.

It is fine.

> dicarded_count='0'

> events_dropped='0'

OK, so your manager is in a good state.

> grep '"id":"60106"' alerts.json
> And there was no alert.

OK, so we don't have to review the indexer (yet)

> grep '"id":"60106"' archives.json

> And it did not show any results as well.

Well, this is not the best search. You should search by data present in the event instead of the rule. Check out other Windows events in the archives.json and try to improve the search. Also, you can try to search by the event IDs: 528, 540, 673, 4624, 4769.

As an example, you could use these commands:

• cat /var/ossec/logs/archives/archives.json | grep '"agent":{"id":"<your_agent_id>"'
• cat /var/ossec/logs/archives/archives.json | grep '"agent":{"id":"<your_agent_id>"' | grep -P "528|540|673|4624|4769|login|logon|success"

Please, check the archives.json, and we continue from there.

Also, review the ossec.log of your agent.

Good luck!

[Cézar]

Hi Jesus,

I tried to check archives.json in real time while I logged into the agent, and the only thing of relevance that it logged in might be this message:

"rule":{"level":5,"description":"WSearch was unavailable to handle a notification event.","id":"60775"

"message":"\"The winlogon notification subscriber <WSearch> was unavailable to handle a notification event.\""}

This did go to the dashboard, but i think this has nothing to do with the issue here.

Also I have checked the windows event viewer and it showed in there the ID 4624 for windows logon:

I am not sure what to look at the ossec.log file, so I have attached it here.

Thanks again,

Cézar

[Jesus Linares]

Hi Cézar,

Sorry for the late reply. 

Well, if the event is in the Windows Event viewer and it is not in archives.json, we know that the error must be in the Wazuh agent.

Checking the ossec.log with "cat ossec.log| grep -i error", I see these kinds of errors:

• 2022/10/18 15:15:34 wazuh-agent: ERROR: Could not EvtSubscribe() for (Security) which returned (15001)

For some reason, the agent is not able to subscribe to the Security channel (where Windows stores the login logs). I think maybe it is related to the syntax that you use:

<localfile>

  <location>Security</location>

  <log_format>eventchannel</log_format> 

  <query>Event/System[EventID!=4673|4674]</query>

</localfile>

Could you try the following one?

<localfile>

  <location>Security</location> 

  <log_format>eventchannel</log_format>

  <query>Event/System[EventID != 4673 and EventID != 4674]</query>

</localfile>

Also, keep in mind that you are overwriting the default configuration: https://github.com/wazuh/wazuh/blob/master/src/win32/ossec.conf#L34 where we ignore some events because they can cause different flooding errors. So, I recommend adding your events to the default ones:

<localfile>
<location>Security</location>
<log_format>eventchannel</log_format>

<query>Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and
    EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and
    EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and
    EventID != 5152 and EventID != 5157 and
    EventID != 4673 and EventID != 4674]</query>
</localfile>

I hope it helps.

[Cézar]

Thank you, it worked. It was something related to the agent.conf. I removed these rules and everything did come back to normal. I guess it was a syntax problem(?), not sure though. Anyway, thanks for the help!

Regards,

Cézar