Automatic daily reports are empty

[Mohamed ZAGHOUANI]

Hello wazuh community,

I have recently configured the daily reports sent by email by adding several report sections in the ossec.conf file. I also have configured SMTP settings and I am able to receive triggered rules notifications.

Otherwise, I just received one daily report for the first time after enabling the feature. Every day I check the log file and I find that the reports are empty but there are many corresponding alerts that are triggered during the day.

For every report, I get these lines in the ossec.log file:

 

2022/03/22 00:00:10 wazuh-monitord: INFO: Starting daily reporting for 'Successful Authentications'
2022/03/22 00:00:10 wazuh-monitord: ERROR: date or location not NULL or p is NULL
2022/03/22 00:00:15 wazuh-monitord: INFO: Report 'Successful Authentications' completed and zero alerts post-filter.

2022/03/22 00:00:15 wazuh-monitord: INFO: Report 'Successful Authentications' empty.

I also tested by using the wazuh-reportd tool and I get the same response.

Can anyone help me to resolve the problem ?

Regards,

Mohamed Zaghouani

[Christian Borla]

Hi Mohamed Zaghouani
I hope you are doing fine!

I would like to know which Wazuh version do you have?
I did some test in my local environment, and it works, the configuration is:

Into Manager side:

ossec.conf file:

  <global>
    .
    .
    .
    <email_notification>yes</email_notification>
    <smtp_server>localhost</smtp_server>
    <email_from>xxx...@gmail.com</email_from>
    <email_to>zzz...@gmail.com</email_to>
    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>
    .
  <global>

  <reports>
    <title>Daily report</title>
    <rule>100001</rule>                       <----- I use a custom rule to test it.
    <email_to>zzz...@gmail.com</email_to>
    <showlogs>yes</showlogs>
  </reports>


Then I defined a local rule /var/ossec/etc/rules/local_rules.xml

<group name="local,syslog,sshd,">

  <rule id="100001" level="5">
    <if_sid>5716</if_sid>
    <srcip>1.1.1.1</srcip>
    <description>sshd: authentication failed from IP 1.1.1.1.</description>
    <group>authentication_failed,pci_dss_10.2.4,pci_dss_10.2.5,</group>
  </rule>

</group>


And the example log that trigger that rule looks like following:   Dec 10 01:02:02 host sshd[1234]: Failed none for root from 1.1.1.1 port 1066 ssh2

Into agent side:

So I defined a test.txt file into Agent, and set following config into ossec.conf agent side.

  <localfile>
    <location>C:\Users\MyUser\test.txt</location>
    <log_format>syslog</log_format>
  </localfile>

Where I collect events from, I pasted 2 o 3 times the example log in text.txt file and save changes,

Added log to file:

Dec 10 01:02:02 host sshd[1234]: Failed none for root from 1.1.1.1 port 1066 ssh2

Agent collect it and sent to manager, the manager process it and trigger my custom rule. I forced a day change by system just to start the report, and the email contains a complete report.

You can use any event number that want, try to use any frequently event just for test, I created a custom rule to have more control of the test.
Could you share your Reports configuration?

I expend a lot of time setting my email account, but now it's working, I let you an example report picture.

Let me know if this information is useful to you!
Regards.

[Mohamed ZAGHOUANI]

Hi Christian,

Thank you for interacting with my case.

First, I use Wazuh 4.2.

Then, this is my global email configuration: (I still receive alerts notifications)

    <email_notification>yes</email_notification>
    <smtp_server>localhost</smtp_server>
    <email_from>xxx...@gmail.com</email_from>

    <email_to>yyy...@gmail.com</email_to>

    <email_maxperhour>12</email_maxperhour>
    <email_log_source>alerts.log</email_log_source>

And this is the reports' configurations: (I tried the same example as you did using a custom rule that I forced to trigger and for which I received an alerte notification by email, I also changed the system time and I got the same log lines that I shared in the previous post while generating the reports)

<reports>
    <title>Daily report</title>

    <rule>100002</rule>
    <email_to>yyy...@gmail.com</email_to>
    <showlogs>yes</showlogs>
  </reports>

 <reports>
    <title>Successful Authentications</title>
    <group>authentication_success,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Failed Authentications</title>
    <group>authentication_failed,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Multiple Failed Authentications</title>
    <group>authentication_failures,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Policy Violations</title>
    <group>policy_violation,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>VirusTotal Scans</title>
    <group>virustotal,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Vulnerability Scans</title>
    <group>vulnerability-detector,</group>
    <location>agent-zzz</location>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Regular File Integrity Checks</title>
    <group>syscheck_file,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Windows Registries Integrity Checks</title>
    <group>syscheck_registry,</group>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

  <reports>
    <title>Anomaly and Malware Detection</title>
    <group>rootcheck,</group>
    <level>7</level>
    <email_to>yyy...@gmail.com</email_to>
  </reports>

All those configured reports are empty.

Finally, I want to mention that this error appears for every report creation process:

"wazuh-monitord: ERROR: date or location not NULL or p is NULL", that could be a parsing error while reading the alerts log file.

Regards,

[Christian Borla]

Hi Mohamed Zaghouani
I hope you are doing fine!

Could you try with just one reports field into ossec.conf?
Try to create a unique report, triggered by a known rule, for example the custom rule 100002 that you already have. To have full control of the test.

 <reports>
    <title>Daily report</title>
    <rule>100002</rule>
    <email_to>yyy...@gmail.com</email_to>
    <showlogs>yes</showlogs>
 </reports>

 Trigger some times the rule and share the alert generated, it will be into /var/ossec/logs/alerts/alerts.json.
 After found some events 100002, force the system time to generate the report.

 I would check:
 A json alert
 The report as you received into email.
 and the wazuh-monitord message into ossec.log file.  

 As you mentioned, the error "wazuh-monitord: ERROR: date or location not NULL or p is NULL" appears into "Extract information from the event" process, maybe it try to check if the alert is from an active-response.
 Please do a test with just one report configured, and let me know the results.

Regards

[Mohamed ZAGHOUANI]

Hi Christian,

As you requested, I kept only one report section in ossec.conf file:

<reports>
    <title>Daily report</title>

    <rule>100002</rule>
    <email_to>yyy...@gmail.com</email_to>
    <showlogs>yes</showlogs>
  </reports>

Then, I triggered the rule 100002 one time, and this is the alert log from alerts.log:

** Alert 1648162630.68261497: mail  - policy_violation,local,login_time,authentication_failed,
2022 Mar 24 23:57:10 (centos-agent) any->/var/log/secure
Rule: 100002 (level 9) -> 'Failed login during non-business hours'
Src IP: x.x.x.x
Mar 24 04:58:10 hostname sshd[22645]: Failed password for invalid user dfddfd from x.x.x.x port x ssh2

Additionally, this is the email notification that I received (I am forcing the email alert for this rule):

Wazuh Notification.
2022 Mar 24 23:57:10

Received From: (centos-agent) any->/var/log/secure
Rule: 100002 fired (level 9) -> "Failed login during non-business hours"
Src IP: x.x.x.x
Portion of the log(s):

Mar 24 04:58:10 hostname sshd[22645]: Failed password for invalid user dfddfd from x.x.x.x port x ssh2

 --END OF NOTIFICATION

Then, I triggered the same rule for a second time, and this is the alert log from alerts.log:

** Alert 1648162684.68310353: mail  - policy_violation,local,login_time,authentication_failed,
2022 Mar 24 23:58:04 (centos-agent) any->/var/log/secure
Rule: 100002 (level 9) -> 'Failed login during non-business hours'
Src IP: x.x.x.x
Src Port: x
User: xyz
Mar 24 04:59:04 hostname sshd[22649]: Failed password for xyz from x.x.x.x port x ssh2

Additionally, this is the email notification that I received:

Wazuh Notification.
2022 Mar 24 23:58:04

Received From: (centos-agent) any->/var/log/secure
Rule: 100002 fired (level 9) -> "Failed login during non-business hours"
Src IP: x.x.x.x
User: xyz
Portion of the log(s):

Mar 24 04:59:04 hostname sshd[22649]: Failed password for xyz from x.x.x.x port x ssh2




 --END OF NOTIFICATION

Finally, these are the wazuh-monitord logs from ossec.log file (I still can't receive report by mail because it is empty):

2022/03/25 00:00:10 wazuh-monitord: INFO: Starting daily reporting for 'Daily report'
2022/03/25 00:00:10 wazuh-monitord: ERROR: date or location not NULL or p is NULL
2022/03/25 00:00:15 wazuh-monitord: INFO: Report 'Daily report' completed and zero alerts post-filter.
2022/03/25 00:00:15 wazuh-monitord: INFO: Report 'Daily report' empty.

Before I finish, I want to mention that when I configured the report section in ossec.conf for the first time when I started using wazuh, I received a report by mail the next day, since that I haven't received any other reports (Unfortunately, there isn't any way to check for old logs).

Christian, I hope that you get all the information that you have requested.

Regards,

[Christian Borla]

Hi  Mohamed ZAGHOUANI.
I hope you are doing well!
Sorry for the delay.
Could you share your custom rule ? rule id 100002, I would like to recreate full environment to test it.
And if you found any alert into  alerts.json file it would be grate, because the process use that file to look for alerts, but it can reproduce it with the rule.
Regrdas.

[Mohamed ZAGHOUANI]

Hi Chrisitian,

There's no problem, I know that you were taking care of other cases.

As you requested, this is the rule id 1000002:

<rule id="100002" level="9">
    <if_group>authentication_failed</if_group>
    <time>06:00 pm-8:30 am</time>
    <description>Failed login during non-business hours</description>
    <group>login_time,authentication_failed,</group>
    <options>alert_by_email</options>
  </rule>

I again have tested the trigger of the rule and this is the alert message recorded in alerts.json file:

{"timestamp":"2022-03-28T23:55:40.490+0100","rule":{"level":9,"description":"Failed login during non-business hours","id":"100002","firedtimes":2,"mail":true,"groups":["policy_violation","local","login_time","authentication_failed"]},"agent":{"id":"001","name":"centos-agent","ip":"x.x.x.x"},"manager":{"name":"wazuh"},"id":"1648508140.81484893","full_log":"Mar 28 04:14:29 hostname sshd[31646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=y.y.y.y","predecoder":{"program_name":"sshd","timestamp":"Mar 28 04:14:29","hostname":"hostname"},"decoder":{"name":"pam"},"data":{"srcip":"y.y.y.y","uid":"0","euid":"0","tty":"ssh"},"location":"/var/log/secure"

{"timestamp":"2022-03-28T23:55:44.454+0100","rule":{"level":9,"description":"Failed login during non-business hours","id":"100002","firedtimes":3,"mail":true,"groups":["policy_violation","local","login_time","authentication_failed"]},"agent":{"id":"001","name":"centos-agent","ip":"x.x.x.x"},"manager":{"name":"wazuh"},"id":"1648508144.81532656","full_log":"Mar 28 04:14:31 hostname sshd[31646]: Failed password for invalid user dfdfdfdf from y.y.y.y port x ssh2","predecoder":{"program_name":"sshd","timestamp":"Mar 28 04:14:31","hostname":"hostname"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"y.y.y.y","srcuser":"dfdfdfdf"},"location":"/var/log/secure"}

Finally, the report is empty as usual:

2022/03/29 00:00:10 wazuh-monitord: INFO: Starting new log after rotation.
2022/03/29 00:00:10 wazuh-monitord: INFO: Starting daily reporting for 'Daily report'
2022/03/29 00:00:10 wazuh-monitord: ERROR: date or location not NULL or p is NULL
2022/03/29 00:00:15 wazuh-monitord: INFO: Report 'Daily report' completed and zero alerts post-filter.
2022/03/29 00:00:15 wazuh-monitord: INFO: Report 'Daily report' empty.

Regards.

[Christian Borla]

Hi Mohamed Zaghouani
I hope you are doing fine!

I simulated same scenario as yours. I will describe the steps and results, please let me know if you see any different.

1. Added a custom rule.

<rule id="100002" level="9">
  <if_group>authentication_failed</if_group>
  <time>06:00 pm-8:30 am</time>
  <description>Failed login during non-business hours</description>
  <group>login_time,authentication_failed,</group>
  <options>alert_by_email</options>
</rule>

2. Added a report configuration

<reports>
  <title>custom-rule</title>
  <rule>100002</rule>
  <email_to>b...@xgmail.com</email_to>
  <showlogs>yes</showlogs>
</reports>

3. I created a test file in my agent, located into (the error mention location different than empty) \Users\asus-user\test.txt and pasted following logs.

Mar 28 04:14:29 hostname sshd[31646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=y.y.y.y

Mar 28 09:14:31 hostname sshd[31646]: Failed password for invalid user dfdfdfdf from y.y.y.y port x ssh2

Mar 28 04:14:29 hostname sshd[31646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=y.y.y.y

Mar 28 09:14:31 hostname sshd[31646]: Failed password for invalid user dfdfdfdf from x.x.x.x port x ssh2

Mar 28 04:14:29 hostname sshd[31646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=y.y.y.y

Mar 28 09:14:31 hostname sshd[31646]: Failed password for invalid user dfdfdfdf from X.x.x.X port x ssh2

Mar 28 04:14:29 hostname sshd[31646]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=y.y.y.y

Mar 28 09:14:31 hostname sshd[31646]: Failed password for invalid user dfdfdfdf from x.x.x.X port x ssh2

4. I forced the manager to create the report, and message logs are:
2022/03/28 18:39:50 wazuh-monitord[480775] generate_reports.c:55 at generate_reports(): INFO: Starting daily reporting for 'custom-rule'
2022/03/28 18:39:55 wazuh-monitord[480775] report_op.c:572 at os_ReportdStart(): INFO: Report 'custom-rule' completed. Creating output...

5. I receive the email as picture show.

I have some doubts about which alert broke your report process, because it process all the previous log alerts.
Could enable log debug level 2 and run the test again?
To enable the debug go to /var/ossec/etc/local_internal_options.conf add following config, and restart the manager.

# Auth daemon debug (server)
monitord.debug=2

Please let me know ossec.log results.
I hope it will useful to solve it.
Regards.

[Mohamed ZAGHOUANI]

Hi Chrisitian

I am fine thank you.

I activiated the debug level 2 for monitord as you mentionned, and did the test again, here is the output of the ossec.log file:

2022/03/30 00:00:10 wazuh-monitord[102359] generate_reports.c:55 at generate_reports(): INFO: Starting daily reporting for 'Daily report'
2022/03/30 00:00:10 wazuh-monitord[102359] read-alert.c:211 at GetAlertData(): ERROR: date or location not NULL or p is NULL
2022/03/30 00:00:15 wazuh-monitord[102359] report_op.c:565 at os_ReportdStart(): INFO: Report 'Daily report' completed and zero alerts post-filter.
2022/03/30 00:00:15 wazuh-monitord[102359] generate_reports.c:75 at generate_reports(): INFO: Report 'Daily report' empty.

I think that the error from the GetAlertData() function, which is called by another function called Read_FileMon(), is returning null value (no data). As a result, the function os_ReportdStart() which calls Read_FileMon() will break from the 'while' section (line 461 in report_op.c) before ever processing the filters and retrieve the fields from the events data.

So the problem is extracting the event data from the alert file.

Also, when i tested using the reportd tool and using alerts.json file as input, I didn't get that error otherwise there's no output (but as I have understood from the issue 8282,

reportd tool is no longer working since version 4.14):

cat logs/alerts/alerts.json | bin/wazuh-reportd -f rule 5715 -d 2>&1

2022/03/29 11:49:10 wazuh-reportd[112143] debug_op.c:70 at _log(): DEBUG: Logging module auto-initialized
2022/03/29 11:49:10 wazuh-reportd[112143] report.c:162 at main(): DEBUG: Wazuh home directory: /var/ossec
2022/03/29 11:49:10 wazuh-reportd[112143] report.c:192 at main(): DEBUG: Chrooted to directory: /var/ossec, using user: ossec
2022/03/29 11:49:10 wazuh-reportd[112143] report.c:205 at main(): INFO: Started (pid: 112143).
2022/03/29 11:49:15 wazuh-reportd[112143] report_op.c:563 at os_ReportdStart(): INFO: Report completed and zero alerts post-filter.

My question is , is there any way to force the monitord process to use alerts.json file to retrieve alerts?

Regards.

[Christian Borla]

Hi Mohamed Zaghouani

I think i found something, take a look to this issue https://github.com/wazuh/wazuh/issues/5872,
You can try pointing to alert.json source, change the ossec.conf configuration:

<email_log_source>alerts.log</email_log_source>

to

<email_log_source>alerts.json</email_log_source>

Looks like this change fix the issue, also they say..

In addition, there is a problem that is being solved with the email_alert_level field. If the error persists after changing the alerts to .json format, try removing that line from the configuration. For more information, you can check out this thread: #5758

Please let me know if this works.
Regards!

[Mohamed ZAGHOUANI]

Hi Christian,

Yes, changing the <email_log_source> option from alerts.log to alerts.json fixed the issue and I am able now to receive the daily report by email.

Thank you Chrisitian for your support and your patience, you have done a great job.

I will certainly come back for any other issue.

Regards,

Mohamed Zaghouani

[Christian Borla]

Hi Mohamed ZAGHOUANI
Great!!!  Thank you for your patience too.
Regards!