Apache logs

82 views
Skip to first unread message

Cyprien Chapelle

unread,
Feb 17, 2022, 4:04:12 AM2/17/22
to Wazuh mailing list
Hello,

I retrieve via a script the logs of WEB sites hosted on OVH. The logs are then sent to a .log file, which is referenced in the ossec.conf file.

When I run the script, I do get the logs in alerts.log.

However, the script is executed by contab at 3:00 in the morning and when I discover Wazuh at 9:00 am, I have no detected alerts. Are there any "sleep" settings or something?

Jonathan Martín Valera

unread,
Feb 17, 2022, 4:51:00 AM2/17/22
to Wazuh mailing list

Hi,

Answering to: Are there any “sleep” settings or something?

No, in principle all monitoring is done continuously, without interruptions, unless you stop the wazuh-agent.

Let’s see what could be happening in your case. You say that you have a script to write all the logs of your servers in a single log file (you centralize all the logs in that file) and that you have monitored that centralized log file with your wazuh-agent.

I imagine that to monitor the log you have applied a configuration in the ossec.conf as the following:

<localfile>
  <location>path_to_your_centralized log file</location>
  <log_format>syslog</log_format>
</localfile>

A note to this comment: When I run the script, I get the logs in alerts.log

When a new line is added to a file that has been monitored, then an event is generated and sent to the wazuh-manager. This event is processed using the ruleset (decoders + rules), and will generate an alert depending on whether it has been decoded and matched with a level 3 or higher rule (default threshold). These alerts generated by the wazuh-manager are stored in the /var/ossec/logs/alerts/alerts.log and /var/ossec/logs/alerts/alerts.json files.

By this I mean, that not all logs have to generate an alert, and consequently, that it appears in the alerts.* file.

Regarding: the script is run by crontab at 3:00 am and when I discover Wazuh at 9:00 am, I have no alerts detected

I recommend that you perform a set of debugging steps to find the possible reason.

  • First of all, I would check if the script has been executed correctly with crontab. To do this verify that in your centralized log file the corresponding lines have been added.

  • I would then check whether or not the logs that have been recorded in the centralized file generate alerts. To do this I would use the /var/ossec/bin/wazuh-logtest tool of the wazuh-manager and I would check if each inserted line should generate an alert or not.

  • Next, it would look if the alerts corresponding to said logs have been stored in the alerts.log or alerts.json file. To do this, you would use some kind of filter with grep.

If you don’t get the answer with these steps, then what I recommend is that you try to replicate your use case in a development environment, where you can schedule crontab for that time and observe if the wazuh-manager is receiving the events or not.

To see the events received by the wazuh-manager, you can enable logging with the <logall>yes</logall> option for the /var/ossec/logs/archives/archives.log file or <logall_json>yes</logall_json> for the /var/ossec/logs/archives/archives.json file and restart the wazuh-manager (systemctl restart wazuh-manager).

From that moment on, all events received by the wazuh-manager will be logged in that/those file(s) (archives.*). Check that the wazuh-manager is receiving the events.

Note: Remember to disable logging of these events (<logall>no</logall> and <logall_json>no</logall_json>) when you don’t need it, because it can fill up and become disk intensive in case the wazuh-manager receives many events.

Let me know the results obtained, or if you have any doubts about what I have told you.

Regards.

​

Cyprien Chapelle

unread,
Apr 5, 2022, 3:50:24 AM4/5/22
to Wazuh mailing list
Hello, Sorry for the very late reply, I have been busy with another project. I am now back Wazuh, with my problem.


- First of all, I would check if the script has been executed correctly with crontab. To do this verify that in your centralized log file the corresponding lines have been added.

Yes, it is verified. I find the corresponding logs.

- I would then check whether or not the logs that have been recorded in the centralized file generate alerts. To do this I would use the /var/ossec/bin/wazuh-logtest tool of the wazuh-manager and I would check if each inserted line should generate an alert or not.

Oui, cela a bien généré une alerte.

- Next, it would look if the alerts corresponding to said logs have been stored in the alerts.log or alerts.json file. To do this, you would use some kind of filter with grep.

Yes, the alerts are well stored in alerts.log but only when I manually run the script which adds the new logs in the files. I tried to configure the Crontabs at 9:00 for example, in order to be able to observe the archives.log file live, here is what I get :(with logall)
2022 Apr 05 07:30:02 wazuh-manager->/var/log/syslog Apr  5 07:30:01 wazuh-manager CRON[8289]: (root) CMD (/etc/recup-logs-web/script-sites-web1.sh)
2022 Apr 05 07:30:02 wazuh-manager->/var/log/syslog Apr  5 07:30:01 wazuh-manager CRON[8288]: (root) CMD (/etc/recup-logs-web/script-sites-web2.sh)
2022 Apr 05 07:30:04 wazuh-manager->/var/log/syslog Apr  5 07:30:02 wazuh-manager postfix/local[8316]: AAEC280E3B: to=<ro...@wazuh-manager.localdomain>, orig_to=<root>, relay=local, delay=1.2, delays=0.23/0/0/0.95, dsn=2.0.0, status=sent (delivered to mailbox)
2022 Apr 05 07:30:04 wazuh-manager->/var/log/syslog Apr  5 07:30:02 wazuh-manager postfix/qmgr[310]: AAEC280E3B: removed
2022 Apr 05 07:30:04 wazuh-manager->/var/log/mail.info Apr  5 07:30:02 wazuh-manager postfix/qmgr[310]: AAEC280E3B: removed
2022 Apr 05 07:30:04 wazuh-manager->/var/log/mail.info Apr  5 07:30:02 wazuh-manager postfix/local[8316]: AAEC280E3B: to=<ro...@wazuh-manager.localdomain>, orig_to=<root>, relay=local, delay=1.2, delays=0.23/0/0/0.95, dsn=2.0.0, status=sent (delivered to mailbox)
2022 Apr 05 07:30:05 (host) x.x.x.x->/var/log/syslog Apr  5 07:30:05 mc pveproxy[1575]: worker 23808 finished
2022 Apr 05 07:30:05 (host) x.x.x.x->/var/log/syslog Apr  5 07:30:05 mc pveproxy[23808]: worker exit
2022 Apr 05 07:30:05 (host) x.x.x.x->/var/log/syslog Apr  5 07:30:05 mc pveproxy[1575]: starting 1 worker(s)
2022 Apr 05 07:30:07 (host) x.x.x.x->/var/log/syslog Apr  5 07:30:05 mc pveproxy[1575]: worker 16235 started
2022 Apr 05 07:30:10 wazuh-manager->/var/log/syslog Apr  5 07:30:09 wazuh-manager filebeat[7566]: 2022-04-05T07:30:09.675Z#011INFO#011[input.harvester]#011log/harvester.go:309#011Harvester started for file.#011{"input_id": "45da0cdc-e8c8-44b0-8c48-8a569d929dbb", "source": "/var/log/auth.log", "state_id": "native::527934-1801", "finished": false, "os_id": "527934-1801", "old_source": "/var/log/auth.log", "old_finished": true, "old_os_id": "527934-1801", "harvester_id": "3a5db5f9-876a-4ec3-9094-16faf40ed4b0"}
2022 Apr 05 07:30:10 wazuh-manager->/var/log/messages Apr  5 07:30:09 wazuh-manager filebeat[7566]: 2022-04-05T07:30:09.675Z#011INFO#011[input.harvester]#011log/harvester.go:309#011Harvester started for file.#011{"input_id": "45da0cdc-e8c8-44b0-8c48-8a569d929dbb", "source": "/var/log/auth.log", "state_id": "native::527934-1801", "finished": false, "os_id": "527934-1801", "old_source": "/var/log/auth.log", "old_finished": true, "old_os_id": "527934-1801", "harvester_id": "3a5db5f9-876a-4ec3-9094-16faf40ed4b0"}
2022 Apr 05 07:30:11 (host) x.x.x.x->df -P ossec: output: 'df -P': Filesystem          1024-blocks       Used Available Capacity Mounted on
​

It does detect that Crontab is running, but it does not detect the modification of website files. I don't understand why this works when I do it without crontab :

** Alert 1649142341.68729: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,nist_800_53_SA.11,nist_800_53_SI.4,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2022 Apr 05 07:05:41 wazuh-manager->/var/log/sites-web/web-2022-04-04.log
Rule: 31101 (level 5) -> 'Web server 400 error code.'
Src IP:  www.site-web.com  
 x.x.x.x  www.site-web.com   - [04/Apr/2022:15:25:41 +0200] "GET /wp-json/ HTTP/1.1" 403 1874 "https:// www.site-web.com  /en/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.17 Safari/537.36"
srcip2:  x.x.x.x

** Alert 1649142341.69410: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,nist_800_53_SA.11,nist_800_53_SI.4,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2022 Apr 05 07:05:41 wazuh-manager->/var/log/sites-web/web-2022-04-04.log
Rule: 31101 (level 5) -> 'Web server 400 error code.'
Src IP:  www.site-web.com  
 x.x.x.x  www.site-web.com  - [04/Apr/2022:15:25:42 +0200] "GET /en/comments/feed/ HTTP/1.1" 404 107410 "https:// www.site-web.com  /en/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.17 Safari/537.36"
srcip2: x.x.x.x

** Alert 1649142341.70102: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,nist_800_53_SA.11,nist_800_53_SI.4,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,
2022 Apr 05 07:05:41 wazuh-manager->/var/log/sites-web/web-2022-04-04.log
Rule: 31101 (level 5) -> 'Web server 400 error code.'
Src IP: www.site-web.com  
 x.x.x.x  www.site-web.com - [04/Apr/2022:15:27:23 +0200] "POST /wp-json/contact-forms/1200/feedback HTTP/1.1" 403 1874 "https://www.site-web.com/en/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.17 Safari/537.36"
srcip2: x.x.x.x
​



Jonathan Martín Valera

unread,
Apr 6, 2022, 4:06:53 AM4/6/22
to Wazuh mailing list

Hi, don’t worry about the late response,

Ok, so you are assuring me that when crontab is run, the log lines are being added to the monitored file, right?

Can you describe the architecture and flow of this case? For example, is it one of the following cases?

  • (1) - Is that script running on a host-x, where logs are stored in a *.log file and in are being sent to the wazuh-manager (host-y) through a wazuh-agent that you have installed on host-x?

  • (2) - Is that script running on the host-y where the logs are stored in a *.log file, and is it being monitored with thewazuh-manager` itself?

Notice that in the events you shared with me, the only thing that appears are /var/log/syslog events from host x.x.x.x.x plus the output from the execution of a periodic command, so nothing related to apache events appears.

Please describe to me the architecture and information flow of the case you are talking about, and provide me with as much information as possible to try to reproduce your case and comment on the results.

If possible, share the following (change possible sensitive information for another string of the same type):

  • Information flow between processes.
  • Added crontab line.
  • Example log lines added in that centralized file.
  • Configuration of ossec.conf used to monitor this file.

Best regards.

​

Cyprien Chapelle

unread,
Apr 6, 2022, 8:10:28 AM4/6/22
to Wazuh mailing list
Thanks for your help.


Ok, so you are assuring me that when crontab is run, the log lines are being added to the monitored file, right?

Yes.

Here is the corresponding case :

(2) - Is that script running on the host-y where the logs are stored in a *.log file, and is it being monitored with thewazuh-manager itself?


  • Information flow between processes.
In alerts.log, we find the lines showing that the Crontab is activated: 
2022 Apr 05 07:30:02 wazuh-manager->/var/log/syslog Apr  5 07:30:01 wazuh-manager CRON[8289]: (root) CMD (/etc/recup-logs-web/script1.sh) 
2022 Apr 05 07:30:02 wazuh-manager->/var/log/syslog Apr  5 07:30:01 wazuh-manager CRON[8288]: (root) CMD (/etc/recup-logs-web/script2.sh)
​
And when I manually run the scripts (./script1.sh) I get good logs in alerts.log

 ** Alert 1649142341.68729: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,nist_800_53_SA.11,nist_800_53_SI.4,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3, 
2022 Apr 05 07:05:41 wazuh-manager->/var/log/sites-web/web-2022-04-04.log 
Rule: 31101 (level 5) -> 'Web server 400 error code.' 
Src IP:  www.site-web.com    
x.x.x.x  www.site-web.com   - [04/Apr/2022:15:25:41 +0200] "GET /wp-json/ HTTP/1.1" 403 1874 "https:// www.site-web.com  /en/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.17 Safari/537.36" 
srcip2:  x.x.x.x  

** Alert 1649142341.69410: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,nist_800_53_SA.11,nist_800_53_SI.4,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3, 
2022 Apr 05 07:05:41 wazuh-manager->/var/log/sites-web/web-2022-04-05.log 

Rule: 31101 (level 5) -> 'Web server 400 error code.' 
Src IP:  www.site-web.com    
x.x.x.x  www.site-web.com  - [04/Apr/2022:15:25:42 +0200] "GET /en/comments/feed/ HTTP/1.1" 404 107410 "https:// www.site-web.com  /en/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.17 Safari/537.36" 
srcip2: x.x.x.x
​

  • Added crontab line.
I'm doing acrontab -e to show you :
00 02 * * * /etc/recup-logs-web/script1.sh
00 02 * * * /etc/recup-logs-web/script2.sh
​
  • Example log lines added in that centralized file.
Files added in this file:

/var/log/sites-web/

​
And here are two examples of files added in folder sites-web by the scripts :

....
web-2022-04-05.log
web-2022-04-05.log
....
​

  • Configuration of ossec.conf used to monitor this file 
<ossec_config> 
...
  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/sites-web/*.log</location>
  </localfile>

</ossec_config>
​

Jonathan Martín Valera

unread,
Apr 6, 2022, 10:43:41 AM4/6/22
to Wazuh mailing list

Hi,

I think I know what might be happening to you. It is related to the use of the *.log when monitoring the /var/log/sites-web directory with the following configuration:

<localfile>
    <log_format>apache</log_format>
    <location>/var/log/sites-web/*.log</location>
</localfile>

I guess your script creates the new file, and then adds all the logs, right?

Wazuh is able to instantly monitor all files in the specified directory at the moment the service (in this case your wazuh-manager) is configured and restarted. Regarding the new files that are added after this restart, there is a refresh time in which the log analysis daemon itself (called wazuh-logcollector) checks if there are new files in the monitored directories to add them to its monitoring list.

By default, this refresh time is 64 seconds (see logcollector.vcheck_files in https://documentation.wazuh.com/current/user-manual/reference/internal-options.html#logcollector).

In your case, as the script executed with crontab is the one that creates the file and adds the logs, there is no time for this new file to be monitored (time between file creation and logs < 64s), and later when you run the script manually, as this file already exists and the 64 seconds after its creation have expired, all the alerts are reported.

The solution you can apply here is as follows:

  • First of all, you can reduce the time to check for new files from 64s to a shorter time, (e.g. 1 or 0s). This leads to a slight increase in CPU usage, but should be almost negligible. You can edit this setting by adding the following to the /var/ossec/etc/local_internal_options.conf file:

    logcollector.vcheck_files=0

    and restarting the wazuh-manager. (because you are using the wazuh-manager itself to monitor this file.)

    systemctl restart wazuh-manager

  • Also, in your script, you should make a short sleep between the creation of the new log file (you can make for example an empty touch) and then add the logs. This time should be 1s + the number you have set in the configuration above. For example, for the above case of logcollector.vcheck_files=0, your script should have the following structure:

    touch <file_path.log>  # Create the empty file_path
sleep 1  # Wait 1 second
get_logs(<file_path.log>)  # Retrieve all logs and add them to the <file_path.log>

Try it and let us know the results.

I hope this information has been useful to you.

Best regards.

​

Cyprien Chapelle

unread,
Apr 7, 2022, 3:08:08 AM4/7/22
to Wazuh mailing list
Hi Jonathan,

I did what you said and by miracle, this morning the logs have been added!

Thank you very much because I would never have thought of this problem.

Have a very nice day.
Reply all
Reply to author
Forward
0 new messages