Importing Apache logs with mod

Michael Mansour

unread,

Feb 4, 2021, 1:23:29 AM2/4/21

to Wazuh mailing list

Hi,

I'm trying to get Apache logs from CentOS 7 machines into Wazuh.

I'm using the latest Wazuh version 4 (installed 3 days ago).

The agent.conf on the web servers have:

<localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/error_log</location>
</localfile>

<localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/access_log</location>
</localfile>

in the "<ossec_config>" section, I didn't need to do anything there those entries were there.

For the Wazuh server I added the above to its ossec.conf at the bottom, after the maillog entry:

<localfile>
    <log_format>syslog</log_format>
    <location>/var/log/maillog</location>
</localfile>

<localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/error_log</location>
</localfile>

<localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/access_log</location>
</localfile>

I restarted wazuh-server on the Wazuh server, and wazuh-agent on the web servers.

On the web servers, I see items in /var/log/httpd/error_log and /var/log/access_log, together with ModSecurity items (which I'm most interested in and should be generating alerts) but nothing in the Wazuh GUI.

I see dovecot, postfix, etc all logging just nothing from Apache.

I don't wish to logall, but what am I doing wrong here?

Thanks.

Michael.

victor...@wazuh.com

unread,

Feb 4, 2021, 6:11:37 AM2/4/21

to Wazuh mailing list

Hi Micoots,
If you want to monitor an HTTP server like Apache, the best way is to deploy a Wazuh agent in the same host.

As you said, there are 2 pre-defined localfile blocks in the default Wazuh agent configuration, so there is no need to an extra configuration in the agent side.
Wazuh agent will monitor /var/log/httpd/error_log and /var/log/httpd/acces_log and send them to Wazuh manager.

By default in Wazuh manager, there are defined rules for apache in /var/ossec/ruleset/rules/0250-apache_rules.xml.
Take a look at that file because there are many ModSecurity rules defined.

The configuration you added on Wazuh manager is not needed unless you have an httpd server in the same host as your Wazuh manager.

I’ve tried to reproduce your environment with 2 CentOS 7 hosts. 1 Wazuh manager and 1 CentOS 7 with Apache. With the two localfile blocks in agent configuration /var/ossec/etc/ossec.conf

<localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/error_log</location>
  </localfile>

  <localfile>
    <log_format>apache</log_format>
    <location>/var/log/httpd/access_log</location>
</localfile>

After setting up the Apache server I made some invalid requests to it like:

curl example.com/invadlireq

And in Wazuh manager I get the following alert:

** Alert 1612436633.3498915: - web,accesslog,attack,pci_dss_6.5,pci_dss_11.4,gdpr_IV_35.7.d,nist_800_53_SA.11,nist_800_53_SI.4,tsc_CC6.6,tsc_CC7.1,tsc_CC8.1,tsc_CC6.1,tsc_CC6.8,tsc_CC7.2,tsc_CC7.3,

2021 Feb 04 11:03:53 (apache) any->/var/log/httpd/access_log
Rule: 31101 (level 5) -> 'Web server 400 error code.'
Src IP: 127.0.0.1
127.0.0.1 - - [04/Feb/2021:11:03:53 +0000] "GET /invadlireq HTTP/1.1" 404 208 "-" "curl/7.29.0"

To troubleshoot your issue, first, we have to make sure that Wazuh manager is receiving events from your agent.
Activate the logall in ossec.conf and check for incoming events in /var/ossec/logs/archives/archives.log https://documentation.wazuh.com/4.0/user-manual/reference/ossec-conf/global.html#logall.
Once you see incoming events from your agent, you should turn logall off, to avoid disk flooding.

Check if Wazuh manager is generating alerts. Make some invalid request to your Apache server while monitoring Wazuh’s manager file /var/ossec/logs/alerts/alerts.log. It should log 404 alerts.

If Wazuh manager is generating alerts, then the events in your logs are not enough critical to generate alerts or there is no default rule for that event.
Please share with us the content of your Apache logs that you think should trigger an alert.
If there is no alert for that logs we could create a custom one. Please take a look at https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwi-kYGHi9DuAhWFWxUIHbaGBRYQFjABegQIBRAC&url=https%3A%2F%2Fwazuh.com%2Fblog%2Fcreating-decoders-and-rules-from-scratch%2F&usg=AOvVaw0RiHBT0ZTiPj_fC8Q8UdY5

If you couldn’t see any events coming from your agent, please check connectivity between your agent and your Wazuh manager. Check that there is no firewall blocking connections. You could make a fast test with ping or checking the Wazuh agent logs /var/ossec/logs/ossec.log to see if it is connected to the Manager. Make sure all required ports are open https://documentation.wazuh.com/4.0/getting-started/architecture.html

Hope it helps, Víctor.

Michael Mansour

unread,

Feb 4, 2021, 7:05:22 AM2/4/21

to victor...@wazuh.com, Wazuh mailing list

Hi Victor,

I removed the config in ossec.conf for the wazuh server, restarted the service.

I did do the logall and yes, after restarting the wazuh-manager service, all sorts of stuff gets logged to the archives.log. I turned it off and restarted the service again.

I made some invalid requests to the web server and nothing was showing up in the alerts.log of the wazuh manager server.

I do get this in the alerts.log (sanitised):

[Thu Feb 04 17:46:03.640514 2021] [autoindex:error] [pid 1399] [client xxx.xxx.xxx.xxx:39266] AH01276: Cannot serve directory /home/xxxxxxx/public_html/: No matching DirectoryIndex (index.html,index.htm,index.php,index.php4,index.php5) found, and server-generated directory index forbidden by Options directive

but what the web server has in the /var/log/httpd/error_log is also bunches of (santised):

[Thu Feb 04 22:35:07.223104 2021] [:error] [pid 28999] [client 127.0.0.1:53332] [client 127.0.0.1] ModSecurity: Warning. Operator LT matched 5 at TX:inbound_anomaly_score. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_60_correlation.conf"] [line "33"] [id "981203"] [msg "Inbound Anomaly Score (Total Inbound Score: 2, SQLi=0, XSS=0): Request Missing an Accept Header"] [hostname "localhost"] [uri "/server-status"] [unique_id "YBvbxxxxMI@-Z-f6HxxxxxxxAAI"]

I get plenty of dovecot, postfix, PAM logs in the alerts.log from the same web servers, so the wazuh-agent on those web servers is working, just not getting the ModSecurity ones and many of the other Apache ones.

For example, I sent a curl blah.com.au/invalidrequest (on a real web server) it shows this in the Apache logs of that server:

xxx.xxx.xxx.xxx somedomain.com - - 80 [04/Feb/2021:22:43:24 +1100] "GET /invalidrequest HTTP/1.1" "" 404 212 "-" "curl/7.29.0" 373 419 1291

but nothing showing in the wazuh manager alerts.log for it.

In regards to the config, I have this on the ossec.conf for the agent:

<ossec_config>
<client>
<server>
<address>xxx.xxx.xxx.xxx</address>
<port>1514</port>
<protocol>tcp</protocol>
</server>
<config-profile>centos, centos7, centos7.9</config-profile>
<notify_time>10</notify_time>
<time-reconnect>60</time-reconnect>
<auto_restart>yes</auto_restart>
<crypto_method>aes</crypto_method>
</client>

<client_buffer>

<disabled>no</disabled>
<queue_size>5000</queue_size>
<events_per_second>500</events_per_second>
</client_buffer>


<rootcheck>
<disabled>no</disabled>
<check_files>yes</check_files>
<check_trojans>yes</check_trojans>
<check_dev>yes</check_dev>
<check_sys>yes</check_sys>
<check_pids>yes</check_pids>
<check_ports>yes</check_ports>
<check_if>yes</check_if>


<frequency>43200</frequency>

<rootkit_files>/var/ossec/etc/shared/rootkit_files.txt</rootkit_files>
<rootkit_trojans>/var/ossec/etc/shared/rootkit_trojans.txt</rootkit_trojans>

<skip_nfs>yes</skip_nfs>
</rootcheck>

<wodle name="cis-cat">
<disabled>yes</disabled>
<timeout>1800</timeout>
<interval>1d</interval>
<scan-on-start>yes</scan-on-start>

<java_path>wodles/java</java_path>
<ciscat_path>wodles/ciscat</ciscat_path>
</wodle>


<wodle name="osquery">
<disabled>yes</disabled>
<run_daemon>yes</run_daemon>
<log_path>/var/log/osquery/osqueryd.results.log</log_path>
<config_path>/etc/osquery/osquery.conf</config_path>
<add_labels>yes</add_labels>
</wodle>


<wodle name="syscollector">
<disabled>no</disabled>
<interval>1h</interval>
<scan_on_start>yes</scan_on_start>
<hardware>yes</hardware>
<os>yes</os>
<network>yes</network>
<packages>yes</packages>
<ports all="no">yes</ports>
<processes>yes</processes>
</wodle>

<sca>
<enabled>yes</enabled>
<scan_on_start>yes</scan_on_start>
<interval>12h</interval>
<skip_nfs>yes</skip_nfs>
</sca>


<syscheck>
<disabled>no</disabled>


<frequency>43200</frequency>

<scan_on_start>yes</scan_on_start>


<directories>/etc,/usr/bin,/usr/sbin</directories>
<directories>/bin,/sbin,/boot</directories>


<ignore>/etc/mtab</ignore>
<ignore>/etc/hosts.deny</ignore>
<ignore>/etc/mail/statistics</ignore>
<ignore>/etc/random-seed</ignore>

<ignore>/etc/adjtime</ignore>
<ignore>/etc/httpd/logs</ignore>
<ignore>/etc/utmpx</ignore>
<ignore>/etc/wtmpx</ignore>
<ignore>/etc/cups/certs</ignore>
<ignore>/etc/dumpdates</ignore>
<ignore>/etc/svc/volatile</ignore>


<ignore type="sregex">.log$|.swp$</ignore>


<nodiff>/etc/ssl/private.key</nodiff>

<skip_nfs>yes</skip_nfs>
<skip_dev>yes</skip_dev>
<skip_proc>yes</skip_proc>
<skip_sys>yes</skip_sys>


<process_priority>10</process_priority>


<max_eps>100</max_eps>


<synchronization>
<enabled>yes</enabled>
<interval>5m</interval>
<max_interval>1h</max_interval>
<max_eps>10</max_eps>
</synchronization>
</syscheck>


<localfile>
<log_format>command</log_format>
<command>df -P</command>
<frequency>360</frequency>
</localfile>

<localfile>
<log_format>full_command</log_format>
<command>netstat -tulpn | sed 's/$[[:alnum:]]\+$\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+$.*$:$[[:digit:]]*$\ \+$[0-9\.\:\*]\+$.\+\ $[[:digit:]]*\/[[:alnum:]\-]*$.*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == $.*$ ==/:\1/' | sed 1,2d</command>
<alias>netstat listening ports</alias>
<frequency>360</frequency>
</localfile>

<localfile>

<log_format>full_command</log_format>
<command>last -n 20</command>
<frequency>360</frequency>
</localfile>


<active-response>
<disabled>no</disabled>
<ca_store>/var/ossec/etc/wpk_root.pem</ca_store>
<ca_verification>yes</ca_verification>
</active-response>


<logging>
<log_format>plain</log_format>
</logging>

</ossec_config>

<ossec_config>

<localfile>
<log_format>apache</log_format>
<location>/var/log/httpd/error_log</location>
</localfile>

<localfile>
<log_format>apache</log_format>
<location>/var/log/httpd/access_log</location>
</localfile>

<localfile>
<log_format>audit</log_format>
<location>/var/log/audit/audit.log</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/ossec/logs/active-responses.log</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/messages</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/secure</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/xferlog</location>
</localfile>

<localfile>
<log_format>syslog</log_format>
<location>/var/log/maillog</location>
</localfile>

</ossec_config>

and the shared agent.conf which adds to that:

<agent_config profile="centos7">

<wodle name="syscollector">
<disabled>no</disabled>
<interval>1h</interval>
<os>yes</os>
<packages>yes</packages>
</wodle>

<wodle name="open-scap">
<content type="xccdf" path="ssg-rhel7-ds.xml">
<profile>xccdf_org.ssgproject.content_profile_pci-dss</profile>
</content>
</wodle>

</agent_config>

<agent_config>

<localfile>
<log_format>apache</log_format>
<location>/var/log/virtualmin/*error_log</location>

</localfile>

<localfile>
<log_format>apache</log_format>

<location>/var/log/virtualmin/*access_log</location>
</localfile>

</agent_config>

and for the ossec.conf on the wazuh server, I was going to post that but it's quite large, the only bit I added to it (with the before and after blocks shown) was:

...

<provider name="nvd">
<enabled>yes</enabled>
<update_from_year>2010</update_from_year>
<update_interval>1h</update_interval>
</provider>

</vulnerability-detector>


<wodle name="open-scap">
<disabled>no</disabled>
<timeout>1800</timeout>
<interval>1d</interval>
<scan-on-start>yes</scan-on-start>

<content type="xccdf" path="ssg-centos-7-ds.xml">
<profile>xccdf_org.ssgproject.content_profile_pci-dss</profile>
<profile>xccdf_org.ssgproject.content_profile_common</profile>
</content>
</wodle>


<syscheck>
<disabled>no</disabled>

...

So not 100% sure what's wrong here. I installed this without too many dramas from the distributed setup docs with one wazuh manager server and the other elasticsearch/kibana server.

Any ideas on this one or something else I can try and trouble-shoot?

On a side note...

There was one error I encountered with the install script, the bit where you run "bash ~/wazuh-server-installation.sh -n wazuhmanagerhost", I documented that as:

"

Found the issue here:

vim wazuh-server-installation.sh

eval "mkdir /etc/filebeat/certs ${debug}"

eval "cp ~/certs.tar /etc/filebeat/certs/ ${debug}"

eval "cd /etc/filebeat/certs/ ${debug}"

eval "tar -xf certs.tar ${iname}.pem ${iname}.key root-ca.pem ${debug}"

if [ ${iname} != "filebeat" ]

then

eval "mv /etc/filebeat/certs/${iname}.pem /etc/filebeat/certs/filebeat.pem ${debug}"

eval "mv /etc/filebeat/certs/${iname}.key /etc/filebeat/certs/filebeat.key ${debug}"

fi

iname is defined in the top as what is given to the -n argument.

When extracted, the filenames are:

filebeat.pem

filebeat.key

So the first clause can’t extract anything, and the then clause then gets activated and can’t work because the names are actually as above, so nothing is done and the script fails.

Changed to:

eval "mkdir /etc/filebeat/certs ${debug}"

eval "cp ~/certs.tar /etc/filebeat/certs/ ${debug}"

eval "cd /etc/filebeat/certs/ ${debug}"

eval "tar -xf certs.tar filebeat.pem filebeat.key root-ca.pem ${debug}"

which makes it work as intended.

"

Not sure where to post that problem though?

Thanks for your assistance so far.

Michael.

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/8TH_cLbthVk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/8ecf0087-8b50-4770-985b-6a21c5f44907n%40googlegroups.com.

Michael Mansour

unread,

Feb 4, 2021, 7:16:31 AM2/4/21

to Wazuh mailing list

Hmm, would my extended Apache log format have anything to do with it (in bold below)?

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%a %v %l %u %{local}p %t \"%r\" \"%q\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O %D" combined
    LogFormat "%a %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here. Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    #CustomLog "logs/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    CustomLog "logs/access_log" combined
</IfModule>

I did have to change those to get the logs ingested into another tool?

Michael.

victor...@wazuh.com

unread,

Feb 4, 2021, 10:58:19 AM2/4/21

to Wazuh mailing list

Hi Michael,
Yes, you are right, your custom Apache logs won’t trigger the default Apache decoder built-in Wazuh.
For example, in my access_log Apache logs looks like:

127.0.0.1 - - [04/Feb/2021:15:44:04 +0000] "GET /invadlireq HTTP/1
.1" 404 208 "-" "curl/7.29.0"

And your logs:

192.168.54.4 somedomain.com - - 80 [04/Feb/2021:22:43:24 +1100] "
GET /invalidrequest HTTP/1.1" "" 404 212 "-" "curl/7.29.0" 373 419 
1291

That’s the main reason for Wazuh not generating alerts. You should create your own decoders/rules.
Feel free to check our blog post: https://wazuh.com/blog/creating-decoders-and-rules-from-scratch/

If you need some help creating those rules/decoders, don’t hesitate to ask again!

Regards,
Víctor.

Reply all

Reply to author

Forward

Importing Apache logs with mod_security

Michael Mansour

victor...@wazuh.com

Michael Mansour

Michael Mansour

victor...@wazuh.com