Issues getting web rules to detect an Nginx log when it's come in via syslog

Miguel Jacq

unread,

Jun 21, 2021, 6:29:56 AM6/21/21

to ossec-list

Hi,

I am running a system whereby Nginx traffic logs are being sent from a Docker container to a remote syslog server, where they arrive in that remote syslog server's /var/log/syslog. This remote server is also the one running OSSEC.

As a result, the Nginx logs look like this in the syslog - note the 'example.com' is effectively the 'program_name' which is the identifier of the container itself.

Jun 20 15:52:09 example.com nginx: 11.22.33.44 - - [20/Jun/2021:15:52:09 +0000] "GET /something/ HTTP/1.1" 500 7910 "https://example.com/" "Mozilla/5.0 (Windows NT 5.1; rv:52.0) Gecko/20100101 Firefox/52.0"

My problem is that the OSSEC rules are not recognising the Nginx logs, because they are in the syslog.

To 'half' solve that, I added this custom decoder which I borrowed from https://github.com/wazuh/wazuh/issues/352:

<decoder name="web-accesslog">
<type>web-log</type>
<program_name>nginx|apache</program_name>
</decoder>

Now, this is good because the above example log message will now appear as rule 31101 'Access log messages grouped'. Progress!

However, note that the log message was a 500 internal server error. It seems that despite landing in 31101 thanks to the custom decoder, the other 'child' rules in web_rules.xml are not applying, e.g 31122:

<rule id="31122" level="5">
    <if_sid>31120</if_sid>
    <id_pcre2>^500</id_pcre2>
    <options>alert_by_email</options>
    <description>Web server 500 error code (Internal Error).</description>
    <group>system_error,</group>
</rule>

It doesn't seem to hit this error, it just stays as 31101 according to ossec-logtest.

I am assuming it's the id_pcre2 not picking up the '500' because of the extra fields when it's from syslog? As a guess?

If I change both rule 31120 and rule 31122 to use <match>50</match> and <match>500</match> respectively, then it works, and rule 31122 fires for the above. But not if it uses id_pcre2 *or* if it uses ^ at the start of the match - both make it skip.

I'm not so great at regexes - so I would really appreciate any help to get the standard web rules detecting the above Nginx log message when it's coming as a 'syslog' message.

I am running OSSEC 3.6.0 on Ubuntu 18.04.

Thanks!

Yana Zaeva

unread,

Jun 21, 2021, 10:55:41 AM6/21/21

to ossec-list

Hi Miguel,

Could you please paste the output coming from ossec-logtest after pasting these logs?

Waiting for your reply,

Yana.

Miguel Jacq

unread,

Jun 21, 2021, 6:00:49 PM6/21/21

to ossec-list

Thanks Yana,

With the original 'id_pcre2' in rules 31120 and 31122, and my custom decoder per the original post, I get this:

ossec-testrule: Type one log per line.

Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 "https://something.com" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"

**Phase 1: Completed pre-decoding.
       full event: 'Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 "https://something.com" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"'
       hostname: 'example.com'
       program_name: 'nginx'
       log: '22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 "https://something.com" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"'

**Phase 2: Completed decoding.
       decoder: 'web-accesslog'

**Phase 3: Completed filtering (rules).
       Rule id: '31100'
       Level: '0'
       Description: 'Access log messages grouped.'

If I change the <id_prce2> to <match> and remove the ^ in the 50/500 match string, for rules 31120 and 31122, I get this:

ossec-testrule: Type one log per line.

Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 "https://something.com" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"

**Phase 1: Completed pre-decoding.
       full event: 'Jun 21 12:35:37 example.com nginx: 22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 "https://something.com" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"'
       hostname: 'example.com'
       program_name: 'nginx'
       log: '22.33.44.55 - - [21/Jun/2021:12:35:37 +0000] "GET /something?bad HTTP/1.1" 500 10372 "https://something.com" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.26 Safari/537.36 Core/1.63.5083.400 QQBrowser/10.0.972.400"'

**Phase 2: Completed decoding.
       decoder: 'web-accesslog'

**Phase 3: Completed filtering (rules).
       Rule id: '31122'
       Level: '5'
       Description: 'Web server 500 error code (Internal Error).'
**Alert to be generated.

Thanks

Yana Zaeva

unread,

Jun 29, 2021, 6:26:06 AM6/29/21

to ossec-list

Hi,

My apologies for the late response. Is your installation a fresh installation? It seems that from version 3.4, you must have the pcre2-10.32 sources installed in src/external. You can obtain them by running:

cd ossec-hids-*

wget https://ftp.pcre.org/pub/pcre/pcre2-10.32.tar.gz

tar xzf pcre2-10.32.tar.gz -C src/external

After this, if you decide to use the pcre2-10.32 sources, you must set the PCRE"_SYSTEM variable to no:

cd ossec-hids-*

PCRE2_SYSTEM=no ./install.sh

If you decide to use the system's PCRE2, set the PCRE2_SYSTEM variable to yes:

cd ossec-hids-*

PCRE2_SYSTEM=yes ./install.sh

You can check more information about this here.

Hope I was helpful. Let me know if you need anything else.

Regards,

Yana.

Miguel Jacq

unread,

Jun 29, 2021, 5:59:59 PM6/29/21

to ossec-list

Hi Yana,

Thank you for the reply.

It's not really a 'fresh' installation. I did install it using the system PCRE2, with `PCRE2_SYSTEM=yes ./install.sh`

I don't think the issue is with PCRE as such but the fact that the nginx logs are arriving in the syslog, and therefore the decoder regex somehow needs tweaking since the format of the log is not this traditional Nginx log:

11.22.33.44 - - [29/Jun/2021:21:55:30 +0000] "GET / HTTP/1.1" 200 467 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4"

... but instead because it's coming into rsyslog, it looks like this:

Jun 29 21:55:30 example.com nginx: 11.22.33.44 - - [29/Jun/2021:21:55:30 +0000] "GET / HTTP/1.1" 200 467 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4"

The web-accesslog decoder looks like this, I wonder if it needs overriding somehow to account for the extra fields and maybe that's why the PCRE matches aren't able to 'detect' the status code in the usual way?

<prematch_pcre2>^\S+ \S+ \S+ \[\S+ \S\d+\] "\w+ \S+ HTTP\S+" </prematch_pcre2>
<pcre2>^(\S+) \S+ (\S+) \[\S+ \S\d+\] </pcre2>
<pcre2>"(\w+) (\S+) HTTP\S+" (\d+) </pcre2>
<order>srcip, srcuser, action, url, id</order>
</decoder>

I'll see if hopefully anyone else has any tips.

Thanks again

Miguel Jacq

unread,

Mar 23, 2022, 6:30:46 PM3/23/22

to ossec-list

Sorry to resurrect an old thread, but I finally got my local decoders to work with Nginx logs when they are being sent to syslog rather than /var/log/nginx.

Here's what worked in order to trigger the standard web rules from web_rules.xml . Note that other agents that have 'regular' Nginx logs in /var/log/nginx also still work (they are parsed by the core OSSEC web-accesslog decoder).

<decoder name="nginx-syslog">
<program_name>^nginx</program_name>
<type>web-log</type>
</decoder>

<decoder name="nginx-syslog_2">
<parent>nginx-syslog</parent>
<type>web-log</type>
<regex offset="after_parent">(\d*.\d*.\d*.\d*)</regex>
<order>srcip</order>
</decoder>

<decoder name="nginx-syslog_2">
<parent>nginx-syslog</parent>
<type>web-log</type>
<regex offset="after_parent">(\d*/\w*/\d*)</regex>
<order>date</order>
</decoder>

<decoder name="nginx-syslog_2">
<parent>nginx-syslog</parent>
<type>web-log</type>
<regex offset="after_parent">:(\d*:\d*:\d*)</regex>
<order>time</order>
</decoder>

<decoder name="nginx-syslog_2">
<parent>nginx-syslog</parent>
<type>web-log</type>

<regex offset="after_parent">\s+\S+\s+"(\S+)\s+(\S+)\s+\S+"\s+(\d+)</regex>
<order>action, url, id</order>
</decoder>

I am sure it could be simplified, but for now I'm happy as it actually trips the rules as desired.

Reply all

Reply to author

Forward