Rule 533, netstat listening ports, email notifications have truncated outputs

1,311 views

Skip to first unread message

Simon Tideswell

unread,

Jul 17, 2018, 12:37:52 AM7/17/18

to Wazuh mailing list

Since I've been running OSSEC/Wazuh (from the early days of OSSEC 2.8.x, to Wazuh 2.x and now on Wazuh 3.3.1) I've always been afflicted by apparently bogus, and certainly not very helpful email notifications about "netstat listening ports". I also know that other people have been similarly afflicted because I've seen plenty of chatter about this on both the OSSEC and Wazuh forums. What I haven't seen anywhere yet is a solution?!

Here's an example notification I received today.

----------------------------------------------------------------------------------------------------------

Wazuh Notification.

2018 Jul 17 13:01:38

Received From: myossecserver->netstat listening ports

Rule: 533 fired (level 7) -> "Listened ports status (netstat) changed (new port opened or closed)."

Portion of the log(s):

ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

Proto Local Address Foreign Address State PID/Program name

tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd

tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1196/sshd

tcp 0.0.0.0:2812 0.0.0.0:* LISTEN 1424/monit

tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:48975 0.0.0.0:* LISTEN 1705/cvfwd

tcp 0.0.0.0:50946 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 1408/nrpe

tcp 0.0.0.0:56927 0.0.0.0:* LISTEN 1694/ClMgrS

tcp 0.0.0.0:59916 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:8400 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:8403 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:25 0.0.0.0:* LISTEN 710/smtpd

tcp 127.0.0.1:5432 0.0.0

Previous output:

ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

--END OF NOTIFICATION

----------------------------------------------------------------------------------------------------------

The aspect that makes this notification most unhelpful is that the output of both the before and after netstats is clearly truncated, apparently arbitrarily. If I look at the entry in the alerts.json log that I think relates to this notification and I expand the JSON to make it easier to read, I get the following.

----------------------------------------------------------------------------------------------------------

timestamp => 2018-07-17T13:01:38.61+1000

rule => {"level"=>7, "description"=>"Listened ports status (netstat) changed (new port opened or closed).", "id"=>"533", "firedtimes"=>1, "mail"=>true, "groups"=>["ossec"], "pci_dss"=>["10.2.7", "10.6.1"], "gpg13"=>["10.1"], "gdpr"=>["IV_35.7.d"]}

agent => {"id"=>"000", "name"=>"myossecserver"}

manager => {"name"=>"myossecserver"}

id => 1531796498.174017

previous_output => ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

Proto Local Address Foreign Address State PID/Program name

tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd

tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1196/sshd

tcp 0.0.0.0:2812 0.0.0.0:* LISTEN 1424/monit

tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:48975 0.0.0.0:* LISTEN 1705/cvfwd

tcp 0.0.0.0:50946 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 1408/nrpe

tcp 0.0.0.0:56927 0.0.0.0:* LISTEN 1694/ClMgrS

tcp 0.0.0.0:59916 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:8400 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:8403 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:25 0.0.0.0:* LISTEN 1883/master

tcp 127.0.0.1:5432 0.0.0.0:* LISTEN 1554/postgres

tcp 127.0.0.1:54384 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:55167 0.0.0.0:* LISTEN 1693/cvd

tcp :::22 :::* LISTEN 1196/sshd

tcp :::55000 :::* LISTEN 1199/nodejs

tcp :::5666 :::* LISTEN 1408/nrpe

tcp :::57657 :::* LISTEN 1705/cvfwd

udp 0.0.0.0:123 0.0.0.0:* 1421/ntpd

udp 0.0.0.0:161 0.0.0.0:* 1489/snmpd

udp 127.0.0.1:123 0.0.0.0:* 1421/ntpd

udp 210.18.251.29:123 0.0.0.0:* 1421/ntpd

udp :::123 :::* 1421/ntpd

full_log => ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

Proto Local Address Foreign Address State PID/Program name

tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd

tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1196/sshd

tcp 0.0.0.0:2812 0.0.0.0:* LISTEN 1424/monit

tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:48975 0.0.0.0:* LISTEN 1705/cvfwd

tcp 0.0.0.0:50946 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 1408/nrpe

tcp 0.0.0.0:56927 0.0.0.0:* LISTEN 1694/ClMgrS

tcp 0.0.0.0:59916 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:8400 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:8403 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:25 0.0.0.0:* LISTEN 710/smtpd

tcp 127.0.0.1:5432 0.0.0.0:* LISTEN 1554/postgres

tcp 127.0.0.1:54384 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:55167 0.0.0.0:* LISTEN 1693/cvd

tcp :::22 :::* LISTEN 1196/sshd

tcp :::55000 :::* LISTEN 1199/nodejs

tcp :::5666 :::* LISTEN 1408/nrpe

tcp :::57657 :::* LISTEN 1705/cvfwd

udp 0.0.0.0:123 0.0.0.0:* 1421/ntpd

udp 0.0.0.0:161 0.0.0.0:* 1489/snmpd

udp 127.0.0.1:123 0.0.0.0:* 1421/ntpd

udp 210.18.251.29:123 0.0.0.0:* 1421/ntpd

udp :::123 :::* 1421/ntpd

decoder => {"name"=>"ossec"}

previous_log => ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

Proto Local Address Foreign Address State PID/Program name

tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd

tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1196/sshd

tcp 0.0.0.0:2812 0.0.0.0:* LISTEN 1424/monit

tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:48975 0.0.0.0:* LISTEN 1705/cvfwd

tcp 0.0.0.0:50946 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 1408/nrpe

tcp 0.0.0.0:56927 0.0.0.0:* LISTEN 1694/ClMgrS

tcp 0.0.0.0:59916 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:8400 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:8403 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:25 0.0.0.0:* LISTEN 1883/master

tcp 127.0.0.1:5432 0.0.0.0:* LISTEN 1554/postgres

tcp 127.0.0.1:54384 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:55167 0.0.0.0:* LISTEN 1693/cvd

tcp :::22 :::* LISTEN 1196/sshd

tcp :::55000 :::* LISTEN 1199/nodejs

tcp :::5666 :::* LISTEN 1408/nrpe

tcp :::57657 :::* LISTEN 1705/cvfwd

udp 0.0.0.0:123 0.0.0.0:* 1421/ntpd

udp 0.0.0.0:161 0.0.0.0:* 1489/snmpd

udp 127.0.0.1:123 0.0.0.0:* 1421/ntpd

udp 210.18.251.29:123 0.0.0.0:* 1421/ntpd

udp :::123 :::* 1421/ntpd

predecoder => {"hostname"=>"myossecserver"}

location => netstat listening ports

----------------------------------------------------------------------------------------------------------

Comparing the values of 'previous_output' and 'previous_log' I see that they are identical. Comparing the values of 'full_log' and 'previous_output' I can see that they differ slightly.

----------------------------------------------------------------------------------------------------------

diff previous_log full_log

16c16

< tcp 127.0.0.1:25 0.0.0.0:* LISTEN 1883/master

---

> tcp 127.0.0.1:25 0.0.0.0:* LISTEN 710/smtpd

----------------------------------------------------------------------------------------------------------

OK, so the alert was triggered for a reason, I'm happy to accept that the alert is not 'bogus'. I'm not going to get into why netstat sometimes shows PID 1883 and sometimes PID 710 in the output (I expect that an instance of smtpd is spawned by Postfix's master process during receipt of an email): that's something that I could presumably filter out in my '<localfile>' definition in ossec.conf (or I could just choose to ignore such alerts). What's really bothering me is the truncation of the output. It seems to have afflicted OSSEC/Wazuh for a long time without any resolution. And the truncation of the output makes the notification totally useless. I'm a busy sysadmin and don't have time to trawl through the alerts.json log every time there is such a notification. I can confirm that the data in Elasticsearch is completely intact i.e. not truncated. I just wondered if someone out there had a solution to this issue? I've seen solutions before which suggest tinkering around with the commands used in the '<localfile>' definition in ossec.conf. But they are not real solutions. The real problem is that the outputs (more specifically the email outputs) are truncated. If I could see the full outputs I could easily see that TCP 25 was being bound to a different process, I'd see what was going on and immediately disregard the event as "of no concern". If I was getting too many of these I could filter out TCP 25 in the command definition, that's not a problem for a sysadmin to deal with, but the truncation of the output means the alert is just irritating and unhelpful.

Simon

alfonso.r...@wazuh.com

unread,

Jul 18, 2018, 5:06:14 AM7/18/18

to Wazuh mailing list

Hello Simon,

Sorry for the late response. The email notification sends the alerts in the same format as the alerts appear in the alerts.log file. This format is different from the one we found in the alerts.json file. If you could find the alert you sent as an example in the alerts.log file, you would see that it is also truncated. This is due to a limitation that has existed, as you say, for a long time.

The problem lies in this code section: https://github.com/wazuh/wazuh/blob/master/src/analysisd/alerts/log.c#L126

As we can see, due to that particular line of code, the alerts that appear in the alerts.log file and that are sent by mail, have a capacity limited to 1256 characters. This makes any alert with a larger characters size appear truncated.

At Wazuh we were aware of this limitation and although we did not have a corresponding issue, we have a solution in development. As we can see in this branch, we have removed the size limitation: https://github.com/wazuh/wazuh/blob/master-increased-buffer/src/analysisd/alerts/log.c#L126

The change has not yet taken effect because it has great relevance and will take some time. We have created the following issue to try to speed up the process: https://github.com/wazuh/wazuh/issues/964

We are sorry we could not be of more help. Do not hesitate to contact us if you have any questions.

Kind regards,

Alfonso Ruiz-Bravo

Simon Tideswell

unread,

Jul 18, 2018, 7:09:25 PM7/18/18

to Wazuh mailing list

Thanks for your response Alfonso.

I can confirm when I look at the corresponding alerts.log file that there is indeed some truncation.

--------------------------------------------------------------------------------------------------------------------------------------

** Alert 1531796498.174017759: mail - ossec,pci_dss_10.2.7,pci_dss_10.6.1,gpg13_10.1,gdpr_IV_35.7.d,

2018 Jul 17 13:01:38 myossecserver->netstat listening ports

Rule: 533 (level 7) -> 'Listened ports status (netstat) changed (new port opened or closed).'

ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

Proto Local Address Foreign Address State PID/Program name

tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd

tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1196/sshd

tcp 0.0.0.0:2812 0.0.0.0:* LISTEN 1424/monit

tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:48975 0.0.0.0:* LISTEN 1705/cvfwd

tcp 0.0.0.0:50946 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 1408/nrpe

tcp 0.0.0.0:56927 0.0.0.0:* LISTEN 1694/ClMgrS

tcp 0.0.0.0:59916 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:8400 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:8403 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:25 0.0.0.0:* LISTEN 710/smtpd

tcp 127.0.0.1:5432 0.0.0

Previous output:

ossec: output: 'netstat listening ports':

Active Internet connections (only servers)

Proto Local Address Foreign Address State PID/Program name

tcp 0.0.0.0:199 0.0.0.0:* LISTEN 1489/snmpd

tcp 0.0.0.0:22 0.0.0.0:* LISTEN 1196/sshd

tcp 0.0.0.0:2812 0.0.0.0:* LISTEN 1424/monit

tcp 0.0.0.0:443 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:48975 0.0.0.0:* LISTEN 1705/cvfwd

tcp 0.0.0.0:50946 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:5666 0.0.0.0:* LISTEN 1408/nrpe

tcp 0.0.0.0:56927 0.0.0.0:* LISTEN 1694/ClMgrS

tcp 0.0.0.0:59916 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:80 0.0.0.0:* LISTEN 1960/apache2

tcp 0.0.0.0:8400 0.0.0.0:* LISTEN 1693/cvd

tcp 0.0.0.0:8403 0.0.0.0:* LISTEN 1705/cvfwd

tcp 127.0.0.1:25 0.0.0.0:* LISTEN 1883/master

tcp 127.0.0.1:5432 0.0.0

--------------------------------------------------------------------------------------------------------------------------------------

Though you will observe that the truncation is more pronounced in the email than is seen in the log entry. So it looks as though there's data truncation in generating the log entry and still more data truncation when generating the alert notification.

I fully understand that any fix may take some time to become fully available. I'll just make a mental note to check back to see of the fix is deployed when I update to later versions in the natural course of events.

Thanks for responding.

Simon

Reply all

Reply to author

Forward

0 new messages