ossec-remoted: WARNING: (1213): Message from 'IP' not allowed

Robert H

unread,

Apr 10, 2018, 12:23:23 PM4/10/18

to Wazuh mailing list

Hi,

We are starting to see this warnings from about all agents. This is from the ossec.log. Where or how can I find what is causing this warning, so I could try to eliminate it?

2018/04/10 08:15:05 ossec-remoted: WARNING: (1213): Message from 'IP' not allowed.

2018/04/10 08:15:07 ossec-remoted: WARNING: (1213): Message from 'IP' not allowed.

2018/04/10 08:15:08 ossec-remoted: WARNING: (1213): Message from 'IP' not allowed.

2018/04/10 08:15:13 ossec-remoted: WARNING: (1213): Message from 'IP' not allowed.

I think this might be (still) causing an issue with ossec-csyslogd as I have noticed it has stopped recently with only 10-20 agents connected. I'd like to find out for sure.

Also, is there an ETA for the 3.2.2 Wazuh manager release?

Regards,

Robert

Robert H

unread,

Apr 10, 2018, 7:50:19 PM4/10/18

to Wazuh mailing list

To add a little more information. I believe this warning is causing or related to the manager eating up excess RAM in the system. The system we're testing has 16GB and only 20 agents. I restarted the manager this morning and the RAM usage went down to about 2GBs. just a few minutes ago, it was using over 15GB.

Please let me know any sources to check for tracking down why this is happening.

The manager is currently 3.2.2. We have 2 in a cluster.

Regards,

Robert

Chema Martinez

unread,

Apr 11, 2018, 4:56:10 AM4/11/18

to Robert H, Wazuh mailing list

Hi Robert,

Typically, that messages received from ossec-remoted appear because the manager doesn`t recognize agents that are reporting events to it.

To ensure that the agents are registered properly, could you check if the 'client.keys' file of the manager contains the correct keys for every agent connected to that manager? The file is located at '/var/ossec/etc/client.keys', and it is formed by one line per agent with the format ' <id> <agent_name> <key> '.

Maybe for any reason, that file is missing or corrupted or the cluster is not synchronizing the keys.

On the other hand, it is very strange that the manager is using as much RAM memory. When you run the manager again, could you use the tool 'top' in order to watch which particular process of Wazuh is having that behavior? It could be very useful for us to trace the issue.

Finally, Wazuh 3.2.2 will be released very soon.

Thank you,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/43e0c5c6-8608-457d-ac7a-b0c1b4d1548b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Robert H

unread,

Apr 11, 2018, 4:56:39 PM4/11/18

to Wazuh mailing list

Thanks for the information Chema,

Regarding the messages from ossec-remoted there were a dozen of so old agents that should have been removed before and were still running. I have requested they remove them and will monitor the log to verify that warning goes away.

For the csyslogd memory issue. The osscec-csyslogd will use all memory until it stopped the csyslogd service from running.

////////////////////////////////////////////////////////

First thing this morning, I noticed the RAM usage was very low and the service was stopped.

# free -m

total used free shared buffers cached

Mem: 16080 1105 14975 0 1 84

-/+ buffers/cache: 1019 15061

Swap: 1999 277 1722

# managerstatus

wazuh-clusterd is running...

ossec-monitord is running...

ossec-logcollector is running...

ossec-remoted is running...

ossec-syscheckd is running...

ossec-analysisd is running...

ossec-maild not running...

ossec-execd not running...

wazuh-modulesd is running...

wazuh-db is running...

ossec-csyslogd: Process 30485 not used by ossec, removing...

ossec-csyslogd not running...

ossec-authd is running...

/////////////////////////////////////////////////////////////

After restart, 8:25am

# managerstatus

wazuh-clusterd is running...

ossec-monitord is running...

ossec-logcollector is running...

ossec-remoted is running...

ossec-syscheckd is running...

ossec-analysisd is running...

ossec-maild not running...

ossec-execd not running...

wazuh-modulesd is running...

wazuh-db is running...

ossec-csyslogd is running...

ossec-authd is running...

# free -m

total used free shared buffers cached

Mem: 16080 1069 15011 0 2 108

-/+ buffers/cache: 958 15122

Swap: 1999 267 1732

///////////////////////////////////////////

After 1.5 hours

# free -m

total used free shared buffers cached

Mem: 16080 3972 12108 0 13 407

-/+ buffers/cache: 3551 12529

14431 ossecm 20 0 2514m 2.4g 748 S 0.0 15.5 0:06.85 ossec-csyslogd <-- %MEM at 15.5% from top

/////////////////////////////////////////

After 3 hours

# free -m

total used free shared buffers cached

Mem: 16080 7264 8816 0 13 454

-/+ buffers/cache: 6797 9283

Swap: 1999 267 1732

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

14431 ossecm 20 0 5629m 5.5g 748 S 0.0 34.9 0:15.72 ossec-csyslogd <-- %MEM upto 34.9%

///////////////////////////////////////////

After 5 hours

# free -m

total used free shared buffers cached

Mem: 16080 9940 6140 0 13 499

-/+ buffers/cache: 9427 6653

Swap: 1999 266 1733

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

14431 ossecm 20 0 8282m 8.1g 748 S 0.0 51.4 0:23.82 ossec-csyslogd <-- %MEM upto 51.4%

1607 root 20 0 1517m 780m 2828 S 2.3 4.9 1384:59 java

14461 ossec 20 0 74696 56m 1052 S 0.7 0.4 2:35.16 ossec-analysisd

As you can see the memory usage increases over time until it's maxed out and then stops the csyslogd service. Can you help me identify why it's happening and/or what is causing this?

Regards,

Robert

Chema Martinez

unread,

Apr 12, 2018, 6:34:39 AM4/12/18

to Robert H, Wazuh mailing list

Hi Robert,

I am trying to reproduce the issue.

Could you show me the revision of the Wazuh manager you have installed? You can see it in the file /etc/ossec-init.conf.

In addition, I think it would be interesting to see the configuration related to csyslogd from your ossec.conf.

Regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/a69fa4cf-e559-4303-bf8c-c3b60d98a083%40googlegroups.com.

Robert H

unread,

Apr 12, 2018, 11:04:33 AM4/12/18

to Wazuh mailing list

Hi Chema,

We are sending syslog out to itself, to a ArcSight smartconnector on the same system.

<syslog_output>

</syslog_output>

Version Information:

DIRECTORY="/<custom_path>/ossec"

NAME="Wazuh"

VERSION="v3.2.2"

REVISION="3221"

DATE="Mon Apr 2 12:49:06 PDT 2018"

TYPE="server"

/////////////////////////////////////////

One other piece of information if relevant. We made custom modifications to CEF mappings and inserted them into the install of the manager. To do this we modified the alert.c file and used 2 custom parser files during the install. Do you think this is relevant to this issue?

Regards,

Robert

Chema Martinez

unread,

Apr 16, 2018, 9:21:42 AM4/16/18

to Robert H, Wazuh mailing list

Hi Robert,

Sorry for the late response.

I guess that the CEF mappings you are using are the same that added to this PR: https://github.com/wazuh/wazuh/pull/422

I have tested that branch and, as you reported, I have found a memory leak when parsing alerts in CEF format. Here you can see a piece of the valgrind output:

==12149== 52,756,040 (1,232,896 direct, 51,523,144 indirect) bytes in 5,504 blocks are definitely lost in loss record 569 of 569
==12149== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12149== by 0x4C2FDEF: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12149== by 0x554354F: re_compile_internal (regcomp.c:747)
==12149== by 0x554B27E: regcomp (regcomp.c:498)
==12149== by 0x404B7B: findMatch (parser.c:140)
==12149== by 0x404AC5: parseString (parser.c:124)
==12149== by 0x4066C7: OS_Alert_SendSyslog (alert.c:142)
==12149== by 0x405AB9: OS_CSyslogD (csyslogd.c:118)
==12149== by 0x4055CD: main (main.c:182)

The memory leak is related to the regex expressions used by the parser, that they weren`t being freed after using them. I already have requested a change in that PR which should solve the memory leak.

Please let me know if it solves the issue and if not, we can look for another solution.

I hope it helps,

Best regards.

Chema Martinez | IT Engineer — Wazuh, Inc.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f0ef73f3-449b-45fd-913c-9eb204a666b5%40googlegroups.com.

Robert H

unread,

Apr 16, 2018, 5:31:34 PM4/16/18

to Wazuh mailing list

Thanks Chema,

That is very helpful. We will correct the cef mappings we created and re-apply them to the wazuh manager.