ossec-remoted Incorrect message size

[Brendan Reekie]

Hi,

We have 2 servers running Wazuh 3.1.0 manager on AWS EC2 and recently both servers logged a number of messages similar to the following:

>2018/09/07 00:00:12 ossec-remoted: ERROR: Incorrect message size from xxx.xxx.xxx.xxx: expecting 375, got 111

We also observed that all clients disconnected from the server (netstat showed no TCP connections to port 1514).  When checking the status of ossec, ossec-control showed that the ossec-remoted process was no longer running.

On one of the servers I had previously added debug to the /etc/local_internal_options.conf file with option 'remoted.debug=2', however I don't see anything specific in the more detailed log as to the cause of the events.

Would anyone be able to help me understand:

1) It seems strange that both servers encountered the same issue around the same time.  In looking over outside logs I don't see any events to correlate to the ossec logs.  Any suggestions as to what might trigger this particular event?

2) Is this a known issue and would an upgrade resolve this?  I can see in subsequent releases that there is more multi-threading in remoted not sure if this would help out for this situation.

3) Any suggestions for additional debugging/understanding of root cause of this issue?

Thanks in advance!

Brendan

[Brendan Reekie]

I did a bit more digging through code and logs.  It appears that when a message with an incorrect size is detected that the socket is closed.  I'm assuming that a new message will reestablish a new socket.   Looking over the syslog I see messages for "TCP: too many orphaned sockets" and my default set on Ubuntu 16.04.3 is set to 8192.  Given the rate of messages of several thousand per hour its possible that the sockets are not being closed in a timely fashion.

I think still needing to understand the root cause of the 'Incorrect message size' messages.

[Pedro Sánchez]

Hi Brendan,

Thanks for sending the feedback and reporting the issue, may I ask if this keeps happening you in the latest Wazuh versions?

As you mention, we are working hard on multithreading Remoted daemon to increase EPS ingestion ratio (8x), as well at the same time we are a focus on keeping up our standards for stability and reliability.

The error you mention should not bring down the daemon (almost anything should take down the daemon), the error indicates the messages have been corrupted at some point in the channel, TCP header shows a size but real size is different, because we are compressing and encrypting all the packets (blowfish in your version, AES in latest versions) the size is critical to uncompress/decrypt.

Are you Manager behind a load balancer or other kind of proxy?

Please, let us know if this continues happening to you in latest Wazuh version so we can dig in and do more troubleshooting.

Regards,

Pedro.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/049a5d55-218c-409c-be5e-379ee956a514%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.