Agent got disconnected and agent log grows rapidly with socketerr

359 views
Skip to first unread message

Vijayakumar U

unread,
May 21, 2019, 7:10:44 AM5/21/19
to Wazuh mailing list

Hello Team,


One of the agent got disconnected and while checking the logs, found below prints and the log file has grown upto 1.6GB size within a day.


What could be the problem? Also if the log in agent grows this much faster, this could fill the agent machines /var partition soon.


Attached agent's ossec.conf file and let me know if any additional details are required.


{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:44","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:45","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:45","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Connection refused)"}

{"timestamp":"2019/05/17 21:58:45","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Connection refused'."}

{"timestamp":"2019/05/17 21:58:45","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:46","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:46","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:46","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:46","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:58:46","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}


{"timestamp":"2019/05/17 21:59:08","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:08","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:08","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:08","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:09","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:09","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:09","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:09","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:10","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:10","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:10","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}


{"timestamp":"2019/05/17 21:59:14","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:14","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:14","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:14","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:15","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:15","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:15","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:15","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:15","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:15","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:15","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd","level":"error","description":"At wm_sendmsg(): Unable to send message to queue: (Bad file descriptor)"}

{"timestamp":"2019/05/17 21:59:16","tag":"wazuh-modulesd:osquery","level":"error","description":"(1210): Queue '/queue/ossec/queue' not accessible: 'Bad file descriptor'."}

{"timestamp":"2019/05/17 21:59:16","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

{"timestamp":"2019/05/17 21:59:16","tag":"ossec-logcollector","level":"error","description":"socketerr (not available)."}

-- 
Cheers,
Vijay.
ossec.conf

Jesús Ángel González

unread,
May 21, 2019, 7:30:09 AM5/21/19
to Wazuh mailing list

Hi Vijay,

Those messages are symptoms of an ossec-agentd fail.

Here is a simple step to reproduce your issue:

pkill -f agentd

Then, you can see some logs like yours:

2019/05/21 11:23:34 ossec-logcollector: ERROR: socketerr (not available).
2019/05/21 11:23:35 ossec-logcollector: ERROR: socketerr (not available).
2019/05/21 11:23:36 ossec-logcollector: ERROR: socketerr (not available).
2019/05/21 11:23:37 ossec-logcollector: ERROR: socketerr (not available).
2019/05/21 11:23:38 ossec-syscheckd: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/21 11:23:38 ossec-syscheckd: ERROR: (1210): Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2019/05/21 11:23:38 rootcheck: CRITICAL: (1211): Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2019/05/21 11:23:38 ossec-syscheckd: CRITICAL: (1211): Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
2019/05/21 11:23:38 ossec-logcollector: ERROR: socketerr (not available).
2019/05/21 11:23:39 ossec-logcollector: ERROR: socketerr (not available).

We can start looking for the daemons that are up in your agent:

ps aux | grep ossec

or you can use ossec-control:

/var/ossec/bin/ossec-control status

In addition, let’s see if there is a process fail:

grep -iR "segfault" /var/log/

Regards,
Jesús

​

Vijayakumar U

unread,
May 21, 2019, 8:21:48 AM5/21/19
to Jesús Ángel González, Wazuh mailing list
Hi Jesús,

Yes, I could see lot of segfault for ossec-agentd & ossec-execd. What is causing this problem and what is the fix?

The agent machine is CentOS6.6

grep -iR "segfault" /var/log/


/var/log/messages.1:May 20 12:29:41 172 kernel: [5877617.130239] ossec-execd[15296]: segfault at 30 ip 0000003788c0a97a sp 00007ffdd36b0af8 error 6 in libpthread-2.12.so[3788c00000+17000]

/var/log/messages.12:May  9 17:16:34 172 kernel: [4944430.137133] ossec-execd[25194]: segfault at 30 ip 0000003788c0a97a sp 00007fa8b24b8938 error 6 in libpthread-2.12.so[3788c00000+17000]

/var/log/messages.12:May  9 17:55:46 172 kernel: [4946781.927741] ossec-execd[25451]: segfault at 30 ip 0000003788c0a97a sp 00007fef0d563938 error 6 in libpthread-2.12.so[3788c00000+17000]

/var/log/messages.12:May  9 17:55:57 172 kernel: [4946793.200944] ossec-execd[11367]: segfault at 30 ip 0000003788c0a97a sp 00007fc9f78b7938 error 6 in libpthread-2.12.so[3788c00000+17000]

/var/log/messages.7:May 14 19:57:25 172 kernel: [5386081.322838] ossec-execd[11657]: segfault at 30 ip 0000003788c0a97a sp 00007f4ecfe42938 error 6 in libpthread-2.12.so[3788c00000+17000]


/var/log/messages.4:May 17 21:58:02 172 kernel: [5652517.849913] ossec-agentd[15305]: segfault at 0 ip 000000378886eaf1 sp 00007f42e313d4a0 error 4 in libc-2.12.so[3788800000+18a000]


/var/log/messages.6:May 15 19:01:43 172 kernel: [5469139.360372] ossec-execd[22839]: segfault at 30 ip 0000003788c0a97a sp 00007ff1d146f938 error 6 in libpthread-2.12.so[3788c00000+17000]

/var/log/messages.6:May 15 19:50:10 172 kernel: [5472046.569788] ossec-execd[1359]: segfault at 30 ip 0000003788c0a97a sp 00007fb7b1d88938 error 6 in libpthread-2.12.so[3788c00000+17000]


Thanks & Regards,
Vijay.


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5a5cbc28-8e35-4017-8b76-08d3ce30fef2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jesús Ángel González

unread,
May 21, 2019, 9:46:39 AM5/21/19
to Wazuh mailing list
Hello again Vijay,

I was using CentOS 7, let me make some tests under CentOS 6.6 and I'll get back to you. 

And thanks for the detailed reports as always!

Regards,
Jesús

Vijayakumar U

unread,
May 22, 2019, 3:59:12 AM5/22/19
to Jesús Ángel González, Wazuh mailing list
Hello Jesús,

Will wait for your update regarding cause/fix for this issue.
-- 
Thanks & Regards,
Vijay.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

Jesús Ángel González

unread,
May 22, 2019, 4:52:56 AM5/22/19
to Wazuh mailing list

Hi Vijay,

I’m preparing for you two special binaries for the daemons that are crashing in your agent and some bash
lines with gdb so we can see exactly why it’s crashing.

Let me prepare all the stuff and I’ll get back to you very soon.

Regards,
Jesús

​

Jesús Ángel González

unread,
May 22, 2019, 5:36:28 AM5/22/19
to Wazuh mailing list

Hello again Vijay,

Since I could not reproduce your issue I’ve built a binary for ossec-agentd with debug mode enabled. Then we can replace your ossec-agentd
binary on the agent and with the help of gdb we’ll obtain a core dump file. That file gives us details about where, when and why the binary crashes.

Note: assuming your agent is 3.9.0, otherwise let me know and do not continue this guide

Stop the agent.

systemctl stop wazuh-agent

Backup your ossec-agentd binary.

cp /var/ossec/bin/ossec-agentd /backup-folder/ossec-agentd

Download the custom binary.

curl -so ossec-agentd https://s3-us-west-1.amazonaws.com/packages-dev.wazuh.com/debug/yum/tmp-agentd-debug/ossec-agentd

The sha512sum for the binary is: 1b9adc4eb0b3125d8c548a0d8c10d9a0af58b4a15007af0c2b75573977e982b1aa2c3b4696a58a511e3f7dd5784328d6ffdf65aaadd81de0a1406504c0301b3c

Replace your binary with the custom binary.

yes | mv /tmp/ossec-agentd /var/ossec/bin

Install gdb

yum install gdb -y

Restart the agent

service wazuh-agent restart

Attach gdb to ossec-agentd process.

gdb /var/ossec/bin/ossec-agentd $(pidof ossec-agentd) -batch -ex "handle SIGPIPE nostop print" -ex "handle SIGTERM nostop noprint" -ex "continue" -ex "generate-core-file /tmp/agentd.core" -ex "quit"

The above line is a foreground process, keep the CLI opened until it fails. You’ll see a message like this:

Program received signal SIGSEGV, Segmentation fault.

Then, it means it crashed and we have the full information in /tmp/agentd.core. Feel free to use a different location for the .core file.

Sorry about the inconveniences.

Best regards,
Jesús

​

Vijayakumar U

unread,
May 24, 2019, 4:47:27 AM5/24/19
to Jesús Ángel González, Wazuh mailing list
Hi Jesús,

Thanks for the efforts and debug patch. I've replaced the ossec-agentd binary and attached the debugger to the new ossec-agentd's process ID.
Waiting for the crash to happen.

Will update here with agentd.core, if it got crashed.

-- 
Thanks & Regards,
Vijay.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

Jesús Ángel González

unread,
May 24, 2019, 9:41:31 AM5/24/19
to Vijayakumar U, Wazuh mailing list

Hello again Vijay,

It would be very great for us so if your agentd fails again, send us the .core result.

Thanks!

Regards,
Jesús

Jesús Ángel González

unread,
May 30, 2019, 7:13:19 AM5/30/19
to Wazuh mailing list
Hi Vijay,

How is your agent? Did it crash again? Have we got a core dump ready to be inspected by us?

Thanks in advance.

Regards,
Jesús

Vijayakumar U

unread,
May 31, 2019, 3:51:36 AM5/31/19
to Jesús Ángel González, Wazuh mailing list
Hi Jesús,

No luck. The agent didn't crashed again.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

For more options, visit https://groups.google.com/d/optout.


--
Cheers,
Vijay.
Reply all
Reply to author
Forward
0 new messages