ossec-control restart doesn't stop logcollector process under OpenBSD 6.6
Carlos Lopez
Hi all,
I am using agent centralized configuration for my OpenBSD agents (using version 3.12 for in all components). When I make a modification in the agent.conf file from the manager, the file is correctly replicated in the OpenBSD agent. But when ossec-control tries to restart the services in the agents, it is not able to kill the previous logcollector process and lifts a new one leaving the old one running.
As an example. These are the processes running when ossec-control tries to restart agent process (it spends 1 min more or less to do this):
23224 ?? DV 0:01.09 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
27931 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
41621 ?? DV 0:00.40 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
5279 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
25846 ?? S 0:00.03 /var/ossec/bin/ossec-execd
65304 ?? S 0:00.36 /var/ossec/bin/ossec-agentd
30300 ?? S 0:02.11 /var/ossec/bin/ossec-syscheckd
64043 ?? DV 0:00.41 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
83237 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
10851 ?? Sp 0:00.50 /bin/sh /var/ossec/bin/ossec-control restart
75321 ?? Sp 0:00.00 sleep 0.1
As you can see, there is other logcollector processes running … And when it's done, ossec-control sets up a new logcollection process while keeping the old ones:
23224 ?? DV 0:01.09 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
27931 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
41621 ?? DV 0:00.40 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
5279 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
52327 ?? S 0:02.83 /usr/local/bin/tor
64043 ?? DV 0:00.41 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
83237 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
80646 ?? S 0:00.01 /var/ossec/bin/ossec-execd
62259 ?? S 0:00.07 /var/ossec/bin/ossec-agentd
40170 ?? S 0:02.02 /var/ossec/bin/ossec-syscheckd
13961 ?? IV 0:00.08 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
68745 ?? I 0:00.03 /var/ossec/bin/wazuh-modulesd
81790 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
However, this behavior does not appear on FreeBSD hosts. It seems a bug, do you agree?
--
Regards,
C. L. Martinez
Carlos Lopez
________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of Carlos Lopez <clo...@outlook.com>
Sent: 02 April 2020 09:52
To: wa...@googlegroups.com
Subject: ossec-control restart doesn't stop logcollector process under OpenBSD 6.6
Hi all,
--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com<mailto:wazuh+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/0857031C-94E7-4EAF-8261-A78CCD7F5E1D%40outlook.com<https://groups.google.com/d/msgid/wazuh/0857031C-94E7-4EAF-8261-A78CCD7F5E1D%40outlook.com?utm_medium=email&utm_source=footer>.
Chema Martinez
- The OpenBSD version.
- How many agents in OpenBSD do you have deployed? Is that happening in all of them?
- Did you upgrade the Wazuh agent to 3.12, or was it a clean installation? If so, did you notice the same behavior in previous versions of Wazuh?
- The agent's ossec.log file. Does it contain any error messages about the Logcollector?
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/DB8PR10MB35951AE4390D5399C6C45E67DBC70%40DB8PR10MB3595.EURPRD10.PROD.OUTLOOK.COM.
Chema Martinez
Carlos Lopez
Good morning Chema,
My answers:
- All OpenBSD agents are 6.6 amd64.
- At this moment, I have 6 agents based in OpenBSD 6.6 (but I expect to install more agents in following days). In all these agents, the behavior is the same and problem is the same.
- Problem appears in a clean 3.12 installation, upgrading from 3.11.4 and doing a 3.11.4 install.
- No errors in ossec.log about logcollector. An example:
2020/04/05 05:42:19 ossec-execd: INFO: No option <ca_store> defined. Using Wazuh default CA (/var/ossec/etc/wpk_root.pem).
2020/04/05 05:42:19 ossec-execd: INFO: Started (pid: 26753).
2020/04/05 05:42:19 ossec-agentd: INFO: Using notify time: 60 and max time to reconnect: 300
2020/04/05 05:42:19 ossec-syscheckd: INFO: (6001): File integrity monitoring disabled.
2020/04/05 05:42:19 ossec-agentd: INFO: Version detected -> OpenBSD |kirkwall.lab.uxdom.org |6.6 |GENERIC#7 |amd64 [BSD|bsd: 6.6] - Wazuh v3.11.4
2020/04/05 05:42:19 ossec-agentd: INFO: (1410): Reading authentication keys file.
2020/04/05 05:42:19 ossec-agentd: INFO: Using AES as encryption method.
2020/04/05 05:42:19 ossec-agentd: INFO: Started (pid: 77755).
2020/04/05 05:42:19 ossec-agentd: INFO: Server IP Address: 172.22.55.1
2020/04/05 05:42:19 ossec-agentd: INFO: Trying to connect to server (172.22.55.1:1575/udp).
2020/04/05 05:42:19 wazuh-modulesd: INFO: Process started.
2020/04/05 05:42:19 sca: INFO: Module started.
2020/04/05 05:42:19 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/sca_unix_audit.yml'
2020/04/05 05:42:19 wazuh-modulesd:syscollector: INFO: Module started.
2020/04/05 05:42:19 sca: INFO: Starting Security Configuration Assessment scan.
2020/04/05 05:42:19 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/sca_unix_audit.yml'
2020/04/05 05:42:20 ossec-logcollector: INFO: Monitoring output of command(360): df -P
2020/04/05 05:42:20 ossec-logcollector: INFO: Monitoring full output of command(360): last -n 5
2020/04/05 05:42:20 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/authlog'.
2020/04/05 05:42:20 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/messages'.
2020/04/05 05:42:20 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/secure'.
2020/04/05 05:42:20 ossec-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'.
2020/04/05 05:42:20 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/xferlog'.
2020/04/05 05:42:20 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/maillog'.
2020/04/05 05:42:20 ossec-logcollector: INFO: Monitoring full output of command(360): netstat -f inet -ln -p tcp | sed 1,2d | awk '{print $4}' | sort -u
2020/04/05 05:42:20 ossec-logcollector: INFO: Monitoring full output of command(360): netstat -f inet -ln -p udp | sed 1,2d | awk '{print $4}' | sort -u
2020/04/05 05:42:20 ossec-logcollector: INFO: Started (pid: 72378).
2020/04/05 05:42:20 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2020/04/05 05:42:23 rootcheck: INFO: Started (pid: 3507).
2020/04/05 05:42:29 ossec-agentd: INFO: (4102): Connected to the server (172.22.55.1:1575/udp).
2020/04/05 05:42:30 wazuh-modulesd:syscollector: WARNING: Opened ports inventory is not available for this OS version.
2020/04/05 05:42:30 wazuh-modulesd:syscollector: WARNING: Running processes inventory is not available for this OS version.
2020/04/05 05:42:30 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2020/04/05 05:42:32 wazuh-modulesd: INFO: Agent is now online. Process unlocked, continuing...
2020/04/05 05:42:32 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/sca_unix_audit.yml'
2020/04/05 05:42:33 sca: INFO: Security Configuration Assessment scan finished. Duration: 14 seconds.
2020/04/05 05:42:38 rootcheck: INFO: Starting rootcheck scan.
2020/04/05 05:42:45 rootcheck: INFO: Ending rootcheck scan.
--
Regards,
C. L. Martinez
From: Chema Martinez <chema.m...@wazuh.com>
Date: Friday, 3 April 2020 at 19:18
To: Carlos Lopez <clo...@outlook.com>
Cc: "wa...@googlegroups.com" <wa...@googlegroups.com>
Subject: Re: ossec-control restart doesn't stop logcollector process under OpenBSD 6.6
I have already realized you specified the OpenBSD version in the thread title.
I will check it at 6.6. However, the other requested information would be still useful.
Thank you Carlos.
Regards.
|
|
|
Chema Martinez |
|
|
On Fri, Apr 3, 2020 at 7:15 PM Chema Martinez <chema.m...@wazuh.com> wrote:
Hi Carlos,
I have tried to reproduce your issue without success in OpenBSD 6.4. It would be very useful if you provide me the following information:
- The OpenBSD version.
- How many agents in OpenBSD do you have deployed? Is that happening in all of them?
- Did you upgrade the Wazuh agent to 3.12, or was it a clean installation? If so, did you notice the same behavior in previous versions of Wazuh?
- The agent's ossec.log file. Does it contain any error messages about the Logcollector?
Best regards,
Chema.
Chema Martinez
Product Core engineer
The Open Source Security Platform
Chema Martinez
- Run /var/ossec/bin/ossec-control stop.
- Ensure that no process related to wazuh/ossec is up:
- Run ps aux | grep -E "ossec|wazuh"
- Check that all PID files have been removed from /var/ossec/var/run.
- If residual Logcollector processes are still running, kill them manually. After that, ensure that they are been stopped correctly.
- Run /var/ossec/bin/ossec-control start.
- Check that only one process for the logcollector daemon has been launched, and its PID corresponds with the created file at /var/ossec/var/run.
 |
| Chema Martinez Product Core engineer |
 The Open Source Security Platform |
  |
Carlos Lopez
HI Chema,
Running command “ps xa | grep -E “ossec|wazuh” my output is:
root 10570 0.0 0.2 9556 1884 ?? S 6:29AM 0:00.44 /var/ossec/bin/ossec-execd
ossec 14891 0.0 0.5 35060 3732 ?? S 6:29AM 0:03.10 /var/ossec/bin/ossec-agentd
root 87486 0.0 0.3 18112 1984 ?? S 6:29AM 0:02.69 /var/ossec/bin/ossec-syscheckd
root 52864 0.0 0.3 50868 2496 ?? IV 6:29AM 0:00.40 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
root 54405 0.0 0.4 43652 3160 ?? I 6:29AM 0:00.05 /var/ossec/bin/wazuh-modulesd
root 3648 0.0 0.0 50864 316 ?? I 6:35AM 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
root 65725 0.0 0.2 344 1200 p0 S+p 11:20AM 0:00.00 grep -E ossec|wazuh
As you can see one logcollector process is idle, but the other one is idle+suspended during a vfork. If I stop wazuh process with ossec-control, all process stops except:
root 52864 0.0 0.3 50864 2544 ?? DV 6:29AM 0:00.41 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
root 3648 0.0 0.0 50864 316 ?? I 6:35AM 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
root 86919 0.0 0.2 332 1172 p0 S+p 11:25AM 0:00.00 grep -E ossec|wazuh
As you can see here, logcollector processes remains as idle and wait to write to disk+ suspended during a vfork. All wazuh pids are removed when I run “ossec-control stop”:
root@obsdwazuh:~# ls -la /var/ossec/var/run/
total 8
drwxrwx--- 2 root ossec 512 Apr 7 11:25 ./
drwxr-x--- 7 root ossec 512 Apr 7 11:25 ../
On the other hand, ossec-control stop spends 1 minute to kill all processes …. On FreeBSD spends only some seconds …
Running “ossec-control start” all pids are created:
root@obsdwazuh:~# ls -la /var/ossec/var/run/
total 32
drwxrwx--- 2 root ossec 512 Apr 7 11:31 ./
drwxr-x--- 7 root ossec 512 Apr 7 11:30 ../
-rw-r----- 1 ossec ossec 6 Apr 7 11:30 ossec-agentd-17112.pid
-rw-r--r-- 1 ossec ossec 540 Apr 7 11:31 ossec-agentd.state
-rw-r----- 1 root ossec 6 Apr 7 11:30 ossec-execd-57315.pid
-rw-r----- 1 root ossec 5 Apr 7 11:30 ossec-logcollector-3596.pid
-rw-r----- 1 root ossec 6 Apr 7 11:30 ossec-syscheckd-42701.pid
-rw-r----- 1 root ossec 6 Apr 7 11:30 wazuh-modulesd-15414.pid
And, “ps xa | grep -E “ossec|wazuh”” output is:
root 57315 0.0 0.2 9560 1900 ?? S 11:30AM 0:00.01 /var/ossec/bin/ossec-execd
ossec 17112 0.0 0.4 34868 3444 ?? S 11:30AM 0:00.11 /var/ossec/bin/ossec-agentd
root 42701 0.0 0.2 18132 1920 ?? S 11:30AM 0:00.99 /var/ossec/bin/ossec-syscheckd
root 3596 0.0 0.3 50940 2684 ?? S 11:30AM 0:00.16 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
root 15414 0.0 0.5 43944 3580 ?? I 11:30AM 0:00.03 /var/ossec/bin/wazuh-modulesd
--
Regards,
C. L. Martinez
From: Chema Martinez <chema.m...@wazuh.com>
Date: Tuesday, 7 April 2020 at 12:46
To: Carlos Lopez <clo...@outlook.com>
Cc: "wa...@googlegroups.com" <wa...@googlegroups.com>
Subject: Re: ossec-control restart doesn't stop logcollector process under OpenBSD 6.6
Hi again Carlos,
I have been trying to reproduce your error in my testing environment without success. I see most of the processes for Logcollector are idle (I) or in uninterruptible sleep (D) when the normal state for the process is interruptible sleep (S). Can you tell me which command are you running to get that process output?
On the other hand, let me tell you how the agent handles the PIDs of the different daemons. I hope this helps to find out the root of the problem.
When a daemon gets launched, a .pid file is created with the process PID under /var/ossec/var/run. That files are read by the ossec-control tool to get noticed about running processes (ossec-control status) as well as killing the processes (ossec-control stop). To determine the cause of the error, it would be useful to clean your environment and restart the daemons step by step.
- Run /var/ossec/bin/ossec-control stop.
- Ensure that no process related to wazuh/ossec is up:
- Run ps aux | grep -E "ossec|wazuh"
- Check that all PID files have been removed from /var/ossec/var/run.
- If residual Logcollector processes are still running, kill them manually. After that, ensure that they are been stopped correctly.
- Run /var/ossec/bin/ossec-control start.
- Check that only one process for the logcollector daemon has been launched, and its PID corresponds with the created file at /var/ossec/var/run.
Best regards,
Chema.
|
|
Chema Martinez
Product Core engineer
Error! Filename not specified. The Open Source Security Platform
Error! Filename not specified.
Error! Filename not specified.
On Fri, Apr 3, 2020 at 7:15 PM Chema Martinez <chema.m...@wazuh.com> wrote:
Hi Carlos,
I have tried to reproduce your issue without success in OpenBSD 6.4. It would be very useful if you provide me the following information:
- The OpenBSD version.
- How many agents in OpenBSD do you have deployed? Is that happening in all of them?
- Did you upgrade the Wazuh agent to 3.12, or was it a clean installation? If so, did you notice the same behavior in previous versions of Wazuh?
- The agent's ossec.log file. Does it contain any error messages about the Logcollector?
Best regards,
Chema.
Chema Martinez
Product Core engineer
Error! Filename not specified. The Open Source Security Platform
Error! Filename not specified.
Error! Filename not specified.
Chema Martinez
|
| Chema Martinez Product Core engineer |
| The Open Source Security Platform |
|
Carlos Lopez
Hi Chema,
After restarting all wazuh’s processes, it seems only one logcollector is up …. But after some minutes before, appears more logcollector processes. As an example. After rebooting one OpenBSD host:
root@obsdwazuh:~# uptime
7:13AM up 1:09, 1 user, load averages: 0.10, 0.07, 0.04
root@obsdwazuh:~# ps xa
PID TT STAT TIME COMMAND
24643 ?? Zp 0:00.00 (sh)
1 ?? I 0:01.00 /sbin/init
33870 ?? IpU 0:00.01 syslogd: [priv] (syslogd)
9347 ?? Ip 0:00.02 /usr/sbin/syslogd
44103 ?? IU 0:00.00 pflogd: [priv] (pflogd)
48947 ?? Sp 0:00.14 pflogd: [running] -s 256 -i pflog0 -f /var/log/pflog (pflogd)
48975 ?? S<p 0:00.08 ntpd: ntp engine (ntpd)
24574 ?? Ip 0:00.02 ntpd: dns engine (ntpd)
54441 ?? I<pU 0:00.00 /usr/sbin/ntpd
64383 ?? I 0:00.00 /usr/sbin/sshd
27885 ?? Ip 0:00.02 smtpd: queue (smtpd)
82285 ?? Ip 0:00.01 smtpd: pony express (smtpd)
78947 ?? Ip 0:00.01 smtpd: lookup (smtpd)
23472 ?? Ip 0:00.01 smtpd: control (smtpd)
76220 ?? Ip 0:00.01 smtpd: klondike (smtpd)
27161 ?? Ip 0:00.00 /usr/sbin/smtpd
50329 ?? Ip 0:00.01 smtpd: scheduler (smtpd)
66847 ?? S 0:04.24 /usr/local/bin/tor
24140 ?? S 0:00.09 /var/ossec/bin/ossec-execd
65219 ?? S 0:01.09 /var/ossec/bin/ossec-agentd
85887 ?? S 0:01.30 /var/ossec/bin/ossec-syscheckd
85068 ?? IV 0:01.05 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
81715 ?? I 0:00.08 /var/ossec/bin/wazuh-modulesd
48753 ?? Sp 0:00.01 /usr/sbin/cron
5614 ?? I 0:00.00 /var/ossec/bin/ossec-logcollector (ossec-logcollect)
44708 ?? S 0:00.04 sshd: root@ttyp0 (sshd)
90963 p0 Sp 0:00.01 -ksh (ksh)
5528 p0 R+pU 0:00.00 ps -xa
42020 00 I+pU 0:00.01 /usr/libexec/getty std.115200 tty00
22493 C0 I+pU 0:00.00 /usr/libexec/getty std.9600 ttyC0
24216 C1 I+pU 0:00.00 /usr/libexec/getty std.9600 ttyC1
My suspicions are that the problem comes with the definition of the <localfile> options (for an unknown reason, logcollector fork it process and tries to write to disk), but I am not sure …. I any case, It doesn't make sense to have multiple logcollector processes ...
My localfile options are:
<localfile>
<log_format>command</log_format>
<command>df -P</command>
<frequency>360</frequency>
</localfile>
<localfile>
<log_format>full_command</log_format>
<command>last -n 5</command>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/authlog</location>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/messages</location>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/secure</location>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/ossec/logs/active-responses.log</location>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/xferlog</location>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/maillog</location>
</localfile>
--
Regards,
C. L. Martinez
From: Chema Martinez <chema.m...@wazuh.com>
Date: Tuesday, 7 April 2020 at 18:52
To: Carlos Lopez <clo...@outlook.com>
Cc: "wa...@googlegroups.com" <wa...@googlegroups.com>
Subject: Re: ossec-control restart doesn't stop logcollector process under OpenBSD 6.6
Hi Carlos,
After the last start it seems everything fine in your outputs, isn't it?
Do you have located the moment the Logcollector process get duplicated? Is it happening every time the agent is restarted?
Regards,
Chema.
Chema Martinez
That way, the module will use a unique thread to read the monitored files and commands, avoiding the issue. It should not affect the performance in a normal situation. To apply the change in the correct way, follow these steps:logcollector.input_threads=1
- Kill the current Logcollector processes by running:
- # ps aux | grep logcollector
- # kill -9 [PID of each Logcollector process]
- Stop the agent.
- Run the following command:
- # echo "logcollector.input_threads=1" >> /var/ossec/etc/local_internal_options.conf
- Restart the agent.
|
| Chema Martinez Product Core engineer |
| The Open Source Security Platform |
|