ossec-remoted not starting with wazuh-manager (start/restart) command

[David Drake]

I've had an OSSEC server running for quite some time on CentOS 7, using the Wazuh repository (yum install wazuh-manager).  

Last night, I was alerted the ossec-remoted process was not running, so I issued a service wazuh-manager restart command and all of the services started except for ossec-remoted.  When I manually issue /var/ossec/bin/ossec-control restart, ossec-remoted starts without any issues.  I can run ossec-remoted in foreground / debug without any issue as well after the wazuh-manager service is started.  

Has anyone else had this issue?  It seems to be targeted to just starting/restarting the service using service command.  

[David Drake]

Forgot to include output:

./ossec-control restart

Deleting PID file '/var/ossec/var/run/ossec-remoted-10367.pid' not used...

Killing ossec-monitord .. 

Killing ossec-logcollector .. 

ossec-remoted not running ..

Killing ossec-syscheckd .. 

Killing ossec-analysisd .. 

ossec-maild not running ..

Killing ossec-execd .. 

Killing wazuh-modulesd .. 

Wazuh v2.0 Stopped

Starting Wazuh v2.0 (maintained by Wazuh Inc.)...

Started wazuh-modulesd...

Started ossec-execd...

Started ossec-analysisd...

Started ossec-logcollector...

Started ossec-remoted...

Started ossec-syscheckd...

Started ossec-monitord...

Completed.

[Victor Fernandez]

Hi David,

sorry about that. Wazuh v2.0 includes many fixes for bugs in Remoted. One of them made Remoted crash when keys were added or removed while Remoted was running. But currently there are no known bugs in Remoted.

Could you describe the problem? I mean, how long Remoted has run before it crashed, if you added or removed agents while Remoted was running or if you are able to reproduce the problem.

Kind regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9593e1c1-f90f-413b-9345-2b8e09f26b94%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Victor M. Fernandez-Castro

IT Security Engineer

Wazuh Inc.

[David Drake]

I can reproduce the issue 100% of the time.  If I issue a service wazuh-manager restart, ossec-remoted will not start successfully.  If I run /var/ossec/bin/ossec-control restart, ossec-remoted works great.  New agents are added often because I use ossec-authd for automating registration.  At this time, all I am trying to ensure is the ability to use the service command to start OSSEC, which suddenly broke.  

For a little more background, this issue first appeared after the number of agents exceeded 1024, which reached the OS limit for open files.  When I raised the limit by issuing command (ulimit -n 4096), ossec-remoted would start using /var/ossec/bin/ossec-control BUT remoted still won't start using service wazuh-manager start.

[David Drake]

For systemctl start wazuh-manager, it appears it tries to start ossec-remoted, but after all other services start, ossec-remoted isn't running as process.

● wazuh-manager.service - SYSV: Starts and stops OSSEC HIDS (Host Intrusion Detection System)

   Loaded: loaded (/etc/rc.d/init.d/wazuh-manager; bad; vendor preset: disabled)

   Active: active (running) since Wed 2017-08-02 12:18:34 EDT; 4s ago

     Docs: man:systemd-sysv-generator(8)

  Process: 12347 ExecStop=/etc/rc.d/init.d/wazuh-manager stop (code=exited, status=0/SUCCESS)

  Process: 12435 ExecStart=/etc/rc.d/init.d/wazuh-manager start (code=exited, status=0/SUCCESS)

   CGroup: /system.slice/wazuh-manager.service

           ├─12465 /var/ossec/bin/wazuh-modulesd

           ├─12472 /var/ossec/bin/ossec-execd

           ├─12476 /var/ossec/bin/ossec-analysisd

           ├─12480 /var/ossec/bin/ossec-logcollector

           ├─12491 /var/ossec/bin/ossec-syscheckd

           └─12495 /var/ossec/bin/ossec-monitord

Aug 02 12:18:30 ossec01 wazuh-manager[12435]: Starting Wazuh v2.0 (maintained by Wazuh Inc.)...

Aug 02 12:18:31 ossec01 wazuh-manager[12435]: Started wazuh-modulesd...

Aug 02 12:18:31 ossec01 wazuh-manager[12435]: Started ossec-execd...

Aug 02 12:18:31 ossec01 wazuh-manager[12435]: Started ossec-analysisd...

Aug 02 12:18:31 ossec01 wazuh-manager[12435]: Started ossec-logcollector...

Aug 02 12:18:31 ossec01 wazuh-manager[12435]: Started ossec-remoted...

Aug 02 12:18:32 ossec01 wazuh-manager[12435]: Started ossec-syscheckd...

Aug 02 12:18:32 ossec01 wazuh-manager[12435]: Started ossec-monitord...

Aug 02 12:18:34 ossec01 wazuh-manager[12435]: Completed.

And for a output of ps right after starting service.

ps -efa | grep ossec

ossec     1019     1  0 Jul30 ?        00:00:43 /bin/node /var/ossec/api/app.js

root     12465     1  8 12:18 ?        00:00:00 /var/ossec/bin/wazuh-modulesd

root     12472     1  0 12:18 ?        00:00:00 /var/ossec/bin/ossec-execd

ossec    12476     1  0 12:18 ?        00:00:00 /var/ossec/bin/ossec-analysisd

root     12480     1  0 12:18 ?        00:00:00 /var/ossec/bin/ossec-logcollector

root     12491     1  0 12:18 ?        00:00:00 /var/ossec/bin/ossec-syscheckd

ossec    12495     1  0 12:18 ?        00:00:00 /var/ossec/bin/ossec-monitord

root     12506 11140  0 12:18 pts/0    00:00:00 grep --color=auto ossec

[Victor Fernandez]

Hi David,

thanks for the information. Ee come to reproduce this problem and are thinking the best solution for it.

It seems that CentOS 7 sets a maximum of 1024 opened files per process for services started with Systemd. But this value cannot be changed though /etc/security/limits.conf or /etc/systemctl.conf.

We only get to start Remoted properly adding this line to function start() at /etc/init.d/wazuh-manager:

start() {

    echo -n "Starting OSSEC: “

    ulimit -n 10000

    ${DIRECTORY}/bin/ossec-control start > /dev/null

    RETVAL=$?

    if [ $RETVAL -eq 0 ]; then

        success

    else

        failure

    fi

    echo

    return $RETVAL

}

We have to check if this change affects to the Wazuh service only. We will change these old SysVinit scripts for modern Systemd init files, that are able to configure the maximum opened files limit per service. 

However in the meantime we’ll try to find a better solution. We’ll keep you informed.

Thank you for reporting.

Best regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2ea0526e-529b-455f-8a66-364efa86e0ff%40googlegroups.com.

[chema.m...@wazuh.com]

Hi David,

As Victor said, modifying wazuh-manager script is a good solution until we change these scripts to Systemd init files.

We are verified that "ulimit -n 10000" line does not affect to other services, so it won't be a problem in that sense.

Finally, I attach to this message the modified script.

We hope it will prove useful.
Best regards.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[David Drake]

Good find, tested and confirmed to work.  I figured it had to do something with the init.d script but didn't think to put ulimit in the service.  Thanks!