Wazuh cannot restart - Process broked

508 views
Skip to first unread message

Luke Lee

unread,
Jan 5, 2020, 10:28:25 PM1/5/20
to Wazuh mailing list
Hi all, I am facing this problem. 

I got this error :: 
ERROR : sys_rpm_packages(): ERROR: sys_rpm_packages(): Failed to open database '/var/lib/rpm/Packages'   


Distributor ID: Ubuntu
Description:    Ubuntu 16.04.6 LTS
Release:        16.04
Codename:       xenial

Wazuh version 3.9.3

Luke Lee

unread,
Jan 5, 2020, 10:32:39 PM1/5/20
to Wazuh mailing list
The other errors found on /var/ossec/logs:

- wazuh-modulesd:download[36207] wm_download.c:165 at wm_download_dispatch(): DEBUG: Downloading 'https://access.redhat.com/labs/securitydataapi/cve.json?after=2010-01-01&per_page=1000&page=10' to '/var/ossec/tmp/cve'

- wazuh-modulesd:vulnerability-detector[36207] wm_vuln_detector.c:2117 at wm_vuldet_run_update(): DEBUG: (5451): Red Hat Enterprise Linux OVAL has been updated correctly.

Why there is some red-hat stuff here? 

Tomás Turina

unread,
Jan 6, 2020, 10:01:32 AM1/6/20
to Luke Lee, Wazuh mailing list

Hi Luke,

 

The logs from vulnerability detector related to Red Hat are because it is necessary to download a feed to populate the CVE database. This feed is not the same for each OS, so depending on the platform that you are running Wazuh, the URL to get the CVE json file will change.

 

Regarding the log from the first email you sent, it is an error related to the Syscollector module. One of Syscollector’s inventories collects the packages installed in the system, which in your case (Red Hat) they are stored in the path ‘/var/lib/rpm/Package’. Syscollector is trying to open that DB but it is failing. I think it can be a problem with the permissions to open that DB file.

Did you try to restart the Wazuh service as root? Did you check if the DB file is corrupted?

 

Regards,

 

Tomás.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/93f90638-702a-42d3-b8d8-015df8dd747c%40googlegroups.com.

 

Luke Lee

unread,
Jan 7, 2020, 2:30:38 AM1/7/20
to Wazuh mailing list
Hi, thanks for replying. 

Did you try to restart the Wazuh service as root? 
Which are the services ? all wazuh-manager, wazuh-api & wazuh-agent? 

Did you check if the DB file is corrupted?
May I know how can I check on this? 


Below are the items which found not running. 

wazuh-clusterd not running...

ossec-logcollector not running...

ossec-remoted not running...

ossec-syscheckd not running...

ossec-agentlessd not running...

ossec-integratord not running...

ossec-dbd not running...

ossec-csyslogd not running...

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

Luke Lee

unread,
Jan 7, 2020, 2:32:47 AM1/7/20
to Wazuh mailing list
but strange enough, 

I am running it on Ubuntu server. Why I am getting this red-hat stuff? 

Tomás Turina

unread,
Jan 7, 2020, 8:54:22 AM1/7/20
to Luke Lee, Wazuh mailing list

Hi Luke,

 

Actually, there are some feeds for vulnerability detector that the manager will download no matter the OS. This is because they are necessary to detect possible problems in the agents connected to it, and these agents can be running on different OS (Ubuntu, RedHat, Windows, etc.).

 

These feeds can be enabled/disabled by configuration in the vulnerability-detector section of ossec.conf:

 

<vulnerability-detector>

    <enabled>no</enabled>

    <interval>5m</interval>

    <ignore_time>6h</ignore_time>

    <run_on_start>yes</run_on_start>

 

    <provider name="canonical">

      <enabled>no</enabled>

      <os>precise</os>

      <os>trusty</os>

      <os>xenial</os>

      <os>bionic</os>

      <update_interval>1h</update_interval>

    </provider>

 

    <provider name="debian">

      <enabled>no</enabled>

      <os>wheezy</os>

      <os>stretch</os>

      <os>jessie</os>

      <os>buster</os>

      <update_interval>1h</update_interval>

    </provider>

 

    <provider name="redhat">

      <enabled>no</enabled>

      <update_from_year>2010</update_from_year>

      <update_interval>1h</update_interval>

    </provider>

 

    <provider name="nvd">

      <enabled>no</enabled>

      <update_from_year>2010</update_from_year>

      <update_interval>1h</update_interval>

    </provider>

 

  </vulnerability-detector>

 

So, if you check your configuration file, it is almost for sure that the provider “redhat” is enabled. That’s why your ubuntu manager is downloading those feeds for RedHat and it shouldn’t be any problem since they are necessary for RedHat agents connected to it.

 

On the other topic, as I can see most of the daemons are not running. To restart all of them, you can use the following command (this is the same for Wazuh manager and Wazuh agent):

 

# /var/ossec/bin/ossec-control restart

 

To get you packages database fixed, I think that this link could be useful for you: https://unix.stackexchange.com/questions/198703/yum-errorrpmdb-open-failed

 

Please, let me know if this fixes your problem.

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/38e1369b-0bc4-4114-b249-7dd810894db1%40googlegroups.com.

 

Luke Lee

unread,
Jan 8, 2020, 9:13:11 PM1/8/20
to Wazuh mailing list
Hi Tomás, 

Thanks for your reply. I would like to seek your further assistance. 

I notice that my server seems running ok, but my agent cannot connect to my server through the port 1514.

On Server side: 
- firewall has been opened. ports: 1514/udp 1514/tcp 

 /var/ossec/bin/ossec-control status
wazuh-clusterd not running...
wazuh-modulesd is running...
ossec-monitord not running...
ossec-logcollector not running...
ossec-remoted not running...
ossec-syscheckd not running...
ossec-analysisd is running...
ossec-maild is running...
ossec-execd is running...
wazuh-db is running...
ossec-authd is running...
ossec-agentlessd not running...
ossec-integratord not running...
ossec-dbd not running...
ossec-csyslogd not running...


Is there anything you can advise me on this? Is it due to the firewall ?

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

Luke Lee

unread,
Jan 9, 2020, 2:45:20 AM1/9/20
to Wazuh mailing list
On top of that there are few errors found. Please advise how to troubleshoot this. 

wazuh-modulesd:download[14986] wm_download.c:165 at wm_download_dispatch(): DEBUG: Downloading 'https://access.redhat.com/labs/securitydataapi/cve.json?after=2010-01-01&per_page=1000&page=19' to '/var/ossec/tmp/cve'

wazuh-modulesd:syscollector[14986] syscollector_linux.c:432 at sys_rpm_packages(): ERROR: sys_rpm_packages(): Failed to open database '/var/lib/rpm/Packages': Invalid argument

wm_osquery_monitor.c:353 at Execute_Osquery(): ERROR: Couldn't execute osquery (osqueryd). Check file and permissions. Sleeping for 10 minutes.

wazuh-modulesd:syscollector[19469] syscollector_linux.c:351 at sys_packages_linux(): DEBUG: Starting installed packages inventory.
wazuh-modulesd:syscollector[19469] syscollector_linux.c:432 at sys_rpm_packages(): ERROR: sys_rpm_packages(): Failed to open database '/var/lib/rpm/Packages': No such file or directory
wazuh-modulesd:syscollector[19469] syscollector_linux.c:362 at sys_packages_linux(): ERROR: Unable to get rpm packages due to: No such file or directory


I discover also when I restart the "service wazuh-manager restart" then the whole wazuh cant work. 

Tomás Turina

unread,
Jan 9, 2020, 8:26:36 AM1/9/20
to Luke Lee, Wazuh mailing list

Hi Luke,

 

As I can see from the output of the ossec-control status command, there are some processes that should be running but they aren’t.

 

Could you please attach the full log so I can check why they aren’t running? Before that, I recommend you to restart all the daemons by running the command:

 

#/var/ossec/bin/ossec-control restart

 

Also, by default the agents connect to the server through port 1515. Following the simple registration service, to get your agent connected to the server you should run (in the agent side):

 

#/var/ossec/agent-auth -m your_server_ip -p 1514

 

Here you can check the full options of the agent-auth tool: https://documentation.wazuh.com/3.10/user-manual/reference/tools/agent-auth.html

 

Please, make sure that the firewall is not blocking connections to port 1514 on your server side.

Regarding the error messages, all of them are from wazuh-modules daemon, so they aren’t the reason why you can’t connect your agent to the manager.

In order to fix them, please make sure that the ‘/var/lib/rpm/Packages’ file exists, and if not please regenerate it following this instructions: https://unix.stackexchange.com/questions/198703/yum-errorrpmdb-open-failed

Please, let me know if this information helps you.

ossec-execd is running...

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/cfe434c1-bc0d-4073-b544-cfdd23b3831a%40googlegroups.com.

 

Message has been deleted

Luke Lee

unread,
Jan 9, 2020, 10:43:40 PM1/9/20
to Wazuh mailing list
Hi Tomas, 

Thanks for your reply and guide. 

May I know in order to get it run, what are the processes needed to run?

I have attached the ossec.log for your reference. 

For the FW matter, I have opened those ports, but I am unable to telnet. 

# sudo firewall-cmd --zone=public --list-ports
1514/udp 5910/tcp 55000/tcp 1514/tcp 55000/udp 8080/tcp

# telnet ip 5910
Trying ip...
Connected to ip.
Escape character is '^]'.
^CConnection closed by foreign host.

# telnet ip 1514
Trying ip...
telnet: connect to address ip: Connection refused

Advise please, is it because I didnt open the port correctly? sudo firewall-cmd --zone=public --permanent --add-port=1514/tcp

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

ossec.log

Luke Lee

unread,
Jan 10, 2020, 1:10:56 AM1/10/20
to Wazuh mailing list
Strange enough, 

Just back from lunch and I notice my wazuh is working and receiving data. May I know why is it happening? Anyone can help me to find out how can this happened and what actually causes these. 

Tomás Turina

unread,
Jan 10, 2020, 8:40:10 AM1/10/20
to Luke Lee, Wazuh mailing list

Hi Luke,

 

Here you can find a description of all the Wazuh daemons: https://documentation.wazuh.com/3.10/user-manual/reference/daemons/index.html

 

In order to get your manager run and receive data from your agents, you need the following processes up and running:

  • ossec-authd: necessary to interchange keys between manager and agents in a secure way.
  • ossec-remoted: this process is in charge of receive data from the agents connected to the manager.
  • ossec-analysisd: it analyzes, decodes and matches data with the rules in order to generate alerts.
  • wazuh-db: this process stores data in DB.

 

From the logs you shared I couldn’t find an error related to these processes, but the process ossec-remoted wasn’t running. I guess that after you restarted the processes with the command that I shared, all the processes listed above were able to get up and run and that’s why your manager is now receiving data from your agents.

 

Let me know if your problem is already solved and don’t hesitate to contact if any problem persists.

 

Kind regards,

Tomás.

 

From: Luke Lee
Sent: Friday, January 10, 2020 3:10 AM
To: Wazuh mailing list
Subject: Re: Wazuh cannot restart - Process broked

 

Strange enough, 

ossec-monitord not running...

ossec-logcollector not running...

ossec-remoted not running...

ossec-syscheckd not running...

ossec-analysisd is running...

ossec-maild is running..

ossec-execd is running...

wazuh-db is running..

ossec-authd is running...

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/de371a53-63e9-4b34-b383-042de9b110a6%40googlegroups.com.

 

Luke Lee

unread,
Jan 14, 2020, 3:28:28 AM1/14/20
to Wazuh mailing list
Hi Tomas,

Thanks for your reply. Today I notice the following scenario. 


ossec-remoted is not running. 

Previously was running on UDP but at one point after I restart the service, it cannot work at all. 

Then we try to switch to TCP by applying a few fixes on the app, then TCP can work for a short period of time, after-which we tried to restart the app, then the Application unable to identify the port, and similarly the app went down again. 

Please advise is there a problem for the application to identify which active port it suppose to pick ?

Thanks ! 

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

Tomás Turina

unread,
Jan 14, 2020, 10:56:45 AM1/14/20
to Luke Lee, Wazuh mailing list

Hi Luke,

 

The problem you face isn’t related to the ports you use. After analyzing the log you sent, we realized that remoted and some of the processes aren’t running because the can’t connect to the process analysisd.

 

Analysisd is the process that receives all the messages from the rest of the processes from a socket located in the path /var/ossec/queue/ossec/queue. In you case, analysisd takes a lot of time to read and load all its initial configuration before starting this socket. That’s why, in the log we can see these lines:

 

2020/01/10 09:47:47 rootcheck[46120] rootcheck.c:250 at rootcheck_connect(): CRITICAL: (1211): Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..

2020/01/10 09:47:47 ossec-logcollector[46135] main.c:183 at main(): CRITICAL: (1211): Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..

2020/01/10 09:47:48 ossec-remoted[46127] secure.c:117 at HandleSecure(): CRITICAL: (1211): Unable to access queue: '/queue/ossec/queue'. Giving up..

2020/01/10 09:47:57 ossec-monitord[46139] monitord.c:63 at Monitord(): CRITICAL: (1211): Unable to access queue: '/queue/ossec/queue'. Giving up..

 

As you can see, those processes can’t connect to the socket and they are exiting.

 

In a normal case, analysisd should take less than 1 second to load all its configuration, but in your case it takes over 45 seconds. In that time, the rest of the processes give up to connect to the socket and exit.

 

In order to understand why it takes so much time to start, could you please share the specifications (CPUs, RAM, etc) of the machine where you are running Wazuh manager?

 

Also, a quick fix you can try is edit the file /var/ossec/bin/ossec-control and add a sleep after starting analysisd but before starting the rest of the processes. After this, you have to restart Wazuh manager with the command:

 

# /var/ossec/bin/ossec-control restart

 

If you get all the processes running, you should get this output:

 

root@tomas-VirtualBox:/var/ossec/bin# ./ossec-control status

wazuh-clusterd not running...

wazuh-modulesd is running...

ossec-monitord is running...

ossec-logcollector is running...

ossec-remoted is running...

ossec-syscheckd is running..

ossec-analysisd is running...

ossec-maild not running...

ossec-execd is running...

wazuh-db is running...

ossec-authd is running...

ossec-agentlessd not running...

ossec-integratord not running...

ossec-dbd not running...

ossec-csyslogd not running...

root@tomas-VirtualBox:/var/ossec/bin#

 

We also have an issue in our backlog to don’t make the processes give up when can’t connect to the socket and, instead, make them retry the connecting every certain period of time, so in future versions this issue should no longer appear.

 

I hope this information helps you.

ossec-authd is running...

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fb3759e2-d93f-48a3-b3ee-351c1833f675%40googlegroups.com.

 

Luke Lee

unread,
Jan 15, 2020, 10:47:15 PM1/15/20
to Wazuh mailing list
Hi Tomas, 

Thanks for your sharing, I manage to get it run again. Just need to manually start those processes which are not running. 

Million thanks ! 


Btw I have a question to ask, I notice that the Wazuh docker has grown quite significantly. From 60% to 70% plus. May I know how do I manage the size of the docker and prevent it from breaking ? 

Thanks 

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

Tomás Turina

unread,
Jan 17, 2020, 3:11:22 PM1/17/20
to Luke Lee, Wazuh mailing list

Hi Luke,

 

The Wazuh documentation says that it is recommended to assign 6GB to Docker host preferences:
https://documentation.wazuh.com/3.10/docker/wazuh-container.html#container-memory

 

Let me know if this answers your question.

telnet: connect to address ip: Connection refused

 

Is there anything you can advise me on this? Is it due to the firewall ?

but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

 

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fc6b15c9-352c-4903-968a-37a5c31e59e8%40googlegroups.com.

 

Message has been deleted

Luke Lee

unread,
Jan 20, 2020, 2:42:53 AM1/20/20
to Wazuh mailing list
Hi, thanks for your reply. 

I found out that my server already with this configuration. sysctl -w vm.max_map_count=262144

By restarting the filebeat process / docker it actually helps to clear up the memory. 


but strange enough, 

 

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

Luke Lee

unread,
Jan 28, 2020, 10:05:15 PM1/28/20
to Wazuh mailing list
Hi, 

I need some advise. For the vulnerability page, I notice that the visualization graph stop displaying for a short while but my logs is still taking in data. May I know what causes this and how can I check the root cause? 

1.PNG

But after I restarted the restart the "/var/ossec/bin/ossec-control" and wazuh-manager, it will resume awhile. 
Reply all
Reply to author
Forward
0 new messages