3005 - Unexpected error checking the Wazuh API

[Aram]

  Hello Wazuh Community, 

I am trying to troubleshoot an issue with the Wazuh API.  When attempting to access it I receive the "Wazuh API seems to be down" with the following error message "3005 - Unexpected error checking the Wazuh API". 

Wazuh Cluster and Version: 

4 node Elasticsearch cluster

2 node Wazuh manager

Wazuh Version  4.0 

Elasticsearch 7.9.3

Manual API Call

When running the API call manually on the Wazuh Manger or Elastic nodes the response below. 

curl -k -X GET "https://wazuh-mgr1:55000/" -H "Authorization: Bearer $(curl -u wazuh:wazuh -k -X GET 'https://wazuh-mgr1:55000/security/user/authenticate?raw=true')"

response from running above command: 

{"data": {"title": "Wazuh API REST", "api_version": "4.0.1", "revision": 40006, "license_name": "GPL 2.0", "license_url": "https://github.com/wazuh/wazuh/blob/4.0/LICENSE", "hostname": "wazuh-mgr1", "timestamp": "2021-08-04T00:49:56+0000"}, "error": 0}

ossec-control status

$ sudo ./ossec-control status

wazuh-clusterd is running...

wazuh-modulesd is running...

ossec-monitord is running...

ossec-logcollector is running...

ossec-remoted is running...

ossec-syscheckd is running...

ossec-analysisd is running...

ossec-maild not running...

ossec-execd is running...

wazuh-db is running...

ossec-authd is running...

ossec-agentlessd not running...

ossec-integratord not running...

ossec-dbd not running...

ossec-csyslogd not running...

wazuh-apid is running...

API Logs (sanitized): 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 193, in send_request_to_wdb

    response.extend(self._send(request))

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 93, in _send

    return json.loads(data[1], object_hook=WazuhDBConnection.json_decoder)

  File "/var/ossec/framework/python/lib/python3.8/json/__init__.py", line 370, in loads

    return cls(**kw).decode(s)

  File "/var/ossec/framework/python/lib/python3.8/json/decoder.py", line 337, in decode

    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "/var/ossec/framework/python/lib/python3.8/json/decoder.py", line 353, in raw_decode

    obj, end = self.scan_once(s, idx)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 111, in json_decoder

    result[k] = datetime.datetime.strptime(v, '%Y/%m/%d %H:%M:%S')

  File "/var/ossec/framework/python/lib/python3.8/_strptime.py", line 568, in _strptime_datetime

    tt, fraction, gmtoff_fraction = _strptime(data_string, format)

  File "/var/ossec/framework/python/lib/python3.8/_strptime.py", line 352, in _strptime

    raise ValueError("unconverted data remains: %s" %

ValueError: unconverted data remains:  ossec-remoted: WARNING: (1218): Unable to send message to '2200': A message could not be delivered completely. [-1]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 266, in execute

    send_request_to_wdb(query_lower, step, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 198, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 198, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 198, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 200, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2 + step % 2, step // 2 + off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 200, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2 + step % 2, step // 2 + off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 198, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 198, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 198, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2, off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 200, in send_request_to_wdb

    send_request_to_wdb(query_lower, step // 2 + step % 2, step // 2 + off, response)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 197, in send_request_to_wdb

    raise WazuhInternalError(2007)

wazuh.core.exception.WazuhInternalError: Error 2007 - Error retrieving data from Wazuh DB

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/cluster/dapi/dapi.py", line 234, in execute_local_request

    data = await asyncio.wait_for(task, timeout=timeout)

  File "/var/ossec/framework/python/lib/python3.8/asyncio/tasks.py", line 483, in wait_for

    return fut.result()

  File "/var/ossec/framework/python/lib/python3.8/concurrent/futures/thread.py", line 57, in run

    result = self.fn(*self.args, **self.kwargs)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/cluster/dapi/dapi.py", line 208, in run_local

    data = self.f(**self.f_kwargs)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/rbac/decorators.py", line 383, in wrapper

    allow = _match_permissions(req_permissions=req_permissions, rbac_mode=rbac.get()['rbac_mode'])

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/rbac/decorators.py", line 245, in _match_permissions

    _single_processor(req_resources, rbac.get().get(req_action, dict()), allow_match)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/rbac/decorators.py", line 200, in _single_processor

    expanded_resource = _expand_resource(user_resource)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/rbac/decorators.py", line 36, in _expand_resource

    return expand_group(value)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/agent.py", line 1726, in expand_group

    data = WazuhDBQueryAgents(select=['group'], limit=None).run()['items']

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/utils.py", line 1297, in run

    return self.general_run() if len(str(rbac_ids)) < common.MAX_QUERY_FILTERS_RESERVED_SIZE else \

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/utils.py", line 1236, in general_run

    self._execute_data_query()

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/utils.py", line 1200, in _execute_data_query

    self._data = self.backend.execute(query_with_select_fields, self.request)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/utils.py", line 929, in execute

    return self.conn.execute(query=self._render_query(query), count=count)

  File "/var/ossec/framework/python/lib/python3.8/site-packages/wazuh-4.0.1-py3.8.egg/wazuh/core/wdb.py", line 272, in execute

    raise WazuhInternalError(2007, str(e))

wazuh.core.exception.WazuhInternalError: Error 2007 - Error retrieving data from Wazuh DB: Error 2007 - Error retrieving data from Wazuh DB

2021/08/03 17:46:15 INFO: wazuh x.x.x.x. "GET /agents" done in 118.99999994784594ms: 500

2021/08/03 17:49:12 INFO: wazuh x.x.x.x. "GET /security/user/authenticate" done in 202.00000004842877ms: 200

2021/08/03 17:49:12 INFO: wazuh x.x.x.x. "GET /" done in 43.99999976158142ms: 200

2021/08/03 17:49:55 INFO: wazuh x.x.x.x. "GET /security/user/authenticate" done in 660.9999998472631ms: 200

2021/08/03 17:49:55 INFO: wazuh x.x.x.x. "GET /security/user/authenticate" done in 553.0000003054738ms: 200

2021/08/03 17:49:55 INFO: wazuh x.x.x.x "GET /security/user/authenticate" done in 416.99999989941716ms: 200

2021/08/03 17:49:55 INFO: wazuh x.x.x.x. "GET /security/user/authenticate" done in 416.99999989941716ms: 200

2021/08/03 17:49:55 INFO: wazuh x.x.x.x "GET /security/user/authenticate" done in 419.9999999254942ms: 200

2021/08/03 17:49:56 INFO: wazuh x.x.x.x. "GET /" done in 64.99999994412065ms: 200

2021/08/03 17:49:56 INFO: wazuh x.x.x.x. "GET /" done in 61.00000021979213ms: 200

2021/08/03 17:49:56 INFO: wazuh x.x.x.113 "GET /" done in 52.99999983981252ms: 200

2021/08/03 17:49:56 INFO: wazuh x.x.x.x. "GET /" done in 83.00000010058284ms: 200

2021/08/03 17:49:56 INFO: wazuh x.x.x.x "GET /" done in 83.00000010058284ms: 200

2021/08/03 17:50:00 INFO: unknown_user x.x.x.x. "GET /nodes" done in 1.0000001639127731ms: 308

2021/08/03 17:50:00 INFO: unknown_user x.x.x.x. "GET /nodes" done in 1.0000001639127731ms: 308

2021/08/03 17:50:00 INFO: wazuh x.x.x.x. "GET /cluster/nodes" done in 111.00000003352761ms: 200

2021/08/03 17:50:00 INFO: wazuh x.x.x.x. "GET /cluster/nodes" done in 75.00000018626451ms: 200

2021/08/03 17:50:00 INFO: unknown_user x.x.x.x. "GET /node01/stats/analysisd" done in 1.0000001639127731ms: 308

2021/08/03 17:50:00 INFO: unknown_user x.x.x.x. "GET /node01/stats/remoted" done in 1.0000001639127731ms: 308

2021/08/03 17:50:00 INFO: wazuh x.x.x.x. "GET /cluster/node01/stats/analysisd" done in 102.999999653548ms: 200

2021/08/03 17:50:00 INFO: wazuh x.x.x.x. "GET /cluster/node01/stats/remoted" done in 55.99999986588955ms: 200

2021/08/03 17:50:00 INFO: unknown_user x.x.x.x. "GET /node02/stats/remoted" done in 1.999999862164259ms: 308

2021/08/03 17:50:01 INFO: unknown_user x.x.x.x. "GET /node02/stats/analysisd" done in 0.9999996982514858ms: 308

2021/08/03 17:50:01 INFO: wazuh x.x.x.x. "GET /cluster/node02/stats/remoted" done in 154.99999979510x.3ms: 200

2021/08/03 17:50:01 INFO: wazuh x.x.x.x. "GET /cluster/node02/stats/analysisd" done in 113.99999959394336ms: 200

Can you please help with the resolution of this issue. 

Thank you

- Aram 

[Maximiliano Ibarra]

Hi Aram.
Thanks for contact us. 

Can you tell me if you have access to Wazuh kibana app? Please, if the app is failing add a screenshot to see if you have an error message.
Also, we need to check if some services are running.
Please, executes these commands in your manager machine and share with me the outputs.

• ps aux | grep ossec
• ps aux | grep wazuh
• service wazuh-manager status

In your API Logs have an error that could be saying that your wazuh-db is failing. Please try these steps:

• Enable WazuhDB in debug mode, write wazuh_db.debug=2 in /var/ossec/etc/local_internal_options.conf
  
• Restart manager with service wazuh-manager restart
• Run grep -E "ERR|WARN" /var/ossec/logs/ossec.log

I looking forward to your reply.

Thank you.

[Aram]

  Hi Maximiliano,

Yes I do have access to the Wazuh Kibana app and have attached screenshot of the error message.  I have also attached screenshots of the commands you listed.  I enabled WazuhDB debug mode and am attaching the results in a .log file.  I have included a sample of the various errors in that file. Please note the IP's in the log file have been sanitized. 

Thank you for your assistance, 

- Aram 

[Maximiliano Ibarra]

Hi Aram,  Thank you for your reply and for sending me all the information that I asked for.
I see that the connection to the API is failing. In 4.0 later versions, the connection username and password have been changed.

Please, change your username and password to wazuh-wui in your wazuh.yml file. 

Check this official article: https://documentation.wazuh.com/current/user-manual/kibana-app/troubleshooting.html#could-not-connect-to-api-with-id-default-3003-missing-param-api-username.

Try this out and tell me how it was.

I looking forward to your reply.

Best Regards.

Maximiliano

[Aram]

Hi  Maximiliano, 

I updated the API password in the wazuh.yml like you suggested however it did not resolve the issue.  For reference the cluster was upgraded to 4.0 in November of 2020 and we have not had issues with the API until now. 

Thanks, 

- Aram 

[Aram]

Hi  Maximiliano,

I've narrowed the issue specifically to the  GET /agents API call. This is the call returns 500 error in the API logs. When running the call manually it returns the following response: 

 curl -k -X GET "https://wazuh-mgr1:55000/cluster/agents" -H "Authorization: Bearer $(curl -u wazuh:wazuh -k -X GET 'https://wazuh-mgr1:55000/security/user/authenticate?raw=true')"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100   265  100   265    0     0    567      0 --:--:-- --:--:-- --:--:--   567

{"title": "Not Found", "detail": "404: Not Found"}

As you can see in the API log the GET /agents is the only API call that is failing: 

..................

2021/08/13 13:15:01 INFO: wazuh 192.168.231.114 "GET /agents" done in 145.0000000000955ms: 500

2021/08/13 13:15:01 INFO: wazuh 192.168.231.114 "GET /cluster/status" done in 67.99999999998363ms: 200

2021/08/13 13:15:02 INFO: wazuh 192.168.231.114 "GET /cluster/local/info" done in 52.99999999999727ms: 200

2021/08/13 13:15:03 INFO: wazuh 192.168.231.114 "GET /cluster/node02/stats/remoted" done in 1076.0000000000218ms: 200

.......................

It seems that Wazuh App needs this call to be able to run successfully to load correctly. 

Hope this helps. 

Thanks, 

Aram 

[Aram]

Hi  Maximiliano and Wazuh community,

Any ideas on what may be going on the issue with the /cluster/agents API call seems to be an issue with wdb.py script or access to the ossec db. Any timely suggestions would be greatly appreciated. 

Thank you 

[Dario Menten]

Hello Aram,
It seems the Endpoint you are trying to use in the curl command does not exist:

curl -k -X GET "https://wazuh-mgr1:55000/cluster/agents"

The correct endpoint should be:

curl -k -X GET "https://wazuh-mgr1:55000/agents"

You can check this in the API Reference for more information about it.

One thing I would like you to check is that the WazuhDB has the proper permission and ownership:

root@theshire:~# ls -l /var/ossec/queue/db/
-rw-r----- 1 ossec ossec    57344 Aug 16 19:55 global.db
-rw-r----- 1 ossec ossec    16928 Aug 16 19:55 global.db-journal
srw-rw---- 1 ossec ossec        0 Aug 16 17:54 wdb
root@theshire:~# ls -l /var/ossec/queue/db/wdb 
srw-rw---- 1 ossec ossec 0 Aug 16 17:54 /var/ossec/queue/db/wdb

Another thing that you can check is if the database is accessible:

root@theshire:~# sqlite3 /var/ossec/queue/db/global.db 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .tables
agent     belongs   group     info      labels    metadata
sqlite> select * from agent;
0|theshire|127.0.0.1|127.0.0.1||Debian GNU/Linux|10|10||buster||debian|Linux |theshire |4.19.0-16-amd64 |#1 SMP Debian 4.19.181-1 (2021-03-19) |x86_64|x86_64|Wazuh v4.1.5|||theshire||1619802068|253402300799||synced|active
2|oel83|10.10.10.152|any|183a13aa7944792a2babd9c0a99f8030db825bb51e98c7ea1dc5f778b9b63238|Oracle Linux Server|8.3|8|3|||ol|Linux |oel83 |5.4.17-2011.7.4.el8uek.x86_64 |#2 SMP Fri Oct 2 14:39:04 PDT 2020 |x86_64|x86_
....
....

If it is all ok, we would need you to put the logs in debug and send us the logs when you replicate the issue:
In the Wazuh Manager go to /var/ossec/etc/local_internal_options.conf and add this configuration:

wazuh_db.debug=2

Reference
Also the API, by going to and adding the lines:

 logs:
  level: "debug"
  path: "logs/api.log"

Please send us the result of the api.log and the ossec.log present in the /var/ossec/etc folder after replicating the issue.

​

[Dario Menten]

Errata:
The logs are in the /var/ossec/logs folder.

​

[Aram]

Hi Dario, 

Thanks for your assistance with looking into this. When running the curl command as you mentioned it produces the error "Error retrieving data from Wazuh DB: Error 2007" similar to what we've seen in the logs: 

curl -k -X GET "https://wazuh-mgr1:55000/agents?pretty" -H "Authorization: Bearer $(curl -u wazuh:wazuh -k -X GET 'wazuh-mgr1:55000/security/user/authenticate?raw=true')"

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100   265  100   265    0     0    259      0  0:00:01  0:00:01 --:--:--   259

{"title": "Wazuh Internal Error", "detail": "Error retrieving data from Wazuh DB: Error 2007 - Error retrieving data from Wazuh DB", "dapi_errors": {"node01": {"error": "Error retrieving data from Wazuh DB: Error 2007 - Error retrieving data from Wazuh DB", "logfile": "WAZUH_HOME/logs/api.log"}}, "error": 2007}

Here is output for commands you listed: 

ls -l /var/ossec/queue/db/

-rw-rw----. 1 ossec ossec 3727360 Aug 16 16:32 global.db

-rw-rw----. 1 ossec ossec 1445120 Aug 16 16:33 global.db-journal

-rw-r-----. 1 ossec ossec 1589248 Nov 23  2020 global.db-oldv0-1606165534

srw-rw----. 1 ossec ossec       0 Aug 16 16:30 wdb

ls -l /var/ossec/queue/db/wdb 

srw-rw----. 1 ossec ossec 0 Aug 16 16:33 /var/ossec/queue/db/wdb

Database

The sqlite3 data is accessible and data is returned when running select * from agent;

Debug

/var/ossec/etc/local_internal_options.conf had already been set to wazuh_db.debug=2 from Maximiliano's recommendation.

API debug level has also been set in /var/ossec/api/configuration/api.yml

 api.log is attached. Ossec is to large so including Drive link (note: pii has been removed from both files):

https://drive.google.com/file/d/13YBCWsEIeSy3FBke6I7GCclIFHLamC1y/view?usp=sharing

Thank you,

- Aram 

[Dario Menten]

Hello Aram,

I was talking about this with our Dev team, and they say it seems to be a race condition that affects the remoted module and it will be solved for Wazuh 4.2 that will be released the next week. This is the issue associated: https://github.com/wazuh/wazuh/pull/8224

I hope this information is helpful.

I look forward to your feedback.

[Aram]

Thanks for the information Dario, are there any other workarounds available that would not require an update to 4.2?

- Aram  

[Dario Menten]

Hello Aram,
If you don’t have too many groups created, you can delete de global.db and restart the Wazuh Manager to re-create the database.
Otherwise, it would be finding the groups in the table that are wrong and erase them.

​

[Aram]

Hi Dario, 

Thanks for this suggestion. We went ahead and tried it but unfortunately the issue persists and we are seeing the same messages in the logs as before. We initially tried deleting global.db with no luck, we then tried removing all files in /var/ossec/queue/db/ with no luck. The process we followed was: 

stop wazuh service

delete

start wazuh service

Is there anything else we can try?

Thank you,

- Aram 

[Dario Menten]

Hello Aram,
Another workaround could be finding the agent-groups files that could be with some error. For that, you can do a grep in the agent-groups folder and erasing the files listed:

root@theshire:~# grep ERROR /var/ossec/queue/agent-groups/*
/var/ossec/queue/agent-groups/019:ERROR

Remove those files and restart the Wazuh Manager.
Please let me know if that helps you, if not I will need you to share your log files to analyze them.

I look forward to your feedback.

Kind Regards

​

[Aram]

Hi Dario, 

Thanks for this suggestion, there were 11 agent group files with error messages.  Unfortunately it did not resolve the issue and we are seeing the same messages in the api and ossec logs. 

The steps I performed were: 

grep ERROR /var/ossec/queue/agent-groups/*

Then delete those files and restart the manager. I did this on both of the Wazuh managers. 

I wanted to make sure I did not miss a step as I was not sure what you meant by: 

/var/ossec/queue/agent-groups/019:ERROR

We are planning on upgrading Wazuh cluster to 4.2 and ELK stack to 7.11.2 tomorrow. 

Thank you, 

- Aram 

[Dario Menten]

Hello Aram,

Regarding your question about this sentence: /var/ossec/queue/agent-groups/019:ERROR it is an example of a result from the grep command in the previous line.

Regarding the persistence of the issue, I would like to have new logs in debug mode in order to help you solve it. But if you are planning the upgrade in next days, I would recommend to move forward on that, and it will solve the issue.

If you have any doubts or concerns about the Upgrade you can ask here or in the Slack channel.

[Aram]

Hi Dario, 

Thanks for the clarification, in that case I did not miss a step however our error messages were a little different. 

As far as the upgrade I did try it out in a lab first.  My only question is about step 11 to "Link Kibana’s socket to privileged port 443".  I noticed that Kibana will not load without executing this command.  In our production environment however 

the Firewall and the client is coming in on 443 and the Firewall is redirecting to Kibana on 5601.  Will this setup work? Do we still need to issue the command to "Link Kibana’s socket to privileged port 443" even though it is configured for 5601?

Thanks, 

- Aram 

[Dario Menten]

Hello Aram,
It is not mandatory to use port 443, you can use the port you want/need, you just need to specify it in the kibana.yml file and restart the Kibana Service.
I hope you can solve your issue.

​

[Aram]

Hi Dario, 

The upgrade was successful and did resolve the issue with API. 

Thanks for all your help! 

Cheers, 

- Aram 

[Aram]

Hi Dario, 

Unfortunately the same or similar problem has returned. The error started occurring after a restart of Wazuh service.  Viewing the logs show the same error as before: 

2021/09/15 05:00:36 ERROR: Error retrieving data from Wazuh DB: Error 2007 - Error retrieving data from Wazuh DB

Traceback (most recent call last):

  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.2.1-py3.9.egg/wazuh/core/wdb.py", line 224, in send_request_to_wdb

    response.extend(self._send(request))

  File "/var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.2.1-py3.9.egg/wazuh/core/wdb.py", line 100, in _send

    raise ValueError

ValueError

Links for OSSEC and API logs are below. Please let us know if you would be able to help with continuation of troubleshooting this issue. 

OSSEC Log

https://drive.google.com/file/d/1VdGBY_KhYhFL8NLgc322BxyNf2fKVe1B/view?usp=sharing

API Log

https://drive.google.com/file/d/19ODYXsMIgrn_I2wRGp3WpC_kfyMSqF-Q/view?usp=sharing

[Dario Menten]

Hello Aram,
My apologies for the delay.
I was checking the logs you sent, and in the API logs it is reflected the error log:

2021/09/17 12:27:07 INFO: wazuh-wui x.x.x.x "GET /agents" with parameters {"offset": "500", "limit": "500", "q": "id!=000"} and body {} done in 16.962s: 500
2021/09/17 12:27:07 ERROR: Error retrieving data from Wazuh DB: Error 2007 - Error retrieving data from Wazuh DB

But no errors like you show from September 15 are in the ossec.log you sent.

Could you please put the API in debug and then execute the following request in Wazuh > _Tools > API Console?

GET /agents
{
  "offset": "500",
  "limit": "500",
  "q": "id!=000"
}

Please send the API logs from the moment you execute this, and also the ossec.log lines.
Thank you in advance!

​

[Aram]

Hi Dario, 

Thanks for getting back, the api logs are in debug as requested and both api and ossec log are attached.  I began to capture them slightly before I issued the command in API console.  There is also a screenshot attached of the response I received in the console. 

As a side note I did a network capture on the manager of the api port 5500. I noticed a significant amount of inbound traffic from the Kibana node, even when Kibana is not being used or page open.  This prompted me to restart Kibana service.  I found that this fixes the issues temporarily however after sever hours it becomes inaccessible again. I do not believe this was the case prior to the 4.2 upgrade. 

Thank you, 

Aram 

[Dario Menten]

Hello Aram,
I’ve been analyzing this, and it is pretty weird, because we don’t see any errors in the ossec.log, just a few warnings, but nothing to be worried.
I am thinking in a performance issue, something related to a lack of resources, so I would like to check the resources available. For that, I will need that you check some things for me:

• Memory usage:

  free -h
  

• CPU Usage

  top
  

• Check for dropped events in remoted state file

  cat /var/ossec/var/run/ossec-remoted.state

  I hope this information could be helpful in solving your issue.

​

[Aram]

Hi Dario, 

Agreed it is a strange issue but we do want to resolve it.  Today I tested and I after restarting Kibana I was able to use the Wazuh App for 3-4 hours then the same issue.  

Here is info on performance: 

free -h 

top

cat /var/ossec/var/run/ossec-remoted.state

While running top I did notice that python3 does fluctuate quite a bit sometimes reaching 50-75% range. Another thing I noticed is that there is an excessive amount of communication between Wazuh manager and Kibana. I attached a tcpdump I ran for a few seconds.  It is a lot of syn/ack traffic. This traffic happens even with the Kibana page closed. Is this normal?

Thanks again for you assistance. 

- Aram 

[Abraham Cruz]

Hello!

I had this same issue. You can find the Slack thread here. But basically in my case it was due to the fact that our agents are rotating a lot. And that was causing authd to spike if the keys are not purged. Here is the documentation.

Once I enabled that and restarted the manager twice, everything worked. So far we have not had any problem at all. 

The error did no show anything, it was just a trial and error and reading lots of documentation that made me think on that. Of course, I don't really know the guts of it to be able to pinpoint the exact reason, but empirically this worked for me. Hopefully it helps you.

[Dario Menten]

Hello Abraham,

I agree, it could be a reason, Aram, could you test that, while I analyze the data you sent me?

I hope this could solve the problem.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/3c0f6705-94d7-4686-843c-4d1fb1626e92n%40googlegroups.com.

[Aram]

Hi Abraham and Dario, 

Abraham thank you for insight into this issue! I see that the "purge" parameter is set to "true" by default in the ossec.conf file.  I verified that this is the case in our deployment.  When you say you enabled it had yours previously been set to false?

Apologies if I am misunderstanding here.

Thank you, 

- Aram 

[Abraham Cruz Sustaita]

Yes, it was false in my case. 

☁ Abraham Cruz Sustaita

On Oct 22, 2021, at 14:18, Aram <pro.ara...@gmail.com> wrote:

Hi Abraham and Dario, 

You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/KRQgvbX64uE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/6e7d05ee-a724-4592-a8fe-4d6a65cd20f9n%40googlegroups.com.

[Aram]

Thanks Abraham, in our case it was set to the true. I went ahead and toggled it (set to false, restart then set to true) however the issue still persists. 

Any other recommendations would be appreciated! 

Thank you Abraham and Dario 

- Aram