Kibana error

497 views
Skip to first unread message

Cristian Radu

unread,
Sep 19, 2022, 3:29:27 PM9/19/22
to Wazuh mailing list
Hello,

I am getting this error on kibana. Any ideas?


Screenshot 2022-09-19 222819.png

Also in Kibana logs I am seeing this:

Sep 19 22:27:41 wazuh-manager kibana[189775]: {"type":"error","@timestamp":"2022-09-19T19:27:41Z","tags":[],"pid":189775,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Er
ror: Internal Server Error\n    at HapiResponseAdapter.toError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:132:19)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/co
re/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:81:17)\n    at Router.handle (/usr/share/kibana/src/cor
e/server/http/router/router.js:164:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":n
ull,"search":null,"query":{},"pathname":"/api/check-stored-api","path":"/api/check-stored-api","href":"/api/check-stored-api"},"message":"Internal Server Error"}
Sep 19 22:27:41 wazuh-manager kibana[189775]: {"type":"response","@timestamp":"2022-09-19T19:27:41Z","tags":[],"pid":189775,"method":"post","statusCode":500,"req":{"url":"/api/check-stored-api","method":"po
st","headers":{"host":"10.1.220.178","connection":"keep-alive","content-length":"16","sec-ch-ua":"\"Google Chrome\";v=\"105\", \"Not)A;Brand\";v=\"8\", \"Chromium\";v=\"105\"","dnt":"1","kbn-xsrf":"kibana",
"sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","content-type":"application/json","accept":"application
/json, text/plain, */*","sec-ch-ua-platform":"\"Windows\"","origin":"https://10.1.220.178","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://10.1.220.178/app
/wazuh","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,ro-RO;q=0.8,ro;q=0.7","securitytenant":""},"remoteAddress":"10.1.140.237","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x6
4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36","referer":"https://10.1.220.178/app/wazuh"},"res":{"statusCode":500,"responseTime":30,"contentLength":9},"message":"POST /api/check-
stored-api 500 30ms - 9.0B"}

BR,
Cristian

Raul Del Pozo Moreno

unread,
Sep 19, 2022, 4:26:20 PM9/19/22
to Cristian Radu, Wazuh mailing list
Hello Cristian,

Have you modified the credentials or the URL located in the wazuh.yml file?

What version of Wazuh are you using? Do you use Wazuh dashboard or Kibana? Depending on what you are using, this file will be located in one directory or another:

  • /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml
  • /usr/share/kibana/data/wazuh/config/wazuh.yml

Is this an All In One or is it a distributed environment?

Please try to avoid posting sensitive data like public IPs or credentials for security reasons.

Regards, Raúl.

WazuhRaúl Del Pozo Moreno
IT Security Engineer - CICD


--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c72d5fc4-2e16-4e03-9549-6310d2275920n%40googlegroups.com.

Cristian Radu

unread,
Oct 3, 2022, 3:03:01 PM10/3/22
to Wazuh mailing list
Hello,  Raúl,

Yes, I have modified the credentials located in the wazuh.yml file and the URL is set to the IP of the VM itself (not localhost).

I'm using a Kibana dashboard. 

Wazuh version is 4.3.7, but I remember installing a All in one deployment and with a lower version, maybe 4.2 if I recall well. I did not perform any update, unless with my troubleshooting if I have upgraded it somehow.

root@wazuh-manager:~# /var/ossec/bin/wazuh-control info | grep WAZUH_VERSION

WAZUH_VERSION="v4.3.7"

What do you suggest to do further?

BR,
Cristian

Cristian Radu

unread,
Oct 3, 2022, 3:24:48 PM10/3/22
to Wazuh mailing list
Hello,

I also delete all the indices with yellow state, even the ones from today. Now I see error under kibana status:

kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-10-03 22:10:47 EEST; 7min ago
   Main PID: 442332 (node)
      Tasks: 11 (limit: 19104)
     Memory: 147.4M
     CGroup: /system.slice/kibana.service
             └─442332 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist -c /etc/kibana/kibana.yml

Oct 03 22:11:19 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:11:19Z","tags":["error","plugins","wazuh","monitoring"],"pid":442332,"message":"{ error: 'no credentials', error_code>
Oct 03 22:13:11 wazuh-manager kibana[442332]: {"type":"error","@timestamp":"2022-10-03T19:13:11Z","tags":["connection","client","error"],"pid":442332,"level":"error","error":{"message":"140108483483456:err>
Oct 03 22:13:12 wazuh-manager kibana[442332]: {"type":"response","@timestamp":"2022-10-03T19:13:11Z","tags":[],"pid":442332,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"p>
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No >
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No >
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No >
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No >
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["error","plugins","wazuh","monitoring"],"pid":442332,"message":"{ error: 'no credentials', error_code>
Oct 03 22:16:19 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:16:19Z","tags":["error","elasticsearch","data"],"pid":442332,"message":"[security_exception]: no permissions for [ind>
Oct 03 22:16:19 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:16:19Z","tags":["error","elasticsearch","data"],"pid":442332,"message":"[illegal_argument_exception]: request [/_lice>

kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-10-03 22:10:47 EEST; 11min ago
   Main PID: 442332 (node)
      Tasks: 11 (limit: 19104)
     Memory: 148.6M
     CGroup: /system.slice/kibana.service
             └─442332 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist -c /etc/kibana/kibana.yml

Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}
Oct 03 22:15:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:15:00Z","tags":["error","plugins","wazuh","monitoring"],"pid":442332,"message":"{ error: 'no credentials', error_code: 1, toString: [Function] }"}
Oct 03 22:16:19 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:16:19Z","tags":["error","elasticsearch","data"],"pid":442332,"message":"[security_exception]: no permissions for [indices:admin/mappings/get] and User [name=kibanaserver, backend_roles=[], requestedTenant=null]"}
Oct 03 22:16:19 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:16:19Z","tags":["error","elasticsearch","data"],"pid":442332,"message":"[illegal_argument_exception]: request [/_license] contains unrecognized parameter: [accept_enterprise]"}
Oct 03 22:20:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:20:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}
Oct 03 22:20:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:20:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}
Oct 03 22:20:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:20:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}
Oct 03 22:20:00 wazuh-manager kibana[442332]: {"type":"log","@timestamp":"2022-10-03T19:20:00Z","tags":["info","plugins","wazuh","cron-scheduler"],"pid":442332,"message":"{\"error\":10001,\"message\":\"No Wazuh host configured in wazuh.yml\"}"}

This is my wazuh.yml

root@wazuh-manager:~# cat /usr/share/kibana/data/wazuh/config/wazuh.yml | grep -v "#"
---




hosts:
  - default:
     url: https://10.1.220.178
     port: 55000
     username: wazuh-wui
     password: NEW_PASS_HERE
     run_as: false

BR,
Cristian

Raul Del Pozo Moreno

unread,
Oct 4, 2022, 1:05:38 PM10/4/22
to Cristian Radu, Wazuh mailing list
Hello Cristian

From what I see, the configuration of wazuh.yml that you have shared is correct.

Could you run the following commands and show me the output of the second command?


With this, we can validate that the password set for the wazuh-wui user is correct.

Regarding the Kibana service messages, I am investigating the cause.

I need clarification on this:


Wazuh version is 4.3.7, but I remember installing a All in one deployment and with a lower version, maybe 4.2 if I recall well. I did not perform any update, unless with my troubleshooting if I have upgraded it somehow.

Do you have an All-in-One environment, which has been upgraded from 4.2 to 4.3.7? If so, did you follow any process?

Regards, Raúl.

WazuhRaúl Del Pozo Moreno
IT Security Engineer - CICD

Cristian Radu

unread,
Oct 5, 2022, 5:50:24 AM10/5/22
to Wazuh mailing list
Hello Raul,

root@wazuh-manager:~# TOKEN=$(curl -u wazuh-wui:<pass> -k -X GET "https://10.1.220.178:55000/security/user/authenticate?raw=true")
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   404  100   404    0     0    252      0  0:00:01  0:00:01 --:--:--   251
root@wazuh-manager:~# curl -k -X GET "https://10.1.220.178:55000/" -H "Authorization: Bearer $TOKEN"
{"data": {"title": "Wazuh API REST", "api_version": "4.3.7", "revision": 40320, "license_name": "GPL 2.0", "license_url": "https://github.com/wazuh/wazuh/blob/4.3/LICENSE", "hostname": "wazuh-manager", "timestamp": "2022-10-05T09:44:26Z"}, "error": 0}root@wazuh-manager:~#

It seems that I get an error. 

Version 4.3.7 is the latest version? So I installed wazuh all-in-one about one year ago. After that I did not follow any upgrade process. So that is why I am saying that it is strange I have version 4.3.7. But would it be a good idea to upgrade everything like on the website? Maybe I'll be solving the issues I have. Following these steps here https://documentation.wazuh.com/4.2/upgrade-guide/index.html

Br,
Cristian

Raul Del Pozo Moreno

unread,
Oct 5, 2022, 9:27:05 AM10/5/22
to Cristian Radu, Wazuh mailing list
Hello Cristian

In case any component has been upgraded, it would be good for you to check the versions of the installed components since generally, it is not enough just to upgrade the package (there are files that need to be upgraded too).

Regarding the command output, it is correct, it gives the Wazuh version and revision so the wazuh-wui password is correct, the indicates that there is no error.

If you installed an All In One environment a year ago, you should have a version of at least 4.2 of Wazuh, since version 4.3.0 was released on May 5, 2022, we are currently in version 4.3.8 version, here you can see the releases notes: https://documentation.wazuh.com/current/release-notes/index.html

Please check that your wazuh.yml file is well indexed since the errors observed also fit with the errors shown in the first post, normally, the entire file should be commented on except for the last part where it should have this format

hosts:
**- default:
******url: https://localhost
******port: 55000
******username: wazuh-wui
******password: wazuh-wui
******run_as: false

Where each * corresponds to a whitespace character

There is a point that seems important, you have the 4.3.7 version of Wazuh server but the version of the Wazuh app has also had to be upgraded since otherwise it would give a version incompatibility error and without knowing exactly what steps you have modified or what has been upgraded It can be a bit difficult to locate the specific error since it can be due to partial upgrades that there are several places where the problem is.

Regarding performing an upgrade, it depends on your needs, but it would be convenient to upgrade to have the latest fixes and added functionalities. Here you will have three options depending on your current deployment:

1. Upgrade to Wazuh central components (Wazuh server, Wazuh dashboard, Wazuh indexer, Filebeat)
     - https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html
     - This option changes Open Distro for Elasticsearch to our packages based on OpenSearch (1.2) (remember that OpenSearch is a fork of Elasticsearch and Kibana)
     - Wazuh dashboard ~ Kibana
     - Wazuh indexer ~ Elasticsearch
     - Starting with version 4.4 of Wazuh, we will be based on OpenSearch 2.3

Regards, Raúl.

WazuhRaúl Del Pozo Moreno
IT Security Engineer - CICD

Cristian Radu

unread,
Oct 11, 2022, 9:21:02 AM10/11/22
to Wazuh mailing list
Hello Raul,

I managed to solve my issue by doing in upgrade to the latest version 4.3.8. I followed this migration guide from here https://documentation.wazuh.com/current/migration-guide/index.html

My question is now how I can improve, optimize my setup so that I do not encounter too many issue in the future. 

For instance my status is now "yellow"

curl -X GET "https://localhost:9200/_cluster/health?pretty" -u admin:pass -k
{
  "cluster_name" : "wazuh-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "active_primary_shards" : 342,
  "active_shards" : 342,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 98.27586206896551
}

Is there any guide on this matter? I know that I had issues in the past with the shards being set to 1000. I know that I have to set cold/hot policies. But is there something else?

Thanks for your help and assisstance!

BR,
Cristian

Raul Del Pozo Moreno

unread,
Oct 13, 2022, 10:26:22 AM10/13/22
to Cristian Radu, Wazuh mailing list
Hello Cristian

I'm glad you were able to fix the problem, we've just released v4.3.9 version, so you might want to take a look at the changes: https://documentation.wazuh.com/current/release-notes/release-4-3-9.html

From what I see, the cluster is in a Yellow state, possibly because you have unassigned shards "unassigned_shards" : 6, this can be due to disk space or multiple reasons, you disabled shard allocation when upgrading to Wazuh indexer? https://documentation.wazuh.com/current/migration-guide/wazuh-indexer.html

Please run the following command to display information about the unassigned shards:

- curl -X GET "https://localhost:9200/_cluster/allocation/explain" -u admin:password -k

Regards, Raúl.
WazuhRaúl Del Pozo Moreno
IT Security Engineer - CICD

Cristian Radu

unread,
Oct 14, 2022, 9:45:23 AM10/14/22
to Wazuh mailing list
Hello Raul,

I'll check the new released version 4.3.9 and maybe do an upgrade.

I disabled the shard allocation as described in the process. I followed everything step by step. 

The output of the command it is as follows:

root@wazuh-manager:~# curl -X GET "https://localhost:9200/_cluster/allocation/explain" -u admin:pass -k
{"index":"security-auditlog-2022.10.06","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2022-10-10T19:01:15.455Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"Wgcxeaj4RM6UkojA65tufg","node_name":"node-1","transport_address":"10.1.220.178:9300","node_attributes":{"shard_indexing_pressure_enabled":"true"},"node_decision":"no","deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[security-auditlog-2022.10.06][0], node[Wgcxeaj4RM6UkojA65tufg], [P], s[STARTED], a[id=MSUJqbmhTOCbRUm9XBF-4w]]"}]}]}root@wazuh-manager:~#

What does this mean? What can I do to solve it?

Again, thank you very much for your assistance and patience. Very much appreciated.

Br,
Cristian

Raul Del Pozo Moreno

unread,
Oct 14, 2022, 2:23:04 PM10/14/22
to Cristian Radu, Wazuh mailing list
Hello Cristian, the problem is that it is a replica, but you only have one Wazuh indexer node, is this correct?

This may be because the security-auditlog-2022.10.06 index has a number_of_replicas value greater than 0.

A quick way to check this is by running the following command:

curl -X GET "https://localhost:9200/security-auditlog-2022.10.06/_settings?pretty" -u admin:pass -k | grep "number_of_replicas"

To specify that the number of replicas is 0 in a specific index, you can use the following command:

curl -k -u admin:pass -XPUT -H 'Content-Type: application/json' 'https://localhost:9200/security-auditlog-2022.10.06/_settings' -d '{ "number_of_replicas": "0 " }'

Now, since said shard is already indexed, it is possible to delete it directly, you can do this by following these steps:
  • Locate unassigned shards (UNASSIGNED):
curl -X GET "https://localhost:9200/_cat/shards" -u admin:pass -k
  • Remove the shard
curl -X DELETE "https://localhost:9200/security-auditlog-2022.10.06" -u admin:pass -k

Regards, Raúl.

WazuhRaúl Del Pozo Moreno
IT Security Engineer - CICD


Cristian Radu

unread,
Oct 15, 2022, 8:29:06 AM10/15/22
to Wazuh mailing list
Hello Raul,

I deleted those indices/shards that you mentioned and now the status is green. :)

{
  "cluster_name" : "wazuh-cluster",
  "status" : "green",

  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "active_primary_shards" : 362,
  "active_shards" : 362,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,

  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

What can I further do now to improve my setup health and make sure I do not have any issues in the future? Is the active_shards limited to 1000? How can I expand that? Or do I need to delete them from time to time?
Or issues I should take into consideration?

BR,
Cristian

Cristian Radu

unread,
Oct 15, 2022, 8:49:55 AM10/15/22
to Wazuh mailing list
Hi Raul,

Also,  under the status some services are with red. How can I solve these?
status.png
BR,
Cristian

Raul Del Pozo Moreno

unread,
Oct 17, 2022, 12:34:38 PM10/17/22
to Cristian Radu, Wazuh mailing list
Hello Cristian

Regarding the Status window:

The daemons in green color are those that are currently activated, those that are in red indicate that they are deactivated or have a problem, but being in red does not mean that there is a problem, to determine if there is a problem, you should consult the Wazuh server or Wazuh agent log file, this log can be found in the path: /var/ossec/logs/ossec.log, for example, I have caused an error in wazuh-maild to display a CRITICAL error with the following command:

# grep -i -E "error|critical|fatal|warning" /var/ossec/logs/ossec.log
2022/10/17 12:56:47 wazuh-maild: CRITICAL: (1501): Invalid SMTP Server: smtp.example.wazuh.com

In this link, you can find information about each daemon of the Wazuh server and Wazuh agent: https://documentation.wazuh.com/current/user-manual/reference/daemons/

Regarding how to improve or maintain your setup in good condition, I share some links to our documentation:

elasticsearch -> wazuh-indexer

I also share a link to Elasticsearch documentation where you will find information about the health of the cluster, keep in mind that Wazuh indexer is based on OpenSearch which in turn is based on Elasticsearch:

In this other link, it is detailed in-depth about the term shard and I hope it solves your doubt:

Regards, Raúl. 

WazuhRaúl Del Pozo Moreno
IT Security Engineer - CICD

Reply all
Reply to author
Forward
0 new messages