kibana server error

[mariano hinjos]

Hi

I have a problem, in kibana log:

{"type":"log","@timestamp":"2023-03-02T19:24:59+01:00","tags":["warning","savedobjects-service"],"pid":11139,"message":"Unable to connect to Elasticsearch. Error: search_phase_execution_exception"}

Any idea?

Thanks in advance

[Juan Nicolás Asselle (Nico Asselle)]

Hello Mariano, can you tell us how your installation is composed?

• Do you have Kibana and Elasticsearch on the same server?
• Can you share with us which versions of Wazuh, Elasticsearch, and Kibana you have installed?
• In case you have Kibana and Elasticsearch on different servers, do you have connectivity from the Kibana server to the Elasticsearch server?
• Can you share with us the configuration of your Kibana and Elasticsearch? (you can hide the information you consider sensitive), but I need to confirm which ports you have configured and if you have connectivity to them.
• is it a persistent error or does it appear when you want to make a particular query?

I look forward to your comments

[mariano hinjos]

Hello

context about your mail

• Do you have Kibana and Elasticsearch on the same server? YES

• Can you share with us which versions of Wazuh, Elasticsearch, and Kibana you have installed?

elasticsearch-7.16.1-1.x86_64

wazuh-manager-4.2.4-1.x86_64

filebeat-7.11.2-1.x86_64

kibana-7.11.2-1.x86_64

• In case you have Kibana and Elasticsearch on different servers, do you have connectivity from the Kibana server to the Elasticsearch server?
• Can you share with us the configuration of your Kibana and Elasticsearch? (you can hide the information you consider sensitive), but I need to confirm which ports you have configured and if you have connectivity to them.

Kibana config

server.host: 0.0.0.0
server.port: 443
elasticsearch.hosts: https://localhost:9200
elasticsearch.password: *****************************

# Elasticsearch from/to Kibana

elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca/ca.crt
elasticsearch.ssl.certificate: /etc/kibana/certs/kibana.crt
elasticsearch.ssl.key: /etc/kibana/certs/kibana.key

# Browser from/to Kibana
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana.crt
server.ssl.key: /etc/kibana/certs/kibana.key

# Elasticsearch authentication
xpack.security.enabled: true
elasticsearch.username: elastic
uiSettings.overrides.defaultRoute: "/app/wazuh"
elasticsearch.ssl.verificationMode: certificate

Elasticsearch

network.host: 127.0.0.1
node.name: elasticsearch
cluster.initial_master_nodes: elasticsearch

# Transport layer
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# HTTP layer
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# Elasticsearch authentication
xpack.security.enabled: true

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

• is it a persistent error or does it appear when you want to make a particular query?

It a persistent error but if I make a manual query I get a response, for example

when i load the page
Kibana server is not ready yet

I don't see the error

[Carlos Ezequiel Bordon]

If you can check the status of the Elasticsearch cluster with this command:

curl -X GET "localhost:9200/_cluster/health?pretty&pretty"

Also if can let us know if you find any other errors in the Elasticsearch logs:

cat /var/log/elasticsearch/elasticsearch.log

On the other hand, is this a new installation of Wazuh? if the answer is no, were there any recent changes that caused the services to stop communicating?

Can you validate if the certificates used by Elasticsearch and Kibana are still valid?

openssl x509 -enddate -noout -in server.crt

[mariano hinjos]

it doesn't work, if i execute

 filebeat test output

elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

and in elastic log

[2023-03-06T08:18:31,153][INFO ][o.e.x.s.a.RealmsAuthenticator] [elasticsearch] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-03-06T08:18:33,652][ERROR][o.e.x.s.a.e.ReservedRealm] [elasticsearch] failed to retrieve password hash for reserved user [elastic]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable

curl -XGET https://localhost:9200 -u elastic:************* -k
{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [elastic] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}[root@

but i don´t find the error config,