filebeat test output - getting certificate error issue

2,609 views
Skip to first unread message

Ramakrushna Panda

unread,
Jun 28, 2022, 9:00:38 AM6/28/22
to Wazuh mailing list
Hello Team,

Getting certificate error issue thrpwoing below error message, can someone help on this issue.

[root@wazuh-indexer kibana]# filebeat test output
elasticsearch: https://10.3.0.238:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.3.0.238
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... ERROR x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Wazuh")

Ramakrushna Panda

unread,
Jun 28, 2022, 9:02:26 AM6/28/22
to Wazuh mailing list
[root@wazuh-indexer kibana]# less /var/log/elasticsearch/elasticsearch.log | grep -i 'WARN\|ERROR'
[root@wazuh-indexer kibana]# less /var/log/filebeat/filebeat.log | grep -i 'WARN\|ERROR'
/var/log/filebeat/filebeat.log: No such file or directory
[root@wazuh-indexer kibana]# less /var/log/filebeat/filebeat | grep -i 'WARN\|ERROR'
[root@wazuh-indexer kibana]#  systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2022-06-28 12:53:45 UTC; 7min ago
     Docs: https://www.elastic.co
  Process: 16446 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 16446 (code=exited, status=1/FAILURE)

Jun 28 12:53:45 wazuh-indexer.novalocal systemd-entrypoint[16446]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun 28 12:53:45 wazuh-indexer.novalocal systemd-entrypoint[16446]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Jun 28 12:53:45 wazuh-indexer.novalocal systemd-entrypoint[16446]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jun 28 12:53:45 wazuh-indexer.novalocal systemd-entrypoint[16446]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Jun 28 12:53:45 wazuh-indexer.novalocal systemd-entrypoint[16446]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jun 28 12:53:45 wazuh-indexer.novalocal systemd-entrypoint[16446]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
Jun 28 12:53:45 wazuh-indexer.novalocal systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 28 12:53:45 wazuh-indexer.novalocal systemd[1]: Failed to start Elasticsearch.
Jun 28 12:53:45 wazuh-indexer.novalocal systemd[1]: Unit elasticsearch.service entered failed state.
Jun 28 12:53:45 wazuh-indexer.novalocal systemd[1]: elasticsearch.service failed.
[root@wazuh-indexer kibana]# systemctl status filebeat.service
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-06-28 10:35:02 UTC; 2h 26min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 1161 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─1161 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/fi...

Jun 28 10:35:02 wazuh-indexer.novalocal systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
[root@wazuh-indexer kibana]#

Roman Luna

unread,
Jun 28, 2022, 10:33:27 AM6/28/22
to Wazuh mailing list
Hi Rama,

Let me explain the process and a few concepts then onto the recommended solution at the end. We will create new certificates for all of the services in order to avoid issues and possible errors, with this guide you should be able to have it running quickly. In this example I used an all-in-one, the process is the same for any distribution, the difference relies on the location of said services. In the installation guide you may see it too:

Here is the link for a single-node elasticsearch in 4.2: Single-Node 4.2

Concepts:

  • ca local certificate authority (CA). Which holds the CA certificate and the private key for the CA.

  • cert X.509 certificates and keys. All certificates that are generated by this command are signed by a CA.

  • systemctl status -l kibana the -l option gives you more information about the log. You can also use which is more flexible: journalctl, check this:

    • journalctl -u kibana --no-pager, it is like cat, it will show you all the log lines.

      journalctl -u kibana --no-pager -f, it is like tail -f, it will show you all the log lines and wait for new lines.

      journalctl -u kibana -f -n 100, it is like tail -f -n 100, it will show you the last 100 log lines and wait for changes.

Requirements:

We will follow the steps from the installation process but with new certificates:

  1. Stop the following services Elasticsearch, Kibana and Filebeat with:

    systemctl stop elasticsearch.service
    systemctl stop kibana.service
    systemctl stop filebeat.service

  2. Move all the current certs into another directory, the certs are located in the following paths:

    /etc/elasticsearch/certs/
    /etc/kibana/certs/
    /etc/filebeat/certs/

  3. Create new directories in each path mentioned before with: mkdir backup then move them with mv <files-name> backup

  4. Run the wazuh-cert-tool.sh it will create a new directory ~/certs/ which contains all the certs that we are going to use.

  5. Taking this guide into consideration All-In-One Certificates we will configure the certs:

Elasticsearch:

mv ~/certs/elasticsearch* /etc/elasticsearch/certs/ mv ~/certs/admin* /etc/elasticsearch/certs/
cp ~/certs/root-ca* /etc/elasticsearch/certs/

Once the certs were moved:

systemctl daemon-reload
systemctl start elasticsearch

Run the Elasticsearch securityadmin script to load the new certificates information and start the cluster:

export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem -icl

  • The -cd option specifies where the security plugin configuration files to upload to the cluster can be found.

  • The -icl (--ignore-clustername) option tells the security plugin to upload the configuration regardless of the cluster name. As an alternative, you can also specify the cluster name with the -cn (--clustername) option.

  • Because the demo certificates are self-signed, we also disable hostname verification with the -nhnv (--disable-host-name-verification) option.

  • The -cacert, -cert and -key options define the location of your root CA certificate, the admin certificate, and the private key for the admin certificate. If the private key has a password, specify it with the -keypass option.

With this command you should get the following output: curl -XGET https://localhost:9200 -u admin:admin -k the admin:admin is user:password from your environment

{ "name" : "node-1", "cluster_name" : "elasticsearch", "cluster_uuid" : "tWYgqpgdRz6fGN8gH11flw", "version" : { "number" : "7.10.2", "build_flavor" : "oss", "build_type" : "rpm", "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9", "build_date" : "2021-01-13T00:42:12.435326Z", "build_snapshot" : false, "lucene_version" : "8.7.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search"
}

 

Filebeat:

cp ~/certs/root-ca.pem /etc/filebeat/certs/
mv ~/certs/filebeat* /etc/filebeat/certs/

Start the service:

systemctl daemon-reload
systemctl start filebeat

To ensure that Filebeat is successfully installed, run the following command: filebeat test output

An example response should look as follows:

elasticsearch: https://127.0.0.1:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 127.0.0.1 dial up... OK TLS... security: server's certificate chain verification is enabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 7.10.2

 

Kibana:

cp ~/certs/root-ca.pem /etc/kibana/certs/ mv ~/certs/kibana* /etc/kibana/certs/
chown kibana:kibana /etc/kibana/certs/*

Start the Kibana service:

systemctl daemon-reload
systemctl start kibana

Upon the first access to Kibana, the browser shows a warning message stating that the certificate was not issued by a trusted authority. An exception can be added in the advanced options of the web browser or, for increased security, the root-ca.pem file previously generated can be imported to the certificate manager of the browser.

Importing a Certificate into Mozilla Firefox

  1. In Firefox, go to Tools > Options.

  2. Go to the Advanced tab and the Security sub-tab and click View Certificates.

  3. Go to the Authorities tab and click Import.

Importing a Certificate into Google Chrome:

  1. Go to Chrome Settings.

  2. Click on "advanced settings"

  3. Under HTTPS/SSL click to "Manage Certificates"

  4. Go to "Trusted Root Certificate Authorities"

  5. Click to "Import"

  6. There will be a pop up window that will ask you if you want to install this certificate. Click "yes".

Remember to delete the cache and cookies from the browser. This is because you now have new certificates.

Let me know if you find this helpful and if you have any problems, if so, we can schedule a call, let me know of your schedule and we will work something out,

Cheers,

Roman from Wazuh!.

Ramakrushna Panda

unread,
Jun 28, 2022, 11:40:54 AM6/28/22
to Wazuh mailing list
Hi Roman,

Thank you so much for your valuable response,  right now I am using a single node Wazuh manager version 4.3.

I think the procedure and reference link you provided is for Version 4.2, Can you confirm the procedure you have posted above that will work Wazuh 4.3 version?

Thank you 

Ramakrushna Panda

unread,
Jun 28, 2022, 1:43:03 PM6/28/22
to Wazuh mailing list
Hello Team/ Roman,

I followed the same steps all the services are up but still getting the same certificate error. Could you please suggest 

se Error"}
qJun 28 17:39:35 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:39:35Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[ResponseError]: Response Error"}
^C
[root@wazuh-indexer certs]# filebeat test output

elasticsearch: https://10.3.0.238:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.3.0.238
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... ERROR x509: certificate signed by unknown authority
[root@wazuh-indexer certs]#

++++++++++++++++++++++++++++
[root@wazuh-indexer certs]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-06-28 17:35:58 UTC; 5min ago
 Main PID: 1035 (node)
   CGroup: /system.slice/kibana.service
           └─1035 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist -c /etc/kibana/kibana.yml

Jun 28 17:41:32 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:32Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:35 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:35Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:37 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:37Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:40 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:40Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:42 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:42Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:45 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:45Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:47 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:47Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:50 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:50Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:52 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:52Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Jun 28 17:41:55 wazuh-indexer.novalocal kibana[1035]: {"type":"log","@timestamp":"2022-06-28T17:41:55Z","tags":["error","elasticsearch","data"],"pid":1035,"message":"[Response...nse Error"}
Hint: Some lines were ellipsized, use -l to show in full.
(reverse-i-search)`ela': cd ../elasticsearch/^Casticsearch.yml
[root@wazuh-indexer certs]# systemctl status filebeat

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-06-28 17:31:29 UTC; 10min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 749 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─749 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/fil...

Jun 28 17:31:29 wazuh-indexer.novalocal systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
[root@wazuh-indexer certs]#  systemctl status elasticsearch

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2022-06-28 17:21:07 UTC; 21min ago
     Docs: https://www.elastic.co
 Main PID: 32069 (java)
   CGroup: /system.slice/elasticsearch.service
           └─32069 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.hea...

Jun 28 17:20:46 wazuh-indexer.novalocal systemd[1]: Starting Elasticsearch...
Jun 28 17:21:07 wazuh-indexer.novalocal systemd[1]: Started Elasticsearch.
[root@wazuh-indexer certs]#

Roman Luna

unread,
Jul 1, 2022, 3:38:42 PM7/1/22
to Wazuh mailing list
Hi Team,

Sorry for the delay. Maybe the deployment of certificates was unsuccessful. Here is the guide we have for 4.3, in which you will find a new script we develop to create the certificates. Is this a new installation?

Remember that, to create the certificates, edit the config.yml file and replace the node names and IP values with the corresponding names and IP addresses. The <node-ip> can be either an IP address or a DNS name. The config.yml template can be found here: config.yml.


Then from the installation guide you will find where does each certificate goes:


Have in mind the permissions too and ownership of the certificates. It is also explained in the guide just in case.

Regards.
Reply all
Reply to author
Forward
0 new messages