Wazuh server

[John Anderson]

After following the procedures in the following url - ( https://documentation.wazuh.com/current/learning-wazuh/vuln-detection.html) to track down vulnerabilities. I now get an error on the elastic log in screen.

Check alerts index pattern

Check monitoring index pattern

Check statistics index pattern

Check timelion:max_buckets setting

Check metaFields setting

Check timepicker:timeDefaults setting

[Alerts index pattern] No template found for the selected index-pattern title [wazuh-alerts-*]

The only changes I made was to configure the Ossec.conf file. I asked a collegue to check the changes I made and we commented them out but still get the errors on the Elastic gui page.

any help would be gratefully received.

Thanks

John

[Antonio David Gutiérrez]

Hi, thank you for using Wazuh!

It seems that you could not have the template that defines settings for the indices related to Wazuh alerts or archives.

I want to know what is your environment:
0.1 Are you using Elastic Stack (Kibana, Elasticsearch, Filebeat) or Wazuh stack (Wazuh dashboard, Wazuh indexer, Filebeat)? What version of Wazuh plugin (if using Kibana) or Wazuh dashboard are you using?
0.2 Did the problem happen in a new environment or this was working before? If it is new, what guide did you use to deploy the environment?

Check the next things:
1. The template is stored in Elasticsearch/Wazuh indexer. Go to the Dev Tools plugin of Kibana/Wazuh dashboard and run the next request:
GET /_cat/templates
You should have a template called wazuh. Share the output.

If you don't have the template, then follow the next step:
2. Check the Filebeat-Elasticsearch/Wazuh indexer connection. In the Filebeat host, run:
filebeat test output

Share the output.

If the output shows all is ok, maybe the problem is due to a misconfiguration so we should review the Fielbeat configuration.
3. Review the Filebeat configuration. Share it and obfuscate the sensitive data.

Other things you should have in account:
- If you changed the user's password used by Filebeat, you should change it in the Filebeat configuration.
Filed

[Antonio David Gutiérrez]

Hi John,

By the output of GET /_cat/templates I see you don't have the wazuh template. This template is added by Filebeat if you configurated correctly.

1. Check the Filebeat-Elasticsearch connection. Run in the host of Wazuh manager (in each one if you have a cluster of Wazuh manager nodes linked to the same Elasticsearch).

filebeat test output

Share the output.

2. If there is no problem with the previous command, then maybe the problem is you didn't configure Filebeat as Wazuh expects to work. Share the Filebeat configuration file. Obfuscate the sensitive data. The file should be located in /etc/filebeat/filebeat.yml.

Note: use the Google group forum instead to reply to me directly through email, so your question/problem could be useful for other users. Thank you.

[John Anderson]

Hi Antonia,

The output of "filebeat test output" is shown below -

filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK

talk to server... ERROR Connection marked as failed because the onConnect callback failed: could not connect to a compatible version of Elasticsearch: 400 Bad Request: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_license], must not start with '_'.","index_uuid":"_na_","index":"_license"}],"type":"invalid_index_name_exception","reason":"Invalid index name [_license], must not start with '_'.","index_uuid":"_na_","index":"_license"},"status":400}

The output of the /etc/filebeat/filebeat.yml file -

 Wazuh - Filebeat configuration file
output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  protocol: https
  username: "admin"
  password: "admin"
  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
  ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.overwrite: true
setup.ilm.enabled: false

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

Thanks again

John

[Antonio David Gutiérrez]

Hi John,

Thank you for the information.

It seems you could have installed Elasticsearch and Filebeat with a different distribution (default and OSS) and they are not compatible. You should ensure you are using the same distribution for both products (Elasticsearch and Filebeat).

If you using the Elastic basic license, then you should have the products (Kibana/Elasticsearch/Filebeat) with the default distribution.

If you are using Open Distro for Elasticsearch, you should have products with the OSS distribution. For this case, Open Distro for Elasticsearch has packages that are wrappers for Kibana/Elasticsearch OSS adding its own plugins and requires Filebeat OSS.

For some reason, you could have mixed how to install Elasticsearch and Filebeat depending on your deployment.

You should review the guide you followed to deploy Wazuh and its components and ensure you did the mentioned steps. Put special attention to the steps related to the installation of Elasticsearch and Filebeat, the repository used and what packages were installed. In the Wazuh documentation, there are 2 types of deployment installing packages: Elastic default distribution or using Open Distro for Elasticsearch that uses package wrappers for Kibana/Elasticserach OSS and requires Filebeat OSS. If you require to change the Filebeat package, take in account you should do this in each Wazuh manager node that belongs to the same Wazuh manager cluster.

If you need some help to solve the problem, share the next information:
- Version of the Wazuh manager/s
- What is your deployment type: Elastic basic license (its Elasticsearch has default distribution) or Open Distro for Elasticsearch?
- What guide did you use to deploy Wazuh and its components? Share the link.

[John Anderson]

Hi Antonio,

The version of Wazuh Manager is {"WAZUH_VERSION":"v4.2.6"},{"WAZUH_REVISION":"40221" 

I used the following command to remove the Elasticsearch performance analyzer plugin      /usr/share/elasticsearch/bin/elasticsearch-plugin remove opendistro-performance-analyzer. The server is using Open Distro for Elasticsearch

I used the following link to install and set up Wazuh - https://documentation.wazuh.com/4.2/installation-guide/open-distro/all-in-one-deployment/all-in-one.html 

Thanks again

John

[Antonio David Gutiérrez]

Hi John,

Thank you for the information.

Verify the distribution of Elasticsearch. Run this request.
curl -XGET -k https://<ELASTICSEARCH_HOST_IP>:9200 -u <ELASTICSEARCH_USER>:<ELASTICSEARCH_USER_PASSWORD>

where:
- <ELASTICSEARCH_HOST_IP>: Elasticsearch host IP.
- <ELASTICSEARCH_USER>: username of a Elasticsearch user with privilegies to do the request
- <ELASTICSEARCH_USER_PASSWORD>: user's password

Share the output.

If the Elasticsearch is an OSS distribution, then the installed Filebeat distribution could be the default distribution and both are not compatibles.

If you followed the guide in a fresh machine, the Filebeat OSS should be installed correctly. In the guide you used to deploy Wazuh and its components, Filebeat is installed with a command depeding on the OS' host, which installs the last version of Filebeat seeing the repositories have the host. At the beginning of the guide, a repository is added that contains the Filebeat OSS package. Did you add or have enabled an Elasticsearch repository? If this case, maybe when you ran the command, installed the default distribution instead of the OSS distribution.

You could try to remove the current Filebeat and ensure you install the Filebeat with OSS distribution. Remember to do the next steps after installing Filebeat: https://documentation.wazuh.com/4.2/installation-guide/open-distro/all-in-one-deployment/all-in-one.html#installing-filebeat

[John Anderson]

Hi Antonio,

I ran the request  "curl -XGET -k https://<ELASTICSEARCH_HOST_IP>:9200 -u <ELASTICSEARCH_USER>:<ELASTICSEARCH_USER_PASSWORD"

It looks like there could be a syntax error as this was the result of running the above command - 

-bash: syntax error near unexpected token `newline'

Thanks 

John

[Antonio David Gutiérrez]

Hi John,

Did you replace the placeholders in the command?

Verify the distribution of Elasticsearch. Run this request.
curl -XGET -k https://<ELASTICSEARCH_HOST_IP>:9200 -u <ELASTICSEARCH_USER>:<ELASTICSEARCH_USER_PASSWORD>

You need to replace the placeholders in the command:
- <ELASTICSEARCH_HOST_IP>
- <ELASTICSEARCH_USER> 
- <ELASTICSEARCH_USER_PASSWORD>

where:
- <ELASTICSEARCH_HOST_IP>: Elasticsearch host IP.

- <ELASTICSEARCH_USER>: username of an Elasticsearch user with privileges to do the request

- <ELASTICSEARCH_USER_PASSWORD>: user's password

For example, assuming that:
The Elasticsearch host IP is: 127.0.0.1
A privilegied user with username: admin
The password for the privilegied user: mypassword

the command will be done:
curl -XGET -k https:// 127.0.0.1:9200 -u admin:mypassword

[John Anderson]

Hi Antonio, 

Really sorry for not picking that up. I have being mainly doing network stuff for the past 7 years so lets say I am very rusty with server configs. The output from the command you asked me to run was:

{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "YKi5PKciQs-XIr6blh5WQQ",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss",
    "build_type" : "deb",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },

 "tagline" : "You Know, for Search"
}

Thanks

John

[Antonio David Gutiérrez]

Hi John,

It seems you have an Elasticsearch with the OSS distribution and version 7.10.2.

Could you share the version of Filebeat you have installed? Run the next command in the host where Filebeat is installed and you got the error.
filebeat version

[John Anderson]

Hi Antonia,

This the version I have:

 filebeat version
filebeat version 7.17.3 (amd64),

Thanks 

John

[Antonio David Gutiérrez]

Hi @John

If you followed the guide, you should have installed Filebeat 7.10.2 OSS distribution but you have Filebeat 7.17.3. For some reason, you installed or updated the Filebeat version.

Try to remove the current Filebeat and install Filebeat 7.10.2 OSS distribution. After reinstalling the mentioned Filebeat, ensure you have the 7.10.2 version (review the filebeat version) of Filebeat and you have the configuration correctly or redo it if necessary as explained in the guide: https://documentation.wazuh.com/4.2/installation-guide/open-distro/all-in-one-deployment/all-in-one.html#installing-filebeat

[John Anderson]

Hi Antonio,

I removed Filebeat and then tried following the instructions in the link to reinstall it using the command - "apt remove --purge filebeat -y". I am at the stage where I am trying to move the certs to the folder /etc/filebeat/certs but get the following error

 mv ~/certs/filebeat* /etc/filebeat/certs/
mv: cannot stat '/root/certs/filebeat*': No such file or directory

The only 2 files in /root/certs is shown below

 ls /root/certs/
root-ca.key  root-ca.pem

For some reason there is no file or directory "filebeat" in root/certs

root@wazuh:/etc/filebeat/certs# cd /root/certs/
root@wazuh:~/certs# ls
root-ca.key  root-ca.pem

I run the  filebeat test output and got the following error because I think there is no filebeat certificates in the folder /etc/filebeat/certs:

Error initializing output: 1 error: open /etc/filebeat/certs/filebeat.pem: no such file or directory /etc/filebeat/certs/filebeat.pem accessing 'output.elasticsearch' (source:'/etc/filebeat/filebeat.yml')

Thanks again

John

[John Anderson]

Hi Antonio,

I have just run the command "Filebeat" version and now I am running -

 filebeat version
filebeat version 7.17.5 (amd64), libbeat 7.17.5

I am on a higher version than before. I used the link you sent 

All the best

John

[John Anderson]

Hi Antonio,

I removed the newer version of filebeat and reinstalled it with the correct version using the command  apt-get install filebeat=7.10.2

I went through the procedure again and got the same error

 filebeat test output

Error initializing output: 1 error: open /etc/filebeat/certs/filebeat.pem: no such file or directory /etc/filebeat/certs/filebeat.pem

Thanks

John

[John Anderson]

HI Antonio,

I renewed the   Elasticsearch certificates but now I get the following error when I test filebeat:

elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK

  TLS...
    security: server's certificate chain verification is enabled
    handshake... ERROR x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Wazuh")

Is there anything obvious - should I try a reinstall and follow the instructions in the link you sent me.

Thanks again

John

[Antonio David Gutiérrez]

HI John,

Did you replace the files related to certificates that Filebeat is using?

[John Anderson]

Hi Antonio,

I followed the procedure in step 5

1. Copy the Elasticsearch certificates into /etc/filebeat/certs:

   # mkdir /etc/filebeat/certs # cp ~/certs/root-ca.pem /etc/filebeat/certs/ # mv ~/certs/filebeat* /etc/filebeat/certs/

After that when I ran "filebeat test output" I got the error above

Thanks

John

[Antonio David Gutiérrez]

Hi John,

When you said you renewed the Elasticsearch certificates, did you mean you redid the certificates used by Elasticsearch? If the response is yes, then did you move the new certificates to the Filebeat host and configurated them correctly? Are you sure Filebeat is using the new certificates and not the older ones?

How do you create the new certificates? Did you use the proposed method in the guide? https://documentation.wazuh.com/4.2/installation-guide/open-distro/all-in-one-deployment/all-in-one.html#certificates-creation

For another hand, you could ignore verifying the SSL by adding the next setting to the Filebeat configuration (filebeat.yml file)
output.elasticsearch.ssl.verification_mode: none

[John Anderson]

Hi Antonio,

I edited the file filebeat.yml and added the line "output.elasticsearch.ssl.verification_mode: none" but when I ran the test -  filebeat test output, I still got the error below.

elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... ERROR x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Wazuh")

Thanks

John

[Antonio David Gutiérrez]

Hi John,

Do you have these lines in the Filebeat configuration?

  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
  ssl.key: "/etc/filebeat/certs/filebeat-key.pem"

If you don't want that verify SSL, then you should remove them and add the output.elasticsearch.ssl.verification_mode: none setting. If you are defining the `output.elasticsearch` setting, you could append to the block: ssl.verification_mode: none
as is done for the settings mentioned before. I mean, something like this block:

output.elasticsearch:
  hosts: ["127.0.0.1:9200"]
  protocol: https
  username: "admin"
  password: "admin"

  ssl.verification_mode: none

Note the ssl.verification_mode setting is specified in the configuration file.

If the Filebeat-Elasticsearch connection is ok, then I would recommend you restarting Filebeat service after that check.

If you continue with the problem, share the Filebeat configuration file. Obfuscate the sensitive data.

[John Anderson]

Hi Antonio,

I disabled the ssl.verification mode then checked filebeat test and got the following output

  security... WARN server's certificate chain verification is disabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR Connection marked as failed because the onConnect callback failed: Filebeat requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Filebeat.

Also I have attached an image of the template alert on the gui.

Thanks 

John

[Antonio David Gutiérrez]

Hi John,

The error that you shared in the screenshot is because the Wazuh plugin for Kibana checks that you have a template defined for the index pattern of alerts. In the Wazuh setup, Filebeat is in charge to send the template to Elasticsearch, but in your case, this can't do it because of Filebeat can't connect to your Elasticsearch.

The next error:

 talk to server... ERROR Connection marked as failed because the onConnect callback failed: Filebeat requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of Filebeat.

is related to you are using different Elasticsearch/Filebeat distributions. You have to use the same distribution of both products to they work and can connect correctly. Taking in account you have an environment with Open Distro for Elasticsearch, this uses under the hood an Elasticsearch OSS distribution. But you have installed a Filebeat default distribution. As I mentioned in a previous message, they are not compatible. You have to install a Filebeat OSS distribution, so Elasticsearch and Filebeat will be OSS distributions and will be compatibles. 

Ensure you install Filebeat OSS distribution. Take a look at the message: https://groups.google.com/g/wazuh/c/MHfwmBIjw7k/m/9AWLYr0_BgAJ

[John Anderson]

Hi Antonio,

I did uninstall filebeat and followed the link you sent.

I followed this link https://documentation.wazuh.com/4.2/installation-guide/open-distro/all-in-one-deployment/all-in-one.html#installing-filebeat so am not sure why I don't have the Filebeat OSS distribution:

# apt-get install filebeat

# curl -so /etc/filebeat/filebeat.yml https://packages.wazuh.com/resources/4.2/open-distro/filebeat/7.x/filebeat_all_in_one.yml

I have uninstalled it again and am going to try reinstalling it. How can I make sure that I am installing the OSS distribution?

When I removed Filebeat there were a couple of warnings as shown below. The certs are still in the /etc/filebeat directory - should I remove the directory /etc/filebeat?

The following packages will be REMOVED:
  filebeat*
0 upgraded, 0 newly installed, 1 to remove and 113 not upgraded.
After this operation, 123 MB disk space will be freed.
(Reading database ... 203688 files and directories currently installed.)
Removing filebeat (7.10.2) ...
(Reading database ... 202483 files and directories currently installed.)
Purging configuration files for filebeat (7.10.2) ...
dpkg: warning: while removing filebeat, directory '/etc/filebeat' not empty so not removed
dpkg: warning: while removing filebeat, directory '/usr/share/filebeat/module' not empty so not removed

Thanks again

John

[Antonio David Gutiérrez]

Hi Jonh,

When uninstalling Filebeat, you could remove the /etc/filebeat/  and /usr/share/filebeat/ directories. After installing the exptected Filebeat OSS distribution, then you follow the steps to configure Filebeat mentioned here: https://documentation.wazuh.com/4.2/installation-guide/open-distro/all-in-one-deployment/all-in-one.html#installing-filebeat (Filebeat configuration, move certificates, download wazuh module for Filebeat, etc...)

I see in the shared logs:

The following packages will be REMOVED:
  filebeat*
0 upgraded, 0 newly installed, 1 to remove and 113 not upgraded.
After this operation, 123 MB disk space will be freed.
(Reading database ... 203688 files and directories currently installed.)
Removing filebeat (7.10.2) ...
(Reading database ... 202483 files and directories currently installed.)
Purging configuration files for filebeat (7.10.2) ...
dpkg: warning: while removing filebeat, directory '/etc/filebeat' not empty so not removed
dpkg: warning: while removing filebeat, directory '/usr/share/filebeat/module' not empty so not removed

that mentions the version of Filebeat is 7.10.2, and this doesn't match with this response that you did in a previous message. Anyways, try to remove the current Filebeat you have installed and install Filebeat OSS distribution 7.10.2.

If you had Filebeat 7.17.5 previously as mentioned in this message, maybe you have added the Elastic package repository and when you run the command to install Filebeat using your package manager, this installed the package of Elastic repository instead of the Wazuh repository or you managed to upgrade of some way.

Anyways, you should ensure you install Filebeat OSS distribution 7.10.2 in each Wazuh manager node.

How can I make sure that I am installing the OSS distribution?

I was researching how to check if the installed package is an OSS distribution, but unfortunately, I didn't find a way to do it, so you should follow the steps to install the correct package.

Alternatives to install Filebeat OSS distribution 7.10.2:
- Using the package manager:
Before installing Filebeat, if you go to use the package manager, check what repositories you have added. If you have the official Elastic repository and you install Filebeat, the package manager could install one package of that repository instead of the Wazuh, causing you finally install a default distribution or a recent version, but you need Filebeat OSS distribution 7.10.2.

- Download the file depending on your operating system and install it: https://www.elastic.co/es/downloads/past-releases/filebeat-oss-7-10-2

[John Anderson]

Hi Antonio,

I followed the procedure in step 5

1. Copy the Elasticsearch certificates into /etc/filebeat/certs:

   # mkdir /etc/filebeat/certs # cp ~/certs/root-ca.pem /etc/filebeat/certs/ # mv ~/certs/filebeat* /etc/filebeat/certs/

After that when I ran "filebeat test output" I got the error above

Thanks

John

[John Anderson]

Hi Antonio,

I followed the procedure in step 5

1. Copy the Elasticsearch certificates into /etc/filebeat/certs:

   # mkdir /etc/filebeat/certs # cp ~/certs/root-ca.pem /etc/filebeat/certs/ # mv ~/certs/filebeat* /etc/filebeat/certs/

I downloaded filebeat  Filebeat OSS distribution 7.10.2. and installed it

After that when I ran "filebeat test output" I got the error below

 filebeat test output
Error initializing beat: error loading config file: type 'string' is not supported on top level of config, only dictionary or list (source:'/etc/filebeat/filebeat.yml').

The contents of the file filebeat.yml is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>resources/4.2/open-distro/filebeat/7.x/filebeat_all_in_one.ym</Key><RequestId>RMFAWE4Y5QSGDP00</RequestId><HostId>Y6TNeVFvajAIScwTx0va3Klxz1UIYehQ2Pbrdm6vvg1hoeHCei+B9TlZMy4pOOtj+BVgFx7rMec=</HostId></Error>

Thanks

John