Errori migrating to wazuh-index from AiO Wazuh 4.2.5 to 4.3.1

[Valerio Vinci]

Hi Team,

I'm trying to migrate from my Wazuh all in one v4.2.5 to Distributed Wazuh 4.3.1.

Before to add new hosts, I'm just updating and migrating the existing one.

I've migrated the wazuh-dashboard with no problem but with wazuh-index I'm having some trouble..

I've followed the guide linked at: https://documentation.wazuh.com/current/migration-guide/wazuh-indexer.html

but when I try to start the service I got these error:

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.cli.Command.main(Command.java:101)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log

Checking wazuh-cluster.log I found:

[2022-05-20T00:08:18,220][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-20T00:08:18,223][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-20T00:08:18,226][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-20T00:08:18,229][ERROR][c.a.o.s.a.BackendRegistry] [node-1] Not yet initialized (you may need to run securityadmin)
[2022-05-20T00:16:45,869][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [node-1] ElasticsearchException[Illegal parameter in http or transport request found.
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]
[2022-05-20T00:16:45,876][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [node-1] ElasticsearchException[Illegal parameter in http or transport request found.
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]
[2022-05-20T00:16:46,858][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [node-1] ElasticsearchException[Illegal parameter in http or transport request found.
[2022-05-20T00:16:46,859][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [node-1] ElasticsearchException[Illegal parameter in http or transport request found.
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[elasticsearch-7.10.2.jar:7.10.2]

Now.. when I try to run securityadmin:

[root@xxx ~]# export JAVA_HOME=/usr/share/elasticsearch/jdk/

[root@xxx ~]# /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh    -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/    -icl -nhnv -cacert /etc/wazuh-indexer/certs/root-ca.pem     -cert /etc/wazuh-indexer/certs/admin.pem     -key /etc/wazuh-indexer/certs/admin-key.pem    -h localhost

Open Distro Security Admin v7

Will connect to localhost:9300

ERR: Seems there is no Elasticsearch running on localhost:9300 - Will exit

Also tried specifing "-P 9200" or the Java and cert path in wazuh-indexer nut nothing change..

Now both elastichsearch and wash-indexer is not starting... what I have to do?

[Federico Gustavo Caffieri]

Hello valeavin,

Seeing the error that the securityadmin returns, it seems that you do not have the indexer running on the localhost.

Could you share your indexer configuration file? It's inside /etc/wazuh-indexer/opensearch.yml. All IPs and domains.

By default the securityadmin tries to connect to wazuh indexer through the localhost ip, if you have it running on another IP, you can add the -h parameter to place the corresponding IP.

Have you repeated the migration steps on each 1 of the cluster nodes?

Could you share the output of the followings commands:
1- journalctl -xe -u wazuh-indexer
2- curl -k -u USER:PASS https://YOUR_INDEXER_IP:9200/_cluster/health?pretty
3- curl -k -u USER:PASS https://YOUR_INDEXER_IP:9200/_cat/indicesCoul you share the indexer logs? You can find it into: /var/log/wazuh-indexer/wazuh-cluster.log

Thanks.

[Valerio Vinci]

Hello,

Thanks for your email. Below the requests:

indexer config file:

 cat /etc/wazuh-indexer/opensearch.yml
network.host: "0.0.0.0"
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
#- "node-2"
#- "node-3"
cluster.name: "wazuh-cluster"
discovery.seed_hosts:
  - "172.20.41.201"
#  - "node-2-ip"
#  - "node-3-ip"
node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false

plugins.security.audit.type: internal_opensearch
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Docu,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=node-1,OU=Docu,O=Wazuh,L=California,C=US"
#- "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

journalctl:

-- Logs begin at Fri 2022-05-20 03:39:44 CEST, end at Fri 2022-05-20 22:36:50 CEST. --

May 20 03:39:51 xxx.xxx systemd[1]: Starting Wazuh-indexer...

-- Subject: Unit wazuh-indexer.service has begun start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit wazuh-indexer.service has begun starting up.

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:431)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/java.nio.file.Files.newDirectoryStream(Files.java:476)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at java.base/java.nio.file.Files.walkFileTree(Files.java:2840)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:232)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:142)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.cli.Command.main(Command.java:101)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)

May 20 03:39:57 xxx.xxx systemd-entrypoint[1208]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log

May 20 03:39:57 xxx.xxx systemd[1]: wazuh-indexer.service: main process exited, code=exited, status=1/FAILURE

May 20 03:39:57 xxx.xxx systemd[1]: Failed to start Wazuh-indexer.

-- Subject: Unit wazuh-indexer.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit wazuh-indexer.service has failed.

--

-- The result is failed.

May 20 03:39:57 xxx.xxx systemd[1]: Unit wazuh-indexer.service entered failed state.

May 20 03:39:57 xxx.xxx systemd[1]: wazuh-indexer.service failed.

curl:

[root@xxx ~]# curl -k -u xxx:xxxx https://127.0.0.1:9200/_cluster/health?pretty
curl: (7) Failed connect to 127.0.0.1:9200; Connection refused

[root@xxx ~]# curl -k -u xxx:xxxx https://127.0.0.1:9200/_cat/indices
curl: (7) Failed connect to 127.0.0.1:9200; Connection refused

[Federico Gustavo Caffieri]

Hello valeavin, sorry for the delay in response.

From what you see in the output of the journal command, it looks like a permissions problem on the /etc/wazuh-indexer/certs directory.

You could grant the permissions indicated by the documentation on this directory:
https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#deploying-certificates

chmod 500 /etc/wazuh-indexer/certs 

chmod 400 /etc/wazuh-indexer/certs/* 

chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

Then try to restart the wazuh indexer service.

If you don't solve the problem, could you send us the requested indexer log? 

You can find it in: /var/log/wazuh-indexer/wazuh-cluster.log

Thanks

[Valerio Vinci]

Hi,

Many Thanks.

I have already run these command in migration process:

-r--------. 1 wazuh-indexer wazuh-indexer 1704 May 20 00:18 admin-key.pem
-r--------. 1 wazuh-indexer wazuh-indexer 1103 May 20 00:18 admin.pem
-r--------. 1 wazuh-indexer wazuh-indexer 1708 May 20 00:18 indexer-key.pem
-r--------. 1 wazuh-indexer wazuh-indexer 1224 May 20 00:18 indexer.pem
-r--------. 1 wazuh-indexer wazuh-indexer 1180 May 20 00:18 root-ca.pem

but reading the guide linked by you I've seen a detail regarding the node certificate that is missing in my environment... 

as you can see, there's only the admin and root certificate but not the "node-1" certificate.

I don't know if the problem could be there or not. this environment is an All in One server 

[Federico Gustavo Caffieri]

Hello valeven, sorry for the delay in response. 

If you have followed all the steps in the documentation: https://documentation.wazuh.com/current/migration-guide/wazuh-indexer.html you should have the correct certificates. 

Could you share the /etc/elasticsearch/elasticsearch.yml file with us? or a comparison against /etc/wazuh-indexer/opensearch.yml ? 

 Thanks

[Valerio Vinci]

HI

elastic search.yml:

[root@xxx ]# cat /etc/elasticsearch/elasticsearch.yml

network.host: 0.0.0.0
node.name: node-1
cluster.initial_master_nodes:
        - node-1
#        - node-2
#        - node-3

# discovery.seed_hosts:
#         - <elasticsearch_ip_node1>
#         - <elasticsearch_ip_node2>
#         - <elasticsearch_ip_node3>

cluster.name: wazuh-cluster

opendistro_security.ssl.transport.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch.pem
opendistro_security.ssl.transport.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch.pem
opendistro_security.ssl.http.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem
opendistro_security.nodes_dn:
- CN=node-1,OU=Docu,O=Wazuh,L=California,C=US
# - CN=node-2,OU=Docu,O=Wazuh,L=California,C=US
# - CN=node-3,OU=Docu,O=Wazuh,L=California,C=US
opendistro_security.authcz.admin_dn:
- CN=admin,OU=Docu,O=Wazuh,L=California,C=US

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
node.max_local_storage_nodes: 3

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

open search.yml:

[root@xxx ]# cat /etc/wazuh-indexer/opensearch.yml

network.host: "0.0.0.0"
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
#- "node-2"
#- "node-3"
cluster.name: "wazuh-cluster"
discovery.seed_hosts:

  - "127.0.0.1"

[Federico Gustavo Caffieri]

Hello valeavin, it seems that you have everything OK.
Taking into account the error that occurs "org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs", i would also like to check the permissions of the directory.
Could you send me the result of the following command?
ls -la /etc/wazuh-indexer
Thanks

[Valerio Vinci]

Hello,

below the output of the request

[root@xxx certs]# ls -la /etc/wazuh-indexer
total 44
drwxr-x---.  6 wazuh-indexer wazuh-indexer   274 May 20 01:46 .
drwxr-xr-x. 90 root          root           8192 May 20 03:39 ..
dr-x------.  2 wazuh-indexer wazuh-indexer   105 May 20 00:18 certs
-rw-rw----.  1 wazuh-indexer wazuh-indexer  2352 Apr  5 19:50 jvm.options
drwxr-x---.  2 wazuh-indexer wazuh-indexer     6 Apr  5 19:50 jvm.options.d
-rw-rw----.  1 wazuh-indexer wazuh-indexer 11646 Apr  5 19:50 log4j2.properties
-rw-rw----.  1 wazuh-indexer wazuh-indexer   196 May 20 00:17 opensearch.keystore
-rw-------.  1           990           990    73 May 20 00:17 .opensearch.keystore.initial_md5sum
drwxr-x---.  2 wazuh-indexer wazuh-indexer    31 May 20 01:46 opensearch-observability
drwxr-x---.  2 wazuh-indexer wazuh-indexer    35 May 20 01:46 opensearch-reports-scheduler
-rw-rw----.  1 wazuh-indexer wazuh-indexer  2086 May 25 11:53 opensearch.yml
-rw-rw----.  1           990           990  1645 May 20 00:19 opensearch.yml.rpmsave

[Federico Gustavo Caffieri]

Valerio, 

After 2 meetings we had, we managed to get the cluster working. 

We encountered several problems, as a summary, I will list them: 

- At first we found that the installation of the wazuh-indexer was wrong and incomplete, with the reinstallation of it following the guide it was corrected. 

- Then Filebeat did not raise correctly. This was due to an incompatibility in the installed versions, on the one hand you had Filebeat 7.17 installed and on the other opensearch 7.10. After correcting this it worked correctly. 

- Another problem was the Dashboard, which in principle we found 3 problems: 

1- The first was the migration process that was cut and for this reason it was left unfinished. The plugin was not installed correctly, we proceeded to reinstall Dashboard and the plugin manually. 

2- The second was due to the fact that the firewall was blocking port 443, this is usually limited in centos 7, after changing the listening port to 5601 it was resolved. 

3- Finally, we found ourselves with the problem of the passwords which were partially modified, in principle we tried to modify them by hand, but at some point and in some service this was incomplete. After modifying all of them again with the wazuh-password-tool, the web correctly raise. 

I close the thread, I hope I have been of help.

Any other questions or queries, you can contact us through the enabled channels. 

Thank you very much in advance for your patience and for the meetings that helped solve your problem. 

Thanks again and regards.