Migrating from Opendistro to opensearch

[Dhruvin Shah]

Hi All

I have migrated to opensearch and upgraded wazuh manager after it. As of now wazuh-manager, wazuh-dashboard, wazuh-indexer and filebeat are running fine but when I access the webpage it says - Wazuh dashboard server is not ready yet  

Can someone assist me to resolve this issue?

[Alberto Rodriguez]

Did you migrate your kibana.yml settings? Does the file contain opensearch.username and password?

If so, remove the opensearch-dashboard-keystore by running the mentioned binary, removing the two entries created by default for the package. 

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/ac1e3b63-9ca0-4c24-b709-39aa710d3b4bn%40googlegroups.com.

[Abdul Samad]

Restart the services

systemctl restart wazuh-manager

systemctl restart wazuh-indexer

systemctl restart wazuh-dashboard

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAFVXG3z7m-APDt0zsVYMuntDsvkuk9Y8659uBn0aWRpC35JW4A%40mail.gmail.com.

[Dhruvin Shah]

Hi 

In  /etc/wazuh-dashboard/opensearch_dashboards.yml following is the configuration, I see the username and password are not mention.

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://127.0.0.1:9200
opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wazuh

For - remove the opensearch-dashboard-keystore by running the mentioned binary, which is the binary? Sorry, I do not know about it.

Appreciate all the support.

[Dhruvin Shah]

Thanks, Samad but the issue still persists.

[Dhruvin Shah]

I am seeing an error now in wazuh-dashboard

"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

[Alberto Rodriguez]

Hello Dhruvin

Sorry for the late reply. The ECONNREFUSED error should mean that the Wazuh indexer is not working. Please check the Wazuh indexer state by running systemctl status wazuh-indexer. If it’s not running, check the reason on the logs /var/log/wazuh-indexer/wazuh-cluster.log (or similar, depending on your configuration file defined cluster name). If you need help with that, please share the logs.

Regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/453cbd7c-30c1-48a8-b8d1-8af4aac4016fn%40googlegroups.com.

[Dhruvin Shah]

Hi  Alberto 

Thanks for assisting.

I see wazuh-indexer service is running but with some issues:

[root@ip-10-184-11-13 ec2-user]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-05-26 11:22:25 UTC; 10min ago
     Docs: https://documentation.wazuh.com
 Main PID: 986 (java)
    Tasks: 164 (limit: 99006)
   Memory: 2.8G
   CGroup: /system.slice/wazuh-indexer.service
           └─986 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=>

May 26 11:22:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: WARNING: An illegal reflective access operation has occurred
May 26 11:22:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicT>
May 26 11:22:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: WARNING: Please consider reporting this to the maintainers of io.protost>
May 26 11:22:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: WARNING: Use --illegal-access=warn to enable warnings of further illegal>
May 26 11:22:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: WARNING: All illegal access operations will be denied in a future release
May 26 11:22:25 ip-10-184-11-13.eu-west-1.compute.internal systemd[1]: Started Wazuh-indexer.
May 26 11:29:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: java.lang.OutOfMemoryError: Java heap space
May 26 11:29:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: Dumping heap to data ...
May 26 11:29:20 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: Unable to create data: File exists
May 26 11:32:19 ip-10-184-11-13.eu-west-1.compute.internal systemd-entrypoint[986]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionH Exception : java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "opensearch[node-1][generic][T#31]"

Also, there were many errors relating to shards which are found in /var/log/wazuh-indexer/wazuh-cluster.log, attaching log file for your reference.

Please help how to resolve these errors.

Thank you

Kind Regards

[Alberto Rodriguez]

Hello Dhruvin
You run on an out-of-memory issue. Please, check the /etc/wazuh-indexer/jvm.options, the values Xmx and Xms should have about a half of your total host RAM, and restart the wazuh indexer. If those values are correct, you must check why an out-of-memory occurred, maybe other processes?

Regarding your log, the only ERROR message you have is this one:

[2022-05-26T11:24:26,757][ERROR][o.o.i.i.MetadataService ] [node-1] failed reason: {"index":".opendistro-ism-config","type":"_doc","id":"4YNzC-k0S4SuStIlEVrK4w#metadata","cause":{"type":"unavailable_shards_exception","reason":"[.opendistro-ism-config][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.opendistro-ism-config][0]] containing [330] requests]"},"status":503}, UnavailableShardsException[[.opendistro-ism-config][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.opendistro-ism-config][0]] containing [330] requests]]

which means that one of your indices resulted orphaned because of an out of memory. You probably need to delete the index .opendistro-ism-config, used for ISM configuration, and re-configure the plugin settings.

Regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/5d63762f-1e6f-445c-ab91-68d6b8581271n%40googlegroups.com.

--

[Dhruvin Shah]

Hi Alberto Rodriguez

Thank you very much, it was the heap size issue only. Wazuh is now up and running. :D

Kind regards

[Dhruvin Shah]

Hi Alberto

Portal is up but I see Wazuh API error now which is as follow:

INFO: Current API id [default] INFO: Checking current API id [default]... INFO: Current API id [default] has some problem: 3020 - EACCES: permission denied, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml' INFO: Getting API hosts... ERROR: Error connecting to API: 2001 - EACCES: permission denied, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml' INFO: Removed [navigate] cookie ERROR: Error connecting to API: 2001 - EACCES: permission denied, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml'

I tried to check wazuh.yml but could not find it inside the mentioned path

[root@ip-10-184-11-13 wazuh-dashboard]# cd data/wazuh/config/
[root@ip-10-184-11-13 config]# ll
total 0

It seems the file is missing

could you assist on this as well?

Kind regards

[Alberto Rodriguez]

Hello 

  Sorry for the late response. Please use the attached file in the mentioned path, and modify the IP, password, etc if necessary. Assign the permissions 600 and owner wazuh-dashboard:wazuh-dashboard. Then, restart the wazuh-dashboard. 

Please let me know if it works.