OpenSearch Keystore does not work on Wazuh-Indexer. Error /usr/share/wazuh-indexer/config: No such file or directory

[Ranjith Kesavan]

We have installed Wazuh-Indexer and trying to configure Snapshot to Azure. 

When running the Opensearch-keystore add command to add the credentials, it works with the error "/usr/share/wazuh-indexer/bin/opensearch-env: line 89: cd: /usr/share/wazuh-indexer/config: No such file or directory" Still the account name and credentials are added to the keystore and we can list it. 

However, when we try to create the snapshot repository, it fails with error  " "Unable to find client with name [default]"

Not sure if this has to do anything with the config directory is no more located in /usr/share/wazuh-indexer/

Any help is appreciated

Please find the images below. 

[Emiliano Zorn]

Hello Ranjith!

Are you using any documentation for this process? If so, could you provide me with a link?

I would like to be able to reproduce the error.

Regards,

Emiliano.

[Ranjith Kesavan]

Hello Emiliano, 

Primaily I was following the steps under the blog :  Index backup management · Wazuh · The Open Source Security Platform to create snapshot from Opensearch. However the blog is not so clear about the steps performed on Opensearch, rather favors more towards Elasticsearch basic. So please find below the steps to recreate the issue :  These steps works perfect with OpenDistro and Elasticsearch but not with Wazuh-Indexer(Opensearch)

1) Create a Storage account in Azure, Create a Container under the storage account and copy the access key and Storage acocunt name(as account name) . Steps can be found at Create a storage account - Azure Storage | Microsoft Docs 

2) Add the Azure storage accout credentials to Opensearch keystore:  

•           Login to the Terminal on Wazuh-Indexer Server
•           Install "repository-azure" plugin and restart wazuh-indexer
•           Run the command "/usr/share/wazuh-indexer/bin/opensearch-keystore add azure.client.default.account" to add the Azure storate account name - 
•           Run the command "/usr/share/wazuh-indexer/bin/opensearch-keystore add azure.client.default.key" and enter the Azure storage account access key 

NOTICE THE ERROR in SECOND LINE

3) Login to Wazuh-Dashboard and from the Devtools run the command to register the snapshot repository

PUT _snapshot/wazuh-es-snapshot
{
  "type": "azure",
  "settings": {
    "container": "es-snapshot",
    "chunk_size": "500MB",
    "compress": true
  }
}

This step should register the respository and is working as expected in Opendistro. However with Wazuh-Indexer, it throws the error "Unable to find client with name [default]".  This means that Opensearch is unable to find the Azure client credentials in Keystore. I suspect its something related to the error with /usr/share/wazuh-indexer/config" directory missing.  I have tested this in multiple Wazuh-indexer instances with the exact same result. Note : I tried with installing wazuh-Indexer with both Step by step installation and initiating the cluster with SecurityAdmin and using the Install script. Both showed the same result. 

Logs from wazuh-cluster.log : 

[2022-06-15T14:09:52,962][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2022-06-15T14:09:52,976][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: false
        at org.opensearch.security.configuration.ConfigurationRepository.getConfigurationsFromIndex(ConfigurationRepository.java:375) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:321) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:306) [opensearch-security-1.2.4.0.jar:1.2.4.0]
        at org.opensearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:166) [opensearch-security-1.2.4.0.jar:1.2.4.0]
[2022-06-15T14:10:01,974][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled
[2022-06-15T14:10:01,975][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node 'node-1' initialized
[2022-06-15T14:17:19,001][INFO ][o.o.r.RepositoriesService] [node-1] put repository [wazuh-es-snapshot]
[2022-06-15T14:17:19,237][WARN ][r.suppressed             ] [node-1] path: /_snapshot/wazuh-es-snapshot, params: {pretty=true, repository=wazuh-es-snapshot}
org.opensearch.repositories.RepositoryVerificationException: [wazuh-es-snapshot] path  is not accessible on master node
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.2.4.jar:1.2.4]
[2022-06-15T14:20:31,219][INFO ][o.o.r.RepositoriesService] [node-1] update repository [wazuh-es-snapshot]
[2022-06-15T14:20:31,289][WARN ][r.suppressed             ] [node-1] path: /_snapshot/wazuh-es-snapshot, params: {pretty=true, repository=wazuh-es-snapshot}
org.opensearch.repositories.RepositoryVerificationException: [wazuh-es-snapshot] path  is not accessible on master node
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1549) ~[opensearch-1.2.4.jar:1.2.4]
        at org.opensearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1544) ~[opensearch-1.2.4.jar:1.2.4]

[Ranjith Kesavan]

Hello Team, 

Can someone help us with this?.. It looks to be a real issue with Wazuh-Indexer

[Emiliano Zorn]

Hello Ranjith!

Sorry for the late reply, i just saw this.

I will be testing this in my environment and will get back to you as soon as possible. Thanks for the detailed step-by-step guide and the attached logs, they are very helpful.

Regards.

[Ranjith Kesavan]

Hello Emiliano, 

Were you able to test this ? 

Thank you,

Ranjith Kesavan. 

[Emiliano Zorn]

Hi Ranjith, sorry for the delay in my reply.

As you say, there is a conflict problem with the path that leads to the config section within the script, the team has created an issue that I leave HERE.

We are testing to see if this issue corresponds to our version of Open search Wazuh indexer, or to Opensearch's own version.

Thanks a lot for your time and the heads up.

Regards,

Emiliano Zorn.

[Ranjith Kesavan]

Thank you Emiliano for your support on this. I will wait for the update on the Issue. 

[Ranjith Kesavan]

Hello Team, 

I see the bug https://github.com/opensearch-project/alerting-dashboards-plugin/issues/283 reported has been closed and could not find any resolution mentioned. Can you confirm if this issue will be fixed in version 4.3 or is there a work arround we can use ? 

[Emiliano Zorn]

Hello Ranjith!

Thank you for following the case with us.

Currently, we have already raised this issue to development, and they are working on it to establish a workaround until OpenSearch solves the problem, I can not ensure 100% but the incident is among the most important to solve before version 4.4.

Regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9eb8be8a-aa20-42b3-a741-aaf0a0d5891en%40googlegroups.com.

--

[Eric Vu]

Hi Wazuh Experts, 

Can you please update the status of the issue? Can you please help us suggest a workaround for that?

 

Regards, 

[Emiliano Zorn]

Hello Eric!

We are still working on this issue.

Here you can follow our progress and the actual status of this:

https://github.com/wazuh/wazuh-packages/issues/1733

Regards.

[elw...@wazuh.com]

Hello,

Just shimming in to share the guides that might help in this case:

• https://medium.com/@karkoubelwali/azure-blob-storage-as-snapshots-repository-for-wazuh-indexer-data-623799091705
  
• https://www.youtube.com/watch?v=9oax1k4wbFQ

Hope it helps.

Regards,
Wali