Wazuh/Opensearch error with the "generate and download" feature
Uni AT
Dear Wazuh community, support team,
Our deployment consists of 3 Wazuh Server docker containers (v4.14.6) with an OpenSearch (v2.19.4) integration for logs indexing.
We are facing an error to generate and download reports from the saved searches. Although we are aware that this might be an issue more related to OpenSearch than Wazuh itself, we’d like to know if anyone ever got this problem and how it was fixed.
The error is reproduced below with an account that has administrative rights to exclude a permission issue.
We saved the search security_logs_last_24h which relates to all the entries contained in the index pattern security-auditlog*. The result is those 2 saved objects:
The saved search loads successfully in the dashboard:
When opening the search and going to Reporting à Generate and download à Generate CSV or Generate XLSX, we get the following error message:
While performing this action, those 2 failed API calls can be seen (multitenancy is NOT enabled):
And this, despite the fact that our saved object does have the same ID cd24f470-9566-11f1-9e17-8f5036b3726d that is reportedly not found, and is referencing well the security-auditlog object with the ID f99a1910-9564-11f1-9e17-8f5036b3726d.
Please see the attached file with the content of the saved object.
Do you have an idea what the issue might be?
Thank you a lot in advance,
Olamilekan Abdullateef Ajani
This usually points to a saved-object scope problem. In other words, the Saved Objects UI can see the object, but the Reporting backend is looking for it in a different tenant or .kibana index.
Please confirm whether multitenancy is disabled on all dashboard containers. Check these settings on every dashboard node /etc/wazuh-dashboard/opensearch_dashboards.yml
opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.enable_private: false
opensearch_security.multitenancy.tenants.enable_global: true
Then check where the saved search actually exists:
GET _cat/indices/.kibana*?v
GET .kibana/_doc/search:cd24f470-9566-11f1-9e17-8f5036b3726d
If it is not in .kibana, repeat the GET against any .kibana_* index you find. If the object exists in a tenant-specific index, recreate or re-import the saved search in the Global tenant.
Since you have multiple server containers, also test each container directly. A config difference between containers could also explain why the search opens in the UI but report generation fails.
I await feedback from you
Uni AT
Hello Olamilekan,
At first, thank you very much for your detailed answer.
I can confirm you that multitenancy is disabled on the dashboard container:
XXX@wazuhdashboard01:/etc/wazuh-dashboard# cat opensearch_dashboards.yml | grep multi
opensearch_security.multitenancy.enabled: false
XXX@wazuhdashboard01:/etc/wazuh-dashboard/local# cat opensearch_dashboards_local.yml | grep multi
opensearch_security.multitenancy.enabled: false
The index where the saved searches is .kibana_wazuh_1 instead of .kibana. You can find attached the outputs of the GET commands.
Is this a custom index or one that comes with a default Wazuh setup? And since the saved search works well when loading it, so is it only the reporting function that relies on the .kibana index ?
If so, we can probably try reindex the saved search from .kibana_wazuh_1 to .kibana using this request:
POST _reindex
{
"source": {
"index": ".kibana_wazuh_1",
"query": {
"term": {
"_id": "search: cd24f470-9566-11f1-9e17-8f5036b3726d"
}
}
},
"dest": {
"index": ".kibana"
}
}
But this is rather suboptimal. Is there any suggested way to fix it?
Have a nice day,
Olamilekan Abdullateef Ajani
.kibana_wazuh_1
not in the default .kibana saved-object index.
So the reporting process is likely looking in the default .kibana alias/index, while your Wazuh Dashboard is using .kibana_wazuh_1. That explains why the search opens normally in the UI but the report generation says the saved object does not exist.
I would robably not use manual _reindex as the main fix. It may work for one object, but it can become disjointed because the saved search also references the index pattern object, and future edits would still be saved to .kibana_wazuh_1.
What to do is, run:
grep -iE "opensearchDashboards.index|kibana.index" \
/etc/wazuh-dashboard/opensearch_dashboards.yml \
/etc/wazuh-dashboard/local/opensearch_dashboards_local.yml
Check this on every dashboard container.
If opensearchDashboards.index is set to .kibana_wazuh, then that is why your objects are stored there. OpenSearch Dashboards stores saved objects such as searches, index patterns, visualizations, and dashboards in its configured saved-object index.
Recommended path:
Export the saved objects from the current UI as .ndjson.
Align all dashboard containers to use the same saved-object index, the default .kibana.
Restart the dashboard containers.
Import the .ndjson again through the UI.
Recreate/generate the report.
Before changing it, confirm whether the existing .kibana_* objects are still used by another dashboard instance.
So yes, .kibana_wazuh_1 looks like a custom/overridden saved-object index, not a tenant issue. The best fix is to standardize the dashboard saved-object index across all containers, not manually copy one saved search.
Uni AT
