Interaction with ElasticSearch db and Kibana for custom integration in Wazuh

[Adriano Pace]

Hello everyone,

i explain my situation:

I have two hosts, manager and agent deployed in a controlled enviroment for a Thesys project.

My goal is to write a python script that uses Machine learning to analyze data collected from Wazuh in the ElasticSearch db and creates a custom dashboard with Kibana.

My questions are:

1) Are these apis the "only" availables for querying elastic search trought my script ?

2) How can i access Kibana from my manager host ? I mean graphically if possible and trought apis in the script to generate a custom dashboard using the data i processed ?

To install Wazuh i used the quick start script that creates the 3 manager entities and then i created and deployed the agent from the Wazuh dashboard, to i have to install Kibana separately or my installation is complete like this ?

thanks so much in advance for the response,

cheers :)

[Antonio David Gutiérrez]

Hi Adriano,

1. When you deploy Wazuh with Elastic stack (Kibana, Elasticsearch and Filebeat), the data is stored in different places:
- Local databases in the Wazuh server/manager hosts. You could use the Wazuh API included in each Wazuh server/manager to query this data. The link you shared, https://documentation.wazuh.com/current/user-manual/api/reference.html, is related to the Wazuh API.
- Data related to alerts is indexed to Elasticsearch through Filebeat from each Wazuh server/manager. You will need to use the Elasticsearch API.

Depending on the data you need to analyze in your script, you could need to do queries to Wazuh API, Elasticsearch API, or both.

2. Kibana allows to creation of dashboards through the Dashboards application. These dashboards are composed of visualizations that use the data stored in Elasticsearch. It means, that if you want to use these dashboards, the data to represent must be indexed in Elasticsearch. For example, if you pretend to display information that your script generates, then you must index this data if doesn't exist previously to Elasticsearch so the dashboards are able to use that data in the visualizations.

Kibana can be accessible through:
- graphically through the browser. You should use the URL of the machine that is running Kibana.
- programmatically through the API endpoints.

Kibana dashboards are usually created through the interaction of a user with the UI using the Dashboards plugin or importing from files. If you want to create the dashboard from your script, then you could need to interact with the saved objects API of Kibana. I found a related topic that metions this: https://discuss.elastic.co/t/creating-dashbard-and-visualisation-with-python/270235.

If you used the quick start script and you mentioned to Wazuh dashboard, I guess you deployed the Wazuh stack. This stack is composed of:
- Wazuh server/manager (with Filebeat)
- Wazuh dashboard
- Wazuh indexer

Wazuh dashboard and Wazuh indexer are applications based on OpenSearch Dashboards and OpenSearch distributions, which are forks of Kibana and Elasticsearch 7.10.2.

Depending on your requirements, you could create the dashboards on the Wazuh dashboard instead of Kibana.

If you need to create the dashboards on Kibana and you deployed the Wazuh stack (Wazuh server, Wazuh dashboard and Wazuh indexer), you will need to install a Kibana and Elasticsearch. If you are not interested in the Wazuh stack, then you could be interested in deploying Wazuh with Elastic stack instead https://documentation.wazuh.com/current/deployment-options/elastic-stack/index.html. But you should know that there will be no Wazuh 4.6.0 plugins for the currently supported Kibana versions (7.16.x and 7.17.x). The Wazuh plugin for Kibana lets to manage and see the data of Wazuh. See the compatibility matrix of the Kibana plugin: https://github.com/wazuh/wazuh-dashboard-plugins/wiki/Compatibility.

[Adriano Pace]

Thank you very much Antonio for your precise and detailed answer.

I defenitely understand the connection between Kibana and ElasticSearch now and i also understand why.

Taking advantage of what you said "Depending on your requirements, you could create the dashboards on the Wazuh dashboard instead of Kibana." , can i ask you how this could be possible ?

Best regards and thank you so much, you were very helpful.

[Antonio David Gutiérrez]

If you don't need Kibana for some reason (feature, organization requirement, etc...) and you choose to deploy Wazuh using the Wazuh stack (Wazuh dashboard and Wazuh indexer), then you could create the dashboard on Wazuh dashboard instead.

If you are asking how to create the dashboard programmatically on the Wazuh dashboard or Kibana, you should know the dashboard (and visualization) definitions are stored in documents of special indices on the Wazuh indexer or Elasticsearch. The name of these indices usually starts with .kibana and they store saved objects of dashboards, visualizations, searches, or configurations. For more information about saved objects, you can review the documentation provided by Kibana docs: https://www.elastic.co/guide/en/kibana/7.10/managing-saved-objects.html. The previous link is related to Kibana 7.10.x and could apply to the current Wazuh dashboard, if you pretend to use another version of Kibana, see the appropriate documentation. It means you could index a compatible document instead of doing it through the dashboard or visualization apps. It should work.

If you want to create the dashboards programmatically, my recommendation is to follow the next steps:
0. If the data to represent in the dashboard is not indexed in Wazuh indexer/Elsaticsearch by Wazuh and your script will do, then resolve this in the first place.
1. Get the document schema definition related to dashboard or visualization
1.1. Create the desired dashboards using the Dashboard and Visualize (or Lens) applications of the Wazuh dashboard/Kibana. Ensure you save the dashboard/visualization definition.
1.2. Search and review the document content that is related to the dashboard/visualization definition. This should be stored in some index whose name starts with .kibana. You could use the Wazuh indexer/Elasticsearch API to search the documents of the mentioned indices.
3. Knowing the document content of the dashboard/visualizations definition, you should be able to index programmatically the same document into the expected index using the Wazuh indexer/Elasticsearch API or Wazuh dashboard/Kibana saved objects API. More information about the saved objects API is provided by Kibana docs https://www.elastic.co/guide/en/kibana/7.10/saved-objects-api.html. The previous link is related to Kibana 7.10.x and could apply to the current Wazuh dashboard, if you pretend to use another version of Kibana, see the appropriate documentation.

[Adriano Pace]

Really helpful, thank you so much.

Best regards,

Adriano