wazuh-statistics-* not created

[Gene Comer]

Recently did a fresh install with the newest version using the distributed deployment guide.  One server Wazuh cluster, one server Elasticsearch Cluster (with Kibana).

Seems to work as expected, but the wazuh-statistics indexes didn't get created, and were generating errors in the log.

Had previously done the all-in-one with the 4.0.x version, which did have those indexes.

On the current system, get _cat/templates?v gives me:

wazuh             [wazuh-alerts-4.x-*, wazuh-archives-4.x-*]
wazuh-alerts      [wazuh-alerts-4.x*]
wazuh-agent       [wazuh-monitoring-*]
security-auditlog [security-auditlog-*] 

of those, only wazuh-alerts.4.x-* and security-auditlog-* have been created.

Did I miss a step somewhere?

Gene

[Yana Zaeva]

Hi Gene,

Do you have any agents registered in this environment? Also, let me know if the <syscollector> module is enabled in the Wazuh manager's ossec.conf file. 

Lastly, send me the output of this command: ls -l /var/ossec/var/run/

Waiting for your reply,

Yana.

[Gene Comer]

Two agents currently, one a 4.0.4 which was setup with the original install and carried over, the second a 4.1.0 which was installed after this configuration.

For syscollector I have:

  <!-- System inventory -->
  <wodle name="syscollector">
    <disabled>no</disabled>
    <interval>1h</interval>
    <scan_on_start>yes</scan_on_start>
    <hardware>yes</hardware>
    <os>yes</os>
    <network>yes</network>
    <packages>yes</packages>
    <ports all="no">yes</ports>
    <processes>yes</processes>
  </wodle>

For the LS command...

 ls -l /var/ossec/var/run/
total 52
-rw-r----- 1 ossec  ossec    5 Feb 19 17:19 ossec-analysisd-1211.pid
-rw-r----- 1 ossec  ossec 2541 Feb 22 21:20 ossec-analysisd.state
-rw-r----- 1 root   ossec    5 Feb 19 17:19 ossec-authd-1139.pid
-rw-r----- 1 root   ossec    5 Feb 19 17:19 ossec-execd-1200.pid
-rw-r----- 1 root   ossec    5 Feb 19 17:19 ossec-logcollector-1357.pid
-rw-r----- 1 ossec  ossec    5 Feb 19 17:19 ossec-monitord-1390.pid
-rw-r----- 1 ossecr ossec    5 Feb 19 17:19 ossec-remoted-1324.pid
-rw-r----- 1 ossecr ossec    5 Feb 19 17:19 ossec-remoted-1325.pid
-rw-r--r-- 1 ossecr ossec  481 Feb 22 21:20 ossec-remoted.state
-rw-r----- 1 root   ossec    5 Feb 19 17:19 ossec-syscheckd-1308.pid
-rw-r----- 1 ossec  ossec    5 Feb 19 17:19 wazuh-apid-1100.pid
-rw-r----- 1 ossec  ossec    5 Feb 19 17:19 wazuh-db-1170.pid
-rw-r----- 1 root   ossec    5 Feb 19 17:19 wazuh-modulesd-1401.pid

Gene

[Yana Zaeva]

Hi Gene,

Thank you for the provided information. I asked you for this as I thought that maybe the syscollector module was disabled, and the necessary files to generate the wazuh-statistics-* index were not being created. These files are ossec-remoted.state and ossec-analysisd.state. Both of these files should be updated every 5 seconds. Please, check that this time is correct in the /var/ossec/etc/local_internal_options.conf, in the tags analysisd.state_interval and remoted.state_interval. 

Also, run the command GET _cat/indices, thus you can see all of the created indexes and search for the wazuh-statistics-* one. Check if when you go to the menu Discover and change the index if you are able to see the wazuh-statistics-* index too. Lastly, send me the erroneous logs that this was generating. 

Waiting for your reply,

Yana.

[Gene Comer]

local_internal_options.conf is empty except for some boilerplate comments at the top.  The two referenced .state files seem to be updating (time on the files is changing).

GET _cat/indices does not show anything wazuh-statistics-* at all.

Discover does have wazuh-statistics-* as an option in the dropdown, but just returns "No results match your search criteria"

The log is showing:

2021/02/23 09:45:00 ERROR Error searching or creating 'wazuh-statistics-2021.9w' due to 'Response Error'

2021/02/23 09:45:00 ERROR Response Error

2021/02/23 09:45:00 ERROR security_exception

[Yana Zaeva]

Hi Gene,

I will test this in order to see how this error happened. Please confirm that this is a new install (not an upgrade) of the newest version (4.1.0). You can check the version by running: cat/var/ossec/etc/ossec-init.conf. Also, correct me if I am wrong but you have one Wazuh manager on one server and one Elasticsearch node and Kibana on another. Let me know if you have more than one Wazuh manager or Elasticsearch nodes and how many. Also, confirm that these erroneous logs that you sent me above are from the /var/ossec/logs/ossec.log file, and if not, let me know from where they are. 

Lastly, please let me know if a pattern for this index is created. You can check it by going to the Kibana interface and selecting the option Stack management -> Index patterns -> Wazuh-statistics-*. 

Waiting for your reply,

Yana.

[Gene Comer]

This is a new install, started with a fresh version of Unbuntu.  Then followed the step by step directions on the website.

ossec-init.conf has:

DIRECTORY="/var/ossec"
NAME="Wazuh"
VERSION="v4.1.0"
REVISION="40106"
DATE="Tue Feb  2 07:27:17 UTC 2021"
TYPE="server"

There is one Wazuh manager on one server, and one elasticsearch node with Kibana on a second server.

The log entries I got from the UI (Wazuh->Settings->Logs), shows the log file located at /usr/share/kibana/data/wazuh/logs/wazuhapp.log

Stack Management->Index Patterns->wazuh-statistics-* does exist, 66 fields, no scripted, no filters.

Thanks for you help!

Gene

[Gene Comer]

What are we missing out on, with out having the statistics being stored?

[Сергей Парфёнов]

Hello, Team. any update on this issue?
I'm currently experiencing the same error in logs: 
Dec 1, 2021 @ 16:30:01 INFO Could not check if the index wazuh-statistics-2021.49w exists due to no permissions for create, delete or check Dec 1, 2021 @ 16:30:01 ERROR Could not check if the index wazuh-statistics-2021.49w exists due to no permissions for create, delete or check I have installation from scratch using provided ansible playbooks. Manager and kibana on the one node and Elastic (opendistro) cluster on another 3 nodes.
elasticsearch-oss 7.10.2 wazuh manager 4.2.5
opendistroforelasticsearch-kibana 1.13.2
Kibana Plugins:

opendistroAl...@1.13.0.0

opendistroAnomal...@1.13.0.0

opendistroGan...@1.13.0.0

opendistroIndex...@1.13.0.1

opendistroNo...@1.13.2.0

opendistroQuery...@1.13.0.0

opendistroR...@1.13.2.0

opendistroSe...@1.13.0.1

opendistroTrace...@1.13.2.0

wa...@4.2.4-4205-1

Stack Management->Index Patterns->wazuh-statistics-*  exists
GET _cat/indices shows nothing about statistics.

I have nothing in /var/ossec/etc/local_internal_options.conf on manager node. Should I put something into? 

четверг, 11 марта 2021 г. в 20:27:37 UTC+3, Gene Comer: