Empty IT Hygiene Section

[john]

Dear Wazuh Team

I am having the issue with IT Hygiene where not data is being displayed. Could you help me troubleshoot this issue and find out what causes it?

Thank you in advance

Best,

John

[Nicolai Romero]

Hi John,

I hope you are doing fine! The IT Hygiene dashboard is populated from the data collected by the syscollector module on the Wazuh agent side. The first step to diagnose the issue is to validate that the syscollector module is enabled on the agent (configuration) and the vulnerability-detector module is enabled on the manager (configuration). If both modules are enabled, then the issue could come from the agent->server->indexer->dashboard connection.

I would recommend you the troubleshooting guide if the problem persists. I will be attentive if the proposed solutions don't work to further diagnose the issue. 

Best regards!
Nicolai R.

[john]

Dear Nicolai

Thanks for your response.

I tried tweaked the agent settings on server side and restart the wazuh-manager to integrate new component into agents. Vulnerability detection is also present and all seems to work fine.

However the page still happens to be empty.

Could you provide any further assistance?

Thank you a lot

Best,

John

[Nicolai Romero]

Hi John,

Sure thing. To further confirm if the information is being sent correctly to the Indexer and accessed by the Dashboard I recommend you perform the following checks:

1.  Check the logs on the agent side to see if the syscollector module is working:
            grep -i 'syscollector' /var/ossec/logs/ossec.log
2. Check that filebeat is correctly set and that it is sending the info to the indexer:
            sudo systemctl status filebeat
3. Check that the indexer is enabled and receiving data, and see if it has indexes:
             Wazuh indexer indices - Wazuh indexer · Wazuh documentation
             curl -k https://127.0.0.1:9200/_cat/indices/wazuh-inventory*?v
 4. Check that the cluster name that you are using in your query is correct:
            API reference · Wazuh documentation
     curl -s -u <USER>:<PASSWORD> https://<WAZUH_MANAGER>:55000/cluster/status?pretty

If the problem persists, please share the result of this commands and I would have more information to help you diagnose the issue.

[john]

Hey,

Sure here are results of commands

root@wazuh:~# grep -i 'syscollector' /var/ossec/logs/ossec.log
2025/10/30 00:32:42 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 00:32:54 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 01:32:55 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 01:33:06 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 02:33:08 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 02:33:19 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 03:33:21 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 03:33:32 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 04:33:34 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 04:33:45 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 05:33:47 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 05:33:59 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 06:34:00 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 06:34:14 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 07:34:16 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 07:34:27 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 08:34:29 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 08:34:41 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 09:34:43 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 09:34:55 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 10:34:56 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 10:35:09 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 11:35:10 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 11:35:23 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 12:35:24 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 12:35:38 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/10/30 13:35:39 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/10/30 13:35:52 wazuh-modulesd:syscollector: INFO: Evaluation finished.
grep: /var/ossec/logs/ossec.log: binary file matches

root@wazuh:~# sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-10-09 14:34:50 MSK; 2 weeks 6 days ago
       Docs: https://www.elastic.co/products/beats/filebeat
   Main PID: 1113619 (filebeat)
      Tasks: 12 (limit: 38367)
     Memory: 98.8M (peak: 1.4G swap: 3.4M swap peak: 6.4M)
        CPU: 7h 27min 56.334s
     CGroup: /system.slice/filebeat.service
             └─1113619 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --pat>
Notice: journal has been rotated since unit was started, output may be incomplete.

root@wazuh:~# curl -k https://127.0.0.1:9200/_cat/indices/wazuh-inventory*?v
root@wazuh:~# curl -s -u REDACTED:REDACTED https://REDACTED:55000/cluster/status?pretty
root@wazuh:~# 

[Nicolai Romero]

Hi John,

Great, thanks for sharing. I see that you didn't add an output for the last two curl commands. Did you add the full information or does that mean that they didn't return a value? If so, it could mean that the issue is most likely on Filebeat or the Indexer. Could you please check that you followed all the steps described in the documentation for the Filebeat configuration and Indexer integration:

-  Installing the Wazuh server step by step - Wazuh server

In particular, you can check Filebeat connection using the command: 

filebeat test output

If it passes, we can check that the information is sent to the manager from the agent by checking the local manager DB. Depending on your installation and the agent id, you could do something like this to query that DB:

  sqlite3 /var/ossec/queue/db/<AGENT_ID>.db  
      SELECT scan_time, hostname, version FROM sys_osinfo;  

If the replication is working, you should see inventory data from the agent OS. Please check again the last curl commands after validating the installation steps and send me the result. I will be glad to assist you. 

Best regards,
- Nicolai R.

[john]

Hello, 

The wazuh didn't return anything.
alright will reach out as soon as have a progress

Best,
John

[Nicolai Romero]

Hi John,

Sure thing, I will be attentive to your response! 

[john]

Hey Nicolai,

Here are the command results

root@wazuh:~# filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.10.2


root@wazuh:~# sqlite3 /var/ossec/queue/db/420.db
SQLite version 3.45.1 2024-01-30 16:01:20
Enter ".help" for usage hints.
sqlite> SELECT scan_time, hostname, version FROM sys_osinfo;
2025/11/05 06:04:22|{REDACTED MACHINE NAME}|

this command returned only the machine name with 420 id, nothing else.

I checked the curl commands from the Wazuh doc and it seems that all files that curl command downloads are present. Please advise further.

Thank you and wish you the best,
John

[john]

Hello Nicolai

Any updates?

Thank you,
John

[Nicolai Romero]

Hi John,

With the last results you sent, it is shown that the manager is getting data correctly from the agent, and that filebeat can connect to the indexer. I believe that the reason you are not getting IT Hygiene information is most likely due to an issue with the manager. My recommendation is the following:

- Check for errors on the `ossec.log` file on MANAGER side. Last time you just shared the logs on agent side and we just validated that it is working as expected. Connect to the manager machine and inspect the log files for the manager, filebeat and indexer, and see if any of them report any errors or warnings:

grep -Ei 'error|warning' /var/ossec/logs/ossec.log
grep -Ei 'error|warning' /var/ossec/logs/filebeat/filebeat-plain.log  
grep -Ei 'error|warning' /var/log/wazuh-indexer/wazuh-indexer.log

- You can also use directly either the Indexer API with the dashboard DevTools (to rule out curl certificate errors) and further explore the indexer status. For instance, you can check the following commands:

1. Check the cluster name to be used on the IT hygiene filter:
GET _cluster/state/cluster_name?pretty

2. Check what indexes are available:
GET _cat/indices/wazuh*?v

3. Check if the syscollector data (wazuh-inventory* indexes) has been indexed:
GET _cat/indices/wazuh-inventory*?v

4. Check if the index has syscollector data:
GET wazuh-inventory-*/_search
{
  "query": {
    "term": {
      "agent.id": "420"
    }
  },
  "size": 5
}


If you find problems doing any of the steps, then maybe one step in the configuration was done incorrectly. Please check the Indexer and Manager installation guides to look for missing steps and additional validation actions. I hope this helps having a better clarity of the process and I will attentive to further support you with this issue. 

Best regards,
Nicolai R.

[john]

Dear Nicolai,

Here please take a look. Note that I have intentionally shortened some of the outputs, for your simplicity and better understanding

grep -Ei 'error|warning' /var/ossec/logs/ossec.log

2025/11/11 00:00:32 wazuh-remoted: WARNING: Unexpected message (hex): '3c34-----REDACTED-----6e0a'
2025/11/11 00:00:32 wazuh-remoted: WARNING: Too big message size from socket [187].
2025/11/11 00:00:33 wazuh-remoted: WARNING: Unexpected message (hex): '3c34363e3120323032352d31312d31315430303a30303a33332e3434373734322b30333a303020707831207379735f6c6f676765722d57617a75685379736c6f67203133343030204d6963726f736f6674636c6f7564736572766963656d6f646966696564205b536563757264656e4c6f67403332343733204163746976697479547970653d224d6963726f736f667420636c6f75642073657276696365206d6f6469666965642220506572666f726d656442793d2253797374656d2220506572666f726d65644f7665723d224e2f412220506572666f726d656446726f6d3d225343444e2d454543412d4757312220526561736f6e3d22536563757264656e2050414d2041646d696e202d205465616d732053746f7261676520636c6f75642073657276696365206d6f6469666965642e20436c6f75642053746f726167652073657276696365287329206173736f6369617465642e20446973636f766572792c20417a7572652053746f7261676520436f6e7461696e6572732c204d61696c20776974682044656c656761746564205065726d697373696f6e2c204d61696c2077697468204170706c69636174696f6e205065726d697373696f6e207365727669636528732920756e2d6173736f6369617465642e225d20536563757264656e0a'
2025/11/11 00:00:33 wazuh-remoted: WARNING: Too big message size from socket [188].
2025/11/11 00:00:33 wazuh-remoted: WARNING: Unexpected message (hex): '3c3133343e3120323032352d31312d31305432313a30303a33305a206872732d696e7465726e6174696f6e612d39647773316c767920436865636b506f696e74203235343936202d205b616374696f6e3a2244656372797074223b-----REDACTED-----  536202b20534841323536223b20'
2025/11/11 00:00:33 wazuh-remoted: WARNING: Too big message size from socket [188].
2025/11/11 00:00:33 wazuh-remoted: WARNING: Unexpected message (hex): '344439384139307d3b6d676d743d4d616e6167656d656e745f536572766963653b646174653d313736313234303236313b706f6c6963795f6e616d653d43502d47572d4d534b2d546573745c5d223b206473743a2235322e35342e3230392e3435223b206d6573736167655f696e666f3a22416464726573732073706f6f66696e67223b2070726f647563743a2256504e2d312026204669726557616c6c2d31223b2070726f746f3a2236223b20735f706f72743a223631303732223b20736572766963653a22343433223b207372633a223139322e3136382e3233302e323534225d0a'
2025/11/11 00:00:33 wazuh-remoted: WARNING: Too big message size from socket [188].
2025/11/11 00:00:34 wazuh-remoted: WARNING: Unexpected message (hex): '3c34363e3120323032352d31312d31315430303a30303a33342e3435313639372b30333a303020707831207379735f6c6f676765722d57617a75685379736c6f67203133343030204d6963726f736f6674636c6f7564736572766963656d6f646966696564205b536563757264656e4c6f674033323437332041637469 -----REDACTED-----  656e0a'
2025/11/11 00:00:34 wazuh-remoted: WARNING: Too big message size from socket [188].
2025/11/11 00:00:34 wazuh-remoted: WARNING: Unexpected message (hex): '3c3133343e3120323032352d31 -----REDACTED-----  6557616c6c2d315b64625f7461673d7b32453434443244302d353038'
2025/11/11 00:00:34 wazuh-remoted: WARNING: Too big message size from socket [196].
2025/11/11 00:00:35 wazuh-remoted: WARNING: Unexpected message (hex): '3c34363e3120323032352d31312d3131543030 -----REDACTED-----  20536563757264656e0a'
2025/11/11 00:00:35 wazuh-remoted: WARNING: Too big message size from socket [196].
2025/11/11 00:00:35 wazuh-remoted: WARNING: Unexpected message (hex): '3c3133343e3120323032352d31312d3 -----REDACTED-----  386463382d306363322d346437392d38'
2025/11/11 00:00:35 wazuh-remoted: WARNING: Too big message size from socket [196].
2025/11/11 00:00:36 wazuh-remoted: WARNING: Unexpected message (hex): '3c34363e3120323 -----REDACTED-----  465642e225d20536563757264656e0a'
2025/11/11 00:00:36 wazuh-remoted: WARNING: Too big message size from socket [196].
2025/11/11 00:00:36 wazuh-remoted: WARNING: Unexpected message (hex): '3c34363e3120323032 -----REDACTED-----  4656e0a'
2025/11/11 00:00:36 wazuh-remoted: WARNING: Too big message size from socket [196].
2025/11/11 00:00:37 wazuh-remoted: WARNING: Unexpected message (hex): '3c34363e3120323032352d31312d31315430303a30303a33372e3631383937332b30333a303020707831207379735f6c6f676765722d57617a75685379736c6f67203133343030204d6963726f736f6674636c6f7564736572766963656d6f646966696564205b536563757264656e4c6f67403332343733204163746976697479547970653d224d6963726f736f667420636c6f75642073657276696365206d6f6469666965642220506572666f726d656442793d2253797374656d2220506572666f726d65644f7665723d224e2f412220506572666f726d656446726f6d3d225343444e2d454d45412d4757312220526561736f6e3d22536563757264656e2050414d2041646d696e202d205465616d732053746f7261676520636c6f75642073657276696365206d6f6469666965642e20436c6f75642053746f726167652073657276696365287329206173736f6369617465642e20446973636f766572792c20417a7572652053746f7261676520436f6e7461696e6572732c204d61696c20776974682044656c656761746564205065726d697373696f6e2c204d61696c2077697468204170706c69636174696f6e205065726d697373696f6e207365727669636528732920756e2d6173736f6369617465642e225d20536563757264656e0a'
2025/11/11 00:00:37 wazuh-remoted: WARNING: Too big message size from socket [198].
2025/11/11 00:00:37 wazuh-remoted: WARNING: Unexpected message (hex): '3c3133343e3120323032352d31312d31305432313a30303a33355a206872732d696e7465726e6174696f6e612d39647773316c767920436865636b506f696e74203235343936202d205b616374696f6e3a2244726f70223b20666c6167733a22343030363434223b2069666469723a22696e626f756e64223b2069666e616d653a22626f6e643130223b206c6f6769643a2230223b206c6f677569643a227b307836393132353237352c3078302c307833343030343036342c307833316530363339387d223b206f726967696e3a223130302e3130302e33312e3634223b206f726967696e7369636e616d653a22434e3d43502d47572d4d534b2c4f3d4d616e6167656d656e745f536572766963652e2e396b71713837223b2073657175656e63656e756d3a2235223b2074696d653a2231373632383038343335223b2076657273696f6e3a2235223b205f5f706f6c6963795f69645f7461673a2270726f647563743d56504e2d312026204669726557616c6c2d315b64625f7461673d7b32453434443244302d353038342d323434462d383136362d3846333435344439384139307d3b6d676d743d4d616e6167656d656e745f536572766963653b646174653d313736313234303236313b706f6c6963795f6e616d653d43502d47572d4d534b2d546573745c5d223b206473743a223235352e3235352e3235352e323535223b206473745f64796e6f626a5f6e616d653a224d795f56504e5f446f6d61696e223b206473745f6f626a6563745f747970653a2264796e616d69635f6f626a656374223b20696e7a6f6e653a22496e7465726e616c223b206c617965725f6e616d653a224e6574776f726b223b206c617965725f757569643a2264363037356462322d623138612d343862612d393435312d353862323533346639636565223b206d617463685f69643a223137223b20706172656e745f72756c653a2230223b2072756c655f616374696f6e3a2244726f70223b2072756c655f6e616d653a22436c65616e75702072756c65223b2072756c655f7569643a2231653261393830632d653464632d343463372d386464302d393162393632626333613164223b206f75747a6f6e653a224c6f63616c223b2070726f647563743a2256504e2d312026204669726557616c6c2d31223b2070726f746f3a223137223b20735f706f72743a223537353239223b20736572766963653a2231393437223b207372633a223137322e31372e39392e3132225d0a3c3133343e3120323032352d31312d31305432313a30303a33355a206872732d696e7465726e6174696f6e612d39647773316c767920436865636b506f696e74203235343936202d205b616374696f6e3a22446574656374223b20666c6167733a22333933323136223b206966646972'
2025/11/11 00:00:37 wazuh-remoted: WARNING: Too big message size from socket [201].

2025/11/11 00:00:39 wazuh-remoted: WARNING: Too big message size from socket [205].



In case of filebeat and Wazuh Indexer logs, that path doesn't even exist
root@wazuh:~# grep -Ei 'error|warning' /var/ossec/logs/filebeat/filebeat-plain.log  
grep: /var/ossec/logs/filebeat/filebeat-plain.log: No such file or directory


root@wazuh:~# grep -Ei 'error|warning' /var/log/wazuh-indexer/wazuh-indexer.log
grep: /var/log/wazuh-indexer/wazuh-indexer.log: No such file or directory


And regarding Dev Tools command all of them returned 500 error, please check the image I attached

Thank you for your help

Best,
aex

[Nicolai Romero]

Hi John,

Sorry, I sent you an incorrect path for the indexer and filebeat logs (at least for version 4.14.0, please confirm what version you are using to reproduce the exact scenario). For what I can see in your logs and the image, the problem is related to the Indexer which is not accepting API calls neither from the dashboard DevTools nor using curl. We need to debug further and see if there is any error executing the indexer.

grep -Ei 'error|warning' /var/log/wazuh-indexer/wazuh-cluster.log

You should be able to see files on the  /var/log/wazuh-indexer/ directory. If not, then we have a problem with the installation, possibly related to permissions. On the other hand, filebeat logs should be in this location:

grep -Ei 'error|warning'  /var/log/filebeat/filebeat

Again, you should be able to see files on the  /var/log/filebeat/ directory, but this could change for versions other than 4.14.0. 

Besides that, you can also check if both services are running ok, or if something failed at initialization. You can run the following commands to check that:

systemctl status wazuh-indexer
journalctl -u wazuh-indexer -n 100 --no-pager

systemctl status filebeat
journalctl -u filebeat -n 100 --no-pager

Please proceed with the latter and I will be attentive to help you investigate further. 

[john]

Hi Nicolai,

Sure, here are the command outputs. Its important to mention that I am currently using the latest Wazuh version v4.14.0

root@wazuh:~# grep -Ei 'error|warning' /var/log/wazuh-indexer/wazuh-cluster.log
[2024-12-07T03:46:58,379][ERROR][o.o.a.a.AlertIndices     ] [node-1] info deleteOldIndices
[2024-12-07T03:46:58,380][ERROR][o.o.a.a.AlertIndices     ] [node-1] info deleteOldIndices
[2024-12-07T08:33:06,929][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1024m, -Xmx1024m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2024-12-07T08:33:38,034][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

root@wazuh:~# grep -Ei 'error|warning'  /var/log/filebeat/filebeat

root@wazuh:~# ls -l /var/log/filebeat/
total 70180
-rw-r--r-- 1 root root      660 Nov  5 09:16 filebeat
-rw-r----- 1 root root  8947334 Nov 12 00:05 filebeat.1
-rw-r----- 1 root root 10482686 Oct 22 02:59 filebeat.2
-rw-r----- 1 root root 10482409 Oct 22 02:57 filebeat.3
-rw-r----- 1 root root 10480816 Oct 22 02:56 filebeat.4
-rw-r----- 1 root root 10484723 Oct 22 02:55 filebeat.5
-rw-r----- 1 root root 10484964 Oct 22 02:54 filebeat.6
-rw-r----- 1 root root 10485544 Oct 22 02:53 filebeat.7

root@wazuh:~# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-10-27 11:48:03 MSK; 2 weeks 2 days ago
       Docs: https://documentation.wazuh.com
   Main PID: 4162074 (java)
      Tasks: 228 (limit: 38367)
     Memory: 9.1G (peak: 28.3G swap: 1.7G swap peak: 1.8G)
        CPU: 2d 6h 10min 11.642s
     CGroup: /system.slice/wazuh-indexer.service
             └─4162074 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+Alw>

Notice: journal has been rotated since unit was started, output may be incomplete.

root@wazuh:~# journalctl -u wazuh-indexer -n 100 --no-pager
-- No entries --

root@wazuh:~# journalctl -u wazuh-indexer -n 100 --no-pager
-- No entries --
root@wazuh:~# systemctl status filebeat

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: enabled)

     Active: active (running) since Thu 2025-10-09 14:34:50 MSK; 1 month 3 days ago

       Docs: https://www.elastic.co/products/beats/filebeat
   Main PID: 1113619 (filebeat)
      Tasks: 12 (limit: 38367)

     Memory: 119.4M (peak: 1.4G swap: 3.5M swap peak: 6.4M)
        CPU: 11h 39min 54.027s

     CGroup: /system.slice/filebeat.service
             └─1113619 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --pat>
Notice: journal has been rotated since unit was started, output may be incomplete.

root@wazuh:~# journalctl -u filebeat -n 100 --no-pager
-- No entries --

Hope this helps to resolve issue.
Best,
John

[Nicolai Romero]

Hi John,

Ok, the services are running, and the logs don't show any relevant information related to the current issue (the wazuh-cluster.log logs are from last year). Now that we confirmed that they are both running, we have to continue diagnosing the Indexer status. In particular, after reproducing the issue on my local machine I found out that the 500 Internal Server Error you had last time was most likely because you executed the commands on the Server management/Dev Tools instead of the Indexer management/Dev Tools, which are targeting different endpoints, sorry about that.

It is worth trying again with curl, using the -k parameter to allow connections without certificate checks. You should do something like this:

curl -k -u <indexer_user>:<indexer_pwd> https://127.0.0.1:9200/_cluster/state/cluster_name?pretty

Could you please confirm if you get the same result running the commands shared above on the Indexer Dev Tools and with -k option? I apologize if this is taking more time than expected to diagnose, I will continue to be attentive to support you and hopefully we can get useful information from the Indexer API. 

Best regards,
Nicolai R.

[john]

Dear Nicolai,

when I run this command I get nothing in return. I don't even see if the creds are correct, which they are. Even if i deliberately enter wrong password, I get no result

root@wazuh:~# curl -k -u  <indexer_user>:<indexer_pwd>   https://127.0.0.1:9200/_cluster/state/cluster_name?pretty
root@wazuh:~# 

Hope that will give you some clues. Thank you for your help

Best,
John

[Nicolai Romero]

Hi John,

I have a couple of follow-up questions:

1. Did you recently upgrade your Wazuh version? If so, could you please confirm that you have installed the same version for the Manager, Indexer and Dashboard components? And could you please tell me if the IT Hygiene section ever showed info on your dashboard?
Please check the troubleshooting guide for upgrade-related issues.

2. Did you try to do the same requests with the Indexer Management/DevTools? We need to confirm not only that the Indexer is working but also that the Dashboard can access this API.
Please check the Indexer API documentation and try to execute the commands listed here. If they fail too, you could have a faulty indexer installation that would require a re-installation/configuration.

I am putting again the commands that you should execute to validate the status of the Indexer

            1. Check the cluster name to be used on the IT hygiene filter:
              GET _cluster/state/cluster_name?pretty

              2. Check what indexes are available:
              GET _cat/indices/wazuh*?v

              3. Check if the syscollector data (wazuh-inventory* indexes) has been indexed:
              GET _cat/indices/wazuh-inventory*?v

              4. Check if the index has syscollector data:
              GET wazuh-inventory-*/_search
              {
                "query": {
                  "term": {
                    "agent.id": "420"
                  }
                },
                "size": 5
              }

3. Could you please check if the Indexer is running on port 9200? You can run the following command on the Indexer machine to check what process is listening on that port.
    sudo ss -tulpen | grep 9200

Please start with the troubleshooting guides and follow the steps described to validate that the indexer and dashboard are working. Right now we need to confirm the status of the connection to the Indexer API which is critical for the IT Hygiene to show data.

I will be attentive to your response. Best regards,
- Nicolai R.

[john]

Hello Nicolai,

Appreciate your reach outs
Let me answer your questions in sequence:

1. I remember upgrading wazuh to latest version which is 4.14.1. Here's the proof:
root@wazuh:~# grep "upgrade" /var/log/dpkg.log | grep wazuh
2025-11-17 09:23:53 upgrade wazuh-indexer:amd64 4.14.0-1 4.14.1-1
2025-11-17 09:25:45 upgrade wazuh-dashboard:amd64 4.14.0-1 4.14.1-1
2025-11-17 09:29:07 upgrade wazuh-manager:amd64 4.14.0-1 4.14.1-1

the thing is that I never noticed the IT hygiene to show any data

2. Regarding the commands you provided, here are their responses
 GET _cluster/state/cluster_name?pretty
{
  "cluster_name": "wazuh-indexer-cluster",
  "cluster_uuid": "REDACTED"
}

GET _cat/indices/wazuh*?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   wazuh-statistics-2025.3w    GgDS6bzFTEyv5jKKorrsUQ   1   0       4040            0        2mb            2mb
green  open   wazuh-alerts-4.x-2025.09.10 73dJg7yRRMitMVSytYk0qQ   3   0    5475414            0      1.8gb          1.8gb
green  open   wazuh-monitoring-2025.43w   Ozn0QGrWSSin6SJD21brHw   1   0     631925            0     90.1mb         90.1mb
green  open   wazuh-monitoring-2024.51w   IG59gmYCQ0S-MOXfzS0CsA   1   0     410817            0     86.8mb         86.8mb
green  open   wazuh-monitoring-2025.31w   2Y_IvIxrTYalORE9tU7z3w   1   0     651322            0    143.7mb        143.7mb
green  open   wazuh-alerts-4.x-2025.04.06 Q_VapG5vRFyUwpTiXwW1ng   3   0     168622            0     54.6mb         54.6mb
......................
green  open   wazuh-alerts-4.x-2025.09.08 TZFM6p38QIyhy1pxtNqVlw   3   0    5936604            0      5.9gb          5.9gb
green  open   wazuh-statistics-2024.49w   jXRaaSiZQUikNCt5y0maHw   1   0        812            0    720.8kb        720.8kb
green  open   wazuh-alerts-4.x-2025.09.05 RDwDmY7aT7SBR4ZA4CAbAg   3   0    5362262            0      5.4gb          5.4gb
green  open   wazuh-alerts-4.x-2025.09.06 -bibp_trR92LPa3ZHIvi3g   3   0    3229488            0   1009.2mb       1009.2mb

GET _cat/indices/wazuh-inventory*?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

GET wazuh-inventory-*/_search
{
 "query": {
   "term": {
       "agent.id": "420"
       }
  },
     "size": 5
}

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0,
    "hits": []
  }
}

3. indexer is running 

root@wazuh:~#     sudo ss -tulpen | grep 9200
tcp   LISTEN 0      4096                 [::ffff:127.0.0.1]:9200             *:*    users:(("java",pid=1150,fd=624)) uid:999 ino:29475 sk:1003 cgroup:/system.slice/wazuh-indexer.service v6only:0 <->  

Hope this information will help you 

I will review the troubleshooting documentation and write to you if I have some updates

Thank you and wish you the best,
John

[Nicolai Romero]

Hi John!

Thanks for your response. With these recent results we can see that the indexer is working properly and accessible from the dashboard, but no inventory indexes are present.  Please let's continue the diagnose with the following steps:

1. In 4.14.0 the inventory indexes are following the format wazuh-states-inventory-<module>, as seen in the documentation. Please run this command on the Indexer Management/DevTools section to confirm that they are not present:
    GET _cat/indices/wazuh-states-inventory*?v

2. The inventory data is sent from the manager using the Inventory Harvester module, which could be failing on your installation. Please enable debug logs and share the result of running these commands:

   First edit the file /var/ossec/etc/local_internal_options.conf and add the following line:
        wazuh_modules.debug=2

    Then restart the manager.
        sudo systemctl restart wazuh-manager

    Finally, after a couple of minutes, check the logs and please share the result:
         grep -iE "inventory-harvester|inventory_harvester|indexer-connector|syscollector" /var/ossec/logs/ossec.log | tail -n 100

     Note: once we finish with the debugging, you should change the debug config back to 0.   

3. For now, I think you may try updating the stored indexer credentials in the wazuh-keystore, as described in this documentation. Please run the following commands with your current indexer username and password:

       echo '<WAZUH_INDEXER_USERNAME>' | /var/ossec/bin/wazuh-keystore -f indexer -k username
   echo '<WAZUH_INDEXER_PASSWORD>' | /var/ossec/bin/wazuh-keystore -f indexer -k password

    Then restart the manager and check if the dashboard shows any difference.
        sudo systemctl restart wazuh-manager


I will be attentive to hear the result of these steps and to continue helping identifying the underlying issue. Best regards!
- Nicolai R.

[john]

Dear Nicolai,

Here are the responses:

GET _cat/indices/wazuh-states-inventory*?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

After altering the debugger file I got this in response:
root@wazuh:~# grep -iE "inventory-harvester|inventory_harvester|indexer-connector|syscollector" /var/ossec/logs/ossec.log | tail -n 100
2025/11/24 08:58:26 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 2 seconds.

2025/11/24 08:58:26 indexer-connector[1205202] indexerConnector.cpp:1251 at operator()(): WARNING: IndexerConnector initialization failed for index 'wazuh-states-inventory-packages-wazuh.domain.com', retrying until the connection is successful.

2025/11/24 08:58:27 indexer-connector[1205202] indexerConnector.cpp:912 at operator()(): DEBUG: Waiting for initialization thread to process events.
2025/11/24 08:58:28 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 4 seconds.
2025/11/24 08:58:32 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 8 seconds.
2025/11/24 08:58:36 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-system-wazuh.domain.com': No available server. Retrying in 2 seconds.
2025/11/24 08:58:36 indexer-connector[1205202] indexerConnector.cpp:1251 at operator()(): WARNING: IndexerConnector initialization failed for index 'wazuh-states-inventory-system-wazuh.domain.com', retrying until the connection is successful.
2025/11/24 08:58:36 indexer-connector[1205202] indexerConnector.cpp:912 at operator()(): DEBUG: Waiting for initialization thread to process events.
2025/11/24 08:58:38 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-system-wazuh.domain.com': No available server. Retrying in 4 seconds.
2025/11/24 08:58:40 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 16 seconds.
------ SHORTENED ------
2025/11/24 09:22:38 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-system-wazuh.domain.com': No available server. Retrying in 60 seconds.

2025/11/24 09:23:28 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 60 seconds.
2025/11/24 09:23:38 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-system-wazuh.domain.com': No available server. Retrying in 60 seconds.
2025/11/24 09:24:14 wazuh-analysisd: WARNING: Syscollector decoder queue is full.
2025/11/24 09:24:28 indexer-connector[1205202] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 60 seconds.
------ SHORTENED ------
2025/11/24 10:58:28 indexer-connector[1221405] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-packages-wazuh.domain.com': No available server. Retrying in 60 seconds.
2025/11/24 10:58:43 indexer-connector[1221405] indexerConnector.cpp:1243 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-inventory-system-wazuh.domain.com': No available server. Retrying in 60 seconds.

echo '<WAZUH_INDEXER_USERNAME>' | /var/ossec/bin/wazuh-keystore -f indexer -k username
echo '<WAZUH_INDEXER_PASSWORD>' | /var/ossec/bin/wazuh-keystore -f indexer -k password

These both are empty and return nothing

Best,
Alex

[Nicolai Romero]

Hi John,

These logs confirm that the issue is a problem with the Manager->Indexer connection. The error Unable to initialize IndexerConnector for index <INDEX>: No available server. means that this connection is not correctly working. We have to check a couple of thinks first.

- Troubleshooting - Upgrade guide · Wazuh documentation

1. I checked the configuration you initially shared. According to it, the Indexer is installed locally and listening port 9200, and the filebeat certificates are in /etc/filebeat/certs/. Please confirm that this is correct or modify the ossec.conf file accordingly. You can check that the keys exist and that they have the expected name and permissions running this command:

ls -l /etc/filebeat/certs
# Expected result:
total 12
-r-------- 1 root root filebeat-key.pem
-r-------- 1 root root filebeat.pem
-r-------- 1 root root root-ca.pem


2. Depending on the output, you should be able to run curl commands using this certificate paths. For instance, you could do something as this, replacing each value with the path to each certificate file, and with the Indexer admin user and password.

curl --cacert <ROOT_CA> --cert <CERTIFICATE_PEM> --key <CERTIFICATE_KEY_PEM> -u <USER>:<PASS> -XGET  https://indexer:9200/_cluster/health

3. Please run again these commands replacing the <WAZUH_INDEXER_USERNAME> and <WAZUH_INDEXER_PASSWORD> with your Indexer credentials (the user and password used to connect to the Dashboard).

echo '<WAZUH_INDEXER_USERNAME>' | /var/ossec/bin/wazuh-keystore -f indexer -k username
echo '<WAZUH_INDEXER_PASSWORD>' | /var/ossec/bin/wazuh-keystore -f indexer -k password

Please run them again to update this values and then restart the manager:

systemctl restart wazuh-manager

4. Once you updated this credentials and restarted the manager, please check again the manager logs.

grep -iE "inventory-harvester|inventory_harvester|indexer-connector|syscollector" /var/ossec/logs/ossec.log | tail -n 100

If the credentials, ip, port and certificates are all correct, you should see no logs for IndexerConnector and the information should be displayed in the dashboard. 


Please try the steps described here and refer to the documentation shared. I hope this actions help you find a solution and I will be attentive to further assist you.

Best regards,
- Nicolai R.

[john]

Hello

1.  you're absolutely correct. here the indexer configuration in ossec.conf file:
  <indexer>
    <enabled>yes</enabled>
    <hosts>
      <host>https://127.0.0.1:9200</host>
    </hosts>
    <ssl>
      <certificate_authorities>
        <ca>/etc/filebeat/certs/root-ca.pem</ca>
      </certificate_authorities>
      <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
      <key>/etc/filebeat/certs/filebeat-key.pem</key>
    </ssl>
  </indexer>

the filebeat keys are used there:
ls -l /etc/filebeat/certs
total 20
-r-------- 1 root root 1204 Sep 27  2024 root-ca.pem
-r-------- 1 root root 1704 Sep 27  2024 wazuh-1-key.pem
-r-------- 1 root root 1277 Sep 27  2024 wazuh-1.pem
-r-------- 1 root root 1704 Dec  3  2024 wazuh-server-key.pem
-r-------- 1 root root 1285 Dec  3  2024 wazuh-server.pem

what I noticed is that the key files specified in ossec.conf do not exist in /etc/filebeat/certs. But after changing config to this :

  <indexer>
    <enabled>yes</enabled>
    <hosts>
      <host>https://127.0.0.1:9200</host>
    </hosts>
    <ssl>
      <certificate_authorities>
        <ca>/etc/filebeat/certs/root-ca.pem</ca>
      </certificate_authorities>
      <certificate>/etc/filebeat/certs/wazuh-server.pem</certificate>
      <key>/etc/filebeat/certs/wazuh-server-key.pem</key>
    </ssl>
  </indexer>

and restarting, i got this on IT hygiene dashboard

No results match your search criteria

and here are logs

2025/11/25 15:25:37 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-inventory-packages-wazuh.hrsinternational.com', retrying until the connection is successful.
2025/11/25 15:25:48 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-inventory-system-wazuh.hrsinternational.com', retrying until the connection is successful.

3. I am also using SSO for Wazuh Login and not fully aware of password. should i still do it?

Best,

John

[Nicolai Romero]

Hi John,

Great! This is likely to be the root cause of the issue. The certificates that you have to use have to match the ones used by Filebeat to communicate with the Indexer. 

1. You mentioned in a previous message that the command filebeat test output was working as expected. Let's verify first that this is still the case. 
        filebeat test output

2. If it works then we should check the filebeat configuration and use the same certificates in the ossec.conf file. Please check the /etc/filebeat/filebeat.yml file and look for something like this:

  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/wazuh-1.pem"
  ssl.key: "/etc/filebeat/certs/wazuh-1-key.pem"

3. Update the ossec.conf with the certificates used here. Consider this mapping:

     Filebeat configuration                                |            Wazuh manager configuration (ossec.conf)
   ssl.certificate_authorities ->    <certificate_authorities><ca>  </ca></certificate_authorities>
   ssl.certificate             ->    <certificate>  </certificate>
   ssl.key                     ->    <key>  </key>


4. First, you can try if it has the default admin:admin credentials and test to see if the problem is fixed. Then, you can change the indexer credentials following this guide:
      Password management - User administration · Wazuh documentation

     Either way, you would have to update the values stored in the Wazuh keystore, replacing the <WAZUH_INDEXER_USERNAME> and <WAZUH_INDEXER_PASSWORD> with the new credentials.

    echo '<WAZUH_INDEXER_USERNAME>' | /var/ossec/bin/wazuh-keystore -f indexer -k username
    echo '<WAZUH_INDEXER_PASSWORD>' | /var/ossec/bin/wazuh-keystore -f indexer -k password

Please try the steps described here and refer to the documentation shared. I hope this actions help you find a solution and I will be attentive to further assist you.

Best regards,
- Nicolai R.

[john]

Dear Nicolai,

Thank you for your help.

Everything works now

Kind Regards,

John

[Nicolai Romero]

Hi John! 

Excellent! I am glad to hear that you solved the issue. Please feel free to ask if you have any additional issue. Have an excellent rest of the week.

Best regards,
-  Nicolai R.