4.12->4.13.1 'archives' no longer being indexed (but 'alerts' are working fine)

[Alan Jackson]

Upgraded wazuh stack (separate manager, indexer, dash nodes, no cluster). 

Mostly everything appears to be working - alerts are entering the system fine & can be seen in the wazuh-alerts-* discovery view.

However, the wazuh-archives-* index hasn't shown any new documents since the upgrade.

wazuh/ossec is writing archives to a json file - and the archive events are definitely appearing in there.

Filebeat, when put into debug mode, showed pipelines being triggered for the archive events just fine.

The relevant archives pipeline exists in the opensearch side (hasn't changed with the upgrade).

When I run a _count query against the the most recent archives index, it is shown to be increasing - implying some data IS entering the index? But I don't know how to query this data - it doesn't return via searches of the usual @timestamp etc. I'm guessing the documents have strange data, but can't work out how to query the new documents to inspect them.

Alerts are working fine, so shard/watermark limits shouldn't apply. 

No related errors are in filebeat or wazuh-indexer logs.

Any help appreciated!

[hasitha.u...@wazuh.com]

Hi Alan,

First, we need to verify the archive logs written to archives.json file.
tail /var/ossec/logs/archives/archives.json

If the latest logs are not written to this file, then verify that the <logall_json> is set to yes in /var/ossec/etc/ossec.conf.
For example:

1. <ossec_config>
2.   <global>
3.     <jsonout_output>yes</jsonout_output>
4. __________
5.     <logall_json>yes</logall_json>
6. __________

<logall> enables or disables archiving of all log messages. When enabled, the Wazuh server stores the logs in a syslog format. The allowed values are yes and no.

<logall_json> enables or disables logging of events. When enabled, the Wazuh server stores the events in a JSON format. The allowed values are yes and no.

Note: However, only the <logall_json>yes</logall_json> option allows you to create an index that can be used to visualize the events on the Wazuh dashboard.

If you have configured the Wazuh manager for enabling logall_json then try restarting the manager.
systemctl restart wazuh-manager

Additionally, edit the Filebeat configuration file /etc/filebeat/filebeat.yml and change the value of archives: enabled from false to true if not:
For example:

1. archives:
2.  enabled: true

Restart Filebeat to apply the configuration changes:
systemctl restart filebeat

If the above steps have already been done and then check wazuh-archives-* index is missing in the dashboard.
Click the upper-left menu icon to open the main menu. Expand Dashboard management and navigate to Dashboards management > Index patterns -> and check wazuh-archives-* index is available in there.

If not, try creating the archives index manually again.
Click Create index pattern. Use wazuh-archives-* as the index pattern name, and set the timestamp in the Time field drop-down list.
Ref: https://documentation.wazuh.com/current/user-manual/manager/event-logging.html

Let me know the update on this.

[Alan Jackson]

Hello!

I can confirm all those settings were correct & as expected.

Interestingly, after HOURS of the 'count increasing but no documents being visible', apparently it all was back-filled. When I looked at the dashboard this morning, all the data is now there. Peculiar!

Regards