Delay in Logs and log lose

[Wazuh Server]

Dear Team, 
We are encountering delay in logs showing in GUI, and sometimes logs lose as well.

Deployment >> 4.3, Wazuh central component in single server. I have attached the below error /Warn messages to the file.

cat /var/log/wazuh-indexer# cat wazuh-indexer-cluster.log | grep -i -E "error|warn"

cat /var/log/filebeat/filebeat | grep -i -E "error|warn"

cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

journalctl -u wazuh-dashboard 

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

Requesting you to please check and update what is causing the issue.

[Wazuh Server]

Hello Team,

Can someone look into this! 

[Mauro Agustín Malara]

Hi! I hope everything is doing great.

Could you please give me further context?

What do you mean by "delay in logs showing in GUI"?

Have you detected any pattern when those logs are lost (a message in a log or something similar)?

Could you detail the specific version of Wazuh and the environment you are using (Docker, VM, etc.)?

Finally, the logs of the Wazuh Dashboard are cropped, could you share with me the output of `journalctl -xn --no-pager`?

I need more information to help you in a better way, so if you provide deep info about this would be helpful.

Regards.

[Wazuh Server]

Hi Mauro,

Please find the requested details in attachment. 

What do you mean by "delay in logs showing in GUI"?

Ans: We are observing delays in logs during the business hours (9-6pm) weekdays and even log loss as well. When it comes to non-business days logs are reported on-time without any delay. 

Have you detected any pattern when those logs are lost (a message in a log or something similar)?

Ans: Yes, After logs are stopped shipping, Next day morning 5:30 AM IST, Again we started receiving the live logs( Here we are losing the delayed/cached logs of before day)

Could you detail the specific version of Wazuh and the environment you are using (Docker, VM, etc.)?

 App version: 4.3.10
App revision: 4311 

Finally, the logs of the Wazuh Dashboard are cropped, could you share with me the output of `journalctl -xn --no-pager`?

Could you run this command  netstat -nputa?
You could also check indexer status with systemctl status wazuh-indexer.

Ans: Attached in txt file.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c6c2e357-4b28-44da-9cc8-03d101c50663n%40googlegroups.com.

[Mauro Agustín Malara]

Hi! Thank you for the information.

I made a mistake while writing the journalctl command, could you run it again as follows: journalctl -u wazuh-dashboard --no-pager | grep -iE 'warn|error' ?

Sorry for the inconvenience,

Regards.

[Wazuh Server]

Here you go, Sharing only a few samples here. Please let me know if you require more visibility.

Feb 24 15:17:05 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:47:05Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82442], primary term [35]. current document has seqNo [82443] and primary term [35]"}
Feb 24 15:20:52 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:50:52Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82452], primary term [35]. current document has seqNo [82453] and primary term [35]"}
Feb 24 15:20:52 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:50:52Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82452], primary term [35]. current document has seqNo [82453] and primary term [35]"}
Feb 24 15:24:40 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:40Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[RequestAbortedError]: Request aborted"}
Feb 24 15:24:40 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:40Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82462], primary term [35]. current document has seqNo [82463] and primary term [35]"}
Feb 24 15:24:40 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:40Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82462], primary term [35]. current document has seqNo [82463] and primary term [35]"}
Feb 24 15:24:41 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:41Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82464], primary term [35]. current document has seqNo [82465] and primary term [35]"}
Feb 24 15:24:41 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:41Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82464], primary term [35]. current document has seqNo [82465] and primary term [35]"}
Feb 24 15:24:41 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:41Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82464], primary term [35]. current document has seqNo [82465] and primary term [35]"}
Feb 24 15:24:41 SOC-Infra opensearch-dashboards[2755102]: {"type":"log","@timestamp":"2023-02-24T09:54:41Z","tags":["error","opensearch","data"],"pid":2755102,"message":"[version_conflict_engine_exception]: [search-telemetry:search-telemetry]: version conflict, required seqNo [82464], primary term [35]. current document has seqNo [82465] and primary term [35]"}

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/1b723c00-46cf-4427-b1a5-fe10aa0c5757n%40googlegroups.com.

[Wazuh Server]

Hi Fabricio,

Please find the requested logs attached to this file.

On Fri, Feb 24, 2023 at 6:24 PM 'Fabricio Brunetti' via Wazuh mailing list <wa...@googlegroups.com> wrote:

Hello, hope you are doing well.
From the error and warnings in the  logs it looks like there is something wrong with wazuh-indexer.

Could you run this command  netstat -nputa?
You could also check indexer status with systemctl status wazuh-indexer.

Regards,
Fabricio

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c263d181-8ce9-497a-84b4-41dfe3e3d9cen%40googlegroups.com.

[Mauro Agustín Malara]

Ok, thank you for sending me the information.

While checking the logs and your answers I saw that both the Wazuh Dashboard and Filebeat have problems while connecting with Elasticsearch. So, I think that this could be a performance issue related to it.

Could you specify your server's memory and CPU?

Could you send me the output of the following commands?

• curl -u user:password -s -k -X GET https://IP:9200/_cat/shards | grep UNASSIGNED
• curl -u user:password -s -k -X GET https://IP:9200/_cluster/health?pretty
• curl -u user:password -s -k -X GET https://10.10.10.100:9200/_cluster/allocation/explain

Regarding the Wazuh Dashboard logs, there is no need to give them any type of relevance for now (reference).

Regards.

[Wazuh Server]

Hi Mauro,

Please find the requested details below

Could you specify your server's memory and CPU?

32 GB RAM and 8Core CPU >> 4.3 Deployment, wherein indexer, manager, dashboard everything in single mentioned server.

Could you send me the output of the following commands?

• curl -u user:password -s -k -X GET https://IP:9200/_cat/shards | grep UNASSIGNED
• curl -u user:password -s -k -X GET https://IP:9200/_cluster/health?pretty
• curl -u user:password -s -k -X GET https://10.10.10.100:9200/_cluster/allocation/explain

Attached all the logs in the file.

[Mauro Agustín Malara]

Hi! Sorry for the delay in responding,

I will need the following configuration of the Wazuh Indexer and Wazuh Dashboard (remember to not share sensitive information):

- /etc/wazuh-indexer/opensearch.yml

- /etc/wazuh-dashboard/opensearch_dashboards.yml

Maybe some events are being lost because of a conflict in a field (for instance: sometimes a field comes as an integer and the next time as a string so that document will be rejected). So, could you verify that no conflicts are displayed in Stack Management/Index Patterns -> wazuh-alerts-*?

Also, could you check if the Wazuh Manager is dropping events by running the following commands?

- grep -E "usage|dropped" /var/ossec/var/run/wazuh-analysisd.state

- cat /var/ossec/var/run/wazuh-remoted.state

Regards.

[Wazuh Server]

Hi Mauro,

Please find the requested details attached. 

/etc/wazuh-indexer/opensearch.yml

- /etc/wazuh-dashboard/opensearch_dashboards.yml

Attached the config in Txt file 

Maybe some events are being lost because of a conflict in a field (for instance: sometimes a field comes as an integer and the next time as a string so that document will be rejected). So, could you verify that no conflicts are displayed in Stack Management/Index Patterns -> wazuh-alerts-*?

 Yes, Everything is normal and Wazuh-alerts is default 

- grep -E "usage|dropped" /var/ossec/var/run/wazuh-analysisd.state

- cat /var/ossec/var/run/wazuh-remoted.state

Attached the output for your reference

[Wazuh Server]

HI Mauro,

Any update on below request !

[Mauro Agustín Malara]

Hi! Sorry for the late reply,

I'll check this with the team and I'll keep you posted.

Regards.

[Mauro Agustín Malara]

Hi! Sorry for taking so long to reply,So, as a summary we could say that:

• Manager is working okey, it does not show any error.
  
• No events are being dropped by Analysisd and no messages are being discarded by Remoted.
  
• Filebeat shows errors saying that it could not publish events and it could not connect to Opensearch because it refuses the connection.
  
• Wazuh App shows errors saying that the connection was refused by Opensearch and some internal errors while trying to execute requests to the Wazuh API.
  
• The status of the Wazuh Indexer cluster is red, which means that at least one primary shard and its replicas are not allocated.
  

-----Finally, the "explain allocation" endpoint shows an error because Opensearch is trying to allocate a replica in the same node (this is not allowed because the replicas must be allocated in a different node than its primary shard is allocated).

• So, it does not make sense to have replicas enabled in a All-in-One installation (1 node). Disable the replicas by setting the number_of_replicas to 0 in the index:
• curl -X PUT "http://localhost:9200/wazuh-alerts-\*/_settings?pretty" -H 'Content-Type: application/json' -d'

{ "settings" : { "number_of_replicas" : 0 } }'

This "solution" is dangerous because replicas prevent data loss, so a better solution is to configure a Wazuh-Indexer cluster. This also prevents having a large quantity of primary shards in one data node, because is recommended to have ~25 shards per GiB of JVM heap memory.

Now, we should check the main reason that is causing the alerts to be lost or delayed in the UI. So, please run the following commands and share with me the output:

• df -h: we should check the disk usage
• curl -u user:pass -k -X GET https://IP:9200/_cat/shards/wazuh-alerts-* | wc -l: we should count the number of shards for the wazuh-alerts-* indices.
• curl -u user:pass -k -X GET https://IP:9200/_cluster/settings?flat_settings=true: we should check some settings.
• egrep -A2 "\-Xms" /etc/wazuh-indexer/jvm.options: we should check the heap memory.

Finally, I need you to monitor the /var/ossec/var/run/wazuh-remoted.state file during business hours and check if is there any discarded message (do the same with /var/ossec/var/run/wazuh-analysisd.state checking if there is any dropped event).

Regards!

[Wazuh Server]

Hi Mauro,

Sorry for the delay in my response. Please find the requested logs for your reference

root@SOC-Infra:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       3.8T  2.4T  1.5T  63% /
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G  5.8M   16G   1% /dev/shm
tmpfs           3.2G  1.8M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/loop1       92M   92M     0 100% /snap/lxd/24061
/dev/loop0       64M   64M     0 100% /snap/core20/1822
/dev/sda15      105M  5.2M  100M   5% /boot/efi
/dev/loop3       50M   50M     0 100% /snap/snapd/17950
/dev/loop4       50M   50M     0 100% /snap/snapd/18357
/dev/loop2       64M   64M     0 100% /snap/core20/1828
tmpfs           3.2G  8.0K  3.2G   1% /run/user/118
tmpfs           3.2G  4.0K  3.2G   1% /run/user/1004
tmpfs           3.2G   16K  3.2G   1% /run/user/1001
tmpfs           3.2G   12K  3.2G   1% /run/user/1003
tmpfs           3.2G  4.0K  3.2G   1% /run/user/1005
tmpfs           3.2G  4.0K  3.2G   1% /run/user/1002
tmpfs           3.2G  4.0K  3.2G   1% /run/user/1000
root@SOC-Infra:~#

root@SOC-Infra:~# curl -u user:pass -k -X GET https://10.X.X.X:9200/_cat/shards/wazuh-alerts-* | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    12  100    12    0     0   1090      0 --:--:-- --:--:-- --:--:--  1090


root@SOC-Infra:~# curl -u user:pass -k -X GET https://10.X.X.X:9200/_cluster/settings?flat_settings=true^[[D
curl: (3) bad range in URL position 65:
https://10.X.X.X:9200/_cluster/settings?flat_settings=true
                                                                ^
root@SOC-Infra:~# egrep -A2 "\-Xms" /etc/wazuh-indexer/jvm.options
## -Xms16g
## -Xmx16g
##
--
-Xms16g
-Xmx16g

/var/ossec/var/run/wazuh-remoted.state

/var/ossec/var/run/wazuh-analysisd.state

Attached complete logs in txt file.

Note: I haven't performed any changes as you mentioned above, Requesting you to please check the provided logs and helps us with best possible solution in this case.

[Wazuh Server]

PFA

[Mauro Agustín Malara]

Hi!

I cannot see the attached files from the last messages, could you attach them again?

I made a mistake while writing the commands, please run them as follows:

• curl -u user:pass -s -k -X GET "https://IP:9200/_cat/shards/wazuh-alerts-*" | wc -l
  

• curl -u user:pass -k -X GET "https://IP:9200/_cluster/settings?flat_settings=true"
  

and, share with me the output.

Regards.

[Wazuh Server]

FYI

root@SOC-Infra:~# curl -u user:pass -s -k -X GET "https://10.176.16.34:9200/_cat/shards/wazuh-alerts-*" | wc -l
0
root@SOC-Infra:~# curl -u user:pass -k -X GET "https://10.176.16.34:9200/_cluster/settings?flat_settings=true"
Unauthorizedroot@SOC-Infra:~#

[Mauro Malara]

Hi!

Remember to replace the "user:pass" with your ES credentials before executing the commands. That's why we are getting that response from the API. Please, run the commands again and send me the output.

On the other hand, I'm not being able to see any attachments. Please, answer and send the attachments via Gmail.

Regards.

You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/xj_-cleKLGs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/47790e1b-85ce-4aa2-9f08-2ad3dac467aan%40googlegroups.com.

--

Mauro Malara QA Engineer +542613678879 wazuh.com mauro....@wazuh.com mauromalara

[Mauro Malara]

Hi, sorry for answering so late.

Let me check it and get back to you.

Regards.

On Thu, Mar 30, 2023 at 10:25 AM Wazuh Server <wazuh...@gmail.com> wrote:

Hi Mauro,

Please find the below requested details and attachements.

 root@SOC-Infra:~# curl -u admin:xxxxxxxxxxxxxxx -s -k -X GET "https://X.X.X.X:9200/_cat/shards/wazuh-alerts-*" | wc -l
276

root@SOC-Infra:~# curl -u admin:xxxxxxxxxxxxxxx -k -X GET "https://X.X.X.X:9200/_cluster/settings?flat_settings=true"
{"persistent":{},"transient":{}}root@SOC-Infra:~# 

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/50182d3f-e57b-495d-b513-b3f12ce5af1bn%40googlegroups.com.

--

Mauro Malara QA Engineer wazuh.com

[Wazuh Server]

Hi Mauro,

Did you get a chance to look at the issue.

[Wazuh Server]

Hi Mauro,

Awaiting for your response