Wazuh cluster integrity sync error

[Minh Pham]

Hi team,

Our product cluster have 3 nodes: 1 master and 2 worker to serve around 2k agent, it's a basic installation from your document guide. But one week ago, we encountered a situation that our manager master node have been down frequently in a short time. After investigated a further more, we saw many entry in cluster.log of both master and worker node like this:

2022/11/10 07:54:35 ERROR: [Worker manager-w1] [Main] Error in synchronization process: Error 3039 - Timeout while waiting to receive a file: Integrity sync at manager-w1
2022/11/10 07:55:37 ERROR: [Worker manager-w1] [Main] Error sending files information: Error 3027 - Unknown received task name: dc347904-c309-470b-9d09-0237b51f9420

Then we found this error related to wazuh-clusterd integrity sync phase that master can't sync agent data to worker because some files is too large or something we don't know.
Could you please help me to fix this or provide me any information related to this situation?

Thanks and best regards.

[Jorge Eduardo Molas]

Hi, I hope you are doing well! Thanks for using Wazuh.
It seems your case is related to this issue, which will be released with the 4.4. Anyway, can you perform a workaround just restarting clusterd ? Then you can check the Integrity Sync logs doing ```tail -1000f /var/ossec/logs/cluster.log | grep 'Integrity sync' ```. 

Here are other issues related:

• Check if wazuh-clusterd process dies when there is no disk space left #14734
• Use less generic exceptions when receiving files in cluster #14735

I hope it's useful. Let me know. 

Regards!

[Minh Pham]

Hello Jorge,

Thank for your quick response, we have been restarted cluster several times, however clusterd.log is still showing nothing other than those generic error, so according to your reply, we just need to wait 4.4 release then update cluster to fix this? Could you please give me some information about its release date ?
If it isn't released soon, we must find a temporary way to make our cluster work normally, current cluster kinda lagging because only master node is working right now.
After searching, we found this issue, is its description true about how master and worker sync together? If it's true, assume that our cluster's error is caused by some reason mention in issue 11945, can we just copy all missing file between those node then restart cluster to make it work ?

Regards.

Vào lúc 20:45:41 UTC+7 ngày Thứ Năm, 10 tháng 11, 2022, jorge...@wazuh.com đã viết:

[Jorge Eduardo Molas]

Hi! So sorry for the delay.
We don't have an exact release date for 4.4, must probably it'll be released next year in Q1.
We can try another workaround. Can you show me the output of this command on the Wazuh manager? 

cat /var/ossec/framework/python/lib/python3.9/site-packages/wazuh-4.2.7-py3.9.egg/wazuh/core/cluster/cluster.json | grep -i  "timeout_receiving_file" (keep in mind that this path could change depending on your Wazuh version).

You can change this value in order to increase the time-out. To perform that, you must change the value on each worker and then restart the service in every worker .

Let me know if this workaround works. 

Regards!

[Minh Pham]

Hello Jorge, 

I tried it and after I changed that value our cluster is working normally right now.
Thank you so much!! 

Best regards!

Vào Th 2, 14 thg 11, 2022 vào lúc 18:54 'Jorge Eduardo Molas' via Wazuh mailing list <wa...@googlegroups.com> đã viết:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/BQPyhdf2L-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/72295aca-c551-41a4-95cf-40eaba9c69c9n%40googlegroups.com.