Agents not synchronized

579 views
Skip to first unread message

frances...@gmail.com

unread,
Jun 30, 2021, 11:07:57 AM6/30/21
to Wazuh mailing list
Hello, I casually discovered that many agents (but not all of them) are not synchronized. They are all active and apparently are working.I tried restart both manager and agent but nothing changed. In log I can't find anything useful.
I'm using Wazuh 3.13.
Any ideas?
Thank you.

Matias Pereyra

unread,
Jun 30, 2021, 5:50:43 PM6/30/21
to Wazuh mailing list
Hi! Thanks for using Wazuh.

It'd be useful to know:
  • Have these agents something in common (OS, group, network, etc.)?
  • Where do you see the lack of synchronization (inventory, centralized configuration, etc.)?
  • Is this a cluster or a single manager installation? 
  • Is there any particular reason for using V3.13? The last stable version is V4.1.5
Also, if you could upload the ossec.log file from the manager and one agent, we could analyze them in detail.

Regards. 

frances...@gmail.com

unread,
Jul 1, 2021, 4:20:38 AM7/1/21
to Wazuh mailing list
Hi, 
  • apparently nothing in common
  • I see with command "/var/ossec/bin/agent_groups -S -i " and wazuh app too
  • single manager
  • No, only a matter of time!
manager's ossec.log i s full of messages like these:

WARNING: (1213): Message from '192.168.153.251' not allowed. Cannot find the ID of the agent. Source agent ID is unknown.

In non synchronized agents I see these:

ossec-agentd: WARNING: Unknown message received. No action defined.

maybe other software is sending packets on the same ports of wazuh and is disturbing it, I'll check it.

frances...@gmail.com

unread,
Jul 1, 2021, 7:03:50 AM7/1/21
to Wazuh mailing list
We stopped the server which was disturbing, but agents are still not synchronized.
Here is the manager log after restart:

2021/07/01 12:53:09 ossec-integratord: INFO: Started (pid: 29758).
2021/07/01 12:53:09 ossec-integratord: INFO: Enabling integration for: 'custom-script'.
2021/07/01 12:53:09 ossec-integratord: INFO: Enabling integration for: 'virustotal'.
2021/07/01 12:53:09 ossec-agentlessd: INFO: Not configured. Exiting.
2021/07/01 12:53:09 ossec-authd: INFO: Started (pid: 29787).
2021/07/01 12:53:09 ossec-authd: INFO: Accepting connections on port 1515. No password required.
2021/07/01 12:53:09 ossec-authd: INFO: Setting network timeout to 1.000000 sec.
2021/07/01 12:53:10 wazuh-db: INFO: Started (pid: 29804).
2021/07/01 12:53:11 ossec-execd: INFO: Started (pid: 29829).
2021/07/01 12:53:12 ossec-maild: INFO: Started (pid: 29841).
2021/07/01 12:53:12 ossec-maild: INFO: Getting alerts in log format.
2021/07/01 12:53:12 ossec-analysisd: INFO: Total rules enabled: '3809'
2021/07/01 12:53:12 ossec-analysisd: INFO: Started (pid: 29848).
2021/07/01 12:53:13 ossec-syscheckd: WARNING: The check_unixaudit option is deprecated in favor of the SCA module.
2021/07/01 12:53:13 ossec-syscheckd: INFO: Started (pid: 29865).
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6003): Monitoring directory/file: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | realtime'.
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6003): Monitoring directory/file: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | realtime'.
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6003): Monitoring directory/file: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | realtime'.
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/krb5.keytab'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/sys/kernel/security'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/sys/kernel/debug'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key'
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6016): Directory set for real time monitoring: '/etc'.
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6016): Directory set for real time monitoring: '/usr/bin'.
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6016): Directory set for real time monitoring: '/usr/sbin'.
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6000): Starting daemon...
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 10800 seconds
2021/07/01 12:53:13 ossec-syscheckd: INFO: (6008): File integrity monitoring scan started.
2021/07/01 12:53:13 rootcheck: INFO: Starting rootcheck scan.
2021/07/01 12:53:14 ossec-remoted: INFO: Started (pid: 29928). Listening on port 1514/UDP (secure).
2021/07/01 12:53:14 ossec-remoted: INFO: (4111): Maximum number of agents allowed: '100000'.
2021/07/01 12:53:14 ossec-remoted: INFO: (1410): Reading authentication keys file.
2021/07/01 12:53:15 ossec-logcollector: WARNING: (1958): Log file '/var/ossec/logs/active-responses.log' is duplicated.
2021/07/01 12:53:15 ossec-logcollector: INFO: Monitoring output of command(360): df -P
2021/07/01 12:53:15 ossec-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d
2021/07/01 12:53:15 ossec-logcollector: INFO: Monitoring full output of command(360): last -n 20
2021/07/01 12:53:15 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/audit/audit.log'.
2021/07/01 12:53:15 ossec-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'.
2021/07/01 12:53:15 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/messages'.
2021/07/01 12:53:15 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/secure'.
2021/07/01 12:53:15 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/maillog'.
2021/07/01 12:53:15 ossec-logcollector: INFO: Started (pid: 29964).
2021/07/01 12:53:16 ossec-monitord: INFO: Started (pid: 30081).
2021/07/01 12:53:17 wazuh-modulesd: INFO: Process started.
2021/07/01 12:53:17 wazuh-modulesd:download: INFO: Module started.
2021/07/01 12:53:17 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2021/07/01 12:53:17 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2021/07/01 12:53:17 wazuh-modulesd:database: INFO: Module started.
2021/07/01 12:53:17 wazuh-modulesd:oscap: INFO: Module disabled. Exiting...
2021/07/01 12:53:17 wazuh-modulesd:control: INFO: Starting control thread.
2021/07/01 12:53:17 wazuh-modulesd:syscollector: INFO: Module started.
2021/07/01 12:53:18 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/07/01 12:53:19 ossec-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2021/07/01 12:53:19 ossec-syscheckd: INFO: (6012): Real-time file integrity monitoring started.
2021/07/01 12:53:24 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/07/01 12:54:37 rootcheck: INFO: Ending rootcheck scan.

Here is agent log:

2021/07/01 12:53:33 ossec-execd: INFO: Started (pid: 18411).
2021/07/01 12:53:35 ossec-agentd: INFO: (1410): Reading authentication keys file.
2021/07/01 12:53:35 ossec-agentd: INFO: Using notify time: 10 and max time to reconnect: 60
2021/07/01 12:53:35 ossec-agentd: INFO: Version detected -> Linux |cloudlinux6 |3.10.0-957.12.2.el7.x86_64 |#1 SMP Tue May 14 21:24:32 UTC 2019 |x86_64 [CentOS Linux|centos: 7.6] - Wazuh v3.13.2
2021/07/01 12:53:35 ossec-agentd: INFO: Started (pid: 18424).
2021/07/01 12:53:35 ossec-agentd: INFO: Server IP Address: 192.168.153.107
2021/07/01 12:53:35 ossec-agentd: INFO: Using AES as encryption method.
2021/07/01 12:53:35 ossec-agentd: INFO: Trying to connect to server (192.168.153.107:1514/udp).
2021/07/01 12:53:35 ossec-agentd: INFO: (4102): Connected to the server (192.168.153.107:1514/udp).
2021/07/01 12:53:36 ossec-syscheckd: INFO: Started (pid: 18439).
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6003): Monitoring directory/file: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | follow_symbolic_links | whodata'.
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6003): Monitoring directory/file: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | whodata'.
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6003): Monitoring directory: '/usr/bin' (/bin), with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | follow_symbolic_links | whodata'.
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6003): Monitoring directory: '/usr/sbin' (/sbin), with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | follow_symbolic_links | whodata'.
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/sys/kernel/security'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/sys/kernel/debug'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/dev/core'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/krb5.keytab'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/prelink.cache'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/aliases.db'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/lvm/cache'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/openvpn/ipp.txt'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/boot/grub2/grubenv'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/selinux/targeted/tmp'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6207): Ignore 'file' sregex '^/proc'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6207): Ignore 'file' sregex '.css$|temp-write-test'
2021/07/01 12:53:36 ossec-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key'
2021/07/01 12:53:37 ossec-logcollector: INFO: Monitoring output of command(360): df -P
2021/07/01 12:53:37 ossec-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d
2021/07/01 12:53:37 ossec-logcollector: INFO: Monitoring full output of command(360): last -n 20
2021/07/01 12:53:37 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/audit/audit.log'.
2021/07/01 12:53:37 ossec-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'.
2021/07/01 12:53:37 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/messages'.
2021/07/01 12:53:37 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/secure'.
2021/07/01 12:53:37 ossec-logcollector: INFO: (1950): Analyzing file: '/var/log/maillog'.
2021/07/01 12:53:37 ossec-logcollector: INFO: Started (pid: 18453).
2021/07/01 12:53:37 wazuh-modulesd: INFO: Process started.
2021/07/01 12:53:37 wazuh-modulesd:oscap: INFO: Module disabled. Exiting...
2021/07/01 12:53:37 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2021/07/01 12:53:37 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2021/07/01 12:53:37 sca: INFO: Module started.
2021/07/01 12:53:37 wazuh-modulesd:control: INFO: Starting control thread.
2021/07/01 12:53:37 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_rhel7_linux.yml'
2021/07/01 12:53:37 sca: INFO: Starting Security Configuration Assessment scan.
2021/07/01 12:53:37 wazuh-modulesd:syscollector: INFO: Module started.
2021/07/01 12:53:37 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_rhel7_linux.yml'
2021/07/01 12:53:38 ossec-agentd: WARNING: Unknown message received. No action defined.
2021/07/01 12:53:38 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2021/07/01 12:53:38 ossec-agentd: WARNING: Unknown message received. No action defined.
2021/07/01 12:53:40 ossec-syscheckd: INFO: (6000): Starting daemon...
2021/07/01 12:53:40 rootcheck: INFO: Starting rootcheck scan.
2021/07/01 12:53:40 ossec-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 14400 seconds
2021/07/01 12:53:40 ossec-syscheckd: INFO: (6008): File integrity monitoring scan started.
2021/07/01 12:53:44 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_rhel7_linux.yml'
2021/07/01 12:53:44 sca: INFO: Security Configuration Assessment scan finished. Duration: 0 seconds.
2021/07/01 12:53:45 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2021/07/01 12:53:45 ossec-agentd: WARNING: Unknown message received. No action defined.
2021/07/01 12:56:03 rootcheck: INFO: Ending rootcheck scan.

Matias Pereyra

unread,
Jul 1, 2021, 2:32:31 PM7/1/21
to Wazuh mailing list
Hello again, thank you very much for the information provided.

I think that this warning message means there is some communication problem between the Manager and the agents, that could lead to a synchronization failure


    WARNING: (1213): Message from '192.168.153.251' not allowed. Cannot find the ID of the agent. Source agent ID is      unknown.

It could happen, for example, that the agents were registered with a fixed IP and not using any. So, if the agent changes its IP, the manager won't be able to read the agent's messages. Please, check this information in your client.keys file and register again the corresponding agents.

Also, you are using UDP but the configuration as default in Wazuh 4.0 is TCP. This connection is more reliable, and you could give it a try to discard any connection issue.

Finally, are we sure these agents are working as expected? Have you checked their last-keepalive frequent update? In Wazuh 3.13, they won't be considered disconnected until 30 minutes have passed since the last message received. 

Regards.

frances...@gmail.com

unread,
Jul 2, 2021, 6:15:12 AM7/2/21
to Wazuh mailing list
Yes, it was a cloned vm, but now we deleted it and these messages are disappeared.
Changing UDP to TCP in every agent is quite expensive at the moment, we have about 200 agents!
Agents not synchronized are working as expected, they send events and last keep alive is now.

frances...@gmail.com

unread,
Jul 2, 2021, 6:32:27 AM7/2/21
to Wazuh mailing list
I checked traffic with tcpdump, on the agent there is traffic on udp port 1514, from manager to agent and from agent to manager.
So, why doesn't it synchronize and can't it get centralized configuration?

Matias Pereyra

unread,
Jul 2, 2021, 11:35:44 AM7/2/21
to Wazuh mailing list
Hi!

If the messages in the manager have disappeared, we can focus on the warning messages in the agent

    ossec-agentd: WARNING: Unknown message received. No action defined.

If the agent hasn't a version greater than the manager, then the only reason I can think for these unknown messages is a packet corruption in your environment. You can find this error message in the source code: receive_msg().
Consider that the shared configuration consists of sending the merged file in various packets, and the UDP protocol isn't reliable. So the keepalive may be arriving, but it could be difficult for a long sequence of messages.

My strong suggestion is to use TCP, Wazuh uses it in the last version. Many issues related to centralized configuration were fixed after this change.
I understand it can be a difficult task, considering that you have to do it manually and the manager doesn't support both protocols at the same time yet. Consider an orchestration tool or a script for this purpose.

I apologize for the inconvenience of protocol change, this difficulty will be solved by the team in future releases.

Regards.

frances...@gmail.com

unread,
Jul 5, 2021, 4:42:04 AM7/5/21
to Wazuh mailing list
Maybe I found the cause of the problem: we have another SIEM, clients send UDP packets to SIEM manager on port 1514, perhaps they conflict with UDP packets from wazuh manager to clients.
I'll try to change port and protocol type to TCP on every agent.
Thank you.

Matias Pereyra

unread,
Jul 5, 2021, 9:53:34 AM7/5/21
to Wazuh mailing list
Hi again Francesco.

I hope that solves these issues. 
But write us again anytime if you still have problems.

Regards. 

Reply all
Reply to author
Forward
0 new messages