Agents duplicated on Wazuh manager

[Cyprien Chapelle]

Hello,

I added multiple agents on Wazuh manager (latest version), which was active till now.

This morning, I arrive on my Debian machine (11) and I see that everything has been duplicated:

root@wazuh-manager:/opt# /var/ossec/bin/agent_control -l

Wazuh agent_control. List of available agents:
   ID: 000, Name: wazuh-manager (server), IP: 127.0.0.1, Active/Local
   ID: 002, Name: machine1, IP: 192.168.1.1, Disconnected
   ID: 004, Name: machine2, IP: 192.168.1.2, Disconnected
   ID: 006, Name: machine3, IP: 192.168.1.3, Disconnected
   ID: 007, Name: machine1, IP: any, Active
   ID: 008, Name: machine2, IP: any, Active
   ID: 009, Name: machine3, IP: any, Active

​

Do you know what this is due to? This is the first time this has happened to me, yet I didn't do anything specific

[Kevin Ledesma]

Hello! how are you?

I configured the same setup that you have, tried to reproduce your error by restarting agents and manager, re-synchronizing the merged.mg file multiple times, turning off the machines,..  but I wasn't able to get to that scenario.

So, to figure out whats going on there, I will need you to give me more info like:

1. What is the version of the manager and the agents?
2. Do you have any warning or error in the logs of any component? (use the command grep "WARN\|ERR" /var/ossec/logs/ossec.log on every machine)
3. All the machines were up the whole time?

Thanks!

[Cyprien Chapelle]

Good morning I am fine and you ?

Ok, so here are the answers to the questions:

- 1. What is the version of the manager and the agents?

wazuh-manager/stable,now 4.3.8-1 amd64
wazuh-agent/stable 4.3.8-1 amd64

- 2. Do you have any warning or error in the logs of any component? (use the command grep "WARN\|ERR" /var/ossec/logs/ossec.log on every machine)

For agent machine2 (and same for every agent):

root@machine2:/etc#  grep "WARN\|ERR" /var/ossec/logs/ossec.log
2022/11/02 02:57:41 wazuh-agentd: WARNING: Server unavailable. Setting lock.
2022/11/02 04:28:32 wazuh-agentd: WARNING: Server unavailable. Setting lock.
2022/11/02 04:30:19 wazuh-syscheckd: WARNING: Process locked due to agent is offline. Waiting for connection...
2022/11/02 04:32:33 wazuh-logcollector: WARNING: Process locked due to agent is offline. Waiting for connection...
2022/11/02 05:08:21 wazuh-agentd: WARNING: Server unavailable. Setting lock.
2022/11/02 05:08:33 wazuh-logcollector: WARNING: Process locked due to agent is offline. Waiting for connection...
2022/11/02 05:12:38 wazuh-syscheckd: WARNING: Process locked due to agent is offline. Waiting for connection...
2022/11/02 05:15:43 wazuh-agentd: ERROR: Duplicate agent name: machine2 (from manager)
2022/11/02 05:15:43 wazuh-agentd: ERROR: Unable to add agent (from manager)
2022/11/02 05:15:53 wazuh-agentd: WARNING: (4101): Waiting for server reply (not started). Tried: '192.168.1.10'.
2022/11/02 05:15:53 wazuh-agentd: WARNING: Unable to connect to any server.
2022/11/02 05:49:14 wazuh-agentd: WARNING: Server unavailable. Setting lock.
2022/11/02 05:50:33 wazuh-logcollector: WARNING: Process locked due to agent is offline. Waiting for connection...
2022/11/02 05:50:48 wazuh-syscheckd: WARNING: Process locked due to agent is offline. Waiting for connection...

​

For Wazuh-manager :

root@wazuh-manager:/opt# grep "WARN\|ERR" /var/ossec/logs/ossec.log
2022/11/02 05:15:42 wazuh-authd: WARNING: Duplicate name 'machine1', rejecting enrollment. Agent '007' can't be replaced since it is not disconnected.
2022/11/02 05:15:43 wazuh-authd: WARNING: Duplicate name 'machine2', rejecting enrollment. Agent '008 can't be replaced since it is not disconnected.
2022/11/02 05:15:43 wazuh-authd: WARNING: Duplicate name 'machine3', rejecting enrollment. Agent '009' can't be replaced since it is not disconnected.

​

- 3. All the machines were up the whole time?

Yes.

Unfortunately, I don't have the logs when the problem occurred, these are the last logs I have access to.

Thank you for your help !

[Cyprien Chapelle]

Ok so I tried to delete agents with any IP (so ID 007 to 009) and they came back !

Log file of Wazuh-Manager give :

2022/11/02 09:45:54 wazuh-remoted: WARNING: (1408): Invalid ID 032 for the source ip: '192.168.1.2' (name 'unknown').
2022/11/02 09:45:54 wazuh-remoted: WARNING: (1408): Invalid ID 032 for the source ip: '192.168.1.2' (name 'unknown').
2022/11/02 09:45:54 wazuh-remoted: WARNING: (1408): Invalid ID 026 for the source ip: '192.168.1.3' (name 'unknown').

​

[Cyprien Chapelle]

I regenerated an API key for the agents (whose IP is filled in), then I turned off the wazuh-agent service on each agent and then added the API key on each agent.
It worked, but how did the problem arise? Keys expired?

[Cyprien Chapelle]

Ok, this morning I saw that the problem had returned. At 6 a.m, the agents disconnected. Except they are still running, still sending data to the Manager, so duplicates are created. I can't figure out why this is happening.

[Cyprien Chapelle]

If I look in client.keys file, I see all different keys corresponding to agents automatically add (with no IP : "any").

So, I have try to delete the value "any" to add the IP address which corresponds. After restart the Manager, the modified agent is overwritten and a new one (with the same name as the modified one) appears with the value "any" and a different key! I do not know what to do ...

[Kevin Ledesma]

Hello! sorry for the delay! I was testing the options that we could use for this case.

So, I have three options to fix it (or at least to try to), but first you should stop all your agents (running systemctl stop wazuh-agent on each machine) and remove them from the manager (remove-agents).

The options are

1. Configure use_source_ip tag on the wazuh-manager's auth block (inside the file /var/ossec/etc/ossec.conf) (use-source-ip). It would force the agent to authenticate with its IP
   
2. Configure in the enrollment block the agent_address tag to register with the agent's IP (inside agent's config /var/ossec/etc/ossec.conf ) (enrollment-agent-adderss). The enrollment block is used to specify how the agent will enroll to the manager, so if you specify the agent_address it will always enroll using its IP.
   
3. Use the agent-auth tool specifying the agent IP with the -I flag (it could be found in the agents machine /var/ossec/bin/agent-auth). Example: agent-auth -m <MANAGER_IP> -I <AGENT_IP> -A <AGENT_NAME>. The only issue with this method is that if the agent gets reconnected it will do it without specifying the IP (it will use the enrollment or auth configuration).

Thanks for your patience! Let me know the results