I am setting up wazuh distributed architecture and using Nginx load balancer.
I have customized ports.
Agent is getting the key and then again it is going for enrollment and it is getting rejected and the agent always shows as never connected in the dashboard.
I have enabled debug logs.
Please find logs below and advise what could be the problem and how to fix that.
2024/04/30 04:11:26 wazuh-remoted[291006] keys.c:448 at OS_UpdateKeys(): DEBUG: move_netdata
2024/04/30 04:11:26 wazuh-remoted[291006] keys.c:454 at OS_UpdateKeys(): DEBUG: Key reloading completed
2024/04/30 04:11:26 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:11:26 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:26 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:11:31 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:36 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:11:36 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:36 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:11:36 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:11:41 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:46 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:11:46 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:46 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:11:46 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:11:51 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:56 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:11:56 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:11:56 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:11:56 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:12:01 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:06 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:12:06 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:06 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:12:06 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:12:11 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:708 at run_dispatcher(): INFO: New connection from 192.168.10.101
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:751 at run_dispatcher(): DEBUG: Request received: <OSSEC PASS: XXXXXXXX OSSEC A:'XX-XXXXXXXXX' V:'v4.7.3' G:'Macbook' K:'64770086548f49f07952b46417fb370b2e8cd04f'
>
2024/04/30 04:12:15 wazuh-authd[290551] auth.c:108 at w_auth_parse_data(): INFO: Received request for a new agent (XX-XXXXXXXXXX) from: 192.168.10.101
2024/04/30 04:12:15 wazuh-authd[290551] auth.c:175 at w_auth_parse_data(): DEBUG: Group(s) is: Macbook
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:781 at run_dispatcher(): INFO: Agent key generated for 'XX-XXXXXXXXXX' (requested by any)
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:958 at run_writer(): DEBUG: Dumping changes into disk.
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:980 at run_writer(): DEBUG: [Writer] OS_WriteKeys(): 1905 µs.
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:990 at run_writer(): DEBUG: [Writer] OS_WriteTimestamps(): 239 µs.
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:998 at run_writer(): DEBUG: [Writer] Performing insert([003] XX-XXXXXXXXX).
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:1005 at run_writer(): DEBUG: [Writer] wdb_insert_agent(): 697 µs.
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:1020 at run_writer(): DEBUG: [Writer] wdb_set_agent_groups_csv(): 5856 µs.
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:1076 at run_writer(): DEBUG: [Writer] Inserted agents: 1
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:1077 at run_writer(): DEBUG: [Writer] Removed agents: 0
2024/04/30 04:12:15 wazuh-authd[290551] main-server.c:1078 at run_writer(): DEBUG: [Writer] Loop: 9 ms.
2024/04/30 04:12:16 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:12:16 wazuh-remoted[291006] sendmsg.c:50 at check_keyupdate(): INFO: (1409): Authentication file changed. Updating.
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:432 at OS_UpdateKeys(): DEBUG: Reloading keys
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:434 at OS_UpdateKeys(): DEBUG: OS_DupKeys
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:437 at OS_UpdateKeys(): DEBUG: Freekeys
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:441 at OS_UpdateKeys(): DEBUG: OS_ReadKeys
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:442 at OS_UpdateKeys(): INFO: (1410): Reading authentication keys file.
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:445 at OS_UpdateKeys(): DEBUG: OS_StartCounter
2024/04/30 04:12:16 wazuh-remoted[291006] msgs.c:83 at OS_StartCounter(): DEBUG: OS_StartCounter: keysize: 1
2024/04/30 04:12:16 wazuh-remoted[291006] msgs.c:109 at OS_StartCounter(): DEBUG: No previous sender counter.
2024/04/30 04:12:16 wazuh-remoted[291006] msgs.c:120 at OS_StartCounter(): DEBUG: Assigning sender counter: 0:0
2024/04/30 04:12:16 wazuh-remoted[291006] msgs.c:140 at OS_StartCounter(): DEBUG: Stored counter.
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:448 at OS_UpdateKeys(): DEBUG: move_netdata
2024/04/30 04:12:16 wazuh-remoted[291006] keys.c:454 at OS_UpdateKeys(): DEBUG: Key reloading completed
2024/04/30 04:12:16 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:16 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:12:16 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:12:21 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:26 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:12:26 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:26 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:12:26 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:12:31 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:36 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:12:36 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:36 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:12:36 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:12:41 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:46 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:12:46 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:46 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:12:46 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:12:51 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:56 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:12:56 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:12:56 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:12:56 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:13:01 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:06 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:13:06 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:06 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:13:06 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:13:11 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:16 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:13:16 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:16 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:13:17 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:13:21 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:25 wazuh-authd[290551] main-server.c:708 at run_dispatcher(): INFO: New connection from 192.168.10.101
2024/04/30 04:13:25 wazuh-authd[290551] main-server.c:751 at run_dispatcher(): DEBUG: Request received: <OSSEC PASS: XXXXXXX OSSEC A:'XX-XXXXXXXX' V:'v4.7.3' G:'Macbook' K:'817ae2f4e82bbef68920cadb40d56f12aa8b504b'
>
2024/04/30 04:13:25 wazuh-authd[290551] auth.c:108 at w_auth_parse_data(): INFO: Received request for a new agent (XX-XXXXXXXXX) from: 192.168.10.101
2024/04/30 04:13:25 wazuh-authd[290551] auth.c:175 at w_auth_parse_data(): DEBUG: Group(s) is: Macbook
2024/04/30 04:13:25 wazuh-authd[290551] auth.c:356 at w_auth_validate_data(): WARNING: Duplicate name 'XX-XXXXXXXXX', rejecting enrollment. Agent '003' doesn't comply with the registration time to be removed.
2024/04/30 04:13:26 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:13:26 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:13:26 wazuh-remoted[291006] secure.c:414 at close_fp_main(): DEBUG: Opened rids queue size: 0
2024/04/30 04:13:26 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:13:26 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:31 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:34 wazuh-remoted[291006] secure.c:838 at key_request_reconnect(): DEBUG: Key-request feature is not available. Retrying connection in 300 seconds.
2024/04/30 04:13:36 wazuh-remoted[291006] manager.c:840 at c_files(): DEBUG: Updating shared files.
2024/04/30 04:13:36 wazuh-remoted[291006] secure.c:391 at rem_keyupdate_main(): DEBUG: Checking for keys file changes.
2024/04/30 04:13:36 wazuh-remoted[291006] manager.c:862 at c_files(): DEBUG: End updating shared files.
2024/04/30 04:13:36 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 04:13:41 wazuh-remoted[291006] state.c:161 at rem_write_state(): DEBUG: Updating state file.
2024/04/30 09:43:24 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/04/30 09:43:30 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:43:30 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:43:40 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:43:40 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:43:50 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:43:50 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:00 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:00 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:00 wazuh-agentd: INFO: Requesting a key from server: 192.168.10.101
2024/04/30 09:44:00 wazuh-agentd: INFO: Using password specified on file: etc/authd.pass
2024/04/30 09:44:00 wazuh-agentd: INFO: Using agent name as: XX-XXXXXXXXX
2024/04/30 09:44:00 wazuh-agentd: INFO: Waiting for server reply
2024/04/30 09:44:00 wazuh-agentd: INFO: Valid key received
2024/04/30 09:44:00 wazuh-agentd: INFO: Waiting 20 seconds before server connection
2024/04/30 09:44:20 wazuh-agentd: INFO: (1410): Reading authentication keys file.
2024/04/30 09:44:20 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:20 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:30 wazuh-agentd: WARNING: (4101): Waiting for server reply (not started). Tried: '192.168.10.101'. Ensure that the manager version is 'v4.7.3' or higher.
2024/04/30 09:44:30 wazuh-agentd: WARNING: Unable to connect to any server.
2024/04/30 09:44:30 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:30 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:40 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:40 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:50 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:44:50 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:00 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:00 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:10 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:10 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:10 wazuh-agentd: INFO: Requesting a key from server: 192.168.10.101
2024/04/30 09:45:10 wazuh-agentd: INFO: Using agent name as: XX-XXXXXXXXX
2024/04/30 09:45:10 wazuh-agentd: INFO: Waiting for server reply
2024/04/30 09:45:10 wazuh-agentd: ERROR: Duplicate agent name: XX-XXXXXXXX (from manager)
2024/04/30 09:45:10 wazuh-agentd: ERROR: Unable to add agent (from manager)
2024/04/30 09:45:20 wazuh-agentd: WARNING: (4101): Waiting for server reply (not started). Tried: '192.168.10.101'. Ensure that the manager version is 'v4.7.3' or higher.
2024/04/30 09:45:20 wazuh-agentd: WARNING: Unable to connect to any server.
2024/04/30 09:45:20 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:20 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:30 wazuh-agentd: INFO: Closing connection to server ([192.168.10.101]:2087/tcp).
2024/04/30 09:45:30 wazuh-agentd: INFO: Trying to connect to server ([192.168.10.101]:2087/tcp).