All wazuh agents disconnected due to SSL error

[Louis Hather]

Hi all,

We're running a wazuh cluster, v4.3.1, with a single manager handling wodles/gcp-pubsub and a single worker handling the events from agents. We register our Wazuh agents with a password which seems to be working fine. However, once registered the manager reports:

{"timestamp":"2022/09/12 13:45:55","tag":"wazuh-authd","pid":30643,"file":"auth.c","line":106,"routine":"w_auth_parse_data","level":"info","description":"Received request for a new agent (AGENT NAME) from: AGENT IP"}
{"timestamp":"2022/09/12 13:45:55","tag":"wazuh-authd","pid":30643,"file":"main-server.c","line":654,"routine":"run_dispatcher","level":"info","description":"Agent key generated for 'AGENT NAME' (requested by any)"}
{"timestamp":"2022/09/12 13:45:55","tag":"wazuh-authd","pid":30643,"file":"main-server.c","line":808,"routine":"run_writer","level":"debug","description":"Dumping changes into disk."}
{"timestamp":"2022/09/12 13:45:55","tag":"wazuh-authd","pid":30643,"file":"main-server.c","line":848,"routine":"run_writer","level":"debug","description":"[Writer] Performing insert([148581] AGENT NAME)."

{"timestamp":"2022/09/12 13:46:02","tag":"wazuh-remoted","level":"info","description":"(1409): Authentication file changed. Updating."}
{"timestamp":"2022/09/12 13:46:02","tag":"wazuh-remoted","level":"info","description":"(1410): Reading authentication keys file."}
{"timestamp":"2022/09/12 13:46:03","tag":"wazuh-authd","pid":30643,"file":"main-server.c","line":584,"routine":"run_dispatcher","level":"debug","description":"SSL Error (-1)"}
{"timestamp":"2022/09/12 13:46:07","tag":"wazuh-authd","pid":30643,"file":"main-server.c","line":591,"routine":"run_dispatcher","level":"info","description":"New connection from AGENT IP"}
{"timestamp":"2022/09/12 13:46:07","tag":"wazuh-authd","pid":30643,"file":"auth.c","line":89,"routine":"w_auth_parse_data","level":"error","description":"Invalid password provided by AGENT IP. Closing connection."}

I've verified that the passwords match - I think this is something to do with SSL rather than the password. The manager and worker also have correct ownership of files around authentication (client.keys, authd.pass). The manager is behind an AWS network load balancer, which forwards 1515/TCP to the manager and 1514/TCP to the workers. The agent logs the following:

2022/09/08 01:11:51 wazuh-agentd: WARNING: (4101): Waiting for server reply (not started). Tried: 'manager.wazuh.gutools.co.uk/54.73.53.14'.
2022/09/08 01:11:51 wazuh-agentd: WARNING: Unable to connect to any server.
2022/09/08 01:11:51 wazuh-agentd: INFO: Closing connection to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:11:51 wazuh-agentd: INFO: Trying to connect to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:13:01 wazuh-agentd: INFO: Closing connection to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:13:01 wazuh-agentd: INFO: Trying to connect to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:14:11 wazuh-agentd: INFO: Closing connection to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:14:11 wazuh-agentd: INFO: Trying to connect to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:14:37 wazuh-agentd: INFO: Closing connection to server (MANAGER URL/MANAGER IP:1514/tcp).
2022/09/08 01:14:37 wazuh-agentd: INFO: Trying to connect to server (MANAGER URL/MANAGER IP/tcp).
2022/09/08 01:14:37 wazuh-agentd: ERROR: (1216): Unable to connect to 'MANAGER IP/tcp': 'Connection refused'.
2022/09/08 01:14:47 wazuh-agentd: INFO: Trying to connect to server (MANAGER URL/MANAGER IP/tcp).
2022/09/08 01:15:47 wazuh-agentd: INFO: Requesting a key from server: MANAGER URL/MANAGER IP
2022/09/08 01:15:47 wazuh-agentd: INFO: No authentication password provided
2022/09/08 01:15:47 wazuh-agentd: INFO: Using agent name as: AGENT NAME
2022/09/08 01:15:47 wazuh-agentd: INFO: Waiting for server reply
2022/09/08 01:15:47 wazuh-agentd: ERROR: Invalid password (from manager)
2022/09/08 01:15:47 wazuh-agentd: ERROR: Unable to add agent (from manager)

It's probably also worth noting that restart the worker briefly allows some of the agents to reconnect, however they time out after 5 minutes. This does suggest the password/registration probably isn't the issue.

Please let me know if I can provide any more information, thank you!

[Matias Pereyra]

Hello! Thanks for using Wazuh.

This seems a connectivity issue and not a registration problem because the agent was successfully registered the first time. 

When it fails to connect to the manager many times (see "Trying to connect to server" messages), the agent will request a new key even when it already has a valid one, just in case that is the cause of the problem.

This re-registration may be failing because

• The agent doesn't have its own authd.pass file. See the message "INFO: No authentication password provided"
• There is a problem with the authorization certificates, have you configured them using the steps in Additional security options? That may be the cause of the message "SSL Error (-1)". This message isn't very descriptive but in this issue, there are some steps described to reproduce it and you can see that the cause is the certificates.

But even when you solve the issues above, the re-registration will fail because the agent has a valid key and it's already registered.

Can you perform some connectivity tests from the agent to verify it can reach the manager?

For example with netcat: nc -vz <IP> <Port>

Regards.

[Louis Hather]

Hi Matias, thanks for the response!

I've verified connectivity between the agents and the manage, that doesn't seem to be the issue. Here's auth config:

 <auth>
    <disabled>no</disabled>
    <port>1515</port>
    <use_source_ip>no</use_source_ip>
    <force_insert>yes</force_insert>
    <force_time>0</force_time>
    <purge>yes</purge>
    <use_password>yes</use_password>
    <limit_maxagents>no</limit_maxagents>
    <ciphers>HIGH:!ADH:!EXP:!MD5:!RC4:!3DES:!CAMELLIA:@STRENGTH</ciphers>
    <!-- <ssl_agent_ca></ssl_agent_ca> -->
    <ssl_verify_host>no</ssl_verify_host>
    <ssl_manager_cert>/var/ossec/etc/sslmanager.cert</ssl_manager_cert>
    <ssl_manager_key>/var/ossec/etc/sslmanager.key</ssl_manager_key>
    <ssl_auto_negotiate>no</ssl_auto_negotiate>
  </auth>

We haven't touched the default configuration for SSL, though it's worked up until now. 

When the wazuh-agent service starts, we've got a script (ExecStartPre in systemd unit file) which authenticates using the password and gets a key - then removes the password. We have many short lived instances, and services are never restarted past initial startup. This has worked fine until the recent monthly instance refresh of the leader, when it stopped working. As it's automated the only real difference is between versions of libraries, namely openssl. 

[Matias Pereyra]

Hello again!

Thank you for the details.

The configurations seem ok.

The certificates are used to validate the communication only during the registration process. That's why I thought it was a communication issue, the agent receives a valid key the first time. If this script you mention is responsible for the password file, now I understand the message in the agent "No authentication password provided".

Now, you say it was working before, can you tell me what was the previous and new openssl versions? What is the OS of both the agent and manager?
your cluster is v4.3.1, what is the Wazuh version of the agent? Can you upload the full ossec.log file of both hosts?

Regards.