Failed upgrades: Agent 001 status: Send lock restart error

[Андрей Горин]

Hi all!
After upgrading wazuh from 4.5.0 to 4.6.0, I tried to run a remote update agents using /var/ossec/bin/agent_upgrade -a, but I get the error: Failed upgrades: Agent 001 status: Send lock restart error" every time. There are no errors when updating locally.
Please help me resolve the issue.

[Raul Del Pozo Moreno]

Hello

This error may be due to a network problem, specifically, the log error has the following explanation:

Before sending the WPK installer, the manager locks the agent to prevent it from restarting due to a shared configuration change (merged.mg).

So it seems that the manager is not being able to communicate correctly with the agent, do you have the following log file in the agent? If it exists, please show me the content

/var/ossec/log/upgrade.log

In general, this problem should be solved by increasing the timeout time and/or the maximum attempts in the Wazuh manager, for this, modify the /var/ossec/etc/internal_options.conf file and restart the Wazuh manager

• remoted.request_timeout
• remoted.max_attempts

Related documentation about the internal options: https://documentation.wazuh.com/current/user-manual/reference/internal-options.html#remoted

After the changes have been made, please try to upgrade the agent again.

Regards

Raúl Del Pozo Moreno QA + Automation engineer

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/50fe0349-2031-421e-ba38-2fd46f31840cn%40googlegroups.com.

[Андрей Горин]

Hello, I set the maximum values for these parameters - 

• remoted.request_timeout
• remoted.max_attempts

but the situation did not change.

There are no new entries in the update.log.

среда, 8 ноября 2023 г. в 16:40:49 UTC+3, Raul Del Pozo Moreno:

[Андрей Горин]

четверг, 9 ноября 2023 г. в 09:43:17 UTC+3, Андрей Горин:

[Raul Del Pozo Moreno]

Just to be sure, did you restart the Wazuh manager?

/var/ossec/bin/wazuh-control restart

Looking at wazuh-remoted messages, it does seem to be a connection problem, can you tell me more about your infrastructure? Is it just one node or is it a Wazuh cluster of several nodes? If this is the case, from which node are you executing the upgrade (master or worker)?

Please share the following information

• From the Wazuh manager (hide sensitive information if necessary please)

/var/ossec/bin/wazuh-control info
/var/ossec/bin/agent-control -i 001

• From the Wazuh agent 001 (this is for a Linux agent, the path may change if macOS or Windows is used)

/var/ossec/bin/wazuh-control info

/var/ossec/bin/wazuh-control status 

Do you have a firewall configured on the agent or manager? Please check the ports necessary for the correct operation of Wazuh and modify the firewall rules if necessary 

• https://documentation.wazuh.com/current/getting-started/architecture.html#required-ports

Raúl Del Pozo Moreno QA + Automation engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b678dfa9-db65-4f1a-a981-d75d19b2ffdbn%40googlegroups.com.

[Андрей Горин]

I use 1 master server, 1 dashboard server, 2 indexer servers, as well as 9 workers and about 400 agents.
All firewall settings are set correctly for each server. Everything worked perfectly before update 4.6.0 =(
There is a connection between the servers, but it is not possible to install it remotely on any of the agents. 
1. screen master
2. screen agent

четверг, 9 ноября 2023 г. в 16:54:34 UTC+3, Raul Del Pozo Moreno:

[Raul Del Pozo Moreno]

I have tried to reproduce your problem but I have not been able to achieve it using a Wazuh manager 4.6.0 and a Wazuh agent 4.5.0, both in a cluster with a worker and individually

I'm going to talk to the team about it, but from the images that you've shown me, it seems that the manager is not able to reach the agent in his requests, and for him to mark the agent as active is strange.

Can you do a little test with the agent you show? If you stop the agent, does the manager update its status to Disconnected?

It is very possible that to investigate further we will have to enable the wazuh-remoted debug mode, but having 400 agents would possibly generate too many logs, so in the meantime, I am going to try to determine something more about the problem you may have

Raúl Del Pozo Moreno QA + Automation engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fb3e193c-3e9d-4b93-93ed-bd7a073c962bn%40googlegroups.com.

[Андрей Горин]

Yes. Then the agent is disconnected, the status changes

четверг, 9 ноября 2023 г. в 18:40:56 UTC+3, Raul Del Pozo Moreno:

[Андрей Горин]

This is really strange, considering that the agent is synchronized with the worker, but when sending an update request remotely, it feels like something is interfering with it.

четверг, 9 ноября 2023 г. в 18:40:56 UTC+3, Raul Del Pozo Moreno:

[Raul Del Pozo Moreno]

Mmm, the Wazuh manager/agent connection "seems" to be OK, that is why I mentioned that it is possibly a connection/infrastructure problem and that is why I mentioned the firewall

Okay, let's look at it in another way.

Since you have a Wazuh cluster, the recommended option to upgrade the agents is using API calls. It is necessary to use the following commands, in this case, for only one agent (we are going to focus on that agent 287)

- Documentation: https://documentation.wazuh.com/current/user-manual/agents/remote-upgrading/upgrading-agent.html#using-the-restful-api

1. Get token

TOKEN=$(curl -u wazuh:wazuh -k -X POST "https://localhost:55000/security/user/authenticate?raw=true")

2. List agents not updated (with respect to the manager)

curl -k -X GET "https://localhost:55000/agents/outdated?pretty=true" -H "Authorization: Bearer $TOKEN"

3. Upgrade the agent

curl -k -X PUT "https://localhost:55000/agents/upgrade?agents_list=287&pretty=true" -H "Authorization: Bearer $TOKEN"

4. Check the upgrade status

curl -k -X GET "https://localhost:55000/agents/upgrade_result?agents_list=287&pretty=true" -H "Authorization: Bearer $TOKEN"

You should see a key status with the value Updating

"status": "Updating",

and when the task finishes, the value should change to Updated

"status": "Updated"

In the meantime, I will continue to try to reproduce the problem

Raúl Del Pozo Moreno QA + Automation engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/ede4fcac-32c6-4913-8344-bddb7f89c076n%40googlegroups.com.

[Андрей Горин]

ty. What am I doing wrong? a little confused

пятница, 10 ноября 2023 г. в 16:14:36 UTC+3, Raul Del Pozo Moreno:

[Raul Del Pozo Moreno]

Possibly related to the wazuh user's password, if you changed the password of said user you must use the one that corresponds to the wazuh user, this password is the one set by default

• https://documentation.wazuh.com/current/user-manual/api/getting-started.html#logging-into-the-wazuh-api

Use the cURL command to log in. The Wazuh API will provide a JWT token upon success. Replace <user> and <password> with yours. By default, the user is wazuh, and the password is wazuh. If SSL (HTTPS) is enabled in the API and it is using the default self-signed certificates, it will be necessary to add the parameter -k

Raúl Del Pozo Moreno QA + Automation engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9bf57f93-409a-4906-ab82-f7538efefbd8n%40googlegroups.com.

[Андрей Горин]

Good afternoon!
I was able to launch the update using the API, I hope that I can also fix the error through a standard remote update

пятница, 10 ноября 2023 г. в 16:50:43 UTC+3, Raul Del Pozo Moreno:

[Raul Del Pozo Moreno]

Interesting, using the tool agent_upgrade it fails but using the API it doesn't, correct?

Since the recommended option to upgrade the agents from a cluster is to use the API, I recommend using this method, but regarding the error using agent_upgrade, I am going to consult with the team about what this could be due to since it is not a usual situation

Have you had any other errors with other agents using the API?

[Raul Del Pozo Moreno]

Hello

I have managed to reproduce this error using a Wazuh master + worker cluster, the conditions have been

1. Connect worker to master (both with firewall disabled)
2. Set worker IP to agent
3. Start agent
4. Wait for the agent to be marked as active in the master node
5. Enable ufw in worker (ufw enable) (disabled in master node)
6. Block port 1514 in worker node (ufw deny 1514)
7. Run upgrade from worker with agent_upgrade (the Send lock restart error message is obtained)

Please make sure that the firewall is correct on all Wazuh manager nodes, both master and worker.

Raúl Del Pozo Moreno QA + Automation engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/bc734607-b5bc-4a48-8b50-4623187e7e15n%40googlegroups.com.

[Андрей Горин]

Good afternoon, thanks for the recommendations. I checked, unfortunately, this is not the problem. The firewall is configured and has not changed properly for many years, all the necessary rules for port 1514.1515 are installed =(

понедельник, 13 ноября 2023 г. в 16:40:31 UTC+3, Raul Del Pozo Moreno:

[Raul Del Pozo Moreno]

Hello, do you think it would be possible to reproduce this behavior outside of that environment? That is, an environment outside that cluster but on the same network

To obtain more information we will need to activate the debug mode of wazuh-remoted, and with the number of agents you have connected I would like to avoid modifying said environment as much as possible since a lot of information will be registered in the Wazuh manager

Raúl Del Pozo Moreno QA + Automation engineer

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/4d2f8298-9e3c-4b67-bd14-709004cf59ebn%40googlegroups.com.