Problem upgrading wazuh windows agents to V4.12.0 via official WPK file

[Fco. Javier C.]

I initially published on Reedit but I think there is more reach here.

I'm having a problem updating Windows agents via WPK that I don't know how to address, or if it's better to wait for v4.12.1.

We have deployed Windows agents with version 4.8.1, and when upgrading the agents using Wazuh's own WPK, version 4.12.0, we find that the process doesn't complete or fails in most cases.

• Sometimes, when running the agent_upgrade command, the process remains running for hours, without completing or timing out.

• Other times result in the message "Upgrade task has appeared to be done, but the notification has never reached the manager."

• Other times, it indicates that it has been successfully updated to v4.12.0. The agent appears connected in the console showing the correct version, but after a few minutes, it appears disconnected. On these servers, the agent is stopped, and when started manually, an error is returned indicating that the service cannot be started.

When trying to review the agent logs locally, it's not possible because, as an administrator, it indicates that we don't have permissions to view the log (it's as if the permissions on the wazuh-agent/ossec-agent directory had become too restrictive).

All I can do is run the upgrade and reinstall the agent using the .msi in cases where the process fails but I haven't tried this yet and it's not feasible in our case.

Any suggestions?

PS: The Linux agent version upgrade were performed correctly using the WPK in all cases.

[Fco. Javier C.]

@Jumpy-Ad-9456 commented to me on Reddit:

Hi, there's a way to avoid this problem with WPK, it is more manual, but will solve the problem. It's requires using .msi installer. You can execute it like this: msiexec /i wazuh-agent-4.12.0.msi /quiet /norestart

But before that, you'll need to fix the permissions problem you mentioned in the folder.

After updating Linux agents without problems using WPK, today i can confirm that updating Windows agents with WPK v4.12.0 causes the service to fail to start in most cases (in the cases where I've tried updating the agent via WPK, I don't know if anyone else has had this same issue).

Investigating the problem, I identified the following in the installer.log:

Action start 13:31:17: SchedSecureObjectsRollback.
SchedSecureObjectsRollback: Entering SchedSecureObjectsRollback in C:\Windows\Installer\MSI4524.tmp, version 3.11.4516.0
SchedSecureObjectsRollback: Error 0x8007007b: Unable to schedule rollback for object:
SchedSecureObjectsRollback: Failed to store ACL rollback information with error 0x8007007b - continuing
Action ended 13:31:17: SchedSecureObjectsRollback. Return value 1.

Repeating tests on another Windows server, I check the permissions on the ossec-agent/wazuh-agent directory:

Initial ACLS:

C:\>icacls "C:\Program Files (x86)\ossec-agent"
C:\Program Files (x86)\ossec-agent NT SERVICE\TrustedInstaller:(F)
NT SERVICE\TrustedInstaller:(CI)(IO)(F)
NT AUTHORITY\SYSTEM:(F)
NT AUTHORITY\SYSTEM:(OI)(CI)(IO)(F)
BUILTIN\Administradores:(F)
BUILTIN\Administradores:(OI)(CI)(IO)(F)
CREATOR OWNER:(OI)(CI)(IO)(F)
ENTIDAD DE PAQUETES DE APLICACIONES\TODOS LOS PAQUETES DE APLICACIONES:(RX)
ENTIDAD DE PAQUETES DE APLICACIONES\TODOS LOS PAQUETES DE APLICACIONES:(OI)(CI)(IO)(GR,GE)
ENTIDAD DE PAQUETES DE APLICACIONES\TODOS LOS PAQUETES DE APLICACIÓN RESTRINGIDOS:(RX)
ENTIDAD DE PAQUETES DE APLICACIONES\TODOS LOS PAQUETES DE APLICACIÓN RESTRINGIDOS:(OI)(CI)(IO)(GR,GE)

After running the update via WPK, the ACLs change and the Wazuh service no longer starts

C:\>icacls "C:\Program Files (x86)\ossec-agent"
C:\Program Files (x86)\ossec-agent NT AUTHORITY\Usuarios autentificados:(RX)
NT AUTHORITY\SYSTEM:(OI)(CI)(F)
BUILTIN\Administradores:(OI)(CI)(F)

 

Checking the Windows logs, I see multiples Windows Event ID: 1000;

Nombre de la aplicación con errores: wazuh-agent.exe, versión: 4.12.0.0, marca de tiempo: 0x681200fa
Nombre del módulo con errores: wazuh-agent.exe, versión: 4.12.0.0, marca de tiempo: 0x681200fa
Código de excepción: 0xc0000005

 

I understand that the problem originates from an ACL modification made during the agent update process using WPK. Perhaps a bug? Resetting the ACLs manually solves the problem, but it's not practical to do so on more than 200 computers.

Could someone check if it's a bug or an error in the official WPK generation and if it's possible to generate a new one?

Thanks for read me!

PS: If I have to create an issue in the repo, I'll die, hahaha

[Fco. Javier C.]

After searching and searching, I found the following issue:  https://github.com/wazuh/wazuh/issues/24078

It indicates that it's being fixed with a PR: https://github.com/wazuh/wazuh/pull/25429

Could this be the same thing? A possible regression of the problem? 

[Ayooluwa Paul Akindeko]

When upgrading via WPK, the installer strips all directory permissions to reconfigure them, but it seems this creates a brief window where the MSI installer itself loses permission access to the directory which then makes the  SchedSecureObjectsRollback to fail.
Since you mentioned that manually resetting the ACL solves the problem but is not feasible due to the number of agents, maybe you can explore remotely running a script to do this on your agents or a group policy.
I am trying to explore options that might be available to you just incase this is not the expected behaviour.

[francisco...@gmail.com]

I'm asking out of curiosity. Has anyone been able to verify if the behavior is as expected or is it just a matter of my environment? Having to remotely execute commands to correct ACLs and restart the agent service is unfortunately not feasible for me. 

Does anyone know when version 4.12.1 will be released? Just in case the new WPK allows me to update without problems :-) 

Thanks to everyone and for everything,

Regards

[Ayooluwa Paul Akindeko]

There is a part of the installation script that first checks the version of your windows before resetting the permission, this means that it is possible that the issue is environment specific. Can you share a bit of detail about the agents, what version of Windows are your agents running?