Troubleshooting FIM and Agent Stopping

[Vicente Cañaveras]

Hi,

I have a problem with an windows agent that stops when FIM is activated.

If I try to activate FIM only in few folders it works as expected but when I try to explore a folder with many subfolders and a lot of files it never ends and Stop agent in 1 hour aprox.

Windows Agent is Windows 2016 with version 4.2.3 and Wazuh master version is 4.2.4

I have try with realtime=yes-->agent stopped; whodata=yes --> agent stopped; only with check_all=yes --> agent stopped.

I've checked with syscheck.debug=2 but no error is displayed, simply agent stops.

Can someone help me with this issue please?

Thanks!

[jeremias...@wazuh.com]

Hi and thank you for using Wazuh!
Let me better understand your scenario so I can investigate what could be happening:
When you observe that Wazuh Agent stopped:
- Is the service stoped in the Service Manager or just stopped logging? You can check the service status i.e. in the Task Manager (See attached screenshot)

- Do you find a log from Wazuh in Windows Event Viewer?  (See attached screenshot)

- Can you share with us the output of Event Viewer? At least the Application section?
- If we identify a crash of the service: Are you agree if we collect a crash dump to investigate the problem?

From the configuration:
- Is  <directories check_all="yes" whodata="yes">e:/programa</directories> the configuration of the directory that you added when Wazuh agent stoped?
- Can you bring me detail of how many folders and files are in this directory? I will try to reproduce the issue in a local environment.

With this information, we can better investigate this problem.
Best regards.

[Vicente Cañaveras]

Hi Jeremias,

What I mean when agent stops is Service is stopped. If I click refresh on Wazuh Agent Manager it shows "Status: Stopped"

And I see this on Application Event Viewer:

Nombre de registro:Application

Origen:        Windows Error Reporting

Fecha:         21/10/2021 17:22:51

Id. del evento:1001

Categoría de la tarea:Ninguno

Nivel:         Información

Palabras clave:Clásico

Usuario:       No disponible

Equipo:        SERVERDADES.intranet.DOMAIN.com

Descripción:

Depósito con errores , tipo 0

Nombre de evento: APPCRASH

Respuesta: No disponible

Identificador de archivo CAB: 0

Firma del problema:

P1: wazuh-agent.exe

P2: 0.0.0.0

P3: 615c2757

P4: msvcrt.dll

P5: 7.0.14393.2457

P6: 5b7e2dd0

P7: c0000005

P8: 00089619

P9: 

P10: 

Archivos adjuntos:

Es posible que estos archivos estén disponibles aquí:

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_wazuh-agent.exe_298eaaf7c8aa5c9c641b1a589aa7e00405b5ad8_31add48b_38a45243

Símbolo de análisis: 

Nueva búsqueda de una solución: 0

Identificador de informe: d82c8baf-2942-4fed-ac07-6d94b49afa95

Estado del informe: 4

Depósito con algoritmo hash: 

XML de evento:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Windows Error Reporting" />

    <EventID Qualifiers="0">1001</EventID>

    <Level>4</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2021-10-21T15:22:51.671369600Z" />

    <EventRecordID>490769</EventRecordID>

    <Channel>Application</Channel>

    <Computer>SERVERDADES.intranet.DOMAIN.com</Computer>

    <Security />

  </System>

  <EventData>

    <Data>

    </Data>

    <Data>0</Data>

    <Data>APPCRASH</Data>

    <Data>No disponible</Data>

    <Data>0</Data>

    <Data>wazuh-agent.exe</Data>

    <Data>0.0.0.0</Data>

    <Data>615c2757</Data>

    <Data>msvcrt.dll</Data>

    <Data>7.0.14393.2457</Data>

    <Data>5b7e2dd0</Data>

    <Data>c0000005</Data>

    <Data>00089619</Data>

    <Data>

    </Data>

    <Data>

    </Data>

    <Data>

    </Data>

    <Data>C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_wazuh-agent.exe_298eaaf7c8aa5c9c641b1a589aa7e00405b5ad8_31add48b_38a45243</Data>

    <Data>

    </Data>

    <Data>0</Data>

    <Data>d82c8baf-2942-4fed-ac07-6d94b49afa95</Data>

    <Data>4</Data>

    <Data>

    </Data>

  </EventData>

</Event>

Here you will find attached how many files and folders are in "programa" folder

The response for: "Is  <directories check_all="yes" whodata="yes">e:/programa</directories> the configuration of the directory that you added when Wazuh agent stoped?" is YES, but if I try with <directories check_all="yes" whodata="yes">e:/programa/LABORATORIO</directories> for example, wazuh also crash.

And of course, I agree with collectting a crash dump...

regards!

[jeremias...@wazuh.com]

Hi.

From the provided information we can confirm this is a crash of Wazuh Agent.

It seems that this issue wasn´t reported yet.

We will be working to reproduce the problem and I will get in touch with other co-workers to confirm which information can we ask you to create the related issue and continue with the investigation.

I will get back to you ASAP.

Best regards.

[Vicente Cañaveras]

Hi,

Thanks for your effort... I'll wait for new version and hope it will be resolved..

Regards!

[jeremias...@wazuh.com]

Hello again and sorry for the delay.
If you agree it will be very useful for our investigation to collect a memory dump of your Wazuh Agent when it crashes.
To do so, the easiest way will be to work with procdump. This is a tool provided by Microsoft to generate memory dumps from any process.
Here is a link to download the tool. Please, observe that this is a link to Microsoft site, that's why we trust in this tool. We will never suggest you download or run applications from untrusted developers.

To collect the memory dump you need to:

- Download and extract the tool.
- Start Wazuh Agent with the faulting configuration

- Open a command prompt with elevated privileges and stand on the tool folder.

- Run procdump.exe -t wazuh on any command prompt with elevated privileges. This will run procdump with default options over wazuh service and collect the memory dump when the process terminates.

- Wait until Wazuh crashes.

- Collect the output file in the same tool folder (wazuh-agent.exe_*.dmp).

- Restart Wazuh.

- Repeat the process but with  procdump.exe -e wazuh. This will collect the memory dump when the process matches an unhandled exception.

- Collect the new output file.

- Send us the collected files for investigation.

On the other hand, this error seems to be a problem when Wazuh tries to recursively scan too many folders. As a workaround, you can set a limit to the folder recursion. i.e. e:/programa</directories could be configured like:

<directories recursion_level="10" check_all="yes" whodata="yes">e:/programa</directories>

and test if this limit satisfies your requirement and avoids the crash.

Let me know how the test goes and if you have further doubts please don`t hesitate to ask.

Best regards.

[jeremias...@wazuh.com]

Hello, 
I want to inform you that this problem is being tracked in the following issue:
https://github.com/wazuh/wazuh/issues/10692

We will keep you updated for any new and hope to bring a solution for this problem ASAP.

Best regards.

[Jose Luis Carreras Marin]

Hello Hello info.viprovic

I have been reading and working on the issue opened by my coworker. I have analyzed this thread and all the information you gave us, but I can't find a way to reproduce the problem.
Could you give us some more information about the environment? I could observe that the monitored directory is "e:\program", is it some remote disk or something special?

On the other hand, is it a machine with limited resources? Any information of this type is relevant and could be of help to us.

Thank you very much and best regards!