Windows agents failing, can't be restarted

[Whit Blauvelt]

We've had some 20 Windows agents disconnect, and they fail again quickly when restarted. Windows System Logs just says "The Wazuh service terminated unexpectedly." 

Wazuh file name and version: wazuh-agent-3.2.0-1.msi

Windows 2012 r2 Version 6.3 build 9600

Last thing in the local ossec.log is: "2018/03/05 10:54:56 wazuh-modulesd:syscollector: INFO: Starting Operating System inventory."

There's no AV running on these systems.

How do we diagnose this?

Thanks,

Whit

[Santiago Bassett]

Hi,

do you have shared syscollector information in agent.conf? If so try removing it (from the manager, and also clean it at the agent just in case) as that was known to cause a segmentation fault in 3.2.0. This has been fixed in 3.2.1

I hope it helps,

Santiago. 

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/15e0a6e8-3925-408e-9375-1fd227665957%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Whit Blauvelt]

On Monday, March 5, 2018 at 11:53:47 AM UTC-5, Santiago Bassett wrote:

do you have shared syscollector information in agent.conf? If so try removing it (from the manager, and also clean it at the agent just in case) as that was known to cause a segmentation fault in 3.2.0. This has been fixed in 3.2.1

 

Hi Santiago,

We have whatever is the default on the clients. On the server all I see for that is:

root@xxx-ossec:/var/ossec/etc/shared/default# more agent.conf

<agent_config>
  <!-- Shared agent configuration here -->
</agent_config>

On the Windows systems, default-ossec.conf has nothing about syscollector. There is no agent.conf file.

Whit 

[Chema Martinez]

Hi Whit,

To debug what is happening in your agents. You could try the following:

Could you see if the Operating System information about that Windows agents has been stored in the database correctly? To do this, go to the manager and type the following (for the agent 001 for example):

root@ubuntu:~# sqlite3 /var/ossec/queue/db/001.db

SQLite version 3.11.0 2016-02-15 17:29:24

Enter ".help" for usage hints.

sqlite> select * from sys_osinfo;

1831895358|2018/03/05 21:13:54|W7|i686|Microsoft Windows 7 Professional|6.1.7601|||6|1|7601|||

It should retrieve the OS information of your agent and check if the scan worked for your agents.

Another thing that could help us is setting the debug mode in the Windows agents, editing the "local_internal_options.conf" file in the agent adding the following line:

windows.debug=2

After that, restart your agent and look for the logs retrieved from the syscollector module.

Apart from that, I recommend you to upgrade your environment to the last version, v3.2.1, where several bugs related to Syscollector have been fixed.

Here you can find the available packages: https://documentation.wazuh.com/3.x/installation-guide/packages-list/index.html

I hope that helps, and please let us know if the issue persists.

Best regards,

Chema. 

Chema Martinez | IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/435eb516-7222-4791-8b45-aa3ff6cbd85c%40googlegroups.com.

[Whit Blauvelt]

Hi Chema,

Here's an example for one of the disconnected agents:

root@rpc-ossec:~# sqlite3 /var/ossec/queue/db/065.db

SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> select * from sys_osinfo;

885645261|2018/03/02 01:29:42|C01-ADS-SVC02|x86_64|Microsoft Windows Server 2012 R2 Standard|6.3.9600||6|3|9600|||

 

Another thing that could help us is setting the debug mode in the Windows agents, editing the "local_internal_options.conf" file in the agent adding the following line:

windows.debug=2

A complication on this side is I'm not a Windows admin. The admin doing the agent installations on Windows claims there is no .conf file other than default-ossec.conf with the agent. I haven't looked. Are there other .conf files stores more obscurely?

After that, restart your agent and look for the logs retrieved from the syscollector module.

Look where?

 

Apart from that, I recommend you to upgrade your environment to the last version, v3.2.1, where several bugs related to Syscollector have been fixed.

Upgrading does allow the disconnect agents to keep running, rather than immediately crash. On the other hand, the majority of Windows agents we'd installed -- all on the same Windows version, haven't crashed yet. So it's a crash that happens after a few days, or not, but that leaves something in a bad state for restarting that agent. Yet the upgrade does let the agent be started again. So if the upgraded agents stay up, we're good.

Thanks,

Whit

[Chema Martinez]

It seems that the OS scan is working and storing the information in the Databases, so I think the best way to catch the issue is to enable the debug mode and wait for another crash, looking for any suspicious log in the "ossec.log" file.

In any Windows agent should be the following configuration files by default:

• default-ossec.conf, ossec.conf, ossec.conf.bak or last-ossec.conf (depending on the situation).
• internal_options.conf (to modify the internal behavior of some components). 
• local_internal_options.conf (this file is used to edit your internal options and not lose changes when upgrading)

All of them located in the installation folder, called "ossec-agent".

Anyway, as I told you, to upgrade your agents is a good point.

Best regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/48d7d1b5-4922-4a20-9f70-50647ed42608%40googlegroups.com.