Agents disconnecting

270 views
Skip to first unread message

Sandeep Renjith

unread,
Jul 18, 2021, 3:42:01 PM7/18/21
to Wazuh mailing list
Hi All, 

I have been facing an issue where a couple of windows agents keep disconnecting. 

  • The hosts in this case are Windows Server 2012 R2.
  • The ossec.conf is with default configuration. 
  • The issue seems to occur only when syscheck is enabled.
  • Only the default directories and registry entries are in syscheck section of the config.
  • I see the below error in Kibana before the agent disconnects.  
"\"Faulting application name: ossec-agent.exe, version: 0.0.0.0, time stamp: 0x60814dcb\r\nFaulting module name: ossec-agent.exe, version: 0.0.0.0, time stamp: 0x60814dcb\r\nException code: 0xc0000005\r\nFault offset: 0x0000e6e5\r\nFaulting process id: 0x5d1c\r\nFaulting application start time: 0x01d77bcd16f7b52f\r\nFaulting application path: C:\\Program Files (x86)\\ossec-agent\\ossec-agent.exe\r\nFaulting module path: C:\\Program Files (x86)\\ossec-agent\\ossec-agent.exe\r\nReport Id: f06c6875-e7c2-11eb-80e1-9418827c4cff\r\nFaulting package full name: \r\nFaulting package-relative application ID: \"",

  • The error   0xc0000005 seems to suggest a memory access violation.
  • The time between the agent starts and stops seems somewhat random. In most cases, it takes about a minute, while in some cases it takes a few hours. 
  • The issue is observed on two Windows Server 2012 R2 hosts. There are other hosts with the same OS version where the agent runs fine. 
  • Memory, CPU and Disk IO on the hosts are minimal.
  • Attaching the ossec.log file with debug level 2 enabled for windows and syscheck. ( Host Names, Domain Names and IPs are replaced with 'x' characters).
Any help would be greatly appreciated.

Regards,
Sandeep
ossec_debug.txt

antonio....@wazuh.com

unread,
Jul 19, 2021, 2:23:51 AM7/19/21
to Wazuh mailing list

Hi Sandeep

We are aware of a bug in the FIM synchronization mechanism that causes a segmentation fault. You can check the issue (that it’s solved already and it will be available in 4.2)
With the information that you are sharing (basically these two fields Exception code: 0xc0000005, Fault offset: 0x0000e6e5\) tell me that is that problem

Having said that, you have two options:

  • Identify the problematic registry entry and stop monitoring it: The problem was caused when a monitored entry in the Windows Registry has a value starting with :. If you follow the steps that I mentioned here, you can easily identify the entry and stop monitoring it.
  • Disable the registry synchronization mechanism: This will avoid the problem, but the Windows Registry inventory won’t be available, as the manager won’t get the information for the agent
  • Downgrade to a version prior to 4.1: The bug was introduced in the 4.1 version in this PR.

If you have any doubts don’t hesitate to contact us

​

Sandeep Renjith

unread,
Jul 19, 2021, 6:26:27 AM7/19/21
to Wazuh mailing list
Hi Antonio, 

Thanks for the prompt reply. 
I will add the registry entries for which 'dbsync no_data' was logged to the registry_ignore list in ossec.conf and monitor for a day. 
I will update my observations here with the status. 

Regards,
Sandeep
Message has been deleted

antonio....@wazuh.com

unread,
Jul 19, 2021, 6:54:02 AM7/19/21
to Wazuh mailing list

Hi again Sandeep.

I have been checking the logs that you shared and I think this one is the key message:

2021/07/17  14:57:15 ossec-agent[16712] run_check.c:78 at fim_send_sync_msg():  DEBUG: (6317): Sending integrity control message:  {"component":"fim_registry","type":"integrity_check_right","data":{"id":1626519424,"begin":"[x32]   HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet  Settings\\ZoneMap\\Ranges\\Range1:::Range","end":"[x32]  HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network  Connections:NC_PersonalFirewallConfig","checksum":"34672e80664a855c498ab26d06da1d3ae323acb0"}} 2021/07/17 14:57:15 ossec-agent[16712] receiver-win.c:128 at  receiver_thread(): DEBUG: Received message: '#!-fim_registry dbsync  no_data {"id":1626519424,"begin":"[x32]  HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet  Settings\\ZoneMap\\Ranges\\Range1:::Range","end":"[x32]  HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network  Connections:NC_PersonalFirewallConfig"}'

If you take a look, these two messages are repeated a lot of times in the log. This part of the message '#!-fim_registry dbsync no_data {"id":1626519424,"begin":"[x32] HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1:::Range","end": makes me think that the problematic key is "[x32] HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1. Could you try first to remove this key from the monitoring and check if the agent keeps crashing?

​

Sandeep Renjith

unread,
Jul 24, 2021, 12:26:42 PM7/24/21
to Wazuh mailing list
Hi Antonio, 

Thanks for the info. Will try it and let you know the status. 

We see the below entries as well in the logs with 'dbsync no_data'. Should I remove them all individually from FIM with a <registry_ignore> entry? 
Would an entry like this work to avoid all the keys with a : character "<registry_ignore type="sregex">^.*\:.*$</registry_ignore> " ?


```
sandeep@jarvis /m/c/U/r/W/temp> grep 'no_data' ossec_200721_1939.txt | awk -F "no_data" '{print $2}'| grep -oE 'HKEY[^"]*'                                                                                           HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\EventLog\\HardwareEvents:                                                                                                                                   HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\RasMan\\PPP\\EAP\\26:ConfigCLSID                                                                                                                            HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\CpqNicMgmt\\CPQNISNMP:Timeout                                                                                                                               HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\EventLog\\Application\\vmicvss:TypesSupported                                                                                                               HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\AppIDSvc:ObjectName                                                                                                                                         HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\CpqNicMgmt\\CPQNISNMP:SetsEnabled                                                                                                                           HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\.NET Data Provider for SqlServer\\Performance:Last Counter                                                                                                  HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\AppIDSvc:ImagePath                                                                                                                                          HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\.NET CLR Data\\Performance:Collect                                                                                                                          HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\.NET Data Provider for SqlServer\\Performance:IsMultiInstance                                                                                               HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Control\\SecurePipeServers\\winreg\\AllowedExactPaths:                                                                                                                HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Services\\.NET CLR Data\\Performance:Close                                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network Connections:                                                                                                                                     HKEY_LOCAL_MACHINE\\Software\\Policies\\SNMP\\Parameters:                                                                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\NetworkConnectivityStatusIndicator:                                                                                                                      HKEY_LOCAL_MACHINE\\System\\CurrentControlSet\\Control\\SecurePipeServers\\winreg:Description                                                                                                                        HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1:::Range                                                                                       HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network Connections:NC_PersonalFirewallConfig                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\DataCollection:DisableEnterpriseAuthProxy                                                                                                                HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\IPSec\\Policy\\Local:                                                                                                                                    HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1:::Range                                                                                       HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network Connections:NC_PersonalFirewallConfig                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1:::Range                                                                                       HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network Connections:NC_PersonalFirewallConfig                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1:::Range                                                                                       HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network Connections:NC_PersonalFirewallConfig                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Classes\\Protocols\\Name-Space Handler:                                                                                                                                                HKEY_LOCAL_MACHINE\\Software\\Classes\\Protocols\\Name-Space Handler\\mk\\*:CLSID                                                                                                                                    HKEY_LOCAL_MACHINE\\Software\\Classes\\cmdfile\\DefaultIcon:                                                                                                                                                         HKEY_LOCAL_MACHINE\\Software\\Classes\\cmdfile\\shell\\edit\\command:@                                                                                                                                               HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range1:::Range                                                                                       HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\Network Connections:NC_PersonalFirewallConfig                                                                                                            HKEY_LOCAL_MACHINE\\Software\\Classes\\Protocols\\Name-Space Handler:                                                                                                                                                HKEY_LOCAL_MACHINE\\Software\\Classes\\Protocols\\Name-Space Handler\\mk\\*:CLSID                                                                                                                                    HKEY_LOCAL_MACHINE\\Software\\Classes\\cmdfile\\DefaultIcon:                                                                                                                                                         HKEY_LOCAL_MACHINE\\Software\\Classes\\cmdfile\\shell\\edit\\command:@   
```

Sandeep Renjith

unread,
Jul 24, 2021, 1:01:43 PM7/24/21
to Wazuh mailing list
Hi Antonio, 

We tried adding  HKEY_LOCAL_MACHINE\\Software\\Policies\\Microsoft\\Windows\\CurrentVersion\\Internet Settings\\ZoneMap\\Ranges\\Range to the <registry-ignore> list. 
This did not work. The agent stopped again.

Regards,
Sandeep

antonio....@wazuh.com

unread,
Jul 29, 2021, 9:35:46 AM7/29/21
to Wazuh mailing list

Hello, Sandeep.

Sorry for the delayed response.

My recommendation is to disable the registry synchronization mechanism because a problem may be a really high number of values starting with : and the agent will keep crashing.
Please, try disabling the syncronization adding the following lines to the ossec.conf inside the syscheck section:

 <synchronization> 

   <enabled>yes</enabled> 

   <interval>5m</interval> 

   <max_interval>1h</max_interval>  

   <max_eps>10</max_eps>
   <registry_enabled>no</registry_enabled> 

 </synchronization> 

</syscheck>
​
Reply all
Reply to author
Forward
0 new messages