Windows Server Disk Space Alerts

[SIIL IT]

After a recent issue with one of our legacy servers, I was tasked with setting up monitoring and alerts for disk utilization on a couple of our Windows servers running 2008 and 2012 until they finally get shutdown in a few months.

I found this thread - https://groups.google.com/g/wazuh/c/hlhqDQeS3Y8
Started by setting up a group for the servers and adding the lines below to the group config

<localfile>
        <log_format>command</log_format>
        <command>
            powershell -command "$disk = Get-PSDrive C | Select-Object Used,Free;$total = $disk.used + $disk.free;if ($total -gt 0){ $PercentFree = [Math]::round((($disk.free/$total) * 100))} else {$PercentFree = 0};if ($PercentFree -le 15){Write-Host 'Device C:/ with less than 15% of free space - '$PercentFree '%'}
        "</command>
        <frequency>21600</frequency>
        <alias>free-space-disk-C</alias>
    </localfile>
    <localfile>
        <log_format>command</log_format>
        <command>
            powershell -command "$disk = Get-PSDrive E | Select-Object Used,Free;$total = $disk.used + $disk.free;if ($total -gt 0){ $PercentFree = [Math]::round((($disk.free/$total) * 100))} else {$PercentFree = 0};if ($PercentFree -le 15){Write-Host 'Device E:/ with less than 15% of free space - '$PercentFree '%'}
        "</command>
        <frequency>21600</frequency>
        <alias>free-space-disk-E</alias>
    </localfile>

On each of the systems, I edited the local_internal_options.conf and added the line

wazuh_command.remote_commands=1

I restarted the agent on each system after adding.

On the master server, I added the following to local_rules.xml

<group name="Appserver,">
  <rule id="100034" level="12">
    <if_sid>530</if_sid>
    <match>ossec: output: 'free-space-disk-C':</match>
    <regex>with less than 15%</regex>
    <description>Device with less than 15% of free space </description>
  </rule>

  <rule id="100035" level="12">
    <if_sid>530</if_sid>
    <match>ossec: output: 'free-space-disk-E':</match>
    <regex>with less than 15%</regex>
    <description>Device with less than 15% of free space </description>
  </rule>
</group>

Restarted the service and checked that the rule replicated out to the workers.
On the master, I also ran wazuh-logtest to test the two rules. The phase 3 result showed the rules were triggered and an alert to be generated.

The above has been running for about 48hrs now and not triggered an alert yet which at least 2 of the systems should have by now.

if I check the ossec.txt file before and after a restart of the service, I'm not seeing the powershell command run.

On two of the systems i've gone in today and added the two <localfile> entries manually into the ossec.conf and restarted the agent. The ossec.txt shows the commands have been run but I'm still not seeing the alert in the events on the dashboard.

My questions are
1. How do I troubleshoot the fact that the settings i'm adding to the group config don't seem to be making to the systems in the group?

2. How can I get the command to run from an admin command prompt as this makes a difference to the output of the command on the server 2008 system?

3. Is there a better way to do this than what i've tried above?

Thanks in advance for the assistance

Baffled, confused 'n sober!

[Mauricio Ruben Santillan]

Hello!

Answering your questions next:

1. How do I troubleshoot the fact that the settings i'm adding to the group config don't seem to be making to the systems in the group?

• Configurations from groups are saved into the agents' C:\Program Files (x86)\ossec-agent\shared\agent.conf. Not in the agents' own ossec.conf. So for starters, you should check the agents' agent.conf which should have a merge of all the configurations sent using Groups.

2. How can I get the command to run from an admin command prompt as this makes a difference to the output of the command on the server 2008 system?

• You can use Windows runas for this. An example here: runas /profile /user:administrator “Drive:\folder\program”. Still, the Wazuh agent by default runs using the LOCAL SYSTEM account which should have no problem running admin commands.

3. Is there a better way to do this than what i've tried above?

• Since Wazuh is capable of ingesting the output of any command (or text content from files), as long as you get this data from a command (or being saved as text into a file), you should be able of getting this information.

Additionally, there are other Powershell commands like Get-Volume that can show disk information.

Now, I've tested this integration you mentioned and run the command from the post you mentioned and got next output:

PS C:\Users\mau_m> $disk = Get-PSDrive C | Select-Object Used,Free;$total = $disk.used + $disk.free;if ($total -gt 0){ $PercentFree = [Math]::round((($disk.free/$total) * 100))} else {$PercentFree = 0};if ($PercentFree -le 15){Write-Host 'Device C:/ with less than 15% of free space - '$PercentFree '%'}
PS C:\Users\mau_m> echo $disk

       Used         Free
       ----         ----
81822896128 396893986816

PS C:\Users\mau_m>

First of all, as you can see here, the command saves the output to a variable called disk (Not sure why). Wazuh will ingest outputs, not variables content unless you run a command to extract the content of such variables. So the commands you're using just won't work.
Then, it is not providing the output the rules are expecting (rules match with ossec: output: 'free-space-disk-C/E': and with less than 15% and it is not in my output at least).

Doing some research, I found here a simple command that would show just the free disk percetage: get-psdrive c | % { $_.free/($_.used + $_.free) } | % tostring p

Output example:

PS C:\Users\mau_m> get-psdrive c | % { $_.free/($_.used + $_.free) } | % tostring p
82,57 %
PS C:\Users\mau_m>

For other drive, you only need to set its drive letter instead of c after get-psdrive.

So for C drive you could run it with next localfile module:

<localfile>
  <log_format>full_command</log_format>
  <command>powershell -command "get-psdrive c | % { $_.free/($_.used + $_.free) } | % tostring p"</command>
  <frequency>10</frequency>
  <alias>free-space-disk-C</alias>
</localfile>

and get alerts with next rules:

<group name="disk-monitoring">
  <rule id="110300" level="1">
    <if_sid>530</if_sid>
    <location>free-space-disk-C</location>
    <description>Free disk check.</description>
  </rule>
  <rule id="110301" level="8">
    <if_sid>110300</if_sid>
    <match type="pcre2">\n2[0-9]</match>
    <description>Low free disk space available.</description>
  </rule>
  <rule id="110302" level="12">
    <if_sid>110300</if_sid>
    <match type="pcre2">\n1[5-9]</match>
    <description>Critically low free disk space available.</description>
  </rule>
  <rule id="110303" level="15">
    <if_sid>110300</if_sid>
    <match type="pcre2">\n1[0-4]|\n\d\D|\n0\D</match>
    <description>Ultra critically low free disk space available.</description>
  </rule>
  <rule id="110304" level="3">
    <if_sid>110300</if_sid>
    <match type="pcre2">\n[3-9][0-9]|\n100</match>
    <description>Enough free disk space available</description>
  </rule>

</group>

Explanation:

• Rule 110300 will just match the events.
• Rule 110301 will match free disk percentage between 20 and 29 %.
• Rule 110302 will match free disk percentage between 15 and 19 %.
• Rule 110303 will match free disk percentage between 0 and 14 %.
• Rule 110304 will match free disk percentage between 30 and 100 %.

Here an alert example run in my PC:

{"timestamp":"2022-12-26T19:14:35.043-0300","rule":{"level":3,"description":"Free disk space available","id":"110304","firedtimes":3,"mail":false,"groups":["disk-monitoring"]},"agent":{"id":"003","name":"probook-mau","ip":"192.168.1.69"},"manager":{"name":"wazuh-manager-testing"},"id":"1672092875.18656","full_log":"ossec: output: 'free-space-disk-C':\n82,77 %","decoder":{"name":"ossec"},"location":"free-space-disk-C"}

I hope this helps. Let me know how it goes.

[SIIL IT]

Thank you, Mauricio. Shortly after I read this, I noticed that I was actually getting alerts from my original setup (can be a lot of lag between setting something up and seeing results) but I thought I would give your code a rules a try as the rules are better than my original.

 As of this morning, I’m not seeing alerts from systems that should be triggering them and checking the ossec.txt file on those systems, I see the entry below

2022/12/28 08:10:45 wazuh-agent: ERROR: Remote commands are not accepted from the manager. Ignoring it on the agent.conf

 I do have the entry in the local_internal_options - wazuh_command.remote_commands=1

Plus, I have restarted the agent since all the changes

 

Is there anything else I need to do to enable remote commands?

[SIIL IT]

Checking internal_options.conf on the monitored systems, I'm seeing 

wazuh_command.remote_commands=0 
Where can I change that option at least for this specific group?