kubelet error [7415] very disruptive docker

[Tengku Arya Saputra]

Hello team,

How can I hide this alert in threat hunting? It is very distracting on my threat hunting dashboard.

I will provide you with alert.json for analysis.

Jan 27 00:40:52 master-node kubelet[7415]: E0127 07:40:52.966536    7415 container_log_manager.go:268] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-scheduler-master-node_ffd0002ad022a7de254446f43179d8b6/kube-scheduler/12.log\": failed to reopen container log \"b477a31a6e1fd1aa081e991d787ffc78781fc8f9853c7b8d57bc49c251d119f9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-scheduler-master-node_ffd0002ad022a7de254446f43179d8b6/kube-scheduler/12.log" containerID="b477a31a6e1fd1aa081e991d787ffc78781fc8f9853c7b8d57bc49c251d119f9"

Jan 27 00:40:54 master-node kubelet[7415]: E0127 07:40:54.305307    7415 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://REDACTED:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master-node?timeout=10s\": dial tcp REDACTED:6443: connect: connection refused" interval="7s"

Jan 27 00:40:53 master-node kubelet[7415]: E0127 07:40:53.319893    7415 kubelet_node_status.go:96] "Unable to register node with API server" err="Post \"https://REDACTED:6443/api/v1/nodes\": dial tcp REDACTED:6443: connect: connection refused" node="master-node"

Jan 27 00:40:52 master-node kubelet[7415]: E0127 07:40:52.967941    7415 remote_runtime.go:751] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="a96b571be61d2b789de9c6e1ba2a3c1e8bd43e58339732c73e608319d15f4107"

Jan 27 00:40:48 master-node agent[121504]: 2026-01-27 07:40:48 WIB | CORE | WARN | (pkg/collector/corechecks/system/disk/diskv2/disk.go:556 in getPartitionUsage) | Unable to get disk metrics for /var/lib/docker/containers/c59b7150991fd4574729359702306dfbada0ebfba153db7d792b1668bfafd33a/mounts/shm: permission denied. You can exclude this mountpoint in the settings if it is invalid.

Jan 27 00:40:46 master-node kubelet[7415]: E0127 07:40:46.601816    7415 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-apiserver\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-master-node_kube-system(cf56b83d983a34714b04c54fcd610e08)\"" pod="kube-system/kube-apiserver-master-node" podUID="cf56b83d983a34714b04c54fcd610e08"

And many more. How can I silence these alerts because they are very distracting when I am doing threat hunting?

Regards

[Olamilekan Abdullateef Ajani]

Hello,

The reason you are seeing that event on the dashboard is simply because the log alert level in your Wazuh manager ossec configuration has changed and the level to trigger an alert on the dashboard has been reduced to accommodate alerts of level 2 and above. OR there is a high alert level triggering it to appear on the dashboard.

You can confirm the first assumption by checking the ossec.conf on the manager:  /var/ossec/etc/ossec.conf

 <alerts>
    <log_alert_level>3</log_alert_level>
    <email_alert_level>12</email_alert_level>
  </alerts>

The log alert level should be 3 by default, this is to reduce the amount of noise, as you have said, that could cause distraction. This level can be increased. You can learn more about this in the documentation below:

https://documentation.wazuh.com/current/user-manual/manager/alert-management.html

The "Unknown problem somewhere in the system." alert triggers on level 2 with rule ID 1002, you can also test this via the logtest engine as described in the attached screenshot to see what level yours triggers at and confirm the issue

To ensure this does not appear on the dashboard anymore, you can either raise the log_alert_level to 3 OR create a custom rule to match the ingested log and discard the alert by assigning the rule level 0.

Something similar to the rule below:

<group name="silent error,">
  <rule id="155874" level="0">
   <if_sid>1002</if_sid>
    <match>failed to reopen container</match>
    <description>silent Unknown problem somewhere in the system logs</description>
  </rule>
</group>

The match tag is to tag the actual logs being ingested from the kubs pod so other events that match rule 1002 will not be wrongly triggered by the just-created custom rule.

I picked the log below from the logs you shared as test, all of them matched rule 1002.

Jan 27 00:40:52 master-node kubelet[7415]: E0127 07:40:52.966536    7415 container_log_manager.go:268] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_kube-scheduler-master-node_ffd0002ad022a7de254446f43179d8b6/kube-scheduler/12.log\": failed to reopen container log \"b477a31a6e1fd1aa081e991d787ffc78781fc8f9853c7b8d57bc49c251d119f9\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_kube-scheduler-master-node_ffd0002ad022a7de254446f43179d8b6/kube-scheduler/12.log" containerID="b477a31a6e1fd1aa081e991d787ffc78781fc8f9853c7b8d57bc49c251d119f9"

Ref:

https://documentation.wazuh.com/current/user-manual/manager/alert-management.html

https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html

Please let me know if you require further assistance on this.

[Tengku Arya Saputra]

Thank you, this works well, but I have a new problem.

 

This is really noisy. I've set it up in these rules, but there's no effect. It's still noisy on the dashboard.

<group name="local,windows,sysmon,">
<!-- Variant A: field path win.eventdata.* -->
<rule id="100952" level="0">
  <if_matched_sid>92052</if_matched_sid>

  <field name="win.eventdata.image" type="pcre2">(?i)\\cmd\.exe$</field>
  <field name="win.eventdata.parentImage" type="pcre2">(?i)\\forfiles\.exe$</field>

  <!-- parentCommandLine di event ada: ForFiles  /p \"D:\\Work\\AUTO_PRINT\" ... -->
  <field name="win.eventdata.parentCommandLine" type="pcre2">
    (?i)forfiles\s+.*?/p\s+\\?"D:\\\\Work\\\\AUTO_PRINT\\?"\s+.*?/d\s+-30\s+.*?cmd\s+/c\s+del\s+/q
  </field>

  <field name="win.eventdata.user" type="pcre2">(?i)^MEGAINS\\svr\.app$</field>

  <description>Suppress AUTO_PRINT cleanup via forfiles->cmd del (svr.app)</description>
</rule>

<!-- Variant B: field path data.win.eventdata.* -->
<rule id="100953" level="0">
  <if_matched_sid>92052</if_matched_sid>

  <field name="data.win.eventdata.image" type="pcre2">(?i)\\cmd\.exe$</field>
  <field name="data.win.eventdata.parentImage" type="pcre2">(?i)\\forfiles\.exe$</field>

  <field name="data.win.eventdata.parentCommandLine" type="pcre2">
    (?i)forfiles\s+.*?/p\s+\\?"D:\\\\Work\\\\AUTO_PRINT\\?"\s+.*?/d\s+-30\s+.*?cmd\s+/c\s+del\s+/q
  </field>

  <field name="data.win.eventdata.user" type="pcre2">(?i)^MEGAINS\\svr\.app$</field>

  <description>Suppress AUTO_PRINT cleanup via forfiles->cmd del (svr.app) (data.*)</description>
</rule>

  <rule id="111789" level="0">
    <if_matched_sid>92052</if_matched_sid>
    <field name="agent.id">004</field>
    <description>
      IGNORE: Sysmon EID1 cmd.exe abnormal parent (rule 92052) on SVR-APPIIS-PRD
    </description>
  </rule>

</group>

<group name="windows,sysmon,ignore,">
<rule id="111788" level="0">
  <if_matched_sid>92052</if_matched_sid>
  <field name="agent.id">004</field>
  <field name="win.system.message" type="pcre2">
    (?is)ParentImage:\s+C:\\\\Windows\\\\System32\\\\forfiles\.exe.*ParentCommandLine:.*D:\\\\Work\\\\AUTO_PRINT.*ForFiles.*?/d\s+-30.*?cmd\s+/c\s+del\s+/q\s+@file
  </field>
  <description>IGNORE: match via sysmon message (fallback) for AUTO_PRINT housekeeping</description>
</rule>

  <rule id="111742" level="0">
    <if_matched_sid>92052</if_matched_sid>

    <!-- hanya agent ini -->
    <field name="agent.id">004</field>

    <!-- process & parent -->
    <field name="win.eventdata.image" type="pcre2">(?i)\\cmd\.exe$</field>
    <field name="win.eventdata.parentImage" type="pcre2">(?i)\\forfiles\.exe$</field>

    <!-- parentCommandLine: toleran spasi & variasi quoting -->
    <field name="win.eventdata.parentCommandLine" type="pcre2">
      (?i)ForFiles\s+/p\s+.*D:\\\\Work\\\\AUTO_PRINT.*\s+/s\s+/d\s+-30\s+/c\s+.*cmd\s+/c\s+del\s+/q\s+@file.*
    </field>

    <!-- user service -->
    <field name="win.eventdata.user" type="pcre2">(?i)^MEGAINS\\\\svr\.app$</field>

    <description>IGNORE: forfiles housekeeping AUTO_PRINT spawning cmd.exe delete (SVR-APPIIS-PRD)</description>
  </rule>

</group>

Do you have any other solutions?

Regards

[Olamilekan Abdullateef Ajani]

Hello ,

The reason your rule did not work is because, from what you shared, you used if_matched_sid in place of if_sid tag. The if_matched_sid works along with timeframe and frequency, typically for chaining events. You can learn more about this in the documentation here.

If you want to silence that rule to discard the log, you can simply use the rule below:

<group name="local,windows,sysmon,">
<!-- Variant A: field path win.eventdata.* -->
<rule id="100952" level="0">

  <if_sid>92052</if_sid>  

  <description>Suppress AUTO_PRINT cleanup via forfiles->cmd del (svr.app)</description>
</rule>

</group>

You can make it more restrictive by adding specific fields to the rule so it does not go outside the required span.

Example:

<group name="local,windows,sysmon,">
<!-- Variant A: field path win.eventdata.* -->
<rule id="100952" level="0">

  <if_sid>92052</if_sid>  

  <field name="win.eventdata.image" type="pcre2">(?i)\\cmd\.exe$</field>
  <field name="win.eventdata.parentImage" type="pcre2">(?i)\\forfiles\.exe$</field>

  <description>Suppress AUTO_PRINT cleanup via forfiles->cmd del (svr.app)</description>
</rule>

</group>

To learn more about rules, you can check out the documentation below:
https://documentation.wazuh.com/current/user-manual/ruleset/ruleset-xml-syntax/rules.html

Please let me know if you require further assistance on this.