Audit Kubernetes issues

[Phạm Q. Đạt]

Hi Wzuh team,
We plan to audit Kubernetes events with Wazuh following the instruction here https://wazuh.com/blog/auditing-kubernetes-with-wazuh/ but we're stuck at some points. We would appreciate your help on it.
There are 3 steps to implement it in your blog:
1. Create a webhook listener on the Wazuh server to receive logs from the Kubernetes cluster.
2. Enable auditing on the Kubernetes cluster and configure it to forward audit logs to the Wazuh webhook listener.
3. Create rules on the Wazuh server to alert about audit events received from Kubernetes.

We did the same steps as the instructions but we can't see any alerts for creating and deleting a deployment.
For debugging, we run the custom-webhook script directly and it shows me that requests sent to our Wazuh are timeout.

To understand more about how it works, we print out the content of the request by adding this line to the custom-webhook.py:

Now we run the test command to create and delete a deployment but we can't find anything on our Wazuh side. It means that Wazuh doesn't receive the logs well.

------------------------------------

The second question is that we got another issue when creating the detection rules. We followed the instruction but when testing the rules with the log which is your sample creation log, it showed that there is no alert for it.

{ "kind": "EventList", "apiVersion": "audit.k8s.io/v1", "metadata": {}, "items": [ { "level": "Metadata", "auditID": "6ae321a6-0735-41a6-a9d9-050f9a75644c", "stage": "ResponseComplete", "requestURI": "/apis/apps/v1/namespaces/default/deployments?fieldManager=kubectl-create&fieldValidation=Strict", "verb": "create", "user": { "username": "minikube-user", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "192.168.132.137" ], "userAgent": "kubectl/v1.25.3 (linux/amd64) kubernetes/434bfd8", "objectRef": { "resource": "deployments", "namespace": "default", "name": "hello-minikube", "apiGroup": "apps", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2022-11-08T15:45:13.929428Z", "stageTimestamp": "2022-11-08T15:45:13.946284Z", "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" } } ] } 

We hope to see your advice. 

Regards.

[Phạm Q. Đạt]

Hello community,

Does anyone know the solution to my questions?

Thank you.

[Awwal Ishiaku]

Hello, 

To investigate this, kindly share your Kubernetes API server configuration file /etc/kubernetes/manifests/kube-apiserver.yaml.

Did you create a systemd service for the webhook as stated in the blog post?

If yes, show the status of the service by running  systemctl status wazuh-webhook.service

Show me the output when you run this command.

Additionally, kindly share your Wazuh rules file  /var/ossec/etc/rules/local_rules.xml

Regards.

[Awwal Ishiaku]

In addition to this, the log test utility will not work for that rule because the rule will only trigger if the log comes direction from the webhook. 

You can confirm this in rule 110002 that has the following content " <location>k8s</location>".
Note that you can change this condition if you wish.

[Phạm Q. Đạt]

Hi

Status of wazuh-webhook:

I also attached 2 configuration files below. Please examine it

Thank you.

[Phạm Q. Đạt]

Hi,

I solved the issue by editing the ruleset.
the original one: 

<group name="k8s_audit,">

<rule id="110002" level="0">

<location>k8s</location>

<field name="apiVersion">audit</field>

<description>Kubernetes audit log.</description>

</rule>

<rule id="110003" level="5">

<if_sid>110002</if_sid>

<regex type="pcre2">requestURI\":.+", \"verb\": \"create</regex>

<description>Kubernetes request to create resource</description>

</rule>

<rule id="110004" level="5">

<if_sid>110002</if_sid>

<regex type="pcre2">requestURI\":.+", \"verb\": \"delete</regex>

<description>Kubernetes request to delete resource</description>

</rule>

</group>

change this tag

<field name="apiVersion">audit</field> 

to 

<field name="apiVersion">audit\.</field>

So It will match the log for example

{...."apiVersion": "audit.k8s.io/v1",....}

Hope it helps someone out there.