Errors retrieving pub/sup messages from GCP
Carlos Lopez
We have configured a pub/sub subscription to retrieve GKE audit logs in GCP. Only "pods.exec.create" and "pods.attach.create" messages arrive to Wazuh. In our GCP console we can see other messages like "pods.delete", "patch", "update", etc when we execute "Pull" option but these messages are not retrieved by Wazuh.
How can I debug/fix ths?
Best regards,
C. L. Martinez
Carlos Lopez
2023 Feb 16 08:26:13 aktwzhwork01->Wazuh-GCloud {"integration": "gcp", "gcp": {"insertId":"ba1ca230-9451-42a6-ae0f-c7771161cb81","labels":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"access granted by IAM permissions."},"logName":"projects/pro-01/logs/cloudaudit.googleapis.com%2Factivity","operation":{"first":true,"id":"ba1ca230-9451-42a6-ae0f-c7771161cb81","last":true,"producer":"k8s.io"},"protoPayload":{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"myu...@domain.com"},"authorizationInfo":[{"granted":true,"permission":"io.k8s.core.v1.pods.delete","resource":"core/v1/namespaces/default/pods/tmp-shell"}],"methodName":"io.k8s.core.v1.pods.delete","request":{"@type":"core.k8s.io/v1.DeleteOptions","apiVersion":"v1","kind":"DeleteOptions","propagationPolicy":"Background"},"requestMetadata":{"callerIp":"34.132.102.240","callerSuppliedUserAgent":"kubectl/v1.26.0 (linux/amd64) kubernetes/b46a3f8"},"resourceName":"core/v1/namespaces/default/pods/tmp-shell","response":{"@type":"core.k8s.io/v1.Pod","apiVersion":"v1","kind":"Pod","metadata":{"creationTimestamp":"2023-02-16T08:03:43Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2023-02-16T08:03:55Z","labels":{"run":"tmp-shell"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:run":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"tmp-shell\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:stdin":{},"f:stdinOnce":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:tty":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}},"manager":"kubectl-run","operation":"Update","time":"2023-02-16T08:03:43Z"},{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.60.33.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"manager":"kubelet","operation":"Update","subresource":"status","time":"2023-02-16T08:03:53Z"}],"name":"tmp-shell","namespace":"default","resourceVersion":"73139912","uid":"e9b2402a-ec31-4903-b789-f9af390e152a"},"spec":{"containers":[{"args":["/bin/bash"],"image":"ubuntu","imagePullPolicy":"Always","name":"tmp-shell","resources":{},"stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true,"volumeMounts":[{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","name":"kube-api-access-hlk2h","readOnly":true}]}],"dnsPolicy":"ClusterFirst","enableServiceLinks":true,"nodeName":"gke-pro-01-default-df76098b-o45m","preemptionPolicy":"PreemptLowerPriority","priority":0,"restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30,"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable","operator":"Exists","tolerationSeconds":300}],"volumes":[{"name":"kube-api-access-hlk2h","projected":{"defaultMode":420,"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"items":[{"key":"ca.crt","path":"ca.crt"}],"name":"kube-root-ca.crt"}},{"downwardAPI":{"items":[{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"},"path":"namespace"}]}}]}}]},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:43Z","reason":"PodCompleted","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:53Z","reason":"PodCompleted","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:53Z","reason":"PodCompleted","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:43Z","status":"True","type":"PodScheduled"}],"containerStatuses":[{"containerID":"containerd://9156e75cbb69a96dd268684f665c7a1fadcd807e3822922bd2e46a253289722d","image":"docker.io/library/ubuntu:latest","imageID":"docker.io/library/ubuntu@sha256:9a0bdde4188b896a372804be2384015e90e3f84906b750c1a53539b585fbbe7f","lastState":{},"name":"tmp-shell","ready":false,"restartCount":0,"started":false,"state":{"terminated":{"containerID":"containerd://9156e75cbb69a96dd268684f665c7a1fadcd807e3822922bd2e46a253289722d","exitCode":0,"finishedAt":"2023-02-16T08:03:53Z","reason":"Completed","startedAt":"2023-02-16T08:03:44Z"}}}],"hostIP":"10.40.10.193","phase":"Succeeded","podIP":"10.60.33.172","podIPs":[{"ip":"10.60.33.172"}],"qosClass":"BestEffort","startTime":"2023-02-16T08:03:43Z"}},"serviceName":"k8s.io","status":{}},"receiveTimestamp":"2023-02-16T08:04:03.115480444Z","resource":{"labels":{"cluster_name":"pro-01","location":"us-central1-a","project_id":"pro-01"},"type":"k8s_cluster"},"timestamp":"2023-02-16T08:03:55.821754Z"}}
But these logs don't arrive to Elastic. Filebeat returns several errors like:
2023-02-15T14:24:11.227Z WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event
with GKE and GCP logs .... How can I fix these?
Best regards,
C. L. Martinez
________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of Carlos Lopez <clo...@outlook.com>
Sent: 16 February 2023 08:23
To: wa...@googlegroups.com
Subject: Errors retrieving pub/sup messages from GCP
Good morning,
--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/PRAP251MB05679FC8090F4B11E34572D4DBA09%40PRAP251MB0567.EURP251.PROD.OUTLOOK.COM.
Nicolas Agustin Guevara Pihen
I will kindly ask for some information to help me identify the problem. Please make sure that no sensitive data is sent in the information that you provide (credentials, tokens, etc.).
- What version of Wazuh are you using? Is it running with Wazuh Indexer and Wazuh dashboard, or with Elasticsearch and Kibana?
- Could you share with me some of the Filebeat logs that contain the Warnings? You can also share the complete file if you prefer, just make sure that no sensitive data is there.
- Could you share examples of the logs that you are sending to Wazuh and not seeing in the web interface?
- Is there any custom rule or decoder made for these logs?
Carlos Lopez
You are right: some (most of them in fact) logs are not shown in Elastic and in fact problem seems with filebeat that returns errors like this:
{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"field name cannot contain only dots"}}, dropping event!
which is correct. As you can see in my previous log, GCP stores a lot of logs with "." in several fields. Maybe is it possible to remove all fields that comes with "."?
I am using Wazuh 4.3.10 with ELK 7.17.6 with default decoders and rules.
Best regards,
C. L. Martinez
________________________________________
From: 'Nicolas Agustin Guevara Pihen' via Wazuh mailing list <wa...@googlegroups.com>
Sent: 16 February 2023 12:18
To: Wazuh mailing list
Subject: Re: Errors retrieving pub/sup messages from GCP
Hello Carlos, thank you for using Wazuh. I will be helping you with this issue.
I understand that the logs are being sent to Wazuh, which apparently receives all of them but just some are shown in Elastic. Additionally, there are some errors in Filebeat. Is that correct?
I will kindly ask for some information to help me identify the problem. Please make sure that no sensitive data is sent in the information that you provide (credentials, tokens, etc.).
* What version of Wazuh are you using? Is it running with Wazuh Indexer and Wazuh dashboard, or with Elasticsearch and Kibana?
* Could you share with me some of the Filebeat logs that contain the Warnings? You can also share the complete file if you prefer, just make sure that no sensitive data is there.
* Could you share examples of the logs that you are sending to Wazuh and not seeing in the web interface?
* Is there any custom rule or decoder made for these logs?
I will be looking forward to your answer.
Kind regards,
On Thursday, February 16, 2023 at 5:36:33 AM UTC-3 Carlos Lopez wrote:
More info. Wazuh retrieves all pub/sub logs. for example:
2023 Feb 16 08:26:13 aktwzhwork01->Wazuh-GCloud {"integration": "gcp", "gcp": {"insertId":"ba1ca230-9451-42a6-ae0f-c7771161cb81","labels":{"authorization.k8s.io/decision<http://authorization.k8s.io/decision>":"allow","authorization.k8s.io/reason<http://authorization.k8s.io/reason>":"access granted by IAM permissions."},"logName":"projects/pro-01/logs/cloudaudit.googleapis.com<http://cloudaudit.googleapis.com>%2Factivity","operation":{"first":true,"id":"ba1ca230-9451-42a6-ae0f-c7771161cb81","last":true,"producer":"k8s.io<http://k8s.io>"},"protoPayload":{"@type":"type.googleapis.com/google.cloud.audit.AuditLog<http://type.googleapis.com/google.cloud.audit.AuditLog>","authenticationInfo":{"principalEmail":"myu...@domain.com"},"authorizationInfo":[{"granted":true,"permission":"io.k8s.core.v1.pods.delete","resource":"core/v1/namespaces/default/pods/tmp-shell"}],"methodName":"io.k8s.core.v1.pods.delete","request":{"@type":"core.k8s.io/v1.DeleteOptions<http://core.k8s.io/v1.DeleteOptions>","apiVersion":"v1","kind":"DeleteOptions","propagationPolicy":"Background"},"requestMetadata":{"callerIp":"34.132.102.240","callerSuppliedUserAgent":"kubectl/v1.26.0 (linux/amd64) kubernetes/b46a3f8"},"resourceName":"core/v1/namespaces/default/pods/tmp-shell","response":{"@type":"core.k8s.io/v1.Pod<http://core.k8s.io/v1.Pod>","apiVersion":"v1","kind":"Pod","metadata":{"creationTimestamp":"2023-02-16T08:03:43Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2023-02-16T08:03:55Z","labels":{"run":"tmp-shell"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:run":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"tmp-shell\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:stdin":{},"f:stdinOnce":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:tty":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}},"manager":"kubectl-run","operation":"Update","time":"2023-02-16T08:03:43Z"},{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.60.33.172\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"manager":"kubelet","operation":"Update","subresource":"status","time":"2023-02-16T08:03:53Z"}],"name":"tmp-shell","namespace":"default","resourceVersion":"73139912","uid":"e9b2402a-ec31-4903-b789-f9af390e152a"},"spec":{"containers":[{"args":["/bin/bash"],"image":"ubuntu","imagePullPolicy":"Always","name":"tmp-shell","resources":{},"stdin":true,"stdinOnce":true,"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","tty":true,"volumeMounts":[{"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount<http://kubernetes.io/serviceaccount>","name":"kube-api-access-hlk2h","readOnly":true}]}],"dnsPolicy":"ClusterFirst","enableServiceLinks":true,"nodeName":"gke-pro-01-default-df76098b-o45m","preemptionPolicy":"PreemptLowerPriority","priority":0,"restartPolicy":"Never","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"default","serviceAccountName":"default","terminationGracePeriodSeconds":30,"tolerations":[{"effect":"NoExecute","key":"node.kubernetes.io/not-ready<http://node.kubernetes.io/not-ready>","operator":"Exists","tolerationSeconds":300},{"effect":"NoExecute","key":"node.kubernetes.io/unreachable<http://node.kubernetes.io/unreachable>","operator":"Exists","tolerationSeconds":300}],"volumes":[{"name":"kube-api-access-hlk2h","projected":{"defaultMode":420,"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"items":[{"key":"ca.crt","path":"ca.crt"}],"name":"kube-root-ca.crt"}},{"downwardAPI":{"items":[{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"},"path":"namespace"}]}}]}}]},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:43Z","reason":"PodCompleted","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:53Z","reason":"PodCompleted","status":"False","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:53Z","reason":"PodCompleted","status":"False","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2023-02-16T08:03:43Z","status":"True","type":"PodScheduled"}],"containerStatuses":[{"containerID":"containerd://9156e75cbb69a96dd268684f665c7a1fadcd807e3822922bd2e46a253289722d","image":"docker.io/library/ubuntu:latest<http://docker.io/library/ubuntu:latest>","imageID":"docker.io/library/ubuntu@sha256:9a0bdde4188b896a372804be2384015e90e3f84906b750c1a53539b585fbbe7f<http://docker.io/library/ubuntu@sha256:9a0bdde4188b896a372804be2384015e90e3f84906b750c1a53539b585fbbe7f>","lastState":{},"name":"tmp-shell","ready":false,"restartCount":0,"started":false,"state":{"terminated":{"containerID":"containerd://9156e75cbb69a96dd268684f665c7a1fadcd807e3822922bd2e46a253289722d","exitCode":0,"finishedAt":"2023-02-16T08:03:53Z","reason":"Completed","startedAt":"2023-02-16T08:03:44Z"}}}],"hostIP":"10.40.10.193","phase":"Succeeded","podIP":"10.60.33.172","podIPs":[{"ip":"10.60.33.172"}],"qosClass":"BestEffort","startTime":"2023-02-16T08:03:43Z"}},"serviceName":"k8s.io<http://k8s.io>","status":{}},"receiveTimestamp":"2023-02-16T08:04:03.115480444Z","resource":{"labels":{"cluster_name":"pro-01","location":"us-central1-a","project_id":"pro-01"},"type":"k8s_cluster"},"timestamp":"2023-02-16T08:03:55.821754Z"}}
But these logs don't arrive to Elastic. Filebeat returns several errors like:
2023-02-15T14:24:11.227Z WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event
with GKE and GCP logs .... How can I fix these?
Best regards,
C. L. Martinez
________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of Carlos Lopez <clo...@outlook.com>
Sent: 16 February 2023 08:23
To: wa...@googlegroups.com
Subject: Errors retrieving pub/sup messages from GCP
Good morning,
We have configured a pub/sub subscription to retrieve GKE audit logs in GCP. Only "pods.exec.create" and "pods.attach.create" messages arrive to Wazuh. In our GCP console we can see other messages like "pods.delete", "patch", "update", etc when we execute "Pull" option but these messages are not retrieved by Wazuh.
How can I debug/fix ths?
Best regards,
C. L. Martinez
--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/PRAP251MB05679FC8090F4B11E34572D4DBA09%40PRAP251MB0567.EURP251.PROD.OUTLOOK.COM.
--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com<mailto:wazuh+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fb2a2b75-9b6e-409f-9dd9-c31172ee466fn%40googlegroups.com<https://groups.google.com/d/msgid/wazuh/fb2a2b75-9b6e-409f-9dd9-c31172ee466fn%40googlegroups.com?utm_medium=email&utm_source=footer>.
Nicolas Agustin Guevara Pihen
As you pointed out, it is failing because some fields contain "." as a name.
The best option if it was possible is to modify the logs at the origin and rename or delete those fields to prevent the error from happening.
If that is not possible, it may be possible to modify the fields using Filebeat processors in the ingest pipeline. If you can share with me some of the GCP logs that should be in /var/ossec/logs/alerts/alerts.json, I can make some tests with the processors in order to try solving this issue.
I will be looking forward to your answer!
Regards,