fluentd not logging cluster name, instance ID and zone with metadata concealment and/or custom service accounts enabled

[daniel.ale...@gmail.com]

Hi,

Today, I:

- switched to a custom service account with reduced scopes for my node pool as per https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#use_least_privilege_service_accounts_for_your_nodes
- enabled metadata concealment as per https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#conceal_node_metadata_beta.

I have been and am still using Stackdriver Logging.

Since making the changes above, my container log fields only contain empty strings for the fields "resource.labels.cluster_name", "resource.labels.instance_id", and "resource.labels.zone". I suspect that my new service account doesn't have sufficient permissions to obtain these, and/or metadata concealment prevents fluentd from obtaining the information to fill into these fields.

Did anyone else experience this before and/or knows how to get these fields filled in again?

FYI, these are the roles configured for my node service account in IAM:

Stackdriver Debugger Agent
Stackdriver Profiler Agent
Cloud Trace Agent
Errors Writer
Logs Writer
Monitoring Metric Writer
Monitoring Viewer
Quota Viewer
Service Controller

Thanks,
Daniel

[Matt Brown]

From looking at the fluent-plugin-google-cloud source code, these labels come from the instance/attributes/kube-env field from the metadata-service - which metadata-concealment blocks.

https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/blob/cf936796cf7182209315db339bb2131f1b1ac159/lib/fluent/plugin/out_google_cloud.rb#L1133-L1139

[daniel.ale...@gmail.com]

Thanks Matt! I also noticed that a Pod's logs in Kubernetes Engine's "Workloads" area are also unavailable ever since enabling this; is it intentional that a feature Google suggests enabling (metadata concealment) blocks Kubernetes Engine features (Pod log viewing)?

[Matt Brown]

I can't speak to whether it is intentional or not. I would speculate though that the logs are not available because the link in the Console is sending you to a query in Stackdriver Logging using some of the labels that are no longer being set in the logs. I believe you can still find the logs using other queries in Stackdriver.