Unable to use Kubernetes Engine (beta)

[Jeroen Op 't Eynde]

Hi all

I've enabled the 'Stackdriver Kubernetes Engine Monitoring'  and disabled the legacy options on a GKE cluster, hoping this would enable the new beta.

After enabling, (I waited a bit and) I checked the Kubernetes Engine beta under Resources, but no dice, it still says 'No nodes in this cluster.'.

The 'Kubernetes migration status' page under settings says there is nothing to migrate.

I've also upgraded the master and nodes to '1.13.7-gke.8' (latest available atm), still nothing.

Am I missing something obvious here? 

Cheers

Jeroen

[Mary Koes]

+Ruxanda Danetiu 

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/42ed1f38-4050-4b89-8ff1-806d8dac6e16%40googlegroups.com.

[Igor Peshansky]

Have you followed the troubleshooting steps at https://cloud.google.com/monitoring/kubernetes-engine/observing#troubleshooting? Can you please check whether there are agents (heapster, fluentd-gcp, metadata-agent) present in the kube-system namespace in your cluster? Can you check the agent logs via "kubectl logs" to see if they are reporting errors communicating with the respective APIs?

Note that we can't really offer anything more than general troubleshooting steps on this list. If you have a Cloud Support contract, opening a support case is the best way to ensure that your case gets a full in-context investigation and tracking.

        Igor

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAJoTiF-Z-XPBfQEjDcnNjSMRw_PnKN%3Drg6T8uqhqKj8Gz-13Mg%40mail.gmail.com.

[Jeroen Op 't Eynde]

Thanks, I missed the Troubleshooting chapter somehow.

It looks like the issue is with the permissions, heapster logs this, the others report similar permission errors:

Error while sending request to Stackdriver rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist).

I'd like to answer 'yes' to "Does your cluster's service account have permission to write data into Stackdriver?". However, I didn't configure a 'service account' to access Stackdriver and I can't find it back where I should have configured it.

Is it part of the OAuth scopes maybe?

Is it the 'Kubernetes Engine Service Agent' managed by Google? Adding 'Monitoring Metric Writer' to this role didn't resolve above error.

Shouldn't 'enabling' the new metrics on the GKE cluster "automatically" configure it correctly?

Thanks for the quick feedback already, I don't mind getting general troubleshooting steps, it's not an urgent task on my end.

Jeroen

On Monday, July 29, 2019 at 4:14:19 PM UTC+2, Igor Peshansky wrote:

Have you followed the troubleshooting steps at https://cloud.google.com/monitoring/kubernetes-engine/observing#troubleshooting? Can you please check whether there are agents (heapster, fluentd-gcp, metadata-agent) present in the kube-system namespace in your cluster? Can you check the agent logs via "kubectl logs" to see if they are reporting errors communicating with the respective APIs?

Note that we can't really offer anything more than general troubleshooting steps on this list. If you have a Cloud Support contract, opening a support case is the best way to ensure that your case gets a full in-context investigation and tracking.

        Igor

On Mon, Jul 29, 2019 at 9:43 AM 'Mary Koes' via Google Stackdriver Discussion Forum <google-stackdriver-discu...@googlegroups.com> wrote:

+Ruxanda Danetiu 

On Mon, Jul 29, 2019, 9:42 AM Jeroen Op 't Eynde <jer...@travis-ci.org> wrote:

Hi all

I've enabled the 'Stackdriver Kubernetes Engine Monitoring'  and disabled the legacy options on a GKE cluster, hoping this would enable the new beta.

After enabling, (I waited a bit and) I checked the Kubernetes Engine beta under Resources, but no dice, it still says 'No nodes in this cluster.'.

The 'Kubernetes migration status' page under settings says there is nothing to migrate.

I've also upgraded the master and nodes to '1.13.7-gke.8' (latest available atm), still nothing.

Am I missing something obvious here? 

Cheers

Jeroen

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/42ed1f38-4050-4b89-8ff1-806d8dac6e16%40googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[Igor Peshansky]

You need to make sure that the service account(s) used by heapster, fluentd-gcp, and metadata-agent all have the respective permissions. Generally, on GKE, the agents use the node's (GCE default) service account, so each node must have the appropriate OAuth scopes (logging.write and monitoring.write) so that the service account token allows performing those operations. You should also double-check that the relevant APIs are all enabled in the project, though you'd get a different error if they were disabled. And yes, enabling monitoring on the GKE cluster should have set this up correctly, but there are always corner cases, so I'd double-check.

As far as I can see, "Kubernetes Engine Service Account" is something managed by Google to enable the connection between Google infrastructure and your project, and shouldn't be used as an identity for any resources running in the project (see https://cloud.google.com/kubernetes-engine/docs/troubleshooting#restore_service_account).

        Igor

On Mon, Jul 29, 2019 at 11:37 AM Jeroen Op 't Eynde <jer...@travis-ci.org> wrote:

Thanks, I missed the Troubleshooting chapter somehow.

It looks like the issue is with the permissions, heapster logs this, the others report similar permission errors:

Error while sending request to Stackdriver rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist).

I'd like to answer 'yes' to "Does your cluster's service account have permission to write data into Stackdriver?". However, I didn't configure a 'service account' to access Stackdriver and I can't find it back where I should have configured it.

Is it part of the OAuth scopes maybe?

Is it the 'Kubernetes Engine Service Agent' managed by Google? Adding 'Monitoring Metric Writer' to this role didn't resolve above error.

Shouldn't 'enabling' the new metrics on the GKE cluster "automatically" configure it correctly?

Thanks for the quick feedback already, I don't mind getting general troubleshooting steps, it's not an urgent task on my end.

Jeroen

On Monday, July 29, 2019 at 4:14:19 PM UTC+2, Igor Peshansky wrote:

Have you followed the troubleshooting steps at https://cloud.google.com/monitoring/kubernetes-engine/observing#troubleshooting? Can you please check whether there are agents (heapster, fluentd-gcp, metadata-agent) present in the kube-system namespace in your cluster? Can you check the agent logs via "kubectl logs" to see if they are reporting errors communicating with the respective APIs?

Note that we can't really offer anything more than general troubleshooting steps on this list. If you have a Cloud Support contract, opening a support case is the best way to ensure that your case gets a full in-context investigation and tracking.

        Igor

On Mon, Jul 29, 2019 at 9:43 AM 'Mary Koes' via Google Stackdriver Discussion Forum <google-stackdr...@googlegroups.com> wrote:

+Ruxanda Danetiu 

On Mon, Jul 29, 2019, 9:42 AM Jeroen Op 't Eynde <jer...@travis-ci.org> wrote:

Hi all

I've enabled the 'Stackdriver Kubernetes Engine Monitoring'  and disabled the legacy options on a GKE cluster, hoping this would enable the new beta.

After enabling, (I waited a bit and) I checked the Kubernetes Engine beta under Resources, but no dice, it still says 'No nodes in this cluster.'.

The 'Kubernetes migration status' page under settings says there is nothing to migrate.

I've also upgraded the master and nodes to '1.13.7-gke.8' (latest available atm), still nothing.

Am I missing something obvious here? 

Cheers

Jeroen

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/42ed1f38-4050-4b89-8ff1-806d8dac6e16%40googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAJoTiF-Z-XPBfQEjDcnNjSMRw_PnKN%3Drg6T8uqhqKj8Gz-13Mg%40mail.gmail.com.

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/9b18a768-8bb0-4f87-b431-dd5cbd2d44ef%40googlegroups.com.

[Jeroen Op 't Eynde]

Hi Igor

I have checked all steps between (for example) Heapster and the actual VM. As I haven't configured anything manually for Heapster (or other processes on kube-system), I think I can safely assume it boils down to the node's service account. 

I went from Workloads->Heapster Deployment->Heapster Pod->K8s Node->GCE VM:

Stackdriver Logging API

Write Only

Stackdriver Monitoring API

Full

I've also checked the APIs, they all are enabled.

I guess I'm stuck here now, the cluster configuration is written in Terraform and can be found here: https://github.com/travis-ci/terraform-config/compare/duo_enable_stackdriver_mon

Thanks for the help, I'm sure I'm missing something stupid.

Jeroen

On Monday, July 29, 2019 at 6:21:24 PM UTC+2, Igor Peshansky wrote:

You need to make sure that the service account(s) used by heapster, fluentd-gcp, and metadata-agent all have the respective permissions. Generally, on GKE, the agents use the node's (GCE default) service account, so each node must have the appropriate OAuth scopes (logging.write and monitoring.write) so that the service account token allows performing those operations. You should also double-check that the relevant APIs are all enabled in the project, though you'd get a different error if they were disabled. And yes, enabling monitoring on the GKE cluster should have set this up correctly, but there are always corner cases, so I'd double-check.

As far as I can see, "Kubernetes Engine Service Account" is something managed by Google to enable the connection between Google infrastructure and your project, and shouldn't be used as an identity for any resources running in the project (see https://cloud.google.com/kubernetes-engine/docs/troubleshooting#restore_service_account).

        Igor

On Mon, Jul 29, 2019 at 11:37 AM Jeroen Op 't Eynde <jer...@travis-ci.org> wrote:

Thanks, I missed the Troubleshooting chapter somehow.

It looks like the issue is with the permissions, heapster logs this, the others report similar permission errors:

Error while sending request to Stackdriver rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist).

I'd like to answer 'yes' to "Does your cluster's service account have permission to write data into Stackdriver?". However, I didn't configure a 'service account' to access Stackdriver and I can't find it back where I should have configured it.

Is it part of the OAuth scopes maybe?

Is it the 'Kubernetes Engine Service Agent' managed by Google? Adding 'Monitoring Metric Writer' to this role didn't resolve above error.

Shouldn't 'enabling' the new metrics on the GKE cluster "automatically" configure it correctly?

Thanks for the quick feedback already, I don't mind getting general troubleshooting steps, it's not an urgent task on my end.

Jeroen

On Monday, July 29, 2019 at 4:14:19 PM UTC+2, Igor Peshansky wrote:

Have you followed the troubleshooting steps at https://cloud.google.com/monitoring/kubernetes-engine/observing#troubleshooting? Can you please check whether there are agents (heapster, fluentd-gcp, metadata-agent) present in the kube-system namespace in your cluster? Can you check the agent logs via "kubectl logs" to see if they are reporting errors communicating with the respective APIs?

Note that we can't really offer anything more than general troubleshooting steps on this list. If you have a Cloud Support contract, opening a support case is the best way to ensure that your case gets a full in-context investigation and tracking.

        Igor

On Mon, Jul 29, 2019 at 9:43 AM 'Mary Koes' via Google Stackdriver Discussion Forum <google-stackdriver-discu...@googlegroups.com> wrote:

+Ruxanda Danetiu 

On Mon, Jul 29, 2019, 9:42 AM Jeroen Op 't Eynde <jer...@travis-ci.org> wrote:

Hi all

I've enabled the 'Stackdriver Kubernetes Engine Monitoring'  and disabled the legacy options on a GKE cluster, hoping this would enable the new beta.

After enabling, (I waited a bit and) I checked the Kubernetes Engine beta under Resources, but no dice, it still says 'No nodes in this cluster.'.

The 'Kubernetes migration status' page under settings says there is nothing to migrate.

I've also upgraded the master and nodes to '1.13.7-gke.8' (latest available atm), still nothing.

Am I missing something obvious here? 

Cheers

Jeroen

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/42ed1f38-4050-4b89-8ff1-806d8dac6e16%40googlegroups.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/CAJoTiF-Z-XPBfQEjDcnNjSMRw_PnKN%3Drg6T8uqhqKj8Gz-13Mg%40mail.gmail.com.

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.