403 Forbidden on ESPv2, GKE AutoPilot, WIF

Tom Klimovski

unread,

Jan 10, 2022, 3:55:49 AM1/10/22

to Google Cloud Endpoints

Hi all,

I'm following the Getting started with Endpoints for GKE with ESPv2 and I've been running into the error:

`F0110 03:46:24.304229 8 server.go:54] fail to initialize config manager: http call to GET https://servicemanagement.googleapis.com/v1/services/name:bookstore.endpoints.<project>.cloud.goog/rollouts?filter=status=SUCCESS returns not 200 OK: 403 Forbidden`

Which ultimately leads to a transport failure error and shut down of the Pod.

My first step was to investigate permission issues, but I could really use some outside perspective on this as I've been going around in circles on this.

Here's my config:

>> gcloud container clusters describe $GKE_CLUSTER_NAME \
--zone=$GKE_CLUSTER_ZONE \
--format='value[delimiter="\n"](nodePools[].config.oauthScopes)'

>> gcloud container clusters describe $GKE_CLUSTER_NAME \
--zone=$GKE_CLUSTER_ZONE \
--format='value[delimiter="\n"](nodePools[].config.serviceAccount)'
default
default

----

Service-Account-Name: test-espv2

Roles

Cloud Trace Agent

Owner

Service Account Token Creator

Service Account User

Service Controller

Workload Identity User

---

I've associated the WIF svc-act with the Cluster with the following yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: test-espv2@<project>.iam.gserviceaccount.com
  name: test-espv2
  namespace: eventing
---

Since the `gcr.io/endpoints-release/endpoints-runtime:2` is limited,

I created a test container and deployed it into the same `eventing` namespace.

Within the container, I'm able to retrieve the endpoint service config with the following command:

curl --fail -o "service.json" -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://servicemanagement.googleapis.com/v1/services/${SERVICE}/configs/${CONFIG_ID}?view=FULL"

And also within the container, I'm running as the impersonated service account, tested with:

`curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/`

Are there any other tests I can run to help me debug this issue?

Thanks in advance,

Xuyang(Jason) Tao

unread,

Jan 10, 2022, 1:34:22 PM1/10/22

to Tom Klimovski, Google Cloud Endpoints

Hi Tom,

Did you use the same k8s yaml config for your test container(replacing ESPv2 container), where you were able to get the service config and correct SA successfully? If yes, I suppose the ESPv2 container should be able to get the config also.

Did you put the KSA under containers section in your yaml also, like

apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: K8S_NAMESPACE
spec:
  containers:
  - image: IMAGE_NAME
    name: ....
    command: ....
  serviceAccountName: KSA_NAME ?

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/84b7080b-6bdd-436b-a468-dde0bf55e2c2n%40googlegroups.com.

--

Xuyang(Jason) Tao

tao...@google.com

Service Infrastructure

Software Engineer

Chen Levi Elenberg

unread,

Jan 10, 2022, 5:57:42 PM1/10/22

to Xuyang(Jason) Tao, Google Cloud Endpoints, Tom Klimovski

Based on the error you pasted I believe your service account does not have service config role on IAM which you will need to add.

Are you using workload identity on this autopilot cluster?

If you do you won’t be able to use the default compue account and you need to specifically create service account and bind it with the needed perm

בתאריך יום ב׳, 10 בינו׳ 2022 ב-20:36 מאת 'Xuyang(Jason) Tao' via Google Cloud Endpoints <google-clou...@googlegroups.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAL6CxQ69rxBsSKqP0yiOrEgyFa4i0EVUmatxux0%3DBO9oX5T9uQ%40mail.gmail.com.

Tom Klimovski

unread,

Jan 10, 2022, 5:57:45 PM1/10/22

to Google Cloud Endpoints

Hey Jason,

That's a good point; yes I've done that. Sample below

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: esp-grpc-bookstore
  namespace: eventing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: esp-grpc-bookstore
  template:
    metadata:
      labels:
        app: esp-grpc-bookstore
    spec:
      serviceAccountName: test-espv2

Tom Klimovski

unread,

Feb 8, 2022, 10:42:47 PM2/8/22

to Google Cloud Endpoints

I've finally figured out the issue. It was in 2 parts.

1. Redeployment of app, paying special attention and verification of the `kubectl annotate serviceaccount` commands

- add-iam-policy-binding for both serviceController and cloudtrace.agent

- omitting `nodeSelector: iam.gke.io/gke-metadata-server-enabled: "true" ` due to Autopilot

Doing this enabled a successful kube deployment as displayed by the logs.

Next error I had was

```

<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>

```

2. This was fixed by turning my attention back to my Kube cluster.

Looking through the events in my ingress service, since I was in a shared-vpc and my security policies only allowed firewall management from the host project, the deployment was failing to update the firewall rules.

Manually provisioning them, as shown here :

https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#manually_provision_firewall_rules_from_the_host_project

solved my issues.

Reply all

Reply to author

Forward