Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Typo in Google Kubernetes Engine ip range results in unusable terraform state

208 views
Skip to first unread message

Andrej

unread,
Apr 28, 2021, 5:31:07 AM4/28/21
to Terraform
Hello, I encountered couple of problems while trying to create a new google kubernetes cluster using the google terraform provider.
The last problem left me unable to make any changes to my gcp project.

First I tried to create a cluster using terraform but I hit a quota on CPUs. I requested a raise for the quota. After it got raised I tried to create the cluster but the process of creating cluster got stuck. Terraform was waiting for GCP to create the cluster. In cloud console I saw that the cluster was being created.
After 30 minutes of waiting for cluster I decided to stop the terraform. Few minutes later I saw that the gcp failed to create the cluster. The error message was:
"Retry budget exhausted (80 attempts): Secondary range "gitlab-runners-cluster-service-cidr" does not exist in network "https://www.googleapis.com/compute/v1/projects/the-project-id/global/networks/standard-vpc", subnetwork "gitlab-runners-cluster-subnet"."

It failed because of a typo in the ip_range_services. I missed the second "s" in the word "services" in "gitlab-runners-cluster-service-cidr" so it should have been "gitlab-runners-cluster-services-cidr".

This probably should have been caught by terraform validation or plan but was not.
Anyway, I deleted the failed cluster (in gcp console ... oops?) and tried again to create the cluster. This resulted in following error message: 

Error: Error waiting for resuming GKE cluster: Retry budget exhausted (80 attempts): Secondary range "gitlab-runners-cluster-service-cidr" does not exist in network "https://www.googleapis.com/compute/v1/projects/the-project-id/global/networks/standard-vpc", subnetwork "gitlab-runners-cluster-subnet".

Even after removing the cluster from the terraform code I still can't do any changes to my infrastructure and I'm left with the same message.

I'm using the "terraform-google-modules/kubernetes-engine/google//modules/private-cluster" module.

Does anyone have any idea what I did wrong and how can I fix this ?
Thanks a lot.

Andrej

unread,
Apr 28, 2021, 10:21:38 AM4/28/21
to Terraform
The problem was broken terraform state.
It included the cluster that was stuck at creating.
I fixed it by running "terraform state rm module.gke_gitlab-runners"

Dátum: streda 28. apríla 2021, čas: 11:31:07 UTC+2, odosielateľ: Andrej
Reply all
Reply to author
Forward
0 new messages