Typo in Google Kubernetes Engine ip range results in unusable terraform state

200 views

Skip to first unread message

Andrej

unread,

Apr 28, 2021, 5:31:07 AM4/28/21

to Terraform

Hello, I encountered couple of problems while trying to create a new google kubernetes cluster using the google terraform provider.

The last problem left me unable to make any changes to my gcp project.

First I tried to create a cluster using terraform but I hit a quota on CPUs. I requested a raise for the quota. After it got raised I tried to create the cluster but the process of creating cluster got stuck. Terraform was waiting for GCP to create the cluster. In cloud console I saw that the cluster was being created.

After 30 minutes of waiting for cluster I decided to stop the terraform. Few minutes later I saw that the gcp failed to create the cluster. The error message was:

"Retry budget exhausted (80 attempts): Secondary range "gitlab-runners-cluster-service-cidr" does not exist in network "https://www.googleapis.com/compute/v1/projects/the-project-id/global/networks/standard-vpc", subnetwork "gitlab-runners-cluster-subnet"."

It failed because of a typo in the ip_range_services. I missed the second "s" in the word "services" in "gitlab-runners-cluster-service-cidr" so it should have been "gitlab-runners-cluster-services-cidr".

This probably should have been caught by terraform validation or plan but was not.

Anyway, I deleted the failed cluster (in gcp console ... oops?) and tried again to create the cluster. This resulted in following error message:

Error: Error waiting for resuming GKE cluster: Retry budget exhausted (80 attempts): Secondary range "gitlab-runners-cluster-service-cidr" does not exist in network "https://www.googleapis.com/compute/v1/projects/the-project-id/global/networks/standard-vpc", subnetwork "gitlab-runners-cluster-subnet".

Even after removing the cluster from the terraform code I still can't do any changes to my infrastructure and I'm left with the same message.

I'm using the "terraform-google-modules/kubernetes-engine/google//modules/private-cluster" module.

Does anyone have any idea what I did wrong and how can I fix this ?

Thanks a lot.

Andrej

unread,

Apr 28, 2021, 10:21:38 AM4/28/21

to Terraform

The problem was broken terraform state.
It included the cluster that was stuck at creating.
I fixed it by running "terraform state rm module.gke_gitlab-runners"