Endpoints on Cloud Run with Terraform

[Etienne Thiery]

Hi all,

The current setup for Endpoints on Cloud Run involves:

1. Deploying gcr.io/endpoints-release/endpoints-runtime-serverless:2 as a Cloud Run service.
2. Deploying the Endpoints configuration, which must includes the URL of the backend Cloud Run service, and the hostname of the Cloud Run service deployed in step 1.
3. Building the Endpoints service config into a new ESPv2 Beta docker image, using a provided script.
   
4. Redeploying the Cloud Run service deployed in step 1 with the new image built in step 3.

I have been trying to automate this setup using Terraform, unsuccessfuly so far.

Step 1 is straightforward using the "google_cloud_run_service" resource of the google terraform provider:

resource "google_cloud_run_service" "myservice_endpoints_first_deploy" {
  name     = "myservice-endpoints"

  template {
    spec {
      containers {
        image = "gcr.io/endpoints-release/endpoints-runtime-serverless:2"
      }
    }
  }

  depends_on = [
    google_project_service.cloud_run_api,
  ]
}

Step 2 is also simple enough using the "google_endpoints_service" resource of the google terraform provider. 

locals {
  backend_url = google_cloud_run_service.myservice.status[0].url  
  endpoints_host = trimprefix(google_cloud_run_service.myservice_endpoints_first_deploy.status[0].url, "https://")
}

resource "google_endpoints_service" "myservice" {
  service_name = local.endpoints_host
  openapi_config = replace(replace(file("${path.module}/swagger.yaml"),
    "BACKEND_ADDRESS", local.backend_url),
    "HOST", local.endpoints_host
  )

  depends_on = [
    google_project_service.endpoints_api,
    google_project_service.service_control_api,
    google_project_service.service_management_api,
  ]
}

Note: The snipet above assumes there is a swagger.yaml endpoints configuration template in the same folder, and uses the replace function to insert the backend Cloud Run service URL, and the endpoint Cloud Run service hostname. 

Step 3 is not so Terraform friendly. 

One solution is to use the local-exec provisioner to executes the provided "gcloud_build_image" script. But this requires the Google Cloud SDK to be installed, which makes applying this terraform config in CI/CD more complicated.

resource "null_resource" "espv2_image" {
  triggers = {
    endpoints_config_id = google_endpoints_service.myservice.config_id
  }

  provisioner "local-exec" {
    command = "${path.module}/gcloud_build_image -p ${google_endpoints_service.myservice.project} -s ${google_endpoints_service.myservice.service_name} -c ${google_endpoints_service.myservice.config_id}"
  }

  depends_on = [
    google_project_service.cloud_build_api,
    google_project_service.container_registry_api,
  ]
}


locals {
  espv2_image = "gcr.io/${google_endpoints_service.myservice.project}/endpoints-runtime-serverless:${google_endpoints_service.myservice.service_name}-${google_endpoints_service.myservice.config_id}"
}

Another solution could be to peak into the provided "gcloud_build_image" script, and reproduce its behaviour using Terraform. This script makes a request to the service management API to get the endpoints service config, and builds a docker image using Google Cloud Build. Unfortunately, the google terraform provider does not provide resources for these two actions.

Step 4 is the real deal breaker. One simply cannot redeploy the service deployed in step 1 as part of the same Terraform apply command. It should be fairly obvious why to anyone with a decent understanding how Terraform works. Just to be sure, you can try to add the following resource:

resource "google_cloud_run_service" "myservice_endpoints" {
  name     = "myservice-endpoints"
  template {
    spec {
      containers {
        image = local.espv2_image
      }
    }
  }

  depends_on = [
    null_resource.espv2_image,
  ]
}

Which, when applied, results in the following error:

"Error: Error creating Service: googleapi: Error 409: Resource 'myservice-endpoints' already exists."

I would be surprised if integrability with IaC tools like Terraform was not taken into account for the design of any GCP service, so could you share with us any plans you might have to solve this issue in the future? Even better, do you have any suggestion on how to mitigate the issue right now?

Best,

Etienne

[Etienne Thiery]

For the record, there is an open issue on the google terraform provider that describes a similar problem with Endpoints for Cloud Functions.

[Teju Nareddy]

For reference, it seems like you did find a workaround.

This deployment model was designed 3+ years ago, before Terraform was on our radar. We have never tested this out before.

I can put this on the backlog, but our team needs to spend some time understanding Terraform first.