gRPC Cloudrun as backend | IAM

[Jan Krynauw]

Hi

We have the following setup:

• a gRPC backend running on Google Cloudrun
• developed in Go
• we use IAM to manage access to the Cloud Run instance - i.e. only users with the following role on the particular Cloud Run instance will have access: roles/run.invoker.
• We therefore control access on a per-service basis, using use per-service IAM
• Any client therefore has to use identity tokens when calling the Cloud Run (fully managed) instance.
  

We are trying to 'wrap' this behind and ESPv2 / Cloud Endpoints, so that we'll get the additional benefits for HTTP transcoding, documentation and particularly using the 'Try this API' feature.

• As per the documentation we have deployed our ESPv2 on Cloudrun, with public access (allUsers have roles/run.invoker on the ESP)
• In the Service Specification we have our service private Cloud Run as the backend.

Looking at the service configuration: https://cloud.google.com/endpoints/docs/grpc-service-config/reference/rpc/google.api#service

It is not clear how we should setup the backend together with Authentication in the service config.  Here is an outline of what we currently have:

The backend Cloud Run is responsible for checking access and the thinking with the 'disable_auth' is that the headers sent to the ESPv2 Cloudrun should be passed onto the Backend Cloud Run, which then authenticates using IAM (i.e. via roles/run.invoker).

How would one setup the above and be able to use the 'Try this API' feature?

[Xuyang(Jason) Tao]

Hi Jan, 

There are two separate JWT tokens existing in the data path of Cloud Endpoints: 

• client(attach a JWT) -> ESPv2(verify the JWT sent from client, jwtAuthn)
• ESPv2(attach a new JWT based on the service config) -> Cloud Run(verify the second JWT)

The `authentication` section is responsible for verifying JWT from clients while the `backend` section is responsible for generating a new JWT for backend. 

 As you want Cloud Run access to be protected, `disable_auth` shouldn't be set and the backend authn will be auto-enabled by using `address` field as audience,

BTW, the `audience` in your `authentication` section should be your public Cloud Endpoints address instead of the real protected Cloud Run address.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Endpoints" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-endp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/276be5f0-ab73-45cb-93d1-8e8001de91aan%40googlegroups.com.

--

Xuyang(Jason) Tao tao...@google.com Service Infrastructure Software Engineer

[Wayne Zhang]

1) Do you need to use jwt in the client (calling ESP),  to ask ESP to verify it,  if no, you can remove the "authentication" section. 

2) after you remove "disable_auth" in the `Backend` rules section,  ESP will attach the ID token from the service account deploying the ESPv2 Cloud Run service, send it to your grpc backend.  Your grpc backand IAM will verify it.  You need to make sure that service account has the roles/run.invoker permission to your grpc backend.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CAL6CxQ7wVzHkLsvYGHsytd1BEmzcNoq70%3Dt28vTd0ct9UNofTA%40mail.gmail.com.

[Jan Krynauw]

Thank you.

My understanding is that Access and Authentication(verifying identity) are two different things.  IAM provides both, the Access part is implemented through the role roles/run.invoker.

How does the Access part work with Cloud Endpoints? 

If I remove the "disable_auth" in the backend, any valid JWT sent by the client will be verified by the ESPv2, and since the backend receives a new token from the ESPv2 and has roles/run.invoker rights the request will always reach the backend.  There is therefore no Access restriction to the backend when one considers the full data path? The only requirement is that the request is authenticated, i.e. identified.

[Xuyang(Jason) Tao]

Hi Jan,

I understand you are talking about the authorization. Yes, ESP only does authentication based on JWT. As for the authorization, achieve that by 

 api key. Here are docs about it.

https://cloud.google.com/endpoints/docs/openapi/restricting-api-access-with-api-keys

https://cloud.google.com/endpoints/docs/openapi/when-why-api-key

They are required by default if you don't set allowUnregisteredCall.

[Jan Krynauw]

Makes sense thank you.

As we are using IAM for the Authorisation to our current cloudrun backends, the only way to also use IAM for Authorisation with Cloud Endpoints would be to lock down the ESPv2 Cloud Run?  i.e. clients without roles/run.invoker rights to the ESPv2 Cloud Run instance won't even reach the proxy.

Managing access via IAM avoids clients having to maintain additional keys.

[Teju Nareddy]

Yes, you can lock down the ESPv2 Cloud Run service to use IAM for Authorization, allowing only select users to call ESPv2. You will then lockdown your backend to only allow ESPv2 to call it.

If you don't want to lock down ESPv2 with IAM, you can add some extra code to your backend to check the original JWT token sent from the client. ESPv2 will forward the original authorization header, documented here. Because ESPv2 has already verified the signature of the JWT, your backend can trust the claims and use them to allow/reject the requests.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CADeeoqrm1xoJdPOEBmC1ytCp8v4xrNzSqHentOGXWbUQvQ_ciA%40mail.gmail.com.

--

Teju Nareddy nare...@google.com Software Engineer

[Jan Krynauw]

Thank you Teju, this is all starting to make a lot of sense.

Extending your example, instead of locking down the ESPv2 Cloud Run Service, what if one make it public and then only lock down the backend Cloud Run such that only clients with roles/run.invoker role will be able to access it.  Now setting the `disable_auth` to true in the backend, won't the client headers sent to the proxy simply be passed onto the backend, and therefore be validated? https://cloud.google.com/endpoints/docs/grpc-service-config/reference/rpc/google.api#google.api.BackendRule

When disable_auth is true, a JWT ID token won't be generated and the original "Authorization" HTTP header will be preserved. If the header is used to carry the original token and is expected by the backend, this field must be set to true to preserve the header.

In this way one can continue to manage access using IAM at the relevant backend Cloud Run instances and then simply add on the ESPv2 for its documentation, try this API and HTTP transcoding benefits?

[Josh Einhorn]

Jan,

Your intuition about using identity-based authorization instead of API Key authorization is spot on. The standard solution in Cloud for identity-based authorization is IAP. You can set up IAP on a GCLB Backend Service, and you can route GCLB to your Cloud Run instance using a Serverless NEG. Hope that helps.

-Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CADeeoqpuAQY_OFDMtns9AObJ-%3Dcg6XPR4%3DLdbBcwVYrSmunvZw%40mail.gmail.com.

--

Josh Einhorn |

Software Engineer |

joshe...@google.com |

1-215-837-1102

[Jan Krynauw]

Thank you Josh.

With IAP the resource is IAP with required role to access the service: roles/iap.httpsResourceAccessor

With Cloud Run the resource is a Cloud Run instance with required role to access the service: roles/run.invoker

Is the latter not already using identity-based authorisation? Why not use the IAM with Cloud Run directly?

[Josh Einhorn]

Why not use the IAM with Cloud Run directly?

The primary reason is that it mixes concerns and decentralizes the access policy. Using Cloud Run IAM means that 

1. You can't ever change the entry point to your application without also re-doing all the permissions for all clients. Consider if/when you migrate to API Gateway, the run.invoker role will not work anymore (but IAP will still work via GCLB).
2. If you have more than one Cloud Run Service (e.g. multi-region), you need to replicate permissions across all of them or grant over-privileged access by binding permissions at the Project level. IAP centralizes control into a single place.
3. It creates a "leaky abstraction" to grant clients access to invoke your Cloud Run Service (which may be considered "infrastructure"), rather than to abstractly invoke your "HTTP resource."
   

And just curious... if you're using Cloud IAM to secure your resources and not using API Keys, are you just using ESPv2 for gRPC transcoding?

-Josh

[Jan Krynauw]

Thank you Josh!

That is correct, only really using ESPv2 for the gRPC transcoding and then ideally use the documentation and 'Try this API' feature on Cloud Endpoints.

Our Cloud Runs instances are mainly used as endpoints between services.  For example:

• Service A, Service B, Service C and Service X are all deployed on their own Cloud Run instances,
• Each with their own respective Service accounts: service-a@my-project.iam.gserviceaccount.com, service-b@my-project.iam.gserviceaccount.com, service-c@my-project.iam.gserviceaccount.com, service-x@my-project.iam.gserviceaccount.com
• We are using the Principle of Least Privilege with these service accounts. 
• Let say only Services A and B are allowed to invoke Service X,  Service C is not allowed to invoke Service X
• Looking at the Permissions on the Service X cloud run instance, only service-a@my-project.iam.gserviceaccount.com and service-b@my-project.iam.gserviceaccount.com will have 'roles/run.invoker' permissions set.
• In this way we use IAM to tightly control access / authorisation to Service X.
• This works nicely with the Application Default Credentials flow within Services A,B and C which calls a standard Google API to get the id_token before making a request to Service X.
• Not having to manage access within each backend is also (i.e. is this authenticated user able to perform this action etc.) ideal - therefore the use of IAM.

Looking at the Documentation I have not been able to get the above IAM driven setup to work with Cloud Endpoints. The documentation refers to (https://cloud.google.com/endpoints/docs/grpc/api-access-overview) API access, in particular referring to "Grant access to your API users so they can enable your API in their own Google Cloud project." - this seems to only provide access to enable the API, not necessarily related to the authorisation of a request (i.e. what we achieve with roles/run.invoker).

It appears that the only native way to manage access / authorisation with Cloud Endpoints is using API Keys?

Kind regards, Jan

[Josh Einhorn]

It appears that the only native way to manage access / authorisation with Cloud Endpoints is using API Keys?

That's mostly correct, however it seems like your particular use case is narrowly scoped to service-to-service authorization; in this case you can do this with Endpoints. It is secure because the Service Account is "hard-coded" as the JWT issuer such that only specified SAs are authorized. Note you can specify multiple JWT requirements using OR logic to authorize multiple SAs.

Of course this is limited to IAM Service Account authentication only! So if you intend to authorize humans, you'll need Cloud IAM.

-Josh

[Jan Krynauw]

Thank you Josh, this makes a lot of sense now.

Kind regards, Jan

[Jan Krynauw]

Hi Josh,   your service-to-service documentation only seem to refer to to the Open API specs, any support for gRPC?   For example, the securityDefinitions object is not defined in your services.proto.

[Josh Einhorn]

Hi Jan,

Absolutely it's supported, it's just under a different name: https://cloud.google.com/endpoints/docs/grpc/service-account-authentication. The settings in the service configuration in the example will also work with API Gateway (which I think is your question).

-Josh

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/a9305d6a-e04f-4154-bd12-c4c4cee469b8n%40googlegroups.com.

[Jan Krynauw]

Thank you Josh.

Looking at your link, in particular https://cloud.google.com/endpoints/docs/grpc/service-account-authentication#generate_a_jwt_token it refers to:

In this example, the Bookstore is using service-to-service authentication where the calling service is purely authenticated by its service account, so creating an appropriate token to send with our requests is simple. Note that you can also require more stringent service-to-service authentication where the generated token must be further authenticated by Google (using a Google Id token).

Looking at https://cloud.google.com/endpoints/docs/grpc/troubleshoot-jwt a valid token is:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "42ba1e234ac91ffca687a5b5b3d0ca2d7ce0fc0a"
}

Payload:
{
  "iss": "myse...@myproject.iam.gserviceaccount.com",
  "iat": 1493833746,
  "aud": "myservice.appspot.com",
  "exp": 1493837346,
  "sub": "myse...@myproject.iam.gserviceaccount.com"
}

In particular it mentions that the "iss" and "sub" both need to be the service account email address.

Now, looking at a Google Id token the "iss" and "sub" are not the same.  Here's an example:

{
  "aud": "https://example.com",
  "azp": "113774264463038321964",
  "email": "gae...@appspot.gserviceaccount.com",
  "email_verified": true,
  "exp": 1550185935,
  "iat": 1550182335,
  "iss": "https://accounts.google.com",
  "sub": "113774264463038321964"
}

This is what say Pub/Sub adds to an authenticated push message.  This is also how we currently authenticate calls to Cloud Run gRPC endpoints.  This is also what 'gcloud auth print-identity-token' provides.

If I understand this, setting the aud to the service account email address is merely an 'proxy' to control 'Access' to the backend via the Gateway?

Could one achieve similar results by setting the audience to something super hard to guess?  For example:

authentication:
  providers:
    - id: google_id_token_test
      issuer: "https://accounts.google.com"
      jwks_uri: "https://www.googleapis.com/oauth2/v3/certs"
      audiences: "super-hard-to-guess-text-here"
  rules:
    # This auth rule will apply to all methods.
    - selector: "*"
      requirements:
        - provider_id: google_id_token_test

Is there a workaround where say Pub/Sub could make an authenticated push via the Gateway?

Kind regards, Jan

[Wayne Zhang]

Hi Jan,

We don't recommend passing Google id token to ESP for API security.  since "issuer" and "jwks_uri" are not service account specific, all ID tokens generated from all service accounts can be verified by ESP, the only way to guard the API is the "audience" field.  "Super-hard-to-guess" is not secure enough. 

Thanks

-Wayne

To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-endpoints/CADeeoqrZd7aHMq%2Bb6XQt1Z%2B7YJib5GJj4tHh1K9twzx%3DUr%3Duow%40mail.gmail.com.

[Josh Einhorn]

In particular it mentions that the "iss" and "sub" both need to be the service account email address.

This is a bit misleading; they don't need to be the SA e-mail, it's just a convention and a very optional one at that. Anyone with a Google SA can put whatever they want in those fields by signing a JWT directly instead of getting an ID token.

If I understand this, setting the aud to the service account email address is merely an 'proxy' to control 'Access' to the backend via the Gateway?

No, the audience field is used to down-scope the token to only work on a particular resource, it is not for access control. The reason for this is that various vulnerabilities in your client applications may lead to token leaks ... without an audience, these tokens would give unfettered access (albeit temporary) to every resource the SA has access to. The audience prevents this.

Could one achieve similar results by setting the audience to something super hard to guess?

Maybe, but this is an anti-pattern; the mechanism you're looking for is commonly known as a "shared secret" i.e. API Keys. Fortunately, API Gateway comes with this feature out of the box! The advantage of API Key based access is that you can track usage by Key and rotate API Keys based on usage patterns e.g. revoke a key that looks like it's being used abusively. You could maybe figure out a way to do this via the audience, but it's not the intent of that feature.

-Josh