Setting Cloud Monitoring uptime checks for non publicly accessible backends

[Fareed Ahmed]

Copy of: https://stackoverflow.com/questions/67426377/setting-cloud-monitoring-uptime-checks-for-non-publicly-accessible-backends

I'm having some trouble setting uptime checks for some Cloud Run services that don't allow unauthenticated invocations.

For context, I'm using Cloud Endpoints + ESPv2 as an API gateway that's connected to a few Cloud Run services.

The ESPv2 container/API gateway allows unauthenticated invocations, but the underlying Cloud Run services do not (since requests to these backends flow via the API gateway).

Each Cloud Run service has an internal health check endpoint that I'd like to hit periodically via Cloud Monitoring uptime checks.

This serves the purpose of ensuring that my Cloud Run services are healthy, but also gives the added benefit of reduced cold boot times as the containers are kept 'warm'

However, since the protected Cloud Run services expect a valid authorisation header all of the requests from Cloud Monitoring fail with a 403.

From the Cloud Monitoring UI, it looks like you can only configure a static auth header, which won't work in this case. I need to be able to dynamically create an auth header per request sent from Cloud Monitoring.

I can see that Cloud Scheduler supports this already. I have a few internal endpoints on the Cloud Run services (that aren't exposed via the API gateway) that are hit via Cloud Scheduler, and I am able to configure an OIDC auth header on each request. Ideally, I'd be able to do the same with Cloud Monitoring.

I can see a few workarounds for this, but all of them are less than ideal:

• Allow unauthenticated invocations for the underlying Cloud Run services. This will make my internal services publicly accessible and then I will have to worry about handling auth within each service.

• Expose the internal endpoints via the API gateway/ESPv2. This is effectively the same as the previous workaround.

• Expose the internal endpoints via the API gateway/ESPv2 AND configure some sort of auth. This sort of works but at the time of writing the only auth methods supported by ESPv2 are API Keys and JWT. JWT is already out of the question but I guess an API key would work. Again, this requires a bit of set up which I'd rather avoid if possible.

Would appreciate any thought/advice on this.

Thanks!

[Igor Peshansky]

As far as I can see, you have two independent issues here. One is keeping your endpoints private, and the other is authentication.

For the former, you can set up uptime checks for services on a private network — see the section on firewall config in https://cloud.google.com/monitoring/uptime-checks#before_you_begin.

For the latter, one approach can be some sort of a secret key in the URL that is only used for uptime checks (like an API key). As you've noticed, uptime checks also support HTTP/HTTPS username/password authentication, but they don't support any sort of dynamic auth as far as I know…

Hope this helps,

        Igor

--
© 2021 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/0f7052ff-baf5-4d53-bf3d-ee841485ab32n%40googlegroups.com.

[Fareed Ahmed]

Awesome, thanks!

I'll take a look at uptime checks for private networks.

Do you know if there are any plans to add dynamic auth headers to uptime checks?

For reference, this is what it looks like when creating a new job in Cloud Scheduler.

As well as setting static headers it also allows you to define dynamic OIDC auth headers using service accounts.

I can imagine this would be quite complicated though as this could lead to privilege escalation if not used correctly.

[Igor Peshansky]

@John Brier is our Uptime Monitoring PM, who may be interested in this use case, and can inform you of any plans regarding Uptime Checks. If you're in a position to open a cloud support case, that would be a way to file a feature request that would allow you to keep track of it.

        Igor

[Fareed Ahmed]

Thanks for the tip!

I’ll speak with some colleagues tomorrow and see how we can go about opening a cloud support case.

Regards,

Fareed