PubSub Push to IAP Protected AppEngine service not working

[Prateek Malhotra]

I have setup a PubSub subscription to push authenticated HTTPS requests to my AppEngine application sitting behind Cloud IAP. With IAP disabled, the requests come through fine - I even logged the Authorization token header and tried it via cURL with IAP enabled again and it works. So I can validate that PubSub is sending the correct token but with IAP enabled, the request never reaches my service.

What I did to troubleshoot:

• Manually send an event to the topic
• Monitor logs for AppEngine service and see nothing come through
• Disable Cloud IAP for AppEngine service
• Monitor logs for AppEngine service and see the request come through as expected, logged all headers
• Enabled Cloud IAP for AppEngine service
• Tried a POST cURL request using Authorization header logged when Cloud IAP was disabled
• Monitor logs for AppEngine and see cURL request come through

Setup:

gcloud beta iam service-accounts create cron-service-accessor --description="Only this user is authorized to make requests to the Cron service on AppEngine" --display-name="Cron Service Accessor" --project=$PROJECT_ID

gcloud beta iap web add-iam-policy-binding --member=serviceAccount:cron-service-accessor@$PROJECT_ID.iam.gserviceaccount.com --role="roles/iap.httpsResourceAccessor" --service=cron --resource-type=app-engine --project=$PROJECT_ID

export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID | grep projectNumber | cut -d "'" -f 2 -)

gcloud beta iam service-accounts add-iam-policy-binding cron-service-accessor@$PROJECT_ID.iam.gserviceaccount.com --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com" --role=roles/iam.serviceAccountTokenCreator --project=$PROJECT_ID

gcloud beta pubsub topics create files-topic --project=$PROJECT_ID

gcloud beta pubsub subscriptions create process-files --topic=files-topic --ack-deadline=600 --retain-acked-messages --push-endpoint="https://cron-dot-$PROJECT_ID.appspot.com/process_files"  --push-auth-service-account=cron-service-accessor@$PROJECT_ID.iam.gserviceaccount.com --push-auth-token-audience=$IAP_CLIENT_ID --expiration-period=never --project=$PROJECT_ID

A similar setup is working with Cron Scheduler, using the oidc authentication options available there for the request. I even came across this StackOverflow question where some users are reporting they got it to work and others (more recently, within the past few weeks) are reporting it does not work.

Is there an issue with PubSub preventing this mechanism from working? Does a project age have anything to do with it?

I'd appreciate any help/guidance on the matter.

Thank you,

Prateek

[Jinjun (Cloud Platform Support)]

Please check the answer in following Stackoverflow post on how to make it work.

https://stackoverflow.com/questions/57817374/

[Prateek Malhotra]

Thanks for the reply! I actually mentioned this in my initial question, I already found that post and even after following the directions the PubSub push still doesn't work!

Do you have any other things I can try? 

Thank you,

Prateek

[Prateek Malhotra]

Ahh - it seems like a comment was added a couple days ago with a solution: the IAP Invoker role needs to be granted on the entire AppEngine application instead of just the service that PubSub requires access to.

This appears to be a bug with PubSub - the same service account is used with Cloud Scheduler and works without issue for that service without granting the Invoker role to the entire application. Why does PubSub require access to the entire AppEngine application vs just the specific service it requires access to in IAP? As mentioned in my initial post, it appears PubSub can generate the correct token to access the service - there must be something previous to that token generation that's incorrectly trying to access the service and failing.

On Friday, December 6, 2019 at 9:14:55 PM UTC-5, Jinjun (Cloud Platform Support) wrote:

[Jinjun (Cloud Platform Support)]

To open a bug with Pub/Sub, please use our Public Issue Tracker at https://cloud.google.com/support/docs/issue-trackers.