Which IP address ranges to allow so PubSub can push to my (flexible) AppEngine?

[Jack Lee]

So this question has been asked a few times here and there:

1. IP Ranges / Blocks for Pub / Sub - Google Groups
   
2. google cloud platform - App Engine Firewall Setup For PubSub Push Messages - Stack Overflow
   
3. firewall - What is the IP range(s) of Google pub/sub? - Stack Overflow
   

Like the guy in link 2 I have set the default rule to deny which explains why my subscriber never receives any push messages from PubSub. If I change it to allow then the messages reach my AppEngine.

But then what is the solution to get PubSub push working with default to set deny? 

First attempt:

I've tried Jordan's suggestion in link 1 of following the Static IP Addresses and App Engine apps' guide and ended up with a bunch of gcloud commands like this:

gcloud app firewall-rules create 1000 --source-range="8.34.208.0/20" --action=allow --description="gcp ip address"

..etc...

Currently, there are 5 _cloud-netblocks and I believe I've added all ranges in each netblock.

...but data still don't flow from PubSub to my AppEngine. Out of curiousity I ping'ed pubsub.googleapis.com and tested the IP against all the IP-ranges I've allowed in AppEngine firewall; turns out it would have been denied though. Adding it to the list with allow didn't help. however.

Second attempt:

I've printed out all request headers and noticed that X-Forwarded-For has value 10.0.0.1. I've allowed that one for a while since it's the same IP for allowing cronjobs.

Third attempt:

The answer in the third link says that one should allow "2002:a00::/24". I've also added that but with no luck.

What else can I try?

[Kir Titievsky]

Jack, Pub/Sub push does not play well with IP based firewall rules.  We strongly encourage you to consider a different security mechanism.  Some options:

1. Switch to pull/streamingPull, where there requests would come from your consumers. 

2. For App Engine Standard Endpoints, rely on service-account based access control.  The path between Pub/Sub and App Engine offers additional security compared to a webhook hosted on a public URL.

3. Set up your own push proxy.  A simple stateless service might pull messages from Pub/Sub and push them from a fixed IP address to your webhook. Pub/Sub offers value in this set up by providing scale and availability. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e88d6ef1-af6d-41ba-9b6a-f842e12cb2b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub 

[Jack Lee]

Hi Kir.

Thank you for your reply.

I have also looked at using PubSub pull as an alternative but is it correct that pull requires that the subscriber needs to run in a loop in order to monitor new published messages, i.e. a cronjob in AppEngine? This would then incur additional cost to the project since my AppEngine has to run frequently as pointed out by Jordan here.

Maybe my use case for using PubSub was weak to begin with since I was just looking into separating my API into two parts. I might as well just have written a simple message queue and used Datastore/SQL for storing new job messages and then installed the worker as a cronjob in Google Cloud. Or spawned a worker thread in the API or something.

Do you know if the upcoming Cloud Tasks push will have the same firewall issue? At least according to the Task Queue documentation for the Standard environment it says that "Task Queues and Cron traffic will be allowed by the firewall, even when the default rule is set to deny".

I wish that it was mentioned somewhere in the documentation that one may run into issues with PubSub when the firewall is in use. It could at least have been mentioned in the troubleshooting section. It seems to me like a strong reason for looking into an alternative framework.

Thanks and have a nice weekend. :)

/Jack

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e88d6ef1-af6d-41ba-9b6a-f842e12cb2b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kir Titievsky]

Fair point on documentation. Can you say more on why your webhook must have a firewall, particularly, if it is running on app engine?

[Jack Lee]

The webservice is currently planned to be available only for specific clients and the AppEngine firewall function seemed fit for us for now. I guess I could remove firewall to let PubSub push work and then look into adding some security mechanism to the solution, something that I've planned to do so at some point too. Thanks.

/Jack

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/e88d6ef1-af6d-41ba-9b6a-f842e12cb2b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub