AppEngine Flex - Correct Firewall Configuration For Inter-Service-Communication

[dvd gsng]

Hi folks, we've deployed multiple GAE Flex services in the same project that talk to eachother using the URLs suggested in the documentation (https://cloud.google.com/appengine/docs/flexible/java/communicating-between-services). This works fine until we update the GAE firewall and change "The default action" from Allow to Deny. We then end up with 403 Forbidden for calls between the services. However, calls from outside GAE to the services succeed (using the same URLs).

After digging a little deeper into documentation, we found that certain IPs need to be whitelisted "to accommodate the IP addresses that are used for service-to-service communication", so we added four more rules to the GAE firewall (0.1.0.40, 10.0.0.1, 0.1.0.30, 10.1.0.41) (https://cloud.google.com/appengine/docs/flexible/java/creating-firewalls#allowing_requests_from_your_services). Unfortunately without success.

Just to make sure, we've also configured the same rules in the regular VPC firewall for the default network, which would make sense since GAE Flex utilizes GCE instances. But no success here too.

The documentation also lists certain request headers that can be added, but the only one that would have been helpful is only available in GAE Standard (X-Appengine-Inbound-Appid).So no point in setting them, AFAICT.

We don't use the default service.

We don't use a dispatch.yml.

We use only the default GAE service accounts to run services, no futher credentials are provided.

We don't use GAE standard.

We're using a custom runtime with OpenJDK11 as base image.

How is the firewall supposed to be configured for inter-service-communication with DENY as the default action? Are missing something?

Thanks in advance!

David

[George (Cloud Platform Support)]

It is not immediately apparent, after reading the documentation page you link to, how certain IPs are to be whitelisted; targeted HTTP requests, service accounts, and Cloud Pub/Sub are mentioned, are recommended solutions on that page. 

The firewall configuration page stipulates, for requests received in the flexible environment: 0.1.0.40 and 10.0.0.1. You need to create two firewall rules to allow requests:

0.1.0.40 - A rule to allow backend_flex to receive URL Fetch requests from backend_std.

10.0.0.1 - A rule to allow the service-to-service communication for the URL Fetch requests in backend_flex.

[dvd gsng]

Hi George, thanks for replying. Unfortunately, we've tried that, but it didn't work out as expected. We added those two IPs (and two more, see initial post) to the GAE firewall as well as the regular VPC firewall. Please have a look that the images for reference:

GAE firewall:

VPC firewall:

Note that the VPC FW rule does apply to the GAE.

WDYT?

Also, we're currently looking into using IAP for securing our endpoints. Is that the recommended way to implement authenticated communication between GAE services?

David

[Nicolas (Google Cloud Platform Support)]

Hi,

Thanks for the screenshots! I’ve reproduced this and was able to confirm that if you switch the App Engine Firewall rules to “Deny All” and then try to granularly allow some IP address the app will return 403s.

This is most likely due to the internal App Engine infrastructure so I would recommend IAP as your best current solution to communicate between GAE services.

[dvd gsng]

Hi Nicolas, thanks for verifying our results. We think that the GAE documentation should be updated to reflect these restrictions more explicitly, are you in a position to forward this to the appropriate team by any chance?

Thanks

David

[Nicolas (Google Cloud Platform Support)]

Hi David,

Thanks for bringing this to our attention, I will gladly submit this documentation improvement to the rightful team.

Thanks again and have a great day!