Connecting from app engine instance to compute engine instance directly

[Ian Childress]

We (our Go dev team) want to connect our app engine apps to our compute engine apps directly using internal IP address. From my understanding through documentation and exhaustive searching, the solution is to use a public IP address. This solution requires whitelisting an entire ip block (Google's ip block for app engine). This both increases security risk as well as charges additional bandwidth. The other solution is to use a messaging service PubSub. This is fine for submitting tasks to be performed by the back end app, but it prevents the app engine from receiving the response from the compute engine. 

Have I missed the solution somewhere that allows for internal communication between app engine and compute engine? Using a job queue or public IP is not a replacement for an internal socket connection.

[Alex Martelli]

On Thu, May 7, 2015 at 10:52 AM, Ian Childress <ianchi...@gmail.com> wrote:

We (our Go dev team) want to connect our app engine apps to our compute engine apps directly using internal IP address. From my understanding through documentation and exhaustive searching, the solution is to use a public IP address. This solution requires whitelisting an entire ip block (Google's ip block for app engine). This both increases security risk as well as charges additional bandwidth. The other solution is to use a messaging service PubSub. This is fine for submitting tasks to be performed by the back end app, but it prevents the app engine from receiving the response from the compute engine. 

How so?  Both GAE and GCE can be both talkers and listeners on Cloud Pubsub topics / subscriptions (see https://cloud.google.com/pubsub/docs ). 

Have I missed the solution somewhere that allows for internal communication between app engine and compute engine? Using a job queue or public IP is not a replacement for an internal socket connection.

This sounds to me like a very reasonable feature request that you could open at https://code.google.com/p/googleappengine/issues/list and/or https://code.google.com/p/google-compute-engine/issues/list -- it does not look as if such a feature request for direct GAE <-> GCE sockets (w/o a public IP) has previously been opened.

Alex

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/db88993d-65b8-4b15-bad4-740fd654b082%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ian Childress]

On Thursday, May 7, 2015 at 2:25:08 PM UTC-4, Alex Martelli wrote:

How so?  Both GAE and GCE can be both talkers and listeners on Cloud Pubsub topics / subscriptions (see https://cloud.google.com/pubsub/docs ). 

I can't think of a good solution for taking many web requests and passing those requests to PubSub and receiving the corresponding request. If 10 requests come in, and 10 go routines submit jobs to the PubSub, and the compute engine processes the jobs, there isn't a way of getting the correct response to the correct go routine short of creating a one time topic that is torn down after the response has been made. I don't think PubSub is designed to be used that way. When I have created topics and subscriptions, it tends to take +1 seconds to reply. The whole process becomes several seconds long as pubsub compute engine uses the PULL method via api. I'm open for suggestions though! 

This sounds to me like a very reasonable feature request that you could open at https://code.google.com/p/googleappengine/issues/list and/or https://code.google.com/p/google-compute-engine/issues/list -- it does not look as if such a feature request for direct GAE <-> GCE sockets (w/o a public IP) has previously been opened.

Thank you very much, I will create a feature request for this. 

[Alex Martelli]

On Thu, May 7, 2015 at 11:39 AM, Ian Childress <ianchi...@gmail.com> wrote:

On Thursday, May 7, 2015 at 2:25:08 PM UTC-4, Alex Martelli wrote:

How so?  Both GAE and GCE can be both talkers and listeners on Cloud Pubsub topics / subscriptions (see https://cloud.google.com/pubsub/docs ). 

I can't think of a good solution for taking many web requests and passing those requests to PubSub and receiving the corresponding request. If 10 requests come in, and 10 go routines submit jobs to the PubSub, and the compute engine processes the jobs, there isn't a way of getting the correct response to the correct go routine short of creating a one time topic that is torn down after the response has been made.

The way I would do it in Python: each GAE request to GCE includes a unique ID, GCE's response to GAE includes the request's ID too, a single listener on the Python GAE side receives all responses and dispatches them via a dictionary mapping the ID to the coroutine (or other processing unit) that needs to process the response.  The dictionary's maintenance (adding the ID -> processing unit entry on sending the request, removing it when not needed any more) can be done either in the sending/processing units, or in the dispatcher, or partly in each, whatever's most convenient.

Unfortunately I'm not "knowledgeable and current" enough in Go to know how best this simple "funnel/dispatcher" architecture is mapped to the language; I'd imagine a mapping to e.g go channels where the appropriate goroutines are waiting/listening might work reasonably well, for example.

Alex

 

I don't think PubSub is designed to be used that way. When I have created topics and subscriptions, it tends to take +1 seconds to reply. The whole process becomes several seconds long as pubsub compute engine uses the PULL method via api. I'm open for suggestions though! 

This sounds to me like a very reasonable feature request that you could open at https://code.google.com/p/googleappengine/issues/list and/or https://code.google.com/p/google-compute-engine/issues/list -- it does not look as if such a feature request for direct GAE <-> GCE sockets (w/o a public IP) has previously been opened.

Thank you very much, I will create a feature request for this. 

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/db88993d-65b8-4b15-bad4-740fd654b082%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/8134e724-6c6a-4547-a063-997f02486ef9%40googlegroups.com.

[Hussein Vastani]

Hi,

At my current gig we've been grappling with the same problem, and there seems to be a solution in the 1.9.19 SDK release (finally!). 

So my context is: my java frontend instances need to hit my elasticsearch cluster hosted on GCE. The only way to reach the cluster was via a reverse proxy exposed through a GCE public IP address. I was using basic auth to authenticate my App Engine frontend instances. As you would expect the latency has been hurting. Search queries that should take 10ms, instead come back in 150ms (10x slow). After additional processing in my frontend request, that adds up pretty quickly to noticeable latency user-side. 

I posted about the problem some time ago in this group here and the issue tracker here. I got an acknowledgement from Google and the fix appeared in the 1.9.19 release, a bit quietly (the issue is still open).

So the solution for me is to run a managed VM on the same network as my elasticsearch cluster, and that way I should be able to access my cluster "directly" using internal network addresses, without going out of the Google network. The 1.9.19 SDK allows you to specify a network in the configuration for your managed vm, that defaults to the default GCE network of your project. Thus I either directly route my search requests to my managed VM or have my front end instances talk to my managed VM over http. 

(For my particular case, I expect additional latency savings because I can run a full elasticsearch client on the managed VM that knows the cluster state, instead of using the naked elasticsearch REST API, as I do now, from my front end instances. That saves me an additional network hop that is needed to figure out which nodes in my cluster to hit.)

I guess, the same should work for you no?

I'll post next week about the results of migrating to a managed VM setup.

Good luck!

H

[Ian Childress]

Thank you Hussein. We are going to move our app from appengine to a managed VM and place it in the same network (default) as our compute engines. Until appengine and compute engine instances can communicate on the same network we'll be forced to use this work around. Please let me know how your method works!

[Robert Dyas]

Has there been any progress on this?

On Thursday, May 7, 2015 at 1:52:39 PM UTC-4, Ian Childress wrote:

[Nicholas (Google Cloud Support)]

This thread is a little dated and may not get the attention you expect.  I'd recommend posting to a new thread and describing exactly the requirements you seek.  When doing so, please describe the criteria you need met.

• Would GAE being issuing request and GCE, responses?  The other way around?  Both?
  
• Are requests and responses tightly coupled like HTTP requests/responses or more independent like tasks in task queues reflecting asynchronous requests?
  
• What type of scaling would be employed for the App Engine component?  Would the Compute Engine instances also be scaled?
  
• What solutions have you tried?  Any of the ones posted above?
  

[Lorne Kligerman]

If a direct connection between an App Engine standard app to a GCE VM via an internal IP address is what you're looking for, work is in progress!

When ready for testing I'll be sure to drop a note to this group.

Cheers,

Lorne.

Product Manager - App Engine

[Robert Dyas]

The sooner the better!

[Bjorn Harvold]

Definitely a needed feature!

[AJ Ross]

If you're willing to migrate to Flex, it appears as though you can do it:

https://groups.google.com/d/msg/google-appengine/5RsoKIPWyTQ/pnxyI-anBAAJ

On Wednesday, April 26, 2017 at 8:03:29 PM UTC-7, Bjorn Harvold wrote:

Definitely a needed feature!

[Justin Beckwith]

No no no no :) We don't enable this directly in flex.  May I ask - why are you trying to connect directly to a single instance?  

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9dc63deb-1bf5-49af-a26f-319a6c86dc80%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Justin Beckwith | Google Cloud Platform | @justinbeckwith | http://jbeckwith.com

[AJ Ross]

I would like to connect to a microservice running on kubernetes cluster from my appengine application, while avoiding the need to transit the public internet. Since flex runs my appengine application as a cluster in GCE it seems like this should be possible?

On Friday, May 26, 2017 at 1:26:50 PM UTC-7, Justin Beckwith wrote:

No no no no :) We don't enable this directly in flex.  May I ask - why are you trying to connect directly to a single instance?  

On Fri, May 26, 2017 at 1:20 PM, AJ Ross <a...@lasersox.net> wrote:

If you're willing to migrate to Flex, it appears as though you can do it:

https://groups.google.com/d/msg/google-appengine/5RsoKIPWyTQ/pnxyI-anBAAJ

On Wednesday, April 26, 2017 at 8:03:29 PM UTC-7, Bjorn Harvold wrote:

Definitely a needed feature!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9dc63deb-1bf5-49af-a26f-319a6c86dc80%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[David Grant]

Any update on this?

[Robert Dyas]

Still waiting here too. 

On Wednesday, June 21, 2017 at 2:45:20 PM UTC-4, David Grant wrote:

Any update on this?

[Lorne Kligerman]

Nothing to report at this point, I'll post more details when ready.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/P2PK6g4TteY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine+unsubscribe@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/2405ea07-9462-43d1-b020-108f1e142deb%40googlegroups.com.

[guillaume blaquiere]

Thanks. I subscribed to this groups for updates. I hope to have good news soon :)

Le mercredi 21 juin 2017 22:34:53 UTC+2, Lorne Kligerman a écrit :

Nothing to report at this point, I'll post more details when ready.

On Wed, Jun 21, 2017 at 11:48 AM, Robert Dyas <rober...@parasql.com> wrote:

Still waiting here too. 

On Wednesday, June 21, 2017 at 2:45:20 PM UTC-4, David Grant wrote:

Any update on this?

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/P2PK6g4TteY/unsubscribe.

To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.

Visit this group at https://groups.google.com/group/google-appengine.

[Marcus Mosttler]

Does anyone know if there has there been any updates on getting the ability to connect from GAE Standard to a GCE instance?

[Sachin Patidar]

Is there any plan to release this feature?

[Osmond Ng]

Will GAE standard different to GAE Flex interm of connecting to GCE instance?  Any timeline can share?  IHAC is waiting for this for their AWS to GCP migration.

[Chris Santos]

Still can't believe we're waiting on this. Are there any updates?

[Jason Collins]

Check out https://cloud.google.com/appengine/docs/standard/python/connecting-vpc

[Nikola Mihajlović]

Hello,

I am trying to connect to an instance on GCE from GAE by using its name: http://INSTANCE.c.PROJECT_ID.internal

Is this not possible?

[Harmit Rishi (Cloud Platform Support)]

Hello, 

My research into the matter allowed me to find a Beta for Serverless VPC Access. This product allows you to connect from your App Engine app directly to Compute Engine VM instances, Cloud Memorystore instances, Cloud SQL instances, and any other resources with an internal IP address. The following documentation highlights how to use it for Python here. You may feel free to navigate to the desired supported language offered on that page for your use case.

Hope this helps!

[Vishnu Mohan]

Hey Hussein,

How did you manage network configurations in app.yaml file for App Engine. I have an App Engine Flex and an Elasticsearch on Compute Engine under same VPC. But i am not able to connect my app engine to the internal ip of compute engine to access elasticsearch.

Thanks.!

[yananc]

Hello Vishnu,

The documentation ‘app.yaml Configuration File - Network settings’ provides information on how to configure network settings in the app.yaml file for App Engine Flexible. You may also navigate to similar pages for other supported language you desire for your specific use case.

And also please make sure that firewall rules for the VPC network are properly configured so the communication between App Engine and Compute Engine is not blocked. The link offers details about firewall settings for VPC network.

Hope the information above is helpful to answer your question.