Managed VMs logs, load-balancing & context

[John Smith]

TL;DR: Need to use WebSockets on AE. Only works on MVM. NewContext fails, BackgroundContext works. Logs are polluted & disorganized (/_ah/background)

Hey,

I would like to communicate with my App Engine app over WebSockets.

I have found that the Java library has an official (github) example of using WebSockets under the Managed VMs framework,

after browsing the implementation I noticed that it queries the internally exposed metadata service to get the external IP of the machine and redirects the client to it.

However I expect my application to receive more traffic than a single server would handle and so I need a load-balanced solution (cannot relay on a single static-IP)

so I've looked into App Engine's "Network Load Balancer" and after an hour (and a long deploy time) I was up and running with now my domain set to the load balancer's static IP.

Now is a good time to note that I use the Go standard runtime.

While porting my app from the Classic App Engine I was facing an issue with the request context (which now "skips" the default HTTP load balancer and goes straight to my MVM(gce) node).

I was getting an error ("appengine: NewContext passed an unknown http.Request").

To overcome my need for datastore/logging/other App Engine facilities I used context.BackgroundContext() everywhere and it seemed to work well for the datastore/urlfetch/etc..

Everything worked but the logging facility which was polluting my logs with empty "/_ah/background" lines and everything else was multiplexed into a single huge "/_ah/background" entry.

This makes it very difficult to debug or monitor my application.

I have browsed the google.golang.org/appengine package and tried to split the log lines to no avail.

My questions are:

1) Can I use the classic App Engine/default http load balancer/custom http load balancer to service my WebSocket requests?

2) Can I, with my current solution, get the X-AppEngine-* headers?

3) How could I, with my current solution, split the log lines to per request entries (very like the classic app engine logs)

4) Is this the right approach?

Thanks.

[Nick (Cloud Platform Support)]

As someone somewhat familiar with the Go runtime but not with your exact use case, I'd like to help, but I'm wondering if you could provide some more concrete specification of the following from your post, in terms of code or logs: 

While porting my app from the Classic App Engine I was facing an issue with the request context (which now "skips" the default HTTP load balancer and goes straight to my MVM(gce) node).

 

I was getting an error ("appengine: NewContext passed an unknown http.Request").

 

To overcome my need for datastore/logging/other App Engine facilities I used context.BackgroundContext() everywhere and it seemed to work well for the datastore/urlfetch/etc..

This might help me to reproduce your situation and attempt different solutions.

As to your numbered questions, here are my first responses:

1. Are you talking about "Network Loadbalancing" or the GCE "Http Load balancer", and could you clarify again what kind of errors you observed when attempting this? What is meant by context "skipping"?

2. Do you notice that you're receiving X-Appengine-* headers? If so, you can certainly use them. If you attempt to add them to requests, do you notice if they're stripped? If they are, it won't work, if they aren't, it will. Although this is a quite empirical way to proceed, and if something isn't explicitly documented, you should file a Public Issue Tracker feature request to have the behaviour defined. 

3. Your log lines are all piling up under one /_ah/background entry, as far as I can tell. You could use the Logging API to write logs, and see if that shows up properly. Another option is logging to a BigQuery or SQL table. The final, most trivial answer, is to figure out how to get request logging to work without this BackgroundContext which seems to flatten all its logs into one expando in the Developers Console UI (if I'm understanding you correctly, again, it seems like this is what you're reporting).

4.  A better approach could only be found through further investigation, if it exists. I'm optimistic that a better approach is out there, it would just take more thought.

[John Smith]

Hey Nick,

Thank you for your response, I would like to further clarify the flow of things:

• I have to use WebSockets
• WebSockets are not supported on the Classic App Engine (when I attempt to open a websocket connection to my-game.appspot.com it is being filtered out and not even making it to my app code)
• WebSockets seem to work on MVM via a network-load-balancer (and not via the default my-game.appspot.com domain)
• using MVM and a network-load-balancer I cannot create a request context [to create a new context I use the supplied appengine.NewContext(r) function and get the mentioned error]
• I need a context to use the datastore.GetMulti() or log.Debugf() APIs
• I am able to create a background context (supported by MVM) and use these APIs but when I use this context (with the appengine/log.Debugf() API) the logs get cluttered (aggregated under /_ah/background)

1. are the above points clarify how my app receives requests?
2. no, when a request is dispatched via network-load-balancer (and not my-game.appspot.com) my application gets no X-Appengine-* headers at all
3. yes all of my logs are piling up under one /_ah/background entry. as mentioned above I am using the golang standard appengine/log library as I did with my previous Classic App engine.

ideally the appengine.NewContext(r) function would work and logging via the appengine/log.Debugf() would split the lines per request.

do you find this explanation sufficient?

Thanks.

[Kyle Finley]

John,

Are you calling appengine.Main() in your main() function?

I'm not sure if it works with the network-load-balancer, but when I switched from classic to MVM I was receiving the same error. Adding the appengine.Main() call fixed the unknown http.Request error.

Kyle

e.g.

package main

import (

"google.golang.org/appengine"

)

func main() {

appengine.Main()

[John Smith]

Hey Kyle,

yes I am using appengine.Main() to start serving requests.

did you try to access your VMs directly? (by IP, or by network-load-balancer)

(note: when using google's default my-game.appspot.com appengine.NewContext works but WebSockets dont)

Thanks

[Nick (Cloud Platform Support)]

Hey John,

Could you clarify the relation between the need for a context object and websockets? Is it that you'd like to access App Engine services and do logging while responding to websocket events? There appears to be two issues here, one of the Golang runtime and context objects with regard to Managed VMs, and another issue of websockets on Managed VMs and how to connect (with the appspot.com domain refusing port connections while port connections are working when the instance is accessed via IP). Does this accurately clarify the issues at play here? 

[Kyle Finley]

John,

I just tested accessing my VM by IP. Unfortunately the request timed out:

curl: (7) Failed to connect to 23.XX.XX.XX port 80: Operation timed out

Incase you haven't seen it yet. There's a thread in the App Engine Go group where the issue with background context logging is discussed. There's a few alternatives suggested.

https://groups.google.com/forum/#!searchin/google-appengine-go/background/google-appengine-go/V90qiZKJ5Tk/CWiXsxkNDgAJ

I hope you find a solution I'll be looking into websockets in the near future.

Kyle

[John Smith]

Hey Nick & Kyle,

@Nick

Yes I do need to call App Engine services in response to WebSocket requests. 

There are multiple issues here and I'm trying to work around them

to sum things up:

• WebSocket connections are not supported in the Classic App Engine OR the HTTP load balancers
• The App Engine log API does not export API for writing specific log entries and the BackgroundContext logs get cluttered

@Kyle

You probably could not connect to your MVM instance because the default firewall rule forbids access to port 80, try :8080 instead

After browsing the thread you linked to I have found this API, which I will look into and might help me mitigate App Engine's lack of finer resolution log infrastructure.

Thanks again,

John

[Nick (Cloud Platform Support)]

Hey John,

The thread linked by Kyle does indeed have some good ideas for background context logging. I believe the explanation for the HTTP load balancer not working with websockets is that it's only meant to transfer HTTP traffic, and won't respond properly to socket connection attempts. If your sockets library is capable of falling back to HTTP long polling (such as socketio tends to do), that might work through the HTTP load balancer. A common pattern to side-step this difficulty is to have an endpoint on each instance which allows the client to request its IP, so that the socket connection can be made to the machine itself, rather than through the appspot front-end or an HTTP load balancer. I think that's your best bet for a scalable solution to websocket connections against a pool of instances, keeping the HTTP traffic for the load balancer.