How to deal with AppEngine Flexible Environment Update

[Prateek Malhotra]

Hello,

The latest announcement about the changes to the AppEngine flexible environment is very troubling for us. Most importantly, we can no longer use the google.golang.org/appengine  package and memcache support is not (yet) available. I have a use-case that I'm not entirely sure I can migrate easily to the new version of the flexible environment or back to the standard environment. It's hard to continue calling the flexible environment "AppEngine" if we can't even use the appengine import package!

Here are some issues I have and want to know if there is anything on the roadmap to ease the transition:

• I use the nds package for the memcache features it provides. I don't see any way to continue using this package in my module/service in the new environment, even if it does add memcache support later on. Is that correct?
• Switching imports from google.golang.org/appengine to cloud.google.com/go/... isn't a simple import path change. In fact, some features on the latter import path for datastore aren't supported on the appengine side and now we have to be extra careful not to leverage those features. I also don't see an equivalent for "appengine.Main()" on the new import package so are we now expected to leverage our web frameworks server listener? How is HTTPS going to be handled in this situation?
• The migration guide for Go provides some guidance on how to migrate, but some features are going to require a lot of work, specifically guarding URL paths with "login:admin" and the Users API. Will further examples/resources be provided for this sort of migration?
• I see no mention that URL routing via a dispatch.yaml will be affected by this change, but URL routing is affected in other ways. Can someone please confirm there will be no issues with the dispatch.yaml routing?
• If the dispatch.yaml routing isn't affected, then does that mean that tasks queued for a specific service/module will continue to route appropriately? Is it only adding tasks to the task queue that would be no longer supported on the flexible environment?

I'd normally migrate back to the standard environment, however, I am utilizing some features only available through the flexible environment that I cannot (easily) migrate back from. I am using grpc, which last I checked wasn't working with the standard environment, to communicate to a server we have on-premise to exchange data with over a Compute Engine network with a VPN setup. If grpc now does work on the standard environment, there is no way for me to utilize the Compute Engine network that has a VPN connection to our on-premise network to communicate over, and would instead have to open up our internal network publically for a grpc client on AppEngine standard. 

We use a mix of standard & flexible services for various parts of our application so maintaining that both sides can communicate with one another and to the same backend database is important. Sharing code between these services/modules is also important but seems as if it will no longer be possible due to the change in import paths. The change to the 'google.golang.org/appengine' import path was useful in that we could unify our code between Standard/Flexible, but now its a shame that effort is being thrown away and we again have to separate the codebase. Would it make sense (and be possible?) to just utilize the new import path for all parts that need to be shared, like datastore access?

I'm hoping the situation will improve over the next 6 months and would like to hear about any such plans or guidance on how to ease this transition. 

Thank you,

Prateek M.

[Chris Broadfoot]

Hi Prateek,

Thanks for giving your feedback. We're definitely listening, and have heard similar concerns from other customers.

I'll put most of my responses inline.

On Mon, Nov 28, 2016 at 2:30 PM, Prateek Malhotra <some...@gmail.com> wrote:

Here are some issues I have and want to know if there is anything on the roadmap to ease the transition:

• I use the nds package for the memcache features it provides. I don't see any way to continue using this package in my module/service in the new environment, even if it does add memcache support later on. Is that correct?

This is a great package, I agree. You're right - we'd need nds (or a similar package) to support some arbitrary caching layer, backed by cloud.google.com/go/datastore rather than the google.golang.org/appengine/datastore package.

• Switching imports from google.golang.org/appengine to cloud.google.com/go/... isn't a simple import path change. In fact, some features on the latter import path for datastore aren't supported on the appengine side and now we have to be extra careful not to leverage those features. I also don't see an equivalent for "appengine.Main()" on the new import package so are we now expected to leverage our web frameworks server listener? How is HTTPS going to be handled in this situation?

Can you elaborate - which features must you be careful not to use? If you're talking about the new entity values encoding, you just need to make sure you're using an up-to-date google.golang.org/appengine/datastore package on the GAE standard side.

appengine.Main() will continue to work on Flex, but it doesn't do very much other than listen with DefaultServeMux on port 8080 and set up a health check handler for /_ah/health. So yes, you can use any listener, as long as it's on port 8080.

HTTPS is handled by App Engine's load balancer - incoming requests are terminated (for example, with HTTP/2 over TLS) and then routed to your application with HTTP/1.1 on port 8080.

The new networking stack opens up possibilities for HTTP/2 all the way through to the application, which may enable things like gRPC. And similarly, we're also looking at Web Sockets.

• The migration guide for Go provides some guidance on how to migrate, but some features are going to require a lot of work, specifically guarding URL paths with "login:admin" and the Users API. Will further examples/resources be provided for this sort of migration?

Good feedback. Some documentation and examples might be good here. I suggest writing some middleware to route users to a login page, sign them in with Google Auth (or Firebase Auth, or Auth0), and store their login info in a signed cookie. See here for an example:

https://cloud.google.com/go/getting-started/authenticate-users

https://github.com/GoogleCloudPlatform/golang-samples/blob/master/getting-started/bookshelf/app/auth.go

If you actually need to check if a user is an owner/editor on the project, then you need to use the IAM API:

https://godoc.org/google.golang.org/api/cloudresourcemanager/v1#OrganizationsService.TestIamPermissions

• I see no mention that URL routing via a dispatch.yaml will be affected by this change, but URL routing is affected in other ways. Can someone please confirm there will be no issues with the dispatch.yaml routing?

I'm fairly certain it's untouched. Please let us know if you do have any issues.

https://cloud.google.com/appengine/docs/flexible/go/how-requests-are-routed#routing_with_a_dispatch_file

• If the dispatch.yaml routing isn't affected, then does that mean that tasks queued for a specific service/module will continue to route appropriately? Is it only adding tasks to the task queue that would be no longer supported on the flexible environment?

I believe so. I'm not sure if the "appspot-preview" redirect has any effect here. If it does, please let us know.

Tasks can be added using the TaskQueue API, which is currently in Alpha:

https://cloud.google.com/appengine/docs/go/taskqueue/rest/

https://godoc.org/google.golang.org/api/taskqueue (we'll be working on an idiomatic client for this API in the coming months)

Sign up link: https://cloud.google.com/appengine/docs/flexible/go/migrating#task_queues

 

I'd normally migrate back to the standard environment, however, I am utilizing some features only available through the flexible environment that I cannot (easily) migrate back from. I am using grpc, which last I checked wasn't working with the standard environment, to communicate to a server we have on-premise to exchange data with over a Compute Engine network with a VPN setup. If grpc now does work on the standard environment, there is no way for me to utilize the Compute Engine network that has a VPN connection to our on-premise network to communicate over, and would instead have to open up our internal network publically for a grpc client on AppEngine standard. 

gRPC should work with the Standard environment using the Sockets API. Unfortunately, you'll need to create a new gRPC client for each incoming request, since you must use it in conjunction with a context associated with an incoming request (appengine.NewContext).

Here's how we do it in our cloud.google.com/go/... client libraries:

https://github.com/google/google-api-go-client/blob/master/transport/dial_appengine.go#L29-L33

However, you'll likely encounter a latency hit, and the Socket API might well end up being more expensive. We're looking at ways to make these libraries work better on standard (e.g., removing the need to use the Socket API).

 

We use a mix of standard & flexible services for various parts of our application so maintaining that both sides can communicate with one another and to the same backend database is important. Sharing code between these services/modules is also important but seems as if it will no longer be possible due to the change in import paths. The change to the 'google.golang.org/appengine' import path was useful in that we could unify our code between Standard/Flexible, but now its a shame that effort is being thrown away and we again have to separate the codebase. Would it make sense (and be possible?) to just utilize the new import path for all parts that need to be shared, like datastore access?

The cloud.google.com/go/... packages should be functional on both on standard and flexible. If you encounter any problems, please file bugs.

Cheers

Chris

[Strom]

Regarding NDB-like caching layers (e.g. nds, goon), just adding support for flex-specific caches isn't enough if flex code can't access standard memcache or standard code can't access the flex cache. These libraries clear the cache with datastore writes, which means they have to clear *all caches* upon writes. I guess some HTTP based protocol could be hand-crafted which proxies the cache clear, however at this point there comes the question why Google can't do this efficiently behind the scenes and just surface the same old memcache API which connects to the standard appengine memcache.

If cloud.google.com/go works on both flex & standard, does that mean that google.golang.org/appengine is on its way to deprecation? If not, are there plans to at least convert the duplicated functionality (e.g. datastore) into shims that call cloud.google.com/go under the hood? Because these two implementations don't seem compatible. I just got done submitting a bugfix today.

[pedro mg]

Hi Chris, 

On Tuesday, November 29, 2016 at 8:04:41 PM UTC, Chris Broadfoot wrote:

On Mon, Nov 28, 2016 at 2:30 PM, Prateek Malhotra <some...@gmail.com> wrote:

Here are some issues I have and want to know if there is anything on the roadmap to ease the transition:

• I use the nds package for the memcache features it provides. I don't see any way to continue using this package in my module/service in the new environment, even if it does add memcache support later on. Is that correct?

This is a great package, I agree. You're right - we'd need nds (or a similar package) to support some arbitrary caching layer, backed by cloud.google.com/go/datastore rather than the google.golang.org/appengine/datastore package.

is there a better timeline for AppEngine Flexible to have Memcache ? I know this is a hard question but from the docs: 

https://cloud.google.com/appengine/docs/flexible/go/migrating

"The Memcache service is currently not available for the App Engine flexible environment. An alpha version of the memcache service will be available shortly. If you would like to be notified when the service is available, fill out thisearly access form. "

it looks its almost there. Filled the form. Yet this was March 17. Just trying to adjust plans as possible here. 

We have a service that needs to request data from an endpoint (that we can't control) that takes 5m to reply. AppEngine Standard request times out, but Flexible wont. We then need to feed Datastore and Memcache so that other services on the same Project can use the data. 

Also, Go 1.6 on Standard is getting quite "old". Our most recent example, on Go 1.8 the mail.ParseAddress has different behavior from Go 1.6. 

e, err := mail.ParseAddress("Alice <alice@exam   ple.com>")

Go 1.6 does not return error and e.Address == "alice@exam"

Go 1.8 returns error.

For us, a mix of Flexible + Standard services per Project is very good (for the nature of faster auto scaling on Standard, etc), specially if they share API's (Datastore, Memcache, etc). Flexible for some workers is pretty great.

Thanks for all the incredible work,

 

Pedro