go112 runtime support with dev app server

1,383 views
Skip to first unread message

dir Ls

unread,
May 28, 2019, 1:56:22 AM5/28/19
to google-appengine-go
When will the dev app server start supporting go112 runtime? At the moment I am getting "RuntimeError: Unknown runtime 'go112';" error


Constantin Konstantinidis

unread,
May 31, 2019, 12:41:45 AM5/31/19
to google-appengine-go
I understand that you're asking about dev_appserver.py. It works for me iwhen keeping go111 in your yaml.
When printing runtime.Version(), the latest displays:
>dev_appserver.py ./src
INFO     2019-05-31 06:38:19,315 devappserver2.py:278] Skipping SDK update check.
INFO     2019-05-31 06:38:19,563 api_server.py:275] Starting API server at: http://localhost:53413
INFO     2019-05-31 06:38:19,566 instance_factory.py:178] Building with dependencies from go.mod.
INFO     2019-05-31 06:38:19,575 dispatcher.py:256] Starting module "default" running at: http://localhost:8080
INFO     2019-05-31 06:38:19,581 admin_server.py:150] Starting admin server at: http://localhost:8000
2019/05/31 06:38:24 Running go1.12.5
INFO     2019-05-31 06:38:24,829 instance.py:294] Instance PID: 30108

dir Ls

unread,
May 31, 2019, 12:59:27 AM5/31/19
to google-appengine-go
Thank you Constantin. Seems like a higher go version can run a lower version runtime and that's how it's working for you. But I want to test my app with go112 runtime as it's supposed to have some additional restrictions. The following doc says this runtime is available and that is what I want to test.

Prateek Malhotra

unread,
May 31, 2019, 11:34:02 AM5/31/19
to google-appengine-go
My apologies if I'm incorrect here - but I don't think you need dev appserver starting with go 1.12 - it's meant to run any standard/typical Go application and it removes access to all standard runtime APIs (User/Memcache/Search/etc.) and features (e.g. the appengine build tag is ignored) - what value does it provide with these in mind that you'd like to use it?

The only thing I can think of is multiple services and the dispatch.yaml - What I've done is keep any such paths configurable via env variables and develop locally with different paths than what I deploy to production, running each service on a seperate port


Thank you,
Prateek

Constantin Konstantinidis

unread,
Jun 1, 2019, 1:02:02 AM6/1/19
to google-appengine-go
Moving to 1.12 improves many things (https://golang.org/doc/go1.12).

dev_appserver.py is indeed failing with:

sorted(repr(k) for k in runtime_factories.FACTORIES))))
RuntimeError: Unknown runtime 'go112'; supported runtimes are 'custom', 'go', 'go111', 'java', 'java7', 'java8', 'php55', 'php72', 'python', 'python-compat', 'python27', 'python37'.

Faillure seems to indicate that runtime: go112 is unsupported for now although using latest gcloud SDK:
>gcloud version
Google Cloud SDK 248.0.0
app-engine-go
app-engine-python 1.9.85
bq 2.0.43
cloud-datastore-emulator 2.1.0
core 2019.05.24
docker-credential-gcr
gcloud
gsutil 4.38
kubectl 2019.05.24


Constantin Konstantinidis

unread,
Jun 11, 2019, 4:09:22 PM6/11/19
to google-appengine-go
RuntimeError: Unknown runtime 'go112'; supported runtimes are 'custom', 'go', 'go111', 'java', 'java7', 'java8', 'php55', 'php72', 'python', 'python-compat', 'python27', 'python37'.

>gcloud version
Google Cloud SDK 250.0.0
app-engine-go
app-engine-python 1.9.86
bq 2.0.43
cloud-datastore-emulator 2.1.0
core 2019.06.07
docker-credential-gcr
gcloud
gsutil 4.38
kubectl 2019.06.07

Steven Buss

unread,
Jun 11, 2019, 4:15:37 PM6/11/19
to Constantin Konstantinidis, google-appengine-go
> I don't think you need dev appserver starting with go 1.12 - it's meant to run any standard/typical Go application and it removes access to all standard runtime APIs

Prateek is correct -- there's no need to use dev_appserver.py with Go 1.12 and beyond. dev_appserver is useful to spin up fake versions of the legacy App Engine APIs, which is not useful for Go 1.12 because those APIs are no longer supported. If you need those APIs, please continue to use runtime: go111 and dev_appserver will work as you expect.

But also, please upgrade to cloud.google.com/go based libraries rather than the google.golang.org/appengine libraries. Your code will be completely portable and no longer constrained to run only on App Engine.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/84bcc7aa-eea1-4a54-9f16-8dc82dba9f5d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Constantin Konstantinidis

unread,
Jun 11, 2019, 4:18:08 PM6/11/19
to google-appengine-go
Everything works fine by running inside the main directory but there is indeed no cloud emulation:
>go run .

dir Ls

unread,
Jun 11, 2019, 11:04:27 PM6/11/19
to google-appengine-go
Running as a standard/typical go app is fine. However, running it via dev app server allowed me to a) use multiple services b) dispatching c) auto-restart on changes to files including front-end files outside the go app d) performance testing as dev app server was launching several instances automatically based on the load.

I wish that this is only temporary and eventually go 1.12 runtime will be supported with dev app server to get all those benefits. Otherwise, this is a big setback.


On Tuesday, June 11, 2019 at 1:15:37 PM UTC-7, Steven Buss wrote:
> I don't think you need dev appserver starting with go 1.12 - it's meant to run any standard/typical Go application and it removes access to all standard runtime APIs

Prateek is correct -- there's no need to use dev_appserver.py with Go 1.12 and beyond. dev_appserver is useful to spin up fake versions of the legacy App Engine APIs, which is not useful for Go 1.12 because those APIs are no longer supported. If you need those APIs, please continue to use runtime: go111 and dev_appserver will work as you expect.

But also, please upgrade to cloud.google.com/go based libraries rather than the google.golang.org/appengine libraries. Your code will be completely portable and no longer constrained to run only on App Engine.

On Tue, Jun 11, 2019 at 1:09 PM Constantin Konstantinidis <constantinko...@gmail.com> wrote:

RuntimeError: Unknown runtime 'go112'; supported runtimes are 'custom', 'go', 'go111', 'java', 'java7', 'java8', 'php55', 'php72', 'python', 'python-compat', 'python27', 'python37'.

>gcloud version
Google Cloud SDK 250.0.0
app-engine-go
app-engine-python 1.9.86
bq 2.0.43
cloud-datastore-emulator 2.1.0
core 2019.06.07
docker-credential-gcr
gcloud
gsutil 4.38
kubectl 2019.06.07

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

Henry Henderson

unread,
Jun 12, 2019, 3:43:41 AM6/12/19
to google-appengine-go
Like dir Ls, I am also wondering what the recommended replacement for this is? Simply running "go run ." is not a viable out-of-the-box replacement for the devappserver when your product uses many of the GCP services. We use the datastore, memcache, pub/sub, task queues, cron jobs, multiple services in AppEngine utilizing the dispatch to route properly, as well as instance scaling. All of these are immensely integral to our system and rebuilding all of these solutions just so we can run locally when in development seems to be counterproductive on a huge scale. I don't understand why there is not more discussion regarding this when there is no proposed replacement.

I can't see realistically being able to move off of Go 1.9 without spending several sprint cycles re-engineering many parts of our system to use the new gcloud APIs and then trying to find how to build a new dev environment to mimic or step-in for the sudden massive loss of tooling that devappengine provided.

How are others solving this? Have I really missed something?

Prateek Malhotra

unread,
Jun 12, 2019, 2:39:31 PM6/12/19
to Henry Henderson, google-appengine-go
I've begrudgingly accepted this, I had to do a lot when moving from Managed VMs ==> Flexible Runtime and since my codebase was utilizing a non-shared environment early on I was able to design my application in a way better suited to not needing the local development environment. My advice:
  • Migrate to the new cloud packages - it hopefully isn't as big a change as you're thinking, and you get additional benefits (e.g. datastore you get nested repeated fields, portability, mutations, etc.)
    • Datastore -> straightforward (mostly)
    • Logging -> Maybe I'll throw my project up on github, but I basically replicated the old App Engine logging API on the StackDriver logging library and no code changes except for pointing to the new package and adding a middleware necessary
    • Tracing -> I think aside from getting the client configured, its just a middleware provided by the OpenCensus package
    • Mail -> I long ago switched to SendGrid
    • urlfetch -> This was always annoying, switching to the standard http package is refreshing
    • search -> I made a "API" hosted on Go 1.11 where I can still access the old search features and have my code send requests for indexing/searching here - I'm not proud and it's the only API I haven't been able to break away from yet
  • Loss of Dispatch.yaml -> make such URL's configurable with env variables or .env.prod and .env.dev for SPAs and run additional services on different ports
    • e.g. My API is accessible via /api in my dispatch.yaml, so on production my URL prefix via an ENV variable (kept in my app.yaml) is HOSTNAME/api, but locally I just run the api module on a different port (e.g. 8083) and locally the ENV is set to localhost:8083
  • User authentication -> There are several middlewares available for OIDC/OAuth2 you can try and configure/plugin
  • Datastore -> there's a local emulator you can use as part of the gcloud cli, the new packages are wired to use this when env vars are set, or point to a test project datastore
  • Tasks -> No substitute, unfortunately - I just test if I'm in a local dev environment and do a standard URL request (works for my needs)
  • Cron -> Go to manually
  • Memcache -> Switch to redis/memcached, you can spin this up locally and use something like redis labs in prod
  • pub/sub -> There's a local emulator (there wasn't anything provided by dev_appserver, was there?)
  • Storage -> No substitute, unfortunately, local devs get to use real buckets in a test project,  just set GOOGLE_APPLICATION_CREDENTIALS
I personally made a few projects: One for Dev, one for Testing, one for Staging, and one for Production

I know this probably isn't a general solution - I'm sure there are intricacies to everyone's projects that makes them more difficult to break out from dev app server (I have one such project in Python forever tied to ndb). Regardless, I hope this helps!

Thank you,

Prateek Malhotra


To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/kHxZ9zIb_QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/3ee1838e-afa5-4d02-83b7-e785e6cb24c0%40googlegroups.com.

Prateek Malhotra

unread,
Jun 12, 2019, 2:45:40 PM6/12/19
to Henry Henderson, google-appengine-go
Forgot to mention - there are libraries out there for hot reloading go apps (air, gin, etc.) and instance scaling locally doesn't seem like the right place to test this (maybe I'm wrong) - you're local dev boxes are probably not replications of what prod looks like, so you should probably be doing load tests and the like on a staging project/app. My opinion anyway.

Prateek Malhotra

Chee How Chua

unread,
Jun 26, 2019, 1:42:34 PM6/26/19
to google-appengine-go
How about the static directories? Does it mean that you have to main the app.yaml config and your source code for the static files and directories?
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/kHxZ9zIb_QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.

Matti Dahlbom

unread,
Nov 11, 2019, 11:19:54 AM11/11/19
to google-appengine-go
Would also be interested in good ideas on how to serve static files (eg. .html, .js) in a local dev setup in go112 runtime, so that extra code would not be needed / would not contradict handlers defined in app.yaml.

Ronoaldo José de Lana Pereira

unread,
Nov 11, 2019, 5:52:36 PM11/11/19
to Matti Dahlbom, google-appengine-go
Hi!

I'm glad my team and I are not alone with some pain points over the migration to 1.12 and beyond. I am pretty frustrated with one particular key feature from App Engine specifically that I loved, and defended for many years ("many years" as in "since the product was beta back in 2010, before GCP"): the fully contained dev environment, emulating the services you use in production for both dev and tests.

My understanding is that GCP has too many services now, and there is no practical way to provide enough emulators for them all; so we are now being directed to create dev/test cloud projects for building the apps and to test and consume the services. This is great in several cenarios, but comes with consequences... I now have to manage billing for each dev/test project that each of my team members will use. We may risk skyrocketing costs if we dont place watchdogs properly. Also, we may have development disrupted due to billing issues and budget constraints. These are new problems we were not facing before.

I do understand the value of a easy take out solution. But honestly, it all comes down to a trade off. We traded-off using GCP services because that accelerated our team development (specially because of the zero-to-hero with the AE SDK setup) at the expense of a vendor specific hosting platform and database. GAE is losing much of it's value, and will eventually be reduced to "a fancy managed Kubernetes", making less and less sense to use it...

I'm open for ideas/clarifications. As a long-time App Engine happy user, I'm really concerned about what is comming next. But maybe I'm just not understanding the benefits of the trades being made here?

Best regards,

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/kHxZ9zIb_QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/a1f67192-2860-4701-80ba-ad062bcf2f0d%40googlegroups.com.


--
Ronoaldo Pereira

Chaoming Li

unread,
Nov 11, 2019, 7:53:56 PM11/11/19
to google-appengine-go
Same here, it's painful to develop under 1.12. I am actually running dev_server.py on 1.11 still so at least I can have a local app engine running.

I think at least there should be a local app engine for 1.12 without any emulator, and it should allow developers to put in a local key file of the service account to connect to the GCP resource. It will be as slow as hell but at least it will be faster than deploying to a dev app engine in GCP.

As GCP uses a lot of open source project, it would not be too bad for this solution. Developers can spin up local Redis to replace GAE memcache, etc.

Cheers,
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/kHxZ9zIb_QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.


--
Ronoaldo Pereira

Sam Thorogood

unread,
Nov 12, 2019, 2:50:26 PM11/12/19
to Matti Dahlbom, google-appengine-go
I've written nicehttp which includes a runner for go112's app.yaml file (see godoc) in development, it parses and matches static/static_dir etc. The package includes other helpers; if the App Engine part alone was interesting I could break it into its own package.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/kHxZ9zIb_QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengin...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/a1f67192-2860-4701-80ba-ad062bcf2f0d%40googlegroups.com.

Ronoaldo José de Lana Pereira

unread,
Nov 12, 2019, 9:35:41 PM11/12/19
to Sam Thorogood, Matti Dahlbom, google-appengine-go
This looks quite cool! Would like to contribute... What is your plan for gae sub-folder?

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine-go/CA%2B2D%2Br0WiBgkU1a%3DZQ_wysrOBjG%2BY7sEfb0BpJ3FTKaLc0iQig%40mail.gmail.com.


--
Ronoaldo Pereira

Jens-Uwe Mager

unread,
Nov 17, 2019, 1:30:45 PM11/17/19
to google-appengine-go
For me migrating anything beyond 1.11 is also much too painful, and my biggest pain point is the missing memcache support. Basically leaving that out means I have to develop my own support. And this means also that I am really developing against some other api's on my development machine and will attempt to mimic that development environment very close on deployment. Apparently this is not necessarily gcp any longer.

-Jens-Uwe

Chee How Chua

unread,
Nov 20, 2019, 11:51:20 PM11/20/19
to google-appengine-go
Nice! Thanks for sharing!
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "google-appengine-go" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine-go/kHxZ9zIb_QE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengine-go+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "google-appengine-go" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine-go+unsub...@googlegroups.com.

Chee How Chua

unread,
Nov 21, 2019, 12:00:02 AM11/21/19
to google-appengine-go
It is. The pain point for me is the removal of the Search API. By removing all the auxiliary services (Search, memcache, tasks, mail, etc.), to me it seems App Engine is no longer a PaaS (platform as a service) offering. If they also remove the auto-scaling capability, that would be the final straw.

For now, it seems the auxiliary services are where they draw the line. And I'm grateful that Google is still offering a free tier for poor developers to test out ideas.
Reply all
Reply to author
Forward
0 new messages