New App Engine Managed VM Docs

[Amir Rouzrokh]

Hi everyone,

I’m one of App Engine’s Product Managers and we’ve just pushed out a new iteration of our App Engine Managed VM docs located at

https://cloud.google.com/appengine/docs/managed-vms/

Give it a try please and send over feedback including any more developer guides you would like to see, any errors, etc. The feedback button is on the top right hand side of every page. Thanks in advance!

[Karl MacMillan]

On Feb 22, 2016, at 1:18 PM, 'Amir Rouzrokh' via Google App Engine <google-a...@googlegroups.com> wrote:

Hi everyone,

I’m one of App Engine’s Product Managers and we’ve just pushed out a new iteration of our App Engine Managed VM docs located at

https://cloud.google.com/appengine/docs/managed-vms/

1. Are there plans to port over NDB for datastore access on managed VMs?

2. Is webapp2 going to be deprecated?

I ask because both of those seem to not be getting much attention with the Managed VM transition. It’s particularly concerning to me because I used both webapp2 and NDB as those were the defaults presented in the App Engine documentation for python (and still are). But now I see no signs of NDB moving to Managed VMs and you are using Flask in these examples.

FWIW - I don’t have a preference about about NDB vs. the new datastore API in gcloud except that I have lots of NDB code that I don’t really want to port. As for webapp2 - I feel really burned by that choice. I only went with it because I thought it would be better supported, but it is far from the best option and not even maintained as far as I can tell. I absolutely think you should push Flask or something similar and just publicly declare that you won’t be maintaining webapp2.

Karl

[Jon Parrott]

Are there plans to port over NDB for datastore access on managed VMs?

There is on-going discussion about NDB for Cloud Datastore here. The tl;dr is that basically it's possible, it's planned, but its contingent on the next release of the Cloud Datastore API and decoupling NDB from the App Engine SDK.

That being said, you can absolutely continue use NDB in Managed VMs by using runtime: python-compat. 

Is webapp2 going to be deprecated

There are no present plans that I'm aware of to deprecate webapp2. Because webapp2 is a WSGI-compliant web framework and is not itself tied to App Engine, you can use it in Standard, Managed VMs, and pretty much anywhere else. It'll continue to work everywhere WSGI works. It's also an open-source project and patches are always welcome.

The choice of Flask for our samples in Managed VMs is to more closely match the broader Python community. In that same effort, we're also creating Django samples.

 only went with it because I thought it would be better supported, but it is far from the best option and not even maintained as far as I can tell. I absolutely think you should push Flask or something similar and just publicly declare that you won’t be maintaining webapp2.

We're recommending developers use what they are comfortable and familiar with and we're working to make sure that common tools and frameworks work on our platform.

[Karl MacMillan]

On Feb 22, 2016, at 3:44 PM, 'Jon Parrott' via Google App Engine <google-a...@googlegroups.com> wrote:

Are there plans to port over NDB for datastore access on managed VMs?

There is on-going discussion about NDB for Cloud Datastore here. The tl;dr is that basically it's possible, it's planned, but its contingent on the next release of the Cloud Datastore API and decoupling NDB from the App Engine SDK.

Great.

That being said, you can absolutely continue use NDB in Managed VMs by using runtime: python-compat. 

Is that new? Last I looked I believe NDB was not supported. If I could suggest it would be helpful to use more specific terminology than Datastore on that page.

Is webapp2 going to be deprecated

There are no present plans that I'm aware of to deprecate webapp2. Because webapp2 is a WSGI-compliant web framework and is not itself tied to App Engine, you can use it in Standard, Managed VMs, and pretty much anywhere else. It'll continue to work everywhere WSGI works.

Sure - I currently use it on GCE as well just to not use two frameworks. But it is abandoned as far as I can tell - there are bugs that have not been addressed in years.

It's also an open-source project and patches are always welcome.

I’m sorry, but that’s a little disingenuous. As far as I can tell there is _no_ activity on webapp2. I submitted a bug report that pointed to an exact line of code that needed to be changed but there was no response (https://code.google.com/archive/p/webapp-improved/issues/102). It’s not clear that there is any process to submit patches (contributing on the website has no information - https://webapp-improved.appspot.com/#contribute), there is no active mailing list / discussion forum, the bug tracker is just an archive on google code, and I can see no evidence that there is anyone other than a Google employee that can commit.

So - I’ll call your bluff. Tell me how to submit a patch for the issue I pointed to and I’ll happily do so.

Otherwise, please consider declaring that the project is abandoned and stop pointing people to webapp2 in App Engine documentation since it’s a dead end.

Karl

[Jon Parrott]

> Is that new? Last I looked I believe NDB was not supported. If I could suggest it would be helpful to use more specific terminology than Datastore on that page.

It's not new as far as I know - ndb and db have traditionally be the way to access datastore in App Engine, so it's somewhat implied. I'll see if I can't make the docs a bit more clear there.

> So - I’ll call your bluff. Tell me how to submit a patch for the issue I pointed to and I’ll happily do so.

You make some excellent points. I will move webapp2 to github, take ownership of it, and I'll be happy to review and merge your patch as well as publish a new version of webapp2 to PyPI. This process takes some time, so I will update this list once that is done.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/HazEmXZRpiw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/AF77E164-0058-455F-9679-87FDB89C1926%40strajillion.com.
For more options, visit https://groups.google.com/d/optout.

[Karl MacMillan]

On Feb 22, 2016, at 4:14 PM, 'Jon Parrott' via Google App Engine <google-a...@googlegroups.com> wrote:

> Is that new? Last I looked I believe NDB was not supported. If I could suggest it would be helpful to use more specific terminology than Datastore on that page.

It's not new as far as I know - ndb and db have traditionally be the way to access datastore in App Engine, so it's somewhat implied. I'll see if I can't make the docs a bit more clear there.

Hmm - maybe I was wanting ndb in a non-compat python runtime. It’s been a while.

> So - I’ll call your bluff. Tell me how to submit a patch for the issue I pointed to and I’ll happily do so.

You make some excellent points. I will move webapp2 to github, take ownership of it, and I'll be happy to review and merge your patch as well as publish a new version of webapp2 to PyPI. This process takes some time, so I will update this list once that is done.

Hah - fair enough. I’ll send you that patch when you’re ready.

BUT - if you ask me, you really should just put webapp2 into maintenance only mode and suggest people move on. There are no advantages over more popular frameworks and plenty of downsides.

One note - last I looked much of the session handling code was completely broken (https://webapp-improved.appspot.com/api/webapp2_extras/sessions.html). I ended up just rolling my own after my initial bug report was ignored.

Karl

On Mon, Feb 22, 2016 at 1:01 PM Karl MacMillan <ka...@strajillion.com> wrote:

On Feb 22, 2016, at 3:44 PM, 'Jon Parrott' via Google App Engine <google-a...@googlegroups.com> wrote:

Are there plans to port over NDB for datastore access on managed VMs?

There is on-going discussion about NDB for Cloud Datastore here. The tl;dr is that basically it's possible, it's planned, but its contingent on the next release of the Cloud Datastore API and decoupling NDB from the App Engine SDK.

Great.

That being said, you can absolutely continue use NDB in Managed VMs by using runtime: python-compat. 

Is that new? Last I looked I believe NDB was not supported. If I could suggest it would be helpful to use more specific terminology than Datastore on that page.

Is webapp2 going to be deprecated

There are no present plans that I'm aware of to deprecate webapp2. Because webapp2 is a WSGI-compliant web framework and is not itself tied to App Engine, you can use it in Standard, Managed VMs, and pretty much anywhere else. It'll continue to work everywhere WSGI works.

Sure - I currently use it on GCE as well just to not use two frameworks. But it is abandoned as far as I can tell - there are bugs that have not been addressed in years.

It's also an open-source project and patches are always welcome.

I’m sorry, but that’s a little disingenuous. As far as I can tell there is _no_ activity on webapp2. I submitted a bug report that pointed to an exact line of code that needed to be changed but there was no response (https://code.google.com/archive/p/webapp-improved/issues/102). It’s not clear that there is any process to submit patches (contributing on the website has no information - https://webapp-improved.appspot.com/#contribute), there is no active mailing list / discussion forum, the bug tracker is just an archive on google code, and I can see no evidence that there is anyone other than a Google employee that can commit.

So - I’ll call your bluff. Tell me how to submit a patch for the issue I pointed to and I’ll happily do so.

Otherwise, please consider declaring that the project is abandoned and stop pointing people to webapp2 in App Engine documentation since it’s a dead end.

Karl

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/HazEmXZRpiw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/AF77E164-0058-455F-9679-87FDB89C1926%40strajillion.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAN6JEymD%2BEv3jnS7KRr6UN4zBwaHVogNuPrA78Wnq2%2B%2BXV%2B6TA%40mail.gmail.com.

[Jon Parrott]

> if you ask me, you really should just put webapp2 into maintenance only mode and suggest people move on. There are no advantages over more popular frameworks and plenty of downsides.

In general we want to align with the Python community. Whether we update our docs to point to Flask over webapp2 is currently a point of discussion - but I'm personally for it. No matter what we do there, there should be some way for people who are currently using webapp2 currently to provide patches.

Putting webapp2 in maintenance only mode is not currently our decision as we do not own the package (it's a third-party package). If we do gain ownership of it, we will likely put it into a community-supported model.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6C7FB369-90C4-4866-9C24-7F21D1E68ADD%40strajillion.com.

[Karl MacMillan]

On Feb 22, 2016, at 4:40 PM, 'Jon Parrott' via Google App Engine <google-a...@googlegroups.com> wrote:

> if you ask me, you really should just put webapp2 into maintenance only mode and suggest people move on. There are no advantages over more popular frameworks and plenty of downsides.

In general we want to align with the Python community. Whether we update our docs to point to Flask over webapp2 is currently a point of discussion - but I'm personally for it.

If you don’t make the change (or maybe until), can I strongly suggest you include some discussion that makes it clear the level of support for webapp2. Like I said - I went with that option on the mistaken impression that it was well supported by Google. That’s the _only_ thing that irritates me about this whole situation. I believe the docs lead me down a path without warning me of the true consequences. Even a basic “webapp2 is community supported framework” description would make it clearer.

No matter what we do there, there should be some way for people who are currently using webapp2 currently to provide patches.

I’d certainly appreciate that. Despite my complaints, I’m not itching to port all of my code off of webapp2 because it’s working fine.

Karl

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAN6JEyk4XWK9D35uF%2B_33q5y6iX8p%2BVx5YJ6q8sGUVvpNbqr_Q%40mail.gmail.com.

[Alex Martelli]

Just a personal input/data point -- writing not as a Googler but as a "Python guru":-)

Currently writing the 3rd edition of Python in a Nutshell (in my copious personal time), I've recently finished the "serving HTTP" chapter (the book is already available in Early Release form at the O'Reilly website, check it out:-). For the purpose, I thoroughly tested lightweight frameworks (rich frameworks such as django, web2py, and the like, though I mentioned them, aren't really "my thang"). Specifically: falcon, flask, bottle, and webapp2.

Of the four, webapp2 remains my favorite (admittedly I started out knowing it much better than the others, which may have biased me). Next in order of preference, falcon, bottle, flask -- all good, but the most popular, flask, is accruing so much extras that it barely qualifies as "lightweight" any more.

If webapp2 becomes easy to offer patches for again, I'll spend personal time (once the Nutshell is finished -- give me a few more months!-) helping there -- indeed I'll volunteer to vet pull requests if gurus to vet them are needed -- to help keep out the kinds of deep integration with unrelated components (such as the admittedly superb templating framework jinjia2) which are fattening up flask:-(.

I sure hope (very personal preference) that webapp2 isn't put out to pasture... should it be, I might either fork it, or give up and move to falcon for new projects (and maybe volunteer to help out on THAT framework).

Again, just personal preferences -- but, ones based on almost 20 years' experience with Python (starting with a personal website project using CGI) and on being a Fellow of the PSF and a winner of the Frank Willison Memorial Award for contributions to Python:-).

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6577D566-C74D-41C5-AD75-B479B8B64246%40strajillion.com.

[troberti]

This is encouraging to hear and hope it happens. I like webapp2 just fine as it is: simple, stable, easy to test, *just works*. The fact that it doesn't 'evolve' is imho a good thing. Bug fixes are of course always welcome.

[Robert Fox]

Hi Amir I really appreciate the documentation.  I used to know it like the back of my hand but lately I've noticed a lot of changes.  Personally I'd like to see more documentation that gets to the delineation between Compute Engine / Managed VMs / App Engine.  Specifically when it comes to services.  

Some services like Datastore work quickly and without authentication in both Managed VMs (java-compat at least) and traditional App Engine.  What is the performance cost of using that service from Compute Engine?  I can test it myself but some examples would be awesome.  

With other services like Cloud SQL the Managed VM is much more like Compute Engine... you have to have a strategy for authentication.  And for my usage performance of Cloud SQL is relatively the same across offerings.

I know these offerings blur--and as a developer I love choice... but I think clear documentation and for me especially some insight to the service/API roadmap would be welcome.  Will Search API be usable from compute engine?[*]  Will Managed VMs eventually scale to custom metrics?  How does scaling in compute engine compare to scaling for Managed VMs?  Is datastore access from Compute Engine and Managed VM environment comparable?  These are the types of questions I think developers need to get on the right platform. 

I'm currently moving from Managed VM over to Compute Engine.  Some things about that decision still leave me uneasy.  I love some of the API offerings and I don't want to lose out on performance.  Roadmaps!  App Engine rocks.  Keep pushing.

[Adam Sah]

hi Amir - awesome stuff!  Question about backup/restore and migration.

I have a 5 year old GAE/python2.7 app that I'd like to port, but don't see how to migrate the data (180+GB).  I tried the datastore admin copy-from-backup system, but it couldn't read the files from cloud storage (/gs/...mybucket...), even after setting permissions on the bucket to world readable.  Am I doing something wrong?

https://cloud.google.com/appengine/docs/managed-vms/python/migrating-an-existing-app

adam

[Jon Parrott]

You don't actually need to create a new project to use Managed VMs. You can deploy to a new version or module within an existing application.

Though we do highly recommend testing and monitoring your application before serving public traffic.

--

You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/HazEmXZRpiw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ad33fcdd-ea21-4b0e-9c7c-95a6352fe742%40googlegroups.com.

[Adam Sah]

thanks!  but alas, I'm getting an error trying to setup gcloud...

$ gcloud init

Welcome! This command will take you through the configuration of gcloud.

Settings from your current configuration [default] are:

Your active configuration is: [default]

[app]

suppress_change_warning = true

[core]

account = [... my email ...]

disable_usage_reporting = False

project = [... existing GAE project ID ...]

Pick configuration to use:

 [1] Re-initialize this configuration [default] with new settings

 [2] Create a new configuration

Please enter your numeric choice:  1

Your current configuration has been set to: [default]

Pick credentials to use:

 [1] [... my email ...]

 [2] Log in with new credentials

Please enter your numeric choice:  1

You are now logged in as: [... my email ...]

Pick cloud project to use:

 [1] ...

 [19] ...

Please enter your numeric choice:  6

Your current project has been set to: [... existing GAE project ID ...].

You can use [gcloud config] to change more gcloud settings.

Your active configuration is: [default]

[app]

suppress_change_warning = true

[core]

account = [... my email ...]

disable_usage_reporting = False

project = [... existing GAE project ID ...].

ERROR: (gcloud.init) Failed command: [compute zones describe us-central2-tmpgae] with exit code [1]

any ideaS?

[Jon Parrott]

Just a quick update on the webapp2 situation. The original author has graciously allowed us to take ownership. My team is working on getting everything moved to github and making sure that we can properly test and release new versions. I'll make an announcement on this group when the github repository is ready. At that point, we'll be able review and accept pull requests from the community to maintain webapp2.

[Karl MacMillan]

On Mar 4, 2016, at 1:57 PM, 'Jon Parrott' via Google App Engine <google-a...@googlegroups.com> wrote:

Just a quick update on the webapp2 situation. The original author has graciously allowed us to take ownership. My team is working on getting everything moved to github and making sure that we can properly test and release new versions. I'll make an announcement on this group when the github repository is ready. At that point, we'll be able review and accept pull requests from the community to maintain webapp2.

Thanks for the update - looking forward to some updates.

One of the missing pieces for some of the problems that I encountered is a modern, native (i.e., fast) implementation of modern password hashing (e.g., PBKDF2, SCrypt, or BCrypt). Am I just missing a python module for this that is available for App Engine?

Thanks,

Karl

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/1bd152ab-9c3b-49ea-ae4b-b235a59e438a%40googlegroups.com.

[Adam Sah]

found it:   there's no us-central2-tmpgae zone, who knows how that got misconfigured?

[Alex Martelli]

On Fri, Mar 4, 2016 at 11:15 AM, Karl MacMillan <ka...@strajillion.com> wrote:

On Mar 4, 2016, at 1:57 PM, 'Jon Parrott' via Google App Engine <google-a...@googlegroups.com> wrote:

Just a quick update on the webapp2 situation. The original author has graciously allowed us to take ownership. My team is working on getting everything moved to github and making sure that we can properly test and release new versions. I'll make an announcement on this group when the github repository is ready. At that point, we'll be able review and accept pull requests from the community to maintain webapp2.

Thanks for the update - looking forward to some updates.

One of the missing pieces for some of the problems that I encountered is a modern, native (i.e., fast) implementation of modern password hashing (e.g., PBKDF2, SCrypt, or BCrypt). Am I just missing a python module for this that is available for App Engine?

You should be able to use the pure-python, open-source https://github.com/dlitz/python-pbkdf2/blob/master/pbkdf2.py -- App Engine does supply the underlying pycrypto which this pure-python module imports (you must request it in the libraries: stanza of your app.yaml -- or yaml for your GAE module if you need PBKDF2 for use in such a module).

As always, to use pure-python modules in your GAE app, you just need to add them to your source tree.

Alex

 

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/82679F8D-836B-4DC3-9C27-1CD16F316F39%40strajillion.com.

[Jon Parrott]

For those of you interested in webapp2, I've posted an update here.