Diving into App Engine is soooooooooooo hard now - My analysis

[Kaan Soral]

I'll whine a lot, but there's a concentrated and crisp TL;DR in the end

I've been using App Engine I guess for 10 years now? It's hard for me too, but also, I'm preparing to open source my current project, at each step I question everything, I was going to open source my game so young developers/kids/gamers could dive into networked game development easier (It's an MMORPG on Google Cloud, my dream was that open sourcing it could enable fast experimentation with the genre), but as it is, at the current state of App Engine, the bottleneck is definitely all the diversified and separated products, separate emulators (missing ones), separate environment variables for everything etc. - I can't imagine a happy scenario where one could just develop an App Engine app locally without spending weeks trying to understand what's what first - Back in the day, I think the Launcher GUI and the simplicity of it all, got us all hooked (This is coming from a CLI user)

Things used to be as simple as "python2.7 /sdk/dev_appserver.py --storage_path=/storage/ --blobstore_path=/blobstore/ --datastore_path=/storage/db.rdbms --host=0.0.0.0 --port=8080 /path --require_indexes" - We could test EVERYTHING locally, now we can almost test nothing locally

It seems that the python2.7 that's currently in the process of being deprecated was the last of it's kind

Instead of refactoring everything for Python 3, I decided to move onto NodeJS instead, the work done, the documentation, the depth of it all, on the surface, is very exciting, until you dive in

First of all, at each step, you're faced with initially arbitrary decisions that's going to eliminate you, flexible or standard? - does it really matter, I don't think so, firestore in datastore mode or firestore? oh my god, it's like each option is designed to torture and drive away a potential user

As a long time App Engine / Datastore user, I've simplified my data design to a singular dimension and simple get/set by key operations wherever possible, so for me the limitations of the new datastore really doesn't matter much, I just wanted to use the new Firestore and all the new exciting stuff that came with it, experiment with it, see how it can improve my approach, I've always designed products that regularly pinged the server for new data, so the realtime features etc. appealed to me a lot - so I'd choose the unlocked Firestore, instead of the irreversibly locked one, at least that decision was easy, but the not so easy part was to find out how I'm going to emulate it locally, authenticate it locally, use it locally

Honestly, it's been months since I've been entertaining idea, whenever I can find some free time, I re-dive back into it, but still, I achieved nothing, yesterday I finally discovered "gcloud beta emulators firestore start" - but the new "environment variable" based approach etc. is making me want to cry, so I gave up again (the examples/docs are from "firebase", the authentication/flow is from "firebase", there are no simple examples to just use and authenticate "firestore" locally) - When I'm configuring a product, I either want to configure it in-code, or configure it in .yaml files, with the new approach, even app id's are set through environment variable's ...

Just before writing this, I wondered if I'm being unjust, and wondered whether I should just pick an arbitrary service, experiment with it using NodeJS, so I picked Taskqueues, or "Cloud Tasks" as they are now called, once again, it's a seemingly well documented and seemingly a well designed product, but no matter what I did, I just couldn't find how I'm going to test it locally, as it turns out, it can't be tested/emulated locally? bummer. (Related: https://stackoverflow.com/questions/53826183/local-development-with-cloud-tasks-cloud-datastore-with-gae-with-python3)

TL;DR/Conclusion:

I believe we need a practical, all in one and turnkey local emulation solution again - There's a vast/overflowing/overwhelming amount of new options and no easy way to test them - I think the best way to learn about something is to use it

Suggestion/Takeaway:

I think local development is no longer feasible or possible, so it seems like a better idea to just deploy and test live, I don't know whether it's the intended way now, but it's seemingly the only way. In the past we assumingly all had a local development version of our products, I assume we now should start with live development versions of our products. So when I'm ready to try again, I'll experiment with it all live. (It's sad that likely this option is only possible for paying customers, I don't know whether the products have safe free tiers now, I assume not)

I also wonder whether I'm the only one who feels this way, are there any success stories out there, happy new App Engine users?

[Kaan Soral]

A small update, after my conclusion, I went ahead with my live trial and finally started a new journey

~/Desktop/APPENGINE/cloudsdk184/bin/gcloud app deploy -> after every small change

~/Desktop/APPENGINE/cloudsdk184/bin/gcloud app logs tail -> replaces the local development server - instead of instantly reading your local logs, you now trigger something, wait 5-10 seconds, and the "logs tail" pulls the live log to your Terminal after a healthy delay

I started with a simple Firestore write, received "UnhandledPromiseRejectionWarning: Error: 9 FAILED_PRECONDITION: The Cloud Firestore API is not available for Cloud Datastore projects." - then upgraded my unused Datastore to Native Firestore and saw my first data entry!

Ironically, while the Firestore advertises live capabilities, you need to refresh https://console.cloud.google.com/firestore/data manually everytime you make a change too (documents auto update, but collections don't)

I'm currently happy and excited, I just wish I came to this obvious conclusion earlier - Don't forget to limit your spending as a live environment isn't forgiving

[Brian Ruuska]

Glad to hear I'm not the only one disappointed with it all. I started using App Engine when it was in beta, and launched a one-person business when it went GA. It's so overwhelming attempting to port my product to the new environments that it's much more than a one-person effort. And with all the land mines and missing functionality I decided to just close the business last year. They seem to be more interested in serving big enterprises, and us little guys are just left behind. :-(

[Matthew Parkes]

I have to totally agree with this.  My experience has been the same coming from AppEngine Standard python 2.7 I decided to try and migrate one of my apps to Python3.7 but was extremely frustrated and put off with the whole experience.  I lost a lot of nice framework that made decisions easy and straight forward to actually just get functionality working. (Endpoints API, User management, Data management, Admin datastore).  I found it very hard to get an actual development environment setup where I could rapidly develop and test, see data in the database etc.

The local testing has become very difficult. It's sad that while google appears to say "Oh AppEngine so much more flexible for you now" sounds like a step forward, when really it's more like 1 step forward, 5 steps backwards.

Disappointed,

Matt

[Rajesh Gupta]

I have been a user of google appengine for 7-8 years.  Yes, its disappointing

The local dev console did not improve.  There is no easy way to import/export from local dev to the cloud.

I am really unhappy that they did not provide tools for migrating data into new schemas.  

First they promoted mapreduce, and they left it orphaned with no documentation and open sourced it.

My biggest scare for my project is about how to do schema changes. egs, adding new properties to the KINDS, removing unused properties etc to name a few.

Then, Google Cloud dataflow was introduced,  It is such an overkill for small teams to learn and master any big technology.   Many SaaS applications are developed for SME's.  Google cloud dataflow is promoted and recommended without knowing the end developer.   

I really wish they get back the mapreduce and help developer ecosystem.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9b64def7-8ce2-4cf3-a6c7-84aba79e7aea%40googlegroups.com.

--

Rajesh

www.servicefolder.com

Field Service Software on Google Cloud Platform and Mobile

[Vitaly Bogomolov]

+1. 

GCP.AppEngine.StandardEnvironment.SecondGeneration looks like a terminal point for me. You must stop using GAE. 10+ years of GAE use.

WBR, Vitaly.

[Nickolas Daskalou]

Same experience for us.

10 years ago App Engine was a revolution. Now it's almost revolting.

Agree with the hard to set up development environment for the newer App Engine installments.

Other pain points too, like no shared Memcache replacement (we're told NDB for Python 3 is fully ready, when in reality a lot of existing apps which rely on the built in feature of the free shared Memcache will either suffer performance issues and/or an increase in billing cost if they switch over).

Google are having a hard time working out how to crack the cloud computing market. Enterprises go with Azure because Microsoft are already heavily embedded and sell the business fluff well, and AWS have a massive head start on startups due to a larger existing user base and service offering because they were first in the game.

Nick

--

[Joshua Smith]

Anyone tried Beanstalk? Is it any better?

I’ve recently had a run-in with Azure where I had to store some stuff for a customer in their S3-like blob store service. It was a nightmare. Insanely over-engineeered to the point that it becomes impossible to figure out how to do the simplest things. Much like GAE’s trajectory.

Ima coin a new law: All PAAS services will expand until they are indistinguishable from IAAS.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAOj3zuA6P676X8EKyt22MB5G01Nv8uLOqfGm14A%2BRbNmgCMUmQ%40mail.gmail.com.

[john]

+1

I also lament the loss of the turnkey environment that app engine used to provide. I can only presume that the business model for it isn’t very profitable, otherwise someone else would reproduce it. 

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Giuliano Ribeiro]

Unfortunately I need to say the same.

Nowadays, it seems Google is forcing us to use GKE or Cloud Run, as all the App Engine "magic" gone.

I've been using App Engine my last 8years, and is clear, the Google strategy is not App Engine anymore.

Giuliano Ribeiro Cloud and Software

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/bb621c1e-5fa5-4dfa-8c7e-8911701666b8%40googlegroups.com.

[Patrice B]

Hi guys,

A few months back, I had a similar reflexion on current trends regarding AppEngine, which I called "the end of paas", which could be called "end of serverless" in a way

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!search/paas|sort:date/google-appengine/DHMMv5r8qjU/A-VO3PXPDwAJ

It's not very long, so I paste it here, since it is quite similar to what is being said here.

It seems most of the services that made AppEngine a proper Platorm as a Service are now scheduled to be shut down, and users are advised to migrate.   Migrate search to ElasticSearch, migrate memcache to Redis, and maybe at some point we'll be asked to migrate Ndb to MongoDB and GCS to whatever.   I'm not complaining about the way the process is handled actually, there is enough time to consider the options and work on a migration scenario, there is no imminent deadline, at least for the moment.   But I'm wondering what went wrong with the PaaS approach, and is it officially dead.

Is this the end of GAE as a PaaS ?   I truly believed PaaS was the future of cloud architectures:  stop thinking in terms of servers, start thinking in terms of services.   When I started working with AppEngine, I dreamed of CPU as a service, with no server granularity, and I was disappointed to find I still had to worry about servers, starting up a new server instance, choosing what type of instance would be best, a scaling strategy, etc.   I was expecting servers as a service, i.e. serving my requests without me ever thinking in terms of servers.   But at least there was Ndb, search, memcache, GCS and a few more.

Now it seems all of these are on their way out, which makes me wonder: was there something wrong with the concept of PaaS itself, or is it just that these products didn't gain traction, and are now too costly to maintain with regards to their user base ?   Actually, the one thing that was wrong with Paas from the very beginning was that it would lock a project into a given cloud.   That was a risk to be reckoned for users, but it could have been seen as a feature for the cloud provider.   Now is it for this reason that the mentionned services didn't make it ?  Because users would have been wary of being locked in, and for that reason would prefer to use leading products deployed on leased servers ?    One thing is for sure: once an application has migrated to more standard services, it will not be tied to GCP anymore.

There was a major benefit to the PaaS concept:  it was very cheap for startups.   Deploying ElasticSearch on a the smallest possible cluster will start at around $200 a month, while the search usage of a small application could cost less than $10.   Same for the shared memcache service offered by AppEngine.   Now you are having to pay for running servers all night with very little transactions to handle.

I'd be happy to hear your thoughts on this matter ?

[Joshua Smith]

It’s interesting to look at what Google is doing with Firebase.

1. They are encouraging us to move more of the processing into the client, which while often architecturally foolish, does eliminate the need for some of the PaaS stuff.

2. They have a serverless/cloud functions thing over there, which overlaps a lot with the PaaS stuff we’ve been able to do in GAE. I haven’t dug into that too much, but on the surface it looks like the spaghetti antipattern.

Given that this is google, and that it’s PaaS all over again, I think the risk of putting any eggs in the Firebase basket seems almost unbearably high. Is Google going to turn their back on Firebase when the next shiny object distracts them? Why should we believe that any service google offers will last more than their committed 3 years?

I’m super tired of having to go back and rewrite my critical business infrastructure every few years.

(Google’s competitors in the PaaS space don’t inspire any more confidence, so this isn’t so much a google problem as a cloud problem, I think.)

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6edf3459-a4d0-4ea4-9282-86cdb6a2672f%40googlegroups.com.

[Luca de Alfaro]

I will sorely miss Appengine, as it was a uniquely simple way to get reliability and scalability.

And it was unique to Google.  Other things, like containers, vms, and databases, AWS also has them.

So I think the strategy is foolish.  If Appengine is de-emphasized, I won't be confident putting resources in Firebase, which seems to be just the next Appengine.  I will move to things that historically have been more stable, and that means, I am afraid, AWS.

As much as I admire Google's technical skills, for the things I and my startups do, we need to be able to focus on developing new features rather than porting what we have from services that are going to be deprecated periodically.

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5644BD34-868F-48F8-8FD7-87B8C0FA6AD6%40gmail.com.

[Rajesh Gupta]

Please Google don't deprecate anything.  Dont' introduce new parallel way of doing things with new stuff and put the developers in confusion on the dev roadmap.

Get  back mapreduce. If I write a simple SAAS that is serving few customers and growing, why do I need  new and big and scary DataFlow technologies.  Dataflow is too enterprisish.

Following issues small companies faced for having been on Google appengine:

 - GWT is not promoted as it was 7-8 years ago

- Image API was deprecated.  It was so hard for finding equivalent things for low cost.

- Mapreduce is made open source and not promoted.  Mapreduce itself went two api iterations and then finally parked away as open source.

- A bunch of new things equivalent to what is available on appengine with new terms like 'cloud tasks'', 'cloud datastore' etc.  

Lastly, give good and easy migration UI tools for making bulk schema changes.

Improve documentation.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPgu2uKUr7mY89NYrOWG0kJa9a1V%2Bc0395-5csqq7JycE7u4nQ%40mail.gmail.com.

[Jeff Schnitzer]

As another longtime user, I have mixed feelings. Some things are better, some things are worse. I certainly wouldn't give up the current GAE to go back to the old one, but mostly because I still have enough of the old one available.

I straddle the old world and the new world, using the "old" memcache and task queue APIs. I guess if I ever want to upgrade from Java8 I'll be forced into Cloud Memorystore and Cloud Tasks. The biggest problem would be converting my test harness. I could use a live Redis that I flush between tests, but lack of an emulator for Cloud Tasks is a pretty serious problem. I'm very dependent on the LocalTaskQueueTestConfig.

Actually this is maybe a symptom of a bigger problem. In the old "integrated whole" GAE, the task queue is a way of deferring execution. The fact that there were URLs and http requests underneath was plumbing that could be ignored. 100% of my task queue usage is "execute this bit of code later", so it's really easy to write tests with an awaitTasks() method that chews through the list of deferred operations. The only level of abstraction provided by Cloud Tasks is "http request" so all the deferring and test harness logic would be up to me to implement. Ugh.

Honestly though, the old memcache and task queue APIs are so rock solid reliable, upgrading seems like a terrible idea. And there's nothing particularly compelling in Java11 (except the 10m deadlines).

On the plus side, Cloud Postgres is pretty fantastic, although balancing the number of database connections with the number of instances is really tricky with autoscaling on GAE. Under load, you either run out of JDBC connections or you run out of instances to serve user requests... if you're used to the datastore, tuning your scaling settings is now hard. PG connections are an expensive resource.

I like the new console quite a lot more than the old one, and it's getting better. The error notifications are great - nicely integrated with logging, unlike rollbar (which I no longer need to pay for).

GCS is vastly superior to the old blobstore.

I agree it's harder to get started with the current GAE offering, but I still recommend it to folks.

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CA%2BS7ijaK%3DutqkAT%2BPHYRg4sf6CteT6a4uT6wqUfNR3a40oPVqw%40mail.gmail.com.

[Alexey]

I feel like Google has made some improvements and some of the deprecations are understandable.  My biggest issue is the way Google has gone about doing this.  GAE started out as a platform coded to a number of languages' interfaces, some standard and some proprietary.  For example, it used to be possible to send emails in Java using standard javax.mail interfaces.  But as time went on, Google felt like there were opportunities and pressures to make changes to the offerings and many of these changes were made in a breaking fashion or with alternatives that didn't add up to the functionality they were replacing.

From day 1, when compared to AWS, Google's documentation felt a bit disjointed, like a patchwork of output from different teams that don't talk to each other.  Now that there's this push to move everything under GCP tooling, this disjointed nature of documentation and sense of lack of direction feels even worse.  I don't mean to downplay GAE's strengths or discourage people from using it, but undoubtedly it feels like you can't put too much trust in Google's willingness to support services or interfaces you build against.  So as a long-term decision, GAE carries more risk than some competing approaches.  Certainly, as others have said, it doesn't seem as friendly of a product to small teams or less technical teams, which is a shame.

Alexey

On Sunday, March 22, 2020 at 2:49:37 PM UTC-4, Jeff Schnitzer wrote:

As another longtime user, I have mixed feelings. Some things are better, some things are worse. I certainly wouldn't give up the current GAE to go back to the old one, but mostly because I still have enough of the old one available.

I straddle the old world and the new world, using the "old" memcache and task queue APIs. I guess if I ever want to upgrade from Java8 I'll be forced into Cloud Memorystore and Cloud Tasks. The biggest problem would be converting my test harness. I could use a live Redis that I flush between tests, but lack of an emulator for Cloud Tasks is a pretty serious problem. I'm very dependent on the LocalTaskQueueTestConfig.

Actually this is maybe a symptom of a bigger problem. In the old "integrated whole" GAE, the task queue is a way of deferring execution. The fact that there were URLs and http requests underneath was plumbing that could be ignored. 100% of my task queue usage is "execute this bit of code later", so it's really easy to write tests with an awaitTasks() method that chews through the list of deferred operations. The only level of abstraction provided by Cloud Tasks is "http request" so all the deferring and test harness logic would be up to me to implement. Ugh.

Honestly though, the old memcache and task queue APIs are so rock solid reliable, upgrading seems like a terrible idea. And there's nothing particularly compelling in Java11 (except the 10m deadlines).

On the plus side, Cloud Postgres is pretty fantastic, although balancing the number of database connections with the number of instances is really tricky with autoscaling on GAE. Under load, you either run out of JDBC connections or you run out of instances to serve user requests... if you're used to the datastore, tuning your scaling settings is now hard. PG connections are an expensive resource.

I like the new console quite a lot more than the old one, and it's getting better. The error notifications are great - nicely integrated with logging, unlike rollbar (which I no longer need to pay for).

GCS is vastly superior to the old blobstore.

I agree it's harder to get started with the current GAE offering, but I still recommend it to folks.

Jeff

-Joshua

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6edf3459-a4d0-4ea4-9282-86cdb6a2672f%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/5644BD34-868F-48F8-8FD7-87B8C0FA6AD6%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAPgu2uKUr7mY89NYrOWG0kJa9a1V%2Bc0395-5csqq7JycE7u4nQ%40mail.gmail.com.

--

Rajesh

www.servicefolder.com

Field Service Software on Google Cloud Platform and Mobile

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-a...@googlegroups.com.

[Olu]

Thank you for your feedback, Alexey and I understand your concerns. I however encourage you to use the issue tracker links for creation of features[1] that you consider would be nice to have on GCP. The team are committed to reviewing such requests and considering how these requests fit into the GCP Road map. 

Similarly, You could always submit issue links to request clarifications and better explanations on particular GCP Documentations that you may be using. These requests will be reviewed and forwarded to the Documentation team who further review the details and provide the needed updates on the Documentation page. 

Our team are always looking for ways to make GCP products and its documentation pages better. 

Thank you for your understanding. 

[1]https://cloud.google.com/support/docs/issue-trackers#feature_requests

[2]https://cloud.google.com/support/docs/issue-trackers#trackers-list

[bFlood]

Is there an official AppEngine Standard Python 2.7 depreciation date? the SDK (appcfg) is gone in July2020 but has anything been announced with regard to the runtime/hosting environment yet?

thanks. 

[barrado]

Hi! You can find the official statements regarding Python2 in this documentation pages [1][2].

[1]: https://cloud.google.com/python/docs/python2-sunset

[2]: https://cloud.google.com/appengine/docs/standard/long-term-support

[MdeA]

I've been developing on Google Application Engine since its beginnings, when Python (v2.5) was the only language supported. I came from the world of ASP/IIS and LAMP stacks, so suddenly not having to worry about scaling-up a web server and having a DB pretty much built-in was very appealing to me and the extremely easy to set up development environment (using Google App Engine Launcher) convinced me to start learning Python so that I could have very scalable web applications running in the cloud.

Now, seeing all those notes of Python 2 and appcfg being deprecated and since I have several web applications (paying applications that bring an income) running on GAE, I decided to give Python 3 and GCloud SDK a try following the Quickstart tutorial.

Boy, oh boy. Was I in for a surprise? Not a very nice one.

I develop on Windows and, of course, I was expecting something easy to use that'd gave me similar functionality as what I already had there.

Long story short, after several days I haven't been able to get the Hello World application to connect to a local Datastore. 

No idea how I could migrate existing Datastores to new ones, if I ever managed to re-write apps in Python 3 using GCloud SDK.

No easy way of switching from developing in one app to developing in another one in that new environment.

At least I haven't wrecked the Python 2 + Google App Engine Launcher environment while trying all this in my dev. computer.

I have to say I've had a pretty good run with GAE and I know technology changes, but this surely doesn't look to me as evolution.

Now I'm just wondering what will happen to all my web apps running on GAE after July, 2020.

[Joshua Smith]

On Apr 19, 2020, at 1:01 AM, MdeA <micde...@gmail.com> wrote:

Now I'm just wondering what will happen to all my web apps running on GAE after July, 2020.

Short answer: They will continue to work as they have been.

We need to migrate from Py2.7 to Py3.7 and the new stack, but google is giving us years to get it done.

At least, that how I read the notices google has sent out.

-Joshua

[Kaan Soral]

As far as I assumed, and I'm too optimistic these days, they'll work indefinitely? https://cloud.google.com/appengine/docs/standard/long-term-support

I'm probably never going to move my old apps to 3.7, they'll die when 2.7 dies - they use pretty much every available original in-house app engine service - almost impossible to completely re-engineer - It's painful but all things die :/

Re-writing with NodeJS / App Engine proved more mentally challenging then I assumed too, you get to experience every little thing we had and now lost - we were developing locally with instant refreshing at every file update, when you move to live development that requires you to do a 2 minute deployment to test every change, the frictional difference is huge - I guess it might even be a good idea to write custom local emulators for everything, routines that'll just return sample values etc. - or maybe in-line conditions to fallback to faked responses when you are locally developing

Like MdeA I started using Python for only App Engine, and I certainly wish services weren't being sunsetted, instead more could've been introduced, images service for example, was the most practical solution to image serving - when video/video sharing apps became more popular, I was anxiously awaiting a "videos" service to be introduced too, but looking at things today, we lost them all instead of gaining more :D