Python 3 standard environment data storage

[Joshua Smith]

I’ve been using AppEngine for about a decade, currently my apps are all Python 2.7, Standard environment, google.appengine.ext.db

I need to create a new app, so I figured it’s a good time to start learning all the technologies that have replaced everything I’ve been using.

I’m trying to understand what database technology I should use. Since I’m eventually going to have to migrate all my existing apps to Python 3, I’d like to use the database that is closest to what I’ve been using.

From my reading of the docs, I think I can use “Cloud Firestore in Datastore Mode” and interface to that using “Cloud NDB” which provides an API very similar to “ndb” which is something I’ve never used, but looks darn similar to “db”.

Does that make sense? I see that there are “this is beta software” warnings around the Cloud NDB part. Should I be worried about that?

Since this is a new project, I don’t have any data migration to worry about. But if I choose this path when porting an existing project, is the underlying database the same database? Can I be reading and writing the same datastore from a Python 2.7 google.appengine.ext.db app and a Python 3.7 google.cloud.datastore app at the same time?

-Joshua

[Pierre-Yves Blain]

Firestore in Datastore mode is indeed recommended for databases that will be used primarily by App Engine apps. Firestore in Native mode could also be an interesting option to look into as it represents the next major version of Datastore. You can take a look at the differences here [1] and see what's most applicable for your use case. If you prioritize a feel closer to what you have been using over the additional features provided by Native mode, than Datastore mode may be more appropriate for your use case.

The Cloud NDB client library is meant to replace App Engine NDB for customers migrating projects/apps to Python 3. As explained in our documentation [2], this library will not support new features of Firestore in Datastore mode. As such, new apps/projects should use the Datastore mode client library [3] instead of Cloud NDB. Reading and writing data from Cloud NDB will allow your Python 2 and Python 3 apps to use the same databases, however the product managing those databases will now be Cloud Firestore in Datastore mode.

Finally, products in Beta mode have usually spent 6 months in the Alpha phase where the majority of testing was done. Products in Beta are therefore generally more robust, publicly announced, and generally feature complete although this might change based on customer feedback. You can read more about the different phases here [4]. It is worth noting that SLAs do not apply for products in that phase, so you may verify the Firestore SLA [5] and see if you require it for your projects. If required, the Datastore mode Client Libraries [6] may be a better option as it is in GA phase.

[1] https://cloud.google.com/datastore/docs/firestore-or-datastore 

[2] https://cloud.google.com/appengine/docs/standard/python3/migrating-to-cloud-ndb 

[3] https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-usage-python

[4] https://cloud.google.com/products/#product-launch-stages

[5] https://cloud.google.com/firestore/sla 

[6] https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-install-python

[Kaan Soral]

I see a lot of questions of this nature lately, good that we're slowly starting to migrate/consider migrating as a community

A bit hijacking the thread; Do we really need to migrate old apps? I have a lot of apps that I keep alive just to keep them alive, they are keepsakes at this point, I don't want to kill them, their income roughly offsets their costs, so for me, the question is keep or kill, and I'd like to keep

Is there any point where old technologies / Python 2.7 is going to become unsupported?

(This Python 2.7 thing is such a bummer too, I certainly wish we could have a Google Python 2.7 fork and keep using it indefinitely :)

[Joshua Smith]

My understanding is that yes, 2.7 will eventually go *poof* like the old pre-HR DB implementation went *poof*. However, it’s a long way off. They haven’t issued a deprecation warning yet, and as a core technology, google's habit is to give you 3 years to move off of it after deprecation.

The launcher, though, is another story. That’s going away this summer, and will be sorely missed. I cannot imagine why they think having to type mysterious incantations into a terminal is preferable to clicking “run” or “deploy”

I managed to get the new 3.7 app I’m using as a test case running, but it took me nearly an hour to figure out all the proper steps. (The step-by-step walk-throughs are nice, but they are misleading. For example, if you use a database (who doesn’t?) you need to install a bunch of other stuff and do a whole bunch of other cryptic things to get a test database running or your simple “hello world” will crash with a page of error messages.)

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/da5bc76b-c9e9-484d-b32f-4e4a0e9162ec%40googlegroups.com.

[Joshua Smith]

Running locally, I’m using this datastore emulator:

https://cloud.google.com/datastore/docs/tools/datastore-emulator

It works, but it is SOOO SLOOOOW. It takes literally 10-20 seconds to respond to a simple query.

Is this a known issue? Is there a fix?

-Joshua

[Joshua Smith]

I moved my test app to the cloud and I’m able to read and write data using Cloud NDB. However, the console still shows my database as empty, and it seems to be ignoring my index.yaml file (or maybe the indexes are just taking a really long time to build, but I can’t tell because the console says I have no indexes).

Questions:

- Does it take a while for the Datastore console notice there’s a datastore on a new project?

- Is there an extra step in the python3.7 environment beyond including an index.yaml file in my deployment?

Also, I’ve figured out how to use sendgrid instead of the built-in email like I had in P2.7, which was surprisingly painless and easy. But is there a replacement for this:

from google.appengine.ext import ereporter

ereporter.register_logger()

That thing is invaluable when you have lots of apps. Is there something similar for P3.7 that will send me exception summaries daily?

I’m familiar with the “Cloud Console” phone app, which will do notifications. But in my experience you can’t rely on that, because sometimes it spontaneously stops notifying.

-Joshua

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/aee16a09-25b0-42f0-8c55-da7b31e115e0%40googlegroups.com.

[Ryan B]

On Friday, January 17, 2020 at 8:42:11 AM UTC-8, Joshua Smith wrote:

Running locally, I’m using this datastore emulator:

https://cloud.google.com/datastore/docs/tools/datastore-emulator

It works, but it is SOOO SLOOOOW. It takes literally 10-20 seconds to respond to a simple query.

huh, that's surprising. just as a data point, i definitely don't see that behavior. i use the datastore emulator locally for both unit tests and interactive development, and it responds many orders of magnitude faster, as expected. for example, one project runs 524 unit tests in ~53s, many of which do multiple operations (both read and write) against the emulator.

i'm on mac os 10.15.2, gcloud 276.0.0, cloud-datastore-emulator 2.1.0, python 3.6.5.

[Ryan B]

On Friday, January 17, 2020 at 9:28:56 AM UTC-8, Joshua Smith wrote:

That thing is invaluable when you have lots of apps. Is there something similar for P3.7 that will send me exception summaries daily?

I’m familiar with the “Cloud Console” phone app, which will do notifications. But in my experience you can’t rely on that, because sometimes it spontaneously stops notifying.

i've been using stackdriver error reporting for this for a while, and it's great, and pretty automatic and built in. i don't think it does daily digest emails, but you can definitely have it email individual new errors when they first appear, or recur after closing them, and those emails have been very reliable in my experience.

[Joshua Smith]

Answering my own questions, for the record...

Questions:

- Does it take a while for the Datastore console notice there’s a datastore on a new project?

Yup. It takes a while but eventually the console starts showing the database.

- Is there an extra step in the python3.7 environment beyond including an index.yaml file in my deployment?

Yup:

gcloud datastore indexes create index.yaml