Cloud NDB is now GA

[Andrew Gorcester]

Hello Pythonistas,

The Cloud NDB client library for Cloud Datastore and Cloud Firestore in Datastore mode, previously introduced as a beta project in September 2019, has been launched to GA with its 1.0 release.

Cloud NDB is an updated version of the App Engine NDB client library for Datastore, adding support for Python 3 and targeting the Datastore API. While it is intended for users migrating from App Engine Python 2.7 to App Engine Python 3, it is fully portable and compatible with other Python platforms. The library can be installed with `pip install google-cloud-ndb` and the source code can be found on Github at https://github.com/googleapis/python-ndb.

We also have a migration guide for NDB at, as part of our broader GAE Python migration documentation. We'll continue updating and adding to this documentation and library as part of our migration support effort, so please also continue sending us your feedback.

Thank you again for your continued support for App Engine and our Cloud platform. I am looking forward to hearing feedback from the community on this and upcoming efforts.

[Remko Tronçon]

We also have a migration guide for NDB at, as part of our broader GAE Python migration documentation. We'll continue updating and adding to this documentation and library as part of our migration support effort, so please also continue sending us your feedback.

 

The migration guide doesn't mention how NDB unit tests can be ported. AFAIK, the testbed that is currently available in NDB is not available for Cloud NDB.

Is there anything provided for unit testing Cloud NDB code, or do you have to abstract out NDB (which sounds like a huge effort and would pretty much make Cloud NDB useless).

thanks,

Remko

[Philip]

In this ticket it was stated that Google does not recommend NDB for new projects. It is merely intended for upgrading from 2.7 to python 3 GAE. It this statement still true given the fact that it is now GA? 

[Andrew Gorcester]

Cloud NDB is compatible with the Cloud Datastore Emulator; we recommend tests be done using that, the same way on Cloud NDB as they would be with the Cloud Datastore library.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/307e34e9-60ed-4534-a330-1adbad15255a%40googlegroups.com.

[Andrew Gorcester]

Cloud NDB is intended as a migration tool for App Engine NDB users who wish to move to either newer versions of App Engine (App Engine Python 3) or to other environments. Customers are welcome to use Cloud NDB however they wish, including on totally new projects that aren't associated with GAE NDB. However, Cloud NDB will not add support for any new features that might be added to Cloud Datastore in the future, because that wouldn't further the goal of aiding migration from GAE NDB; also the NDB concurrency model was designed around GAE Python 2.7 limitations and might not be the best model for greenfield projects with no history rooted in App Engine. We recommend the use of the Cloud Datastore library for new projects when possible.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/322768f1-6716-436c-b4da-e8d17ed4ad77%40googlegroups.com.

[Remko Tronçon]

On Monday, February 24, 2020 at 9:52:25 PM UTC+1, Andrew Gorcester wrote:
> Cloud NDB is compatible with the Cloud Datastore Emulator; we recommend tests be done using that, the same way on Cloud NDB as they would be with the Cloud Datastore library.

Using an external emulator is way too slow to use for unit tests, even if you are able to get all your code changed so you don’ have to tear down and clean start the emulator to avoid interference between tests. This is pretty disappointing, and means I (and I expect many others) will be forced to rewrite everything if your app isn’t in pure maintenance mode.

Remko

[Andrew Gorcester]

I'm afraid none of the Cloud API-targeting libraries have testbed support, as testbed was a solution to App Engine Gen 1-specific problems at the time. If you have an example of how much slower the datastore emulator is vs. testbed for your use case, that would be very helpful in prioritizing further work, either in speeding up the emulator or replicating the testbed system for Cloud. I'd recommend filing a bug with your experiences at https://github.com/googleapis/python-ndb/issues (though this would eventually be tackled by the entire Cloud Client libraries team, rather than just the NDB team).

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/9f3246de-d324-4e33-b2d9-86f00a642aa5%40googlegroups.com.

[Ryan B]

congratulations andrew! and to chris, carlos, danny, and everyone else who built python 3 ndb. it's been a huge boon for migrating from appengine python 2 to python 3. we appreciate all the hard work you all put into it!

[Ryan B]

the datastore emulator is indeed slower than testbed for unit tests, but just as another data point, it's not way too slow for me. i routinely use it as part of my edit/compile/test loop on an app with a cumulative total of >1600 tests. around 500 of them use the datastore heavily, and running just those tests takes ~52s. not fast, but manageable, especially when i iterate on just a few relevant tests, and then expand to all of them before committing.

also, you don't have to entirely tear down and restart the emulator just to clear its data between tests. just POST to http://localhost:8089/reset (or whichever port you run the emulator on). example: https://github.com/snarfed/bridgy/blob/master/tests/testutil.py#L282-L289

[Jon Grover]

I've found that entering/exiting context in the setUp/tearDown methods (as you do in your example) is adding about 0.7s to each of my tests, so I'm wondering how you're able to do 500 tests in 52s (works out to 0.1s per test). Perhaps it's machine dependent; I'm running these tests on a 2015 MacBook Pro, so maybe they'd be faster on something newer.

[Ryan B]

On Sunday, March 1, 2020 at 2:44:04 PM UTC-8, Jon Grover wrote:

I've found that entering/exiting context in the setUp/tearDown methods (as you do in your example) is adding about 0.7s to each of my tests, so I'm wondering how you're able to do 500 tests in 52s (works out to 0.1s per test). Perhaps it's machine dependent; I'm running these tests on a 2015 MacBook Pro, so maybe they'd be faster on something newer.

hmm! i'm actually on a 2014 MBP, so i doubt it's your hardware. feel free to poke at that repo, stack, and test suite if you want to see if you can find a difference. all of the tests use that base HandlerTest class, unittest, and mox3.

[Jon Grover]

I've gone through your repos for code examples but it hadn't occurred to me to actually pull them down and run them for performance comparisons, but I'll do that now. Thanks once again!

Also, so as not to detract from the thread origin too much: Thanks also to Andrew and the rest of the Cloud NDB team. They've been incredibly responsive on the Github issues tracker and this library is a lifesaver for people trying to migrate legacy apps off Python 2. There's a really good chance that without this library we might simply have scrapped our app completely. Your work is appreciated, guys!