How to download data from my datastore to the local machine

[Delyan Spasov]

I've wasted lot of time trying to download the datatstore to my local machine. Nothing works!

The appcfg download_data doesn't seem to work any more. There is nothing about it in the new docs. It's not even mentioned even as deprecation notice.

It is still there in the appcfg. But it doesn't work. 

I've tried to upload some old data. It doesn't work.

I've tried the remote_api. Tones of errors.

They all used to work. What happened?

How to work with my database on the local machine?  

[Nicola Spreafico]

The appcfg client cannot be used anymore because it use a username/password authentication, which has been deprecated in favor of oauth2 (and not supported by the client itself).

About 1 year ago I had a very very long ticket with google support (issue number #08922843) where we analyzed all this situation, here a couple of comments:

We ended up writing a very simple remote API handler (both server and client) in order to reproduce the same need as you.

You need to deploy online a version with remote_api enabled and the write a simple client which will download the data (for all the kinds you need) from online and load it inside the local dev-server

In my case the client was written in Java, but because the remote_api exists in python I'm very confident that you can build a client of your own.

The main thing to take care is (this example is for java language, i think a python counterpart exists as well)

if ( ... remote api connected to development server ...) {
    options.useDevelopmentServerCredential();
} else {
    options.useApplicationDefaultCredential();
}

When you need to connect with the online environment, the application-default is used which need to be configured using GCloud SDK

https://cloud.google.com/sdk/gcloud/reference/beta/auth/application-default/login

[Alex (Cloud Platform Support)]

To add up to Nicola’s reply, you may be able to download your data to your local machine by opening the local Datastore viewer as proposed on this Issue Tracker thread. To do so, you will have to access the viewer locally at “http://localhost:8000/_ah/admin/datastore” and get the whole data flushed to you local disk.

[Delyan Spasov]

I read it but I don't understand how to download my database. 

The lack of any normal way of exporting my database is making me nervous about using GAE in the future. What if I decide to migrate because of some reason? 

I will find some workaround for the current projects (I will have to). But I don't see myself starting a new project on GAE. 

Thanks!

[Delyan Spasov]

Thanks Nicola,

From the interactive shell I can access the remote_api. But not from my application. It is raising NotSupportedOnThisPlatform(). Since this is in Python you probably don't know about this error. I found it as an issue but I didn't find a solution of it. 

I don't know how to write a client to download the data. I don't know how to format it in a way that the local server will accept it and I also don't know how to import it in the local server. There is not any documentation about it. 

But I think this should be included. It was but they have changed it because of the Oauth. OK, then include it again using Oauth. It isn't normal to not be able to export your db. This is very low quality service. 

[Alex (Cloud Platform Support)]

In fact, current Cloud Datastore’s NoSQL implementation of relationships between data objects differs from traditional databases. And the observed feature differences in this thread can be related to what is described on the Google Cloud Datastore Overview documentation.

Nevertheless, in case you would want to migrate your data to another Google Cloud application, you can use the Cloud Storage backing up functionality to export a datastore backup into a Storage bucket (See this Backing up data section of the Managing Datastore guide). Note that using this method, also allows to export your data to other Google Cloud Platform services, such as Google BigQuery (See Backup and restore considerations).

Lastly, here is a guide about Accessing App Engine with Remote API, that may get you further insights on how to write a client to download the data according to your specifications. See step 6 of Using the Remote API in a local client section for an example on how to format and retrieve data from a Datastore instance.

Hope you’ll find this information insightful and feel free to look at different Cloud products as Cloud SQL that may more accurately suit your needs. In case you would like to see the previously available "appcfg.py” [download_data] available in this Cloud Datastore implementation, it is possible to open a Features request on this Issue Tracker and making sure to specify “Feature Requests” under Component.

Regards,

[Delyan Spasov]

Thank you Alex,

I have read the docs for the remote_api. For some reason it isn't working and the error isn't very useful to find our why - NotSupportedOnThisPlatform.

I have found bug reports about this, but nobody is giving a working solution. 

Fortunately I can connect the remote_api from the console and I wrote a simple script to download the data to a file (Good that my database was a small one).

Regards

[Alex (Cloud Platform Support)]

Glad to hear that you’ve made it work, as the Remote API may in fact require additional setup according to your environment. Nevertheless, I would recommend another alternative potentially easier to configure for accessing the Datastore in the future. The alternative consists in using the Python NDB Client Library as part of an App Engine instance written in Python to access the Cloud Datastore. More information on how to customize queries to retrieve Datastore’s content can be retrieved from this NDB Query Class reference.

Also, here is the referenced Issue Tracker link that was missing in my previous response. Feel free to open a Feature Request regarding the re-implementation of the "appcfg.py” [download_data] feature there.

Hope it helps!