Importing data into Datastore. What's the deal?

[Randy L]

It's been a week of my time trying to figure out how to import ~150000 rows of JSON data into Datastore. Nearly everything that I find about how to do this is from 2014 or earlier. The best info that I have found tells me to use AppEngine mapreduce to process the data. How would anyone get anything done when using Datastore? It should not take a good (ok maybe I'm just decent) engineer a week to import some data. I even opened a ticket with our Cloud Support account and they were not any help at all really. They pointed me to some python script for pulling chunks of data down from Google Cloud Storage. Sure I'll need to do that at some point, but that's missing the point.

[Nickolas Daskalou]

Can you adapt the python script to work with chained Tasks using the Task Queue?

Which python script did they refer you to?

Nick

[George (Cloud Platform Support)]

Datastore is beneficial in some cases, and your use case is not the most frequently encountered in this context. There are alternative storage solutions you might have a look at, listed in the Other storage options from the Cloud Datastore Overview document.

You may use the Rest API to store JSON data in Datastore, with methods: “Method: projects.beginTransaction” and then “Method: projects.commit”.

One needs to create objects from JSON lines and then transfer them to the Datastore in an economical way, so as to reduce time and cost.

More detail about the options available for doing so is to be found in a relatively recent article from Stackoverflow: “How to upload data in bulk to the appengine datastore?”, that you may consider reading.

[Niklas B]

Randy, did you ever find a solution? Using put_multi would take me 15-23 days to upload my 15 GB of data ;)

Regards,

Niklas