Uploading database

[Niall]

Hi,

I have a database in csv format on my computer. Not really a database, I know, but I'd like to have all this data on my web-app as a real database. 

I tried uploading it row-by-row, but that was very very slow process (even if I do multiple rows at a time). If I to upload lots in one go it has a tendency to time out. 

I was wondering if there's a way I've missed that'd allow me to upload the data to a database. 

Thanks for any advice for any advice!

[Niall]

OK. I was able to get something working. I used the bulk-upload thing. but the problem is that there are over 200k items, and I used 100% of my quota to upload uploading less than 10%. 

Is there a way of doing this better? Or do I have to wait 10 days to upload everything? 

Also, what's more processor intensive--posting to the remote api, or parsing? 

[Robert Kluin]

Perhaps you could try uploading the csv to the blobstore, then parsing
it out in chunks. I've used this method to upload lots of data
before. It is simple to set up and pretty fast. Not sure you are
going to be able to import 200k records under the free quota though.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/yk4A4bftfF4J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[Niall]

I'll try that now. 

I assume it's in danger of timing out if I leave the whole thing run. What might be a safe amount of data to run through every time? 

I was hoping I might be able to make the datastore locally in a test (localhost:8080 or something). Then download that. And upload that to my appspot. Might this be possible? Where are the local datastores stored? 

Thanks,

Niall

[Niall]

Before I do this, are the local and appspot databases interchangeable? 

[Robert Kluin]

Hi Niall,
There would be little point in doing it locally (aside from testing,
which I would encourage). You'd then wind up using the same tools...
gaining you nothing.

The process could look something like this:
1) upload the CSV as one big blob to the blobstore.
2) Start a task to process the blob.
2.1) Read a chunk of data from the blob, maybe around 100 entities worth.
2.2) Parse that data into entities, write those entities in bulk
using db.put.
2.3) Insert a new task to resume at step 2.1 where this one left off.

You'll just need to make sure you handle the cases when the data you
fetch ends with a partial record. It isn't difficult logic. You will
also want to adjust the approximate batch size you're doing in each
put based on your data (number of indexes, etc...).

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/FSSFIhfeS4wJ.

[Niall]

Yeah. That's exactly what I ended up doing. Thanks for the advice. 

Niall

[Ed Estes]

Another option is to develop the import functionality in a App Engine Backend.

--Ed