[google-appengine] Datastore inserts

[Geoff]

Hi,

I have an application which requires some relatively big inserts into
the datastore. The amount of data isn't huge, but there are often
>100,000 rows in each file upload (and I could be doing about 500-1000
per day). At the moment I'm doing something like

for row in file:
new_row = RowType( somedata )
new_row.put()

However this is way too slow (CPU-wise) for my needs; a 40,000 line
file is taking about 4 CPU hours.

Is there some obviously way of optimising the insert which I don't
know about?

Geoff.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Tristan]

oh wow.. yes,

try this for starters (pseudocode):

new_rows = new List(500)
i = 0

for row in file:
new_row = RowType( somedata)

if (i < 500)
new_rows.add(new_row)
i++
else
i = 0
new_rows.put()
new_rows.clear()
new_rows.add(new_row)
i++

that should decrease the amount of puts you do into the datastore by a
factor of 2