Import XML data to a High Replication Datastore

21 views
Skip to first unread message

Hosam

unread,
Sep 5, 2011, 4:13:43 PM9/5/11
to Google App Engine
Hi,

I have the follow problem

- I have around 25,000 XML files I need to pre populate the data store
with
- I am using a High Replication Datastore therefore I can't use the
bulk uploader for that

I can't of course upload those files as part of my WEB-INF because the
limit is 3000 files.

One solution I can think of is to write a client that send one file at
a time to my Web App then I parse it and insert it into the Data Store
but
that will eat up my entire BW.

Is there another solution for my problem?

Cheers and many thanks in advance

Robert Kluin

unread,
Sep 7, 2011, 1:42:45 AM9/7/11
to google-a...@googlegroups.com
I think you can use the bulk loader if you pass in the server argument
or app id, I can't recall exactly though.

How big are the files? Perhaps a better solution would be uploading
them in batches then bulk-writing them on the server. I've used that
method many times quite happily.


Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

Gerald Tan

unread,
Sep 7, 2011, 3:23:32 AM9/7/11
to google-a...@googlegroups.com
Archive all the XMLs into one large ZIP file, then use java.util.zip for random access to the XML files within?
Reply all
Reply to author
Forward
0 new messages