Search API: Creating documents and indexing using bulk loader or similar feature?

[Sai Gaddam]

Hi,

We have about 100k entities that we'd like to upload to the datastore. This seems doable (albeit slow) through the bulk upload functionality. However, we also want to create a document for each entity, which will contain a keyword string from the entity. 

We are linking up datastore entities and documents in this manner to take advantage of the search functionality of documents.  However, to do so will also require us to create documents on the fly as the datastore entities are being created via bulk upload.

What's the recommended way to do this? Should we define a post hook inside the data class that automatically creates a document? Or should we handle the documents separately?

There's little or no information about this in the references.  Any pointers would be appreciated. Thanks!

[Vinny P]

On Tue, Jun 24, 2014 at 3:57 AM, csai <chaita...@gmail.com> wrote:

We have about 100k entities that we'd like to upload to the datastore. This seems doable (albeit slow) through the bulk upload functionality. However, we also want to create a document for each entity, which will contain a keyword string from the entity. 

We are linking up datastore entities and documents in this manner to take advantage of the search functionality of documents.  However, to do so will also require us to create documents on the fly as the datastore entities are being created via bulk upload.

There's many ways to handle this, but I would suggest handling the documents separately. You can use the bulkloader to upload all the entities, then use a custom script to run through entities and create your documents. 

This is also better because you can use the script to periodically verify that your data was uploaded successfully.

 

 

-----------------

-Vinny P

Technology & Media Consultant

Chicago, IL

App Engine Code Samples: http://www.learntogoogleit.com