Bulk Loading into Search Index

[Gaylord Aulke]

Hi,

at first: i am new to appengine go.

I would like to download a csv file from a remote server, then parse it and put all contents into the search index.

This does take some time since the file contains about 50K lines.

I got from the docs that i would use a push queue or cron job for this. On my local env it works, but on GAE it takes more than 10 minutes to load these rows into the index.

Either i am doing something the system is not made for, or i missed some important lesson...

So: is there any other way to load csv into the index than index.Put for each line?

And: how am i expected to do bulk loads or any other operation that takes more than 10 minutes?

cheers,

Gaylord

[si...@intesoft.net]

Other than running on Flex (where there isn't such a deadline) you generally want to look at using a task queue to schedule a continuation process if the request timeout is close to hitting.

If the API does have any batch call available then using that will also speed things up.