Access to bulk data from DB possible?

[Luca de Alfaro]

We need to write an app/site whose general usage pattern matches quite well AppEngine, except for one thing: the site will be collecting data / logs, and every now and then, we need to download bulk data from the database tables collecting such data / logs, for offline analysis. 

If we went with a "non-appengine" solution, we could obviously do this by generating e.g. csv files from DB queries or dumps, and then processing the files. 

Will it be possible to have the same kind of simple access to bulk data on AppEngine?  We can of course build a (web) API that issues DB queries and produces csv files for download, but my concerns are:

1. I remember that there is a limit to how many records can be extracted from a DB using a query (1000?), so that we would have to implement the query with continuation parameters etc -- feasible, but complicating the design.
2. I worry about the execution time of the query (is it still true that processes taking over 1s are killed?). 

I guess access to log files / bulk data must be a pretty common requirement for many apps, so I am hoping someone has good words of advice... 

Many thanks!!

Luca

[Ikai Lan (Google)]

1. This limit is gone, but obviously the time it takes to do a query scales with the result set. The design is actually quite easy: get back a cursor, pass the cursor on the next iteration. I don't trivially say things are easy; I err on the side of saying something is too complex.

2. Nope, not anymore.

You might want to look into something like appengine-mapreduce. In theory you should be able to aggregate large numbers of entities into a few blobstore (App Engine files API) entities:

http://code.google.com/p/appengine-mapreduce/

I say "in theory" because this tool is still experimental. We have every intention of making this tool a core part of the platform but it isn't there yet. Still, many developers are using this in production now for offline computation:

http://googleappengine.blogspot.com/2010/08/practical-report-generation-on-app.html

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

Luca

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FD50slblUYEJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.