Cursor support

[Jeff Schnitzer]

We must decide upon an API for cursors.

The simplest solution is probably to change all the Iterable's to
QueryResultIterable's and add a cursor(Cursor) method to Query:

public interface Query<T> extends QueryResultIterable<T>
{
...
public QueryResultIterable<T> fetch();
public QueryResultIterable<Key<T>> fetchKeys();
public cursor(Cursor c);
...
}

I'm open to other suggestions if anyone has them.

Jeff

[David Chandler]

This works for me. I'm assuming QueryResultIterable implements
java.lang.Iterable so I wouldn't have to change all my DAOs again...

Presumably, QueryResultIterable would offer a getCursor() method that
would return the AppEngine cursor (or if not, wrap it with an impl
that provides AppEngine's to/from WebSafe methods).

/dmc

[Jeff Schnitzer]

Yup, QueryResultIterable is part of the low-level api. It extends
Iterable. It returns a QueryResultIterator extends Iterator which has
a getCursor() method. There would not be any API breakage.

The big question in my mind is if we wanted to hide the Cursor object
and just use the String version. It would be an easier-to-use API.
On the other hand, if Google decided to add other methods to Cursor
we'd have to redesign the API.

So I'm leaning towards the simple approach, exposing QueryResultIterable.

Jeff

[David Chandler]

I would think the String version is good enough for now. That keeps it
simple.

/dmc

On Feb 11, 1:51 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
> Yup, QueryResultIterable is part of the low-level api.  It extends
> Iterable.  It returns a QueryResultIterator extends Iterator which has
> a getCursor() method.  There would not be any API breakage.
>
> The big question in my mind is if we wanted to hide the Cursor object
> and just use the String version.  It would be an easier-to-use API.
> On the other hand, if Google decided to add other methods to Cursor
> we'd have to redesign the API.
>
> So I'm leaning towards the simple approach, exposing QueryResultIterable.
>
> Jeff
>

[Jeff Schnitzer]

Example of using datastore types:

QueryResultIterator<Foo> it = ofy.query(Foo.class).iterator();
while (it.hasNext()) {
Foo foo = it.next();
...
if (nearing deadline) {
Cursor cur = it.getCursor();
// store cursor, possibly calling cur.toWebSafeString()
break;
}
}

...later:
Cursor cur = // get cursor, possibly calling Cursor.fromWebSafeString(str);
QueryResultIterator<Foo> it = ofy.query(Foo.class).cursor(cur).iterator();
...continue

If we call to/fromWebSafeString ourselves and make the cursor a String
we will need to create our own CursorableIterable and
CursorableIterator classes. All in all, I favor exposing the
datastore Cursor and QueryResult* classes. It keeps Objectify thinner
and maybe makes interop with any future third-party libraries that use
the datastore types easier.

Jeff

[Matt Quail]

Looks good to me!

> The big question in my mind is if we wanted to hide the Cursor object
> and just use the String version.  It would be an easier-to-use API.
> On the other hand, if Google decided to add other methods to Cursor
> we'd have to redesign the API.
>
> So I'm leaning towards the simple approach, exposing QueryResultIterable.

Yes, I agree, we should explicitly expose QueryResultIterable and Cursor.

I think you could meet in the middle by offering cursor(String) and
cursor(Cursor) on query. To to-string on Query would use the web-safe
version of the Cursor.

=Matt

[David Chandler]

If you've got a way to expose the Datastore QueryResultIterator such
that it iterates over Objectified objects instead of raw entities,
that would be really cool. Exposing the Datastore Cursor is fine, too.

/dmc

On Feb 11, 2:32 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
> Example of using datastore types:
>
> QueryResultIterator<Foo> it = ofy.query(Foo.class).iterator();
> while (it.hasNext()) {
>     Foo foo = it.next();
>     ...
>     if (nearing deadline) {
>         Cursor cur = it.getCursor();
>         // store cursor, possibly calling cur.toWebSafeString()
>         break;
>     }
>
> }
>
> ...later:
> Cursor cur = // get cursor, possibly calling Cursor.fromWebSafeString(str);
> QueryResultIterator<Foo> it = ofy.query(Foo.class).cursor(cur).iterator();
> ...continue
>
> If we call to/fromWebSafeString ourselves and make the cursor a String
> we will need to create our own CursorableIterable and
> CursorableIterator classes.  All in all, I favor exposing the
> datastore Cursor and QueryResult* classes.  It keeps Objectify thinner
> and maybe makes interop with any future third-party libraries that use
> the datastore types easier.
>
> Jeff
>

[Jeff Schnitzer]

On Thu, Feb 11, 2010 at 2:50 PM, David Chandler <turbo...@gmail.com> wrote:
> If you've got a way to expose the Datastore QueryResultIterator such
> that it iterates over Objectified objects instead of raw entities,
> that would be really cool. Exposing the Datastore Cursor is fine, too.

Something other than what it is doing now? QueryResultIterator is a
generified class, if you generate one with an Objectify Query it
returns objectified objects...

Are you asking for a hook so you can wrap an arbitrary QueryResultIterator?

Jeff

[David Chandler]

Nope, was just trying to figure out the wizardry that lets you expose
an AppEngine QueryResultIterator that iterates over Objectified
objects. I (incorrectly) assumed that an AppEngine Iterator class
would return Datastore Entities, not Objectified objects, but
overlooked that it's generified, as you point out. Which, come to
think of it, probably helps out JDO and JPA, as well. Really cool
stuff.

Thanks,
/dmc

On Feb 11, 6:13 pm, Jeff Schnitzer <j...@infohazard.org> wrote:

[Stefano Ciccarelli]

As stated in the docs (http://code.google.com/p/objectify-appengine/
wiki/IntroductionToObjectify#Querying) the Query interface mimics the
Query class from GAE/Python, so why not continue on that way?
http://code.google.com/intl/it-IT/appengine/docs/python/datastore/queriesandindexes.html#Query_Cursors

[Jeff Schnitzer]

On Thu, Feb 11, 2010 at 11:56 PM, Stefano Ciccarelli
<scicc...@gmail.com> wrote:
> As stated in the docs (http://code.google.com/p/objectify-appengine/
> wiki/IntroductionToObjectify#Querying) the Query interface mimics the
> Query class from GAE/Python, so why not continue on that way?
> http://code.google.com/intl/it-IT/appengine/docs/python/datastore/queriesandindexes.html#Query_Cursors

There are inherent differences in the way things work in Java vs the
way things work in Python. One of them is that Javaland likes
Iterators and Iterables. Queries in Objectify are not tied to the
state of any particular iteration - that's for an iterator.

However, Python's example is clearly a vote for String rather than the
raw Cursor object. On the other hand... I actually like having the
typed Cursor object. I'm using cursors with the svn trunk code now
and it's really quite nice. Way easier than tracking the last id
myself.

In fact, it might be useful to post some example code. Here's a base
processing task that works with the newatlanta Deferred servlet:

/*
* $Id$
*/

package us.mobcasting.server.admin;

import java.io.IOException;

import javax.servlet.ServletException;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import unsuck.gae.GAETimer;

import com.google.appengine.api.datastore.Cursor;
import com.google.appengine.api.datastore.QueryResultIterator;
import com.newatlanta.appengine.taskqueue.Deferred;
import com.newatlanta.appengine.taskqueue.Deferred.Deferrable;

/**
* Base class for most tasks which iterate through the dataset and track stats.
*
* @author Jeff Schnitzer
*/
abstract public class IteratingTask<T> implements Deferrable
{
private static final long serialVersionUID = 1L;

/** */
private static final Logger log = LoggerFactory.getLogger(IteratingTask.class);

/** The cursor for navigating the results */
private Cursor cursor;

/** Total number of nodes processed */
private int totalCount;

/**
* @param cur can be null to start without a cursor
* @return the query we should execute, setup with the cursor
*/
protected abstract QueryResultIterator<T> iterator(Cursor cur);

/**
* Process one thing.
*/
protected void process(T thing) {}

/**
* Create the next task in the series.
*/
protected abstract IteratingTask<T> next();

/**
* Called when we're done processing everything.
*/
protected void onFinished() {}

/* */
@Override
final public void doTask() throws ServletException, IOException
{
GAETimer timer = new GAETimer();

if (log.isInfoEnabled())
log.info("Executing {} starting at cursor {}",
this.getClass().getSimpleName(), this.cursor);

QueryResultIterator<T> it = this.iterator(this.cursor);

// Initialize to null, if it stays null we know we're done
this.cursor = null;
int countThisTime = 0;

while (it.hasNext())
{
T thing = it.next();

countThisTime++;

this.process(thing);

// If we're near the deadline, just spawn this task again
if (timer.isNearDeadline() && it.hasNext())
{
this.cursor = it.getCursor();
break;
}
}

if (log.isInfoEnabled())
log.info("Processed " + countThisTime + " records");

this.totalCount += countThisTime;

if (this.cursor == null)
{
if (log.isInfoEnabled())
log.info("Finished processing " + this.totalCount + " total records");

this.onFinished();
}
else
{
IteratingTask<T> next = this.next();
next.cursor = this.cursor;
next.totalCount = this.totalCount;
Deferred.defer(next);
}
}
}

[Matt Quail]

Hi Stefano,

Jeff has added Cursor support to trunk, which will be available in the
next release.

You can retrieve the current cursor while iterating a query's result,
and use that cursor when constructing a query later.

=Matt