Documentation for ext.search module?

[glenc]

Hi,

I'm new to appengine and to this group, so apologies if I have missed
something, but I can't see docs anywhere on the web for the
google.appengine.ext.search module that is referenced in this article:
http://code.google.com/appengine/articles/bulkload.html

Searching for 'search api' and various other sensible variations has
not turned anything up, and I am not on my dev machine at the moment
and so haven't gone through the usual Python help() on the module
itself to see how well it is documented via docstrings.

Is the full appengine API documented on the web, or are modules not
listed on the left of the appengine code site in a RTFS state at the
moment as far as documentation goes?

Any help much appreciated, and very much looking forward to developing
with appengine.

[Duncan]

On Jun 8, 7:58 pm, glenc <glen.coa...@gmail.com> wrote:
> Hi,
>
> I'm new to appengine and to this group, so apologies if I have missed
> something, but I can't see docs anywhere on the web for the
> google.appengine.ext.search module that is referenced in this article:http://code.google.com/appengine/articles/bulkload.html
>

It's in the file google_appengine/google/appengine/ext/search/
__init__.py

You can either read it directly, or if you add google_appengine to
your Python path then you can use pydoc to format it which gives you
the following (but formatted and in pink and purple!):

google.appengine.ext.search index
c:\program files\google\google_appengine\google\appengine\ext\search
\__init__.py

Full text indexing and search, implemented in pure python.

Defines a SearchableModel subclass of db.Model that supports full text
indexing and search, based on the datastore's existing indexes.

Don't expect too much. First, there's no ranking, which is a killer
drawback.
There's also no exact phrase match, substring match, boolean
operators,
stemming, or other common full text search features. Finally, support
for stop
words (common words that are not indexed) is currently limited to
English.

To be indexed, entities must be created and saved as SearchableModel
instances, e.g.:

class Article(search.SearchableModel):
text = db.TextProperty()
...

article = Article(text=...)
article.save()

To search the full text index, use the SearchableModel.all() method to
get an
instance of SearchableModel.Query, which subclasses db.Query. Use its
search()
method to provide a search query, in addition to any other filters or
sort
orders, e.g.:

query = article.all().search('a search
query').filter(...).order(...)
for result in query:
...

The full text index is stored in a property named
__searchable_text_index. If
you want to use search() in a query with an ancestor, filters, or sort
orders,
you'll need to create an index in index.yaml with the
__searchable_text_index
property. For example:

- kind: Article
properties:
- name: __searchable_text_index
- name: date
direction: desc
...

Note that using SearchableModel will noticeable increase the latency
of save()
operations, since it writes an index row for each indexable word. This
also
means that the latency of save() will increase roughly with the size
of the
properties in a given entity. Caveat hacker!


Package Contents




Classes


google.appengine.api.datastore.Entity(__builtin__.dict)

SearchableEntity

google.appengine.api.datastore.Query(__builtin__.dict)

SearchableQuery

google.appengine.ext.db.Model(__builtin__.object)

SearchableModel


class SearchableEntity(google.appengine.api.datastore.Entity)
A subclass of datastore.Entity that supports full text indexing.

Automatically indexes all string and Text properties, using the
datastore's
built-in per-property indices. To search, use the SearchableQuery
class and
its Search() method.



Method resolution order:
SearchableEntity
google.appengine.api.datastore.Entity
__builtin__.dict
__builtin__.object

Methods defined here:

__init__(self, kind_or_entity, *args, **kwargs)
Constructor. May be called as a copy constructor.

If kind_or_entity is a datastore.Entity, copies it into this
Entity.
datastore.Get() and Query() returns instances of datastore.Entity,
so this
is useful for converting them back to SearchableEntity so that
they'll be
indexed when they're stored back in the datastore.

Otherwise, passes through the positional and keyword args to the
datastore.Entity constructor.

Args:
kind_or_entity: string or datastore.Entity

Methods inherited from google.appengine.api.datastore.Entity:

ToXml(self)
Returns an XML representation of this entity. Atom and
gd:namespace
properties are converted to XML according to their respective
schemas. For
more information, see:

http://www.atomenabled.org/developers/syndication/
http://code.google.com/apis/gdata/common-elements.html

This is *not* optimized. It shouldn't be used anywhere near code
that's
performance-critical.

__setitem__(self, name, value)
Implements the [] operator. Used to set property value(s).

If the property name is the empty string or not a string, raises
BadPropertyError. If the value is not a supported type, raises
BadValueError.

app(self)
Returns the name of the application that created this entity, a
string.

copy(self)
The copy method is not supported.

entity_group(self)
Returns this entitys's entity group as a Key.

Note that the returned Key will be incomplete if this is a a root
entity
and its key is incomplete.

key(self)
Returns this entity's primary key, a Key instance.

kind(self)
Returns this entity's kind, a string.

parent(self)
Returns this entity's parent, as a Key. If this entity has no
parent,
returns None.

setdefault(self, name, value)
If the property exists, returns its value. Otherwise sets it to
value.

If the property name is the empty string or not a string, raises
BadPropertyError. If the value is not a supported type, raises
BadValueError.

update(self, other)
Updates this entity's properties from the values in other.

If any property name is the empty string or not a string, raises
BadPropertyError. If any value is not a supported type, raises
BadValueError.

Data descriptors inherited from google.appengine.api.datastore.Entity:

__dict__
dictionary for instance variables (if defined)

__weakref__
list of weak references to the object (if defined)

Methods inherited from __builtin__.dict:

__cmp__(...)
x.__cmp__(y) <==> cmp(x,y)

__contains__(...)
D.__contains__(k) -> True if D has a key k, else False

__delitem__(...)
x.__delitem__(y) <==> del x[y]

__eq__(...)
x.__eq__(y) <==> x==y

__ge__(...)
x.__ge__(y) <==> x>=y

__getattribute__(...)
x.__getattribute__('name') <==> x.name

__getitem__(...)
x.__getitem__(y) <==> x[y]

__gt__(...)
x.__gt__(y) <==> x>y

__hash__(...)
x.__hash__() <==> hash(x)

__iter__(...)
x.__iter__() <==> iter(x)

__le__(...)
x.__le__(y) <==> x<=y

__len__(...)
x.__len__() <==> len(x)

__lt__(...)
x.__lt__(y) <==> x<y

__ne__(...)
x.__ne__(y) <==> x!=y

__repr__(...)
x.__repr__() <==> repr(x)

clear(...)
D.clear() -> None. Remove all items from D.

get(...)
D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.

has_key(...)
D.has_key(k) -> True if D has a key k, else False

items(...)
D.items() -> list of D's (key, value) pairs, as 2-tuples

iteritems(...)
D.iteritems() -> an iterator over the (key, value) items of D

iterkeys(...)
D.iterkeys() -> an iterator over the keys of D

itervalues(...)
D.itervalues() -> an iterator over the values of D

keys(...)
D.keys() -> list of D's keys

pop(...)
D.pop(k[,d]) -> v, remove specified key and return the
corresponding value
If key is not found, d is returned if given, otherwise KeyError is
raised

popitem(...)
D.popitem() -> (k, v), remove and return some (key, value) pair as
a
2-tuple; but raise KeyError if D is empty

values(...)
D.values() -> list of D's values

Data and other attributes inherited from __builtin__.dict:

__new__ = <built-in method __new__ of type object at 0x1E1CB048>
T.__new__(S, ...) -> a new object with type S, a subtype of T

fromkeys = <built-in method fromkeys of type object at 0x024F3318>
dict.fromkeys(S[,v]) -> New dict with keys from S and values equal
to v.
v defaults to None.


class SearchableModel(google.appengine.ext.db.Model)
A subclass of db.Model that supports full text search and
indexing.

Automatically indexes all string-based properties. To search, use the
all()
method to get a SearchableModel.Query, then use its search() method.



Method resolution order:
SearchableModel
google.appengine.ext.db.Model
__builtin__.object

Class methods defined here:

all(cls) from google.appengine.ext.db.PropertiedClass
Returns a SearchableModel.Query for this kind.

Data and other attributes defined here:

Query = <class 'google.appengine.ext.search.Query'>
A subclass of db.Query that supports full text search.

Methods inherited from google.appengine.ext.db.Model:

__init__(self, parent=None, key_name=None, _app=None, **kwds)
Creates a new instance of this model.

To create a new entity, you instantiate a model and then call
save(),
which saves the entity to the datastore:

person = Person()
person.name = 'Bret'
person.save()

You can initialize properties in the model in the constructor with
keyword
arguments:

person = Person(name='Bret')

We initialize all other properties to the default value (as
defined by the
properties in the model definition) if they are not provided in
the
constructor.

Args:
parent: Parent instance for this instance or None, indicating a
top-
level instance.
key_name: Name for new model instance.
_app: Intentionally undocumented.
args: Keyword arguments mapping to properties of model.

delete(self)
Deletes this entity from the datastore.

Raises:
TransactionFailedError if the data could not be committed.

dynamic_properties(self)
Returns a list of all dynamic properties defined for instance.

instance_properties(self)
Alias for dyanmic_properties.

is_saved(self)
Determine if entity is persisted in the datastore.

New instances of Model do not start out saved in the data.
Objects which
are saved to or loaded from the Datastore will have a True saved
state.

Returns:
True if object has been persisted to the datastore, otherwise
False.

key(self)
Unique key for this entity.

This property is only available if this entity is already stored
in the
datastore, so it is available if this entity was fetched returned
from a
query, or after save() is called the first time for new entities.

Returns:
Datastore key of persisted entity.

Raises:
NotSavedError when entity is not persistent.

parent(self)
Get the parent of the model instance.

Returns:
Parent of contained entity or parent provided in constructor,
None if
instance has no parent.

parent_key(self)
Get the parent's key.

This method is useful for avoiding a potential fetch from the
datastore
but still get information about the instances parent.

Returns:
Parent key of entity, None if there is no parent.

put(self)
Writes this model instance to the datastore.

If this instance is new, we add an entity to the datastore.
Otherwise, we update this instance, and the key will remain the
same.

Returns:
The key of the instance (either the existing key or a new key).

Raises:
TransactionFailedError if the data could not be committed.

save = put(self)
Writes this model instance to the datastore.

If this instance is new, we add an entity to the datastore.
Otherwise, we update this instance, and the key will remain the
same.

Returns:
The key of the instance (either the existing key or a new key).

Raises:
TransactionFailedError if the data could not be committed.

to_xml(self, _entity_class=<class
'google.appengine.api.datastore.Entity'>)
Generate an XML representation of this model instance.

atom and gd:namespace properties are converted to XML according to
their
respective schemas. For more information, see:

http://www.atomenabled.org/developers/syndication/
http://code.google.com/apis/gdata/common-elements.html

Class methods inherited from google.appengine.ext.db.Model:

entity_type(cls) from google.appengine.ext.db.PropertiedClass
Soon to be removed alias for kind.

fields(cls) from google.appengine.ext.db.PropertiedClass
Soon to be removed alias for properties.

from_entity(cls, entity) from google.appengine.ext.db.PropertiedClass
Converts the entity representation of this model to an instance.

Converts datastore.Entity instance to an instance of cls.

Args:
entity: Entity loaded directly from datastore.

Raises:
KindError when cls is incorrect model for entity.

get(cls, keys) from google.appengine.ext.db.PropertiedClass
Fetch instance from the datastore of a specific Model type using
key.

We support Key objects and string keys (we convert them to Key
objects
automatically).

Useful for ensuring that specific instance types are retrieved
from the
datastore. It also helps that the source code clearly indicates
what
kind of object is being retreived. Example:

story = Story.get(story_key)

Args:
keys: Key within datastore entity collection to find; or string
key;
or list of Keys or string keys.

Returns:
If a single key was given: a Model instance associated with key
for provided class if it exists in the datastore, otherwise
None; if a list of keys was given: a list whose items are either
a Model instance or None.

Raises:
KindError if any of the retreived objects are not instances of
the
type associated with call to 'get'.

get_by_id(cls, ids, parent=None) from
google.appengine.ext.db.PropertiedClass
Get instance of Model class by id.

Args:
key_names: A single id or a list of ids.
parent: Parent of instances to get. Can be a model or key.

get_by_key_name(cls, key_names, parent=None) from
google.appengine.ext.db.PropertiedClass
Get instance of Model class by its key's name.

Args:
key_names: A single key-name or a list of key-names.
parent: Parent of instances to get. Can be a model or key.

get_or_insert(cls, key_name, **kwds) from
google.appengine.ext.db.PropertiedClass
Transactionally retrieve or create an instance of Model class.

This acts much like the Python dictionary setdefault() method,
where we
first try to retrieve a Model instance with the given key name and
parent.
If it's not present, then we create a new instance (using the
*kwds
supplied) and insert that with the supplied key name.

Subsequent calls to this method with the same key_name and parent
will
always yield the same entity (though not the same actual object
instance),
regardless of the *kwds supplied. If the specified entity has
somehow
been deleted separately, then the next call will create a new
entity and
return it.

If the 'parent' keyword argument is supplied, it must be a Model
instance.
It will be used as the parent of the new instance of this Model
class if
one is created.

This method is especially useful for having just one unique entity
for
a specific identifier. Insertion/retrieval is done
transactionally, which
guarantees uniqueness.

Example usage:

class WikiTopic(db.Model):
creation_date = db.DatetimeProperty(auto_now_add=True)
body = db.TextProperty(required=True)

# The first time through we'll create the new topic.
wiki_word = 'CommonIdioms'
topic = WikiTopic.get_or_insert(wiki_word,
body='This topic is totally
new!')
assert topic.key().name() == 'CommonIdioms'
assert topic.body == 'This topic is totally new!'

# The second time through will just retrieve the entity.
overwrite_topic = WikiTopic.get_or_insert(wiki_word,
body='A totally different
message!')
assert topic.key().name() == 'CommonIdioms'
assert topic.body == 'This topic is totally new!'

Args:
key_name: Key name to retrieve or create.
**kwds: Keyword arguments to pass to the constructor of the
model class
if an instance for the specified key name does not already
exist. If
an instance with the supplied key_name and parent already
exists, the
rest of these arguments will be discarded.

Returns:
Existing instance of Model class with the specified key_name and
parent
or a new one that has just been created.

Raises:
TransactionFailedError if the specified Model instance could not
be
retrieved or created transactionally (due to high contention,
etc).

gql(cls, query_string, *args, **kwds) from
google.appengine.ext.db.PropertiedClass
Returns a query using GQL query string.

See appengine/ext/gql for more information about GQL.

Args:
query_string: properly formatted GQL query string with the
'SELECT * FROM <entity>' part omitted
*args: rest of the positional arguments used to bind numeric
references
in the query.
**kwds: dictionary-based arguments (for named parameters).

kind(cls) from google.appengine.ext.db.PropertiedClass
Returns the datastore kind we use for this model.

We just use the name of the model for now, ignoring potential
collisions.

properties(cls) from google.appengine.ext.db.PropertiedClass
Returns a dictionary of all the properties defined for this model.

Data descriptors inherited from google.appengine.ext.db.Model:

__dict__
dictionary for instance variables (if defined)

__weakref__
list of weak references to the object (if defined)

Data and other attributes inherited from
google.appengine.ext.db.Model:

__metaclass__ = <class 'google.appengine.ext.db.PropertiedClass'>
Meta-class for initializing Model classes properties.

Used for initializing Properties defined in the context of a
model.
By using a meta-class much of the configuration of a Property
descriptor becomes implicit. By using this meta-class,
descriptors
that are of class Model are notified about which class they
belong to and what attribute they are associated with and can
do appropriate initialization via __property_config__.

Duplicate properties are not permitted.


class SearchableQuery(google.appengine.api.datastore.Query)
A subclass of datastore.Query that supports full text search.

Only searches over entities that were created and stored using the
SearchableEntity or SearchableModel classes.



Method resolution order:
SearchableQuery
google.appengine.api.datastore.Query
__builtin__.dict
__builtin__.object

Methods defined here:

Search(self, search_query)
Add a search query. This may be combined with filters.

Note that keywords in the search query will be silently dropped if
they
are stop words or too short, ie if they wouldn't be indexed.

Args:
search_query: string

Returns:
# this query
SearchableQuery

Methods inherited from google.appengine.api.datastore.Query:

Ancestor(self, ancestor)
Sets an ancestor for this query.

This restricts the query to only return result entities that are
descended
from a given entity. In other words, all of the results will have
the
ancestor as their parent, or parent's parent, or etc.

Raises BadArgumentError or BadKeyError if parent is not an
existing Entity
or Key in the datastore.

Args:
# the key must be complete
ancestor: Entity or Key

Returns:
# this query
Query

Count(self, limit=None)
Returns the number of entities that this query matches. The
returned
count is cached; successive Count() calls will not re-scan the
datastore
unless the query is changed.

Raises BadQueryError if the Query has more than one filter.
Multiple
filters aren't supported yet.

Args:
limit, a number. If there are more results than this, stop short
and
just return this number. Providing this argument makes the count
operation more efficient.
Returns:
The number of results.

Get(self, limit, offset=0)
Fetches and returns a maximum number of results from the query.

This method fetches and returns a list of resulting entities that
matched
the query. If the query specified a sort order, entities are
returned in
that order. Otherwise, the order is undefined.

The limit argument specifies the maximum number of entities to
return. If
it's greater than the number of remaining entities, all of the
remaining
entities are returned. In that case, the length of the returned
list will
be smaller than limit.

The offset argument specifies the number of entities that matched
the
query criteria to skip before starting to return results. The
limit is
applied after the offset, so if you provide a limit of 10 and an
offset of 5
and your query matches 20 records, the records whose index is 0
through 4
will be skipped and the records whose index is 5 through 14 will
be
returned.

The results are always returned as a list. If there are no results
left,
an empty list is returned.

If you know in advance how many results you want, this method is
more
efficient than Run(), since it fetches all of the results at once.
(The
datastore backend sets the the limit on the underlying
scan, which makes the scan significantly faster.)

Args:
# the maximum number of entities to return
int or long
# the number of entities to skip
int or long

Returns:
# a list of entities
[Entity, ...]

Hint(self, hint)
Sets a hint for how this query should run.

The query hint gives us information about how best to execute your
query.
Currently, we can only do one index scan, so the query hint should
be used
to indicates which index we should scan against.

Use FILTER_FIRST if your first filter will only match a few
results. In
this case, it will be most efficient to scan against the index for
this
property, load the results into memory, and apply the remaining
filters
and sort orders there.

Similarly, use ANCESTOR_FIRST if the query's ancestor only has a
few
descendants. In this case, it will be most efficient to scan all
entities
below the ancestor and load them into memory first.

Use ORDER_FIRST if the query has a sort order and the result set
is large
or you only plan to fetch the first few results. In that case, we
shouldn't try to load all of the results into memory; instead, we
should
scan the index for this property, which is in sorted order.

Note that hints are currently ignored in the v3 datastore!

Arg:
one of datastore.Query.[ORDER_FIRST, ANCESTOR_FIRST,
FILTER_FIRST]

Returns:
# this query
Query

Order(self, *orderings)
Specify how the query results should be sorted.

Result entities will be sorted by the first property argument,
then by the
second, and so on. For example, this:

> query = Query('Person')
> query.Order('bday', ('age', Query.DESCENDING))

sorts everyone in order of their birthday, starting with January
1.
People with the same birthday are sorted by age, oldest to
youngest.

The direction for each sort property may be provided; if omitted,
it
defaults to ascending.

Order() may be called multiple times. Each call resets the sort
order
from scratch.

If an inequality filter exists in this Query it must be the first
property
passed to Order. Any number of sort orders may be used after the
inequality filter property. Without inequality filters, any number
of
filters with different orders may be specified.

Entities with multiple values for an order property are sorted by
their
lowest value.

Note that a sort order implies an existence filter! In other
words,
Entities without the sort order property are filtered out, and
*not*
included in the query results.

If the sort order property has different types in different
entities - ie,
if bob['id'] is an int and fred['id'] is a string - the entities
will be
grouped first by the property type, then sorted within type. No
attempt is
made to compare property values across types.

Raises BadArgumentError if any argument is of the wrong format.

Args:
# the properties to sort by, in sort order. each argument may be
either a
# string or (string, direction) 2-tuple.

Returns:
# this query
Query

Run(self)
Runs this query.

If a filter string is invalid, raises BadFilterError. If a filter
value is
invalid, raises BadValueError. If an IN filter is provided, and a
sort
order on another property is provided, raises BadQueryError.

If you know in advance how many results you want, use Get()
instead. It's
more efficient.

Returns:
# an iterator that provides access to the query results
Iterator

__delitem__(self, filter)
Implements the del [] operator. Used to remove filters.

__init__(self, kind, filters={}, _app=None)
Constructor.

Raises BadArgumentError if kind is not a string. Raises
BadValueError or
BadFilterError if filters is not a dictionary of valid filters.

Args:
# kind is required. filters is optional; if provided, it's used
# as an initial set of property filters.
kind: string
filters: dict

__iter__(self)

__setitem__(self, filter, value)
Implements the [] operator. Used to set filters.

If the filter string is empty or not a string, raises
BadFilterError. If
the value is not a supported type, raises BadValueError.

copy(self)
The copy method is not supported.

setdefault(self, filter, value)
If the filter exists, returns its value. Otherwise sets it to
value.

If the property name is the empty string or not a string, raises
BadPropertyError. If the value is not a supported type, raises
BadValueError.

update(self, other)
Updates this query's filters from the ones in other.

If any filter string is invalid, raises BadFilterError. If any
value is
not a supported type, raises BadValueError.

Data descriptors inherited from google.appengine.api.datastore.Query:

__dict__
dictionary for instance variables (if defined)

__weakref__
list of weak references to the object (if defined)

Data and other attributes inherited from
google.appengine.api.datastore.Query:

ANCESTOR_FIRST = 2

ASCENDING = 1

DESCENDING = 2

FILTER_FIRST = 3

FILTER_REGEX = <_sre.SRE_Pattern object at 0x00CC3E20>

INEQUALITY_OPERATORS = frozenset(['<', '<=', '>', '>='])

OPERATORS = {'<': 1, '<=': 2, '=': 5, '==': 5, '>': 3, '>=': 4}

ORDER_FIRST = 1

Methods inherited from __builtin__.dict:

__cmp__(...)
x.__cmp__(y) <==> cmp(x,y)

__contains__(...)
D.__contains__(k) -> True if D has a key k, else False

__eq__(...)
x.__eq__(y) <==> x==y

__ge__(...)
x.__ge__(y) <==> x>=y

__getattribute__(...)
x.__getattribute__('name') <==> x.name

__getitem__(...)
x.__getitem__(y) <==> x[y]

__gt__(...)
x.__gt__(y) <==> x>y

__hash__(...)
x.__hash__() <==> hash(x)

__le__(...)
x.__le__(y) <==> x<=y

__len__(...)
x.__len__() <==> len(x)

__lt__(...)
x.__lt__(y) <==> x<y

__ne__(...)
x.__ne__(y) <==> x!=y

__repr__(...)
x.__repr__() <==> repr(x)

clear(...)
D.clear() -> None. Remove all items from D.

get(...)
D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.

has_key(...)
D.has_key(k) -> True if D has a key k, else False

items(...)
D.items() -> list of D's (key, value) pairs, as 2-tuples

iteritems(...)
D.iteritems() -> an iterator over the (key, value) items of D

iterkeys(...)
D.iterkeys() -> an iterator over the keys of D

itervalues(...)
D.itervalues() -> an iterator over the values of D

keys(...)
D.keys() -> list of D's keys

pop(...)
D.pop(k[,d]) -> v, remove specified key and return the
corresponding value
If key is not found, d is returned if given, otherwise KeyError is
raised

popitem(...)
D.popitem() -> (k, v), remove and return some (key, value) pair as
a
2-tuple; but raise KeyError if D is empty

values(...)
D.values() -> list of D's values

Data and other attributes inherited from __builtin__.dict:

__new__ = <built-in method __new__ of type object at 0x1E1CB048>
T.__new__(S, ...) -> a new object with type S, a subtype of T

fromkeys = <built-in method fromkeys of type object at 0x024F7E40>
dict.fromkeys(S[,v]) -> New dict with keys from S and values equal
to v.
v defaults to None.

[glenc]

Thanks for the reply, I discovered that too when I returned to my dev
machine :)

On Jun 8, 5:18 pm, Duncan <kupu...@googlemail.com> wrote:
> On Jun 8, 7:58 pm,glenc<glen.coa...@gmail.com> wrote:> Hi,

> ...
>
> read more »