Indexes on MongoDB

[Joaquin Cuenca Abela]

Hi,

the docs in MongoDB say:

- In order to actually create indexes the dev_appserver must be run with
the --require-indexes option. Running with this option will probably
add significant overhead, since each time the dev_appserver checks to
see if it should create an index a query is performed.

When I grepped require-indexes the only match I got was in this
README.rst file. In typhoonae/__init__.py the class is instantiated,
and if I manually set the require_indexes argument to True, then I got
an error in the python code (I guess it was coded against an outdated
version of mongodb).

The index is not created, however.

I just added them in the mongodb console, is there an automatic way to
create the indexes from index.yaml?

Cheers,

--
Joaquin Cuenca Abela

[Tobias]

Hi Joaquin,

Thanks for pointing me to this one! I fixed the require_index flag.

Automatic index creation is not supported yet. But it's fairly easy to
add. See these changes http://code.google.com/p/typhoonae/source/detail?r=db3e1332b3d98106ebd1725d86ee8d2e57a628fa
to get an idea.

Maybe apptool should do that. What do you think?

Cheers,
Tobias

On Oct 28, 9:14 am, Joaquin Cuenca Abela <joaq...@cuencaabela.com>
wrote:

[Joaquin Cuenca Abela]

On Thu, Oct 28, 2010 at 12:23 PM, Tobias <tobias....@googlemail.com> wrote:
> Hi Joaquin,
>
> Thanks for pointing me to this one! I fixed the require_index flag.
>
> Automatic index creation is not supported yet. But it's fairly easy to
> add. See these changes http://code.google.com/p/typhoonae/source/detail?r=db3e1332b3d98106ebd1725d86ee8d2e57a628fa
> to get an idea.
>
> Maybe apptool should do that. What do you think?

That sounds like a good idea. While we're talking about it, TAE will
complain if we try to do a query like this:

Foo.ancestor(parent_key).sort('-__key__')

or simply this:

Foo.sort('-__key__')

These two queries can be served by the default index on _id of
mongodb. I'm not familiar enough with this part of the code, do you
know if it's easy enough to convince the mongodb driver to go ahead
without the extra index (other than disabling the required_index
flag)?

--
Joaquin Cuenca Abela

[Tobias]

Hi Joaquin,

I'm sorry, but I don't really get why TAE complains. From your samples
I would expect an AttributeError, since Foo is very likely not an
Instance of db.Query.

Assuming Foo is a Model, have you tried the following instead?

Foo.all().ancestor(parent_key).order('-__key__')

Best,
Tobias

On Oct 28, 6:02 pm, Joaquin Cuenca Abela <joaq...@cuencaabela.com>
wrote:

> On Thu, Oct 28, 2010 at 12:23 PM, Tobias <tobias.rodae...@googlemail.com> wrote:
> > Hi Joaquin,
>
> > Thanks for pointing me to this one! I fixed the require_index flag.
>
> > Automatic index creation is not supported yet. But it's fairly easy to

> > add. See these changeshttp://code.google.com/p/typhoonae/source/detail?r=db3e1332b3d98106eb...

[Joaquin Cuenca Abela]

Sorry Tobias, I was not clear, and I made a mistake. First of all,
yes, Foo is a Model.

I was actually using Foo.all().ancestor(parent_key).order('-__key__').

This query will need an index:

- kind: Foo
ancestor: yes
- name: __key__
direction: desc

The MongoDB backend can solve this query with its native index {"_id":
1}, it doesn't need any extra index.

Thinking about it, my only question is, if you generate MongoDB
indexes automatically from index.yaml declarations (something that is
still not possible, but it's easy as you showed), what kind of index
will be generated by the above index.yaml? Is the driver able to
detect that this index is redundant with the native index on _id?

Cheers,

> --
> You received this message because you are subscribed to the Google
> Groups "typhoonae" group.
> To post to this group, send email to typh...@googlegroups.com
> To unsubscribe from this group, send email to
> typhoonae+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/typhoonae?hl=en

--
Joaquin Cuenca Abela

[Tobias]

On Oct 29, 10:36 am, Joaquin Cuenca Abela <joaq...@cuencaabela.com>
wrote:

> Thinking about it, my only question is, if you generate MongoDB
> indexes automatically from index.yaml declarations (something that is
> still not possible, but it's easy as you showed), what kind of index
> will be generated by the above index.yaml? Is the driver able to
> detect that this index is redundant with the native index on _id?

This is indeed an interesting question which needs some investigation.
Since the stub method _Dynamic_CreateIndex takes care of index
creation, we should look there if we can avoid redundant indices. As
you can see here http://code.google.com/p/typhoonae/wiki/GettingStarted#Other_requirements
it's relatively dumb and straightforward.

[Tobias]

Sorry, incorrect link. Here is the correct one
http://code.google.com/p/typhoonae/source/browse/src/typhoonae/mongodb/datastore_mongo_stub.py#893

Tobias

> you can see herehttp://code.google.com/p/typhoonae/wiki/GettingStarted#Other_requirem...