How does TyphoonAE handle...

[hawkett]

Hi All,

I've experimented with TyphoonAE on a couple of occasions, and the last one I nearly took the plunge to move our local dev environment and (non production) server environment to TyphoonAE. Thanks to Tobias for his help last time in proving our app will run on TyphoonAE. I think I'm nearly ready to make that change now, but I had a couple of questions about how TyphoonAE handles the following situations -

index.yaml - on GAE, if you want to do a query that is not supported by an entry in index.yaml you are out of luck. Nextgen queries was promised at Google IO last year, and there is some rumblings it is coming asap, but no sign yet. Even when it does there will still be situations where some queries won't run without an entry. How does an app running on TyphoonAE handle these situations? I know it is based on the SDK, and the SDK will run any query, placing an entry in index.yaml for those that need it. Can I run any query, whether it has index entry or not on TyphoonAE? Will it automatically build indexes for new queries that don't have one? How does it achieve this with the various datastore backends?

exploding index - related to the last question, can I run queries on TyphoonAE that would cause an exploding index on GAE? e.g. "WHERE listAttr = X AND listAttr = Y AND listAttr = Z"

5000 attribute limit - does TyphoonAE enforce the 5000 attribute limit on a model? This also impact the length of a list attribute. What limits are there?

performance and indexes - how does TyphoonAE manage indexing - e.g. with the MySQL backend.

Thanks again,

Colin

[Tobias]

Hi Colin - Great to hear from you!

Generally said, neither the available Datastore API Proxy Stubs
(MongoDB, MySQL) nor TyphoonAE's appserver nor the configuration part
interpret index.yaml definitions right now. Joaquin and I discussed
that a while ago. In other words, all queries are translated to the
particular query API of the configured Datastore backend without using
index definitions from the index.yaml file.

I've filed this http://code.google.com/p/typhoonae/issues/detail?id=80
issue.

However, most queries should work as expected. We have a few
limitations when using the MongoDB backend, though. It does not
support Expandos and kindless ancestor queries.

You could review the test suites for our Datastore backends to get an
overview what's working.

http://code.google.com/p/typhoonae/source/browse/src/typhoonae/mongodb/tests/test_datastore_mongo.py

http://code.google.com/p/typhoonae/source/browse/src/typhoonae/mysql/tests/test_datastore_mysql.py

Keeping the above in mind, we don't have exploding indexes in the way
GAE has. Although, there are other possible bottlenecks depending on
the particular backend. Furthermore, we have no hard limits on the
number of attributes. To be honest, I've never tested a situation
where a model provides more than 10 or 20 attributes.

I'm going to review the code and get beck to you on these issues.
Sorry, that I can't give you more detailed answers at the moment and
thanks a lot for pointing to this and for your patience.

- Tobias

[Joaquin Cuenca Abela]

A small extra comment on the mongodb backend. Another of its limitations is that if you have a single entity with a missing field in your collection, queries against this field can fail. I have to catch a plane, see the archives of the ML for more details.

Cheers

> --
> You received this message because you are subscribed to the Google
> Groups "typhoonae" group.
> To post to this group, send email to typh...@googlegroups.com
> To unsubscribe from this group, send email to
> typhoonae+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/typhoonae?hl=en

[Tobias]

Hi all,

Haven't been browsing the code for some time now, but Joaquin is
absolutely correct. This http://groups.google.com/group/typhoonae/browse_thread/thread/6c4ccbe4ddb208c8
is the related thread.

Before I go into more details, I'd like to do a quick poll.

Regarding Datastore, where would you prefer to put efforts into?

a) Refactoring the MongoDB Stub and implementing support for Expando,
kindless ancestor queries, index.yaml and MongoDB create_index support

b) Implementing a Cassandra Datastore Stub

c) ...

Since MySQL isn't the perfect Datastore backend from an architectural
point of view, I wouldn't put much effort into it apart from
bugfixing.

Thanks,
Tobias

On Jan 29, 4:49 pm, Joaquin Cuenca Abela <e98cu...@gmail.com> wrote:
> A small extra comment on the mongodb backend. Another of its limitations is
> that if you have a single entity with a missing field in your collection,
> queries against this field can fail. I have to catch a plane, see the
> archives of the ML for more details.
>
> Cheers

> El 29/01/2011 13:51, "Tobias" <tobias.rodae...@googlemail.com> escribió:
>
>
>
> > Hi Colin - Great to hear from you!
>
> > Generally said, neither the available Datastore API Proxy Stubs
> > (MongoDB, MySQL) nor TyphoonAE's appserver nor the configuration part
> > interpret index.yaml definitions right now. Joaquin and I discussed
> > that a while ago. In other words, all queries are translated to the
> > particular query API of the configured Datastore backend without using
> > index definitions from the index.yaml file.
>

> > I've filed thishttp://code.google.com/p/typhoonae/issues/detail?id=80

> > issue.
>
> > However, most queries should work as expected. We have a few
> > limitations when using the MongoDB backend, though. It does not
> > support Expandos and kindless ancestor queries.
>
> > You could review the test suites for our Datastore backends to get an
> > overview what's working.
>

> http://code.google.com/p/typhoonae/source/browse/src/typhoonae/mongod...
>
> http://code.google.com/p/typhoonae/source/browse/src/typhoonae/mysql/...

> > typhoonae+...@googlegroups.com<typhoonae%2Bunsubscribe@googlegroups .com>

[Colin Hawkett]

Hi Tobias,

I was actually able to validate my app on MySQL last year :) It held some appeal as the AWS RDS instances could be useful. Mostly my questions relate to performance issues as data increases when there are no indexes. Perhaps you are thinking that indexes should be created manually using the backend tooling? I was also wondering about TyphoonAE's ability to release me from the exploding index, which is a related problem. I think these questions would be relevant regardless of which backend datastore you work on.

From my perspective, if you think cassandra is going to be a better backend in the long term, then it makes sense to spend time there. Are the issues around MySQL simply that it is relational? Or also around scaling? I also wonder how you manage GAE transactions in these backends which do not support them - do you use a memcache lock or similar device?

Also when you mention index.yaml support in mongoDB - are you talking about auto-creating indexes as the SDK auto-populates index.yaml, or static creation at deploy time? If you worked on Cassandra would you consider the same functionality for that backend?

Cheers,

Colin

[Tobias]

Hi Colin,

The MySQL Datastore stub is basically implemented like Nick Johnson's
SQLite stub from the SDK. Both share large portions of the same code.
So, it uses the same tables for storing entities and properties as
described in Nick's post
http://blog.notdot.net/2010/03/Announcing-the-SQLite-datastore-stub-for-the-Python-App-Engine-SDK

And yes, it's much harder to scale with MySQL than with MongoDB. On
the other hand, our MongoDB backend doesn't support GAE transactions.
Snapshot isolation and transaction serialization are missing. I was
looking for a way to avoid locks lately.

Regarding indexes, MongoDB provides a
db[collection].create_index(spec) method. The good news is, our
MongoDB stub already has an experimental implementation for creating
MongoDB indexes. I think the corresponding methods should be triggered
at deploy time when reading index.yaml. Using the development server
from the SDK to auto-generate index.yaml might be an acceptable
workaround.

Cheers,
Tobias