Django on Google App Engine via SQL (not nonrel)
wesley chun
I got in touch with Jacob who suggested I contact you all together. As
you may (or may not) know, at Google I/O back in May, we announced a
hosted cloud SQL service as a new feature for our App Engine
application cloud-hosting platform:
http://code.google.com/appengine/business/#features
http://googlecode.blogspot.com/2010/05/announcing-google-app-engine-for.html
Our offering is mature enough now that we are exploring the final APIs
that we’d like to ship as part of the App Engine Python SDK. As such,
we’re exploring what’s required to enable the Django to use our
MySQL-compatible library, thus providing ORM access to the App Engine
datastore. We’re writing you at this time to get a sense of the effort
that would be required to make this happen. We are excited about this
new feature and would love to have some community involvement to get
Django on-board as an option for users!
Best regards,
-Wesley Chun, Guido van Rossum, and Sean Lynch, Google App Engine team
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
"Core Python Programming", Prentice Hall, (c)2007,2001
"Python Fundamentals", Prentice Hall, (c)2009
http://corepython.com
wesley.chun : wesc+api at google.com : @wescpy
developer relations :: google app engine
@app_engine :: googleappengine.blogspot.com
Alex Gaynor
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
It depends what you mean by "MySQL" compatible. If by that you mean
that it's importable as mysqldb and implements the exact API it should
work out of the box, more or less. However, if by that you mean it
implements PEP-249 you really need to ship a database backend for
Django (you probably want to do this anyway, I'll explain). Since
this is SQL backend this isn't quiet so much work. Database backends
basically consist of 2 things: a compiler, which is the core of the
SQL generation, and "the rest of it" which is a series of flags and
methods which make very small scale decisions about what features are
available and how certain constructs are translated into SQL.
The good thing is, assuming you've implemented SQL in something close
to the standard (or any of the other DBs we support), the compiler
will work more or less out of the box, and all you have to do is fill
in the methods/flags on the backend for it to work, you can look to
the external MSSQL, Sybase, or DB2 backend to see how this works. Of
course if you support only a subset of SQL you'll probably need to
override the compiler to either work around this (e.g. JOIN emulation)
or raise an appropriate error.
The links you provided don't give a ton of info about the exact nature
of the SQL support (and I was fairly ignorant about it going in), but
hopefully that helps and we can continue to help with more specific
questions,
Alex
--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
Russell Keith-Magee
> Dear Django developers,
>
> I got in touch with Jacob who suggested I contact you all together. As
> you may (or may not) know, at Google I/O back in May, we announced a
> hosted cloud SQL service as a new feature for our App Engine
> application cloud-hosting platform:
>
> http://code.google.com/appengine/business/#features
> http://googlecode.blogspot.com/2010/05/announcing-google-app-engine-for.html
>
> Our offering is mature enough now that we are exploring the final APIs
> that we’d like to ship as part of the App Engine Python SDK. As such,
> we’re exploring what’s required to enable the Django to use our
> MySQL-compatible library, thus providing ORM access to the App Engine
> datastore. We’re writing you at this time to get a sense of the effort
> that would be required to make this happen. We are excited about this
> new feature and would love to have some community involvement to get
> Django on-board as an option for users!
Hi Wesley,
Sounds exciting!
It's difficult to judge the amount of effort that would be involved
without knowing exactly how comprehensive the SQL-compatibility layer
is. The best way to give us an idea of how much work needs to be done
would be to run Django's test suite over your AppEngine SQL datastore.
Django's database backends are fairly well abstracted representations
of the eccentricities of SQL datastores, so if your MySQL emulation is
reasonably complete, it should be possible to either use Django's
MySQL backend, or adapt the MySQL backend into a custom 'AppEngine
SQL' backend. Django allows (and encourages) the use of external
backends, so shipping an external backend would be a natural way to
provide AppEngine support.
If it turns out that there are some additional interfaces needed on
the backend in order to support the eccentricities of AppEngine, we're
certainly open to introducing those interfaces. The window for
committing new features is open right now (and will be till the end of
November); if you can give an indication of the eccentricities that
need to be accommodated (by giving us a list of test failures and an
indication of why those tests are failing), we can look at the changes
that we may need to make.
Yours,
Russ Magee %-)
James Bennett
> It depends what you mean by "MySQL" compatible. If by that you mean
> that it's importable as mysqldb and implements the exact API it should
> work out of the box, more or less. However, if by that you mean it
> implements PEP-249 you really need to ship a database backend for
> Django (you probably want to do this anyway, I'll explain). Since
> this is SQL backend this isn't quiet so much work. Database backends
> basically consist of 2 things: a compiler, which is the core of the
> SQL generation, and "the rest of it" which is a series of flags and
> methods which make very small scale decisions about what features are
> available and how certain constructs are translated into SQL.
Also important is a creation module which knows how to map Django
model field types to whatever the appropriate column types are in the
data store.
--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."
Alex Gaynor
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
What James says is completely correct, this is another element of "how
close to MySQL is it", for example all of the creation module may be a
no-op if it retains the BigTable semantic of arbitrary key-value
storage, or it may be exactly the same as the existing MySQL one, or
somewhere in between.
Antonio Ognio
Maybe it would also be helpful to point out in advance what known
behaviors (often considered as limitations) of the datastore will be
carried to the MySQL compatibility layer that Django would have to
deal with.
For example, currently using GQL counting the number of rows returned
by a query is/was really tricky because of the 1000 rows limit.
I also recall there was a 1 MB limit on the size of every item you
could store.
I've collected some links regarding this kind of stuff that I'm
including here:
http://aralbalkan.com/1504
http://stackoverflow.com/questions/421751/whats-the-best-way-to-count-results-in-gql
http://stackoverflow.com/questions/264154/google-appengine-how-to-fetch-more-than-1000
http://stackoverflow.com/questions/703892/whats-your-experience-developing-on-google-app-engine
http://blog.burnayev.com/2008/04/gql-limitations.html
Hope this help a bit discuss this matters,
Antonio
Lima-Peru