I got in touch with Jacob who suggested I contact you all together. As you may (or may not) know, at Google I/O back in May, we announced a hosted cloud SQL service as a new feature for our App Engine application cloud-hosting platform:
Our offering is mature enough now that we are exploring the final APIs that we’d like to ship as part of the App Engine Python SDK. As such, we’re exploring what’s required to enable the Django to use our MySQL-compatible library, thus providing ORM access to the App Engine datastore. We’re writing you at this time to get a sense of the effort that would be required to make this happen. We are excited about this new feature and would love to have some community involvement to get Django on-board as an option for users!
Best regards, -Wesley Chun, Guido van Rossum, and Sean Lynch, Google App Engine team - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - "Core Python Programming", Prentice Hall, (c)2007,2001 "Python Fundamentals", Prentice Hall, (c)2009 http://corepython.com
wesley.chun : wesc+api at google.com : @wescpy developer relations :: google app engine @app_engine :: googleappengine.blogspot.com
On Tue, Oct 26, 2010 at 6:56 PM, wesley chun <wes...@gmail.com> wrote: > Dear Django developers,
> I got in touch with Jacob who suggested I contact you all together. As > you may (or may not) know, at Google I/O back in May, we announced a > hosted cloud SQL service as a new feature for our App Engine > application cloud-hosting platform:
> Our offering is mature enough now that we are exploring the final APIs > that we’d like to ship as part of the App Engine Python SDK. As such, > we’re exploring what’s required to enable the Django to use our > MySQL-compatible library, thus providing ORM access to the App Engine > datastore. We’re writing you at this time to get a sense of the effort > that would be required to make this happen. We are excited about this > new feature and would love to have some community involvement to get > Django on-board as an option for users!
> wesley.chun : wesc+api at google.com : @wescpy > developer relations :: google app engine > @app_engine :: googleappengine.blogspot.com
> -- > You received this message because you are subscribed to the Google Groups "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
It depends what you mean by "MySQL" compatible. If by that you mean that it's importable as mysqldb and implements the exact API it should work out of the box, more or less. However, if by that you mean it implements PEP-249 you really need to ship a database backend for Django (you probably want to do this anyway, I'll explain). Since this is SQL backend this isn't quiet so much work. Database backends basically consist of 2 things: a compiler, which is the core of the SQL generation, and "the rest of it" which is a series of flags and methods which make very small scale decisions about what features are available and how certain constructs are translated into SQL.
The good thing is, assuming you've implemented SQL in something close to the standard (or any of the other DBs we support), the compiler will work more or less out of the box, and all you have to do is fill in the methods/flags on the backend for it to work, you can look to the external MSSQL, Sybase, or DB2 backend to see how this works. Of course if you support only a subset of SQL you'll probably need to override the compiler to either work around this (e.g. JOIN emulation) or raise an appropriate error.
The links you provided don't give a ton of info about the exact nature of the SQL support (and I was fairly ignorant about it going in), but hopefully that helps and we can continue to help with more specific questions, Alex
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me
On Wed, Oct 27, 2010 at 6:56 AM, wesley chun <wes...@gmail.com> wrote: > Dear Django developers,
> I got in touch with Jacob who suggested I contact you all together. As > you may (or may not) know, at Google I/O back in May, we announced a > hosted cloud SQL service as a new feature for our App Engine > application cloud-hosting platform:
> Our offering is mature enough now that we are exploring the final APIs > that we’d like to ship as part of the App Engine Python SDK. As such, > we’re exploring what’s required to enable the Django to use our > MySQL-compatible library, thus providing ORM access to the App Engine > datastore. We’re writing you at this time to get a sense of the effort > that would be required to make this happen. We are excited about this > new feature and would love to have some community involvement to get > Django on-board as an option for users!
Hi Wesley,
Sounds exciting!
It's difficult to judge the amount of effort that would be involved without knowing exactly how comprehensive the SQL-compatibility layer is. The best way to give us an idea of how much work needs to be done would be to run Django's test suite over your AppEngine SQL datastore.
Django's database backends are fairly well abstracted representations of the eccentricities of SQL datastores, so if your MySQL emulation is reasonably complete, it should be possible to either use Django's MySQL backend, or adapt the MySQL backend into a custom 'AppEngine SQL' backend. Django allows (and encourages) the use of external backends, so shipping an external backend would be a natural way to provide AppEngine support.
If it turns out that there are some additional interfaces needed on the backend in order to support the eccentricities of AppEngine, we're certainly open to introducing those interfaces. The window for committing new features is open right now (and will be till the end of November); if you can give an indication of the eccentricities that need to be accommodated (by giving us a list of test failures and an indication of why those tests are failing), we can look at the changes that we may need to make.
On Tue, Oct 26, 2010 at 7:23 PM, Alex Gaynor <alex.gay...@gmail.com> wrote: > It depends what you mean by "MySQL" compatible. If by that you mean > that it's importable as mysqldb and implements the exact API it should > work out of the box, more or less. However, if by that you mean it > implements PEP-249 you really need to ship a database backend for > Django (you probably want to do this anyway, I'll explain). Since > this is SQL backend this isn't quiet so much work. Database backends > basically consist of 2 things: a compiler, which is the core of the > SQL generation, and "the rest of it" which is a series of flags and > methods which make very small scale decisions about what features are > available and how certain constructs are translated into SQL.
Also important is a creation module which knows how to map Django model field types to whatever the appropriate column types are in the data store.
-- "Bureaucrat Conrad, you are technically correct -- the best kind of correct."
On Tue, Oct 26, 2010 at 9:31 PM, James Bennett <ubernost...@gmail.com> wrote: > On Tue, Oct 26, 2010 at 7:23 PM, Alex Gaynor <alex.gay...@gmail.com> wrote: >> It depends what you mean by "MySQL" compatible. If by that you mean >> that it's importable as mysqldb and implements the exact API it should >> work out of the box, more or less. However, if by that you mean it >> implements PEP-249 you really need to ship a database backend for >> Django (you probably want to do this anyway, I'll explain). Since >> this is SQL backend this isn't quiet so much work. Database backends >> basically consist of 2 things: a compiler, which is the core of the >> SQL generation, and "the rest of it" which is a series of flags and >> methods which make very small scale decisions about what features are >> available and how certain constructs are translated into SQL.
> Also important is a creation module which knows how to map Django > model field types to whatever the appropriate column types are in the > data store.
> -- > "Bureaucrat Conrad, you are technically correct -- the best kind of correct."
> -- > You received this message because you are subscribed to the Google Groups "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
What James says is completely correct, this is another element of "how close to MySQL is it", for example all of the creation module may be a no-op if it retains the BigTable semantic of arbitrary key-value storage, or it may be exactly the same as the existing MySQL one, or somewhere in between.
Alex
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me
Maybe it would also be helpful to point out in advance what known
behaviors (often considered as limitations) of the datastore will be
carried to the MySQL compatibility layer that Django would have to
deal with.
For example, currently using GQL counting the number of rows returned
by a query is/was really tricky because of the 1000 rows limit.
I also recall there was a 1 MB limit on the size of every item you
could store.
I've collected some links regarding this kind of stuff that I'm
including here:
