App Engine port
Waldemar Kornewald
first of all, let me introduce myself. I'm the main developer behind
app-engine-patch which is a port of Django to App Engine:
http://code.google.com/p/app-engine-patch/
Our repository version of Django now even supports the admin interface
and almost all of Django's other features, so our code could be reused
in an official port.
The problem here is that I have to use App Engine's Model class
everywhere. The port could've been much easier if Django's Model were
supported. App Engine's field classes don't even take a help_text
parameter and the whole DB API is a little bit disappointing compared
to Django. Also, I have to apply monkey-patches in a few places in
order to fix bugs which Google doesn't seem to have enough time for. :
(
A native Django port could provide us with a clean, more powerful, and
more actively developed API for App Engine development. Moreover, it
could provide a solution for provider lock-in by abstracting the
underlying API and thus making easy migrations possible.
I know that a lot of changes in the storage/query backend code need to
be done in order to support the App Engine datastore and similar
databases and I doubt that I can do the necessary API changes, but I
could at least help with implementing the App Engine backends
(sessions, cache, mail, db/storage). As a starting point, I've
documented a few high-level requirements here:
http://code.google.com/p/app-engine-patch/wiki/DjangoAppEnginePort
Is there any roadmap towards App Engine support?
Is anyone working on it, already?
When will someone work on it and in which Django release can we expect
native support? Is 1.2 possible?
BTW, could someone please activate my "wkornewald" Trac account, so I
can move the documentation into your wiki? I just don't receive any
activation mail when registering. I tried sending to
webm...@djangoproject.com, but that address doesn't exist. Who can I
talk to?
Bye,
Waldemar Kornewald
Russell Keith-Magee
>
> A native Django port could provide us with a clean, more powerful, and
> more actively developed API for App Engine development. Moreover, it
> could provide a solution for provider lock-in by abstracting the
> underlying API and thus making easy migrations possible.
You won't get any disagreement here. Native support for AppEngine
within Django would be a nice feature. Support for other storage
backends (like CouchDB, Amazon SimpleDB, and so on) would also be nice
to have.
> I know that a lot of changes in the storage/query backend code need to
> be done in order to support the App Engine datastore and similar
> databases and I doubt that I can do the necessary API changes, but I
> could at least help with implementing the App Engine backends
> (sessions, cache, mail, db/storage). As a starting point, I've
> documented a few high-level requirements here:
> http://code.google.com/p/app-engine-patch/wiki/DjangoAppEnginePort
>
> Is there any roadmap towards App Engine support?
Is there a formal roadmap? No. App Engine support wasn't mentioned on
our v1.1 roadmap, and we don't plan beyond the next release.
> Is anyone working on it, already?
I'm not aware of any public projects. Reading between the lines, I
suspect I know at least one person that is working on App Engine
integration. However, I don't want to steal their thunder, so I'll
leave it up to them to show their hand if they feel it is appropriate.
> When will someone work on it and in which Django release can we expect
> native support? Is 1.2 possible?
The code isn't going to magically write itself, and it will be a
non-trivial job. You can expect native support right after someone
commits to implementing it :-)
Whether this is possible for v1.2 is very much dependent on who puts
in the work, and how much time they can put into the task. They will
also need to convince a core developer that you are serious, and that
they are capable of bringing the project to fruition. Providing an
initial patch -- even if it is incomplete -- will go a long way to
providing this proof.
I would also note that our (that is, the Core Developers) initial
reaction to non-SQL storage backends will be the same as it is for SQL
database backends -- we will want to see public interest and continued
support as a standalone project before we commit to integrating
anything into the Django core. This policy is why we have an Oracle
backend, but we don't have an MSSQL backend. There are, however,
several external projects providing MSSQL integration.
I would suggest to you that the broader project of "modifying the
django.db.models interface to be fully independent of SQL" is much
more likely to get core developer support. We (the Django core) will
be very receptive to any suggestions on how django.db.models needs to
be modified in order to support non-SQL backends -- especially if you
can demonstrate that your suggestions aren't just theoretical, but are
clearly required by an actual backend implementation.
The goal here isn't to get AppEngine support into the core - it is to
provide an interface in Django that allows you to implement an
AppEngine backend (or any other backend, for that matter) as a
completely standalone project. In time, this may lead to integration
into the Django core, but in the interim, we would rather see a
vibrant ecosystem of external plugins, rather than try and force
everything into Django's trunk.
> BTW, could someone please activate my "wkornewald" Trac account, so I
> can move the documentation into your wiki? I just don't receive any
> activation mail when registering. I tried sending to
> webm...@djangoproject.com, but that address doesn't exist. Who can I
> talk to?
I can mail you activation code privately. Expect an email from me shortly.
Yours,
Russ Magee %-)
Russell Keith-Magee
> can move the documentation into your wiki?
After I pressed send, something else occurred to me:
Before you starting loading documentation into Django's wiki, please
consider if Django's wiki is the right place for that documentation.
If you are maintaining an external project handling AppEngine support
for Django, then that project's wiki is the right place for
documentation about AppEngine support within Django. While AppEngine
support isn't part of Django's core, Django's wiki isn't the right
place to be providing AppEngine specific instructions - especially if
the first instruction is "install this third party project that isn't
formally affiliated with the Django project at this time".
Yours,
Russ Magee %-)
Waldemar Kornewald
On 6 Feb., 11:34, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
> I would suggest to you that the broader project of "modifying the
> django.db.models interface to be fully independent of SQL" is much
> more likely to get core developer support. We (the Django core) will
> be very receptive to any suggestions on how django.db.models needs to
> be modified in order to support non-SQL backends -- especially if you
> can demonstrate that your suggestions aren't just theoretical, but are
> clearly required by an actual backend implementation.
complicated) part. Do the people secretly working on App Engine
integration try to implement that part?
On 6 Feb., 11:53, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
necessary for a completely new port that has practically nothing to do
with app-engine-patch apart from being able to reuse some of the
existing code. It could be useful for any porting effort. If I keep it
in my wiki nobody else can work on that page. It's almost useless that
way.
Bye,
Waldemar Kornewald
Russell Keith-Magee
>
> Hi Russell,
>
> On 6 Feb., 11:34, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
>> I would suggest to you that the broader project of "modifying the
>> django.db.models interface to be fully independent of SQL" is much
>> more likely to get core developer support. We (the Django core) will
>> be very receptive to any suggestions on how django.db.models needs to
>> be modified in order to support non-SQL backends -- especially if you
>> can demonstrate that your suggestions aren't just theoretical, but are
>> clearly required by an actual backend implementation.
>
> Agreed, making models SQL-independent is the most important (and most
> complicated) part. Do the people secretly working on App Engine
> integration try to implement that part?
I haven't seen their code, so I can't say for certain, but I imagine
that they have a small collection of patches for the main Django tree
that enables their backend to be independent of SQL.
>> If you are maintaining an external project handling AppEngine support
>> for Django, then that project's wiki is the right place for
>> documentation about AppEngine support within Django. While AppEngine
>> support isn't part of Django's core, Django's wiki isn't the right
>> place to be providing AppEngine specific instructions - especially if
>> the first instruction is "install this third party project that isn't
>> formally affiliated with the Django project at this time".
>
> I can't find anything like that in my wiki page. It describes what is
> necessary for a completely new port that has practically nothing to do
> with app-engine-patch apart from being able to reuse some of the
> existing code. It could be useful for any porting effort. If I keep it
> in my wiki nobody else can work on that page. It's almost useless that
> way.
Ok - sounds like they could be some good contributions. I just wanted
to make sure you were not planning on making the Django wiki the home
of the "how to use app-engine-patch" documentation.
Yours,
Russ Magee %-)
Waldemar Kornewald
> Ok - sounds like they could be some good contributions. I just wanted
> to make sure you were not planning on making the Django wiki the home
> of the "how to use app-engine-patch" documentation.
I've moved the document here:
http://code.djangoproject.com/wiki/AppEngine
Bye,
Waldemar Kornewald
David Stenglein
App Engine? It would
seem to me that App Engine has been a feather in the cap for Django. A
lot of people don't know Django and at a previous job, I was able to
say that "Google chose django" for App Engine to help validate my use
of it.
I just glanced at the app engine project page and there is even an
article about porting your apps to run in any standard Django
environment.
It may be coincidental, but I thought that a regularized release
schedule (rather than just "use SVN") seemed to come about after the
App Engine announcement.
Has App Engine created a noticeable increase in developers interested
in using Django?
If so, why the seeming lack of excitement around App Engine? Are
google engineers not involved in the community?
Sorry to be posing so many questions, but I've found this aspect of
Django development particularly puzzling.
-Dave
On Fri, Feb 6, 2009 at 7:10 AM, Russell Keith-Magee
<freakboy3...@gmail.com> wrote:
James Bennett
<david.s...@gmail.com> wrote:
> I have to ask a question here. Why is there such reticence regarding
> App Engine?
I'm curious as to what reticence you think there is; since, as Russell
said, code to get a straight Django with all features working on App
Engine would be welcome, I don't really get the same vibe. The only
thing is that we're past the point where major features like this can
be proposed for Django 1.1; we're aleady a bit behind schedule getting
out the first alpha release, and delaying it even further isn't
something we can do at the moment.
--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."
Russell Keith-Magee
<david.s...@gmail.com> wrote:
>
> I have to ask a question here. Why is there such reticence regarding
> App Engine? It would
> seem to me that App Engine has been a feather in the cap for Django. A
> lot of people don't know Django and at a previous job, I was able to
> say that "Google chose django" for App Engine to help validate my use
> of it.
Reticence isn't a particularly accurate description of the situation.
The Django Core developers are all volunteers, so our time is limited.
We work on the things that scratch our particular itches. Speaking
personally (and I suspect the same is probably true of the rest of the
core developers), I don't have a lot of use for App Engine at the
moment, and I have a lot of other pressing concerns, so I'm not
spending a lot of personal time on the problem.
For me, App Engine is in a similar camp to Python 3: a very
interesting idea, but not a development priority _right now_. Given
time, I expect this to change (commercial hosts will start providing
Python 3 support/Google will take App Engine out of preview and into
full release), but for the moment, I have a lot of other issues that
affect me on a daily basis, and are therefore much more compelling for
me to work on.
That said, if Google (or anyone else, for that matter) wants to throw
some engineers at addressing this issue, we will gratefully accept the
assistance. We are in no way opposed to improving App Engine support.
If a well designed, well written patch were to arrive on our doorstep
today, I suspect you would find that patch in Django's trunk in short
order.
> It may be coincidental, but I thought that a regularized release
> schedule (rather than just "use SVN") seemed to come about after the
> App Engine announcement.
This was complete coincidence. Our v1.0 timeline was decided upon
independent of any App Engine announcement or plans. Google hasn't
actively engaged the Django community to coordinate release plans,
schedules, or feature lists.
> Has App Engine created a noticeable increase in developers interested
> in using Django?
There has certainly been some interest from App Engine that flows into
Django. You don't have to hang out on Django-users for too long to
find questions that reference App Engine.
> If so, why the seeming lack of excitement around App Engine?
This question presupposes that there should be excitement. Rephrasing
the question, why should we be excited?
On the one hand - the tools they are exposing (BigTable, easy
deployment environment, etc) are very exciting developments, and they
clearly work well for Google itself.
However, it's not like Apache and *SQL installations are going to go
away overnight. There are some very good reasons to _not_ use App
Engine - not the least of which is the fact that Google lists App
Engine as a "preview" product (it doesn't even make it to Googles
infamous "Beta" stage :-). At the moment, the vast majority of
Django's audience (and 100% of the audience that pays my salary)
aren't using App Engine, and wouldn't be able to (for commercial
reasons) even if they wanted to.
Yes, it's flattering to have one of the largest software companies in
the world using Django. However, that doesn't mean that the Django
Core developers are all going to drop everything and make App Engine
their development priority.
> Are
> google engineers not involved in the community?
There are a few tickets in Django's ticket tracker that I know have
come from Google engineers, but those individuals haven't made any big
noises about App Engine support. By way of evidence: when the v1.1
feature suggestion list was opened, nobody suggested App Engine
support as a potential feature.
Django is an open source project. If Google wants App Engine support
in Django, they're free to engage with the community and make it
happen. If any other company wants App Engine Support in Django,
they're free to engage with the community and make it happen. If any
individual wants App Engine Support in Django, they're free to engage
with the community and make it happen. If you're not willing to write
the code yourself, then you either need to convince someone to engage
with the community and make it happen, pay someone to engage with the
community and make it happen, or wait until it becomes a big enough
itch for someone that it gets scratched.
Yours,
Russ Magee %-)
Malcolm Tredinnick
> I have to ask a question here. Why is there such reticence regarding
> App Engine? It would
> seem to me that App Engine has been a feather in the cap for Django. A
> lot of people don't know Django and at a previous job, I was able to
> say that "Google chose django" for App Engine to help validate my use
> of it.
To echo everybody else; There simply isn't any reticence.
>
> If so, why the seeming lack of excitement around App Engine? Are
> google engineers not involved in the community?
Possibly. If they are, they aren't posting from their @google.com
addresses, as far as I can see on this list.
I would point out that the original GAE integration with Django could
have been implemented quite differently and a lot of this could have
Just Worked out of the box. However, there were reasonably (not
perfectly) valid corporate and legal reasons why Google chose not to
and/or couldn't do that. So the thing works both ways. Google made some
choices and this thread is now about working around those choices to
integrate more nicely.
Regards,
Malcolm
Malcolm Tredinnick
> Hi Russell,
>
> On 6 Feb., 11:34, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
> > I would suggest to you that the broader project of "modifying the
> > django.db.models interface to be fully independent of SQL" is much
> > more likely to get core developer support. We (the Django core) will
> > be very receptive to any suggestions on how django.db.models needs to
> > be modified in order to support non-SQL backends -- especially if you
> > can demonstrate that your suggestions aren't just theoretical, but are
> > clearly required by an actual backend implementation.
>
> Agreed, making models SQL-independent is the most important (and most
> complicated) part. Do the people secretly working on App Engine
> integration try to implement that part?
I have a reasonably fleshed out plan to make things easier here in the
Django 1.2 timeframe. The rough idea is that everything under
django/db/models/sql/ could be replaced with a module of the developer's
choosing (probably set via a setting). That package is the only place
that really cares much about SQL.
So somebody wanting a GAE backend or an Hadoop backend or something else
would write something that behaved the same way as the Query class (and
subclasses) and could be called by the QuerySet class appropriately.
There are a few more details involved there than just that one setting,
since it requires documenting the expectations (vs the currently
convenient assumptions) that Queryset will make on the Query class --
probably that's pretty much what's there now, although I have a list
somewhere of a couple/three things I wanted to look at. There's also
simple enough stuff like removing the last vestiges of raw SQL from a
couple of places (most notable related fields, but that should probably
go after, or in tandem with, a rewrite of related fields to clean them
up in general).
There's also some thinking I have to do about where to draw the line for
things like custom lookup types on fields -- how much do we encourage
(with the noisy end of a cattle prod) people to be good
extension-oriented citizens down the track and where can we help.
We've tried pretty hard to keep the SQL-agnostic portions of the API
separates from the database bits. I've made Russell's life a bit harder
lately by asking for -- fairly insistently -- such changes in the
aggregates patch to help with this goal going forwards, for example. The
public API for querysets is fairly SQL agnostic, with the exception of
Black Sheep extra(), and in that latter case, sometimes there's just
nothing you can do about that. We take a fair bit of grief both here and
in django-users for having this position, actually. Lots of requests for
compromises to allow some half-breed combinations of raw SQL and
querysets, which has hidden traps at both the design level and the
extension level we're talking about here. So the changes you're talking
about are certainly possible.
Regards,
Malcolm
David Stenglein
Thanks for taking the time to respond so thoroughly to my questions.
I think that these questions are rooted in the unrealistic idea that
something that
looks like it would be good relationship- or PR-wise is also a good development
priority.
As I expressed, I've simply thought that GAE would be a no-brainer, but that is
obviously not the case when viewed against other development work with more
concrete demand.
Now it is time for me to put-up or shut-up. Hopefully it will not be
the latter (and
making a post like this can be a good motivator)!
-Dave
Vinay Sajip
On Feb 10, 4:25 am, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> Django 1.2 timeframe.
class which would work well as far as querying is concerned, but
doesn't seem to be so useful for inserting, updating and deleting, and
date and count queries. That's because these are implemented as
subclasses of Query, and although we can substitute our own Query
subclass for Query, as is done in the Oracle backend, what we actually
need is to be able to also affect the behaviour of the subclasses. If
one wanted e.g. some special syntax for an INSERT query, how would one
hook into the machinery to deliver custom SQL for insertions?
Regards,
Vinay Sajip
David Larlet
Le 10 févr. 09 à 05:25, Malcolm Tredinnick a écrit :
> There's also
> simple enough stuff like removing the last vestiges of raw SQL from a
> couple of places (most notable related fields, but that should
> probably
> go after, or in tandem with, a rewrite of related fields to clean them
> up in general).
That's the goal of #10109. Good to know there is eventually an
interest in this.
Regards,
David
Alex Gaynor
Alex
--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero
David Larlet
Le 10 févr. 09 à 17:53, Alex Gaynor a écrit :
> David, I think I mentioned this to you, but I believe that by using
> the new F() objects we can actually remove the raw SQL entirely, I'd
> need to put some more thought into this(and perhaps Malcolm already
> has) but it should be possible.
Alex, I agree with you (not yet tried though), it's only a timing
issue. There are two possibilities: moving then reF()actoring or doing
both in the meantime/same patch. I adopted the first one because I got
no answer to my previous email and that's the blocking point to me, I
thought there were more chance that it get commited quickly keeping
the patch as simple as possible.
Now, if you think it's important to do both and provide some feedback
on the way I did the move I'll be glad to work on it.
Regards,
David
Alex Gaynor
Waldemar Kornewald
On 10 Feb., 05:25, Malcolm Tredinnick <malc...@pointy-stick.com>
wrote:
> Django 1.2 timeframe. The rough idea is that everything under
> django/db/models/sql/ could be replaced with a module of the developer's
> choosing (probably set via a setting). That package is the only place
> that really cares much about SQL.
>
> So somebody wanting a GAE backend or an Hadoop backend or something else
> would write something that behaved the same way as the Query class (and
> subclasses) and could be called by the QuerySet class appropriately.
App Engine support requires at least that Model and probably a few
other classes can be overridden (maybe partially). For example,
save_base() makes a few queries to check if the row already exists,
but this would have to be done differently on App Engine (hopefully
with a transaction).
BTW, Mitchell Garnaat, the creator of boto (popular AWS library for
Python), might be interested in helping with a SimpleDB port:
http://groups.google.com/group/boto-users/msg/c68e9456d2c6393e
Bye,
Waldemar Kornewald
Malcolm Tredinnick
> Hi Malcolm,
>
> On 10 Feb., 05:25, Malcolm Tredinnick <malc...@pointy-stick.com>
> wrote:
> > I have a reasonably fleshed out plan to make things easier here in the
> > Django 1.2 timeframe. The rough idea is that everything under
> > django/db/models/sql/ could be replaced with a module of the developer's
> > choosing (probably set via a setting). That package is the only place
> > that really cares much about SQL.
> >
> > So somebody wanting a GAE backend or an Hadoop backend or something else
> > would write something that behaved the same way as the Query class (and
> > subclasses) and could be called by the QuerySet class appropriately.
>
> Is the plan somewhere on the wiki?
No, because it's only something I'm pulling together slowly in my head.
> App Engine support requires at least that Model and probably a few
> other classes can be overridden (maybe partially)
> For example,
> save_base() makes a few queries to check if the row already exists,
> but this would have to be done differently on App Engine (hopefully
> with a transaction).
No. Backends should require piecemeal futzing around at that level --the
Model level should be reasonably abstracted from backend specifics. We
should, instead, fix the problem more generically by moving those tests
into the django/db/models/sql/ level in some fashion.
That's one of those "we'll do it when we need it" tasks. Since this is
the first time anybody's mentioned, we now have a reason to look at it.
Regards,
Malcolm
Waldemar Kornewald
wrote:
>
> No, because it's only something I'm pulling together slowly in my head.
enough to work even with App Engine's limitations (e.g., it's
practically impossible to implement a TransactionMiddleware):
http://code.djangoproject.com/wiki/AppEngine
SimpleDB is even worse because of "eventual consistency" which means
there are no transactions and absolutely no integrity guarantees.
> > App Engine support requires at least that Model and probably a few
> > other classes can be overridden (maybe partially)
> > For example,
> > save_base() makes a few queries to check if the row already exists,
> > but this would have to be done differently on App Engine (hopefully
> > with a transaction).
>
> No. Backends should require piecemeal futzing around at that level --the
> Model level should be reasonably abstracted from backend specifics. We
> should, instead, fix the problem more generically by moving those tests
> into the django/db/models/sql/ level in some fashion.
> That's one of those "we'll do it when we need it" tasks. Since this is
> the first time anybody's mentioned, we now have a reason to look at it.
help with implementing the actual App Engine backend.
Bye,
Waldemar Kornewald
Malcolm Tredinnick
> On 12 Feb., 03:51, Malcolm Tredinnick <malc...@pointy-stick.com>
> wrote:
> > > Is the plan somewhere on the wiki?
> >
> > No, because it's only something I'm pulling together slowly in my head.
>
> I hope you've read my wiki page, so that in the end Django is flexible
> enough to work even with App Engine's limitations (e.g., it's
> practically impossible to implement a TransactionMiddleware):
> http://code.djangoproject.com/wiki/AppEngine
I've had a quick read of it, but nothing serious, since I have a million
other things to do at the moment that are higher priority. However, the
overall impression from reading that is that most of those problems
aren't something Django itself will need to worry about. It's things
that the Query (and other django/db/models/sql/*) replacement will need
to handle.
Ditto with things like replacing Permissions and ContentTypes -- that
will be up to you to fake somehow in the eventual AppEngine code.
They're part of Django, so if you choose not to use them (and I really
don't understand why you wouldn't, since they're just normal models --
the argument seems to be that you don't want to have an equivalent of
syncdb for some reason, but that doesn't necessarily seem to be a
required constraint), you have to work around that.
> SimpleDB is even worse because of "eventual consistency" which means
> there are no transactions and absolutely no integrity guarantees.
No, it's easier. Since there are no constraints of that nature, it's not
something we have to worry. Designing in features that are impossible
isn't a good goal to have. If you're using a storage backend that
doesn't support some kind of transactional integrity, then you simply
don't have transactional integrity. That's a constraint you accept when
choosing that storage backend.
Regards,
Malcolm
sub...@gmail.com
issues (which, as the DjangoCon presentation proved, there were many)
involving App Engine and Django, then I (and the demographic of
developers I statistically represent) will become excited.
-S
On Feb 6, 9:51 am, David Stenglein <david.stengl...@gmail.com> wrote:
> If so, why the seeming lack of excitement around App Engine? Are
> google engineers not involved in the community?
>
Waldemar Kornewald
wrote:
> > On 12 Feb., 03:51, Malcolm Tredinnick <malc...@pointy-stick.com>
> > wrote:
> > > > Is the plan somewhere on the wiki?
>
> > > No, because it's only something I'm pulling together slowly in my head.
>
> > I hope you've read my wiki page, so that in the end Django is flexible
> > enough to work even with App Engine's limitations (e.g., it's
> > practically impossible to implement a TransactionMiddleware):
> >http://code.djangoproject.com/wiki/AppEngine
>
> I've had a quick read of it, but nothing serious, since I have a million
> other things to do at the moment that are higher priority. However, the
> overall impression from reading that is that most of those problems
> aren't something Django itself will need to worry about. It's things
> that the Query (and other django/db/models/sql/*) replacement will need
> to handle.
on the Django side:
* many-to-many relations need to be handled more flexibly (could be
stored as an attribute on some model)
* batch save/delete (at least very simple/naive implementations) must
be supported (otherwise efficient websites are impossible with App
Engine)
> Ditto with things like replacing Permissions and ContentTypes -- that
> will be up to you to fake somehow in the eventual AppEngine code.
> They're part of Django, so if you choose not to use them (and I really
> don't understand why you wouldn't, since they're just normal models --
> the argument seems to be that you don't want to have an equivalent of
> syncdb for some reason, but that doesn't necessarily seem to be a
> required constraint), you have to work around that.
server, something like syncdb would have to be emulated with a view
that is remotely accessible or using the provider's background
processing facility. Unfortunately, you might hit a request limitation
if you run syncdb in a single request (also applies to background
processing), so syncdb should be spread across multiple requests, but
it's not designed to do that and the implementation wouldn't be easy.
In general, whenever syncdb isn't absolutely necessary it shouldn't be
used.
Regarding faked Permission/ContentType models: We could probably
create all entities within a single request (up to 500 individual
write operations in a single request should be doable). My point,
though, is that faked models would be more efficient and much simpler
(e.g., auth backend's get_group_permissions() could be reduced from 3
queries plus one client-side JOIN to 1 query without JOINs on App
Engine). Also, installations/upgrades are easier if you don't need
syncdb and everything magically just works. This change might not be
an option for SQL, but couldn't Django at least provide a more
efficient alternative for cloud backends?
> > SimpleDB is even worse because of "eventual consistency" which means
> > there are no transactions and absolutely no integrity guarantees.
>
> No, it's easier. Since there are no constraints of that nature, it's not
> something we have to worry. Designing in features that are impossible
> isn't a good goal to have. If you're using a storage backend that
> doesn't support some kind of transactional integrity, then you simply
> don't have transactional integrity. That's a constraint you accept when
> choosing that storage backend.
that some backends might have very restrictive limitations and that it
might be difficult to handle that in a general way. In the end, the
API might look similar, but depending on the backend the semantics
could be ridiculously different. That's still better than no common
API, though.
Bye,
Waldemar Kornewald
Waldemar Kornewald
I'd really like to know if that is an option because if it's not I
should better make app-engine-patch 1.0 compatible with the less
efficient implementation. Hmm, or I could emulate those db tables in
the backend, but that wouldn't be very nice.
Bye,
Waldemar Kornewald
Malcolm Tredinnick
> No comments on the more efficient perms/contenttypes implementation?
If you were hoping for something from me, I don't have any time to think
about it. I'm not in the business of predicting the future in a general
sense, beyond things I've thought hard about and are working on
directly.
Malcolm