Message from discussion proposal for lazy foreignkeys
Received: by 10.216.90.138 with SMTP id e10mr619289wef.4.1285481465094;
Sat, 25 Sep 2010 23:11:05 -0700 (PDT)
Received: by 10.216.242.202 with SMTP id i52ls423380wer.0.p; Sat, 25 Sep 2010
23:10:54 -0700 (PDT)
Received: by 10.216.37.211 with SMTP id y61mr267116wea.14.1285481454142;
Sat, 25 Sep 2010 23:10:54 -0700 (PDT)
Received: by 10.216.37.211 with SMTP id y61mr267115wea.14.1285481453657;
Sat, 25 Sep 2010 23:10:53 -0700 (PDT)
Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [184.108.40.206])
by gmr-mx.google.com with ESMTP id r4si1381536wec.14.2010.09.25.23.10.52;
Sat, 25 Sep 2010 23:10:52 -0700 (PDT)
Received-SPF: pass (google.com: domain of freakboy3...@gmail.com designates 220.127.116.11 as permitted sender) client-ip=18.104.22.168;
Authentication-Results: gmr-mx.google.com; spf=pass (google.com: domain of freakboy3...@gmail.com designates 22.214.171.124 as permitted sender) smtp.mail=freakboy3...@gmail.com; dkim=pass (test mode) header...@gmail.com
Received: by wyb33 with SMTP id 33so5122995wyb.13
for <email@example.com>; Sat, 25 Sep 2010 23:10:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
DomainKey-Signature: a=rsa-sha1; c=nofws;
Received: by 10.227.146.213 with SMTP id i21mr4797509wbv.99.1285481447617;
Sat, 25 Sep 2010 23:10:47 -0700 (PDT)
Received: by 10.227.129.207 with HTTP; Sat, 25 Sep 2010 23:10:47 -0700 (PDT)
Date: Sun, 26 Sep 2010 14:10:47 +0800
Subject: Re: proposal for lazy foreignkeys
From: Russell Keith-Magee <russ...@keith-magee.com>
Content-Type: text/plain; charset=ISO-8859-1
On Sun, Sep 26, 2010 at 1:47 AM, Carl Meyer <carl.j.me...@gmail.com> wrote:
> Hi all,
> I've seen some level of interest in the idea of a lazy foreign key
> (one whose target table is determined by project configuration in some
> way, not hardcoded by the app/model in which it lives). The idea was
> most recently brought up again in Eric Florenzano's keynote at
> DjangoCon. I have some ideas regarding possible API for this, and
> would be glad for feedback.
> First, a couple motivating use cases:
> 1. Reusable apps overuse GenericForeignKey. GFKs are inefficient and
> smell bad. They're good to have around when you really need to link to
> any one of a possibly-growing set of models. But currently reusable
> apps often use them anytime they want to link to "some domain model
> but we don't know which one" - even if in practice in most cases it
> will be only one! A lazy foreign key would be a better solution.
> 2. Standardization with flexibility: i.e. possible-future replacement
> of contrib.auth.User. To be clear, I am not at this point proposing
> any changes at all to contrib.auth. But in some future possible
> contrib.auth refactoring, a lazy foreign key could provide a way for
> reusable apps to point to a common User model, without Django having
> to provide a concrete implementation of that model.
> The concept:
> We introduce the "virtual" model, which is an abstract model with the
> following additional characteristics:
> - It can be the target of a ForeignKey.
> - It may only have one direct concrete subclass, and if it is the
> target of any ForeignKey it must have exactly one.
> At runtime, any ForeignKeys pointing to a virtual model are resolved
> to actually point to the concrete subclass of that virtual model.
> Like an abstract model, a virtual model may include fields, methods,
> etc. These can be considered the specification of an interface: any
> model with a ForeignKey to this virtual model can expect the concrete
> model to satisfy that interface. This is particularly helpful for a
> contrib.auth-type use case: reusable apps don't only need a User model
> to point FKs at, they also often need at least some minimal set of
> fields/properties they can rely on being present.
> It's not required for the virtual model to have any fields, of course:
> in some use cases (voting, tagging) the reusable app doesn't need to
> know anything at all about its target model. A hypothetical voting app
> could simply provide an empty "VotableObject" virtual model, which
> would be inherited by the domain model which can receive votes. Since
> Django already supports multiple inheritance for abstract models, this
> is a minimal and non-restrictive requirement for the domain model
> Advantages of this proposal:
> 1. No new settings.
> 2. In terms of new code API, almost nothing: a new "virtual = True"
> Meta keyword.
> 3. Very little conceptual overhead; reuses existing constructs as much
> as possible.
> I plan to put together a patch to show working code for this, but I'd
> be glad for any feedback at this point, especially if there are
> obvious conceptual problems I'm overlooking. Thanks!
On first inspection, absent of an implementation, I think this is an
My biggest technical concern is the same as Alex's -- that it doesn't
address the 'FK to multiple models' problem. While I agree with your
'no silver bullet' response to Alex, I also don't want to end up with
two (or more) completely different ways of solving the same problem.
At the very least, I'd like to have some certainty that the solution
for single concrete class problem will be conceptually similar to the
multiple concrete class problem.
I also have two technical concerns about how the virtual keyword will
work in practice.
Firstly, take the likely migration path for contrib.auth. We would
introduce an AbstractUser that encompasses the 'basic' concept of a
user, and is marked as a virtual. But we still need to ship a concrete
User that provides the current implementation. At which point, we now
have our single allowed concrete instantiation, and users can't define
their own User class.
I can see this being a common pattern; if you want your app to work
out of the box, you will provide a bare-bones concrete implementation,
which will then block anyone else from providing a concrete
implementation. This necessitates introducing either the ability to
hide a model, or provide a different way of registering models so that
unneeded concrete types aren't instantiated.
Of course, the simple solution here would be to split the concrete
model out into a different app, so that you optionally include
auth.User if you actually want it.
Secondly, this approach requires that content objects that are to be
the target of these relationships must share a virtual base class.
This makes for great pure-OO, but it sucks from the point of view of
For example, consider the case of tagging -- if you want to put a Tag
in a relationship with some content object, then you need to make that
content object inherit from a virtual "TaggableObject" base class.
This means that you need to have control of the base class so that you
can install that mixin. If you're using a reusable app from a third
party to provide your content object (e.g., a Blog model from a
blogging app), you don't have access to the model definition.
The rest of this email is thinking out loud, and probably has a whole
bunch of sharp edges.
It seems to me that what we need isn't a 'virtual' keyword on the
content class, but a virtual representation on the referrer class,
plus a way of registering instances of concrete subclasses.
Here's how it might work:
* Allow a model to have a ForeignKey to an abstract base class; but
in doing so, you make the model with the FK a virtual model.
* Introduce a "VirtualForeignKey" that can point at *any* content
object. Again, having a VirtualForeignKey on a model makes the entire
* In configuration code (I'm a little hazy on exactly where would be
best), provide a way to instantiate virtual models as concrete models.
So, since blog entries can be owned, we would need to register:
MyBlogEntry = make_concrete("MyBlogEntry", model=BlogEntry, owner=MyUser)
So - when we want to allow blogs to be tagged, we might register:
BlogTag = make_concrete("BlogTag", model=Tag, content=MyBlogEntry)
Conceptually, a model could even have multiple virtual extension points:
BlogTag = make_concrete("BlogTag", Tag, content=MyBlogEntry,
As a result of these changes, code like:
wouldn't work out of the box, because Blog is a virtual class.
However, you could use the app cache:
MyBlogEntry = get_model('blog','MyBlogEntry')
We could also include a shortcut on model classes to find their
In practice, this would mean that reusable apps that had virtualized
components will need to be careful to ensure that the concrete model
is appropriately realized, and that the concrete model is then passed
around the app as required. For example, Django's admin (in a
virtualized User world) would need to make all the internal models
with FKs to user were rendered concrete.
This also fits in nicely with the app-cache refactor, because the
'app' object provides a convenient place to specify models and
perform the concreting process.
This would also be backwards compatible, because FKs to abstract base
classes are forbidden at the moment. This new behaviour would only be
required for models that introduced virtualizing foreign keys.
Again - this isn't completely thought through, and I have written even
less code than you have :-), but I'm putting it out on the stoop to
see if the cat licks any of it up.
Russ Magee %-)