State of concrete fields

[Daniel Pyrathon]

Hi All,

The current Options implementation has properties for "concrete fields".

Technically speaking, concrete fields are data fields without a column. The only concrete field in the codebase is ForeignObject. There are 2 classes that inherit from ForeignObject: GenericRelation and ForeignKey, but each of them fall into different categories (virtual and data). The only instances of a ForeignObject in the codebase are in the unit tests, and those occurrences can be replaced with a ForeignKey (not tested).

So, my question is:

- Do we really need concrete fields, if ForeignObject is internal and not used in any place other than unit tests?

- Are there third-party packages that inherit from ForeignObject? if yes, can anyone point which ones?

Perhaps akaariai and loic84 can help, as I know they have done work in this area.

As we are formalising the Options API (https://groups.google.com/forum/#!topic/django-developers/XfMUjK59Sls) it would be great to reduce complexity even more by removing the entire term "concrete fields".

Thanks,

Daniel Pyrathon

[Anssi Kääriäinen]

On Mon, 2014-08-04 at 13:02 -0700, Daniel Pyrathon wrote:
> Hi All,
>
>
> The current Options implementation has properties for "concrete
> fields".
> Technically speaking, concrete fields are data fields without a
> column.

Concrete fields are data fields *with* a column, not the other way
around.

> The only concrete field in the codebase is ForeignObject. There are 2
> classes that inherit from ForeignObject: GenericRelation and
> ForeignKey, but each of them fall into different categories (virtual
> and data). The only instances of a ForeignObject in the codebase are
> in the unit tests, and those occurrences can be replaced with a
> ForeignKey (not tested).

No, they can't be replaced with ForeignKeys. The ForeignObject tests do
various things that normal ForeignKeys can't do. For one, the tests
exercise multicolumn joins. But multicolumn joins aren't yet available
through public APIs. Another use case not possible using ForeignKey is
fetching unique reverse related object (this could be for example latest
edit of an object from related list of edits).

> - Do we really need concrete fields, if ForeignObject is internal and
> not used in any place other than unit tests?

Maybe the right approach is to make ForeignObject a relation field or a
virtual field. I don't remember why ForeignObject wasn't implemented as
a virtual field.

> - Are there third-party packages that inherit from ForeignObject? if
> yes, can anyone point which ones?

Perhaps not popular, but
https://github.com/akaariai/django-reverse-unique/ is something that
uses ForeignObject. GitHub search gives only a lot of clones of Django's
repo. There aren't likely that many users of ForeignObject yet.

>
> As we are formalising the Options API
> (https://groups.google.com/forum/#!
> topic/django-developers/XfMUjK59Sls) it would be great to reduce
> complexity even more by removing the entire term "concrete fields".

I'd like to keep the ability to use something like ForeignObject around.
That is, there should be some way to implement the use cases mentioned
above. If this requires changes to user projects that is OK for me, but
if it is impossible to achieve the use cases mentioned above, then lets
not remove concrete fields.

In general, if at all possible we should avoid removing non-public
features without offering some upgrade path to users.

- Anssi

[Loic Bistuer]

Hi Daniel,

I'm a little confused because if I understand correctly you introduced the concept of "data" field that you describe as "any field that has an entry on the database", but how is that different from the earlier concept of "concrete" fields.

For now FO, FK and O2O all fall under the "concrete" fields category but my take is that eventually all relations should be "virtual", some of them relying on one or more concrete fields (i.e. composite). Currently virtual fields are not part of the public API, and IIRC they are not allowed to have form fields. Eventually we'll need the concept of "data" fields that span a subset of both concrete and virtual fields of a given model for the purpose of acquiring and presenting data (i.e. forms or admin). For instance:

class Model:
name = model.CharField() # concrete & data
fk_col1 = models.IntegerField() # concrete only
fk_col2 = models.IntegerField() # concrete only
fk = models.ForeignKey(col1, col2) # virtual & data

--
Loic

[Anssi Kääriäinen]

On Tue, 2014-08-05 at 16:26 +0200, Loic Bistuer wrote:
> Hi Daniel,
>
> I'm a little confused because if I understand correctly you introduced the concept of "data" field that you describe as "any field that has an entry on the database", but how is that different from the earlier concept of "concrete" fields.
>
> For now FO, FK and O2O all fall under the "concrete" fields category but my take is that eventually all relations should be "virtual", some of them relying on one or more concrete fields (i.e. composite). Currently virtual fields are not part of the public API, and IIRC they are not allowed to have form fields. Eventually we'll need the concept of "data" fields that span a subset of both concrete and virtual fields of a given model for the purpose of acquiring and presenting data (i.e. forms or admin). For instance:
>
> class Model:
> name = model.CharField() # concrete & data
> fk_col1 = models.IntegerField() # concrete only
> fk_col2 = models.IntegerField() # concrete only
> fk = models.ForeignKey(col1, col2) # virtual & data

This categorization seems very good.

GenerigForeignKey behaves a bit differently from the other virtual field
we have in core (that is, ForeignObject). GenerigForeignKeys can't be
used in ORM lookups in select_related(), values() or in filters. This is
one corner case which isn't covered by the above distinction. Adding
support for filtering is a TODO, but select_related and values do not
make sense (as we would also need to do select_related in that case). I
guess we just have to exclude some/all virtual fields by their
properties in ORM methods, where different ORM methods allow different
subsets of virtual fields to be used.

In addition, we have an interesting case in model initialization where
ForeignObjects can be used in model initialization when used as a
keyword argument, but not when used as positional argument. This makes
sense when considering the use cases - positional arguments are (mostly)
used as a fast path when initializing models from database. And, when
initializing the model from database, there are only concrete fields
present.

GFKs can't be used in kwargs model initialization, but ForeignObjects
can be used (currently ForeignObject is in meta.fields, but not in
concrete_fields, whereas GFKs are in virtual_fields but not in
fields...). I don't see a reason why we couldn't add support for GFKs in
model init kwargs, too. This way we could move ForeignObject to the new
API's virtual fields category, and make virtual_fields a subset of all
fields.

In summary I think we could try categorization to concrete, virtual and
data fields. Concrete fields are those that have backing database
columns, virtual fields do not have database columns, and data fields
are fields that present data to users. Unfortunately it is hard to say
if this categorization is sufficient without trying.

- Anssi