Proposal: relationships based on arbitrary predicates

[Alexander Hill]

Hi all,

I've run into many situations during my time using Django where I've wanted to be able to express relations based on some other criteria than foreign key equality. A few examples:

- descendants or children of a node in a tree structure

- saved search terms to search results

- a model containing a date range to timestamped items falling within that date range

Currently to do this kind of thing, you might write a getter which returns a queryset - think for example mptt's get_descendants(). But you don't get any of the nice things a real relation field gives you - you can't use that relationship in filters, you can't select/prefetch_related() or values(), there's no reverse relationship, etc.

I've written a Relationship field[0] that lets you define relations in terms of arbitrary Q filters containing objects of a new type, L. An L is like an F, but represents a field on the "local" or "left" side of the relation, where the Q is filtering against the remote "to" side of the relation. For example, in a materialised path tree, this is how you might express descendants:

class Node(models.Model):

    path = models.TextField()

    descendants = Relationship(

        "self",

        Q(path__startswith=L('path'), pk__ne=L('pk')),

        multiple=True,

        reverse_multiple=True,

        related_name='ascendants',

    )

Now you can use the descendants field like any other many-to-many field in all the places I mentioned above, but the relationship is based purely on prefix-matching on the path field. You also get an ascendants field on Node, which represents the path back to the root and can be used in the same way.

I think this could make a nice new feature for Django. It would give a usability boost to anyone using MPTT or treebeard, for example. It works OK as a third-party library, but the current implementation relies heavily on undocumented ORM internals, and there are a few features I'd like to implement that are impractical without making some ORM changes.

Thoughts/feedback/questions welcome!

Thanks,

Alex

[0] https://github.com/alexhill/django-relativity

[charettes]

Hello Alex!

Thanks for your work on this project, this is definitely something that I believe would be useful in Django's core based on the number of times I implemented a filtered queryset getter on Models.

I'm not totally sold on the API but having an analog of what ForeignObject is to ForeignKey for ManyToManyField would definitely be useful.

From what I can see in relativity.fields[0] most of the additional logic revolves around the extra filtering capabilites through Restriction.

Do you have an idea of what the fields.related inheritance chain would look like if it was part of core? I feel like having Relation(RelatedField), ForeignObject(Relation), ManyToManyField(Relation) and adding the filtering logic to Relation could work but I'd be interested to hear what you think here. FWIW Anssi implemented something similar[1] for reverse unique relationship before FilteredRelation() was introduced.

In a sense Relation would be form of virtual field like ForeignObject since it's not in charge of any database field handling.

Simon

[0] https://github.com/AlexHill/django-relativity/blob/3802608c64e86c62ab6268efc37a3f5ca8539221/relativity/fields.py

[1] https://github.com/akaariai/django-reverse-unique

[Alexander Hill]

Hi Simon,

Thanks for looking at this and for providing some context - I had looked at FilteredRelation but I hadn't seen reverse-unique. It makes me more confident that this is a good direction to take. I've reimplemented ReverseUnique using Relationship [0] and the tests pass, with the only code carried over that for discovery of the FK link.

> I'm not totally sold on the API but having an analog of what ForeignObject is to ForeignKey for ManyToManyField would definitely be useful.

I'm not tied to the API, but I think passing a Q as a predicate makes sense especially given that it's what both FilteredRelation and ReverseUnique do. The core of the idea is that we can express a relationship as a combination of predicate and arity. In practise I don't think this would be used all that much by users directly - more by third-party apps like mptt, and perhaps Django internally.

> From what I can see in relativity.fields[0] most of the additional logic revolves around the extra filtering capabilites through Restriction.

Yeah that's what it boils down to. We return no columns to join against, and return a compilable Restriction from get_extra_restriction to provide all the ON conditions. The rest of it is making the descriptors, rels, prefetch, etc work.

> Do you have an idea of what the fields.related inheritance chain would look like if it was part of core?

The least intrusive, and probably a good starting point, would be to introduce Relationship alongside the other relation fields as a standalone feature, modifying the ORM to allow the implementation to be less hacky. It would remain a subclass of ForeignObject (or perhaps RelatedField - I'll give that a try).

In the future there's potential for a nice refactor of the ORM to generalise join conditions from key-equality to arbitrary predicates of which key equality is just one case, at which point Relationship could sit comfortably as a base class of all the other relations. The assumption that join==key-equality is pervasive and I think that's an unnecessarily large chunk of work to take on at this point - it would be better to get the feature in, then have a release cycle or so to think about the best way to approach that problem and if we even want to.

I would be happy to write up a DEP expanding on an implementation plan and potential future work.

Thanks,

Alex

[0] https://github.com/AlexHill/django-reverse-unique/blob/624c8b19406a40b8e1a2c969c23a6b45bed5a896/reverse_unique/fields.py

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c87af098-baa4-4d45-9a4b-757166b41734%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Silvio]

Alex,

This is a very useful pattern, that like many others, I've also implemented in an ad-hoc fashion using a ton of undocumented internal APIs. So I fully agree standardizing it would be great. Something similar is:

https://groups.google.com/forum/#!topic/django-developers/ADSuUUuZp3Q

Essentially, I've ended up with the need for:

ComputedField

and

ComputedRelationship

where both have all of the niceties that regular fields and foreign relationships have.

So I'd love to see this in Django.

-

Silvio

[Alexander Hill]

Hi Silvio,

Thanks for your feedback. David Sanders brought up something similar to ComputedField on GitHub[0] - a ForwardedField which can traverse relationships and present a field from a remote instance as if it were a local field. As long as it can traverse relationships, that use case would be covered by ComputedField, so that's another tick for that idea.

I think these two features are quite distinct - except that they may both require a similar change to the ORM to allow traversing relations in the referred-to fields. One limitation of relativity now is that you can only form conditions with local fields of your models. I'd like to be able to traverse relationships in those conditions.

Anybody who's interested - please try out relativity and see if it works for you. The mptt and treebeard helpers are a good place to start :)

Simon - any further thoughts on this before I start working up a patch?

Thanks,

Alex

[0] https://github.com/AlexHill/django-relativity/pull/1#issuecomment-418239406

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6619c2f7-e698-4906-b11e-68c9bcf0a23f%40googlegroups.com.

[charettes]

Hello Alex,

While investigating an issue with reverse known related objects assignment[0] I stumbled

upon a little something that might interest you.

It looks like there's currently some tested cases of abusing the ForeignObject interface

to achieve something similar to what you are suggesting here[1][2].

The contenttypes.GenericRelation object happens to implement a similar interface

while it's actually a many-to-many field. I assume it's not setting many_to_many=True

because a ton of internal Django checks duck-type this attribute to figure whether or

not "through" exists but it should really be marked this way.

I'd say giving a shot at reimplementing GenericRelation on top of your suggested

Relationship object would be a good way to test whether or not it's flexible enough

and be a strong argument for inclusion in Django core to get rid of the current hacks

GenericRelation currently employs.

Simon

[0] https://code.djangoproject.com/ticket/29908

[1] https://github.com/django/django/blob/ecac6d7a2a510cb0a5d772deeca633c99c9687e5/tests/foreign_object/models/empty_join.py#L22-L84

[2] https://github.com/django/django/blob/ecac6d7a2a510cb0a5d772deeca633c99c9687e5/tests/foreign_object/models/empty_join.py#L98-L103