Feature Request: Enabling hooking into the autodetector from a third-party app.

[Emil Madsen]

Hello follow Djangonauts.

I'm writing this post in an attempt to ignite a discussion about the autodetector, and how to make it expandable, such that third-party apps can generate their custom migrations automatically.

This has previously been discussed here:
* https://groups.google.com/d/msg/django-developers/qRNkReCZiCk/I8dIxxhoBwAJ
* https://groups.google.com/d/msg/django-developers/pbj7U7FBL6Q/eJtRV7QwDwAJ


Motivation:

My motivation for writing this post is two fold:

1. First, I recently attended DjangoCon Europe 2019, where Carlton Gibson had a keynote about contributing back to Django, I want to do so, and thus this is my first post on the mailing list.
   
2. Secondly, I believe there are various cases where being able to expand the automigrator enables libraries to be a lot cleaner. For instance:
   

• Enforcing constriants on data-models, such as it's done with unique_together, and CheckConstraints. Currently these are implemented directly in Django, but given an expandable autodector they could have been implemented as libraries. In the long section at the end of this post, I mention a similar case to this, that was my original motivation for writing this post.
• Enabling rebust auditing models via database triggers, by simply adding 'audit_trigger = True' to the Meta class of models that should be audited, as done by: 
• https://github.com/carta/postgres_audit_triggers
• Automatically enabling database extensions, when fields which require these extensions are utilized. As an example, utilizing PostGIS or HStore currently requires activating their corresponding extensions manually using CreateExtension('extension'), which could be automated.
• Really any case, in which one would like to automatically generate migrations, and which cannot be achieved by custom fields.

The proposal:

Create a plugin system where plugins can register handlers / callbacks to be run from within the automigrator. Potentially this could even be implemented using the signals mechanism currently in Django, as presented in 'Building plugin ecosystems with Django', by Raphael Michel at DjangoCon Europe 2019.

I imagine the signal should be fired here:

• https://github.com/django/django/blob/master/django/db/migrations/autodetector.py#L192

And that signals handlers should act as combination between the 'create_altered_X' and 'generate_altered_X' methods, resulting either to calls of 'self.add_operation', or returning a list of operations to add to the migrationsfile.

There is an open question with regards to how dependencies and shared state should be handled.

- I do not have enough insight to answer this.

I am very open to other solutions, this is merely what I came up with, without having any throughout insight into how Django works internally.

- I am willing to implement the plugin system, and create a PR for this, if a consensus is found.

My personal motivation (long section):

I have a database design, which includes several data invariants, which I want to uphold. I've been able to implement most of these data invariants using uniqueness (on fields, unique_together, UniqueConstraint), and CheckConstraint.

CheckConstraint is excellent, and I'm very happy that made it to Django with 2.2, but for enforcing data invariants it has one major shortcoming. - It only operates on a single table.

Currently to implement data invariants across multiple tables, I believe the suggestion is to override save methods, or to use save signals.
- However these signals are not emitted for queryset operations, such as bulk_inserts or update s, and as such cannot be used to maintain data invariants, without relying on manual procedures that are error-prone, and take up review time. On another note, save methods and save signals, also do not help if the databsae is accseed outside of the Django ORM.

Thus if you need reliable constraints that work across multiple tables, I don't believe that a robust solution is currently in place, thus I've been working on a prototype for a library to fulfill this role. The library enables very strong data invariant checking by allowing one to write database constraint triggers via Querysets.

The envisioned library is very simple, and works as follows (based upon the pizza toppings example):

class Topping(models.Model):
    name = models.CharField(max_length=30)

    def __str__(self):
        return self.name

class PizzaTopping(models.Model):
    class Meta:
        unique_together = ("pizza", "topping")

    pizza = models.ForeignKey('Pizza')
    topping = models.ForeignKey('Topping')

class Pizza(models.Model):
    name = models.CharField(max_length=30)
    toppings = models.ManyToManyField('Topping', through=PizzaTopping)

    def __str__(self):
        return self.name

We envision having a data invariant on the number of toppings allowed per pizza, and with the library we can now enforce that, by adding the 'constraint_triggers' option to the the Meta class of PizzaTopping:

class PizzaTopping(models.Model):
    class Meta:
        unique_together = ("pizza", "topping")
        constraint_triggers = [ConstraintTrigger(
            name='At most 5 toppings',
            query=M('PizzaTopping').objects.values(
                'pizza'
            ).annotate(
                num_toppings=Count('topping')
            ).filter(
                num_toppings__gt=5
            ),
            error='Trying to add more than 5 toppings to one pizza.',

        }]
    ...

After which one can simply run:

python manage.py makemigration

Which will generate the migration similar to:

django_constraint_triggers.operations.AddConstraintTrigger(
            model_name='pizzatopping',
            trigger_name='At most 5 toppings',
            query=django_constraint_triggers.utils.M(app_label='...', model='PizzaTopping', operations=[...])
            error='Trying to add more than 5 toppings to one pizza.',
        ),

That can then be applied using:

python manage.py migrate

To generate and install the trigger on the database:

Triggers:
    dct__trig__1688bfdc185c490f AFTER INSERT OR UPDATE ON app_pizzatopping DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE PROCEDURE dct__func__1688bfdc185c490f()

Where the database function is just:

IF EXISTS (
    QUERYSET_SQL

) THEN
    RAISE check_violation USING MESSAGE = '{}';
END IF;
RETURN NULL;

Upon which we are now protected against adding too many toppings to a pizza:

cheese = Topping.objects.create(name="Cheese")
ham = Topping.objects.create(name="Ham")
pepperoni = Topping.objects.create(name="Pepperoni")
mushrooms = Topping.objects.create(name="Mushrooms")
onions = Topping.objects.create(name="Onions")
# TODO: Add CheckConstraint against pineapple
pineapple = Topping.objects.create(name="Pineapple")

pizza = Pizza.objects.create(name="Django Special")
PizzaTopping.objects.create(pizza=pizza, topping=cheese)
PizzaTopping.objects.create(pizza=pizza, topping=ham)
PizzaTopping.objects.create(pizza=pizza, topping=pepperoni)
PizzaTopping.objects.create(pizza=pizza, topping=mushrooms)
PizzaTopping.objects.create(pizza=pizza, topping=onions)
# NOTE: Atleast pineapple was added last, and rejected this way.

with self.assertRaises(IntegrityError):
    PizzaTopping.objects.create(pizza=pizza, topping=pineapple)

The way the library works right now, is by monkey patching the autodetector to pick up the constraint_triggers entry on the meta class, and by expanding the OPTIONS.DEFAULT_NAMES to allow for setting this variable on the Meta class.

- The combination of these are then used to automatically generate entries in migrationfiles with custom operations.

The fact that monkey-patching is used means this library is not interoperable with any other libraries changing the autodetector like this, and that code smells.

The library althrough in it's early stage, and very dirty, can be found here:

• https://github.com/magenta-aps/django_constraint_triggers

[Ian Foote]

Hi Emil,

This is a very interesting idea and I find your use-case compelling. Adding support for custom triggers to Django itself would resolve your main use case this time but I also see value in a general solution that enables Django's third-party ecosystem.

I think the next step is to open a new ticket (or find an existing one).

Ian

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/96703d89-be3e-44fe-9888-9aa5b3b9ceb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carlton Gibson]

Hi Emil. 

Thanks for this. 

Ticket was https://code.djangoproject.com/ticket/30849

Closed as wontfix, since at least constraints are now handled. 

I don't think that covers everything you propose but see again Simon and Josh's comments on the first thread you opened. These seem to me to be really on the right track. 

If you had appetite to work on the migrations to open the auto-decector up a bit, there are quite a few migrations tickets that need somebody's love and attention. 🙂

Kind Regards,

Carlton