Streamlining Model validation of ForeignKeys

[Shaheed Haque]

Hi,

I have two models Source and Input, where Input has an FK to Source like this:

class Source(db_models.Model):
    scheme = db_models.CharField(...)
    path = db_models.CharField(...)
    display_as = db_models.CharField(...)

class Input(db_models.Model):
    name = db_models.CharField(...)
    description = db_models.CharField(...)

    type = db_models.CharField(...)
    source = db_models.ForeignKey(Source, on_delete=db_models.CASCADE)
    reuse = db_models.CharField(...)
    value = db_models.CharField(...)

The instances of Source are statically created, and number 20 or so in total. The instances of Input number in the millions, and may be loaded from external files/network endpoints. When Input instances are created, we perform normal Django validation processing, involving form.is_valid()/form.errors. As we know, Django 5.x does something like:

1. Form field validation using Form.clean_xxx().
   
2. Form cross-field validation using Form.clean().
3. Model validation, including validating the FK.

like this:

...

File "/.../django/forms/forms.py", line 197, in is_valid return self.is_bound and not self.errors File "/.../django/forms/forms.py", line 192, in errors self.full_clean() File "/.../django/forms/forms.py", line 329, in full_clean self._post_clean() File "/.../django/forms/models.py", line 496, in _post_clean self.instance.full_clean(exclude=exclude, validate_unique=False) File "/.../django/db/models/base.py", line 1520, in full_clean self.clean_fields(exclude=exclude) File "/.../django/db/models/base.py", line 1572, in clean_fields setattr(self, f.attname, f.clean(raw_value, self)) File "/.../django/db/models/fields/__init__.py", line 830, in clean self.validate(value, model_instance) File "/.../django/db/models/fields/related.py", line 1093, in validate if not qs.exists():

...

In one experiment, I observe that this results in 143k queries, taking a total of 43s. Is there a way to short circuit this Model-level validation? Since I know the ~20 instances of Source, even a cache of Source.objects.all() would be a very cheap, but I cannot see any way to inject that into the code for line 1093:

   def validate(self, value, model_instance):
       if self.remote_field.parent_link:
           return
       super().validate(value, model_instance)
       if value is None:
           return

       using = router.db_for_read(self.remote_field.model, instance=model_instance)
       qs = self.remote_field.model._base_manager.using(using).filter(   <<<< queryset created here
           **{self.remote_field.field_name: value}
       )
       qs = qs.complex_filter(self.get_limit_choices_to())
       if not qs.exists():                                               <<<< queryset evaluated here, line 1093
           raise exceptions.ValidationError(...)

We actually have several versions of this problem so any ideas are welcome.

Thanks, Shaheed

[Ryan Nowakowski]

On Fri, May 10, 2024 at 12:48:09PM +0100, Shaheed Haque wrote:
> In one experiment, I observe that this results in 143k queries, taking a
> total of 43s. Is there a way to short circuit this Model-level validation?
> Since I know the ~20 instances of Source, even a cache of
> Source.objects.all() would be a very cheap, but I cannot see any way to
> inject that into the code

I you've already determined that repeated queries are the bottleneck, you
might look into a generic query cache like https://django-cachalot.readthedocs.io

That way you don't have to muck around with customizing validation.

[Shaheed Haque]

Thanks for the kind tip. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20240511150753.GF28137%40fattuba.com.