Dynamic model, reverse foreign key not working

[Roman Akopov]

Hello,

My problem is very rare so I'll try ad add as much useful detail as possible.

I am using python 3.5 and 3.6, Django 1.11.4

I am creating complex security related application and need to generate additional models based on other applications' models. In general, everything works fine, models are created, migrated without a problem. But there is single problem I totally failed to solve.

Let's say there is preexisting model Alpha of some other application and I dynamically create model Beta which references Alpha with ForeignKey. The problem is that Beta does not get reverse relation, so I can query for beta.alpha, but not for alpha.betas. I receive "django.core.exceptions.FieldError: Cannot resolve keyword 'betas' into field. Choices are: x, y, z"

Here is my code

    def _create_dynamic_model(self, model_label, fields, attributes=None, options=None):
        from django.db import models

        class Meta:
            pass

        setattr(Meta, 'app_label', '_talos')

        if options is not None:
            for key, value in options.items():
                setattr(Meta, key, value)

        attrs = {'__module__': '_talos', '_talos_dynamic': True, 'Meta': Meta}

        if attributes:
            attrs.update(attributes)

        if fields:
            attrs.update(fields)

        model = type(model_label, (models.Model,), attrs)

        return model

I call it like this (where self.model is referenced model class)

    def _create_object_permission_model(self):
        from django.db import models

        return self._create_dynamic_model(
            'o_{0}_{1}'.format(self.model._meta.app_config.label, self.model.__name__),
            fields={
                'role': models.ForeignKey(
                    'talos.Role',
                    related_name='+',
                    on_delete=models.CASCADE),
                'permission': models.ForeignKey(
                    'talos.ObjectPermission',
                    related_name='+',
                    on_delete=models.CASCADE),
                'target': models.ForeignKey(
                    self.model,
                    related_name='talos_permissions', # self.model does not receive reverse relation!
                    on_delete=models.CASCADE)
            },
            options={
                'unique_together': [('role', 'permission', 'target')],
                'index_together': [
                    ('target', 'permission', 'role'),
                    ('target', 'role', 'permission'),
                    ('role', 'target', 'permission')]
            })


I tried calling contribute_to_class and contribute_to_related_class methods manually, tried to call this code at different moments (My application ready method, class_prepared signal handler), with absolutely no luck.

I tried to read Django sources, and got that contribute_to_related_class is called from contribute_to_class with delay, but it did not help me realize what exactly I am doing wrong.

I am trying to use reverse relation from custom view and admin, so everything should be pretty much initialized already.

Roman

[Michal Petrucha]

On Tue, Sep 05, 2017 at 03:47:37AM -0700, Roman Akopov wrote:
> Hello,
>
> My problem is very rare so I'll try ad add as much useful detail as
> possible.
>
> I am using python 3.5 and 3.6, Django 1.11.4
>
> I am creating complex security related application and need to generate
> additional models based on other applications' models. In general,
> everything works fine, models are created, migrated without a problem. But
> there is single problem I totally failed to solve.
>
> Let's say there is preexisting model Alpha of some other application and I
> dynamically create model Beta which references Alpha with ForeignKey. The
> problem is that Beta does not get reverse relation, so I can query for
> beta.alpha, but not for alpha.betas. I receive
> "django.core.exceptions.FieldError: Cannot resolve keyword 'betas' into
> field. Choices are: x, y, z"

So the deal is that each model's _meta caches a bunch of structures
storing the list of fields, reverse relations, and so on, the first
time you access any of them. If you add a new field (or a new reverse
relation) after those caches have already been filled, the new field
or relation won't be reflected in them, leading to errors just like
yours.

There's an internal undocumented API that takes care of this,
Model._meta._expire_cache(), which will clear all those caches. It
should do the trick for you, but as always with private APIs, be aware
that it might break or change in the future.

Cheers,

Michal

[Roman Akopov]

Michael,

Thanks for great hint!

Unfortulately, it did not help. I have added "model._meta._expire_cache()" call almost everywhere, before generating dynamic model, after, between steps, it did not help a bit, error is exactly the same.

Also, I have additionally tested my application against django 1.10 and django 1.9 and got exactly the same result.

Roman

[Michal Petrucha]

On Tue, Sep 05, 2017 at 04:56:10AM -0700, Roman Akopov wrote:
> Unfortulately, it did not help. I have added "model._meta._expire_cache()"
> call almost everywhere, before generating dynamic model, after, between
> steps, it did not help a bit, error is exactly the same.
> Also, I have additionally tested my application against django 1.10 and
> django 1.9 and got exactly the same result.

On which models did you call that? You should call it on the target
model of any relationship that you create dynamically. So if you have
existing models Target1, and Target2, and create a new model Dynamic
with a ForeignKey(Target1) and ManyToManyField(Target2), you'd need to
call _expire_cache() on Target1 and Target2 right after creating the
dynamic model, but before trying to make any queries using those new
reverse relations.

If this doesn't help, then you might have to investigate if there's
perhaps some cached attribute that doesn't get cleared.

Good luck,

Michal

[Roman Akopov]

I call it on target model, the one with reverse relation missing.

[Roman Akopov]

I have investigated a bit more and looks lite it is Options._relation_tree property, it's value is calculated by _populate_directed_relation_graph only once and it is @cached_property, so I see no valid way to reset value