Odd problem: some database updates do not appear on other pages until server restart

[bobhaugen]

I'm running django 1.8.14, I have an odd problem, that I have reproduced both locally using runserver and sqlite, and also using Apache with Postgres.

I create a new model instance on one page, and go to another page where that same object should appear in a form selection list. It does not.

After I restart the server (both with runserver and Apache) it appears.

I also see other places where the same odd behavior with template variables showing  foreign key relationships, like {{ object.foreign_key_field }}.

This is not on all pages, just some. It happens in both of the situations I mentioned above (form selection choices and template variables for foreign key fields).

I grepped for cache. None in my code, a lot in site-packages, but I have no idea where to look and what to look for.

I would gratefully follow any clues.

Thanks.

[Tim Graham]

I'd guess you're doing a query for the form field's choices at the module level which will be executed once and cached for as long as the server runs. See if https://docs.djangoproject.com/en/stable/ref/forms/fields/#fields-which-handle-relationships helps, otherwise please post the code for the form in question.

[bobhaugen]

On Thursday, August 18, 2016 at 1:34:29 PM UTC-5, Tim Graham wrote:

I'd guess you're doing a query for the form field's choices at the module level which will be executed once and cached for as long as the server runs. See if https://docs.djangoproject.com/en/stable/ref/forms/fields/#fields-which-handle-relationships helps, otherwise please post the code for the form in question.

Ooooohhh, Tim! You might just have nailed it! Yes I am.

Here's the relevant code for the form field:

 

```

from_agent = forms.ModelChoiceField(

        required=False,

        queryset=EconomicAgent.objects.with_user(),

```

with_user() is a method of

class AgentManager(models.Manager)

 executed once and cached for as long as the server runs.

Is that behavior documented anywhere? Regardless, got any ideas how to avoid it?

But thank you very much for the likely suspect. 

[bobhaugen]

Looks like it works if I "specify queryset=None when declaring the form field and then populate the queryset in the form’s__init__() method:"

Does that make sense to you?

[bobhaugen]

Also, how pervasive is this behavior? Does it affect all querysets generated by model methods? I do that all over the place. This could be bug heaven!

[Sergiy Khohlov]

Hello,
This is trivial mistake. Use form.__init__ if you would like to change it dynamically

18 серп. 2016 22:14 "bobhaugen" <bob.h...@gmail.com> пише:

Also, how pervasive is this behavior? Does it affect all querysets generated by model methods? I do that all over the place. This could be bug heaven!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/16924d52-c9e7-4666-b80d-7baaa49e59e7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bobhaugen]

Yes, that's what I did. It worked for the form field.

But, still, how pervasive is this behavior? (That was the question in the message you answered).

On Thursday, August 18, 2016 at 2:18:00 PM UTC-5, Sergiy Khohlov wrote:

Hello,
This is trivial mistake. Use form.__init__ if you would like to change it dynamically

18 серп. 2016 22:14 "bobhaugen" <bob.h...@gmail.com> пише:

Also, how pervasive is this behavior? Does it affect all querysets generated by model methods? I do that all over the place. This could be bug heaven!

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Michal Petrucha]

On Thu, Aug 18, 2016 at 11:57:40AM -0700, bobhaugen wrote:
> On Thursday, August 18, 2016 at 1:34:29 PM UTC-5, Tim Graham wrote:
> >
> > I'd guess you're doing a query for the form field's choices at the module
> > level which will be executed once and cached for as long as the server
> > runs. See if
> > https://docs.djangoproject.com/en/stable/ref/forms/fields/#fields-which-handle-relationships
> > helps, otherwise please post the code for the form in question.
> >
> > Ooooohhh, Tim! You might just have nailed it! Yes I am.
>
> Here's the relevant code for the form field:
>
> ```
> from_agent = forms.ModelChoiceField(
> required=False,
> queryset=EconomicAgent.objects.with_user(),
> ```
>
> with_user() is a method of
> class AgentManager(models.Manager)

Could you show us the code of with_user? Maybe it does not return an
unevaluated queryset?

Cheers,

Michal

[bobhaugen]

On Friday, August 19, 2016 at 5:20:45 AM UTC-5, Michal Petrucha wrote:

Could you show us the code of with_user? Maybe it does not return an
unevaluated queryset?

 

    def with_user(self):

        all_agents = EconomicAgent.objects.all()

        ua_ids = []

        for agent in all_agents:

            if agent.users.all():

                ua_ids.append(agent.id)

        return EconomicAgent.objects.filter(id__in=ua_ids)

Moving the call to with_user to form.__init__ solved the problem in the form ModelChoiceField.

These questions remain unanswered, although I intend to do a bunch more testing:

1. How pervasive is this problem? Does it affect template variables like {{ object.foreign_key_method }} where the foreign_key_method returns a queryset?
2. Is this behavior clearly documented anywhere?

[ludovic coues]

You were calling the method in the class definition. The class is
"defined" when the module is imported. That's why things where
"cached". Module is imported only once.
The init method on the other hand is called every time an instance of
the class is created. I believe that method will be called in template
everytime the template is rendered.

I don't think this is clearly documented, but that's how python work.
there is nothing magic happening.

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-users/e21e3ec0-dfba-4d0f-a4f0-828a552ee0af%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--

Cordialement, Coues Ludovic
+336 148 743 42

[Michal Petrucha]

On Fri, Aug 19, 2016 at 05:02:39AM -0700, bobhaugen wrote:
> On Friday, August 19, 2016 at 5:20:45 AM UTC-5, Michal Petrucha wrote:
> >
> > Could you show us the code of with_user? Maybe it does not return an
> > unevaluated queryset?
> >
> >
> def with_user(self):
> all_agents = EconomicAgent.objects.all()
> ua_ids = []
> for agent in all_agents:
> if agent.users.all():
> ua_ids.append(agent.id)
> return EconomicAgent.objects.filter(id__in=ua_ids)
>
> Moving the call to with_user to form.__init__ solved the problem in the
> form ModelChoiceField.

Thanks for sharing the code. The problem is that the ``with_user``
method, which was called during import, eagerly evaluates
``EconomicAgent.objects.all()`` right away, and returns a QuerySet
filtered by the result of that. So even though the final queryset
returned by ``with_user`` is not yet evaluated, the filtering
condition in that queryset is already fixed.

You could fix this by doing something along the lines of::

def with_user(self):
return EconomicAgent.objects.filter(users__isnull=False).distinct()

The important difference is that this version of ``with_user`` does
not evaluate anything eagerly, but rather sets up the necessary
filters to get the correct result set at the time of evaluation of
that queryset.

Depending on what you do with the queryset further, you might need to
change it a little bit to remove the ``distinct()`` call, and use a
subquery instead.

> These questions remain unanswered, although I intend to do a bunch more
> testing:
>

> 1. How pervasive is this problem? Does it affect template variables like

> {{ object.foreign_key_method }} where the foreign_key_method returns a
> queryset?

> 2. Is this behavior clearly documented anywhere?

Honestly, I'm not sure what exactly you're asking here. Your
implementation of ``with_user`` was hard-wiring a queryset filter
based on the state of the database at the time ``with_user`` was
called. The rest is just standard Python behavior – since the method
was called during import (in the ModelForm definition), the result of
that call was used for the rest of the process' lifetime.

Cheers,

Michal

[bobhaugen]

On Friday, August 19, 2016 at 7:38:45 AM UTC-5, Michal Petrucha wrote:

Honestly, I'm not sure what exactly you're asking here. Your
implementation of ``with_user`` was hard-wiring a queryset filter
based on the state of the database at the time ``with_user`` was
called. The rest is just standard Python behavior – since the method
was called during import (in the ModelForm definition), the result of
that call was used for the rest of the process' lifetime.

 

Ok, so from your response and that of Coues Ludovic upthread, I clearly need to study Python more deeply. Will do.

Thanks for all the responses.

[Michal Petrucha]

Actually, let me rephrase that. This problem you encountered here
wasn't a case of “you need to set the ``queryset`` argument of
``ModelChoiceField`` in your form's ``__init__``,” but rather a case
of “don't do too much filtering in your Python code – let your
database do the filtering”.

Using ``__init__`` to set the queryset is not really a very standard
thing to do. It's just a workaround to make a form work if it uses a
manager or queryset method that does not behave entirely correctly.

Does this help make things at least a little bit clearer?

Michal

[bobhaugen]

Michal, thanks for following up.

I'll try your suggested code from your other response upthread and see if it solves the problem as well as the form.__init__.

And do more digging about how Python does things. I am a self-taught programmer who focuses on community economic development and I usually just learn as much about software as I need to do what I want to do. Sometimes that is not enough learning...;-)

[Fred Stluka]

Bob,

I think this is very pervasive.  That is, it happened a few times
to my team until we educated all developers to never use the
choices, limit_choices_to, or queryset parameters of a Model
field or Form field.  Instead, we always explicitly set them in
__init__() instead.

I assume other teams also make this mistake often.

For us, this mistake caused 3 separate type of problems:

1. The problem you described:
    Old data from the DB was "cached" until we restarted the
    Django server, because the DB query was done only
    when the Model or Form class was defined, not when each
    Model or Form instance was created.  So, in answer to
    your question above, yes, it would also affect template
    variables that refer to the data that was loaded from the
    DB when the class was defined.

2. Occasional import loops, which prevented the Django
    server from starting the app, because the value of the
    choices, limit_choices_to, or queryset parameter caused
    another model to be loaded.  This would trip us up badly
    because the import loop would often occur in our TEST or
    PROD environments (Django loaded via Apache WSGI) but
    not in our DEV environments (Django DEV server), since
    the order of class imports could be different in those 2
    different configurations.  See:
    - https://groups.google.com/forum/#!topic/django-users/ONGCSO37UWY

3. There's a warning here about this mistake being able to
    pollute your TEST execution with PROD data:
    - https://docs.djangoproject.com/en/dev/topics/testing/overview/#the-test-database

>     Is this behavior clearly documented anywhere?

It didn't jump out at us.  We got burned by and it and put in
some time to diagnose it.  Then we found the above link that
warns of one of the problems.

--Fred

---

Fred Stluka -- mailto:fr...@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.

---