Complicated, strange bug in ``prefetch_related`` when used in the context of a test client

[Yo-Yo Ma]

I'm still working on forming some sort of understanding of what exactly causes this and/or what is even going on, so any help is much appreciated.

The resultant attached "bars" in the following example:

    queryset = Foo.objects.all()
    queryset = queryset.prefetch_related('bars__baz')
    obj = queryset.get(pk=1)

are different from the results of:

    obj = Foo.objects.all().prefetch_related('bars__baz').get(pk=1)

And, the results attached in the second example are inconsistent as well.

In order to reproduce this, you need to first query for the "Foo", then add some "bars" to the DB, then perform the above examples.

This only seems to occur, at least measurably, during tests when using ``self.client.get(...``, where the view that is being requested performs the queries.

I'm using ``TransactionTestCase``, sqlite3, and Django latest master, pulled about an hour ago, and I've also tried the new live server test case with the same results.

Note: If I modify django.db.models.query # 590 to the following, the bug doesn't occur (not to suggest that the following is a solution, but it might help to debug):

        prefetch_related_objects([], self._prefetch_related_lookups)

[Anssi Kääriäinen]

On 29 loka, 06:05, Yo-Yo Ma <baxterstock...@gmail.com> wrote:
> I'm still working on forming some sort of understanding of what exactly
> causes this and/or what is even going on, so any help is much appreciated.

Based on this bug report there is no way to help. We don't have the
models, or how the results are different. This part looks especially
suspicious:

> queryset = Foo.objects.all()
> queryset = queryset.prefetch_related('bars__baz')
> obj = queryset.get(pk=1)
>
> are different from the results of:
>
> obj = Foo.objects.all().prefetch_related('bars__baz').get(pk=1)

The queries are identical. Only possibility is that there is something
really, really strange going on with the garbage collector. I don't
recall us defining any __del__ methods which could cause anything
serious, so it seems that if there is actually a bug here, it means
the bug is in Python. Actually, I am not sure of the semantics of
Python's garbage collector, it might be that the two above queries are
_exactly_ identical in Python.

My wild guesses:
- some sort of state leak in testing
- non-deterministic bug in prefetch_related (likely caused by random
ordering of the rows in the resultset)

I hope you can provide us more data about this situation,
- Anssi

[Yo-Yo Ma]

Hi Anss, thanks for the reply.

The queries are identical.

That's the exact reason I posted. Those two examples should essentially be the same, save for some sort of operator overloading.

My note at the bottom about providing an empty list instead of the related cache when calling ``prefetch_related_objects`` implies that something weird is going on with the related cache, and the fact that it only happens when using the test client (at least, thus far) makes me think that it could possibly even be related to some sort of ``threading.local``, or something to that effect.

The models are simple (named according to the example in the original post):

class Foo(models.Model):
    title = models.CharField(max_length=255)


class Baz(models.Model):
    name = models.CharField(max_length=255)


class Bar(models.Model):
    foo = models.ForeignKey('Foo')
    baz = models.ForeignKey('Baz')
    quantity = models.IntegerField()