[Django] #31097: StringAgg And ArrayAgg with filtering in subquery generates invalid string_agg() SQL function call

600 views
Skip to first unread message

Django

unread,
Dec 16, 2019, 2:58:53 PM12/16/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
--------------------------------------------+------------------------
Reporter: Laurent Tramoy | Owner: (none)
Type: Bug | Status: new
Component: contrib.postgres | Version: 2.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
--------------------------------------------+------------------------
This issue is similar to issue #30315 but using the `filter` keyword
instead of `ordering`, so I'll reuse the same structure. I tested it on
Django 2.2.8

Consider the following models (in the people app):

{{{
from django.db import models
from django.contrib.postgres.fields import ArrayField


class Person(models.Model):
"""Person model."""

first_name = models.TextField()
last_name = models.TextField()
country = models.TextField(null=True, blank=True)


class Book(models.Model):
"""Book model."""
category = ArrayField(models.CharField(max_length=32), null=True,
default=list)
people = models.ManyToManyField(Person)
}}}

with the following objects:
{{{
p1 = Person.objects.create(first_name="John", last_name="Doe")
p2 = Person.objects.create(first_name="Jane", last_name="Doe")

b1 = Book.objects.create()
b1.people.add(p1)

b2 = Book.objects.create()
b2.people.add(p1, p2)

b3 = Book.objects.create()
}}}

This fails:
{{{
from django.contrib.postgres.aggregates import StringAgg
from django.db.models import Subquery, OuterRef, Q

from people.models import Book

subquery = (
Book.objects.annotate(
_annotated_value=StringAgg(
"people__first_name", ",",
filter=Q(people__first_name__startswith="Ja")
),
)
.filter(pk=OuterRef("pk"),)
.exclude(_annotated_value="")
.values("id")
)
Book.objects.filter(id__in=Subquery(subquery))
}}}

The generated SQL is

{{{
SELECT
"people_book"."id",
"people_book"."category"
FROM
"people_book"
WHERE
"people_book"."id" IN (
SELECT
U0."id"
FROM
"people_book" U0
LEFT OUTER JOIN "people_book_people" U1 ON (U0."id" =
U1."book_id")
LEFT OUTER JOIN "people_person" U2 ON (U1."person_id" = U2."id")
WHERE
U0."id" = ("people_book"."id")
GROUP BY
U0."id"
HAVING
NOT (STRING_AGG(, ',') FILTER (WHERE U2."first_name") =))
}}}

as we can see, the `STRING_AGG argument is wrong.

The same query without the `filter` works fine:

{{{
subquery = (
Book.objects.annotate(
_annotated_value=StringAgg(
"people__first_name", ","
),
)
.filter(pk=OuterRef("pk"),)
.exclude(_annotated_value="")
.values("id")
)
Book.objects.filter(id__in=Subquery(subquery))
}}}

SQL query:

{{{
SELECT
"people_book"."id",
"people_book"."category"
FROM
"people_book"
WHERE
"people_book"."id" IN (
SELECT
U0."id"
FROM
"people_book" U0
LEFT OUTER JOIN "people_book_people" U1 ON (U0. "id" =
U1."book_id")
LEFT OUTER JOIN "people_person" U2 ON (U1."person_id" = U2."id")
WHERE
U0."id" = ("people_book"."id")
GROUP BY
U0."id"
HAVING
NOT (STRING_AGG(U2."first_name", ',') =))
}}}


as well as the same query without using `Subquery`:

{{{
query = (
Book.objects.annotate(
_annotated_value=StringAgg(
"people__first_name", ",",
filter=Q(people__first_name__startswith="Ja")
),
)
.exclude(_annotated_value="")
)
}}}

SQL query:

{{{
SELECT
"people_book"."id",
"people_book"."category",
STRING_AGG("people_person"."first_name", ',') FILTER (WHERE
"people_person"."first_name"::text LIKE Ja %) AS "_annotated_value"
FROM
"people_book"
LEFT OUTER JOIN "people_book_people" ON ("people_book"."id" =
"people_book_people"."book_id")
LEFT OUTER JOIN "people_person" ON ("people_book_people"."person_id" =
"people_person"."id")
GROUP BY
"people_book"."id"
HAVING
NOT (STRING_AGG("people_person"."first_name", ',') FILTER (WHERE
("people_person"."first_name"::text LIKE Ja %)) =)
}}}

Just to make sure I wasn't using an old version, I tried the query from
#30315, which works fine.

NB: I originally noticed that bug using ArrayAgg instead of StringAgg, but
I encountered another bug using ArrayAgg (or at least what I think is a
bug) while writing the example code, I'll report it later if needed

--
Ticket URL: <https://code.djangoproject.com/ticket/31097>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

Django

unread,
Dec 16, 2019, 3:17:20 PM12/16/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
----------------------------------+--------------------------------------

Reporter: Laurent Tramoy | Owner: (none)
Type: Bug | Status: new
Component: contrib.postgres | Version: 2.2
Severity: Normal | Resolution:

Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+--------------------------------------

Comment (by Simon Charette):

The fact the grouping conditions are surfaced to the outer query makes me
believe it could be related to another issue that was addressed in 3.0.
Could you try reproducing with Django 3.0 as well.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:1>

Django

unread,
Dec 16, 2019, 4:32:46 PM12/16/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
----------------------------------+------------------------------------

Reporter: Laurent Tramoy | Owner: (none)
Type: Bug | Status: new
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
----------------------------------+------------------------------------
Changes (by Baptiste Mispelon):

* version: 2.2 => 3.0
* stage: Unreviewed => Accepted


Comment:

Reproduced on current master (ff00a053478fee06bdfb4206c6d4e079e98640ff)
using the code in the ticket.

Here's the prettified query reported by `--debug-sql` after the testcase
failure:
{{{#!sql lineno=1 marks=20
SELECT
"aaaaaaticket31097_book"."id",
"aaaaaaticket31097_book"."category"
FROM
"aaaaaaticket31097_book"
WHERE
"aaaaaaticket31097_book"."id" IN (
SELECT
U0."id"
FROM
"aaaaaaticket31097_book" U0
LEFT OUTER JOIN "aaaaaaticket31097_book_people" U1 ON (U0."id" =
U1."book_id")
LEFT OUTER JOIN "aaaaaaticket31097_person" U2 ON (U1."person_id" =
U2."id")
WHERE
U0."id" = "aaaaaaticket31097_book"."id"


GROUP BY
U0."id"
HAVING
NOT (
STRING_AGG(, ',') FILTER (
WHERE
U2."first_name"

) = ''
)
)
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:2>

Django

unread,
Dec 28, 2019, 2:29:00 PM12/28/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned

Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by David Wobrock):

* status: new => assigned
* owner: (none) => David Wobrock


Comment:

Hi, I'm claiming the ticket if nobody bothers.
I already wrote a testcase that reproduced the bug and started to pin down
the source of the issue.

The first (very) technical info I found:
It seems to be linked to the logic of relabeling aliases of the the filter
clause in {{{django.db.models.sql.query.Query.change_aliases}}}, in the
{{{self.where.relabel_aliases(change_map)}}} logic.
The first node itself is a {{{WhereNode}}} containing {{{(AND: (NOT (AND:
<django.db.models.lookups.Exact object at 0x7f9ea9bfc9b0>)))}}}
When going down the filter tree, we first hit again a {{{WhereNode}}} with
the negation: {{{(NOT (AND: <django.db.models.lookups.Exact object at
0x7f9ea9bfc9b0>))}}} for our exclude clause.
Finally coming to the {{{Exact}}} object, we relabel the left hand-side of
the condition, being the {{{StringAgg}}}.
This calls the
{{{django.db.models.expressions.BaseExpression.relabeled_clone}}} where we
depend on the fields found by {{{self.get_source_expressions()}}}

Either, those source expressions are incomplete because of the way the
{{{OrderableAggMixin}}} plays with the {{{self.get_source_expressions()}}}
(not calling the super), or the {{{change_map}}} which is passed down is
missing some potential aliases.

I'll try to keep investigating and propose a patch in the coming weeks :)

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:3>

Django

unread,
Dec 28, 2019, 2:55:22 PM12/28/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* cc: Simon Charette (added)


Comment:

Did you manage to reproduce against the latest `master` as well? It looks
like it might have been fixed by commits related to #31094.

If it's still the case you seem to be heading in the right direction;
there's likely an issue with `OrderableAggMixin` and the use of `filter`
given how both require some special source expressions handling.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:4>

Django

unread,
Dec 28, 2019, 4:17:26 PM12/28/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by David Wobrock):

Yes, I do reproduce on the latest {{{master}}}. I'll keep digging in that
direction.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:5>

Django

unread,
Dec 28, 2019, 5:04:36 PM12/28/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by David Wobrock):

* has_patch: 0 => 1


Comment:

The PR https://github.com/django/django/pull/12259

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:6>

Django

unread,
Dec 30, 2019, 3:09:13 PM12/30/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* needs_better_patch: 0 => 1


Comment:

Left comments for improvement on the tests but the corrective seems good
to me.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:7>

Django

unread,
Dec 30, 2019, 6:06:55 PM12/30/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Simon Charette):

* needs_better_patch: 1 => 0


Comment:

Adjusted patch seems ready for a final review.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:8>

Django

unread,
Dec 31, 2019, 5:02:49 AM12/31/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mariusz Felisiak <felisiak.mariusz@…>):

* status: assigned => closed
* resolution: => fixed


Comment:

In [changeset:"2f565f84aca136d9cc4e4d061f3196ddf9358ab8" 2f565f84]:
{{{
#!CommitTicketReference repository=""
revision="2f565f84aca136d9cc4e4d061f3196ddf9358ab8"
Fixed #31097 -- Fixed crash of ArrayAgg and StringAgg with filter when
used in Subquery.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:10>

Django

unread,
Dec 31, 2019, 5:02:50 AM12/31/19
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: assigned

Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"7d44aeb388a33ac56353a658b41a6119e93931ff" 7d44aeb3]:
{{{
#!CommitTicketReference repository=""
revision="7d44aeb388a33ac56353a658b41a6119e93931ff"
Refs #31097 -- Added tests for filter in ArrayAgg and StringAgg.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:9>

Django

unread,
Jan 7, 2020, 10:34:08 AM1/7/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by yozzo):

Replying to [comment:10 Mariusz Felisiak <felisiak.mariusz@…>]:


> In [changeset:"2f565f84aca136d9cc4e4d061f3196ddf9358ab8" 2f565f84]:
> {{{
> #!CommitTicketReference repository=""
revision="2f565f84aca136d9cc4e4d061f3196ddf9358ab8"
> Fixed #31097 -- Fixed crash of ArrayAgg and StringAgg with filter when
used in Subquery.
> }}}

Can this fix be backported to v2.2.x also, please?

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:11>

Django

unread,
Jan 7, 2020, 2:12:54 PM1/7/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by felixxm):

Replying to [comment:11 John Fieldman]:
> Can this fix be backported to 3.0.x and 2.2.x also, please?

This patch doesn't qualify for a backport.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:12>

Django

unread,
Jan 7, 2020, 4:17:14 PM1/7/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by John Fieldman):

Replying to [comment:12 felixxm]:


> This patch doesn't qualify for a backport.

The issue is happening for 3.0.x and 2.2.x also, should it be solved
differently for these versions, or won't be solved at all?

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:13>

Django

unread,
Jan 7, 2020, 4:30:29 PM1/7/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

John, based on the investigation that lead to this patch the issue has
been present since the feature was introduced.

Per our backporting policy this means it doesn't qualify for a backport to
3.0.x or 2.2.x anymore.

See https://docs.djangoproject.com/en/3.0/internals/release-process/ for
more details.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:14>

Django

unread,
Jan 8, 2020, 4:50:12 AM1/8/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Reupen Shah):

It does seem that there was a regression in Django 2.2 to me.

The following query works in Django 2.1.15 and master, but not 2.2.9 and
3.0.x:

{{{#!python
from django.contrib.postgres.aggregates import StringAgg
from django.db.models import CharField, OuterRef, Q, Subquery

subquery = Book.objects.annotate(


_annotated_value=StringAgg(
'people__first_name', ', ',

filter=Q(people__first_name='Jane'),
output_field=CharField(),
),
).filter(
pk=OuterRef('pk'),
).values(
'_annotated_value',
)

queryset = Book.objects.annotate(names=Subquery(subquery))

print([book.names for book in queryset])

}}}

(I can file a separate bug for that if you want, though.)

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:15>

Django

unread,
Jan 8, 2020, 10:57:04 AM1/8/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Simon Charette):

Reupen, even if this was a regression in 2.2 this release series is only
receiving security and data loss backport at this point.

Can you confirm this is the case Mariusz?

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:16>

Django

unread,
Jan 9, 2020, 1:48:17 AM1/9/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by felixxm):

Yes it's a regression in 96199e562dcc409ab4bdc2b2146fa7cf73c7c5fe (Django
2.2) so when we added support for `ordering` we also introduced a
regression in `filter`. Nevertheless Django 2.2 is already in extended
support so this doesn't qualify for a backport.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:17>

Django

unread,
Jan 9, 2020, 3:07:09 AM1/9/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

But maybe a backport to 3.0.x?

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:18>

Django

unread,
Jan 9, 2020, 9:05:51 AM1/9/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by felixxm):

Normally we backport only regression from the latest version (in the
mainstream support), e.g. we'll not backport regressions introduced in
Django < 3.0 to the Django 3.0, unless it's "critical".

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:19>

Django

unread,
Jan 9, 2020, 11:53:02 AM1/9/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

Our documentation is not clear about that. As for me, a regression is a
regression and as such should be backported to supported versions.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:20>

Django

unread,
Jan 9, 2020, 3:24:48 PM1/9/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by felixxm):

Do you really think that we should backport fixes for regressions created
few versions ago? and treat them as release blockers. IMO backporting
regressions from the latest version is a reasonable policy, i.e. if no one
reported a regression in 9 months then it's not really crucial.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:21>

Django

unread,
Jan 10, 2020, 1:16:37 AM1/10/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

Tim beat me with a stick plenty of times when I was starting :) — Data
loss bugs and security issues to all supported versions. Regressions and
bugs in new features to current main version. I did think we should be
more liberal but backporting itself isn't risk free, so the more I see it
the more I think this is correct.

A change would be a change in policy though so we'd need to go via the
discussion group and technical board I think.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:22>

Django

unread,
Jan 10, 2020, 7:36:44 AM1/10/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

Carlton, I'm not sure about your comment. Does "Regressions and bugs in
new features to current main version." mean you also are in favor of a
3.0.x backport?
Mariusz is suggesting treating differently the case where a regression is
found after a release cycle passed. I'm not in favor of this distinction.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:23>

Django

unread,
Jan 14, 2020, 3:46:30 AM1/14/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson):

Hi Claude. Good. I think "Regressions" here and in the
[https://docs.djangoproject.com/en/dev/internals/release-process
/#supported-versions supported versions] docs isn't quite clear: it should
say "Regressions introduced since the last release" — that's how it's been
applied that I've seen, and I think it's also
[https://groups.google.com/d/topic/django-
developers/P4Vsa8zqFyo/discussion the historical intention]. (Guess but, I
suspect widening that might lead us to the land of several hundred release
blockers.) I shall propose a small edit to the docs.

Having discussed with Mariusz, we're happy to backport this fix. It's
similar enough to #30315, and is a quite central use-case.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:24>

Django

unread,
Jan 14, 2020, 7:13:35 AM1/14/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Claude Paroz):

Carlton,


> Guess but, I suspect widening that might lead us to the land of several
hundred release blockers

I don't think so. Thanks to its extensive test suite and its wide usage, I
think regressions from older versions are very rare in Django.

Good if this is backported to 3.0.x.

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:25>

Django

unread,
Jan 14, 2020, 8:35:44 AM1/14/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson <carlton@…>):

In [changeset:"927c903f3cd25c817c21738328b53991c035b415" 927c903f]:
{{{
#!CommitTicketReference repository=""
revision="927c903f3cd25c817c21738328b53991c035b415"
Refs #31097 -- Added release notes for
2f565f84aca136d9cc4e4d061f3196ddf9358ab8.

.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:26>

Django

unread,
Jan 14, 2020, 8:42:14 AM1/14/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson <carlton.gibson@…>):

In [changeset:"6aac9f6b1148bdc24dd282b9932c1ae55b489724" 6aac9f6b]:
{{{
#!CommitTicketReference repository=""
revision="6aac9f6b1148bdc24dd282b9932c1ae55b489724"
[3.0.x] Refs #31097 -- Added release notes for
2f565f84aca136d9cc4e4d061f3196ddf9358ab8.

.

Backport of 927c903f3cd25c817c21738328b53991c035b415 from master
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:28>

Django

unread,
Jan 14, 2020, 8:42:16 AM1/14/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Carlton Gibson <carlton.gibson@…>):

In [changeset:"0e6cf4393c84597273d119cf64feec2ef66e35c7" 0e6cf439]:
{{{
#!CommitTicketReference repository=""
revision="0e6cf4393c84597273d119cf64feec2ef66e35c7"
[3.0.x] Fixed #31097 -- Fixed crash of ArrayAgg and StringAgg with filter
when used in Subquery.

Backport of 2f565f84aca136d9cc4e4d061f3196ddf9358ab8 from master
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:27>

Django

unread,
Jan 17, 2020, 3:13:57 AM1/17/20
to django-...@googlegroups.com
#31097: StringAgg And ArrayAgg with filtering in subquery generates invalid
string_agg() SQL function call
-------------------------------------+-------------------------------------
Reporter: Laurent Tramoy | Owner: David
| Wobrock
Type: Bug | Status: closed
Component: contrib.postgres | Version: 3.0
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mariusz Felisiak <felisiak.mariusz@…>):

In [changeset:"5b6778b8b98a3999d15ec3560ad605cdd302e23b" 5b6778b8]:
{{{
#!CommitTicketReference repository=""
revision="5b6778b8b98a3999d15ec3560ad605cdd302e23b"
[3.0.x] Refs #31097 -- Added django.db.models.Q import to contrib.postgres
aggregates tests.
}}}

--
Ticket URL: <https://code.djangoproject.com/ticket/31097#comment:29>

Reply all
Reply to author
Forward
0 new messages