--
Ticket URL: <https://code.djangoproject.com/ticket/16735>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
* needs_better_patch: => 0
* needs_docs: => 0
* needs_tests: => 0
* stage: Unreviewed => Accepted
Comment:
Interesting idea. From a quick inspection of the source (`ValuesQuerySet`)
this looks doable.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:1>
* owner: nobody => nate_b
* status: new => assigned
* has_patch: 0 => 1
Comment:
This patch is the most direct way I could see of adding this feature.
It passes all run tests in {{{./runtests.py --settings=test_sqlite }}}
I tried to be careful, but if it is missing anything, I'd be happy to take
another look at it.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:2>
* needs_better_patch: 0 => 1
Comment:
While adding this to {{{values()}}} is perfectly fine, doing the same with
{{{values_list()}}} feels weird. Also, it's buggy:
{{{#!python
# from ValuesListQuerySet in the patch
fields = list(self._fields) + self._aliased_fields.keys()
}}}
This makes the order of values in returned tuples depend on order of
dictionary keys, which is _undefined_. Just try:
{{{#!python
print Model.objects.values_list(a2="field", a3="other_field")
# on Python 2.7 and 3.2 values should be reversed.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:3>
* stage: Accepted => Design decision needed
Comment:
Ostensibly, I agree with you - it's a bit odd, and I don't presume to know
why one would use it. Initially, when I added that to
{{{values_list()}}}, it was to "harmonize" it with the features added to
{{{values()}}}. However, similar behavior results when you {{{annotate}}}
or {{{aggregate}}} on a {{{values_list()}}} call - the aliased names are
added in the order of the keys.
So, I would propose one of three solutions:
1. Remove aliased names in the {{{values_list()}}} call entirely,
replaced with a dummy empty dictionary.
2. Force the sorting of the aliases, presumably alphabetically; this
would also suggest repairing this defect with annotations and
aggregations, which may not be quite so simple.
3. Leave as is in my updated patch, with indeterminate order returned.
I will be happy to implement which ever one seems best to a core
developer. As such, I'm setting this to DDN.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:4>
Comment (by akaariai):
What is the rationale of doing this for values_list()? If I am not
mistaken, you will get a list back anyways. So the aliases are used for
what?
The .annotate() + values_list() seems to be a bug. Basically, multiple
annotates in one .annotate() call have indeterminate order, which results
in indeterminate order in the values_list(). I bet some users will be hit
if/when Python randomizes the hashing algorithm. So, indeterminate order
is not good. Unfortunately I don't see a fix other than disallowing
multiple annotates in one call. Which is backwards incompatible.
In addition, some benchmark that this doesn't slow down fetching large
sets of objects from the DB is needed. .values() is mostly an
optimization, so it should remain as fast as possible. Maybe django-bench
contains some benchmark already?
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:5>
* cc: django@… (added)
Comment:
I would disagree with the observation that .values() is just an
optimization. In combination with .annotate() it is vital if you need to
annotate / group by multiple fields. This is also one of the use cases
where the missing aliases are most painful.
Example:
{{{
>>> Region.objects.values(
"name",
"orders__items__product__category__parent__name"
).annotate(quantity = Sum("orders__items__quantity"))
[
{'quantity': 10, 'name': u'...',
'orders__items__product__category__parent__name': u'Something'},
{'quantity': 20, 'name': u'...',
'orders__items__product__category__parent__name': u'Something else'},
]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:6>
Comment (by jrs_66@…):
I'm guessing that the seemingly trivial, and widely used, concept of
aliasing values() lists is still not possible? Ever try a union of self
joined tables? Is there any hope of this being put in place in the
future?
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:7>
Comment (by Karthik):
Replying to [comment:6 UloPe]:
> I would disagree with the observation that .values() is just an
optimization. In combination with .annotate() it is vital if you need to
annotate / group by multiple fields. This is also one of the use cases
where the missing aliases are most painful.
>
> Example:
> {{{
> >>> Region.objects.values(
> "name",
> "orders__items__product__category__parent__name"
> ).annotate(quantity = Sum("orders__items__quantity"))
> [
> {'quantity': 10, 'name': u'...',
'orders__items__product__category__parent__name': u'Something'},
> {'quantity': 20, 'name': u'...',
'orders__items__product__category__parent__name': u'Something else'},
> ]
> }}}
I agree. I'm reading this thread because I have this exact same problem at
the moment.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:8>
* stage: Design decision needed => Accepted
Comment:
Unless I missed something, there isn't any objection to adding this
feature to `values`.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:9>
Comment (by erik.telepovsky):
Replying to [comment:8 Karthik]:
> Replying to [comment:6 UloPe]:
> > I would disagree with the observation that .values() is just an
optimization. In combination with .annotate() it is vital if you need to
annotate / group by multiple fields. This is also one of the use cases
where the missing aliases are most painful.
> >
> > Example:
> > {{{
> > >>> Region.objects.values(
> > "name",
> > "orders__items__product__category__parent__name"
> > ).annotate(quantity = Sum("orders__items__quantity"))
> > [
> > {'quantity': 10, 'name': u'...',
'orders__items__product__category__parent__name': u'Something'},
> > {'quantity': 20, 'name': u'...',
'orders__items__product__category__parent__name': u'Something else'},
> > ]
> > }}}
> I agree. I'm reading this thread because I have this exact same problem
at the moment.
I agree as well. It is important to have alias functionality in values()
method.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:10>
Comment (by akaariai):
Quick observation: it might be better to have the aliasing feature as
separate queryset operation:
{{{
Region.objects.alias(
orders__items__product__category__parent__name="parentname"
).values(
"name", "parentname"
).annotate(quantity=Sum("orders__items__quantity"))
}}}
Why this way? There are a couple other places where aliasing could be
useful. For example in multijoin situations you could explicitly define if
you want to filter the same join or different join by using the same alias
or two different aliases (currently the way is "if filtered in same
.filter() condition, then same joins, else different joins). Also, this
way you might be able to inject extra SQL directly into the query:
{{{
Region.objects.alias(
somesql=RawSQL("case when somecol > 0 then 1 else -1 end")
).order_by('somesql')
}}}
Of course, this ticket should not try to do more than just the bare
minimum to get the aliasing to work for .values(). The point is aliasing
could be useful in other operations, too, so lets be prepared for that.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:11>
Comment (by Wraithan):
Replying to [comment:9 aaugustin]:
> Unless I missed something, there isn't any objection to adding this
feature to `values`.
So my understanding is that this ticket is being held back by touching
`values_list` as well as `values`. So a patch only affecting `values` (as
well as docs and tests of course) is what is needed to get accepted?
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:12>
Comment (by russellm):
@Wraithan As far as I can make out, yes. I can't see any benefit to having
API consistency between `values()` and `values_list()` - if only because
`values_list()` already accepts a keyword argument, which introduces a
whole world of pain in the API (what if you want an alias called `flat`?).
The "`values()` is an optimisation" argument doesn't hold water for me.
Yes, it's an optimisation. The source of the optimisation is passing less
information on the wire during the database query (i.e., only returning
two of 15 fields). An alias is already being used for this operation - it
just isn't user specified. Making it user specified is a single dictionary
lookup in a couple of key locations is a minor change in implementation
with huge usability benefits.
So - clean up the patch and we can get this into trunk.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:13>
Comment (by mjtamlyn):
Agreed this should be possible.
As a side note Russ - I've found the main optimisation for using values
was in fact avoiding calling `Model.__init__` a few thousand times, not
the lack of data on the wire.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:14>
Comment (by django@…):
I've been wanting something like this for a while. Just another use case
(when sending emails with pre-defined, admin-editable text.
{{{
text = Settings.objects.get(name='email_text').value # text is something
like "Hi {name}, you are {age} years old. You work at {job}"
email_texts = [text.format(**kw) for kw in
Person.objects.all().values(name='name', age='age', job="job__name")]
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:15>
* cc: bendavis78 (added)
* version: 1.3 => master
Comment:
I've created a branch on github for a fix against the latest version of
django: https://github.com/bendavis78/django/tree/issues/16735
The patch can be found here:
https://github.com/bendavis78/django/commit/cdff83bed850a631e7c6d3cb12359b9b1d3e9bc4
This patch only changes values() and not values_list().
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:16>
Comment (by paveluc.alexandr@…):
Will this patch be included in next Django release?
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:17>
Comment (by charettes):
Unfortunately this feature didn't make it to the 1.7.x branch before it
was feature frozen.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:18>
Comment (by abdulhaq-e):
Replying to [comment:18 charettes]:
> Unfortunately this feature didn't make it to the 1.7.x branch before it
was feature frozen.
Since the feature freeze deadline for v1.8 is approaching, will this be
added to 1.8??
I tried @bendavis78 patch and it works fine!
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:19>
Comment (by timgraham):
The patch likely needs to be updated to apply cleanly and then we need a
pull request so the patch can be reviewed.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:20>
Comment (by jarshwah):
Just quickly, with the changes to annotate that have landed, it is now
possible to create aliases yourself, and reference them from the values
call:
{{{
Model.objects.annotate(my_alias=F('some__long__name__to__alias')).values('my_alias')
}}}
This is fairly similar to the suggestion Anssi made upthread of using a
new method `alias()`. Not sure if this would be preferable though, as it
is more verbose and requires an extra round of queryset copying.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:21>
* status: assigned => new
* owner: nate_b =>
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:22>
Comment (by timgraham):
Related to "Allow expressions in `values()` queryset method (#25871)" and
[https://groups.google.com/d/topic/django-
developers/m4lPPfFChNY/discussion django-developers discussion].
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:23>
* owner: => Ian-Foote
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:24>
Comment (by timgraham):
[https://github.com/django/django/pull/7088 PR] with some comments for
improvement.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:25>
* status: assigned => closed
* resolution: => wontfix
Comment:
Following the further discussion on [https://groups.google.com/d/topic
/django-developers/m4lPPfFChNY/discussion django-developers], I think the
consensus is not to support aliasing beyond an explicit
{{{.values(alias=F('field'))}}} as supported since #25871 was fixed. I'm
therefore updating this to {{{wontfix}}}.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:26>
Comment (by Ionuț Ciocîrlan):
This being marked as wontfix is confusing. It's in fact been fixed by
#25871, which allows for richer aliases.
For anyone landing here jumping to the conclusion the ticket was rejected,
it wasn't: see Ian's note above -- it's available starting with 1.11.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:27>
* resolution: wontfix => duplicate
Comment:
Fixed by https://code.djangoproject.com/ticket/25871
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:28>
Comment (by Tim Graham):
I've clarified the ticket summary to reflect what was wontfixed.
--
Ticket URL: <https://code.djangoproject.com/ticket/16735#comment:29>