first() and last(), earliest() and latest()

[Wim Feijen]

Hi all,

We struggled to get a proper definition for a first() and last() method vs. earliest() and latest() . I'd like to make one proposal. After that, I really like your opinion on which syntax you prefer.

First, let me give you a lenghty introduction. We discussed several use cases on this mailing list. Then, I realized that:

.filter(last_name__startswith='b').order_by('last_name').first()

is an acceptable compromise for me to use in stead of:

.first(last_name__startswith='b').order_by('last_name')

Last weekend Aymeric explained to me that earliest can actually accomplish the same:

.filter(last_name__startswith='b').earliest('last_name')

Then, I find "earliest" an inappropriate name, because it does not describe functioning well.

Therefore, my proposal is, if we are going to implement the earliest and latest method, we should definitely rename them to: first and latest.

After that, there is only one question left:

Which style do you prefer?

.filter(last_name__startswith='b').order_by('last_name').first()    # clear and long

.first(last_name__startswith='b').order_by('last_name')    # first method has filter syntax.

.filter(last_name__startswith='b').first('last_name')   # first method has order_by syntax.

So, what do you think?

Best regards,

Wim

[Aymeric Augustin]

On 27 févr. 2013, at 23:34, Wim Feijen <w...@go2people.nl> wrote:

> Therefore, my proposal is, if we are going to implement the earliest and latest method, we should definitely rename them to: first and latest.

I believe that latest() was introduced with dates in mind, and then earliest() for consistency with latest().
Note that models also have a "get_latest_by" meta attribute defining the order if none is specified.

You're absolutely right: this feature is useful for any field with a meaningful order, and latest / earliest aren't an intuitive API.

I would support renaming them to first / last through a deprecation path. The new aliases would be available immediately, the old ones would be removed in two versions.

And while we're there, I suggest to rely on the existing "ordering" meta attribute and to remove "get_latest_by". I suspect that in many cases these two attributes have the same value, and you can specify an explicit ordering otherwise.

--
Aymeric.

[Shai Berger]

On Thursday 28 February 2013, Aymeric Augustin wrote:
>
> I would support renaming them to first / last through a deprecation path.
> The new aliases would be available immediately, the old ones would be
> removed in two versions.
>

+1

> And while we're there, I suggest to rely on the existing "ordering" meta
> attribute and to remove "get_latest_by". I suspect that in many cases
> these two attributes have the same value, and you can specify an explicit
> ordering otherwise.

Consistent with the above, +1

and as far as Wim's original question is concerned:

> Which style do you prefer?
>
> .filter(last_name__startswith='b').order_by('last_name').first() # clear
> and long
> .first(last_name__startswith='b').order_by('last_name') # first method
> has filter syntax.
> .filter(last_name__startswith='b').first('last_name') # first method has
> order_by syntax.

ordering is given by position, filtering by keyword arguments -- why not
support both?

def first (self, *ordering, **filtering):
...

My only concern is one that Anssi raised -- the check for multiple objects is
discarded, and there is no convenient way to get 0-or-1 objects (which is the
semantics of the try-get-except-DoesNotExist-return-None pattern this is
designed to replace). I don't think it has been given enough attention in the
discussion so far.

One option -- with my suggested syntax -- is that if no ordering is given,
then it means only one is expected (and, say, None could be used to say "use
default ordering"). I suspect, though, that this might be a confusing (and
conflated) interface.

Or maybe it can be saved by saying you must use one or the other but not both;
then it's "overloaded", but nothing really surprising happens. This way, None
could be used to say "No ordering -- there should be only one", which is even
more intuitive.

We get (semantically):

qset.first('last_name') ==>
qset.order_by('last_name')[0] if qset.exists() else None

qset.first(None) ==>
qset.get() if qset,exists() else None

qset.first(last_name__startswith='b') ==>
qset.filter(last_name__startswith='b').first(None)

qset.first("last_name", last_name__startswith='b') ==>
raise TypeError("first() takes either all positional args or all keywords")

qset.first() ==>
qset.first(qset.model.ordering)

Note that with this suggestion:

qset.filter(a=b).first(c)

is not equivalent to

qset.order_by(c).filter(a=b)

Because the latter checks for multiple objects and the former doesn't; this.
IMO, justifies raising a type-error when trying to use both.

Shai.

[Shai Berger]

Oopsie:

On Thursday 28 February 2013, Shai Berger wrote:
>
> Note that with this suggestion:
>
> qset.filter(a=b).first(c)
>
> is not equivalent to
>
> qset.order_by(c).filter(a=b)
>

That should have read

qset.order_by(c).first(a=b)

Shai.

[AK]

On 02/27/2013 05:34 PM, Wim Feijen wrote:
> Hi all,
>
> We struggled to get a proper definition for a first() and last()
> method vs. earliest() and latest() . I'd like to make one proposal.
> After that, I really like your opinion on which syntax you prefer.
>
> First, let me give you a lenghty introduction. We discussed several
> use cases on this mailing list

> <https://groups.google.com/forum/?fromgroups=#%21searchin/django-developers/get_default/django-developers/3RwDxWKPZ_A/mPtAlQ2b0DwJ>.

> Then, I realized that:
>
> .filter(last_name__startswith='b').order_by('last_name').first()
> is an acceptable compromise for me to use in stead of:
> .first(last_name__startswith='b').order_by('last_name')
>
> Last weekend Aymeric explained to me that earliest can actually
> accomplish the same:
> .filter(last_name__startswith='b').earliest('last_name')
>
> Then, I find "earliest" an inappropriate name, because it does not
> describe functioning well.
>
> Therefore, my proposal is, if we are going to implement the earliest
> and latest method, we should definitely rename them to: first and latest.
>
> After that, there is only one question left:
>
> Which style do you prefer?
>
> .filter(last_name__startswith='b').order_by('last_name').first() #
> clear and long
> .first(last_name__startswith='b').order_by('last_name') # first
> method has filter syntax.
> .filter(last_name__startswith='b').first('last_name') # first method
> has order_by syntax.
>
> So, what do you think?
>
> Best regards,
>
> Wim
>

How about first as a function? I think it's very often useful not just
for querysets but
for all kind of iterables:

def first(iterable, default=None):
try:
return next(iter(iterable))
except StopIteration:
return default

-ak

[Ian Kelly]

On Wed, Feb 27, 2013 at 3:34 PM, Wim Feijen <w...@go2people.nl> wrote:
> Which style do you prefer?
>
> .filter(last_name__startswith='b').order_by('last_name').first() # clear
> and long
> .first(last_name__startswith='b').order_by('last_name') # first method
> has filter syntax.
> .filter(last_name__startswith='b').first('last_name') # first method has
> order_by syntax.

+1 for the first syntax. The others are duplicating functionality
that is already present via more aptly named methods. The first
syntax is also more consistent with other ORMs.

[Tom Christie]

> +1 for the first syntax.  The others are duplicating functionality 
> that is already present via more aptly named methods.  The first 
> syntax is also more consistent with other ORMs. 

Seconded.

Seems much more obvious, simple and explicit than the other options to me.

[Camilo Nova]

+1 for first and last

[Stephen Burrows]

So if I understand correctly, you want something that returns 0-or-1 objects, without having to do try/except? (Or alternately, try: qs[:1][0] except IndexError.)

First of all, I don't understand why first/last are the names you've chosen. This isn't about getting the first object or the last object. It's about getting first/last or None. That behavior is inconsistent with the way that Django's ORM works. (Though granted, it wouldn't be the only internal inconsistency.) A name like get_or_none() would make more sense to me, but apparently that was vetoed in the previous thread.

Second, I don't understand what you expect to gain from adding these methods. Yes, I know you save a few lines of code. If it doesn't matter whether you get an object or not, you save three lines. If you have to take different paths, you only save one line. Essentially you're replacing this:

try:

 obj = Model.instance.order_by(...)[:1][0]

except IndexError:
 # Do something
else:

 # Do something else

With this:

obj = Model.instance.order_by(...).first()

if obj is None:

 # Do something

else:

 # Do something else

The gains don't seem to justify the means.

Third, this seems to be about replacing latest/earliest. Personally, I don't have much use for either method, since they can be replaced with the first snippet above without any trouble. If anything, I'd rather see latest/earliest removed from the API than add new methods that do essentially the same thing. What I might see being useful would be a method to reverse the ordering of a queryset - which it turns out already exists.

Fourth, the methods proposed don't seem to do anything beyond act as shortcuts. In contrast, get_or_create (which on the surface is a similar shortcut) also adds protection against database inconsistencies by manually handling transactions.

[Selwin Ong]

+1 for the first syntax too :)

[Anssi Kääriäinen]

On 28 helmi, 01:34, Wim Feijen <w...@go2people.nl> wrote:
> Hi all,
>
> We struggled to get a proper definition for a first() and last() method vs.
> earliest() and latest() . I'd like to make one proposal. After that, I
> really like your opinion on which syntax you prefer.
>
> First, let me give you a lenghty introduction. We discussed several use

> cases on this mailing list<https://groups.google.com/forum/?fromgroups=#!searchin/django-develop...>.

> Then, I realized that:
>
> .filter(last_name__startswith='b').order_by('last_name').first()
> is an acceptable compromise for me to use in stead of:
> .first(last_name__startswith='b').order_by('last_name')
>
> Last weekend Aymeric explained to me that earliest can actually accomplish
> the same:
> .filter(last_name__startswith='b').earliest('last_name')
>
> Then, I find "earliest" an inappropriate name, because it does not describe
> functioning well.
>
> Therefore, my proposal is, if we are going to implement the earliest and
> latest method, we should definitely rename them to: first and latest.
>
> After that, there is only one question left:
>
> Which style do you prefer?
>
> .filter(last_name__startswith='b').order_by('last_name').first()    # clear
> and long
> .first(last_name__startswith='b').order_by('last_name')    # first method
> has filter syntax.
> .filter(last_name__startswith='b').first('last_name')   # first method has
> order_by syntax.
>
> So, what do you think?

While investigating the queryset iterator memory leak issue I spotted
some instances of this anti-pattern:
if qs:
obj = qs[0]
else:
obj = ...
This has the side effect of fetching the whole dataset into memory
(except on Oracle).

.get() doesn't work for this use-case as multiple objects returned
will cause an error. One solution is:
qs = qs[0:1]
if qs:
obj = qs[0]
else:
obj = ...
but this isn't exactly clean.

It would be nice if one could write:
obj = qs.first() or ...
this is clean and short. (Or equivalently: obj = qs.first(); if not
obj: obj = ...).

I propose the following solution: deprecate earliest()/latest() and
replace them by first()/last(). When using first()/last() no match
isn't an exception, instead None is returned. The method returns the
first/last object by:
1) the given by_fields
2) if no by_fields, then Meta.ordering (for earliest/latest this was
by Meta.get_latest_by)
3) raise error (similar to earilest/latest() - another option would
be implicit ordering by pk, but explicit is better...).

The underlying problem is that there is no simple way to return the
first object of the queryset in a way that multiple or no objects
matched isn't an exception. So, users are either going to write code
using the above mentioned antipattern or forced to use exceptions for
logic.

- Anssi

[Selwin Ong]

Hi everyone,

I submitted a pull request implementing "first()" and "last()" here:  https://github.com/django/django/pull/1054

Comments welcome!

Best,

Selwin

[Shai Berger]

Hi Selwin,

On Saturday 11 May 2013, Selwin Ong wrote:
> Hi everyone,
>
> I submitted a pull request implementing "first()" and "last()" here:
> https://github.com/django/django/pull/1054
>
> Comments welcome!
>

You implemented the "order_by" parameter as taking only one field name; this is
inconsistent with the way ordering is done elsewhere -- both the queryset
order_by() method, and the model meta ordering option, take a list of field
names, each optionally prefixed by "-".

While I can see reason in going for a simpler implementation, I think it is
much preferable to have a consistent API; and even if the simpler
implementation is preferred, you should take care of a more friendly error
message for someone who would call qset.first("a","b") or even qset.last("-a")
[the latter may seem to make no sense -- "if that's what you want, call
qset.first('a')" -- but may reasonably arise in situations where the ordering
field is received as an argument].

Shai.

[Selwin Ong]

Hi everyone,

I opened a new pull request implementing Shai's suggestions (I didn't overwrite the current branch so we can still compare the two implementations if needed).

I initially modeled "first()" and "last()"'s behaviors to mimic "latest()", but in this new pull request, you can pass multiple field names into "first()" and "last()" so it behaves like "order_by()". It's more flexible and requires less typing, but I wonder if we should just get rid of the optional field arguments and rely on "order_by" for ordering. "There should be one-- and preferably only one --obvious way to do it".

Thoughts?

The new pull request is here: https://github.com/django/django/pull/1056

Best,

Selwin

[Wim Feijen]

Hi Selwin,

Considering "There should be one-- and preferably only one --obvious way to do it", I definitely prefer to rely on order_by to do the ordering, not on first. 

.order_by('name').first()

is clear and readable in my opinion.

Wim

[Michal Petrucha]

> > I initially modeled "first()" and "last()"'s behaviors to mimic
> > "latest()", but in this new pull request, you can pass multiple field names
> > into "first()" and "last()" so it behaves like "order_by()". It's more
> > flexible and requires less typing, but I wonder if we should just get rid
> > of the optional field arguments and rely on "order_by" for ordering. "There
> > should be one-- and preferably only one --obvious way to do it".
>

> Considering "There should be one-- and preferably only one --obvious way to
> do it", I definitely prefer to rely on order_by to do the ordering, not on
> first.
>
> .order_by('name').first()
>
> is clear and readable in my opinion.

My thoughts exactly, we already have one method that does ordering, I
don't think it is necessary to make these methods incorporate that
functionality. If we did, we might argue that other QuerySet
operations could be supported as well and that would just result in a
bloated API. Especially if there's no performance gain (the QuerySet
would be cloned anyway), and it only saves a few lines of code.

Also, skimming through this thread, I think there was a consensus on
first() and last() not taking any ordering arguments, i.e. the first
proposed syntax:

.filter(last_name__startswith='b').order_by('last_name').first()

Michal

[Selwin Ong]

I've updated the first() and last() to not accept any arguments. Please review it and let me know if there's anything else I need to change. Hopefully this can get merged in during the sprints and make it into 1.6 :).

The pull request is here: https://github.com/django/django/pull/1056 

Best,

Selwin

[Lee Trout]

Is qs[:1][0] better form than list(qs)[0]?

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Alex Ogier]

Significantly better. The latter method loads every single model in the queryset into Python, potentially the whole database!

[Shai Berger]

Two notes:

1) I think it is better to leave the *args, **kw on the manager methods; since
they are just forwarding to the qset anyways, there's no harm in that, and it
makes them more "future proof" (i.e. you wouldn't need to change them next
time you change the interface of the qset methods).

Case in point:

2) I'd like to add an argument to the first() method, something like
def first(self, only=True):

Where the sense of only=True is that if there is more than one object to get,
that's an error (MultipleObjects, like in get).

As this feature was mainly designed to replace the anti-pattern,

try:
x = qset.get(...)
except DoesNotExist
x = None

I think the default for "only" should indeed be True. Other people may
disagree. I think without at least the ability to set only=True, this feature
is essentially broken -- since it is the equivalent of

try:
x = qset[0]
except IndexError
x = None

it encourages people to give up an integrity check.

My 2 cents,
Shai.

[Ryan Hiebert]

Only is a different purpose than first, and as such it would make sense to me if it was a separate method. First has some implication that there may be second, third as well. It comes down to taste, but I think my buds would prefer a separate method.

Ryan

—
Sent from Mailbox for iPhone

[Lee Trout]

That's what I thought- But why not just qs[0]?

Doesn't qs[:1] and qs[0] both cause a LIMIT 1 on the query? It seems that the [:1] is unnecessary. I would expect both to raise IndexError.

[Lee Trout]

Let me clarify that I would expect both qs[:1][0] and qs[0] to raise IndexError if there were no results.

[Alex Ogier]

QuerySets don't support indexing. I'm not entirely sure why, but I
think the reason is to encourage more efficient database usage, since
naive usage of indexing would lead to many repeated LIMIT 1 OFFSET N
type queries. Forcing people to slice first encourages them to think
about exactly how many results they need and evaluate them all at
once.

[Alex Gaynor]

Querysets definitely support indexing.

Alex

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

GPG Key fingerprint: 125F 5C67 DFE9 4084

[Selwin Ong]

Hi Shai,

We can add args and kwargs later when it's actually needed, it's not needed right now and it' cleaner, I think.

As for "only=True", these methods are meant to replace "latest()". If someone ones to ensure that the queryset returns only one object, I dont think "Queryset.get()" is going anywhere soon so they should use that. Besides, "first()" and "last()" don't really make sense if you expect a queryset to only return a single object - ordering doesn't matter.

That's just what I think though.

Best,
Selwin

[Anssi Kääriäinen]

I just pushed the patch to master. I didn't do anything to the API in
the latest pull. No only=True or .only(), and earliest() and latest()
still exist. I didn't feel like bikeshedding this issue any more.

I am happy that we now have some way to get single object from a
queryset without using try-except.

Thanks to all who contributed!

- Anssi

[Jacob Kaplan-Moss]

On Tue, May 21, 2013 at 10:45 AM, Anssi Kääriäinen
<anssi.ka...@thl.fi> wrote:
> I just pushed the patch to master. I didn't do anything to the API in
> the latest pull. No only=True or .only(), and earliest() and latest()
> still exist. I didn't feel like bikeshedding this issue any more.
>
> I am happy that we now have some way to get single object from a
> queryset without using try-except.

Woohoo, thanks!

Jacob