Split a list into two parts based on a filter?

[Roy Smith]

I have a list, songs, which I want to divide into two groups.
Essentially, I want:

new_songs = [s for s in songs if s.is_new()]
old_songs = [s for s in songs if not s.is_new()]

but I don't want to make two passes over the list. I could do:

new_songs = []
old_songs = []
for s in songs:
if s.is_new():
new_songs.append(s)
else:
old_songs.append(s)

Which works, but is klunky compared to the two-liner above. This
seems like a common enough thing that I was expecting to find
something in itertools which did this. I'm thinking something along
the lines of:

matches, non_matches = isplit(lambda s: s.is_new, songs)

Does such a thing exist?

[Roel Schroeven]

Roy Smith schreef:

You could do something like:

new_songs, old_songs = [], []
[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

But I'm not sure that that's any better than the long version.

--
"People almost invariably arrive at their beliefs not on the basis of
proof but on the basis of what they find attractive."
-- Pascal Blaise

ro...@roelschroeven.net

[Chris Angelico]

On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:
> new_songs = [s for s in songs if s.is_new()]
> old_songs = [s for s in songs if not s.is_new()]

Hmm. Would this serve?

old_songs = songs[:]
new_songs = [songs.remove(s) or s for s in songs if s.is_new()]

Python doesn't, AFAIK, have a "destructive remove and return"
operation, and del is a statement rather than an expression/operator,
but maybe this basic idea could be refined into something more useful.
It guarantees to call is_new only once per song.

The iterator version strikes my fancy. Maybe this isn't of use to you,
but I'm going to try my hand at making one anyway.

>>> def iterpartition(pred,it):
"""Partition an iterable based on a predicate.

Returns two iterables, for those with pred False and those True."""
falses,trues=[],[]
it=iter(it)
def get_false():
while True:
if falses: yield falses.pop(0)
else:
while True:
val=next(it)
if pred(val): trues.append(val)
else: break
yield val
def get_true():
while True:
if trues: yield trues.pop(0)
else:
while True:
val=next(it)
if not pred(val): falses.append(val)
else: break
yield val
return get_false(),get_true()
>>> f,t=iterpartition(lambda x: x%3,range(100000000))
>>> next(t)
1
>>> next(t)
2
>>> next(t)
4
>>> next(t)
5
>>> next(f)
0
>>> next(f)
3
>>> next(f)
6
>>> next(f)
9
>>> next(f)
12
>>> next(t)
7
>>> next(t)
8

Ha. :) Useless but fun.

ChrisA

[Chris Rebert]

itertools.groupby() is kinda similar, but unfortunately doesn't fit
the bill due to its sorting requirement.
There is regrettably no itertools.partition(). And given how dead-set
Raymond seems to be against adding things to the itertools module,
there will likely never be.
Maybe more-itertools (https://pypi.python.org/pypi/more-itertools )
would accept a patch?

Cheers,
Chris

[Tim Chase]

On 2013-06-11 08:50, Chris Angelico wrote:
> The iterator version strikes my fancy. Maybe this isn't of use to
> you, but I'm going to try my hand at making one anyway.
>
> >>> def iterpartition(pred,it):
> """Partition an iterable based on a predicate.
>
> Returns two iterables, for those with pred False and those
> True.""" falses,trues=[],[]

This is really nice. I think the only major modification I'd make is
to use a collections.deque() instead of lists here:

trues, falses = collections.deque(), collections.deque()

as I seem to recall that list.pop(0) has some performance issues if
it gets long, while the deque is optimized for fast (O(1)?) push/pop
on either end.

-tkc

[Fábio Santos]

On 10 Jun 2013 23:54, "Roel Schroeven" <ro...@roelschroeven.net> wrote:
>
> You could do something like:
>
> new_songs, old_songs = [], []
> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>
> But I'm not sure that that's any better than the long version.

This is so beautiful!

[Chris Angelico]

Sure. If it's that good I might submit it to more-itertools... heh.

ChrisA

[Peter Otten]

Chris Angelico wrote:

> On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:

>> new_songs = [s for s in songs if s.is_new()]
>> old_songs = [s for s in songs if not s.is_new()]
>

> Hmm. Would this serve?
>
> old_songs = songs[:]
> new_songs = [songs.remove(s) or s for s in songs if s.is_new()]
>
> Python doesn't, AFAIK, have a "destructive remove and return"
> operation, and del is a statement rather than an expression/operator,
> but maybe this basic idea could be refined into something more useful.
> It guarantees to call is_new only once per song.
>

> The iterator version strikes my fancy. Maybe this isn't of use to you,
> but I'm going to try my hand at making one anyway.

> >>> def iterpartition(pred,it):
> """Partition an iterable based on a predicate.
>
> Returns two iterables, for those with pred False and those
True."""
> falses,trues=[],[]

> it=iter(it)
> def get_false():
> while True:
> if falses: yield falses.pop(0)
> else:
> while True:
> val=next(it)
> if pred(val): trues.append(val)
> else: break
> yield val
> def get_true():
> while True:
> if trues: yield trues.pop(0)
> else:
> while True:
> val=next(it)
> if not pred(val):
falses.append(val)
> else: break
> yield val
> return get_false(),get_true()

An alternative implementation, based on itertools.tee:

import itertools

def partition(items, predicate=bool):
a, b = itertools.tee((predicate(item), item) for item in items)
return ((item for pred, item in a if not pred),
(item for pred, item in b if pred))

if __name__ == "__main__":
false, true = partition(range(10), lambda item: item % 2)
print(list(false))
print(list(true))

def echo_odd(item):
print("checking", item)
return item % 2

false, true = partition(range(10), echo_odd)

print("FALSE", [next(false) for _ in range(3)])
print("TRUE", next(true))
print("FALSE", list(false))
print("TRUE", list(true))

[Roy Smith]

In article <mailman.2992.1370904...@python.org>,

Roel Schroeven <ro...@roelschroeven.net> wrote:

> new_songs, old_songs = [], []
> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Thanks kind of neat, thanks.

I'm trying to figure out what list gets created and discarded. I think
it's [None] * len(songs).

[Peter Otten]

Fábio Santos wrote:

> On 10 Jun 2013 23:54, "Roel Schroeven" <ro...@roelschroeven.net> wrote:
>>
>> You could do something like:
>>

>> new_songs, old_songs = [], []
>> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>>

>> But I'm not sure that that's any better than the long version.
>
> This is so beautiful!

It makes me cringe.

This code does spurious work to turn what is naturally written as a for loop
into a list comprehension that is thrown away immediately. I think I'd even
prefer this "gem"

>>> evens = []
>>> odds = [item for item in range(10) if item % 2 or evens.append(item)]
>>> evens, odds
([0, 2, 4, 6, 8], [1, 3, 5, 7, 9])

but if I have my way every side effect in a list comprehension should be
punished with an extra hour of bug-hunting ;)

[Jonas Geiregat]

On 11 Jun 2013, at 01:08, Fábio Santos wrote:

On 10 Jun 2013 23:54, "Roel Schroeven" <ro...@roelschroeven.net> wrote:
>
> You could do something like:
>
> new_songs, old_songs = [], []
> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>
> But I'm not sure that that's any better than the long version.

This is so beautiful!

I must disagree , this is unreadable and in my honor opinion not Pythonic at all.

I've learned it's always better to be explicit then implicit, and this snippet of code does a lot in an implicit way.

[Fábio Santos]

On 11 Jun 2013 07:47, "Peter Otten" <__pet...@web.de> wrote:
>
> Fábio Santos wrote:
>
> > On 10 Jun 2013 23:54, "Roel Schroeven" <ro...@roelschroeven.net> wrote:
> >>
> >> You could do something like:
> >>
> >> new_songs, old_songs = [], []
> >> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
> >>
> >> But I'm not sure that that's any better than the long version.
> >
> > This is so beautiful!
>

> It makes me cringe.
>
> This code does spurious work to turn what is naturally written as a for loop
> into a list comprehension that is thrown away immediately. I think I'd even
> prefer this "gem"
>
> >>> evens = []
> >>> odds = [item for item in range(10) if item % 2 or evens.append(item)]
> >>> evens, odds
> ([0, 2, 4, 6, 8], [1, 3, 5, 7, 9])
>
> but if I have my way every side effect in a list comprehension should be
> punished with an extra hour of bug-hunting ;)
>

What I like so much about it is the .. if .. else .. Within the parenthesis and the append() call outside these parenthesis.

I agree it would be best written as a for loop, to increase readability and avoid confusion. I always expect list comprehensions to be used as expressions, and its use as a (single-expression) statement is rather odd.

On 11 Jun 2013 07:52, "Jonas Geiregat" <jo...@geiregat.org> wrote:
> I must disagree , this is unreadable and in my honor opinion not Pythonic at all.
> I've learned it's always better to be explicit then implicit, and this snippet of code does a lot in an implicit way.
>

(Read above)

[Joshua Landau]

On 11 June 2013 01:11, Peter Otten <__pet...@web.de> wrote:
> def partition(items, predicate=bool):
> a, b = itertools.tee((predicate(item), item) for item in items)
> return ((item for pred, item in a if not pred),
> (item for pred, item in b if pred))

I have to tell you this is the coolest thing I've read today. I'd
never have thought of that.

[Serhiy Storchaka]

11.06.13 07:11, Roy Smith написав(ла):

It is the same as your klunky code, but consumes more memory.

[Serhiy Storchaka]

11.06.13 01:50, Chris Angelico написав(ла):

> On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:

>> new_songs = [s for s in songs if s.is_new()]
>> old_songs = [s for s in songs if not s.is_new()]
>

> Hmm. Would this serve?
>
> old_songs = songs[:]

> new_songs = [songs.remove(s) or s for s in songs if s.is_new()]

O(len(songs)**2) complexity.

[rusi]

On Jun 11, 6:48 pm, Fábio Santos <fabiosantos...@gmail.com> wrote:
>
> What I like so much about it is the .. if .. else .. Within the parenthesis
> and the append() call outside these parenthesis.

You can do this -- which does not mix up functional and imperative
styles badly and is as much a 2-liner as Roy's original.

new_songs, old_songs = [], []
for s in songs: (new_songs if s.is_new() else old_songs).append(s)

[Of course I would prefer a 3-liner where the body of the for is
indented :-) ]

[Fábio Santos]

On 11 Jun 2013 17:47, "rusi" <rusto...@gmail.com> wrote:
> [Of course I would prefer a 3-liner where the body of the for is
> indented :-) ]

Is this an aside comprehension?

[rusi]

On Jun 11, 10:05 pm, Fábio Santos <fabiosantos...@gmail.com> wrote:

> On 11 Jun 2013 17:47, "rusi" <rustompm...@gmail.com> wrote:
>
> > [Of course I would prefer a 3-liner where the body of the for is
> > indented :-) ]
>
> Is this an aside comprehension?

Eh?

Its a for-loop. Same as:

for s in songs:
(new_songs if s.is_new() else old_songs).append(s)

Maybe you are unfamiliar with the line
... A suite can be one or more semicolon-separated simple statements
on the same line as the header,
??

from http://docs.python.org/2/reference/compound_stmts.html

Or you are joking??

What with the offside rule (indentation rule in Haskell) becoming the
'aside rule' causing (in)comprehension in reading comprehensions??

Ok... Jokes aside... Dunno what you are asking.

[Chris Angelico]

On Wed, Jun 12, 2013 at 1:28 AM, Serhiy Storchaka <stor...@gmail.com> wrote:
> 11.06.13 01:50, Chris Angelico написав(ла):
>
>> On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:
>>>

>>> new_songs = [s for s in songs if s.is_new()]
>>> old_songs = [s for s in songs if not s.is_new()]
>>
>>

>> Hmm. Would this serve?
>>
>> old_songs = songs[:]

>> new_songs = [songs.remove(s) or s for s in songs if s.is_new()]
>
>
> O(len(songs)**2) complexity.

Which isn't significant if len(songs) is low. We weren't told the
relative costs - is the is_new call ridiculously expensive? Everything
affects algorithmic choice.

ChrisA

[Chris Angelico]

On Wed, Jun 12, 2013 at 3:23 AM, rusi <rusto...@gmail.com> wrote:
> On Jun 11, 10:05 pm, Fábio Santos <fabiosantos...@gmail.com> wrote:
>> On 11 Jun 2013 17:47, "rusi" <rustompm...@gmail.com> wrote:
>>
>> > [Of course I would prefer a 3-liner where the body of the for is
>> > indented :-) ]
>>
>> Is this an aside comprehension?
>
> Eh?

I know they always say "don't explain the joke", but I'll have a shot
at it. It's somewhat like an autopsy though - you find out why it
ticks, but in the process, you prove that it's no longer ticking...

An aside comprehension is a means of generating an aside in-line, as
an expression. In this case, Fabio believes that you were iterating
over the smiley to generate comments about three-liners, and the
square brackets delimit the comprehension just as they do in a list
comprehension.

It's another case of code syntax cropping up in English, like this
example of ternary punctuation from a C++ program...

//TODO?: blah blah blah

That doesn't translate too well into Python though.

ChrisA

[Fábio Santos]

On 11 Jun 2013 18:48, "Chris Angelico" <ros...@gmail.com> wrote:
>
> On Wed, Jun 12, 2013 at 3:23 AM, rusi <rusto...@gmail.com> wrote:

> > On Jun 11, 10:05 pm, Fábio Santos <fabiosantos...@gmail.com> wrote:
> >> On 11 Jun 2013 17:47, "rusi" <rustompm...@gmail.com> wrote:
> >>
> >> > [Of course I would prefer a 3-liner where the body of the for is
> >> > indented :-) ]
> >>
> >> Is this an aside comprehension?
> >
> > Eh?
>

> I know they always say "don't explain the joke", but I'll have a shot
> at it. It's somewhat like an autopsy though - you find out why it
> ticks, but in the process, you prove that it's no longer ticking...
>
> An aside comprehension is a means of generating an aside in-line, as
> an expression. In this case, Fabio believes that you were iterating
> over the smiley to generate comments about three-liners, and the
> square brackets delimit the comprehension just as they do in a list
> comprehension.
>
> It's another case of code syntax cropping up in English, like this
> example of ternary punctuation from a C++ program...
>
> //TODO?: blah blah blah
>
> That doesn't translate too well into Python though.
>
> ChrisA

> --
> http://mail.python.org/mailman/listinfo/python-list

That was the joke. Unfortunately I didn't have a better synonym for "aside" so it wasn't so obvious.

[rusi]

On Jun 11, 10:37 pm, Chris Angelico <ros...@gmail.com> wrote:

> On Wed, Jun 12, 2013 at 3:23 AM, rusi <rustompm...@gmail.com> wrote:
> > On Jun 11, 10:05 pm, Fábio Santos <fabiosantos...@gmail.com> wrote:
> >> On 11 Jun 2013 17:47, "rusi" <rustompm...@gmail.com> wrote:
>
> >> > [Of course I would prefer a 3-liner where the body of the for is
> >> > indented :-) ]
>
> >> Is this an aside comprehension?
>
> > Eh?
>
> I know they always say "don't explain the joke", but I'll have a shot
> at it. It's somewhat like an autopsy though - you find out why it
> ticks, but in the process, you prove that it's no longer ticking...

Ok :-)

Missed that my aside looked like a comprehension. Or rather did not
comprehend!

[Peter Otten]

Chris Angelico wrote:

> On Wed, Jun 12, 2013 at 1:28 AM, Serhiy Storchaka <stor...@gmail.com>
> wrote:
>> 11.06.13 01:50, Chris Angelico написав(ла):
>>
>>> On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:
>>>>

>>>> new_songs = [s for s in songs if s.is_new()]
>>>> old_songs = [s for s in songs if not s.is_new()]
>>>
>>>

>>> Hmm. Would this serve?
>>>
>>> old_songs = songs[:]

>>> new_songs = [songs.remove(s) or s for s in songs if s.is_new()]

I think you meant old_songs.remove(s).

>> O(len(songs)**2) complexity.
>
> Which isn't significant if len(songs) is low. We weren't told the
> relative costs - is the is_new call ridiculously expensive? Everything
> affects algorithmic choice.

But is it correct? In the general case, no:

>>> numbers = [1, 1.0, 2.0, 2]
>>> ints = numbers[:]
>>> floats = [ints.remove(n) or n for n in numbers if isinstance(n, float)]
>>> floats
[1.0, 2.0]
>>> ints
[1.0, 2] # hmm

[Peter Otten]

:)

[Chris Angelico]

On Wed, Jun 12, 2013 at 4:13 AM, Peter Otten <__pet...@web.de> wrote:
> Chris Angelico wrote:
>
>> On Wed, Jun 12, 2013 at 1:28 AM, Serhiy Storchaka <stor...@gmail.com>
>> wrote:
>>> 11.06.13 01:50, Chris Angelico написав(ла):
>>>
>>>> On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:
>>>>>

>>>>> new_songs = [s for s in songs if s.is_new()]
>>>>> old_songs = [s for s in songs if not s.is_new()]
>>>>
>>>>

>>>> Hmm. Would this serve?
>>>>
>>>> old_songs = songs[:]
>>>> new_songs = [songs.remove(s) or s for s in songs if s.is_new()]
>
> I think you meant old_songs.remove(s).

Ah, yes, editing fail. I started by mutating the original list, then
thought "Oh, better to work with a copy"... and forgot to complete the
edit.

>>> O(len(songs)**2) complexity.
>>
>> Which isn't significant if len(songs) is low. We weren't told the
>> relative costs - is the is_new call ridiculously expensive? Everything
>> affects algorithmic choice.
>
> But is it correct? In the general case, no:
>
>>>> numbers = [1, 1.0, 2.0, 2]
>>>> ints = numbers[:]
>>>> floats = [ints.remove(n) or n for n in numbers if isinstance(n, float)]
>>>> floats
> [1.0, 2.0]
>>>> ints
> [1.0, 2] # hmm

Sure, but the implication of the original is that they're uniquely
identifiable. Anyway, it wasn't meant to be absolutely perfect, just
another notion getting thrown out there... and then thrown out :)

ChrisA

[Roel Schroeven]

Peter Otten schreef:

> Fábio Santos wrote:
>
>> On 10 Jun 2013 23:54, "Roel Schroeven" <ro...@roelschroeven.net> wrote:
>>> You could do something like:
>>>
>>> new_songs, old_songs = [], []
>>> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>>>
>>> But I'm not sure that that's any better than the long version.
>> This is so beautiful!
>
> It makes me cringe.
>
> This code does spurious work to turn what is naturally written as a for loop
> into a list comprehension that is thrown away immediately.

I agree. I see two problems with it: it's not very readable or pythonic,
and it creates a list that is not needed and is immediately thrown away.

I wrote that code just to see if I could make it work that way. In real
code I would never write it that way, let's make that clear :)

[Roy Smith]

In article <mailman.3032.1370971...@python.org>,

Chris Angelico <ros...@gmail.com> wrote:

> On Wed, Jun 12, 2013 at 1:28 AM, Serhiy Storchaka <stor...@gmail.com> wrote:
> > 11.06.13 01:50, Chris Angelico написав(ла):
> >
> >> On Tue, Jun 11, 2013 at 6:34 AM, Roy Smith <r...@panix.com> wrote:
> >>>
> >>> new_songs = [s for s in songs if s.is_new()]
> >>> old_songs = [s for s in songs if not s.is_new()]
> >>
> >>
> >> Hmm. Would this serve?
> >>
> >> old_songs = songs[:]
> >> new_songs = [songs.remove(s) or s for s in songs if s.is_new()]
> >
> >
> > O(len(songs)**2) complexity.

If I didn't want to make two passes over songs, I probably don't want
something that's O(len(songs)^2) :-)

> Which isn't significant if len(songs) is low. We weren't told the
> relative costs - is the is_new call ridiculously expensive? Everything
> affects algorithmic choice.

Assume is_new() is cheap. It's essentially:

return (datetime.utcnow() - self.create_time) < [[a pre-defined timedelta]]

[Roy Smith]

In article <mailman.3023.1370964...@python.org>,

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list :-)

[alex23]

On Jun 11, 9:08 am, Fábio Santos <fabiosantos...@gmail.com> wrote:

> On 10 Jun 2013 23:54, "Roel Schroeven" <r...@roelschroeven.net> wrote:
> > new_songs, old_songs = [], []
> > [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>

> This is so beautiful!

No, it's actually pretty terrible. It creates a list in order to
populate _two other lists_ and then throws away the one made by the
comprehension. There's nothing ugly about multiline for-loops.

[Phil Connell]

You could equivalently pass the generator to deque() with maxlen=0 - this consumes the iterator with constant memory usage.

We are of course firmly in the twilight zone at this point (although this can be a useful technique in general).

Cheers,
Phil

>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

[Roy Smith]

In article <mailman.3050.1371018...@python.org>,

Phil Connell <pcon...@gmail.com> wrote:

> > Well, continuing down this somewhat bizarre path:
> >
> > new_songs, old_songs = [], []
> > itertools.takewhile(
> > lambda x: True,
> > (new_songs if s.is_new() else old_songs).append(s) for s in songs)
> > )
> >
> > I'm not sure I got the syntax exactly right, but the idea is anything
> > that will iterate over a generator expression. That at least gets rid
> > of the memory requirement to hold the throw-away list :-)
>
> You could equivalently pass the generator to deque() with maxlen=0 - this
> consumes the iterator with constant memory usage.
>
> We are of course firmly in the twilight zone at this point (although this
> can be a useful technique in general).

We've been in the twilight zone for a while. That's when the fun
starts. But, somewhat more seriously, I wonder what, exactly, it is
that freaks people out about:

>>>> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Clearly, it's not the fact that it build and immediately discards a
list, because that concern is addressed with the generator hack, and I
think everybody (myself included) agrees that's just horrible.

Or, is it the use of the conditional to create the target for append()?
Would people be as horrified if I wrote:

for s in songs:

(new_songs if s.is_new() else old_songs).append(s)

or even:

for s in songs:
the_right_list = new_songs if s.is_new() else old_songs
the_right_list.append(s)

[Fábio Santos]

On 12 Jun 2013 12:43, "Roy Smith" <r...@panix.com> wrote:
>
> In article <mailman.3050.1371018...@python.org>,
>  Phil Connell <pcon...@gmail.com> wrote:
>
> > > Well, continuing down this somewhat bizarre path:
> > >
> > > new_songs, old_songs = [], []
> > > itertools.takewhile(
> > >     lambda x: True,
> > >     (new_songs if s.is_new() else old_songs).append(s) for s in songs)
> > >     )
> > >
> > > I'm not sure I got the syntax exactly right, but the idea is anything
> > > that will iterate over a generator expression.  That at least gets rid
> > > of the memory requirement to hold the throw-away list :-)
> >
> > You could equivalently pass the generator to deque() with maxlen=0 - this
> > consumes the iterator with constant memory usage.
> >
> > We are of course firmly in the twilight zone at this point (although this
> > can be a useful technique in general).
>
> We've been in the twilight zone for a while.  That's when the fun
> starts.  But, somewhat more seriously, I wonder what, exactly, it is
> that freaks people out about:
>
> >>>> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>
> Clearly, it's not the fact that it build and immediately discards a
> list, because that concern is addressed with the generator hack, and I
> think everybody (myself included) agrees that's just horrible.

I think someone has already said that the problem is that you're creating the list comprehension just for the side effects.

> Or, is it the use of the conditional to create the target for append()?

I particularly liked that part.

[Jussi Piitulainen]

Roy Smith writes:

> We've been in the twilight zone for a while. That's when the fun
> starts. But, somewhat more seriously, I wonder what, exactly, it is
> that freaks people out about:
>
> >>>> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>
> Clearly, it's not the fact that it build and immediately discards a
> list, because that concern is addressed with the generator hack, and
> I think everybody (myself included) agrees that's just horrible.

I expect e(s) in [e(s) for s in songs] to be an expression that is
evaluated for its value and not for an effect. The comprehension
should stand for a list of those values. The one above violates this
expectation.

> Or, is it the use of the conditional to create the target for append()?
> Would people be as horrified if I wrote:
>
> for s in songs:
> (new_songs if s.is_new() else old_songs).append(s)
>
> or even:
>
> for s in songs:
> the_right_list = new_songs if s.is_new() else old_songs
> the_right_list.append(s)

These are fine.

[Serhiy Storchaka]

12.06.13 09:32, Phil Connell написав(ла):
> On 12 Jun 2013 01:36, "Roy Smith" <r...@panix.com <mailto:r...@panix.com>>

> wrote:
> > Well, continuing down this somewhat bizarre path:
> >
> > new_songs, old_songs = [], []
> > itertools.takewhile(
> > lambda x: True,
> > (new_songs if s.is_new() else old_songs).append(s) for s in songs)
> > )
> >
> > I'm not sure I got the syntax exactly right, but the idea is anything
> > that will iterate over a generator expression. That at least gets rid
> > of the memory requirement to hold the throw-away list :-)
>

> You could equivalently pass the generator to deque() with maxlen=0 -
> this consumes the iterator with constant memory usage.

any((new_songs if s.is_new() else old_songs).append(s) for s in songs)

[Fábio Santos]

Why is there no builtin to consume a generator? I find that odd.

On Wed, Jun 12, 2013 at 5:28 PM, Serhiy Storchaka <stor...@gmail.com> wrote:
> 12.06.13 09:32, Phil Connell написав(ла):
>>
>> On 12 Jun 2013 01:36, "Roy Smith" <r...@panix.com <mailto:r...@panix.com>>
>>
>> wrote:

>> > Well, continuing down this somewhat bizarre path:
>> >
>> > new_songs, old_songs = [], []
>> > itertools.takewhile(
>> > lambda x: True,
>> > (new_songs if s.is_new() else old_songs).append(s) for s in songs)
>> > )
>> >
>> > I'm not sure I got the syntax exactly right, but the idea is anything
>> > that will iterate over a generator expression. That at least gets rid
>> > of the memory requirement to hold the throw-away list :-)
>>

>> You could equivalently pass the generator to deque() with maxlen=0 -
>> this consumes the iterator with constant memory usage.
>
>

> any((new_songs if s.is_new() else old_songs).append(s) for s in songs)
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list



--
Fábio Santos

[Terry Reedy]

On 6/12/2013 7:39 AM, Roy Smith wrote:

> starts. But, somewhat more seriously, I wonder what, exactly, it is
> that freaks people out about:
>
>>>>> [(new_songs if s.is_new() else old_songs).append(s) for s in songs]
>
> Clearly, it's not the fact that it build and immediately discards a
> list, because that concern is addressed with the generator hack, and I
> think everybody (myself included) agrees that's just horrible.

It is an example of comprehension abuse. Comprehensions express and
condense a stylized pattern of creating collections from another
collection or collections, possibly filtered. They were not mean to
replace for statements and turn Python into an fp languages. Indeed,
they do replace and expand upon the fp map function. Python for loops
are not evil.

> Or, is it the use of the conditional to create the target for append()?
> Would people be as horrified if I wrote:
>
> for s in songs:
> (new_songs if s.is_new() else old_songs).append(s)

No. That succinctly expresses and implements the idea 'append each song
to one of two lists.

> or even:
>
> for s in songs:
> the_right_list = new_songs if s.is_new() else old_songs
> the_right_list.append(s)
>

--
Terry Jan Reedy

[Terry Reedy]

On 6/12/2013 12:57 PM, Fábio Santos wrote:
> Why is there no builtin to consume a generator? I find that odd.

There are several builtins than consume generators -- and do something
useful with the yielded objects. What you mean is "Why is there no
builtin to uselessly consume a generator?" The question almost answers
itself. A generator generates objects to be used.

> On Wed, Jun 12, 2013 at 5:28 PM, Serhiy Storchaka <stor...@gmail.com> wrote:

>> 12.06.13 09:32, Phil Connell написав(ла):

>>> You could equivalently pass the generator to deque() with maxlen=0 -
>>> this consumes the iterator with constant memory usage.

>> any((new_songs if s.is_new() else old_songs).append(s) for s in songs)

The problem here is that the generator generates and yields an unwanted
sequence of None (references) from the append operations. The proper
loop statement

for s in songs:

(new_songs if s.is_new() else old_songs).append(s)

simply ignores the None return of the appends. Since it does not yield
None over and over, no extra code is needed to ignore what should be
ignored in the first place.

--
Terry Jan Reedy

[Oscar Benjamin]

On 12 June 2013 19:47, Terry Reedy <tjr...@udel.edu> wrote:
> The proper loop statement
>
> for s in songs:

> (new_songs if s.is_new() else old_songs).append(s)

I think I would just end up rewriting this as

for s in songs:
if s.is_new():
new_songs.append(s)
else:
old_songs.append(s)

but then we're back where we started. I don't think any of the
solutions posted in this thread have been better than this. If you
want to make this a nice one-liner then just put this code in a
function.


Oscar