Negative subscripts

[Frank Millman]

Hi all

In my program I have a for-loop like this -

>>> for item in x[:-y]:
...    [do stuff]

'y' may or may not be 0. If it is 0 I want to process the entire list
'x', but of course -0 equals 0, so it returns an empty list.

In theory I can say

>>> for item in x[:-y] if y else x:
...    [do stuff]

But in my actual program, both x and y are fairly long expressions, so
the result is pretty ugly.

Are there any other techniques anyone can suggest, or is the only
alternative to use if...then...else to cater for y = 0?

Thanks

Frank Millman

[Edmondo Giovannozzi]

First assign your long expressions to variables x and y and then you may write:

for item in x[:len(x) - y]:
...

[Antoon Pardon]

Op 26/11/2021 om 10:17 schreef Frank Millman:

> Hi all
>
> In my program I have a for-loop like this -
>
> >>> for item in x[:-y]:
> ...    [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list
> 'x', but of course -0 equals 0, so it returns an empty list.
>
> In theory I can say
>
> >>> for item in x[:-y] if y else x:
> ...    [do stuff]
>
> But in my actual program, both x and y are fairly long expressions, so
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only
> alternative to use if...then...else to cater for y = 0?

It depends. Since you are talking about x being a fairly long expression,
how about adapting that expression so that is produces the reverse of
what is now produced and then use the [y:] slice?

You can also adapt the y expression so that instead of y it produces
len(x) - y.

You may consider writing a reverse_view function, that takes a list
as argument and produces something that behaves as the reversed list
without actually reversing the list and then as in the first option
use the [y:] slice on that.

--
Antoon Pardon.

[Rob Cliffe]

You could evaluate y separately:

yval = <long expression>
for item in x[:-yval] if yval else x:
    [do stuff]

or you could do it using the walrus operator:

for item in x[:-yval] if (yval := <long expression>) else x:
    [do stuff]

or, perhaps simplest, you could do

for item in x[:-y or None]: # a value of None for a slice argument means
"don't slice here"
    [do stuff]

Best wishes
Rob Cliffe

On 26/11/2021 09:17, Frank Millman wrote:
> Hi all
>
> In my program I have a for-loop like this -
>
> >>> for item in x[:-y]:
> ...    [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list
> 'x', but of course -0 equals 0, so it returns an empty list.
>
> In theory I can say
>
> >>> for item in x[:-y] if y else x:
> ...    [do stuff]
>
> But in my actual program, both x and y are fairly long expressions, so
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only
> alternative to use if...then...else to cater for y = 0?
>

> Thanks
>
> Frank Millman

[Chris Angelico]

On Fri, Nov 26, 2021 at 10:11 PM Rob Cliffe via Python-list
<pytho...@python.org> wrote:
> or, perhaps simplest, you could do
>
> for item in x[:-y or None]: # a value of None for a slice argument means
> "don't slice here"
> [do stuff]
>

This is the one I'd recommend. If you're negating a slice like this,
just add "or None" to handle negative zero as the other end of the
list.

Be aware that this will still slice even if you're going to use the
whole list, unlike the "x[-y] if y else x" approach, which will
iterate over the original list. That might be important.

ChrisA

[Frank Millman]

On 2021-11-26 11:17 AM, Frank Millman wrote:
> Hi all
>
> In my program I have a for-loop like this -
>
> >>> for item in x[:-y]:
> ...    [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list
> 'x', but of course -0 equals 0, so it returns an empty list.
>
> In theory I can say
>
> >>> for item in x[:-y] if y else x:
> ...    [do stuff]
>
> But in my actual program, both x and y are fairly long expressions, so
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only
> alternative to use if...then...else to cater for y = 0?
>

Thanks for all the replies. A selection of neat ideas for me to choose from.

Much appreciated.

Frank

[Pieter van Oostrum]

If you put it in a function with x and y as parameters, then both x and y are just a simple identifier inside the function.
And then you can then even eliminate the if with

for item in x[:len(x)-y]:
--
Pieter van Oostrum <pie...@vanoostrum.org>
www: http://pieter.vanoostrum.org/
PGP key: [8DAE142BE17999C4]

[dn]

On 26/11/2021 22.17, Frank Millman wrote:
> In my program I have a for-loop like this -
>
>>>> for item in x[:-y]:
> ...    [do stuff]
>
> 'y' may or may not be 0. If it is 0 I want to process the entire list
> 'x', but of course -0 equals 0, so it returns an empty list.

...

> But in my actual program, both x and y are fairly long expressions, so
> the result is pretty ugly.
>
> Are there any other techniques anyone can suggest, or is the only
> alternative to use if...then...else to cater for y = 0?

Here's hoping that this good-looking boy has not missed the point about
"ugly".

Is this the problem, re-stated to be fully explicit, with (almost)
zero-complexity? [Zen of Python]

>>> x = [ "a", "b", "c", "d", "e" ]
>>> for y in [ 1, 2, 3, 4, 5 ]:
... print( y, x[ :-y ] )
...
1 ['a', 'b', 'c', 'd']
2 ['a', 'b', 'c']
3 ['a', 'b']
4 ['a']
5 []

Whilst the code 'works', it lacks the 'zero-case' of what is not to be
included - which is an overly-complicated way of saying: misses the
'all-case'.

As observed, if we add a zero to the slice/loop-control that fails
logically (and we get more religion: "the first shall be last, and the
last shall be first". [Christian Bible])

>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
... print( y, x[ :-y ] )
...
0 []
1 ['a', 'b', 'c', 'd']
2 ['a', 'b', 'c']
3 ['a', 'b']
4 ['a']
5 []

The problem, as expressed, is that whilst there is a negative of 1, ie
-1, and other positive-integers, there is no (material) negative of 0,
ie 0 == -0 == +0.


(if the following seems pedantic and 'talking down', it is because I'm
filleting a tutorial prepared for Python-Apprentices)


Remember that 'salvation' may lie in the somewhat mysterious rules of
slicing's default values. According to the Python rule-book (not exactly
a religious text, but we must tend it with some fervor... [Built-in
Types] https://docs.python.org/3/library/stdtypes.html?highlight=slice)

«
s[i] ith item of s, origin 0 (3)

s[i:j] slice of s from i to j (3)(4)

s[i:j:k] slice of s from i to j with step k (3)(5)

...

(3) If i or j is negative, the index is relative to the end of sequence
s: len(s) + i or len(s) + j is substituted. But note that -0 is still 0.
»

Which anticipates the problem 'here', and only rubs salt into the wound.

Pressing on...

«
(4) The slice of s from i to j is defined as the sequence of items with
index k such that i <= k < j. If i or j is greater than len(s), use
len(s). If i is omitted or None, use 0. If j is omitted or None, use
len(s). If i is greater than or equal to j, the slice is empty.


(5) The slice of s from i to j with step k is defined as the sequence of
items with index x = i + n*k such that 0 <= n < (j-i)/k. In other words,
the indices are i, i+k, i+2*k, i+3*k and so on, stopping when j is
reached (but never including j). When k is positive, i and j are reduced
to len(s) if they are greater. When k is negative, i and j are reduced
to len(s) - 1 if they are greater. If i or j are omitted or None, they
become “end” values (which end depends on the sign of k). Note, k cannot
be zero. If k is None, it is treated like 1.
»

...and so we see the open-closed nature of Python's ranges and
range-alike objects!


The 'secret' lies in the 'sins of omission'. As above, if either the
'start' or the 'stop' attribute is omitted, then it is treated as None, ie

x[ :-y ] becomes x[ None:-y ]

Thereafter we look at how None is interpreted. In the above example, the
'start' None becomes zero, thus:

x[ :-y ] becomes x[ None:-y ] and thus x[ 0:-y ]


The pertinent topic is the other end of the range. A 'stop' None will
become len( x ), thus putting it together with the 'start' points, the
short-hand form of a copy-operation is:

x[ : ] which becomes x[ None:None ] and thus x[ 0:len( x ) ]


Let's combine that with the idea of negative indexes/indices.
Referring-back to (3) (above).

If all of x is to be included: x[ :] is the way to go - but another way
to express that is:

x[ :len( x ) ]


If the last item in x is to be excluded: x[ :-1 ]

If the last two items ... excluded: x[ :-2 ]

(etc)

and another way to write those two specific examples is

x[ :len( x ) - 1 ] and x[ :len( x ) - 2 ]


and while we are taking things 'away' from the end of the iterable, the
missing x[ :len( x ) ]

is the same as x[ :len( x ) - 0 ]


Thus, by now, you've leaped ahead of me, to:

>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
... print( y, x[ :len( x ) - y ] )
...
0 ['a', 'b', 'c', 'd', 'e']
1 ['a', 'b', 'c', 'd']
2 ['a', 'b', 'c']
3 ['a', 'b']
4 ['a']
5 []

and yes, if computing y is expensive/ugly, for extra-credit, calculate
the 'stop' value outside/prior-to the for-loop!
--
Regards,
=dn

[Greg Ewing]

> On 2021-11-26 11:17 AM, Frank Millman wrote:
>> Are there any other techniques anyone can suggest, or is the only
>> alternative to use if...then...else to cater for y = 0?

x[:-y or None]

Seems to work:

>>> l
['a', 'b', 'c', 'd', 'e']
>>> def f(x): return l[:-x or None]
...
>>> f(3)
['a', 'b']
>>> f(2)
['a', 'b', 'c']
>>> f(1)
['a', 'b', 'c', 'd']
>>> f(0)
['a', 'b', 'c', 'd', 'e']

--
Greg

[Frank Millman]

On 2021-11-26 11:24 PM, dn via Python-list wrote:
> On 26/11/2021 22.17, Frank Millman wrote:
>> In my program I have a for-loop like this -
>>
>>>>> for item in x[:-y]:
>> ...    [do stuff]
>>
>> 'y' may or may not be 0. If it is 0 I want to process the entire list
>> 'x', but of course -0 equals 0, so it returns an empty list.
> ...
>

[...]

That was an interesting read - thanks for spelling it out.

>
>>>> for y in [ 0, 1, 2, 3, 4, 5 ]:
> ... print( y, x[ :len( x ) - y ] )
> ...
> 0 ['a', 'b', 'c', 'd', 'e']
> 1 ['a', 'b', 'c', 'd']
> 2 ['a', 'b', 'c']
> 3 ['a', 'b']
> 4 ['a']
> 5 []
>
> and yes, if computing y is expensive/ugly, for extra-credit, calculate
> the 'stop' value outside/prior-to the for-loop!
>

Ignoring the 'ugly' for the moment, what if computing y is expensive?

To check this, I will restate the example to more closely match my use case.

>>> x = [1, 2, 3, 4, 5, 6, 7]
>>> y = [5, 4, 3]
>>> z = []
>>>
>>> for i in x[ : len(x) - len(y) ]:
... i
...
1
2
3
4
>>>
>>> for i in x[ : len(x) - len(z) ]:
... i
...
1
2
3
4
5
6
7
>>>

So it works perfectly (not that I had any doubts).

But what if it is expensive to compute y? Or to rephrase it, is y
computed on every iteration, or only on the first one?

Without knowing the internals, it is not possible to tell just by
looking at it. But there is a technique I learned from Peter Otten
(haven't heard from him for a while - hope he is still around).

>>> def lng(lst):
... print(f'*{len(lst)}*')
... return len(lst)
...
>>>
>>> for i in x[ : lng(x) - lng(y) ]:
... i
...
*7*
*3*
1
2
3
4
>>>
>>> for i in x[ : lng(x) - lng(z) ]:
... i
...
*7*
*0*
1
2
3
4
5
6
7
>>>

From this it is clear that y is only computed once, when the loop is
started. Therefore I think it follows that there is no need to
pre-compute y.

Hope this makes sense.

Frank

[dn]

Perfect sense (== @Peter sense).


To confirm:

«8.3. The for statement¶

The for statement is used to iterate over the elements of a sequence
(such as a string, tuple or list) or other iterable object:

for_stmt ::= "for" target_list "in" expression_list ":" suite
["else" ":" suite]

The expression list is evaluated once; it should yield an iterable
object. An iterator is created for the result of the expression_list.
The suite is then executed once for each item provided by the iterator,
in the order returned by the iterator. Each item in turn is assigned to
the target list using the standard rules for assignments (see Assignment
statements), and then the suite is executed. When the items are
exhausted (which is immediately when the sequence is empty or an
iterator raises a StopIteration exception), the suite in the else
clause, if present, is executed, and the loop terminates.
»
https://docs.python.org/3/reference/compound_stmts.html#the-for-statement


That said, I'm wondering if all is strictly true when the
expression_list is a generator, which by definition features
lazy-execution rather than up-front list production. So, things may
depend upon the full-nature of the application.


Assuming, as-per these simple examples, there is no advantage to
pre-computation, it may be more readable to 'prepare' the control-values
separately, thereby simplifying the (appearance of the) for-statement.


The problem at this level, are the terms: "fairly long", "pretty ugly"*,
and "expensive". There are various tactics which might be brought to
bear, but will only be applicable in specific situations. So, too little
information, too much speculation...

For example if there is repetition inside the loop, where a particular
x[ i ] is related to something multiple times - perhaps in multiple
executions of the loop, 'memoisation' may save recalculation -
particularly if this happens inside a nested-loop. This has nothing to
do with the for-loop itself. It trades memory-space for execution-time -
one "expense" for another. However, only you can tell if it, or any
other idea, might be relevant in this application...


* can you make up your mind? Is it pretty, or ugly?
--
Regards,
=dn

[Chris Angelico]

On Sat, Nov 27, 2021 at 7:21 PM dn via Python-list
<pytho...@python.org> wrote:
> The expression list is evaluated once; it should yield an iterable
> object. An iterator is created for the result of the expression_list.
> The suite is then executed once for each item provided by the iterator,
> in the order returned by the iterator. Each item in turn is assigned to
> the target list using the standard rules for assignments (see Assignment
> statements), and then the suite is executed. When the items are
> exhausted (which is immediately when the sequence is empty or an
> iterator raises a StopIteration exception), the suite in the else
> clause, if present, is executed, and the loop terminates.
> »
> https://docs.python.org/3/reference/compound_stmts.html#the-for-statement
>
>
> That said, I'm wondering if all is strictly true when the
> expression_list is a generator, which by definition features
> lazy-execution rather than up-front list production. So, things may
> depend upon the full-nature of the application.

Yes, it is. Evaluating a generator expression gives you a generator
object, which is an iterable.

ChrisA

[dn]

You (and the text) are correct - the expression list is "evaluated once"
and produces a generator object. For a particular understanding of
"evaluate".

However, as described, all that has been "evaluated" is a
generator-object. Unlike (say) a list's iterator, a generator only
eventually produces a 'list' - and each time the generator is
called-upon to yield the next value, that value has to be "evaluated",
ie there's some further evaluation loop-by-loop.

Further, that the values-returned can be amended during the life of the
generator (in ways anticipated and unanticipated by coder and
interpreter alike). Thus, it seems that the "list" doesn't actually
exist, as in, hasn't been "evaluated" as data-values, when the loop is
enacted.

What has been 'evaluated' are the terms by which the looping will
proceed, and terminate.


--
Regards,
=dn

[Chris Angelico]

On Tue, Nov 30, 2021 at 6:45 AM dn via Python-list

That's true of ALL iterators though. If you get a list iterator, and
while you're stepping through it, the underlying list changes, you'll
see those changes. Nothing gets "snapshot" unless you explicitly
request it (eg by slicing the list first).

ChrisA