Not x.islower() has different output than x.isupper() in list output...

[Christopher Reimer]

Greetings,

I was playing around with a piece of code to remove lowercase letters
and leave behind uppercase letters from a string when I got unexpected
results.

string = 'Whiskey Tango Foxtrot'

list(filter((lambda x: not x.islower()), string))

['W', ' ', 'T', ' ', 'F']

Note the space characters in the list.

list(filter((lambda x: x.isupper()), string))

['W', 'T', 'F']

Note that there are no space characters in the list.

Shouldn't the results between 'not x.islower()' and 'x.isupper()' be
identical?

The final form of this code is this:

list(filter(str.isupper, string))

['W', 'T', 'F']

Thank you,

Chris R.

[Stephen Hansen]

On Fri, Apr 29, 2016, at 06:17 PM, Christopher Reimer wrote:
> Greetings,
>
> I was playing around with a piece of code to remove lowercase letters
> and leave behind uppercase letters from a string when I got unexpected
> results.
>
> string = 'Whiskey Tango Foxtrot'
>
> list(filter((lambda x: not x.islower()), string))
>
> ['W', ' ', 'T', ' ', 'F']
>
> Note the space characters in the list.
>
> list(filter((lambda x: x.isupper()), string))
>
> ['W', 'T', 'F']
>
> Note that there are no space characters in the list.
>

> Shouldn't the results between 'not x.islower()' and 'x.isupper()' be
> identical?

Absolutely not. There are characters which are neither uppercase or
lowercase: specifically, whitespace in your situation.

"x.islower()" is true for everything that is lowercase, and false for
everything else. (Including things which have no case). If you invert
with not, then its false for everything that is lowercase, but true for
everything else: including things which have no case.

"x.isupper()" is true for everything that is uppercase, and false for
everything else. This is therefore a narrower test then "not
x.islower()"

x.isupper() is a narrower, more specific test then "not x.islower()". If
you used "not x.isupper()" you'd get the whitespace in it too (along
with all the lower case characters).

If isupper/islower were perfect opposites of each-other, there'd be no
need for both. But since characters can be upper, lower, or *neither*,
you run into this situation.

--
Stephen Hansen
m e @ i x o k a i . i o

[Christopher Reimer]

On 4/29/2016 6:29 PM, Stephen Hansen wrote:
> If isupper/islower were perfect opposites of each-other, there'd be no
> need for both. But since characters can be upper, lower, or *neither*,
> you run into this situation.

Based upon the official documentation, I was expecting perfect opposites.

str.islower(): "Return true if all cased characters [4] in the string
are lowercase and there is at least one cased character, false otherwise."

https://docs.python.org/3/library/stdtypes.html?highlight=islower#str.islower

str.isupper(): "Return true if all cased characters [4] in the string
are uppercase and there is at least one cased character, false otherwise."

https://docs.python.org/3/library/stdtypes.html?highlight=isupper#str.isupper

Here's the footnote that may or not be relevant to this discussion: "[4]
Cased characters are those with general category property being one of
“Lu” (Letter, uppercase), “Ll” (Letter, lowercase), or “Lt” (Letter,
titlecase)."

A bug in the docs?

Thank you,

Chris R.

[Chris Angelico]

>>> def case(ch):
... return (
... ("Lower " if ch.islower() else "") +
... ("Upper " if ch.isupper() else "") +
... unicodedata.category(ch)
... )

Both functions require that there be at least one cased character:

>>> case("@")
'Po'

And then each one requires a specific character class:

>>> case("a")
'Lower Ll'
>>> case("A")
'Upper Lu'

Other classes don't count:

>>> case("\u01C5")
'Lt'

There are three categories of "cased characters". Both functions check
for exactly one of those categories. Thus they are not opposites.

ChrisA

[Gregory Ewing]

Christopher Reimer wrote:

> str.islower(): "Return true if all cased characters [4] in the string
> are lowercase and there is at least one cased character, false otherwise."
>

> str.isupper(): "Return true if all cased characters [4] in the string
> are uppercase and there is at least one cased character, false otherwise."

A string consisting of a single space doesn't contain any
cased characters, so both islower(" ") and isupper(" ")
return false according to these rules. The docs are correct.

--
Greg

[Stephen Hansen]

On Fri, Apr 29, 2016, at 06:55 PM, Christopher Reimer wrote:
> On 4/29/2016 6:29 PM, Stephen Hansen wrote:
> > If isupper/islower were perfect opposites of each-other, there'd be no
> > need for both. But since characters can be upper, lower, or *neither*,
> > you run into this situation.
>
> Based upon the official documentation, I was expecting perfect opposites.
>

> str.islower(): "Return true if all cased characters [4] in the string
> are lowercase and there is at least one cased character, false
> otherwise."

The thing is, your use of filter is passing a string of 1 character long
to your function, and so when it comes up against a string " ", there
are no cased characters. Therefore, false.

The documentation holds true.

Your use of filter is breaking "Whiskey Tango Foxtrot" into ["W", "h",
"i", ... "o" t"] and calling not x.islower() vs x.isupper() on each.

When it comes up against " ", it doesn't pass the documented definition
of what islower defines.

The official documentation is accurate.

[Christopher Reimer]

On 4/29/2016 11:43 PM, Stephen Hansen wrote:
> The official documentation is accurate.

That may be true on a technical level. But the identically worded text
in the documentation implies otherwise. Maybe I'm nitpicking this. Even
if I submitted a bug to request a clearer explanation in the
documentation, I doubt it would get change. The programmer still has an
obligation to test the code to make sure that it works as expected,
which was what I did that until I found the concise statement that
worked exactly as I wanted it to.

Thank you,

Chris R.

[Chris Angelico]

Let's take it away from letter case and to something that everyone
should be able to grok.

isalpha(...) method of builtins.str instance
S.isalpha() -> bool

Return True if all characters in S are alphabetic
and there is at least one character in S, False otherwise.

isdigit(...) method of builtins.str instance
S.isdigit() -> bool

Return True if all characters in S are digits
and there is at least one character in S, False otherwise.

It should be pretty obvious that there are some strings which are
neither alphabetic nor digits. Thus these two are not opposites,
despite the identical wording in the docs.

It's exactly the same with letter case. Some strings are neither
uppercase nor lowercase, and thus they'll be False on both checks.

ChrisA

[Stephen Hansen]

On Sat, Apr 30, 2016, at 09:48 AM, Christopher Reimer wrote:
> On 4/29/2016 11:43 PM, Stephen Hansen wrote:
> > The official documentation is accurate.
>
> That may be true on a technical level. But the identically worded text
> in the documentation implies otherwise.

That's the thing -- no it doesn't. The documentation is very specific:
if all cased characters in the string are lowercase, and there's at
least one cased character.

It only seems to because you're packing a loop and test on substrings
into an operation.

list(filter((lambda x: not x.islower()), string))

Let's unpack that. This is basically what you're doing:

result = []
for ch in string:
if not ch.islower():
result.append(ch)

You're thinking of the whole "string", but you're operating on
single-character substrings, and when " ".islower() is run, its false.
Because the two-pronged test, a) if all cased characters are lowercase
and b) there is at least one cased character. b) is failing. Ergo,
you're getting the underscores.

[Christopher Reimer]

On 4/30/2016 10:11 AM, Stephen Hansen wrote:
> You're thinking of the whole "string", but you're operating on
> single-character substrings, and when " ".islower() is run, its false.
> Because the two-pronged test, a) if all cased characters are lowercase
> and b) there is at least one cased character. b) is failing. Ergo,
> you're getting the underscores.

I see where the problem lies in my thinking. I went looking for a single
line solution for the whole string. If I had constructed a for loop and
tried to reduce it to a single line, I *may* have understood the
relationship between the different parts. Or maybe not. Blaming the
documentation is a lot easier. ;)

Thank you,

Chris R.

[pavlove...@gmail.com]

On Friday, April 29, 2016 at 6:55:56 PM UTC-7, Christopher Reimer wrote:
> On 4/29/2016 6:29 PM, Stephen Hansen wrote:
> > If isupper/islower were perfect opposites of each-other, there'd be no
> > need for both. But since characters can be upper, lower, or *neither*,
> > you run into this situation.
>
> Based upon the official documentation, I was expecting perfect opposites.
>
> str.islower(): "Return true if all cased characters [4] in the string
> are lowercase and there is at least one cased character, false otherwise."
>
> https://docs.python.org/3/library/stdtypes.html?highlight=islower#str.islower
>
> str.isupper(): "Return true if all cased characters [4] in the string
> are uppercase and there is at least one cased character, false otherwise."
>
> https://docs.python.org/3/library/stdtypes.html?highlight=isupper#str.isupper

Just to take this discussion in a more pure logic direction. What you call perfect opposites (that is, the functions being negations of each other) is not what the similar wording in the documentation actually implies: you shouldn't have been expecting that.

What you should have been expecting is a symmetry. Say you have a string G. islower(G) will return a certain result. Now take every letter in G and swap the case, and call that string g. isupper(g) will always return the same result is islower(G).

More succinctly, for any string x, the following is always ture:

islower(x) == isupper(swapcase(x))

But that is not the same thing, and does not imply, as the following identity (which it turns out is not always true, as we've seen):

islower(x) == not isupper(x)


Another example of functions that behave like this are ispositive and isnegative. The identity "ispositive(x) == isnegative(-x)" is always true. However, "ispositive(x) == not isnegative(x)" is false if x == 0.


However, I can understand your confusion, because there are some pairs of functions where both identities are true, and if you've seen a few of them it's fairly easy for your intuition to overgeneralize a bit. An example I can think of offhand is iseven(x) and isodd(x), for any integer x. The identities "iseven(x) == isodd(x^1)" and "iseven(x) == not isodd(x)" are both always true.


Carl Banks

[Chris Angelico]

On Tue, May 3, 2016 at 8:00 PM, <pavlove...@gmail.com> wrote:
>
> What you should have been expecting is a symmetry. Say you have a string G. islower(G) will return a certain result. Now take every letter in G and swap the case, and call that string g. isupper(g) will always return the same result is islower(G).
>
> More succinctly, for any string x, the following is always ture:
>
> islower(x) == isupper(swapcase(x))
>
> But that is not the same thing, and does not imply, as the following identity (which it turns out is not always true, as we've seen):
>
> islower(x) == not isupper(x)
>
>
> Another example of functions that behave like this are ispositive and isnegative. The identity "ispositive(x) == isnegative(-x)" is always true. However, "ispositive(x) == not isnegative(x)" is false if x == 0.
>

This assumes, of course, that there is a function swapcase which can
return a string with case inverted. I'm not sure such a function
exists.

ChrisA

[Jussi Piitulainen]

Chris Angelico writes:

> This assumes, of course, that there is a function swapcase which can
> return a string with case inverted. I'm not sure such a function
> exists.

str.swapcase("foO")
'FOo'

[Chris Angelico]

I suppose for this discussion it doesn't matter if it's imperfect.

>>> "\N{ANGSTROM SIGN}".swapcase().swapcase() == "\N{ANGSTROM SIGN}"
False
>>> "\N{LATIN SMALL LETTER SHARP S}".swapcase().swapcase()
'ss'

But drawing the analogy with the negation of real numbers implies
something that doesn't exist.

ChrisA

[DFS]

On 5/3/2016 8:00 AM, Chris Angelico wrote:
> On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen
> <jussi.pi...@helsinki.fi> wrote:
>> Chris Angelico writes:
>>
>>> This assumes, of course, that there is a function swapcase which can
>>> return a string with case inverted. I'm not sure such a function
>>> exists.
>>
>> str.swapcase("foO")
>> 'FOo'
>
> I suppose for this discussion it doesn't matter if it's imperfect.

What was imperfect?

[Chris Angelico]

It doesn't invert, the way numeric negation does. And if you try to
define exactly what it does, you'll come right back to
isupper()/islower(), so it's not much help in defining those.

ChrisA

[DFS]

On 5/3/2016 9:13 AM, Chris Angelico wrote:
> On Tue, May 3, 2016 at 11:01 PM, DFS <nos...@dfs.com> wrote:
>> On 5/3/2016 8:00 AM, Chris Angelico wrote:
>>>
>>> On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen
>>> <jussi.pi...@helsinki.fi> wrote:
>>>>
>>>> Chris Angelico writes:
>>>>
>>>>> This assumes, of course, that there is a function swapcase which can
>>>>> return a string with case inverted. I'm not sure such a function
>>>>> exists.
>>>>
>>>>
>>>> str.swapcase("foO")
>>>> 'FOo'
>>>
>>>
>>> I suppose for this discussion it doesn't matter if it's imperfect.
>>
>>
>>
>> What was imperfect?
>
> It doesn't invert, the way numeric negation does.

What do you mean by 'case inverted'?

It looks like it swaps the case correctly between upper and lower.

[Chris Angelico]

On Tue, May 3, 2016 at 11:19 PM, DFS <nos...@dfs.com> wrote:
> What do you mean by 'case inverted'?
>
> It looks like it swaps the case correctly between upper and lower.

I gave two examples in my previous post. Did you read them? You
trimmed them from the quote.

ChrisA

[Jussi Piitulainen]

Chris Angelico writes:

> On Tue, May 3, 2016 at 9:25 PM, Jussi Piitulainen wrote:
>> Chris Angelico writes:
>>
>>> This assumes, of course, that there is a function swapcase which can
>>> return a string with case inverted. I'm not sure such a function
>>> exists.
>>
>> str.swapcase("foO")
>> 'FOo'
>
> I suppose for this discussion it doesn't matter if it's imperfect.

Not sure. I may have misunderstood what this discussion is about - I
thought you had forgotten that Python has this function :)

>>>> "\N{ANGSTROM SIGN}".swapcase().swapcase() == "\N{ANGSTROM SIGN}"
> False
>>>> "\N{LATIN SMALL LETTER SHARP S}".swapcase().swapcase()
> 'ss'
>
> But drawing the analogy with the negation of real numbers implies
> something that doesn't exist.

I missed the relevance of that analogy.

Ceterum censeo, the only suggested use for .swapcase I've ever heard of
is encryption.

[Jussi Piitulainen]

DFS writes:

> On 5/3/2016 9:13 AM, Chris Angelico wrote:

>> It doesn't invert, the way numeric negation does.
>
> What do you mean by 'case inverted'?
>
> It looks like it swaps the case correctly between upper and lower.

There's letters that do not come in exact pairs of upper and lower case,
so _some_ swaps are not invertible: you swap twice and end up somewhere
else than your starting point.

The "\N{ANSGTROM SIGN}" looks like the Swedish upper-case
a-with-ring-above but isn't the same character, yet Python swaps its
case to the actual lower-case a-with-ring above. It can't go back to
_both_ the Angstrom sign and the actual upper case letter.

(Not sure why the sign is considered a cased letter at all.)

[DFS]

Thanks for the explanation.

Does that mean:

lower(Å) != å ?

and

upper(å) != Å ?

[Jussi Piitulainen]

It means "\N{ANGSTROM SIGN}" != "Å", yet both lower to "å", which then
uppers back to "Å" (U+00c5).

The Ångström sign (U+212b) looks like this: Å. Indistinguishable from Å
in the font that I'm seeing - for all I know, it's the same glyph.

[Grant Edwards]

On 2016-05-03, Jussi Piitulainen <jussi.pi...@helsinki.fi> wrote:

>> Does that mean:
>>
>> lower(Å) != å ?
>>
>> and
>>
>> upper(å) != Å ?
>
> It means "\N{ANGSTROM SIGN}" != "Å", yet both lower to "å", which then
> uppers back to "Å" (U+00c5).
>
> The Ångström sign (U+212b) looks like this: Å. Indistinguishable from Å
> in the font that I'm seeing - for all I know, it's the same glyph.

Interesting. FWIW, Å and Å definitely look different with the terminal
and font I'm using (urxvt with -misc-fixed-medium-r-normal-*-18-120-*-*-*-90-iso10646-*)

Expecting upper/lower operations to be 100% invertible is probably a
ASCII-centric mindset that will falls over as soon as you start
dealing with non-ASCII encodings.

--
Grant Edwards grant.b.edwards Yow! Xerox your lunch
at and file it under "sex
gmail.com offenders"!

[Terry Reedy]

On 5/3/2016 11:42 AM, Grant Edwards wrote:

> Interesting. FWIW, Å and Å definitely look different with the terminal
> and font I'm using (urxvt with -misc-fixed-medium-r-normal-*-18-120-*-*-*-90-iso10646-*)

In the fixed pitch font used by Thunderbird (Courier?), Angstrom Å has
the circle touching the A while letter Å has the circle spaced above.

--
Terry Jan Reedy

[Steven D'Aprano]

On Wed, 4 May 2016 12:42 am, Jussi Piitulainen wrote:

> Ceterum censeo, the only suggested use for .swapcase I've ever heard of
> is encryption.

iF YOU'RE PROGRAMMING AN EDITOR, sWAP cASE IS REALLY USEFUL FOR THOSE LITTLE
capslock ACCIDENTS THAT PLAGUE TYPISTS.



--
Steven

[Steven D'Aprano]

On Wed, 4 May 2016 12:49 am, Jussi Piitulainen wrote:

> DFS writes:
>
>> On 5/3/2016 9:13 AM, Chris Angelico wrote:
>
>>> It doesn't invert, the way numeric negation does.
>>
>> What do you mean by 'case inverted'?
>>
>> It looks like it swaps the case correctly between upper and lower.
>
> There's letters that do not come in exact pairs of upper and lower case,

Languages with two distinct lettercases, like English, are called bicameral.
The two cases are technically called majuscule and minuscule, but
colloquially known as uppercase and lowercase since movable type printers
traditionally used to keep the majuscule letters in a drawer above the
minuscule letters.

Many alphabets are unicameral, that is, they only have a single lettercase.
Examples include Hebrew, Arabic, Hangul, and many others. Georgian is an
interesting example, as it is the only known written alphabet that started
as a bicameral script and then became unicameral.

Consequently, many letters are neither upper nor lower case, and have
Unicode category "Letter other":

py> c = u'\N{ARABIC LETTER FEH}'
py> unicodedata.category(c)
'Lo'
py> c.isalpha()
True
py> c.isupper()
False
py> c.islower()
False


Even among bicameral alphabets, there are a few anomalies. The three most
obvious ones are Greek sigma, German Eszett (or "sharp S") and Turkish I.

(1) The Greek sigma is usually written as Σ or σ in uppercase and lowercase
respectively, but at the end of a word, lowercase sigma is written as ς.

(This final sigma is sometimes called "stigma", but should not be confused
with the archaic Greek letter stigma, which has two cases Ϛ ϛ, at least
when it is not being written as digamma Ϝϝ -- and if you're confused, so
are the Greeks :-)

Python 3.3 correctly handles the sigma/final sigma when upper- and
lowercasing:

py> 'ΘΠΣΤΣ'.lower()
'θπστς'

py> 'ΘΠΣΤΣ'.lower().upper()
'ΘΠΣΤΣ'



(2) The German Eszett ß traditionally existed in only lowercase forms, but
despite the existence of an uppercase form since at least the 19th century,
when the Germans moved away from blackletter to Roman-style letters, the
uppercase form was left out. In recent years, printers in Germany have
started to reintroduce an uppercase version, and the German government have
standardized on its use for placenames, but not other words.

(Aside: in Germany, ß is not considered a distinct letter of the alphabet,
but a ligature of ss; historically it derived from a ligature of ſs, ſz or
ſʒ. The funny characters you may or may not be able to see are the long-S
and round-Z.)

Python follows common, but not universal, German practice for eszett:

py> 'ẞ'.lower()
'ß'
py> 'ß'.upper()
'SS'

Note that this is lossy: given a name like "STRASSER", it is impossible to
tell whether it should be title-cased to "Strasser" or "Straßer". It also
means that uppercasing a string can make it longer.


For more on the uppercase eszett, see:

https://typography.guru/journal/germanys-new-character/
https://typography.guru/journal/how-to-draw-a-capital-sharp-s-r18/


(3) In most Latin alphabets, the lowercase i and j have a "tittle" diacritic
on them, but not the uppercase forms I and J. Turkish and a few other
languages have both I-with-tittle and I-without-tittle.

(As far as I know, there is no language with a dotless J.)

So in Turkish, the correct uppercase to lowercase and back again should go:

Dotless I: I -> ı -> I

Dotted I: İ -> i -> İ

Python does not quite manage to handle this correctly for Turkish
applications, since it loses the dotted/dotless distinction:

py> 'ı'.upper()
'I'
py> 'İ'.lower()
'i'

and further case conversions follow the non-Turkish rules.

Note that sometimes getting this wrong can have serious consequences:

http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two-people-puts-three-more-in-jail



--
Steven

[Gregory Ewing]

Jussi Piitulainen wrote:
> Ceterum censeo, the only suggested use for .swapcase I've ever heard of
> is encryption.

Yep, all the smart terrorists these days are using a
combination of swapcase and rot13. Totally bamboozles
the FBI.

--
Greg

[Steven D'Aprano]

Heh, the Australian government is getting their panties in a twist over the
whole encryption thing, because Apple versus FBI proves that encryption is
evil or something. Which lead to this exchange in the IRC channel we use at
work:

(handles have been removed to anonymize the discussion)

XXXX: newsflash, most criminals are too stupid to use encryption
XXXX: https://en.wikipedia.org/wiki/Mujahedeen_Secrets
XXXX: It's hilarious, they turn off all the "western" ciphers
YYYY: the great thing about Mujahedeen Secrets (apart fromt he crypto
apparently being poor) is the cyphertext is trivially detectable
YYYY: like "-----BEGIN PGP MESSAGE-----" but it's "-----DEATH TO
INFIDELS-----" or something
YYYY: so XKeyscore can just go "this person is using Mujahedeen Secrets" in
their regular traffic matching
ZZZZ: Wait, Mujahedeen Secrets was released anonymously, it has easily
broken encryption and is trivially detectable? Can you say "schmuck bait"? I
think I know who the authors were...



--
Steve

[DFS]

Linguist much?

[Chris Angelico]

On Thu, May 5, 2016 at 12:09 AM, DFS <nos...@dfs.com> wrote:
> On 5/3/2016 11:28 PM, Steven D'Aprano wrote:

>> [ lengthy piece about text, Unicode, and letter case ]
>
> Linguist much?

As an English-only speaker who writes code that needs to be used
around the world, you end up accruing tidbits of language and text
trivia in the form of edge cases that you need to remember to test.
Among them:

* Turkish dotless and dotted i
* Greek medial and final sigma
* German eszett
* Hebrew and Arabic right-to-left text
* Chinese non-BMP characters
* Combining characters (eg diacriticals starting U+0300)
* Non-characters eg U+FFFE

And then a post like Steven's basically comes from pulling up all
those from your memory, and maybe doing a spot of quick testing and/or
research to get some explanatory details. You don't have to be a
linguist, necessarily - just a competent debugger.

ChrisA

[Steven D'Aprano]

On Thu, 5 May 2016 12:09 am, DFS wrote:

> On 5/3/2016 11:28 PM, Steven D'Aprano wrote:

>> Languages with two distinct lettercases, like English, are called
>> bicameral.

[...]

> Linguist much?


Possibly even a cunning one.




Somebody-had-to-say-it-ly y'rs,


--
Steven

[DFS]

On 5/4/2016 11:37 AM, Steven D'Aprano wrote:
> On Thu, 5 May 2016 12:09 am, DFS wrote:
>
>> On 5/3/2016 11:28 PM, Steven D'Aprano wrote:
>
>>> Languages with two distinct lettercases, like English, are called
>>> bicameral.
> [...]
>
>> Linguist much?
>
>
> Possibly even a cunning one.

I see you as more of a Colonel Angus.