Allowing non-ASCII identifiers
"Martin v. Löwis"
> I wonder if the proposal would be more palatable if it were restricted
> to 8-bit encodings (what we used to call "code pages"). This is at least
> a first step in the right direction that would help westerners and could
> be made to work even if Python were compiled without Unicode support.
> (it is still possible to compile Python without Unicode isn't it?)
I doubt that it would matter much to those currently opposed; I know
that *I* would be opposed to such a strategy: Allowing arbitrary source
code encoding is no technical challenge whatsoever, and restricting
it to single-byte encodings is an arbitrary restriction.
I believe Guido's concern is more along the lines "How do I call a
function that has a ł in its name, or a Σ?", or, even, "How can I
find out what the function does, by looking at its name and doc
string, if that is in Polish or Greek?" The fact that there is
a single-byte encoding for either character doesn't really help
here.
So this is about social issues, coding policies, guidelines, etc -
not about technical issues.
Regards,
Martin
Paul Prescod
> Paul Prescod wrote:
>
>> I wonder if the proposal would be more palatable if it were restricted
>> to 8-bit encodings (what we used to call "code pages"). This is at
>> least a first step in the right direction that would help westerners
>> and could be made to work even if Python were compiled without Unicode
>> support. (it is still possible to compile Python without Unicode isn't
>> it?)
>
>
> I doubt that it would matter much to those currently opposed; I know
> that *I* would be opposed to such a strategy: Allowing arbitrary source
> code encoding is no technical challenge whatsoever, and restricting
> it to single-byte encodings is an arbitrary restriction.
You are right. Re-reading Guido's complaint I understand what you mean.
But I have heard the argument in the past that Unicode source files
would break introspection tools. If that isn't a concern this time
around then disregard my suggestion.
Paul Prescod
John Roth
"Paul Prescod" <pa...@prescod.net> wrote in message
news:mailman.1391.1076369...@python.org...
Martin v. Löwis wrote:
[JR]
I believe that unicode (actually UTF-8) source code files
are legitimate if you declare them properly in the encoding
line. In fact, UTF-8 is the example in the documentation.
I'm all in favor of going to unicode all the way. I'd like to
have the proper mathematical symbols for logical and set
operations, as well as integer divide. They're all there in the
unicode character set, after all; why should we have to
settle for archaic character restrictions?
John Roth
[/JR]
Paul Prescod
AdSR
> I'm all in favor of going to unicode all the way. I'd like to
> have the proper mathematical symbols for logical and set
> operations, as well as integer divide. They're all there in the
> unicode character set, after all; why should we have to
> settle for archaic character restrictions?
Java allows for Unicode identifiers and I'm yet to see a single source
file that uses anything but ASCII. Actually, so far I have only seen
non-ASCII in Polish Logo many years ago, and that was only for
educational purposes.
As a non-native English speaker, coming from Polish and Portuguese
background, I could argue in favor of non-ASCII identifiers, but I'm
against them. Do we really need those? Even if program output is in
Polish, all my code is "identified" and commented in English, which I
think of as of a good habit. (With exception of HTML, where comments
are closely related to content.)
I don't have any _really_ solid reasons against Unicode identifiers,
except for simplicity. It's just the way I feel about programming.
On a side note, one place where I think non-ASCII really should be
avoided are domain names, something that is being much debated
recently.
AdSR
Michael Hudson
> On a side note, one place where I think non-ASCII really should be
> avoided are domain names, something that is being much debated
> recently.
And something Python supports already :-)
Cheers,
mwh
--
Windows XP: Big cow. Stands there, not especially malevolent
but constantly crapping on your carpet. Eventually you have to
open a window to let the crap out or you die.
-- Jim's pedigree of operating systems, asr
Scott David Daniels
> ...
> I believe that unicode (actually UTF-8) source code files
> are legitimate if you declare them properly in the encoding
> line. In fact, UTF-8 is the example in the documentation.
>
> I'm all in favor of going to unicode all the way. I'd like to
> have the proper mathematical symbols for logical and set
> operations, as well as integer divide. They're all there in the
> unicode character set, after all; why should we have to
> settle for archaic character restrictions?
incapable of displaying such symbols. Never mind whether we
can read them.
Also, we would have to solve the issue of multiple representations
for the same identifier (normalized identifiers)? There are four
equivalent representations:
(u'\N{Latin small letter e with acute}l'
u'\N{Latin small letter e with grave}ve')
(u'\N{Latin small letter e with acute}l'
u'e\N{Combining grave accent}ve')
(u'e\N{Combining acute accent}l'
u'\N{Latin small letter e with grave}ve')
(u'e\N{Combining acute accent}l'
u'e\N{Combining grave accent}ve')
Unicode says we should treat these four identically. Further,
they each have a distinct hash code, so a dictionary will not
necessarily even try to compare them to find them equal.
--
-Scott David Daniels
Scott....@Acm.Org
"Martin v. Löwis"
> You are right. Re-reading Guido's complaint I understand what you mean.
> But I have heard the argument in the past that Unicode source files
> would break introspection tools. If that isn't a concern this time
> around then disregard my suggestion.
That might be a problem, indeed. OTOH, those tools likely also
break if you use non-ASCII byte strings for identifiers.
Regards,
Martin
Dietrich Epp
On Feb 10, 2004, at 8:59 AM, Scott David Daniels wrote:
> Also, we would have to solve the issue of multiple representations
> for the same identifier (normalized identifiers)? There are four
> equivalent representations:
>
> (u'\N{Latin small letter e with acute}l'
> u'\N{Latin small letter e with grave}ve')
>
> (u'\N{Latin small letter e with acute}l'
> u'e\N{Combining grave accent}ve')
>
> (u'e\N{Combining acute accent}l'
> u'\N{Latin small letter e with grave}ve')
>
> (u'e\N{Combining acute accent}l'
> u'e\N{Combining grave accent}ve')
>
> Unicode says we should treat these four identically. Further,
> they each have a distinct hash code, so a dictionary will not
> necessarily even try to compare them to find them equal.
You could require that all identifiers be the canonically decomposed
Unicode representations encoded into UTF-8. This would mean that no
matter which string is chosen from the above, the result is always the
same sequence of characters. This is how many filesystems use unicode,
i.e., Mac HFS+ works this way (but filesystems usually also require a
specific version of Unicode for backwards compatibility).
I personally think that Unicode identifiers would be catastrophic.
With Unicode on the web, if you can't represent some characters, you
can't read the web page. With programming, it could mean that you are
unable to use a particular module, altering the functionality for
people who can't enter certain codes. There is also the issue of which
characters to allow, because some characters look like numbers. Is
unicode 'IV' a number or an identifier? What about a circled 4? What
about unicode line breaks and paragraph breaks? What about opening and
closing quote marks? What about right-to-left characters? What about
ligatures? Non-breaking spaces? Function application?
I think the assumption some people have is that Unicode will only ever
be used for things that are like the roman alphabet: adding diacritical
marks, etc. It sounds like the most worthless extension ever, and the
only language I think of when I think of special characters is
Intercal.
Scott David Daniels
> You could require that all identifiers be the canonically decomposed
> Unicode representations encoded into UTF-8. This would mean that no
> matter which string is chosen from the above, the result is always the
> same sequence of characters. This is how many filesystems use unicode,
> i.e., Mac HFS+ works this way (but filesystems usually also require a
> specific version of Unicode for backwards compatibility).
There are several "Normal forms" for Unicode letters. You'd need to
choose one.
> I personally think that Unicode identifiers would be catastrophic.....
{lotsa examples, some good, some not-so-good elided)
I'm reluctant to endorse it because I _know_ I'll see "Why doesn't my
program work?" accompanied by characters I'm not used to distinguishing.
> I think the assumption some people have is that Unicode will only ever
> be used for things that are like the roman alphabet: adding diacritical
> marks, etc. It sounds like the most worthless extension ever, and the
> only language I think of when I think of special characters is Intercal.
And this is why I had to comment. You obviously never dealt with APL.
I actually used it without an APL type ball, which was painful in the
extreme. When I give language summaries, my quote for APL is,
"APL is the only language where you regularly see one programmer walk
into another's office (well, cube now, but in the day....) and say,
'I bet you cannot guess what this one-line program does.'"
"Martin v. Löwis"
> Because some of us use archaic systems and/or fonts which are
> incapable of displaying such symbols. Never mind whether we
> can read them.
Right. However, policy whether to use non-ASCII identifiers
because of such issues should be with the source code authors,
not with the language implementation. Being able to use non-ASCII
identifiers does not mean you *have* to; not being able means
you *cannot*.
> Also, we would have to solve the issue of multiple representations
> for the same identifier (normalized identifiers)?
I would use NFC, because it has the best chances of being displayed
properly even on terminals that don't do combining characters.
For the language itself, the specific choice of normalization form
is irrelevant - any form would do (but I agree that normalization
should happen).
> Unicode says we should treat these four identically. Further,
> they each have a distinct hash code, so a dictionary will not
> necessarily even try to compare them to find them equal.
If identifiers are Unicode-normalized, this is not an issue -
all copies of the normal form will hash identical.
Regards,
Martin
"Martin v. Löwis"
> You could require that all identifiers be the canonically decomposed
> Unicode representations encoded into UTF-8.
That would be unpythonic: non-ASCII identifiers should be represented
as Unicode objects, not as UTF-8 byte strings.
> I personally think that Unicode identifiers would be catastrophic. With
> Unicode on the web, if you can't represent some characters, you can't
> read the web page. With programming, it could mean that you are unable
> to use a particular module, altering the functionality for people who
> can't enter certain codes.
It is the case that some people would have problems invoking certain
functions. Why would that be a catastrophy? Authors of Python software
should make a choice whether they prefer readability of the source code,
or accessibility to everyone. Depending on the situation, one choice
or the other may be appropriate. Python should not police that decision
for the developer.
> There is also the issue of which characters
> to allow, because some characters look like numbers.
Yes. I would go with a list similar to the Java one, except with a
few obvious restrictions (e.g. disallow currency symbols: Python
does not allow the DOLLAR SIGN in identifiers, whereas Java does).
> Is unicode 'IV' a number or an identifier?
It is certainly *not* a number. I propose to change the syntax of
identifiers, not of numbers. Whether this specific character Ⅳ is
an identifier or should give a syntax error is a choice one needs
to make, certainly. What would be your choice?
> What about a circled 4? What about unicode
> line breaks and paragraph breaks? What about opening and closing quote
> marks? What about right-to-left characters? What about ligatures?
> Non-breaking spaces? Function application?
The Unicode consortium gives guidance on all these questions. As I said,
I would closely follow the Java principles, which were derived from
the Unicode consortium guidance. Here is my proposal:
Legal non-ASCII identifiers are what legal non-ASCII
identifiers are in Java, except that Python may use
a different version of the Unicode character database.
Python would share the property that future versions
allow more characters in identifiers than older versions.
If you are too lazy too look up the Java definition,
here is a rough overview:
An identifier is "JavaLetter JavaLetterOrDigit*"
JavaLetter is a character of the classes Lu, Ll,
Lt, Lm, or Lo, or a currency symbol (for Python:
excluding $), or a connecting punctuation character
(which is unfortunately underspecified - will
research the implementation).
JavaLetterOrDigit is a JavaLetter, or a digit,
a numeric letter, a combining mark, a non-spacing
mark, or an ignorable control character.
I believe this specification allows you to answer your questions
yourself.
> I think the assumption some people have is that Unicode will only ever
> be used for things that are like the roman alphabet: adding diacritical
> marks, etc. It sounds like the most worthless extension ever, and the
> only language I think of when I think of special characters is Intercal.
That is certainly not my assumption. Instead, I expect that this
extension will primarily be used by developers whose native language
is Russian, Japanese, Chinese, Korean, or Arabic. Atleast, I've heard
developers from these cultures ask for the specific feature in the
past (I've also heard French and German people ask for the feature,
but that fits with your expectation).
Regards,
Martin
Neil Hodgson
> Because some of us use archaic systems and/or fonts which are
> incapable of displaying such symbols. Never mind whether we
> can read them.
For such circumstances, I would like to see hex escape sequences allowed
in identifiers as in Java. That means that there is a representation of last
resort that can be used by those using less capable tools. A simple filter
could translate to and from this format for the extremely rare occasions it
would be needed.
Neil
Joe Mason
> It is the case that some people would have problems invoking certain
> functions. Why would that be a catastrophy?
Oh, it wouldn't be. Not being catastrophic doesn't make it good.
> Authors of Python software should make a choice whether they prefer
> readability of the source code, or accessibility to everyone.
Yeah, they should, but they won't. They'll go nuts with the cool
features and not stop to think about the consequences. Those of us
stuck cleaning up after them will then be hindered by the cool features
that don't work. History has shown us this.
If non-ASCII characters are allowed, they'll be used frivolously.
Somebody will put "et tu, Bruté" in a comment, or start their career
planning package with "import resumé", and these otherwise working
programs would break for people without Unicode support.
> Python should not police that decision for the developer.
Why not? It polices everything else. Isn't Python still the "only one
way to do it" language?
If you were suggesting this for Perl or Ruby, I'd be all in favour (in fact,
it'd be especially apropriate for Ruby). But in Python it's perfectly
appropriate to restrict something that many people would find useful in
favour of simplicity and consistency.
Joe
Paul Prescod
>
> I personally think that Unicode identifiers would be catastrophic.
This is an overstatement. One of the great things about Python is that
it borrows from other langauges. VB and C# for sure and I think Java
allow non-ASCII identifiers and there was no catastrophe. VB has its
problems but Unicode identifiers is not a big one.
I am +0 on this proposal because I really doubt it will cause me big
problems and at least some foreign language speakers claim it will make
their lives much easier. If they post to c.l.py asking for help with
code I can't read I'll tell them I can't read it. If they write
extension modules I can't use I'll just ask them to put an ASCII API
alongside their Unicode one (language is likely to be a bigger
readability problem than encoding anyhow)
Paul Prescod
Brian Quinlan
> it borrows from other langauges. VB and C# for sure and I think Java
> allow non-ASCII identifiers and there was no catastrophe. VB has its
> problems but Unicode identifiers is not a big one.
I think that Python should borrow the C# identifier syntax verbatim. That
way we can have code that looks like this:
class @class:
def @def(@def):
pass
cl\u0061ss.d\u0065f(true)
Cheers,
Brian
Neil Hodgson
> I think that Python should borrow the C# identifier syntax verbatim. That
> way we can have code that looks like this:
>
> class @class:
> def @def(@def):
> pass
>
> cl\u0061ss.d\u0065f(true)
While ugly, the ability to access identifiers defined with other
languages which are keywords in Python is useful. Unicode identifiers will
also interoperation with other languages in environments which allow Unicode
identifiers such as .NET, Java and COM. If Unicode identifiers are not
possible in Python then there is a need for an mechanism to associate a
valid Python identifier with the real Unicode name.
Neil
"Martin v. Löwis"
> If non-ASCII characters are allowed, they'll be used frivolously.
> Somebody will put "et tu, Bruté" in a comment
People can (and do) already put their natural language into comments;
whether or not non-ASCII characters are allowed in identifiers is
irrelevant for that usage.
Also, people don't need "Unicode support" to read those comments.
They just need an editor that can display the character set that
the people wrote their comments in.
Assuming you speak the language in which the comments are written,
you very likely have a text editor which can display them. Or you
use IDLE.
>>Python should not police that decision for the developer.
>
>
> Why not? It polices everything else. Isn't Python still the "only one
> way to do it" language?
And that wouldn't change: There would be only a single way to do
import resumé
Currently, there is no way, which is less than "only one way".
> If you were suggesting this for Perl or Ruby, I'd be all in favour (in fact,
> it'd be especially apropriate for Ruby). But in Python it's perfectly
> appropriate to restrict something that many people would find useful in
> favour of simplicity and consistency.
And indeed, using non-ASCII characters in identifiers is simple and
consistent.
Regards,
Martin
"Martin v. Löwis"
> I think that Python should borrow the C# identifier syntax verbatim. That
> way we can have code that looks like this:
>
> class @class:
> def @def(@def):
> pass
>
> cl\u0061ss.d\u0065f(true)
If I ever get to write a PEP on this, I'll happily copy the second
syntax. I'm not sure I like the first one, and there are plenty
conventions to get around the restriction (e.g. in Tkinter, you add
an underscore if a Tcl property is a Python keyword).
Regards,
Martin
Paul Prescod
>>This is an overstatement. One of the great things about Python is that
>>it borrows from other langauges. VB and C# for sure and I think Java
>>allow non-ASCII identifiers and there was no catastrophe. VB has its
>>problems but Unicode identifiers is not a big one.
>
>
> I think that Python should borrow the C# identifier syntax verbatim. That
> way we can have code that looks like this:
>
> class @class:
> def @def(@def):
> pass
>
> cl\u0061ss.d\u0065f(true)
Have you ever seen real code like that? If not, what are you worried
about? That C# programmers are reasonable but Python programmers are
devious and will go to extra effort to make your life difficult?
Paul Prescod
Brian Quinlan
> > class @class:
> > def @def(@def):
> > pass
> >
> > cl\u0061ss.d\u0065f(true)
>
> Have you ever seen real code like that?
I've never seen any non-ASCII code in any language.
> If not, what are you worried about? That C# programmers are reasonable
> but Python programmers are devious and will go to extra effort to make
> your life difficult?
I don't think that there is any reasonable usage of syntax like that, so why
have it in the language?
Cheers,
Brian
Paul Prescod
>>Have you ever seen real code like that?
>
> I've never seen any non-ASCII code in any language.
So other programming languages make non-ASCII characters possible but
English-speakers do not end up paying any price. So what's the problem?
Panic in the Python world strikes me as just FUD.
>>If not, what are you worried about? That C# programmers are reasonable
>>but Python programmers are devious and will go to extra effort to make
>>your life difficult?
>
>
> I don't think that there is any reasonable usage of syntax like that, so why
> have it in the language?
We are getting a little off track. Nobody really proposed to borrow C#'s
identifier syntax whole hog (i.e have a keyword-escaping mechanism). I
proposed to learn from C# and Java. They allow non-ASCII keywords and
nobody seems to be hurt by it.
As far as "reasonable usage" of non-English identifiers. Non-English
speakers tell me that this is reasonable and I trust them to understand
their situation better than I do!
But as an aside, the idea of allowing keywords to also be identifiers
(with appropriate escaping syntax) is not a bad ones.
Let's say you are calling an XML-RPC server that uses a method name that
happens to also be a Python keyword (e.g. "yield"). Of course your
XML-RPC API could have a specific way of working around this but knowing
the way the Python world works, it will use a different convention than
your COM API which will use a different convention than your SOAP API
which will use a different convention than your PyObjectiveC API, etc.
Why not have the language standardize?
Paul Prescod
Brian Quinlan
> English-speakers do not end up paying any price. So what's the problem?
If we ever need to read any Python code that uses non-ASCII characters then
we pay a price.
> Panic in the Python world strikes me as just FUD.
I'm not panicking; I'm just arguing that expending considerable effort to
add a feature that may cause problems might not be a good idea. I don't have
a very strong opinion on this issue.
> We are getting a little off track. Nobody really proposed to borrow C#'s
> identifier syntax whole hog (i.e have a keyword-escaping mechanism). I
> proposed to learn from C# and Java. They allow non-ASCII keywords and
> nobody seems to be hurt by it.
Did you mean to say that they "allow non-ASCII keywords"? Assuming you meant
identifiers, I don't find that argument convincing because the lack of
widespread problems might be related to the lack of widespread usage.
> As far as "reasonable usage" of non-English identifiers. Non-English
> speakers tell me that this is reasonable and I trust them to understand
> their situation better than I do!
This is a more convincing argument.
> But as an aside, the idea of allowing keywords to also be identifiers
> (with appropriate escaping syntax) is not a bad ones.
But it isn't really necessary in Python since defining/calling a method
using a string is trivial. In C# it is a lot of work.
> Let's say you are calling an XML-RPC server that uses a method name that
> happens to also be a Python keyword (e.g. "yield"). Of course your
> XML-RPC API could have a specific way of working around this but knowing
> the way the Python world works, it will use a different convention than
> your COM API which will use a different convention than your SOAP API
> which will use a different convention than your PyObjectiveC API, etc.
> Why not have the language standardize?
I'm not sure that the API necessarily needs to know anything about this
problem. The API user just has to use the usual Python mechanisms to
work-around the issue e.g.
someComObject.getattr('yield')(2,3)
someRPCServer.getattr('yield')(2,3)
class myComObject:
def _yield(self, x, y):
setattr(myComObject, "yield", myComObject._yield)
Cheers,
Brian
Rainer Deyke
> But as an aside, the idea of allowing keywords to also be identifiers
> (with appropriate escaping syntax) is not a bad ones.
You can already have attributes with keywords as names.
>>> class C(object): pass
...
>>> c = C()
>>> setattr(c, "class", 5)
>>> getattr(c, "class")
5
>>>
--
Rainer Deyke - rai...@eldwood.com - http://eldwood.com
Dan Bishop
> Paul Prescod wrote:
> > > class @class:
> > > def @def(@def):
> > > pass
> > >
> > > cl\u0061ss.d\u0065f(true)
> >
> > Have you ever seen real code like that?
>
> I've never seen any non-ASCII code in any language.
The code posted above is all-ASCII.
> > If not, what are you worried about? That C# programmers are reasonable
>
> > but Python programmers are devious and will go to extra effort to make
> > your life difficult?
>
> I don't think that there is any reasonable usage of syntax like that, so
> why have it in the language?
C# was designed to work with classes written in other languages, which
have different sets of keywords. Therefore, C#'s needed to deal with
questions like "How do you use a class named 'operator' (which would
be a valid identifier in VB.NET but not in C#)?", and the "@operator"
construct was a very reasonable solution.
I don't see any need of it in CPython, however.
Paul Prescod
>>So other programming languages make non-ASCII characters possible but
>>English-speakers do not end up paying any price. So what's the problem?
>
>
> If we ever need to read any Python code that uses non-ASCII characters then
> we pay a price.
We're going around in circles. In languages where non-ASCII identifiers
are allowed you say that you've never once had a problem with them. You
spend a lot of time in one such language and I've spent a lot of time in
other such languages. Why would it be a problem for Python if it is not
for them?
> ... I don't find that argument convincing because the lack of
> widespread problems might be related to the lack of widespread usage.
Some people claim (emphatically) that they would use it and also say
that they would use it responsibly in a way that doesn't inconvenience
other users. Perhaps this is what happens in the C#, Java, etc. worlds.
I'm not sure that the API necessarily needs to know anything about this
> problem. The API user just has to use the usual Python mechanisms to
> work-around the issue e.g.
>
> someComObject.getattr('yield')(2,3)
> someRPCServer.getattr('yield')(2,3)
I think you mean getattr(someComObject('yield'))(2,3,)
It's a little verbose but I agree that it is probably a sufficient way
to work around the problem.
Paul Prescod
Brian Quinlan
> > someRPCServer.getattr('yield')(2,3)
>
> I think you mean getattr(someComObject('yield'))(2,3,)
We both need a Python refresher :-)
getattr(someComObject, 'yield')(2,3)
Cheers,
Brian