Re: Why no warnings when re-assigning builtin names?
Philip Semanchuk
On Aug 15, 2011, at 5:52 PM, Gerrat Rickert wrote:
> With surprising regularity, I see program postings (eg. on
> StackOverflow) from inexperienced Python users accidentally
> re-assigning built-in names.
>
>
>
> For example, they'll innocently call some variable, "list", and assign a
> list of items to it.
>
> ...and if they're _unlucky_ enough, their program may actually work
> (encouraging them to re-use this name in other programs).
Or they'll assign a class instance to 'object', only to cause weird errors later when they use it as a base class.
I agree that this is a problem. The folks on my project who are new-ish to Python overwrite builtins fairly often. Since there's never been any consequence other than my my vague warnings that something bad might happen as a result, it's difficult for them to develop good habits in this regard. It doesn't help that Eclipse (their editor of choice) doesn't seem to provide a way of coloring builtins differently. (That's what I'm told, anyway. I don't use it.)
> If they try to use an actual keyword, both the interpreter and compiler
> are helpful enough to give them a syntax error, but I think the builtins
> should be "pseudo-reserved", and a user should explicitly have to do
> something *extra* to not receive a warning.
Unfortunately you're suggesting a change to the language which could break existing code. I could see a use for "from __future__ import squawk_if_i_reassign_a_builtin" or something like that, but the current default behavior has to remain as it is.
JMO,
Philip
Chris Angelico
<gric...@coldstorage.com> wrote:
> With surprising regularity, I see program postings (eg. on StackOverflow)
> from inexperienced Python users accidentally re-assigning built-in names.
>
> For example, they’ll innocently call some variable, “list”, and assign a
> list of items to it.
It's actually masking, not reassigning. That may make it easier or
harder to resolve the issue.
If you want a future directive that deals with it, I'd do it the other
way - from __future__ import mask_builtin_warning or something - so
the default remains as it currently is. But this may be a better job
for a linting script.
ChrisA
Ethan Furman
> What sayest the Python community about having an explicit warning
> against such un-pythonic behaviour (re-assigning builtin names)?
What makes you think this behavior is unpythonic? Python is not about
hand-holding.
~Ethan~
Seebs
It seems like something which is sufficiently likely to be a mistake might
deserve a warning -- especially since, so far as I can tell, there's never
going to be a program which can't easily be written to avoid the problematic
behavior.
-s
--
Copyright 2011, all wrongs reversed. Peter Seebach / usenet...@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
I am not speaking for my employer, although they do rent some of my opinions.
Ethan Furman
> On 2011-08-15, Ethan Furman <et...@stoneleaf.us> wrote:
>> Gerrat Rickert wrote:
>>> What sayest the Python community about having an explicit warning
>>> against such un-pythonic behaviour (re-assigning builtin names)?
>
>> What makes you think this behavior is unpythonic? Python is not about
>> hand-holding.
>
> It seems like something which is sufficiently likely to be a mistake might
> deserve a warning -- especially since, so far as I can tell, there's never
> going to be a program which can't easily be written to avoid the problematic
> behavior.
"sufficiently likely" depends entirely on who is doing the coding. I
use `open()` for opening my files, and so regularly use `file` as a
name. It can also be very handy to mask a built-in when doing something
even more fun and entertaining and I, for one, have zero desire to have
Python start warning me about perfectly legitimate code.
Programmers need to learn whichever language they are choosing to code
in, and if extra help is needed beyond whatever is basic for that
language, find (or write! ;) the third-party tool to help out. There
are at least two linters for Python, and multiple IDEs that can help
with these, and other, problems. (I don't much care for IDEs, but I am
thinking of starting to use a linter, myself.)
~Ethan~
Steven D'Aprano
> On Mon, Aug 15, 2011 at 10:52 PM, Gerrat Rickert
> <gric...@coldstorage.com> wrote:
>> With surprising regularity, I see program postings (eg. on StackOverflow)
>> from inexperienced Python users accidentally re-assigning built-in
>> names.
>>
>> For example, they’ll innocently call some variable, “list”, and assign a
>> list of items to it.
>
> It's actually masking, not reassigning. That may make it easier or
> harder to resolve the issue.
The usual term is "shadowing builtins", and it's a feature, not a bug :)
> If you want a future directive that deals with it, I'd do it the other
> way - from __future__ import mask_builtin_warning or something - so
> the default remains as it currently is. But this may be a better job
> for a linting script.
Agreed. It's a style issue, nothing else. There's nothing worse about:
def spam(list):
pass
compared to
class thingy: pass
def spam(thingy):
pass
Why should built-ins be treated as more sacred than your own objects?
--
Steven
Philip Semanchuk
Because built-ins are described in the official documentation as having a specific behavior, while my objects are not.
Yes, it can be useful to replace some of the builtins with one's own implementation, and yes, doing so fits in with Python's "we're all consenting adults" philosophy. But replacing (shadowing, masking -- call it what you will) builtins is not everyday practice. On the contrary, as the OP Gerrat pointed out, it's most often done unwittingly by newcomers to the language who have no idea that they've done anything out of the ordinary or potentially confusing.
If a language feature is most often invoked accidentally without knowledge of or regard for its potential negative consequences, then it might be worth making it easier to avoid those accidents.
bye,
Philip
Steven D'Aprano
>
> On Aug 15, 2011, at 9:32 PM, Steven D'Aprano wrote:
>
>> On Tue, 16 Aug 2011 08:15 am Chris Angelico wrote:
>>
>>> If you want a future directive that deals with it, I'd do it the other
>>> way - from __future__ import mask_builtin_warning or something - so
>>> the default remains as it currently is. But this may be a better job
>>> for a linting script.
>>
>> Agreed. It's a style issue, nothing else. There's nothing worse about:
>>
>> def spam(list):
>> pass
>>
>> compared to
>>
>> class thingy: pass
>>
>> def spam(thingy):
>> pass
>>
>> Why should built-ins be treated as more sacred than your own objects?
>
> Because built-ins are described in the official documentation as having a
> specific behavior, while my objects are not.
*My* objects certainly are, because I write documentation for my code. My
docs are no less official than Python's docs.
You can shadow anything. Sometimes shadowing is safe, sometimes it isn't. I
don't see why we should necessarily fear safe shadowing of built-ins more
than we fear unsafe shadowing of non-built-ins.
(I'm not even convinced that making None a reserved word was the right
decision.)
A warning that is off by default won't help the people who need it, because
they don't know enough to turn the warning on. A warning that is on by
default will be helpful to the newbie programmer for the first week or so,
and then will be nothing but an annoyance for the rest of their career.
(For some definition of a week -- some people are slower learners than
others.)
> Yes, it can be useful to replace some of the builtins with one's own
> implementation, and yes, doing so fits in with Python's "we're all
> consenting adults" philosophy. But replacing (shadowing, masking -- call
> it what you will) builtins is not everyday practice. On the contrary, as
> the OP Gerrat pointed out, it's most often done unwittingly by newcomers
> to the language who have no idea that they've done anything out of the
> ordinary or potentially confusing.
Protecting n00bs from their own errors is an admirable aim, but have you
considered that warnings for something which may be harmless could do more
harm than good? Beginners often lack the skill to distinguish between
harmless warnings that can safely be ignored, and fatal errors that need to
be fixed. Even "user friendly" warning or error messages tend to unnerve
some beginner coders.
There's not much we can do about outright errors, except to make sure that
the error string is as useful as possible, but we can avoid overloading
beginners with warnings they don't need to care about:
WARNING WARNING WARNING WILL ROBINSON, DANGER DANGER DANGER:
YOUR SISTER'S NAME 'PENNY' SHADOWS THE BRITISH CURRENCY,
POTENTIAL AMBIGUITY ALERT DANGER DANGER DANGER!
*wink*
Depending on their personality, you may end up teaching them to ignore
warnings, or a superstitious dread of anything that leads to a warning.
Neither is a good outcome.
> If a language feature is most often invoked accidentally without knowledge
> of or regard for its potential negative consequences, then it might be
> worth making it easier to avoid those accidents.
Perhaps. But I'm not so sure it is worth the cost of extra code to detect
shadowing and raise a warning. After all, the average coder probably never
shadows anything, and for those that do, once they get bitten *once* they
either never do it again or learn how to shadow safely. I don't see it as a
problem.
--
Steven
Danny Wong (dannwong)
If I get multiline standard output from a command. How can I
retrieve this part of the string "(1006)"
Example:
#Committing...
#Workspace: (1003) "My OS_8.12.0 Work" <-> (1004) "OS_8.12.0"
# Component: (1005) "he-Group" <-> (1004) "OS_8.12.0"
# Outgoing:
# Change sets:
# (1006) *--@ <No comment>
# Changes:
# ---c- /he-Group/o-PI/target/config/common-ngp/makefile
Chris Rebert
Assuming the #s aren't in the actual output:
import re
pat = re.compile("^ *(\\([^)]+\\))", re.MULTILINE)
print(pat.search(your_str).group(1))
Obviously can vary depending on how you want to go about defining the
target string.
Cheers,
Chris
P.S. If you reply, please remove my email addresses from the quotation
due to "On Behalf Of" brokenness.
--
Freakin' Outlook/Exchange, I'm telling ya...
http://rebertia.com
Chris Angelico
<steve+comp....@pearwood.info> wrote:
> On Tue, 16 Aug 2011 08:15 am Chris Angelico wrote:
>
>> It's actually masking, not reassigning. That may make it easier or
>> harder to resolve the issue.
>
> The usual term is "shadowing builtins", and it's a feature, not a bug :)
Sorry, shadowing. And yeah, that's why I said that "fixing" this
"issue" was the domain of linting scripts.
ChrisA
Roy Smith
Chris Rebert <cl...@rebertia.com> wrote:
> pat = re.compile("^ *(\\([^)]+\\))", re.MULTILINE)
First rule of regexes in Python is to always use raw strings, to
eliminate the doubled backslashes:
> pat = re.compile(r"^ *(\([^)]+\))", re.MULTILINE)
Is easier to read.
Philip Semanchuk
On Aug 16, 2011, at 1:15 AM, Steven D'Aprano wrote:
> On Tue, 16 Aug 2011 01:23 pm Philip Semanchuk wrote:
>
>>
>> On Aug 15, 2011, at 9:32 PM, Steven D'Aprano wrote:
>>
>>> On Tue, 16 Aug 2011 08:15 am Chris Angelico wrote:
>>>
>>>> If you want a future directive that deals with it, I'd do it the other
>>>> way - from __future__ import mask_builtin_warning or something - so
>>>> the default remains as it currently is. But this may be a better job
>>>> for a linting script.
>>>
>>> Agreed. It's a style issue, nothing else. There's nothing worse about:
>>>
>>> def spam(list):
>>> pass
>>>
>>> compared to
>>>
>>> class thingy: pass
>>>
>>> def spam(thingy):
>>> pass
>>>
>>> Why should built-ins be treated as more sacred than your own objects?
>>
>> Because built-ins are described in the official documentation as having a
>> specific behavior, while my objects are not.
>
> *My* objects certainly are, because I write documentation for my code. My
> docs are no less official than Python's docs.
I'm sure they are no less official to you. But you are you, and then there's...everyone else. =)
I (and I think most people) give far more credibility to the Python docs than to the documentation of an individual. That's not a reflection on you, it reflects the limits of one person's ability versus organizationally produced docs which are heavily used, discussed, and have been iteratively developed over many years.
> Sometimes shadowing is safe, sometimes it isn't.
"Sometimes X is safe and sometimes it isn't" can be said of many, many things, from taking a walk down the street to juggling with knives. But it has little to do with whether or not Python should issue a warning in the specific case we're talking about.
> A warning that is off by default won't help the people who need it, because
> they don't know enough to turn the warning on.
I agree that it wouldn't help the people who need it most (absolute raw newcomers). But you're asserting that once one learned the incantation to enable the theoretical warning we're discussing, one would have graduated to a level where it's no longer useful. That's not the case. There's a lot of ground to cover between "newcomer who has learned about a particular warning" and "coder who regularly shadows builtins on purpose".
I am an example. I know enough to turn the theoretical warning on, and I would if I could. I have never shadowed a builtin deliberately. I've done it accidentally plenty of times. There are 84 builtins in my version of Python and I don't have them all memorized. The fact that my editor colors them differently is the only thing I have to back up my leaky memory. Not all editors are so gracious.
>> Yes, it can be useful to replace some of the builtins with one's own
>> implementation, and yes, doing so fits in with Python's "we're all
>> consenting adults" philosophy. But replacing (shadowing, masking -- call
>> it what you will) builtins is not everyday practice. On the contrary, as
>> the OP Gerrat pointed out, it's most often done unwittingly by newcomers
>> to the language who have no idea that they've done anything out of the
>> ordinary or potentially confusing.
>
> Protecting n00bs from their own errors is an admirable aim, but have you
> considered that warnings for something which may be harmless could do more
> harm than good?
Isn't the whole point of a warning to highlight behavior that's not strictly wrong but looks iffy? Sort of, "I can't be sure, but this looks like trouble to me. I hope you know what you're doing". If we are to eschew warnings in cases where they might be highlighting something harmless, then we would have no warnings at all.
Again, shadowing builtins is not everyday practice. I have been trying to remember if I've ever seen it done deliberately, and I can't remember a case. Now, a comment like that is an invitation for people come out of the woodwork with cases where they found it useful, and I would welcome some examples as I'm sure they'd be interesting. But I think it's safe to say that if you look at random samples of code, builtins are shadowed unintentionally hundreds of times for every time they're shadowed deliberately and usefully.
>> If a language feature is most often invoked accidentally without knowledge
>> of or regard for its potential negative consequences, then it might be
>> worth making it easier to avoid those accidents.
>
> Perhaps. But I'm not so sure it is worth the cost of extra code to detect
> shadowing and raise a warning. After all, the average coder probably never
> shadows anything,
One need look no further than the standard library to see a strong counterexample. grep through the Python source for " file =". I see dozens of examples of this builtin being used as a common variable name. I would call contributors to the standard library above-average coders, and we can see them unintentionally shadowing builtins many times.
> and for those that do, once they get bitten *once* they
> either never do it again or learn how to shadow safely.
I have done it plenty of times, never been bitten (thankfully) and still do it by accident now and again.
You can coerce any example to apply to an argument for or against such a warning, but I think the general case is that Python could reduce unintended consequences by warning when vars erase builtins. (<=== How many builtins did I use in that sentence?)
Cheers
Philip
Chris Angelico
> I am an example. I know enough to turn the theoretical warning on, and I would if I could. I have never shadowed a builtin deliberately. I've done it accidentally plenty of times. There are 84 builtins in my version of Python and I don't have them all memorized. The fact that my editor colors them differently is the only thing I have to back up my leaky memory. Not all editors are so gracious.
>
Rather than "turn a warning on" you can "run your code through a
linting script". There are several excellent ones. Add it to your
makefile or test suite; then you get the testing done over _all_ of
your code, instead of waiting until the moment when you actually
execute it.
> One need look no further than the standard library to see a strong counterexample. grep through the Python source for " file =". I see dozens of examples of this builtin being used as a common variable name. I would call contributors to the standard library above-average coders, and we can see them unintentionally shadowing builtins many times.
>
There are several types of shadowing:
1) Deliberate shadowing because you want to change the behavior of the
name. Extremely rare.
2) Shadowing simply by using the name of an unusual builtin (like
'file') in a context where you never use it. Very common.
3) Unintentional shadowing where you create a variable, but then
intend to use the builtin. This is the only one that's a problem.
list = [...]
is not a problem unless you later on use
foo = list(map(...))
which is more common in Python 3 than Python 2, but fortunately, it'll
throw a nice quick error - nobody's going to use list operations on
the normal 'list' type, nor is anybody going to call an instance of
list.
Definitely a job for linting.
ChrisA
Ethan Furman
> On Aug 16, 2011, at 1:15 AM, Steven D'Aprano wrote:
>> considered that warnings for something which may be harmless could do more
>> harm than good?
>
> wrong but looks iffy? Sort of, "I can't be sure, but this looks like
trouble
> to me. I hope you know what you're doing". If we are to eschew
warnings in
> cases where they might be highlighting something harmless, then we would
> have no warnings at all.
Sounds good to me. ;) Keep such things in the IDE's, and then those
who desire such behavior can have it there. Do not clutter Python with
such.
>> Perhaps. But I'm not so sure it is worth the cost of extra code to detect
>> shadowing and raise a warning. After all, the average coder probably never
>> shadows anything,
>
> One need look no further than the standard library to see a strong
> counterexample. grep through the Python source for " file =". I see
dozens
> of examples of this builtin being used as a common variable name. I would
> call contributors to the standard library above-average coders, and
we can
> see them unintentionally shadowing builtins many times.
What makes you think it's unintentional? file makes a good variable
name, and if you don't need it to actually open a file there's nothing
wrong with using it yourself.
>> and for those that do, once they get bitten *once* they
>> either never do it again or learn how to shadow safely.
>
> I have done it plenty of times, never been bitten (thankfully) and still
> do it by accident now and again.
Seems to me the real issue is somebody using a builtin, such as str or
int, and that they somehow manage to do this without realizing, "wait a
sec', that's one of my variables!" I don't see that as a problem that
Python needs to solve.
~Ethan~
Philip Semanchuk
On Aug 16, 2011, at 11:12 AM, Chris Angelico wrote:
> On Tue, Aug 16, 2011 at 3:13 PM, Philip Semanchuk <phi...@semanchuk.com> wrote:
>
>> One need look no further than the standard library to see a strong counterexample. grep through the Python source for " file =". I see dozens of examples of this builtin being used as a common variable name. I would call contributors to the standard library above-average coders, and we can see them unintentionally shadowing builtins many times.
>>
>
> There are several types of shadowing:
>
> 1) Deliberate shadowing because you want to change the behavior of the
> name. Extremely rare.
> 2) Shadowing simply by using the name of an unusual builtin (like
> 'file') in a context where you never use it. Very common.
> 3) Unintentional shadowing where you create a variable, but then
> intend to use the builtin. This is the only one that's a problem.
Yes, but before you get to #3 you have to go through #2. The way I see it, #2 is setting a trap, #3 is actually stepping in it. I don't want to do either. Neither do I like working with code that has set trap #2 for me.
Cheers
Philip
Philip Semanchuk
On Aug 16, 2011, at 11:41 AM, Ethan Furman wrote:
> Philip Semanchuk wrote:
>> On Aug 16, 2011, at 1:15 AM, Steven D'Aprano wrote:
>>> Protecting n00bs from their own errors is an admirable aim, but have you
>>> considered that warnings for something which may be harmless could do more
>>> harm than good?
>> Isn't the whole point of a warning to highlight behavior that's not strictly
> > wrong but looks iffy? Sort of, "I can't be sure, but this looks like trouble
> > to me. I hope you know what you're doing". If we are to eschew warnings in
> > cases where they might be highlighting something harmless, then we would
> > have no warnings at all.
>
> Sounds good to me. ;) Keep such things in the IDE's, and then those who desire such behavior can have it there. Do not clutter Python with such.
You wink, yet you sound serious. What's with the mixed message? Do you honestly advocate removing all warnings from Python, or not? I sincerely would like to know what you think.
>>> Perhaps. But I'm not so sure it is worth the cost of extra code to detect
>>> shadowing and raise a warning. After all, the average coder probably never
>>> shadows anything,
>> One need look no further than the standard library to see a strong
> > counterexample. grep through the Python source for " file =". I see dozens
>> of examples of this builtin being used as a common variable name. I would
> > call contributors to the standard library above-average coders, and we can
> > see them unintentionally shadowing builtins many times.
>
> What makes you think it's unintentional? file makes a good variable name, and if you don't need it to actually open a file there's nothing wrong with using it yourself.
"Unintentional" as in, "I'm using file as a variable name because it's handy" as opposed to intentional as in "Yes, I am deliberately changing the meaning of this builtin".
>>> and for those that do, once they get bitten *once* they
>>> either never do it again or learn how to shadow safely.
>> I have done it plenty of times, never been bitten (thankfully) and still
> > do it by accident now and again.
>
> Seems to me the real issue is somebody using a builtin, such as str or int, and that they somehow manage to do this without realizing, "wait a sec', that's one of my variables!"
Yes
> I don't see that as a problem that Python needs to solve.
"need" is a strong word. Python will be fine regardless of whether this changes or not. I believe Python could be improved; that's all I'm arguing.
Cheers
Philip
Ethan Furman
> On Aug 16, 2011, at 11:41 AM, Ethan Furman wrote:
>> Philip Semanchuk wrote:
>>> If we are to eschew warnings in
>>> cases where they might be highlighting something harmless, then we would
>>> have no warnings at all.
>>
>> Sounds good to me. ;) Keep such things in the IDE's, and then those
>> who desire such behavior can have it there. Do not clutter Python with
>> such.
>
> You wink, yet you sound serious.
The smiley is an attempt to not sound harsh.
> What's with the mixed message? Do you honestly advocate removing all
> warnings from Python, or not? I sincerely would like to know what you
think.
I think warnings should be reserved for language changes and such (like
DeprecationWarning, RuntimeWarning, and FutureWarning), not for possible
programmer mistakes.
>> What makes you think it's unintentional? file makes a good variable name...
>
> "Unintentional" as in, "I'm using file as a variable name because it's handy"
> as opposed to intentional as in "Yes, I am deliberately changing the
meaning
> of this builtin".
That's not what 'unintentional' means. Further, there's no way to tell
whether it was or was not from the code alone. Unless it caused a bug,
in which case I'd be willing to call it unintentional. ;)
>> I don't see that as a problem that Python needs to solve.
>
> "need" is a strong word. Python will be fine regardless of whether this changes
> or not. I believe Python could be improved; that's all I'm arguing.
Python can be improved -- I don't see 'hand-holding' as an improvement.
IDEs and lints can do this.
~Ethan~
Tim Chase
Chris succinctly described the times I've done shadowing.
Fortunately, the shadowing done in #3 (which you appropriately
describe as being a superset of #2) is fairly remedied with most
editors...since it usually occurs when you have "oh, I
accidentally shadowed builtin X", so you just do a global
search-and-replace for all those places you shadowed X and rename
it to something like "my_X" and proceed to use X as the builtin.
The bigger issue I have is module shadowing which is trickier to
catch and produces weird symptoms (i.e. cryptic errors). The
most common one I see is creating a local module called
"email.py" and then having issues when trying to use
standard-library email calls which find your local email.py
before they find the email.py file in the standard library. I
actually wrote a tool to scan for duplicate modules in
$PYTHONPATH (pretty dumb tool, could easily have broken on things
like zipfile imports, DLLs, etc), but it made diagnosing the
issue easier.
-tkc
Philip Semanchuk
On Aug 16, 2011, at 12:19 PM, Ethan Furman wrote:
> Philip Semanchuk wrote:
>> On Aug 16, 2011, at 11:41 AM, Ethan Furman wrote:
>>> Philip Semanchuk wrote:
>>>> If we are to eschew warnings in
>>>> cases where they might be highlighting something harmless, then we would
>>>> have no warnings at all.
> >>
>>> Sounds good to me. ;) Keep such things in the IDE's, and then those
> >> who desire such behavior can have it there. Do not clutter Python with
> >> such.
>> You wink, yet you sound serious.
>
> The smiley is an attempt to not sound harsh.
Thanks. It's hard to know on the Internet.
>>> I don't see that as a problem that Python needs to solve.
>> "need" is a strong word. Python will be fine regardless of whether this changes
> > or not. I believe Python could be improved; that's all I'm arguing.
>
> Python can be improved -- I don't see 'hand-holding' as an improvement. IDEs and lints can do this.
When you say "hand-holding", I hear a pejorative. That makes "I don't see 'hand-holding' as an improvement" a tautology. Have I misheard you?
I think Python does lots of beneficial hand-holding. Garbage collection is a good example. $DIETY knows, people have been struggling with manual memory management in C and its ilk for a long time. Even though there are good tools to help, memory leaks still happen. Python increases our productivity by allowing us to forget about manual memory management altogether. I can do it with tools like valgrind, but Python's makes the point moot. Is that hand-holding? If so, I'm all for it.
Cheers
Philip
Ethan Furman
> I think Python does lots of beneficial hand-holding. Garbage collection
> is a good example. $DIETY knows, people have been struggling with manual
> memory management in C and its ilk for a long time. Even though there
> are good tools to help, memory leaks still happen. Python increases our
> productivity by allowing us to forget about manual memory management
> altogether. I can do it with tools like valgrind, but Python's makes
> the point moot. Is that hand-holding? If so, I'm all for it.
Good point. There is an important difference, however, between offering
warnings to newbie programmers (which is how this thread started), and
memory management.
~Ethan~
Seebs
> What makes you think it's unintentional?
Programming experience.
People *often* do things unintentionally.
> Seems to me the real issue is somebody using a builtin, such as str or
> int, and that they somehow manage to do this without realizing, "wait a
> sec', that's one of my variables!" I don't see that as a problem that
> Python needs to solve.
I think the word "my" prejudices the case.
Imagine stepping into a large project that uses multiple frameworks and
class libraries and so on. You know Python but you're new to the project.
Under which circumstance will you have more problems?
1. There is not a single shadowed built-in in the entire project.
2. There are dozens of shadowed built-ins based on when the original
programmer felt there wasn't going to be a need for a given built-in
feature, or possibly just didn't know about it.
Also, how easy or hard do you think it will be to debug those problems?
What's your first response going to be if, on a screen which doesn't
contain the word file at all, you try to use the file built-in and you
get some cryptic error? Are you going to know to go looking for the
shadow right away?
Seebs
> On Tue, 16 Aug 2011 01:23 pm Philip Semanchuk wrote:
>>> Why should built-ins be treated as more sacred than your own objects?
>> Because built-ins are described in the official documentation as having a
>> specific behavior, while my objects are not.
> *My* objects certainly are, because I write documentation for my code. My
> docs are no less official than Python's docs.
Sure they are. I can't get yours from python.org.
> You can shadow anything. Sometimes shadowing is safe, sometimes it isn't. I
> don't see why we should necessarily fear safe shadowing of built-ins more
> than we fear unsafe shadowing of non-built-ins.
I think largely because anyone coming to your code will have expectations
about the built-ins. For non-built-ins, they'll have to look around the code
to see what they do, but for built-ins, they come to the table with an
otherwise-reasonable expectation that they already know what that word means.
> A warning that is off by default won't help the people who need it, because
> they don't know enough to turn the warning on. A warning that is on by
> default will be helpful to the newbie programmer for the first week or so,
> and then will be nothing but an annoyance for the rest of their career.
Will it?
I am pretty sure that I'd keep it on and fix anything that triggered it,
because shadowing built-ins strikes me as Asking For Trouble.
> Protecting n00bs from their own errors is an admirable aim, but have you
> considered that warnings for something which may be harmless could do more
> harm than good?
I would distinguish between "may not be causing bugs" and "may be harmless".
I think code which shadows a built-in has a pretty real risk of being
harmful at some unspecified future point when some maintainer who hasn't
memorized every last line of the code makes the totally reasonable assumption
that basic language features are still working and available.
> Perhaps. But I'm not so sure it is worth the cost of extra code to detect
> shadowing and raise a warning. After all, the average coder probably never
> shadows anything, and for those that do, once they get bitten *once* they
> either never do it again or learn how to shadow safely. I don't see it as a
> problem.
I would guess that it happens moderately often entirely by accident.
Not a Python example, but I recently had a heck of a time with some
JavaScript I was trying to maintain. Couldn't get it to work on a slightly
older Firefox, and the diagnostics from Firebug came across as basically
illucid.
Problem: I was declaring an array named 'history'.
My thoughts would be:
1. It's hard to avoid shadowing anything unless you know the entire language
and never forget things.
2. In particular, Python likes to use clear, obvious, names for things.
Meaning that your choice of a clear, obvious, name for a similar thing
could be the name of a thing in the language.
3. I am not sure at all that shadowing can be "safe" in code which will
some day be maintained.
The chances of someone coming along and trying to write clean, idiomatic,
Python which happens to blow up interestingly because their code runs in
an environment where part of the standard language has been shadowed
strike me as Too High.
MHO, but speaking only for myself, I'd want that warning on and I'd not
consider it harmless.
Seebs
> DeprecationWarning, RuntimeWarning, and FutureWarning), not for possible
> programmer mistakes.
I disagree, on the basis of the following:
The quality of C code I have to deal with has increased dramatically as
gcc's aggressive use of warnings has spread.
> Python can be improved -- I don't see 'hand-holding' as an improvement.
> IDEs and lints can do this.
-W ignore
Gerrat Rickert
> A warning that is off by default won't help the people who need it,
> because
> they don't know enough to turn the warning on. A warning that is on by
> default will be helpful to the newbie programmer for the first week or
> so,
> and then will be nothing but an annoyance for the rest of their
career.
>
I think that best practices would suggest that one shouldn't use
variable
names that shadow builtins (except in specific, special circumstances),
so I don't really think this would be an annoyance at all. The number
of
*unwanted* warnings they'd get would be pretty close to zero. OTOH, in
response to a question I asked on StackOverflow, someone posted a large
list of times where this isn't followed in the std lib, so there seems
to be a precedent for just using the builtin names for anything
one feels like at the time.
Thinking about it, what about if, by default, python was configured to
emit warnings about this sort of thing, but a simple environment
variable
or config option would turn them off. That way, new users would get
warnings
when doing this sort of thing, and any experienced user that wanted the
option
of using these variables anywhere would just have a one-time thing to
change
when they installed python (or any time later). They'd turn these
warnings off
when they installed Python, and would never have to think about it
again. New
users (or experienced ones that prefer this) would be warned by default,
and then could make the conscious decision to shadow the builtin names,
instead of accidentally doing it.
All the suggestions to just use a linter aren't helpful, since
new users aren't likely to start with one & and they are the thrust
of this discussion.
Andrew Berg
> One word: SYNTAX HILIGHT
And I had thought your troll skills had disappeared. Good one.
--
CPython 3.2.1 | Windows NT 6.1.7601.17592 | Thunderbird 5.0
PGP/GPG Public Key ID: 0xF88E034060A78FCB
Ethan Furman
> On 2011-08-16, Ethan Furman <et...@stoneleaf.us> wrote:
>> I think warnings should be reserved for language changes and such (like
>> DeprecationWarning, RuntimeWarning, and FutureWarning), not for possible
>> programmer mistakes.
>
> I disagree, on the basis of the following:
>
> The quality of C code I have to deal with has increased dramatically as
> gcc's aggressive use of warnings has spread.
With gcc you pay the cost once, with Python you would pay it with every
run. A linter would be more along the lines of 'pay it once'.
~Ethan~
Ethan Furman
> On 2011-08-16, Ethan Furman <et...@stoneleaf.us> wrote:
>> What makes you think it's unintentional?
>
> Programming experience.
>
> People *often* do things unintentionally.
>
>> Seems to me the real issue is somebody using a builtin, such as str or
>> int, and that they somehow manage to do this without realizing, "wait a
>> sec', that's one of my variables!" I don't see that as a problem that
>> Python needs to solve.
>
> I think the word "my" prejudices the case.
But fits in well with the OP.
> Imagine stepping into a large project that uses multiple frameworks and
> class libraries and so on. You know Python but you're new to the project.
>
> Under which circumstance will you have more problems?
>
> 1. There is not a single shadowed built-in in the entire project.
> 2. There are dozens of shadowed built-ins based on when the original
> programmer felt there wasn't going to be a need for a given built-in
> feature, or possibly just didn't know about it.
My first course of action would be to learn what I'm patching before I
patch it. Scope helps in the case of shadowing; lint helps to give
warnings about that and other things as well.
> Also, how easy or hard do you think it will be to debug those problems?
> What's your first response going to be if, on a screen which doesn't
> contain the word file at all, you try to use the file built-in and you
> get some cryptic error? Are you going to know to go looking for the
> shadow right away?
I should have a pretty good clue from the traceback, and a couple prints
or logs around the offending lines should give me the rest.
~Ethan~
Chris Angelico
> Seebs wrote:
>>
>> On 2011-08-16, Ethan Furman <et...@stoneleaf.us> wrote:
>>>
>>> I think warnings should be reserved for language changes and such (like
>>> DeprecationWarning, RuntimeWarning, and FutureWarning), not for possible
>>> programmer mistakes.
>>
>> I disagree, on the basis of the following:
>>
>> The quality of C code I have to deal with has increased dramatically as
>> gcc's aggressive use of warnings has spread.
>
> A linter would be more along the lines of 'pay it once'.
Agreed; in addition, it's spam. Spam at compile time isn't much of an
issue; you type ./configure and make, and a billion messages scroll
past you. Spam at run time? Everyone who USES the program has to deal
with it, and in amongst the program's own output. I don't know how to
have Python emit warnings that wouldn't cause these issues. The only
thing I can think of is to have the interactive interpreter default to
showing more warnings, which is far from perfect itself.
As to 'file' specifically, my 3.2 doesn't seem to have it as a
builtin. The return value from open() is _io.TextIOWrapper, so the
whole issue of file=open(...) may have been completely dodged.
ChrisA
Tim Chase
> Under which circumstance will you have more problems?
>
> 1. There is not a single shadowed built-in in the entire project.
> 2. There are dozens of shadowed built-ins based on when the original
> programmer felt there wasn't going to be a need for a given built-in
> feature, or possibly just didn't know about it.
In practice, I've never hit such a snag. For #2, the only way it
would impact me is if I did the ill-advised
from shadowy import *
which might tromp on things I care about. Otherwise, I'd just do
something like
from shadowy import list as shadowy_list, id as shadowy_id
If I'm altering another person's ill-created module, I might
consider doing a search for builtins and then just simply
patching the module so that it renamed all the shadowing usages.
Unless the author was being particularly obscure (using scope
to impact where a builtin meant the builtin, and other scopes to
shadow the builtins), most authors are pretty consistent in their
shadowing (usually ignorant of the fact they're shadowing the
builtin) which makes the search-n-replace an easy tweak.
-tkc
Seebs
>> The quality of C code I have to deal with has increased dramatically as
>> gcc's aggressive use of warnings has spread.
> With gcc you pay the cost once, with Python you would pay it with every
> run. A linter would be more along the lines of 'pay it once'.
Huh!
That is a really good point, which I had not considered. I still prefer
to get warnings, but... Hmm.
I wonder whether there's a way to mitigate the cost of these things by
messing with -W settings, such that runtime that wants to be fast can
omit the checks, but the default could still be to, well, prevent likely
errors.
Danny Wong (dannwong)
-----Original Message-----
From: ch...@rebertia.com [mailto:ch...@rebertia.com] On Behalf Of Chris Rebert
Sent: Tuesday, August 16, 2011 12:26 AM
To: Danny Wong (dannwong)
Cc: pytho...@python.org
Subject: Re: regular expression
On Tue, Aug 16, 2011 at 12:00 AM, Danny Wong (dannwong)
<dann...@cisco.com> wrote:
> Hi All,
> If I get multiline standard output from a command. How can I
> retrieve this part of the string "(1006)"
> Example:
>
> #Committing...
> #Workspace: (1003) "My OS_8.12.0 Work" <-> (1004) "OS_8.12.0"
> # Component: (1005) "he-Group" <-> (1004) "OS_8.12.0"
> # Outgoing:
> # Change sets:
> # (1006) *--@ <No comment>
> # Changes:
> # ---c- /he-Group/o-PI/target/config/common-ngp/makefile
Assuming the #s aren't in the actual output:
import re
Terry Reedy
> I think that best practices would suggest that one shouldn't use
> variable
> names that shadow builtins (except in specific, special circumstances),
> so I don't really think this would be an annoyance at all. The number
> of
> *unwanted* warnings they'd get would be pretty close to zero. OTOH, in
> response to a question I asked on StackOverflow, someone posted a large
> list of times where this isn't followed in the std lib, so there seems
> to be a precedent for just using the builtin names for anything
> one feels like at the time.
If you run across that again and email me the link, I will take a look
and see if I think the issue should be raised on pydev. Of course, some
modules *intentionally* define an open function, intended to be accessed
as 'mod.open' and not as 'from mod import *; open'. Also, class/instance
attributes can also reuse builtin names. But 'open = <True/False>' would
be bad.
--
Terry Jan Reedy
Terry Reedy
> I wonder whether there's a way to mitigate the cost of these things by
> messing with -W settings, such that runtime that wants to be fast can
> omit the checks, but the default could still be to, well, prevent likely
> errors.
Warning messages have a cost even if suppressed.
--
Terry Jan Reedy
Seebs
Yes, but is it a *significant* cost? My assumption is that the suppression
would be of checking, not just of displaying messages.
Chris Angelico
> Yes, but is it a *significant* cost? My assumption is that the suppression
> would be of checking, not just of displaying messages.
>
It mightn't be very significant, but there'd still be some cost.
However, IMHO the greatest cost is the spamminess; forcing the user to
deal with lines and lines of warnings is not a useful way to design a
language.
ChrisA
Philip Semanchuk
Hi Terry,
To generalize from your example, are you saying that there's a mild admonition against shadowing builtins with unrelated variable names in standard lib code?
Here's an example from Python 3.2.1's argparse.py, lines 466-473. "open" is shadowed on the second line.
# clean up separators for mutually exclusive groups
open = r'[\[(]'
close = r'[\])]'
text = _re.sub(r'(%s) ' % open, r'\1', text)
text = _re.sub(r' (%s)' % close, r'\1', text)
text = _re.sub(r'%s *%s' % (open, close), r'', text)
text = _re.sub(r'\(([^|]*)\)', r'\1', text)
text = text.strip()
Thanks
Philip
Steven D'Aprano
> On 2011-08-16, Steven D'Aprano <steve+comp....@pearwood.info>
> wrote:
>> On Tue, 16 Aug 2011 01:23 pm Philip Semanchuk wrote:
>>>> Why should built-ins be treated as more sacred than your own objects?
>
>>> Because built-ins are described in the official documentation as having
>>> a specific behavior, while my objects are not.
>
>> *My* objects certainly are, because I write documentation for my code. My
>> docs are no less official than Python's docs.
>
> Sure they are. I can't get yours from python.org.
And what makes that unofficial?
python.org is not the sole arbiter of official documentation in the world.
You can't get docs for Django or Scipy or NLTK from python.org either, but
just try telling the authors of those libraries that their docs are somehow
unofficial and see how far that gets you.
[...]
> I think code which shadows a built-in has a pretty real risk of being
> harmful at some unspecified future point when some maintainer who hasn't
> memorized every last line of the code makes the totally reasonable
> assumption that basic language features are still working and available.
Am I the only person who writes functions and methods any more? *wink*
Modern languages, and by modern I mean most languages more than, oh, about
fifty years old, provide ways to limit the scope of variables. You don't
need to memorise "every last line of the code" to safely edit a function.
def process(list, arg):
spam(list)
ham(arg)
cheese(list, arg)
The scope of parameter "list" is limited to within the function process
itself. Inside, it shadows the built-in list. Outside, it doesn't do squat.
[...]
> My thoughts would be:
> 1. It's hard to avoid shadowing anything unless you know the entire
> language and never forget things.
Define the "entire language". Does that include the names of all the
plethora of exceptions? How about the standard library?
For what it's worth, I don't care about memorising all the built-ins. I
delegate that job to my editor, which has a syntax highlighter for Python.
It never forgets built-ins. (In fact, sometimes this is a nuisance. When
I'm editing Python 3 code, it still insists that apply and reduce are
built-ins.)
> 2. In particular, Python likes to use clear, obvious, names for things.
> Meaning that your choice of a clear, obvious, name for a similar thing
> could be the name of a thing in the language.
Yes, and Python also encourages the use of scopes, so that the clear,
obvious name for something in one scope does not clash with the clear,
obvious, identical name for something completely different in another
scope.
> 3. I am not sure at all that shadowing can be "safe" in code which will
> some day be maintained.
Oh there's no doubt that shadowing *can* be unsafe. But then, very few
things can't be abused.
As I see it, there are (at least) four stages related to shadowing.
(1) At first, you don't even know enough to be concerned by shadowing. You
blithely call variables "list" and "str" without a care in the world...
until something goes wrong.
(2) Second stage, you know enough to realise that shadowing can be bad. You
avoid shadowing everything. Your code is full of variables called "my_list"
and "astr" and "some_tuple". You call your functions things like "izip"
even though it is designed as a replacement for zip, because the caller
might use from itertools import * and accidentally replace the built-in zip
with my zip.
You even avoid using "string" as a variable name because it might shadow the
string module -- even though you haven't actually imported or used the
string module in the last four years.
(3) Eventually, you get frustrated writing doc strings like this:
def function(astr, myint=0):
"""function(astr [, myint]) -> list
Arguments:
astr - string
myint - integer
Do something tool-like to a string and optional integer...
"""
and begin writing them like this:
def function(astr, myint):
"""function(str [, int]) -> list
Do something tool-like to a string and optional integer...
"""
(4) And then, after a while, you decide that perhaps shadowing is not always
so bad. (It helps if the Effbot tells you off for objecting to shadowing in
a two-line function.) At first, you limit yourself to using parameter names
that shadow built-ins in small tool-like functions. Then when the world
doesn't implode, you rethink the use of top-level function names
like "izip" and realise that namespaces exist so that you don't need to
care about shadowing functions you don't use, and if people call import *
they have nobody to blame but themselves if things break.
More-or-less-auto-biographically-ly y'rs,
--
Steven
Steven D'Aprano
> I think Python does lots of beneficial hand-holding. Garbage collection is
> a good example. $DIETY knows, people have been struggling with manual
> memory management in C and its ilk for a long time. Even though there are
> good tools to help, memory leaks still happen. Python increases our
> productivity by allowing us to forget about manual memory management
> altogether. I can do it with tools like valgrind, but Python's makes the
> point moot. Is that hand-holding? If so, I'm all for it.
Hand-holding is not a well-defined term. According to Mel, garbage
collectors would be hand-holding designed for soft nellies.
http://www.cs.utah.edu/~elb/folklore/mel.html
But Mel would probably make his own nails rather than buy them from the
hardware store too. To some secret recipe involving rare metals that most
people have never even heard of. And use a hammer forged from a single
piece of meteor iron, shaped to fit his hand precisely. And it would be too
heavy for mortals to lift.
Of course the term "hand-holding" is a pejorative. The implication is that
the person needing the hand-holding is like a child who needs somebody to
hold their hand while they cross the road. Adults are supposed to know
better (if only it were true in real life!). On the other hand, a real
adult knows their limitations, and if you need a bit of hand-holding in a
certain area of your life, so be it.
Some errors aren't exactly a matter of inexperience and poor judgement, and
so everyone can benefit from guard rails, seat belts and garbage
collection. In that case, it isn't hand-holding, because there's no
expectation that you should grow up from it. Even Donald Knuth probably
would get benefit from a good garbage collector.
I have no objection to lint tools. But separation of concerns should apply:
the Python compiler should just compile what I tell it to, the linter
should warn me if I'm running with scissors.
--
Steven
Seebs
> On Wed, Aug 17, 2011 at 12:49 AM, Seebs <usenet...@seebs.net> wrote:
>> would be of checking, not just of displaying messages.
> It mightn't be very significant, but there'd still be some cost.
> However, IMHO the greatest cost is the spamminess; forcing the user to
> deal with lines and lines of warnings is not a useful way to design a
> language.
Lines and lines?
I'd say if it's to be allowed to shadow built-ins (and I'm not sure that's
a good thing at all), you'd still be looking at one warning per shadowing,
which shouldn't be too many.
Terry Reedy
> On 2011-08-16, Terry Reedy<tjr...@udel.edu> wrote:
>> On 8/16/2011 2:56 PM, Seebs wrote:
>>> I wonder whether there's a way to mitigate the cost of these things by
>>> messing with -W settings, such that runtime that wants to be fast can
>>> omit the checks, but the default could still be to, well, prevent likely
>>> errors.
>
>> Warning messages have a cost even if suppressed.
>
> Yes, but is it a *significant* cost? My assumption is that the suppression
> would be of checking, not just of displaying messages.
The -W settings suppress display or printing of warnings, not their
generation.
import warnings
if some_condition:
warnings.warn(message, Warning)
The test and warn call are made regardless of disposition. Filters
determine what is done with each Warning subclass. The logging module
can grab them too.
--
Terry Jan Reedy
Philip Semanchuk
On Aug 16, 2011, at 9:29 PM, Steven D'Aprano wrote:
> I have no objection to lint tools. But separation of concerns should apply:
> the Python compiler should just compile what I tell it to, the linter
> should warn me if I'm running with scissors.
This point (also made by Ethan) I can agree with. I haven't looked through all the warnings the Python compiler emits, but it seems like it currently doesn't dispense advice (unlike, say, gcc). It only warns about changes in the language & standard library. In that context, asking it to warn about shadowing builtins would be an expansion of scope.
bye,
Philip
Terry Reedy
> Hi Terry,
> To generalize from your example, are you saying that there's a mild admonition
> against shadowing builtins with unrelated variable names in standard
lib code?
I would expect that there might be. I would have to check PEP8.
> Here's an example from Python 3.2.1's argparse.py, lines 466-473.
> "open" is shadowed on the second line.
>
> # clean up separators for mutually exclusive groups
> open = r'[\[(]'
> close = r'[\])]'
> text = _re.sub(r'(%s) ' % open, r'\1', text)
> text = _re.sub(r' (%s)' % close, r'\1', text)
> text = _re.sub(r'%s *%s' % (open, close), r'', text)
> text = _re.sub(r'\(([^|]*)\)', r'\1', text)
> text = text.strip()
--
Terry Jan Reedy
Philip Semanchuk
On Aug 16, 2011, at 10:15 PM, Terry Reedy wrote:
> On 8/16/2011 8:18 PM, Philip Semanchuk wrote:
>
>> Hi Terry,
>> To generalize from your example, are you saying that there's a mild admonition
> > against shadowing builtins with unrelated variable names in standard lib code?
>
> I would expect that there might be. I would have to check PEP8.
I was curious, so I checked. I didn't see anything specifically referring to builtins. This is as close as it gets:
"If a function argument's name clashes with a reserved keyword, it is generally better to append a single trailing underscore rather than use an abbreviation or spelling corruption. Thus "print_" is better than "prnt". (Perhaps better is to avoid such clashes by using a synonym.)"
bye
Philip
Seebs
> Seebs wrote:
>> On 2011-08-16, Steven D'Aprano <steve+comp....@pearwood.info>
>> wrote:
>>> *My* objects certainly are, because I write documentation for my code. My
>>> docs are no less official than Python's docs.
>> Sure they are. I can't get yours from python.org.
> And what makes that unofficial?
> python.org is not the sole arbiter of official documentation in the world.
But it is the sole arbiter of official *Python* documentation. If I am
looking at code in Python, I reasonably expect the Python docs to be
correct about that code. Incomplete, sure, but *correct*.
>> I think code which shadows a built-in has a pretty real risk of being
>> harmful at some unspecified future point when some maintainer who hasn't
>> memorized every last line of the code makes the totally reasonable
>> assumption that basic language features are still working and available.
> Am I the only person who writes functions and methods any more? *wink*
Yes. Everyone else converted to single giant blocks of code because they
are easier to develop. :P
> Modern languages, and by modern I mean most languages more than, oh, about
> fifty years old, provide ways to limit the scope of variables. You don't
> need to memorise "every last line of the code" to safely edit a function.
Only the parts that are in scope, but...
I seem to recall (and I'm pretty Python newbiesh, so I could be wrong)
that Python classes create a scope in which bits of the class become
visible, and some classes are sorta biggish.
> def process(list, arg):
> spam(list)
> ham(arg)
> cheese(list, arg)
> The scope of parameter "list" is limited to within the function process
> itself. Inside, it shadows the built-in list. Outside, it doesn't do squat.
Yes. But what about the built-in spam? :)
> Define the "entire language". Does that include the names of all the
> plethora of exceptions? How about the standard library?
I'd think "standard library" or close to it.
> For what it's worth, I don't care about memorising all the built-ins. I
> delegate that job to my editor, which has a syntax highlighter for Python.
> It never forgets built-ins. (In fact, sometimes this is a nuisance. When
> I'm editing Python 3 code, it still insists that apply and reduce are
> built-ins.)
Heh.
I mostly don't use syntax highlighters; at best, they distract me, at worst,
they distract me a lot. I also don't use one in English, although I am
sure some people would love to have nouns in blue and punctuation in green.
> Yes, and Python also encourages the use of scopes, so that the clear,
> obvious name for something in one scope does not clash with the clear,
> obvious, identical name for something completely different in another
> scope.
"Another" scope is normally a horizontal thing -- you're talking about
a different scope such that you are *either* in this one *or* in that
one.
Built-ins are not in a scope you are never not in.
> Oh there's no doubt that shadowing *can* be unsafe. But then, very few
> things can't be abused.
Yup.
> As I see it, there are (at least) four stages related to shadowing.
> (1) At first, you don't even know enough to be concerned by shadowing. You
> blithely call variables "list" and "str" without a care in the world...
> until something goes wrong.
> (2) Second stage, you know enough to realise that shadowing can be bad. You
> avoid shadowing everything. Your code is full of variables called "my_list"
> and "astr" and "some_tuple". You call your functions things like "izip"
> even though it is designed as a replacement for zip, because the caller
> might use from itertools import * and accidentally replace the built-in zip
> with my zip.
> You even avoid using "string" as a variable name because it might shadow the
> string module -- even though you haven't actually imported or used the
> string module in the last four years.
Heh. (I got advised by pylint not to grab something from it, but I no
longer remember why; I seem to recall being totally unable to find a way
to avoid that warning and still have the string processing I needed.)
> (3) Eventually, you get frustrated writing doc strings like this:
>
> def function(astr, myint=0):
> """function(astr [, myint]) -> list
>
> Arguments:
> astr - string
> myint - integer
>
> Do something tool-like to a string and optional integer...
> """
> and begin writing them like this:
> def function(astr, myint):
> """function(str [, int]) -> list
>
> Do something tool-like to a string and optional integer...
> """
That seems safe enough to me. :)
> (4) And then, after a while, you decide that perhaps shadowing is not always
> so bad. (It helps if the Effbot tells you off for objecting to shadowing in
> a two-line function.) At first, you limit yourself to using parameter names
> that shadow built-ins in small tool-like functions. Then when the world
> doesn't implode, you rethink the use of top-level function names
> like "izip" and realise that namespaces exist so that you don't need to
> care about shadowing functions you don't use, and if people call import *
> they have nobody to blame but themselves if things break.
Hmm. See, I've never reached that, in Python or any other language. I
figure it creates a new potential for confusion, and that I would rather
avoid any ambiguity. I don't *like* ambiguity in code.
So I don't shadow stuff that's part of the language, because doing that
makes it possible for a line of code to have a clear and obvious meaning to
someone who looks at that line out of context, and a *completely different*
clear and obvious meaning to someone who looks at it with a bit more
context. I don't like that! It means that someone reading my code can
never, ever, assume that a standard language feature is actually itself
and not a local shadow which does something different unless they go
walking back up the scope chain checking. And that's a pretty big cost
to attach to stuff that is, by design, basic and universally available.
I might feel differently about names which were only visible after you
imported some specific thing, but if we're talking built-ins that are
always visible...
Hmm. No, actually, even then, I just think it's Bad Mojo to overlap
names that were taken by the language itself. If a name is in a different
scope which I'm not in, I don't care about it, but if the name would already
be in scope if I didn't declare it, then I am creating a nasty ambiguity
if I use it.
Chris Angelico
> On 2011-08-17, Chris Angelico <ros...@gmail.com> wrote:
>> It mightn't be very significant, but there'd still be some cost.
>> However, IMHO the greatest cost is the spamminess; forcing the user to
>> deal with lines and lines of warnings is not a useful way to design a
>> language.
>
> Lines and lines?
>
> I'd say if it's to be allowed to shadow built-ins (and I'm not sure that's
> a good thing at all), you'd still be looking at one warning per shadowing,
> which shouldn't be too many.
One warning per shadowing. Okay.
def foo(list):
"""Foo's the list provided and returns True on success or False on
failure."""
def bar(list):
"""Counts the number of bars in the list, assuming it to be made
of music."""
if not foo(list): return
You call foo() once and bar() twice. How many shadowings have there
been? How many warnings do you get?
A simple implementation would give five warnings for this case - once
for each invocation that shadows a builtin. Another simple
implementation would give two warnings, at the time that the def
statements are executed; this is preferred, but it's still two
warnings, and if you have a huge set of functions that do this, that
can easily be "lines and lines" of warnings. Or should it set a flag
and say "I've already warned this session about shadowing 'list', so
suppress all others"? That seems confusing.
ChrisA
Chris Angelico
<steve+comp....@pearwood.info> wrote:
>> My thoughts would be:
>> 1. It's hard to avoid shadowing anything unless you know the entire
>> language and never forget things.
>
> Define the "entire language". Does that include the names of all the
> plethora of exceptions? How about the standard library?
The shadowing issue applies to the standard library as well as the
builtins, so yes; to avoid shadowing *anything*, you would have to
know the entire language. I posit that this is a practical
impossibility, and that unexpected shadowing will always be possible
(and won't always be prevented by syntax highlighting). Some day
you'll discover that you can't use module X because you have a
function called X, and you'll have to rename.
ChrisA
Steven D'Aprano
[...]
> "Another" scope is normally a horizontal thing -- you're talking about
> a different scope such that you are *either* in this one *or* in that
> one.
>
> Built-ins are not in a scope you are never not in.
That's technically incorrect. Built-ins are a scope you are never in, if
by "in" you mean "code is executed in this scope".
You have three scopes in general:
Local
Global
Built-ins
(There's also zero or more nonlocal scopes, between local and global, that
applies to closures and nested functions, but never mind that.)
Code is almost(?) always executed in the local scope. (eval and exec let you
mess around with that, somewhat.) E.g. an assignment "x = 1" applies to the
local namespace unless you explicitly declare it global. If you are in the
top level of a module, the local namespace is also the global one, the
global statement does nothing, and the assignment occurs in the global
namespace.
However, name lookup rules are such that while assignments are always local,
unsuccessful lookups may fall through to global then built-in.
See also: http://www.python.org/dev/peps/pep-0227/
The only way to put something into the built-in namespace is by using the
fully-qualified name:
>>> import builtins # Use __builtin__ in Python 2
>>> builtins.x = 1 # Kids! Don't try this at home!
>>>
>>> x
1
>>> del x # prove it isn't a global or local
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
So a top-level (global) assignment, say:
def sum(list):
...
*shadows* the built-ins sum and list, it doesn't replace them. It defines a
local variable "list" and a global variable "sum", but it doesn't touch
either built-in. They are still available via the fully qualified name
builtins.sum and builtins.list.
[...]
>> (4) And then, after a while, you decide that perhaps shadowing is not
>> always so bad. (It helps if the Effbot tells you off for objecting to
>> shadowing in a two-line function.) At first, you limit yourself to using
>> parameter names that shadow built-ins in small tool-like functions. Then
>> when the world doesn't implode, you rethink the use of top-level function
>> names like "izip" and realise that namespaces exist so that you don't
>> need to care about shadowing functions you don't use, and if people call
>> import * they have nobody to blame but themselves if things break.
>
> Hmm. See, I've never reached that, in Python or any other language. I
> figure it creates a new potential for confusion, and that I would rather
> avoid any ambiguity. I don't *like* ambiguity in code.
Ah, well you see the thing is, this is Python. As soon as you call any
function you don't control, you no longer know what your environment is
with any certainty. For all you know, the harmless-looking function is
monkey-patching builtins like there's no tomorrow. Speaking broadly,
perhaps too broadly, we're incredibly proud of the ability to modify nearly
everything at runtime.
Fortunately, while we are proud of having that ability, actually *using* it
is considered a mortal sin. We're not Ruby developers -- if you actually
monkey-patch something, especially built-ins, you can expect to be taken
outside and slapped around with a fish if you get caught.
http://www.youtube.com/watch?v=IhJQp-q1Y1s
> So I don't shadow stuff that's part of the language, because doing that
> makes it possible for a line of code to have a clear and obvious meaning
> to someone who looks at that line out of context, and a *completely
> different* clear and obvious meaning to someone who looks at it with a bit
> more
> context. I don't like that! It means that someone reading my code can
> never, ever, assume that a standard language feature is actually itself
> and not a local shadow which does something different unless they go
> walking back up the scope chain checking. And that's a pretty big cost
> to attach to stuff that is, by design, basic and universally available.
Sure. But they can't have that certainty regardless of whether you shadow
something, because how do they know whether you've shadowed it or not?
In theory, anything could be anything at any time, and you have no
protection. In practice, I worry more about being eaten by
genetically-engineered flying piranhas than about rogue monkey-patching
code.
--
Steven
Gerrat Rickert
> Terry Jan Reedy
>
See the accepted answer to this question:
http://stackoverflow.com/questions/7079519/is-there-python-code-in-the-python-standard-library-that-uses-variable-names-that
Seebs
> def foo(list):
> """Foo's the list provided and returns True on success or False on
> failure."""
>
> def bar(list):
> """Counts the number of bars in the list, assuming it to be made
> of music."""
> if not foo(list): return
> You call foo() once and bar() twice. How many shadowings have there
> been? How many warnings do you get?
I'd say two, one when def foo... is parsed, one when def bar... is parsed.
> A simple implementation would give five warnings for this case - once
> for each invocation that shadows a builtin. Another simple
> implementation would give two warnings, at the time that the def
> statements are executed; this is preferred, but it's still two
> warnings, and if you have a huge set of functions that do this, that
> can easily be "lines and lines" of warnings. Or should it set a flag
> and say "I've already warned this session about shadowing 'list', so
> suppress all others"? That seems confusing.
I guess I don't object to multiple warnings if I do the same thing multiple
times. I was just thinking in terms of a single parse-time warning for the
actual point at which something is shadowed, rather than, say, a warning
every time a name is hit during execution of statements and refers to a
shadow.
Seebs
> [...]
>> "Another" scope is normally a horizontal thing -- you're talking about
>> a different scope such that you are *either* in this one *or* in that
>> one.
>> Built-ins are not in a scope you are never not in.
> That's technically incorrect. Built-ins are a scope you are never in, if
> by "in" you mean "code is executed in this scope".
No, by "in" I mean "lookups from your code will reach this scope if they
don't find something sooner".
>> Hmm. See, I've never reached that, in Python or any other language. I
>> figure it creates a new potential for confusion, and that I would rather
>> avoid any ambiguity. I don't *like* ambiguity in code.
> Ah, well you see the thing is, this is Python. As soon as you call any
> function you don't control, you no longer know what your environment is
> with any certainty. For all you know, the harmless-looking function is
> monkey-patching builtins like there's no tomorrow. Speaking broadly,
> perhaps too broadly, we're incredibly proud of the ability to modify nearly
> everything at runtime.
Heh.
> Fortunately, while we are proud of having that ability, actually *using* it
> is considered a mortal sin. We're not Ruby developers -- if you actually
> monkey-patch something, especially built-ins, you can expect to be taken
> outside and slapped around with a fish if you get caught.
Okay, so.
Here's what I don't get.
If it's such a bad thing, *why is it allowed*? Why are you proud of the
ability to do something that you are never socially-allowed to do?
I have mixed feelings about Ruby's general tolerance for monkey-patching,
although I've seen it do some pretty awesome things, so I am somewhat
inclined to accept that it may be worth it. But it's clearly a feature
which is used intentionally.
It sounds to me like Python is very proud of having a feature which no
one would ever use. ... Why?
> Sure. But they can't have that certainty regardless of whether you shadow
> something, because how do they know whether you've shadowed it or not?
Well, they could trust me. :)
> In theory, anything could be anything at any time, and you have no
> protection. In practice, I worry more about being eaten by
> genetically-engineered flying piranhas than about rogue monkey-patching
> code.
I do too, if I know that people don't shadow built-ins. If I know that
people are shadowing built-ins, then some of the time, when I'm editing
other peoples' code, they HAVE effectively monkey-patched it.
Ethan Furman
> On 2011-08-17, Steven D'Aprano wrote:
>> On Wed, 17 Aug 2011 01:17 pm Seebs wrote:
>>> Hmm. See, I've never reached that, in Python or any other language. I
>>> figure it creates a new potential for confusion, and that I would rather
>>> avoid any ambiguity. I don't *like* ambiguity in code.
>
>> Ah, well you see the thing is, this is Python. As soon as you call any
>> function you don't control, you no longer know what your environment is
>> with any certainty. For all you know, the harmless-looking function is
>> monkey-patching builtins like there's no tomorrow. Speaking broadly,
>> perhaps too broadly, we're incredibly proud of the ability to modify nearly
>> everything at runtime.
>
> Heh.
>
>> Fortunately, while we are proud of having that ability, actually *using* it
>> is considered a mortal sin. We're not Ruby developers -- if you actually
>> monkey-patch something, especially built-ins, you can expect to be taken
>> outside and slapped around with a fish if you get caught.
>
> Okay, so.
>
> Here's what I don't get.
>
> If it's such a bad thing, *why is it allowed*? Why are you proud of the
> ability to do something that you are never socially-allowed to do?
Monkey-patching built-ins would be something along the lines of
import sys
sys.modules['__builtin__'].str = my_super_string
and is what stands you in jeopardy of being fish-slapped. ;)
Merely shadowing a built-in, or stdlib, or whatever, isn't monkey-patching.
~Ethan~
Seebs
> Seebs wrote:
>> On 2011-08-17, Steven D'Aprano wrote:
>>> Ah, well you see the thing is, this is Python. As soon as you call any
>>> function you don't control, you no longer know what your environment is
>>> with any certainty. For all you know, the harmless-looking function is
>>> monkey-patching builtins like there's no tomorrow. Speaking broadly,
>>> perhaps too broadly, we're incredibly proud of the ability to modify nearly
>>> everything at runtime.
>>> Fortunately, while we are proud of having that ability, actually *using* it
>>> is considered a mortal sin. We're not Ruby developers -- if you actually
>>> monkey-patch something, especially built-ins, you can expect to be taken
>>> outside and slapped around with a fish if you get caught.
>> Here's what I don't get.
>> If it's such a bad thing, *why is it allowed*? Why are you proud of the
>> ability to do something that you are never socially-allowed to do?
> Monkey-patching built-ins would be something along the lines of
> import sys
> sys.modules['__builtin__'].str = my_super_string
> and is what stands you in jeopardy of being fish-slapped. ;)
> Merely shadowing a built-in, or stdlib, or whatever, isn't monkey-patching.
Oh, I know. I was just noticing that Steven's post is specifically talking
about how Python users are proud of having the ability to monkey-patch.
If monkey-patching like that is a mortal sin, leads to fish-slapping, and
so on..
Why is it possible? Why not just reject such things as invalid code?
Chris Angelico
> If it's such a bad thing, *why is it allowed*? Why are you proud of the
> ability to do something that you are never socially-allowed to do?
>
Going back to my original three examples:
> 1) Deliberate shadowing because you want to change the behavior of the
> name. Extremely rare.
> 2) Shadowing simply by using the name of an unusual builtin (like
> 'file') in a context where you never use it. Very common.
> 3) Unintentional shadowing where you create a variable, but then
> intend to use the builtin. This is the only one that's a problem.
All three are allowed, but it's the first one that's considered
unusual. The second one is simply that Python doesn't have a million
and one reserved words. Yes, you probably don't want to use 'print' as
a variable name, but shadowing it with an exact equivalent would be
fine (eg to automatically date-stamp or log your output, without
changing your code). And as described above, using list/str/id etc is
not uncommon.
I greatly prefer this to the alternative, which is another 133
reserved words (based on Python 3.2 for Windows).
ChrisA
Seebs
>> ability to do something that you are never socially-allowed to do?
> Going back to my original three examples:
I may have been unclear about jumping topics; that comment was about
monkey-patching, not about shadowing.
> I greatly prefer this to the alternative, which is another 133
> reserved words (based on Python 3.2 for Windows).
You have a point there.
I guess I'm just used to the assumption that the confusion caused by
shadowing names outweighs the benefits of using them -- the world is rich
with plausible names.
Ethan Furman
Well, the mortal sin part is a bit of an exaggeration -- it's more like
you'd better have a really darn good reason to do it. And it is
absolutely one of my favorite parts about Python. If I want to inject a
custom Path class into builtins so it's readily available, and then
change os.listdir to return it instead of normal strings, I can. If my
application is truly case-insensitive, I can make my own istr class and
monkey-patch builtins so it's what is used. Can this blow-up in my
face? Certainly. But I would rather have the option open to me instead
of being told "No, I'm sorry, you can't do that because I (developers in
question) didn't imagine a good use case for it".
Part of the fun of Python is experimentation. And how much fun is it to
be told over and over, "No, you can't do that"?
As an example of something that could easily have been outlawed, but
wasn't, check out http://stackoverflow.com/questions/7068340
~Ethan~
Chris Angelico
> I may have been unclear about jumping topics; that comment was about
> monkey-patching, not about shadowing.
>
Ah, apologies.
Monkey-patching is a way of using the middle of a stack of code
without using what's below it. If everything is statically linked,
then every function does exactly what it was written to do and no
more; if it accepts a file object as an argument, you could give it
some other file-like object, but that's all. However, if the function
was written to use print() and you don't want it to print to the
screen, what do you do? Obviously it's your responsibility to ensure
that your replacement is 100% equivalent in the required
functionality, but it's a great flexibility to simply change the
meaning of "print"; the alternative is to copy and paste the code,
make one change (or even no change at all, if your replacement print()
is a global), and run that.
As Ethan posted while I was typing this, you're allowed to do it, but
you just need to have a good reason for it. I like to explain/justify
myself in comments when these things crop up in my code.
ChrisA
Seebs
> Part of the fun of Python is experimentation. And how much fun is it to
> be told over and over, "No, you can't do that"?
Okay, I buy that.
Actually, this sort of fits with my experience of how (sane) people do it
in Ruby.
And I'm really the wrong person to criticize people for monkey-patching:
http://www.yoctoproject.org/projects/pseudo
(In which I monkeypatch *the entire Unix filesystem API*. On Linux and
OS X. For C programs.)
I guess maybe my question shouldn't have been "why is it allowed", but "and
why is it bad to use it?" It just seemed like "you should never do this,
it's horrible" and "we're proud of being able to do this" were not entirely
compatible.
Steven D'Aprano
> On 2011-08-17, Steven D'Aprano <steve+comp....@pearwood.info>
> wrote:
>> Fortunately, while we are proud of having that ability, actually *using*
>> it is considered a mortal sin. We're not Ruby developers -- if you
>> actually monkey-patch something, especially built-ins, you can expect to
>> be taken outside and slapped around with a fish if you get caught.
>
> Okay, so.
>
> Here's what I don't get.
>
> If it's such a bad thing, *why is it allowed*? Why are you proud of the
> ability to do something that you are never socially-allowed to do?
Why does any language allow monkey-patching? Because:
(1) it is a consequence of a clean language design that doesn't special case
things unless absolutely necessary -- monkey-patching wasn't added to the
language, it just emerged given the basic language design;
(2) consequently it isn't some special technique, it is just a special name
given to ordinary, humdrum, everyday things like name-binding within a
namespace;
(3) and yet it is a powerful and useful ability that lets you extend both
the language and libraries (yours or third party) while still writing very
clean code;
(4) it's also pretty cool that you can do these things; and most importantly
(5) all of the above.
Even if we shouldn't (ab)use it in production, it is still greatly useful
for quick and dirty scripts, testing, experimentation and debugging.
And sometimes monkey-patches end up in production. For example, the standard
library site.py file adds help() and quit() commands to builtins at
startup. Or you might grab an instance of some third-party class, and
dynamically adding a method or an attribute to it. Why bother sub-classing
it?
There's a scene in James Clavell's "Shogun" which is relevant. Toranaga, the
Japanese warlord, discovers that the Englishman John Blackthorne has
betrayed his rightful ruler. Blackthorne protests that there are mitigating
circumstances. Toranaga says that there can never be any mitigating
circumstances for disloyalty to one's liege-lord.
Blackthorne replies that there is one: if you win.
The same applies for monkey-patching and other dangerous techniques. There
can be no excuse for doing something like that in production... unless it
is better than the alternatives.
[...]
> It sounds to me like Python is very proud of having a feature which no
> one would ever use. ... Why?
Don't get me wrong, there are plenty of people who would give up Python's
extreme dynamicism is a heartbeat, if it were up to them. It does play
havoc with the ability to optimise Python code, and is one of the reasons
why CPython isn't as fast as (say) Lua. But the PyPy people are well on the
way to blunting that criticism, with a fast, effective JIT optimising
compiler that will be fast in the common case and no worse off in the rare
times that built-in functions have been shadowed or modified.
(PyPy is already about twice as fast as CPython, and in a few carefully
crafted examples faster than C code.)
--
Steven
Chris Torek
I read through the whole thread, though, and did not see anyone
bring up this one particular point: there is already a linting
script that handles this.)
>On Mon, Aug 15, 2011 at 10:52 PM, Gerrat Rickert
><gric...@coldstorage.com> wrote:
>> With surprising regularity, I see program postings (eg. on StackOverflow)
>> from inexperienced Python users accidentally re-assigning built-in names.
>>
>> For example, they'll innocently call some variable, `list', and assign a
>> list of items to it.
In article <mailman.22.13134465...@python.org>
Chris Angelico <ros...@gmail.com> wrote:
>It's actually masking, not reassigning. That may make it easier or
>harder to resolve the issue.
>
>If you want a future directive that deals with it, I'd do it the other
>way - from __future__ import mask_builtin_warning or something - so
>the default remains as it currently is. But this may be a better job
>for a linting script.
The pylint program already does this:
$ cat shado.py
"module doc"
def func(list):
"func doc"
return list
$ pylint shado.py
************* Module shado
W0622: 2:func: Redefining built-in 'list'
...
Your code has been rated at 6.67/10
If your shadowing is done on purpose, you can put in a pylint
comment directive to suppress the warning.
Pylint is the American Express Card of Python coding: "don't leave
$HOME without it!" :-)
--
In-Real-Life: Chris Torek, Wind River Systems
Intel require I note that my opinions are not those of WRS or Intel
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
email: gmail (figure it out) http://web.torek.net/torek/index.html
Seebs
> (I realize this thread is old. I have been away for a few weeks.
> I read through the whole thread, though, and did not see anyone
> bring up this one particular point: there is already a linting
> script that handles this.)
Yes. I've found pylint... A weird mix of "very helpful, thanks" and
"oh, come off it". A thread about pylint is where I got my example of
the natural Python way to express a parabola:
theValueRepresentingTheYAxisLocationOfThePoint =
theValueRepresentingTheXAxisLocationOfThe Point *
theValueRepresentingTheXAxisLocationOfThe Point
I still say that there are times when short names are natural and
idiomatic, and much clearer than long names. :P
But I do think that, given the basic assumption that pylint is a core
tool for vetting code, it is probably adequate for it to provide the
warnings.