How much is set in stone?
Jive Dadson
couple of days. -- Considering using it for a scripting language for a
product at work. I'm very impressed with the system. Of course there
are things I would like to change. :-) So the question is, how much is
set in stone at this point? Is the definition of 2.x now the law of the
land, or can we still make suggestions?
Jive
Roeland Rengelink
Hi Jive,
Welcome to Python
Well, you can still make suggestions. There's even a formal procedure
for it. Look at http://python.sourceforge.net/peps/ for the Python
Enhancement Proposals. However, Python is over 10 years old, quite
mature, and does have a considerable user base, so you shouldn't expect
to get proposed changes accepted without some pretty convincing
arguments (i.e 'neat' won't cut it ;).
Hope this helps,
Roeland
--
r.b.ri...@chello.nl
"Half of what I say is nonsense. Unfortunately I don't know which half"
Emile van Sebille
"Jive Dadson" <jda...@ix.netcom.com> wrote in message
news:3BE4DF39...@ix.netcom.com...
2.2 is at feature lock, but everything is open after that. Review the
existing PEPs to see what has been suggested.
http://python.sourceforge.net/peps/
The bottom of http://www.python.org/Help.html has a section titled _Want to
Contribute?_ with additional info.
--
Emile van Sebille
em...@fenx.com
---------
Delaney, Timothy
1. Choose a baselined version - usually, 1.5.2, 2.0.1 or 2.1.1 (soon to be
2.1.2?). Distribute the version with your app (if you are embedding it in an
app, this is almost certainly what you want to do).
2. Stick with the features in 2.1. Apart from the addition of the "yield"
keyword (and hence generators), no features of the language will change
until 3.0 (although you can access new features using from __future__
statements).
3. If there is a particular feature that you need that is missing from a
distribution, you can always add it yourself - perferably as a library, but
if you need to change the core, you can. It's unlikely that such a change
would make it into the standard distribution, but if you make a good enough
argument for it, and (most importantly) provide the source to the change, it
just may.
Tim Delaney
Cross Avaya R&D
+61 2 9352 9079
> -----Original Message-----
> From: Jive Dadson [mailto:jda...@ix.netcom.com]
> Sent: Sunday, 4 November 2001 5:25 PM
> To: pytho...@python.org
> Subject: How much is set in stone?
>
>
> I'm a newbie at Python, but I've been up to my ears in it for the last
> couple of days. -- Considering using it for a scripting language for a
> product at work. I'm very impressed with the system. Of course there
> are things I would like to change. :-) So the question is,
> how much is
> set in stone at this point? Is the definition of 2.x now the
> law of the
> land, or can we still make suggestions?
>
> Jive
> --
> http://mail.python.org/mailman/listinfo/python-list
>
Jive Dadson
I've looked through all the PEPS and I was surprised that my suggestion
is not already there. So, here it is. Please comment.
Other languages have started life with the feature that the first
occurrence of an identifier defines it. The first time a programmer
spends an afternoon chasing a bug introduced by misspelling an
identifier, he begins to doubt the wisdom of that feature. Suppose you
are called on to modify a long and complex module written by someone
else. At some point you forget that the original programmer did not
comply with your company's variable naming conventions. He named a
variable "keepRecord" when it should have been "keep_record". After
you've made your changes, the code usually works fine, but occasionally
it deletes a record it is supposed to keep. You stare at the code for
hours and you can see absolutely nothing wrong with it.
Here's another one. A programmer who is new to Python code might write
the following:
initialized = 0
def init_everything():
blah()
blah()
blah()
initialized = 1
# Oh noooooo!
I would have complete sympathy for this hypothetical unfortunate fellow.
Visual Basic (and perhaps other Basics, so far as I know) implement a
way to have your cake and yet not spoil the old cake. It will
correctly process code that is written to the old standard, but it
allows you to require a defining occurrence of an identifier. If you
want it the new way, you put a line at the top of your module that says,
option explicit
Of course you must know to put that in your source code, but most books
on Visual Basic tell you up front always to do it. It gets to be a
habit.
Here's how it might work in Python. Again the module would have
something at the top, like "option explicit" ("import explicit"? or
what?) telling the interpreter that within that module, variables must
be declared before use. Then the defining occurrence of a variable
could look similar to the declaration of a function:
option explicit
def initialized = 0
def init_everything():
blah()
blah()
blah()
initialized = 1
The above code would work as anticipated. No pesky "global" qualifier.
The following would generate an error message, just like trying to call
an undefined function does:
>>>option explicit
>>>initialized = 0
Traceback (most recent call last):
File "<stdin>", line 1, in ?
NameError: initialized is not defined.
After due warning (a year or two), "option explicit" could become the
default. Old code might need "option implicit" or something added at
the top to keep it working.
Jive Dadson
I should have changed the subject line on the message I'm replying to. Make it, 'PEP "option explicit" [Re: How much is set in stone?]'
Grant Griffin
Sure, you're free to make suggestions. (We tried to stop folks in the
past, but it didn't work <wink>.) However, bear in mind that nearly
anything any newbie might suggest has very likely already been suggested
and debated *many* times in Python's 10-year history, so if it isn't
already in Python, it has probably been rejected, and you are very
likely wasting your breath. (The Python FAQ at
http://www.python.org/doc/FAQ.html addresses some of these things in its
"Python's Design" section.)
That being said, in observing Python's change over the last two years, I
have been unable to discern any overall direction to them. Python
embodies several different design ethics, some of which are
contradicatory, so its designer, Guido van Rossum, has had to make
choices. For example, one ethic is to minimize the number of features;
even so, features get added on a regular basis. Another ethic is to
make Python run faster; however, one new feature, "rich comparisons",
has actually slowed it. Another ethic is to make only
backwards-compatible changes; however, a new feature, the "__future__"
statement, was added to allow backwards-incompatible changes to be
phased in; fortunately, though, the __future__ statement is itself
backwards-compatible.
(Who but Picasso could know exactly which noses to paint sideways?
<wink>)
Version 1.5.2 was current when I started, and several new features have
been added since then. I suppose every thinks the "true" version of
Python is the one in use when they started. But in all honesty, I
haven't found the newer features to be very useful (to me, at least)
except for string methods (you used to have to import the string library
to do basic string manipulations) and "augmented assignments" (things
like "+=", "*=", etc.). I also use "list comprehensions" (which are too
incomprehinsible to summarize here <wink>) on rare occasions--though I'm
usually ashamed of myself later.
Overall, if you find a simple feature to be "missing" (e.g. C's
increment and decrement operators: "++", "--") it probably has been
intentionally omitted (or at least "steadfastly not added"). As you
become experienced in Python, you will eventually discover the reasons
for the choices Guido has made, and you will most likely agree with them
(though I never *did* figure out why Python 1.5.2 didn't have augmented
assignments <wink>.) If not, you can always demand your money back.
What I'm getting at, of course, is that most things that newbies suggest
(including myself) result from a lack of understanding of The Big
Picture, which you might eventually discover--heck, I'm still hoping to
<wink>. Or, maybe it just hasn't happened yet. In any case, be assured
that somebody has thought of it, somebody has suggested it, and somebody
has debated it--probably many times.
if-Python-doesn't-already-have-it,-you-probably-don't-need-it-ly y'rs,
=g2
--
_____________________________________________________________________
Grant R. Griffin g...@dspguru.com
Publisher of dspGuru http://www.dspguru.com
Iowegian International Corporation http://www.iowegian.com
Grant Griffin
...
> Other languages have started life with the feature that the first
> occurrence of an identifier defines it.
> Visual Basic (and perhaps other Basics, so far as I know) implement a
> way to have your cake and yet not spoil the old cake.
Perl has an option like this. However, I like the way Python does it
better.
In Perl, you can use an unitialized variable (unless you disallow that
via the option). Then, unitialized variables get automatically
initialized to sensible "zero" or "empty" values on first use. Whenever
I used the auto-initialize thing, I regretted it. However, I found the
declaritive form to be awkward.
Python's system is basically "you must write to a variable before you
can read it". This has the net effect of catching the kind of error you
refer to, in nearly all cases--and I haven't had a significant problem
with the cases that can get missed.
Therefore, this simple but powerful idea gives you nearly all of the
benefit of variable declarations, yet none of the baggage.
Also, to do automatic zeroing of undeclared variables, Perl has to use
special punctuation as a back-door way of declaring the type of the
variable. Then, you have to carry along this punctuation as you _use_
the variable.
Again, I like Python's approach better. Since there is no automatic
declaration, the types of variables are specified when you assign to
them:
x = 1 # x is an integer
x = 1.0 # x is a float
x = [] # x is a list
x = {} # x is a dictionary
print y # oops!--"y" hasn't been assigned (declared) yet
wish-I'd-thought-of-that-ly y'rs,
Paul Rubin
> Python's system is basically "you must write to a variable before you
> can read it". This has the net effect of catching the kind of error you
> refer to, in nearly all cases--and I haven't had a significant problem
> with the cases that can get missed.
>
> Therefore, this simple but powerful idea gives you nearly all of the
> benefit of variable declarations, yet none of the baggage.
Of course it doesn't. If you misspell a symbol in some rarely used
path through the code, it's quite possible your testing will miss it.
Compile time checking wouldn't.
Prabhu Ramachandran
>> Therefore, this simple but powerful idea gives you nearly all
>> of the benefit of variable declarations, yet none of the
>> baggage.
PR> Of course it doesn't. If you misspell a symbol in some rarely
PR> used path through the code, it's quite possible your testing
PR> will miss it. Compile time checking wouldn't.
To catch these, maybe you should look at:
http://pychecker.sourceforge.net/
prabhu
Chris Barker
cemented in.
Jive Dadson wrote:
> Thanks. Before I submit a formal proposal, let me run it by you guys.
That is a good idea, most PEPs start as a discussion here. AN a whle lot
more start that way and nver make it to a PEP.
> I've looked through all the PEPS and I was surprised that my suggestion
> is not already there. So, here it is. Please comment.
You can bet that this has been proposed by people new to Python again
and again and again, and I can promise you that this is one thing that
is very well cemented in. If there is one thing that makes Python
Python, is that it is a highly dynamic language. THat is not going to
change.
> option explicit
>
> def initialized = 0
> def init_everything():
> blah()
> blah()
> blah()
> initialized = 1
>
> The above code would work as anticipated. No pesky "global" qualifier.
Actually, no it wouldn't. The second initialized here is a local
variable, and it should remain so. Having a variable in a function be
either local or global depending on something that is outside of the
funciton is asking for trouble. If you start changing that too, you
really won't have Python anymore!!
-Chris
--
Christopher Barker,
Ph.D.
ChrisH...@home.net --- --- ---
http://members.home.net/barkerlohmann ---@@ -----@@ -----@@
------@@@ ------@@@ ------@@@
Oil Spill Modeling ------ @ ------ @ ------ @
Water Resources Engineering ------- --------- --------
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------
Jive Dadson
>
> While nothing is actually set in stone, some things are pretty well
> cemented in.
>
> Jive Dadson wrote:
> > Thanks. Before I submit a formal proposal, let me run it by you guys.
>
> That is a good idea, most PEPs start as a discussion here. AN a whle lot
> more start that way and nver make it to a PEP.
>
> > I've looked through all the PEPS and I was surprised that my suggestion
> > is not already there. So, here it is. Please comment.
>
> You can bet that this has been proposed by people new to Python again
What does that tell you?
> ... and I can promise you that this is one thing that
> is very well cemented in. If there is one thing that makes Python
> Python, is that it is a highly dynamic language. THat is not going to
> change.
You must have misunderstood what I wrote. I'm not proposing anything
that would make it "lowly dynamic".
>
> > option explicit
> >
> > def initialized = 0
> > def init_everything():
> > blah()
> > blah()
> > blah()
> > initialized = 1
> >
> > The above code would work as anticipated. No pesky "global" qualifier.
>
> Actually, no it wouldn't. The second initialized here is a local
> variable, and it should remain so. Having a variable in a function be
> either local or global depending on something that is outside of the
> funciton is asking for trouble.
I didn't propose anything like that. The compiler will know that
"initialized" is not local to init_everything because it is not declared
there with a "def". That has absolutely nothing to do with anything
outside the function init_everything.
> If you start changing that too, you
> really won't have Python anymore!!
>
The changes I've proposed do not conflict in any way with the
"Python-ness" of the language. If you don't put "option explicit" at
the top of your module, it will behave exactly the same as it did
before.
What we are discussing is not uniquely "Python". Other languages have
had the dubious feature that the first assignment to a variable is its
declaration. Invariably, people discover that it was a lousy idea. The
people who USE it discover that. They misspell a variable name and in
doing so they introduce a bug. The bug may not manifest itself until
the code is already shipped. Even if they do notice the bug before it's
too late, it may be very difficult to track down. I've been down that
road a lot of times.
There's a pragmatic reason I'm asking for this feature. One of our
customers has suggested that we implement scripting using Python. I've
looked into Python, and I am very impressed with what I see. But I
would have grave misgivings giving it to customers with variable
declarations the way they are now. I'm not sure I could sell the boss
on it, and don't know if I would want to. The situation is quite
different from writing code on ones own computer to be used in-house.
These programs will run equipment, often in foreign countries, that can
break literally a million dollars worth of merchandise, and could even
hurt somebody. It is very important to find bugs BEFORE a program goes
into production use. I won't have any say over who writes these
programs, but bugs that appear to be compiler bugs ("Why doesn't that
variable change when I assign to it?") will get bounced right to me.
Of course, since the source code is freely available, I could implement
it on our copy only. But that would be inefficient. By your own
account, lots of other people want the feature also.
Jive
John Roth
"Prabhu Ramachandran" <pra...@aero.iitm.ernet.in> wrote in message
news:mailman.1004964308...@python.org...
Or PyCover, at the Vaults if you want to insure that
your unit tests were sufficiently comprehensive.
John Roth
>
Roeland Rengelink
Jive Dadson wrote:
>
> Chris Barker wrote:
> >
> > While nothing is actually set in stone, some things are pretty well
> > cemented in.
> >
> > Jive Dadson wrote:
> > > Thanks. Before I submit a formal proposal, let me run it by you guys.
> >
> > That is a good idea, most PEPs start as a discussion here. AN a whle lot
> > more start that way and nver make it to a PEP.
> >
> > > I've looked through all the PEPS and I was surprised that my suggestion
> > > is not already there. So, here it is. Please comment.
> >
> > You can bet that this has been proposed by people new to Python again
> > and again and again ...
>
> What does that tell you?
>
That people who are new to a language tend to perceive that language in
terms of languages they have used previously. The most common requests
for language additions by new users tend to fall in the "can't you add
this really nice feature from language X to Python" category.
> > ... and I can promise you that this is one thing that
> > is very well cemented in. If there is one thing that makes Python
> > Python, is that it is a highly dynamic language. THat is not going to
> > change.
>
> You must have misunderstood what I wrote. I'm not proposing anything
> that would make it "lowly dynamic".
>
Well, you use the term 'variable declaration' which for most of us
carries the connotation of both the definition of a name, and a
restriction of the type of values that can be assigned to it. aka static
typing. The request for some kind of static typing is quite common.
Reasons given are usually 'catching bugs (using the same arguments you
give)' or speed. Python, however, happens to be dynamically typed.
Most of us feel that the advantages of dynamic typing far outweigh the
disadvantages.
However, I gather from the rest of your post, that you just want the
name definition part of variable declaration, and not the type
definition part. Why don't you want static typing. It allows the
compiler to catch even more bugs?
> >
> > > option explicit
> > >
> > > def initialized = 0
> > > def init_everything():
> > > blah()
> > > blah()
> > > blah()
> > > initialized = 1
> > >
> > > The above code would work as anticipated. No pesky "global" qualifier.
> >
> > Actually, no it wouldn't. The second initialized here is a local
> > variable, and it should remain so. Having a variable in a function be
> > either local or global depending on something that is outside of the
> > funciton is asking for trouble.
>
> I didn't propose anything like that. The compiler will know that
> "initialized" is not local to init_everything because it is not declared
> there with a "def". That has absolutely nothing to do with anything
> outside the function init_everything.
>
Yes you did ;)
Most local variables shadowing global names, are intended to be local.
Therefore, the only reasonable thing for your 'option explicit' to do,
would be to raise an Error with "undefined variable initialized in
init_everything". Because, assuming that for the simple reason that
there is a global with the same name that was defined properly, the
undefined local variable is the same as the global one, will usually be
_wrong_. Unfortunately the most likely fix someone would apply is to
write 'def initialized' at the beginning of the function, which,
although usually the right solution, in this case, is an error which
will go just as undetected as the missing global.
By the way, the only reasonable idiom for this example would be:
initialized = 0
def init_everything()
# global initialized <- forgotten
if not initialized: # and therefore this will be an error
blah()
blah()
blah()
initialized = 1
And the compiler catches that just nicely. Also, I just grepped approx
10k lines of Python code and I couldn't find a single 'global' statement
in there.
>
> What we are discussing is not uniquely "Python". Other languages have
> had the dubious feature that the first assignment to a variable is its
> declaration. Invariably, people discover that it was a lousy idea. The
> people who USE it discover that.
Yet, there are thousands of people happily using a language (Python)
that does have that feature. I'm sure you don't wish to imply that they
don't really USE the language ;)
> They misspell a variable name and in
> doing so they introduce a bug. The bug may not manifest itself until
> the code is already shipped. Even if they do notice the bug before it's
> too late, it may be very difficult to track down. I've been down that
> road a lot of times.
>
I think it would be reasonable to assume that either
- Those of us that USE Python really don't encounter this problem that
often, or
- Those of us that USE Python have found ways to deal with this problem
that work so well that a language change is just a very expensive
alternative
Have fun,
Jive Dadson
>
> Jive Dadson wrote:
> >
> > Chris Barker wrote:
> > >
> > > While nothing is actually set in stone, some things are pretty well
> > > cemented in.
> > >
> > > Jive Dadson wrote:
> > > > Thanks. Before I submit a formal proposal, let me run it by you guys.
> > >
> > > That is a good idea, most PEPs start as a discussion here. AN a whle lot
> > > more start that way and nver make it to a PEP.
> > >
> > > > I've looked through all the PEPS and I was surprised that my suggestion
> > > > is not already there. So, here it is. Please comment.
> > >
> > > You can bet that this has been proposed by people new to Python again
> > > and again and again ...
> >
> > What does that tell you?
> >
>
> That people who are new to a language tend to perceive that language in
> terms of languages they have used previously.
I can't say that answer disappoints me, because I expected it.
> The most common requests
> for language additions by new users tend to fall in the "can't you add
> this really nice feature from language X to Python" category.
> ...
> > You must have misunderstood what I wrote. I'm not proposing anything
> > that would make it "lowly dynamic".
> >
>
> Well, you use the term 'variable declaration' which for most of us
> carries the connotation of both the definition of a name, and a
> restriction of the type of values that can be assigned to it. aka static
> typing.
If you had read what I wrote, you would know I didn't mean anything like
that.
> The request for some kind of static typing is quite common.
Off topic.
> ...
> >
> > I didn't propose anything like that. The compiler will know that
> > "initialized" is not local to init_everything because it is not declared
> > there with a "def". That has absolutely nothing to do with anything
> > outside the function init_everything.
> >
>
> Yes you did ;)]
Did not. (Did too.) Did not.
>
> Most local variables shadowing global names, are intended to be local.
> Therefore, the only reasonable thing for your 'option explicit' to do,
> would be to raise an Error with "undefined variable initialized in
> init_everything".
That's right. If you had read my the message you are arguing with, you
would see that I proposed exactly that, even spelling out the syntax of
the exception message.
I'm dumping the rest of your message unread.
Bye now.
Jive
Roeland Rengelink
Jive Dadson wrote:
>
> Roeland Rengelink wrote:
> >
> > Jive Dadson wrote:
<snip>
> > > You must have misunderstood what I wrote. I'm not proposing anything
> > > that would make it "lowly dynamic".
> > >
> >
> > Well, you use the term 'variable declaration' which for most of us
> > carries the connotation of both the definition of a name, and a
> > restriction of the type of values that can be assigned to it. aka static
> > typing.
>
> If you had read what I wrote, you would know I didn't mean anything like
> that.
>
Since, in the next paragraph, I acknowledged that you didn't mean that,
I'm somewhat surprised by that remark. Let me quote myself:
> However, I gather from the rest of your post, that you just want the
> name definition part of variable declaration, and not the type
> definition part.
It's somewhat ironic that my honest attempt to understand what you meant
by 'variable declaration' --rereading what you wrote several times--
leads to this response, and selective quoting.
> > > I didn't propose anything like that. The compiler will know that
> > > "initialized" is not local to init_everything because it is not declared
> > > there with a "def". That has absolutely nothing to do with anything
> > > outside the function init_everything.
> > >
> >
> > Yes you did ;)]
>
> Did not. (Did too.) Did not.
>
> >
> > Most local variables shadowing global names, are intended to be local.
> > Therefore, the only reasonable thing for your 'option explicit' to do,
> > would be to raise an Error with "undefined variable initialized in
> > init_everything".
>
> That's right. If you had read my the message you are arguing with, you
> would see that I proposed exactly that, even spelling out the syntax of
> the exception message.
>
No, you said that it wouldn't raise an error in that case, because,
and I quote:
> > > ... The compiler will know that
> > > "initialized" is not local to init_everything because it is not declared
> > > there with a "def". That has absolutely nothing to do with anything
> > > outside the function init_everything.
The error you suggested was for an assignment to an undefined variable
at global scope.
If I understand you correctly, you're proposing that an undefined
variable at local scope is assumed to be global scope, and only if
there's no global definition it becomes an error.
I'm saying that that's a bad idea because the missing definition at
local scope would usually be a bug, which would then get hidden because
there happens to be an (unrelated) global definition. You're basically
replacing hard to detect bugs due to missing global statements with
similarly hard to detect bugs due to missing defs.
I was also implying that that's a bad trade-off because 'global' is used
rarely.
Chris Barker
> > You can bet that this has been proposed by people new to Python again
> > and again and again ...
>
> What does that tell you?
As others have mentioned, it means that when people come to Python with
experience with other languages, they immediately start proposing that
Python add their favorite features from their previous language.
> > ... and I can promise you that this is one thing that
> > is very well cemented in. If there is one thing that makes Python
> > Python, is that it is a highly dynamic language. THat is not going to
> > change.
>
> You must have misunderstood what I wrote. I'm not proposing anything
> that would make it "lowly dynamic".
While you did propose that "option explicit" was optional, that
essentially creates two dialects of the language, which is frowned upon,
unless it's temorary (see various from __future__ options)
And, yes you are proposing a reduction in dynamisism. Would this be
legal?
option explicit
class dummy_class:
pass
spam = dummy_class()
spam.eggs = 5
Python currently allows you to dynamically add attributes to a class
instance. some people consider that dangerous, but it really is part of
the dynamic nature of Python.
Anyway, I in no way intend or intended to argue the merits of your
proposal, what I am saying is that it is highly unlikely that it would
ever get added to Python.
> There's a pragmatic reason I'm asking for this feature. One of our
> customers has suggested that we implement scripting using Python. I've
> looked into Python, and I am very impressed with what I see. But I
> would have grave misgivings giving it to customers with variable
> declarations the way they are now.
Clearly you have these misgivings, and based on your experience with
other languages, you think this is important, but you have probably
figured out from this thread that there are a whole lot of people using
Python out there that do not see this as a big issue at all. A bug is a
bug, and this feature would only catch very few bugs. While this is a
pretty limited proposal, there have been many, many proposals for
features that would help the compiler catch minor bugs for you. The
consensus among most Python users is that they add little value.
Nicholas FitzRoy-Dale
> Chris Barker wrote:
>> If you start changing that too, you
>> really won't have Python anymore!!
>
> The changes I've proposed do not conflict in any way with the
> "Python-ness" of the language. If you don't put "option explicit" at
> the top of your module, it will behave exactly the same as it did
> before.
Good point! Based on that, here is my Python "Enhancement" proposal:
Let's make it so that placing "option disturbing" at the start of a
module turns on "C Mode". In this mode, Python's behaviour would be
modified in the following ways:
* Use of curly braces to delineate syntax blocks, rather than
indentation;
* The C ternary operator would be fully supported, as would
fall-through "switch" statements (as long as the type being
switch()ed is some kind of number, of course);
* Class support would be removed, but everyone knows all you
need is support for struct{} anyway. Structs could only
contain the types "IntType", "FloatType" and "StringType",
but when you get down to it it's all numbers anyway;
* We'd finally get our beloved "goto" statement.
Fortunately, none of this will affect the "Python-ness" of the language
- without "option disturbing", it will behave exactly as before!
--
- Nicholas FitzRoy-Dale
http://www.lardcave.net
I'm thinking of getting a pet cheese. I already have the cheese food.
- http://www.enweirdenment.org/cgi-bin/cube-hof.html
Jive Dadson
Jive
Kalle Svensson
> I expected better from this group.
You did? What would be better in this case? That anyone who doesn't like your
proposal (most of the c.l.py subscribers, I would guess) stay quiet?
Peace,
Kalle
--
[ Thought control, brought to you by the WIPO! ]
[ http://anti-dmca.org/ http://eurorights.org/ ]
Chris Barker
> I expected better from this group.
That last message was a littel snide, but what exactly di you expect.
As someone new to Python, you have proposed something that you think is
absolutely indespensible, and despite a number of people politely
explaining why they don't think your suggestion is likely to to be
incorporated into Python, you continue to imply that the language is
practically unusable because of this one missing feature. People are
only going to listen to that for so long before they get snide.
By the way, every single one of the modifications listed as being part
of "C-mode" have been suggested by people new to Python, who insisted
that Python would be much better if that feature were incorporated!
(except maybe removing class, but I wouldn't be too surprised!)
Cliff Wells
> I expected better from this group.
I had skipped most of this thread because after reading you initial post, I
saw that the feature you were requesting didn't interest me (and yes, I USE
Python in commercial applications). However, after seeing this post, I had
to wonder what happened later so I went back and read the entire thread.
What I saw was people trying to have an intelligent conversation with you
(with the occasional misunderstanding, as per usual in conversations on
technical topics) and you getting more belligerent as you realized that
others weren't seeing it your way.
You expected better than... you? Don't bother responding as I'll simply
"[dump] the rest of your message unread."
--
Cliff Wells
Software Engineer
Logiplex Corporation (www.logiplex.net)
(503) 978-6726 x308
(800) 735-0555 x308
William Tanksley
>Roeland Rengelink wrote:
>I'm dumping the rest of your message unread.
>Bye now.
That's often a good idea, but in this case the rest of his message
addresses your proposal very well. I highly recommend that you read it.
>Jive
--
-William "Billy" Tanksley
Fredrik Lundh
> I expected better from this group.
well, I expected better from you.
</F>
Huaiyu Zhu
>I expected better from this group.
>
>Jive
Presumably this is your reaction to the comments about your proposal.
Suppose someone says, "It's an idiotic idea", don't take it personally.
Suppose someone says, "Only idiot can have this idea", do take it personally.
The comments in this thread have been much nicer than both.
If there is an impression (real or imagined) that certain proposal was due
to unfamiarity with Python or prejudice carried from other languages, and
have well-known deficiencies, don't you expect people to say so?
It appears that's exactly what people are saying.
If you care about the idea, the only way to proceed is to demonstrate that
it is well thought-out, by explaining it well and rebuke all the counter
arguments.
Believe me, for any comments on any article, there are many more people who
have given it a thought but let others comment. An idea posted here is
often considered more thoroughly than many papers in journals.
Huaiyu
Andreas Kostyrka
Hash: SHA1
Am Mittwoch, 7. November 2001 00:29 schrieben Sie:
> I expected better from this group.
Well, normally, when I'm a newbie, and this happens everytime I try a new
language community, I not only try to learn the rules, but also the useful
idioms. Basically while it's usually quite easy to learn a language, it's
more complicated to get a feeling for the language.
I've also got the impression that you ignore the fact that Python is quite a
mature language (I personally have been maintaining a commercial solution
written in Python since 1996 ;) ), and that is not a wise thing to do.
And lastly your proposition is quite popular only with newbies that don't
know Python. Actually it is already implementable with Python now:
UnknownName="UnknownName"
class ProtectedValues:
def __init__(self):
self.a=1
self.b=2
self.classsetup=1
def __setattr__(self,k,v):
if self.__dict__.has_key("classsetup"):
if not self.__dict__.has_key(k):
raise UnknownName
self.__dict__ [k]=v
x=ProtectedValues()
x.c=100 # -> UnknownName
# Warning: I've typed this in my Mailer, and didn't test it. So the
# indentation is propably wrong ;)
But then, You said You were a newbie. ;) You just didn't behave like a
newbie, by proposing obviously a very non-Pythonish change. But then, perhaps
I should tell the guys at Audi, that my A6 should have 6 tires instead of the
4. As a newbie in car design I know it better than them.
Andreas
- --
Andreas Kostyrka; Raiffeisenstr. 16/9; 2320 Zwölfaxing
Tel: +43/676/4091256; Fax: +43/1/7065299
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE76WVSHJdudm4KnO0RAgM/AKCOMyzQmXApy8rNUlNvkA5BUdaPuwCfa/Kh
AZehM280nUQ9hmNu6xB9Xoc=
=iMk5
-----END PGP SIGNATURE-----
Ype Kingma
you wrote:
> [snip]
>
> There's a pragmatic reason I'm asking for this feature. One of our
> customers has suggested that we implement scripting using Python. I've
> looked into Python, and I am very impressed with what I see. But I
> would have grave misgivings giving it to customers with variable
> declarations the way they are now. I'm not sure I could sell the boss
> on it, and don't know if I would want to. The situation is quite
> different from writing code on ones own computer to be used in-house.
I suppose you know that using an unitialised variable will cause a NameError
exception?
> These programs will run equipment, often in foreign countries, that can
> break literally a million dollars worth of merchandise, and could even
> hurt somebody. It is very important to find bugs BEFORE a program goes
If that is the case you might consider not using python, but a more typesafe
language, and make sure that your customer only uses his own code _after_
inspecting and testing it thoroughly.
Python is dynamic and not typesafe, which means that any object can do
almost anything to any object at any point in time. For example when a class
defines a method, that method can be overriden for any object of that class
_dymically_, even while the overriden method is being executed.
I have not tried it, but I would not be surprised if the method can even
be removed from the class while the program is running.
By the way, this dynamic nature would also render your proposed enhancement
non functional: a variable can be deleted from some namespaces at any time.
Using such a variable after its deletion will cause a NameError exception.
> into production use. I won't have any say over who writes these
> programs, but bugs that appear to be compiler bugs ("Why doesn't that
> variable change when I assign to it?") will get bounced right to me.
>
> Of course, since the source code is freely available, I could implement
> it on our copy only. But that would be inefficient. By your own
> account, lots of other people want the feature also.
>
> Jive
Regards,
Ype
--
email at xs4all.nl
Peter Hansen
>
> I've looked into Python, and I am very impressed with what I see. But I
> would have grave misgivings giving it to customers with variable
> declarations the way they are now. I'm not sure I could sell the boss
> on it, and don't know if I would want to. The situation is quite
> different from writing code on ones own computer to be used in-house.
> These programs will run equipment, often in foreign countries, that can
> break literally a million dollars worth of merchandise, and could even
> hurt somebody. It is very important to find bugs BEFORE a program goes
> into production use.
A necessary step to get the robustness you claim you need is to do
extensive testing. If one writes an application with Python and it
breaks a million dollars of merchandise it will not be Python's
fault, but the programmer's fault for inadequate testing.
The only thing those other languages give you which Python doesn't
is compiler warnings which lull you into a false sense that your program
is clean when in fact you were lazy and didn't do proper testing.
If you do the same amount of testing with Python that you would need
to do with other languages to be very safe, you would have just as
robust an end result.
--
----------------------
Peter Hansen, P.Eng.
pe...@engcorp.com
Paul Rubin
> A necessary step to get the robustness you claim you need is to do
> extensive testing. If one writes an application with Python and it
> breaks a million dollars of merchandise it will not be Python's
> fault, but the programmer's fault for inadequate testing.
>
> The only thing those other languages give you which Python doesn't
> is compiler warnings which lull you into a false sense that your program
> is clean when in fact you were lazy and didn't do proper testing.
> If you do the same amount of testing with Python that you would need
> to do with other languages to be very safe, you would have just as
> robust an end result.
That makes it sound as if any non-robustness in the application is the
fault of how the application is written. However, the Python
interpreter itself can also have bugs. That affects application
robustness. Bugs in the Python interpreter can make the application
crash even if the application code is itself bug-free.
Part of the question "can I write robust applications in Python" is
"how reliable is the interpreter, really?". I've seen Python 2.1.1
inexplicably crash several times. One thing that another language
might give you besides compiler warnings and false senses of security
is implementations that don't crash.
Overall Python seems pretty reliable but I think its implementation is
not yet as well shaken out as some other languages' implementations.
I'm happy writing web applications in Python. I could not in good
conscience recommend writing life critical apps (medical implant
firmware, avionics, etc.) in Python in its present form.
Steve Holden
news:7xofmcy...@ruckus.brouhaha.com...
But of course you'd be happy to trust them to IIS-based, NT-based web
applications, right?
My own opinion, for what it's worth (and I am known to bend ears on this
topic):
<rant>Python's delivered software quality is higher
than many commercial language implementations, and
the level of support available is an order of magnitude
better: with Python you end up communicating with
peoiple who actually understand the software</rant>.
I suspect, without any reasonable data to support it, that most such
assertions (but not necessarily yours) boil down to "I'd like to have
someone to sue if things come badly unstuck". Have you READ the typical
commercial language system license agreement?
regards
Steve
--
http://www.holdenweb.com/
Paul Rubin
> But of course you'd be happy to trust them to IIS-based, NT-based web
> applications, right?
No of course not.
> My own opinion, for what it's worth (and I am known to bend ears on this
> topic):
>
> <rant>Python's delivered software quality is higher
> than many commercial language implementations, and
> the level of support available is an order of magnitude
> better: with Python you end up communicating with
> peoiple who actually understand the software</rant>.
>
> I suspect, without any reasonable data to support it, that most such
> assertions (but not necessarily yours) boil down to "I'd like to have
> someone to sue if things come badly unstuck". Have you READ the typical
> commercial language system license agreement?
I feel that perl and gcc are both more solid than python right now,
and their support is at least as good. Gcc is is used in some
military projects where they simply weren't willing to trust anything
they didn't have source for.
Cliff Wells
> about 2.1.1 before, and I can assure you we didn't pay everyone on c.l.py
Who's "we", Tim? I see you've finally shown yo
Tim Peters
> ...
> Part of the question "can I write robust applications in Python" is
> "how reliable is the interpreter, really?". I've seen Python 2.1.1
> inexplicably crash several times. One thing that another language
> might give you besides compiler warnings and false senses of security
> is implementations that don't crash.
>
> Overall Python seems pretty reliable but I think its implementation is
> not yet as well shaken out as some other languages' implementations.
> I'm happy writing web applications in Python. I could not in good
> conscience recommend writing life critical apps (medical implant
> firmware, avionics, etc.) in Python in its present form.
I invite anyone who cares to study the bug lists for Perl, gcc and Python,
and draw an informed conclusion from that. You (Paul) have spread FUD about
2.1.1 before, and I can assure you we didn't pay everyone on c.l.py to
refrain from posting "me too!", neither to write about their Python apps
running for months at a time. Your experience remains unique,
irreproducible (last I heard), and *could* be caused by any number of
computer problems, both hardware and software. While *a priori* a crash in
a Python program is most likely a bug in Python, the lack of any confirming
reports also requires explanation.
but-on-your-advice-i'm-going-to-drive-my-pacemaker-with-perl<wink>-ly
y'rs - tim
Paul Rubin
> I invite anyone who cares to study the bug lists for Perl, gcc and
> Python, and draw an informed conclusion from that. You (Paul) have
> spread FUD about 2.1.1 before, and I can assure you we didn't pay
> everyone on c.l.py to refrain from posting "me too!", neither to
> write about their Python apps running for months at a time. Your
> experience remains unique, irreproducible (last I heard), and
> *could* be caused by any number of computer problems, both hardware
> and software. While *a priori* a crash in a Python program is most
> likely a bug in Python, the lack of any confirming reports also
> requires explanation.
It's not intended as FUD. Other people did report similar crashes.
It's conceivable that my crashes weren't caused by Python bugs, but
signs don't point that way.
Comparing Python with Perl, generally I find Python better designed
but its implementation more likely to take short cuts. The security
issue with pickle.loads that we spent a long time discussing is
something I think the perl developers would not have tolerated.
There's all kinds of other missing functionality in the runtime system
as well, that doesn't result directly in unrobust programs, but does
make it more difficult to write robustly. A lot of this ng is about
the resulting issues.
I think Python is promising and exciting and I like programming in it.
It's just inaccurate to say it's as far along in development as some
other languages.
Paul Winkler
Why do people's posts keep getting cut off on this ng?
Is there some kind of consp
Paul Rubin
Erno Kuusela
<phr-n...@nightsong.com> writes:
| The security issue with pickle.loads that we spent a long time
| discussing is something I think the perl developers would not have
| tolerated.
could you describe the security issue in some detail?
-- erno
Andrew Dalke
>Comparing Python with Perl, generally I find Python better designed
>but its implementation more likely to take short cuts.
I've reported several core dump bugs in Perl over time. The most
recent was a couple months back. I've used Python a lot more than
Perl, and I work out of CVS, so it's hard to compare the two,
but I feel that they are comparable in implementation solidity.
Yes, I've reported Python core dumps as well.
I find it impressive you can compare implementation details. When
I've found problems or had questions with Perl's C implementation,
I haven't been able to figure out heads nor tails of the code. In
Python, I've never had that problem except once where there was
a 'tstate' bug related to how threads are done.
> The security
>issue with pickle.loads that we spent a long time discussing is
>something I think the perl developers would not have tolerated.
I thought most of those pickle bugs have been addressed. I know
I sent in fixes for a couple of them. As I recall, it wasn't
tolerated, but no one wanted to go fix.
As for security, I'm astonished that Perl passes NUL containing
strings to system calls, which opens up a Perl script to all sorts
of subtle attacks. Perhaps the most famous is
http://www.mail-archive.com/mod...@apache.org/msg00396.html
Python raises an exception in this case. There are other concerns
I have, but the point is I don't see Perl being the best example
with which to compare.
>There's all kinds of other missing functionality in the runtime system
>as well, that doesn't result directly in unrobust programs, but does
>make it more difficult to write robustly. A lot of this ng is about
>the resulting issues.
As I said, I've done Perl coding before, as well as Tcl, C++, and
others. I've also hung around those newsgroups. I don't see
specifically unusual here indicative of a lack of robustness. It
may be because I read c.l.perl from the 4.0.38 -> 5.2 days (so
you could say it wasn't as robust then) or c.l.tcl in the 7.x days.
Wow! Google says my first post to c.l.py was in Sept. 1995 -- but
I was mostly a lurker back then.
Could you remind me what sorts of functionality you consider to
be missing? Are you talking about things like taintedness and
sandboxing? (In which case language like C++ also fit under the
category of "more difficult to write robustly", with which I'll
agree.) I've written a whole lot of Python code and I can't
think of anything in the run-time which needed to be improved to
increase robustness.
There have been improvements in the language which have improved
expressibility and maintainability, but I believe that to be a
different though related issue.
Andrew
da...@dalkescientific.com
Paul Rubin
> | The security issue with pickle.loads that we spent a long time
> | discussing is something I think the perl developers would not have
> | tolerated.
>
> could you describe the security issue in some detail?
Basically if you unpickle a string that came from an untrusted source
(say, a browser cookie from the Cookie module), the string can make
pickle load arbitrary modules and call arbitrary object constructors
in your application. The docs for the cookie module mention this and
there's an bug open on sourceforge to fix the pickle docs.
Tim Peters
> It's not intended as FUD. Other people did report similar crashes.
For vague enough values of "similar", which amounts to "something crashed, I
have no idea why, and can't correlate to anything I was doing except running
Python at the time". Perhaps I'm too used to that from rebooting Windows
ten times a day, though <0.7 wink>.
> It's conceivable that my crashes weren't caused by Python bugs, but
> signs don't point that way.
To my eye they didn't point at Python either. But when a Perl program
crashes, neither do I assume it's Perl's fault, etc.
> Comparing Python with Perl, generally I find Python better designed
> but its implementation more likely to take short cuts.
Heh -- I take it you haven't looked much at either implementation.
> The security issue with pickle.loads that we spent a long time
> discussing is something I think the perl developers would not have
> tolerated.
Can't guess, but would be very surprised if it didn't vary by developer.
Some developers think "potential exploits" are occasion for worldwide alarm,
others yawn, and most of us are somewhere in between. Overall, you appear
to be to the left of the PythonLabs mean on that scale, but we're not glued
to right-hand wall either. Barry Warsaw is inclined to try to make some
time for that one for 2.2b1 -- which was predictable, since he's the only
one of us who disables JavaScript in his browser <0.9 wink>.
> There's all kinds of other missing functionality in the runtime system
> as well, that doesn't result directly in unrobust programs, but does
> make it more difficult to write robustly.
This seems a stretch.
> A lot of this ng is about the resulting issues.
>
> I think Python is promising and exciting and I like programming in it.
> It's just inaccurate to say it's as far along in development as some
> other languages.
I didn't say it was, and I agree it isn't: in person-years devoted to its
development so far, Python is near the bottom of the pack. But the issue
was stability, and the first pre-alpha 0.9.0 Python I got from Guido was
already orders of magnitude more reliable than the production Perl 4 I was
using at the time.
My last employer used gcc and Perl and Python too: no bug reports were
filed against Python as a result of that; several bad codegen reports were
filed against gcc (not counting many filed against a new-platform gcc port);
several more against Intel's compiler; a few against MSVC; and we saved a
collection of Perl binaries and hardcoded paths to them in each script,
because we were wasting too much time "pinning the blame" on the Perl du
jour for crashes before that.
It's impossible to say whether that's "typical", but it was my experience,
and after countless hours staring at the implementations, it's consistent
with what I'd expect.
Andrew Dalke
>Basically if you unpickle a string that came from an untrusted source
>(say, a browser cookie from the Cookie module), the string can make
>pickle load arbitrary modules and call arbitrary object constructors
>in your application. The docs for the cookie module mention this and
>there's an bug open on sourceforge to fix the pickle docs.
I've been trying to figure out what Perl does to prevent this problem
in, I assume, Data::Dumper. As usual, I'm confused. It appears that
Perl has the same problems Python has, in that arbitrary Thaws methods
can be called. (See below for how creating an object, even without
calling a constructor or special thawing function, can cause problems
because the destructor might have side effects, as with
TemporaryFileWrapper.)
BTW, the line
> the string can make pickle load arbitrary modules and call arbitrary
> object constructors in your application.
should be replaced with "call arbitrary callables". For example,
here's a way to remove a file using pickle.loads:
>>> t = "(S'filename.txt'\012p1\012ios\012unlink\012p2\012(dp3\012b."
>>> import pickle
>>> open("filename.txt", "w").write("Hello\n")
>>> ^Z
Suspended
[dalke@pw600a src]$ cat filename.txt
Hello
[dalke@pw600a src]$ fg
python
>>> pickle.loads(t)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/home/dalke/local/lib/python2.0/pickle.py", line 900, in loads
return Unpickler(file).load()
File "/home/dalke/local/lib/python2.0/pickle.py", line 516, in load
dispatch[key](self)
File "/home/dalke/local/lib/python2.0/pickle.py", line 856, in load_build
inst.__dict__.update(value)
AttributeError: 'None' object has no attribute '__dict__'
>>> ^Z
Suspended
[dalke@pw600a src]$ cat filename.txt
cat: filename.txt: No such file or directory
[dalke@pw600a src]$
Is the following useful?
=============
# safe_pickle.py
import pickle
class SafeUnpickler(pickle.Unpickler):
def __init__(self, file, legit_classes = [], legit_modules = []):
pickle.Unpickler.__init__(self, file)
self.legit_classes = legit_classes
self.legit_modules = legit_modules
def find_class(self, module, name):
if module in self.legit_modules and \
name in self.legit_classes:
return pickle.Unpickler.find_class(self, module, name)
else:
raise SystemError("Tries to unpickle a soured item (%s, %s)" % \
(module, name))
def load(file, legit_classes = [], legit_modules = []):
return SafeUnpickler(file, legit_classes, legit_modules).load()
===============
>>> import cStringIO, pickle, safe_pickle
>>>
>>> class Spam:
... def __init__(sel, x, y):
... print "Called with", x, y
... def __getinitargs__(self):
... return (9, 8)
...
>>> spam = Spam(1, 2)
Called with 1 2
>>> s = pickle.dumps(spam)
>>> pickle.loads(s)
Called with 9 8
<__main__.Spam instance at 0x12034a988>
>>> safe_pickle.load(cStringIO.StringIO(s))
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "safe_pickle.py", line 18, in load
return SafeUnpickler(file, legit_classes, legit_modules).load()
File "/home/dalke/local/lib/python2.0/pickle.py", line 516, in load
dispatch[key](self)
File "/home/dalke/local/lib/python2.0/pickle.py", line 682, in load_inst
klass = self.find_class(module, name)
File "safe_pickle.py", line 14, in find_class
raise SystemError("Tries to unpickle a soured item (%s, %s)" % \
SystemError: Tries to unpickle a soured item (__main__, Spam)
>>> safe_pickle.load(cStringIO.StringIO(s), ["Spam"], ["__main__"])
Called with 9 8
<__main__.Spam instance at 0x1202ee1c8>
>>>
It's also possible to check that find_class really returns a ClassType,
and override load_obj (or load_inst or both?) so the constructor is
never called. Hmm, and what about returning the Bastionized form of
the found class?
It's still tricky as things aren't automatically safe -- what if you
unpickle a tempfile.TemporaryFileWrapper? It's possible for the
destructor there to unlink the named file. So skipping the constructor
and any deserialization function still doesn't guarantee the safeness
of unpickling.
But at least with this 'safe_pickle' you get to determine what you trust.
Andrew
da...@dalkescientific.com
Erno Kuusela
<phr-n...@nightsong.com> writes:
oh, that. but it is a deliberate design choice. if you want to take
python to task for these sorts of features, eval() or input() are much
"worse". or even marshal. i agree the pickle documentation should
mention this as the first thing in big friendly letters.
-- erno
Paul Rubin
> | Basically if you unpickle a string that came from an untrusted source
> | (say, a browser cookie from the Cookie module), the string can make
> | pickle load arbitrary modules and call arbitrary object constructors
> | in your application. The docs for the cookie module mention this and
> | there's an bug open on sourceforge to fix the pickle docs.
>
> oh, that. but it is a deliberate design choice. if you want to take
> python to task for these sorts of features, eval() or input() are much
> "worse". or even marshal. i agree the pickle documentation should
> mention this as the first thing in big friendly letters.
In fact it does the opposite--both the documentation and the pickle
implementation (look at the "security" check for pickled strings)
appear written with the idea that unickling is intended to be safe for
untrusted strings. If you look at docs for the Cookie module, you see
that it originally used pickle in precisely this dangerous way. If
that got past the maintainers who allowed the Cookie module to be
shipped with the Python library, what chance do most ordinary users
have?
Re marhsal: is it dangerous to unmarshal an untrusted string? Yes, I
know that marshalled strings can contain code objects, but
unmarshalling the strings doesn't actually RUN such code objects, as
far as I know. I haven't examined marshal super-carefully though.
I think that input() evalling the stuff it reads is also a poor design
choice, but at least it's documented.
Erno Kuusela
<phr-n...@nightsong.com> writes:
| In fact it does the opposite--both the documentation and the pickle
| implementation (look at the "security" check for pickled strings)
| appear written with the idea that unickling is intended to be safe for
| untrusted strings.
i can't see that idea in the documentation even if i try.
i don't know what the "security" check in the source code is about,
perhaps the pickle author was planning to make it safe for untrusted
data at some point.
the fact that pickle shouldn't be fed untrusted data has been common
knowledge in the python user and developer communities as long as i
can remember.
| If you look at docs for the Cookie module, you see
| that it originally used pickle in precisely this dangerous way. If
| that got past the maintainers who allowed the Cookie module to be
| shipped with the Python library, what chance do most ordinary users
| have?
iirc the cookie module had this erroneous code before it was accepted
into the library.
when it was put in the library, warnings were put in its documentation
to warn against anyone ever using the functionality.
| Re marhsal: is it dangerous to unmarshal an untrusted string? Yes, I
| know that marshalled strings can contain code objects, but
| unmarshalling the strings doesn't actually RUN such code objects, as
| far as I know. I haven't examined marshal super-carefully though.
hmm, you may be right.
-- erno
Tim Hammerquist
[ snip ]
> Good point! Based on that, here is my Python "Enhancement" proposal:
>
> Let's make it so that placing "option disturbing" at the start of a
> module turns on "C Mode". In this mode, Python's behaviour would be
> modified in the following ways:
>
> * Use of curly braces to delineate syntax blocks, rather than
> indentation;
> * The C ternary operator would be fully supported, as would
> fall-through "switch" statements (as long as the type being
> switch()ed is some kind of number, of course);
> * Class support would be removed, but everyone knows all you
> need is support for struct{} anyway. Structs could only
> contain the types "IntType", "FloatType" and "StringType",
> but when you get down to it it's all numbers anyway;
> * We'd finally get our beloved "goto" statement.
>
> Fortunately, none of this will affect the "Python-ness" of the language
> - without "option disturbing", it will behave exactly as before!
What a cruel, cruel joke. Have you no shame?! ;)
Tim Hammerquist
--
It takes a smart husband to have the last word and not use it.
Paul Rubin
> | In fact it does the opposite--both the documentation and the pickle
> | implementation (look at the "security" check for pickled strings)
> | appear written with the idea that unickling is intended to be safe for
> | untrusted strings.
>
> i can't see that idea in the documentation even if i try.
The pickle docs mention the non-pickle-ability of code objects as a
security advantage of pickle over marshal. Clearly the doc writer
wasn't aware that unpickling is insecure for other reasons. Also, the
docs for marshal recommend using pickle instead of marshal for RPC.
Any pickle-based RPC server exposed to the internet (see for example
pyro.sourceforge.net) would of course have a security hole.
> i don't know what the "security" check in the source code is about,
> perhaps the pickle author was planning to make it safe for untrusted
> data at some point.
More likely the original version of pickle was intended to be safe
for untrusted data, and the __getinitargs__ scheme was added later
without considering the security implications.
> the fact that pickle shouldn't be fed untrusted data has been common
> knowledge in the python user and developer communities as long as i
> can remember.
It sure shocked the heck out of me! I discovered it as a result of
making a sourceforge bug report (#467384) requesting that marshal be
documented so it could be used for RPC, and Tim suggested I use pickle
instead. So even Tim (one of the main authorities on Python's
implementateion) wasn't aware of the problem at the time. The authors
of the Cookie and Pyro modules weren't aware of it either. So this
type of "common knowledge" needs, at the very least, to be clearly
documented! It's not reasonable to expect programmers of a supposedly
easy to learn language to absorb all kinds of unwritten folklore
before they can safely write as common an application as a simple cgi
script.
> iirc the cookie module had this erroneous code before it was accepted
> into the library.
>
> when it was put in the library, warnings were put in its documentation
> to warn against anyone ever using the functionality.
However, "Cookie" is still aliased to SmartCookie in the module,
supposedly for backward compatibility. That is, to leave a security
hole in any existing servers using the module. SmartCookie and
SerialCookie should be disabled altogether. If the holes were known
before the module was accepted to the library, that's even worse. It
should never have been accepted with holes like that.
Andrew Dalke
>the fact that pickle shouldn't be fed untrusted data has been common
>knowledge in the python user and developer communities as long as i
>can remember.
I've been involved with c.l.py for 6 years. I recall reading the
documentation that pickles don't save code objects so should be
safer than marshalls for sending data between intelligent agents.
I had always assumed it was safe. It wasn't until this thread
came up that I knew there was a problem. (Eg, elsewhere I posted
a pickle that could be used to remove an arbitrary file.)
So I don't think it's common enough. Now I need to revisit how
I've done some of my quick&dirty network protocols (pickles over
an http session) as I now know it's highly insecure for both
the client and the server.
Andrew
da...@dalkescientific.com
A. Keyton Weissinger
any other app server that uses cookie-based state management)? I need to set
up an app that calls a security JSP page, logs in, and then calls a second
page, sending the cookie it received.
Surely someone has done this....
Any help would be MOST appreciated.
Thank you.
Keyton
Tim Peters
> I've been involved with c.l.py for 6 years. I recall reading the
> documentation that pickles don't save code objects so should be
> safer than marshalls for sending data between intelligent agents.
>
> I had always assumed it was safe. It wasn't until this thread
> came up that I knew there was a problem. (Eg, elsewhere I posted
> a pickle that could be used to remove an arbitrary file.)
Oddly enough, cPickle rejects that pickle (it only deletes the file when
using pickle.py).
> So I don't think it's common enough. Now I need to revisit how
> I've done some of my quick&dirty network protocols (pickles over
> an http session) as I now know it's highly insecure for both
> the client and the server.
On the bright side of darkness <wink>, not all of cPickle got documented.
We'll try to repair that for 2.2. In particular, you can set a find_global
attribute on a cPickle Unpickler to supply your own function for resolving
names, and of special interest is that if you set it to None, the Unpickler
won't import modules or unpickle class instances at all. You could, of
course, also supply a function that accetps only a set of blessed names
(etc).
hiding-security-is-our-middle-name-ly y'rs - tim
Paul Rubin
> On the bright side of darkness <wink>, not all of cPickle got
> documented. We'll try to repair that for 2.2. In particular, you
> can set a find_global attribute on a cPickle Unpickler to supply
> your own function for resolving names, and of special interest is
> that if you set it to None, the Unpickler won't import modules or
> unpickle class instances at all. You could, of course, also supply
> a function that accetps only a set of blessed names (etc).
Great! This sounds like a pretty flexible way to fix the problem, so
it's cool that the code was already there for it.
Erno Kuusela
<phr-n...@nightsong.com> writes:
| Erno Kuusela <erno...@erno.iki.fi> writes:
|| | In fact it does the opposite--both the documentation and the pickle
|| | implementation (look at the "security" check for pickled strings)
|| | appear written with the idea that unickling is intended to be safe for
|| | untrusted strings.
||
|| i can't see that idea in the documentation even if i try.
| The pickle docs mention the non-pickle-ability of code objects as a
| security advantage of pickle over marshal. Clearly the doc writer
| wasn't aware that unpickling is insecure for other reasons. Also, the
| docs for marshal recommend using pickle instead of marshal for RPC.
i see it now. quite amazing!
|| the fact that pickle shouldn't be fed untrusted data has been common
|| knowledge in the python user and developer communities as long as i
|| can remember.
| It sure shocked the heck out of me! I discovered it as a result of
| making a sourceforge bug report (#467384) requesting that marshal be
| documented so it could be used for RPC, and Tim suggested I use pickle
| instead. So even Tim (one of the main authorities on Python's
| implementateion) wasn't aware of the problem at the time. The authors
| of the Cookie and Pyro modules weren't aware of it either. So this
| type of "common knowledge" needs, at the very least, to be clearly
| documented! It's not reasonable to expect programmers of a supposedly
| easy to learn language to absorb all kinds of unwritten folklore
| before they can safely write as common an application as a simple cgi
| script.
indeed. it seems this common knowledge was more local to my friends
than i thought.
guido seems to have known about it long ago though
(the google archives only go back to 1995):
<URL: http://groups.google.com/groups?
selm=199608222029.QAA08777%40monty&output=gplain>
|| iirc the cookie module had this erroneous code before it was accepted
|| into the library.
||
|| when it was put in the library, warnings were put in its documentation
|| to warn against anyone ever using the functionality.
| However, "Cookie" is still aliased to SmartCookie in the module,
| supposedly for backward compatibility. [...]
i'd guess this is a bug.
-- erno
Andrew Dalke
>Oddly enough, cPickle rejects that pickle [which passes an arbitrary
>string to os.remove] (it only deletes the file when
>using pickle.py).
I assume it's checking for a ClassType. By testing, yes.
The following is a pickle of
tempfile.TemporaryFileWrapper(StringIO.StringIO(""), "remove.me")
"(itempfile\012TemporaryFileWrapper\012p1\012(dp2\012S'file'" + \
"\012p3\012(iStringIO\012StringIO\012(dp4\012S'closed'\012p5" + \
"\012I0\012sS'len'\012p6\012I0\012sS'buf'\012p7\012S''\012sS'" + \
"buflist'\012p8\012(lsS'softspace'\012p9\012I0\012sS'pos'\012" + \
"p10\012I0\012sbsS'path'\012p11\012S'remove.me'\012sb."
[dalke@pw600a ~]$ touch remove.me
[dalke@pw600a ~]$ cat remove.me
[dalke@pw600a ~]$ ./python
Python 2.2b1+ (#8, Nov 8 2001, 20:45:09)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s = "(itempfile\012TemporaryFileWrapper\012p1\012(dp2\012S'file'" + \
... "\012p3\012(iStringIO\012StringIO\012(dp4\012S'closed'\012p5" + \
... "\012I0\012sS'len'\012p6\012I0\012sS'buf'\012p7\012S''\012sS'" + \
... "buflist'\012p8\012(lsS'softspace'\012p9\012I0\012sS'pos'\012" + \
... "p10\012I0\012sbsS'path'\012p11\012S'remove.me'\012sb."
>>> import cPickle
>>> cPickle.loads(s)
<tempfile.TemporaryFileWrapper instance at 0x1203492d8>
>>> 0
0
>>> ^D
[dalke@pw600a ~]$ cat remove.me
cat: remove.me: No such file or directory
[dalke@pw600a ~]$
The '0' is to force '_' to get rid of the reference to a
TemporaryFileWrapper, whose destructor closes the StringIO then
removes the given filename, which is 'remove.me'
>We'll try to repair that for 2.2. In particular, you can set a find_global
>attribute on a cPickle Unpickler to supply your own function for resolving
>names, and of special interest is that if you set it to None, the Unpickler
>won't import modules or unpickle class instances at all. You could, of
>course, also supply a function that accetps only a set of blessed names
>(etc).
Sure, and it was doable in pickle.py as well by making a subclass
and overriding the find_class method (example shown elsewhere).
But I didn't know I needed to do that, and it's hard to figure out
which classes are okay to pickle safely. Some of my own classes
have these sorts of filesystem cleanup code in __del__. So
pickles are hard to use safely for insecure-but-non-trivial data.
Hope this is all documented in nice friendly letters.
Andrew
da...@dalkescientific.com
Barry A. Warsaw
Just to follow up on this thread while I'm waiting for the Python
2.2b2 test to complete...
I've rewritten the pickle and cPickle documentation hopefully to both
update them to the current state of the modules (they were woefully
out-of-date and underdocumented), and to describe the security
implications and how you might program defensively to avoid them.
I did an update of copy_reg and marshal as well, but didn't get a
chance to look at the Cookie docs. I'll do that before the next
release.
My own feeling is that, with enough pain, you could create unpicklers
that are safe(r), but that to do so requires different approaches
depending on whether you use pickle or cPickle, and exactly what
you're trying to protect against!
I don't see us making any substantive code changes for Python 2.2,
including not changing the alias for Cookie.Cookie. I'd like to see a
pickle PEP for Python 2.3 which would design a better protocol for
letting the unpickling environment control exactly what gets called
and what instances can be created.
Also, I'd like to see pickle.py go away eventually, to be replaced
with cPickle. The only reason pickle still exists is for subclassing;
I'd like to tease out why people need to subclass pickle.Pickler and
pickle.Unpickler and see if we can't provide similar functionality in
a different way through cPickle. There's also the possibility of
using new-style classes in cPickle to still allow subclassing.
Anyway, I think we've addressed things as best we can for Python 2.2.
Please take a look at the new documentation and let me know if you
have any corrections or other suggestions.
-Barry
Barry A. Warsaw
>>>>> "TP" == Tim Peters <tim...@home.com> writes:
TP> On the bright side of darkness <wink>, not all of cPickle got
TP> documented. We'll try to repair that for 2.2. In particular,
TP> you can set a find_global attribute on a cPickle Unpickler to
TP> supply your own function for resolving names, and of special
TP> interest is that if you set it to None, the Unpickler won't
TP> import modules or unpickle class instances at all. You could,
TP> of course, also supply a function that accetps only a set of
TP> blessed names (etc).
And pickle users can override the load_global() method on the
Unpickler to do something similar, although it's a bit more painful.
All documented in the freshly minted pickle docs for 2.2b2. :)
-Barry
Barry A. Warsaw
[Andrew posts a cute way to trick cPickle into deleting a file...]
Note that pickle isn't so easily tricked, but only because of an
implementation accident in TemporaryFileWrapper. This was a fun one
that had Tim and me stumped for a bit while we tried to figure out
exactly what was going on.
Turns out that if you use pickle for this example, you'll get an
exception during loads() because TemporaryFileWrapper implements a
__getattr__() that fails when the Unpickler tries to see if the object
has a __setstate__ attribute. This is because the instance's __dict__
doesn't yet have a 'file' key, so you get a KeyError at that point.
cPickle is vulnerable because it masks all exceptions during the
search for inst.__setstate__; if that fails it just interprets that to
mean that it had no __setstate__ attribute and continues onward.
pickle OTOH only catches AttributeErrors so the KeyError shorts out
pickle.loads() and thus your TemporaryFileWrapper instance never gets
a chance to run its __del__().
AD> Sure, and it was doable in pickle.py as well by making a
AD> subclass and overriding the find_class method (example shown
AD> elsewhere). But I didn't know I needed to do that, and it's
AD> hard to figure out which classes are okay to pickle safely.
AD> Some of my own classes have these sorts of filesystem cleanup
AD> code in __del__. So pickles are hard to use safely for
AD> insecure-but-non-trivial data.
Indeed. I'm curious on how we can systematically make it better, but
that's for another day, hopefully to be captured in a pickle PEP for
2.3.
sniffing-pickles-is-bad-for-your-health-ly y'rs,
-Barry
Paul Rubin
> Anyway, I think we've addressed things as best we can for Python 2.2.
> Please take a look at the new documentation and let me know if you
> have any corrections or other suggestions.
Barry, I can't find the new documentation. Can you say where to look?
Thanks.
Warren Postma
I felt the same way when I first encountered Python. After I had spent a
little more time with it,
and had read the C code implementing the various types, I realized the
Necessity and Completeness of Python, which in a certain way, is the most
harmonious convergence of theoretical Perfection and practical Usability of
any language I have yet encountered.
Option Explicit would wreck all that.
Now, I've always hated VB, but I had not thought of why, until I started
using Python.
Now that I have decided I really like Python, I explain my choices of
Language as follows:
(1) C is a great language, because its the only language Python could have
been implemented in, and be both portable to a wide set of platforms, yet
provide acceptably high performance, and acceptable levels of OS-specific
integration. Nevertheless, without Python, I wouldn't want to try to
manipulate polymorphic hierachical data sets in bare C or C++. The object
model of C and C++ is Insufficient. Obviously a layer on top is necessary
for that. That layer is Python.
(2) Python is a great language, but I wouldn't want to write a C compiler
using it. Well, not just yet, anyways. <grin>
(3) Visual Basic, at least until Visual Studio.NET is not capable of either
#1 or #2, and it remains to be seen if .NET can reverse that. (VB.net will
be the first truly Object Oriented version of VB, but will probably still
not be fully dynamic like Python.)
Warren Postma
Jeffrey Drake
only thing VB doesn't support is implementation inheritance. This
isn't necessarily needed though to be classified as OO because it does
support Interface inheritance.
Regards,
Jeffrey Drake
Jeffrey Drake
only thing VB doesn't support is implementation inheritance. This
isn't necessarily needed though to be classified as OO because it does
support Interface inheritance.
Regards,
Jeffrey Drake
>
>(3) Visual Basic, at least until Visual Studio.NET is not capable of either
>#1 or #2, and it remains to be seen if .NET can reverse that. (VB.net will
>be the first truly Object Oriented version of VB, but will probably still
>not be fully dynamic like Python.)
>
>
>
>Warren Postma
>
>
========= WAS CANCELLED BY =======:
Path: news.sol.net!spool1-nwblwi.newsops.execpc.com!newsfeeds.sol.net!news-out.visi.com!hermes.visi.com!news.maxwell.syr.edu!feeder.kornet.net!news1.kornet.net!ua4canc3ll3r
From: jp...@rogers.com (Jeffrey Drake)
Newsgroups: comp.lang.python
Subject: cmsg cancel <3c2ed3d9.18033070@nntp>
Control: cancel <3c2ed3d9.18033070@nntp>
Date: Mon, 31 Dec 2001 03:12:21 GMT
Organization: A poorly-installed InterNetNews site
Lines: 2
Message-ID: <cancel.3c2ed3d9.18033070@nntp>
NNTP-Posting-Host: 211.57.49.2
X-Trace: news2.kornet.net 1009774300 27193 211.57.49.2 (31 Dec 2001 04:51:40 GMT)
X-Complaints-To: use...@news2.kornet.net
NNTP-Posting-Date: Mon, 31 Dec 2001 04:51:40 +0000 (UTC)
X-No-Archive: yes
X-Unac4ncel: yes
X-Commentary: I love NewsAgent 1.10 and the Sandblaster Cancel Engine Build 74 (19 March 1999)
This message was cancelled from within Mozilla.