Efficient python programming...
Shagshag13
I'm looking for any state of art, optimizing techniques, tips, that a
python beginner couldn't know by itself, but should learn and
would be aware to write "efficient and good looking"python's code...
(not the kind of thing which made you an obfuscated python nerd,
but that "map" should be preferred to "for in loop" if possible, that
dictionary are very well optimized, etc.)
Thanks,
s13.
Emile van Sebille
I think the advise you'll generally get is to write first, then profile,
then optimize where needed. Those optimizations may well involve using
map, or numpy, or c-extensions, etc... But emphasis on optimization
first is probably time mis-spent.
--
Emile van Sebille
em...@fenx.com
---------
François Pinard
> I'm looking for any state of art, optimizing techniques, tips, that a
> python beginner couldn't know by itself, but should learn and would be
> aware to write "efficient and good looking"python's code...
There is a lot of Python code all around to look it, including all the
Python which comes with the Python distribution itself. I guess you will
quickly get used to grasp in a glimpse which code is worth looking and
which looks less interesting.
One feels when code smells written by people having good taste and
experience. There is practical principle that people generally who do
well in many aspect of their work, which seem good to you right now,
are likely to do equally well in the aspects which are more new to you.
Also, on the Python site, there are a few essays written by Guido van
Rossum about how one writes and optimises, and on Python in general,
and which I read soon after starting to learn Python. I found these to
be well written, quite instructive, and very welcome at the time.
--
François Pinard http://www.iro.umontreal.ca/~pinard
gabor
so you need general programming advices?
then by a book about programming or something like that....
but if you want to know about specific python issues, then you have to
learn things like "dictionary are very well optimized etc.."
those things doesn't make your code obfuscated..
for file in directory:
file.printsize()
ran = range(len(directory))
for r in ran:
directory[r].printsize()
[file.printsize() for file in directory]
they're the same, and everyone would recommend you the last one..
it's python specific, but it won't make your code obfuscated..
so to summarize this: what do you want to know/learn?
bye,
gabor
William Park
>
> python beginner couldn't know by itself, but should learn and
> would be aware to write "efficient and good looking"python's code...
Actually, I don't recommend this approach. I recommend you look at your
algorithm and try to translate that to Python syntax. Optimization should
occur at the modelling and algorithm side of things.
>
> (not the kind of thing which made you an obfuscated python nerd,
> but that "map" should be preferred to "for in loop" if possible, that
> dictionary are very well optimized, etc.)
--
William Park, Open Geometry Consulting, <openge...@yahoo.ca>
8-CPU Cluster, Hosting, NAS, Linux, LaTeX, python, vim, mutt, tin
Sean 'Shaleh' Perry
the consensus on this list over the last year seems to be that optimized code is
usually the simplest implementation. If you feel you are going through
hoops getting something working you probably need to take a step back, ponder
and maybe ask a question.
BTW tu...@python.org is a great place for new python coders to hang out, read,
and learn.
Shagshag13
"gabor" <ga...@realtime.sk> a écrit dans le message de news: mailman.1023401650...@python.org...
> but if you want to know about specific python issues, then you have to
> learn things like "dictionary are very well optimized etc.."
That's what i'm asking : learning and understanding... but where should i go ???
I'm wondering if there is a place to go to find "why this solution is better in this
appropriate case... ". For example, i was looking for the best pythoner's way to
remove duplicate in a list, and after reading many posts, i find at the end of a
cookbook recipe that it was :
set = {}
map(set.__setitem__, alist, [])
alist = set.keys()
But that's not trivial !!! And quite difficult to find...
> those things doesn't make your code obfuscated..
I agree :)
> [file.printsize() for file in directory]
>
> they're the same, and everyone would recommend you the last one..
Yes but i would have asked why this is the best... And i think that
your answer would have helped me to become a better pythoner...
I mean there should be a place to go where you could read generalities
like "as dictionary are very well optimized, you can use it without
trouble and for example it's best in : removing duplicates, ..." or
"if you need to do an operation on a list use map because it's
c-optimized", there are "object overhead if you...", and so on
that's not in the books, that's not in 'dive in python',
it's spreaded in Guido's essays, and posts over there... But i worry,
you could miss it if you don't spend many time to search, and say
"there should be a place for that"...
> so to summarize this: what do you want to know/learn?
What a beginner should learn to do it the pythoner way...
s13.
ps : and sorry for my poor english and typos...
Eddie Corns
A friend is fond of quoting this to me:
Rules of Optimization:
Rule 1: Don't do it.
Rule 2 (for experts only): Don't do it yet.
-- M.A. Jackson
I wasn't going to bother contributing since others have said the same thing
but the more people say the more you'll believe it :)
Especially for a beginner the important thing is to get clear and obvious
code. Later, IF IT ISN'T FAST ENOUGH, then you think about ways to speed it
up.
There are may reasons for this, one of them is that it is often counter
intuitive exactly what the fastest approach is anyway.
Just concentrate on finding sensible algorithms as you would in any other
procedural language.
Eddie
Peter Hansen
>
> A friend is fond of quoting this to me:
>
> Rules of Optimization:
> Rule 1: Don't do it.
> Rule 2 (for experts only): Don't do it yet.
> -- M.A. Jackson
>
> I wasn't going to bother contributing since others have said the same thing
> but the more people say the more you'll believe it :)
>
> Especially for a beginner the important thing is to get clear and obvious
> code. Later, IF IT ISN'T FAST ENOUGH, then you think about ways to speed it
> up.
You forgot the even more important first thing for a beginner:
get it correct! The only good way to do this is to write unit
tests that prove it. If you don't have tests, you don't have
working code (or how can you prove it?).
-Peter
Emile van Sebille
> You forgot the even more important first thing for a beginner:
> get it correct! The only good way to do this is to write unit
> tests that prove it. If you don't have tests, you don't have
> working code (or how can you prove it?).
We always used absence of bug reports as a gauge... and always
questioned if that indicated usage or correctness. ;-)
Peter Hansen
>
> Peter Hansen
> > You forgot the even more important first thing for a beginner:
> > get it correct! The only good way to do this is to write unit
> > tests that prove it. If you don't have tests, you don't have
> > working code (or how can you prove it?).
>
> We always used absence of bug reports as a gauge... and always
> questioned if that indicated usage or correctness. ;-)
For those who are serious when they say that, it's all well
and good if you can get your customer to test the code for
you.... ;-)
-Peter
Emile van Sebille
news:3D00ABB6...@engcorp.com...
Even better, as most of the time they're not sure what they asked for in
the first place, blurring the line separating implementation error and
user specification error. ;-))
(we really did think like that then -- for some set of we that includes
me anyway)
holger krekel
yeah. writing tests is *the way* to go if you want other people use
your code. Although successfull unittests theoretically don't prove
anything<wink> they tend to do in reality<doublewink>.
if-it's-not-proven-then-test-it-ly yours, holger
Roman Suzi
To prove that code is correct is very difficult (practically impossible).
Unittests are there to prove that you have errors, not the other way
around.
Of course, they can increase your confidence in that optimized code is
equivalent to what you had before.
> -Peter
Sincerely yours, Roman A.Suzi
--
- Petrozavodsk - Karelia - Russia - mailto:r...@onego.ru -
Steve Holden
news:adpktk$17j3k$1...@ID-146704.news.dfncis.de...
>
> "gabor" <ga...@realtime.sk> a écrit dans le message de news:
mailman.1023401650...@python.org...
>
> > but if you want to know about specific python issues, then you have to
> > learn things like "dictionary are very well optimized etc.."
>
> That's what i'm asking : learning and understanding... but where should i
go ???
>
to be your best guides. I am a much better programmer now than I was thirty
years ago. Why? Because I've done things wrong, and corrected my errors.
I've repeated the same tasks many times, and found out by practical
experience what works best (for me) and what doesn't. The meat between your
ears will provide you with important knowledge if you take the time to
analyse your experience. If you don't then you should remember that "those
who cannot remember history are doomed to repeat it".
> I'm wondering if there is a place to go to find "why this solution is
better in this
> appropriate case... ". For example, i was looking for the best pythoner's
way to
> remove duplicate in a list, and after reading many posts, i find at the
end of a
> cookbook recipe that it was :
>
> set = {}
> map(set.__setitem__, alist, [])
> alist = set.keys()
>
> But that's not trivial !!! And quite difficult to find...
>
... and not always appropriate. One of the things experience will teach you
is that when your list only contains ten elements the more obvious solutions
are to be preferred, since they will execute in maybe 100 microseconds
instead of 63.
> > those things doesn't make your code obfuscated..
>
> I agree :)
>
> > [file.printsize() for file in directory]
> >
> > they're the same, and everyone would recommend you the last one..
>
> Yes but i would have asked why this is the best... And i think that
> your answer would have helped me to become a better pythoner...
>
Of course, comp.lang.python itself is a good place to ask questions like
that. The more trivial the question, the more people will have their own
opinions, of course, so you will still need to use your intelligence to make
sense of the profusion of replies.
> I mean there should be a place to go where you could read generalities
> like "as dictionary are very well optimized, you can use it without
> trouble and for example it's best in : removing duplicates, ..." or
> "if you need to do an operation on a list use map because it's
> c-optimized", there are "object overhead if you...", and so on
> that's not in the books, that's not in 'dive in python',
> it's spreaded in Guido's essays, and posts over there... But i worry,
> you could miss it if you don't spend many time to search, and say
> "there should be a place for that"...
>
Perhaps you should think about writing a book? I did, and I know there are
lots of things I would have liked to include but didn't have space and/or
time.
> > so to summarize this: what do you want to know/learn?
>
> What a beginner should learn to do it the pythoner way...
>
> s13.
>
> ps : and sorry for my poor english and typos...
>
Don't worry about that. Your concern for knowing these things suggests that
you will be a good Python programmer. But the most important part of
becoming a good Python programmer is becoming a good programmer. Knowing the
right question sto ask is as important as getting the answers, so you are
already a good way down the path. Many of the required skills aren't
specific to Python...
regards
Steve
--
-----------------------------------------------------------------------
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/pwp/
-----------------------------------------------------------------------
Brian McErlean
>
> for file in directory:
> file.printsize()
>
>
> ran = range(len(directory))
> for r in ran:
> directory[r].printsize()
>
> [file.printsize() for file in directory]
>
> they're the same, and everyone would recommend you the last one..
I wouldn't. The first one is better in this case. Using the list
comprehension implys to me that you're interested in the result -
while in fact you're discarding it. If there are a lot of files in
the directory (And after all, thats the only reason to optimise it),
then you'll end up with a large list of [None,None,None,...] that is
completely unneeded.
testing gives:
[foo(i) for i in range(100000)]
- took 0.240000009537 secs
for i in range(100000): foo(i)
- took 0.149999976158 secs
(foo(i) is just an empty function)
Using the list comprehension here is both misleading (because you are
relying on a side-effect, not using the returned list), and less
efficient. Using the for loop is better, because it makes clear that
you're using an imperative technique, relying on the side-effect
(printing the size) and are uninterested in building a list at all.
On the other hand, if the code is:
l=[]
for file in directory:
l.append(file.getsize())
It is both clearer, and more efficient if implemented as:
l= [file.getsize() for file in directory]
Brian.
Daniel Fackrell
news:adpktk$17j3k$1...@ID-146704.news.dfncis.de...
...
> set = {}
> map(set.__setitem__, alist, [])
> alist = set.keys()
>
> But that's not trivial !!! And quite difficult to find...
The reason that this example does not seem trivial to you or to me is most
likely that we don't fully grasp the concept and usage of map(). Once we
get that behind us, I'm sure that this example will turn into something that
we use all the time without much thinking about it.
>
> > those things doesn't make your code obfuscated..
>
> I agree :)
>
> > [file.printsize() for file in directory]
> >
> > they're the same, and everyone would recommend you the last one..
>
> Yes but i would have asked why this is the best... And i think that
> your answer would have helped me to become a better pythoner...
This concept (I believe it's called "list comprehensions") is another like
the map() usage. Obscure and arcane to those who don't fully grasp it (like
me, again), but very powerful. I'm not sure what either of them has to do
with optimization, though, except that they are both probably very well
optimized at the underlying C level.
Daniel Fackrell
James Kew
> Eddie Corns wrote:
> >
> > A friend is fond of quoting this to me:
> >
> > Rules of Optimization:
> > Rule 1: Don't do it.
> > Rule 2 (for experts only): Don't do it yet.
> > -- M.A. Jackson
> >
> get it correct!
I favour Kent Beck's aphorism: "Make it work, make it right, make it fast."
James
James Kew
> file.printsize()
>
>
> ran = range(len(directory))
> for r in ran:
> directory[r].printsize()
>
> [file.printsize() for file in directory]
>
> they're the same, and everyone would recommend you the last one..
Well, as a relative newbie, I'd recommend the first as much more
comprehensible and maintainable by whoever inherits the code.
The explicit loop clearly describes, in the code itself, the intent of the
code. Both the alternatives exercise darker corners of the language and are
less immediately descriptive, at least to my inexperienced eyes.
I'm a little uneasy about the list comprehension: it feels like it almost
abuses the spirit of list comprehensions by using an expression with
side-effects and then discarding the result of the comprehension.
Finally, "printsize" as a method name suggests to me that the loop is more
likely to be IO-bound than CPU-bound, so attempts to optimise the loop
overhead may buy very little benefit.
If it were me, I'd go with the first option for clarity; leave it unless and
until a performance problem emerges and profiling points at this loop;
consider algorithmic changes first (possibly merge the multiple IO
operations into a single larger one?); and only then start sacrificing
clarity for speed.
James
Peter Hansen
>
> Peter Hansen wrote:
> > You forgot the even more important first thing for a beginner:
> > get it correct! The only good way to do this is to write unit
> > tests that prove it. If you don't have tests, you don't have
> > working code (or how can you prove it?).
>
> yeah. writing tests is *the way* to go if you want other people use
> your code. Although successfull unittests theoretically don't prove
> anything<wink> they tend to do in reality<doublewink>.
Could you explain why you think unittests don't prove anything
when they are successful?
Are you pointing out that if you write a test after the code
and the test passes, you have learned almost nothing? I agree
with that.
With Test-Driven Development, however, you write the test first,
run it immediately, and ensure that it fails. Only after you
have code that makes it pass are you done.
In this case, making sure the test fails when the code is wrong,
wouldn't you say that the test mostly proves that the code works?
(Not quibbling about the fact you are still not 100% sure...)
-Peter
Peter Hansen
I disagree, if the tests are written first (and run, proving that
they will fail when the code is wrong or missing).
The tests are executable requirement specifications. Running
them "proves" (yes, not 100%... nothing could) that the code
meets the specs. The specs could be wrong of course, but that
doesn't mean the code is broken...
-Peter
Sean 'Shaleh' Perry
> I disagree, if the tests are written first (and run, proving that
> they will fail when the code is wrong or missing).
>
> The tests are executable requirement specifications. Running
> them "proves" (yes, not 100%... nothing could) that the code
> meets the specs. The specs could be wrong of course, but that
> doesn't mean the code is broken...
>
this also assumes the test covers every possible failure. this also assumes
one can write tests for that section of code.
holger krekel
> holger krekel wrote:
> > Peter Hansen wrote:
> > > You forgot the even more important first thing for a beginner:
> > > get it correct! The only good way to do this is to write unit
> > > tests that prove it. If you don't have tests, you don't have
> > > working code (or how can you prove it?).
> >
> > yeah. writing tests is *the way* to go if you want other people use
> > your code. Although successfull unittests theoretically don't prove
> > anything<wink> they tend to do in reality<doublewink>.
>
> Could you explain why you think unittests don't prove anything
> when they are successful?
I meant to play with the academic notion ('theoretically') of
proving algorithms to be correct. From a stricter perspective
unittests are just examples and examples do not prove anything.
But 'in reality' they do prove that intended usage of an API
basically works and nothing broke while fixing bugs or adding
new features.
> Are you pointing out that if you write a test after the code
> and the test passes, you have learned almost nothing? I agree
> with that.
>
> With Test-Driven Development, however, you write the test first,
> run it immediately, and ensure that it fails. Only after you
> have code that makes it pass are you done.
>
> In this case, making sure the test fails when the code is wrong,
> wouldn't you say that the test mostly proves that the code works?
*Mostly*, yes. Unfortunately mathematicians impose a boolean
view regarding proofs on us poor coders: proven or not proven.
> (Not quibbling about the fact you are still not 100% sure...)
which was the whole point of my posting :-)
holger
Jimmy Retzlaff
> I disagree, if the tests are written first (and run, proving that
> they will fail when the code is wrong or missing).
I think the other perspective here is that the writing of the unit tests
is design/coding, and hence subject to the same types of mistakes as the
designing/writing of the code they test. A contrived example:
def addEmUp(x, y):
return x * y
def testAddEmUp():
# try some limit cases and something in between
for testCase in (0, 2, 1.79e+308):
assert addEmUp(testCase, testCase) == testCase+testCase
if __name__ == '__main__':
testAddEmUp()
print 'Success!!!'
That test runs successfully on Python 2.2.1 on Windows despite the
flawed implementation of the function the test is exercising. Having
unit tests may make you more confident in correctness than not having
them, but they are still prone to mistakes themselves. In order for unit
tests to prove the code is correct, you'd need to prove the unit tests
are correct and exhaustive (despite its correctness, the above unit test
is not exhaustive which leads to a false validation).
> The tests are executable requirement specifications. Running
> them "proves" (yes, not 100%... nothing could) that the code
> meets the specs. The specs could be wrong of course, but that
> doesn't mean the code is broken...
You can say the tests are your spec and hence your program is correct
with regards to the spec, but that doesn't make the user any happier if
your code costs them time, money, safety, etc. just because the part of
the code you call the spec was also flawed or incomplete.
Unit testing is just one of many important tools. On any significant
project, it is important to have a well-rounded quality strategy that
also employs things like design/code reviews, pair programming, user
testing, etc.
But if you're really after correctness, there may be only one true
measure - let a sales person demo the product! Good old Murphy ensures
that bugs are very good at exposing themselves during important sales
presentations. :)
Jimmy
Peter Hansen
And all these sorts of things are the direct focus of test-driven
development. For example, on the issue of testability: code developed
this way is inherently testable because the test was written first,
and the code had to be written to pass the test. Code developed in
a more traditional way is much less likely to be testable and often
ends up tightly coupled and very hard to test or maintain as a result.
This, of course, tends to lead to having no tests at all, which is
a terrible thing.
I guess I was just taking a moment to point out there are some new
approaches which are more useful than the traditional (and generally
neglected) approach of writing a handful of tests after the fact.
I was also rebelling momentarily against the idea of shipping code
to a customer without having a suite of unit tests in place to give
confidence in the code.
I'll step back from any claim that unit tests really prove anything,
but in a very Pythonic and pragmatic fashion they are Good Enough to
make them a /necessity/ for anyone trying to deliver reliable software.
-Peter
Dmitri I GOULIAEV
On Fri, Jun 07, 2002 at 07:20:17PM -0400, Peter Hansen wrote:
> Roman Suzi wrote:
> >
> > On Fri, 7 Jun 2002, Peter Hansen wrote:
> > > You forgot the even more important first thing for a beginner:
> > > get it correct! The only good way to do this is to write unit
> > > tests that prove it. If you don't have tests, you don't have
> > > working code (or how can you prove it?).
> >
> > To prove that code is correct is very difficult (practically impossible).
> > Unittests are there to prove that you have errors, not the other way
> > around.
> >
> > Of course, they can increase your confidence in that optimized code is
> > equivalent to what you had before.
>
> I disagree, if the tests are written first (and run, proving that
> they will fail when the code is wrong or missing).
That's what Roman said. If you have the results of tests for your code and your optiimazed code, you show only that in regard of THIS tests your code remains the same ("increase your confidence in that optimized code is equivalent to what you had before").
> The tests are executable requirement specifications. Running
> them "proves" (yes, not 100%... nothing could) that the code
> meets the specs. The specs could be wrong of course, but that
> doesn't mean the code is broken...
... if you can write such tests (I mean the tests covering the specifications). In most case, all you can "prove" is that for this particular tests you obtain this particular results.
Best regards,
--
DIG (Dmitri I GOULIAEV)
All below this line is added by my e-mail provider.
Sean 'Shaleh' Perry
> I'll step back from any claim that unit tests really prove anything,
> but in a very Pythonic and pragmatic fashion they are Good Enough to
> make them a /necessity/ for anyone trying to deliver reliable software.
>
my current project is a window manager for X Windows. It is all X lib calls
and UI events. I can maybe test 5% of it. Seems most of the programs I have
worked on turn out that way. I write as many tests as I can but usually it
just doesn't work out for me.
But you are right, test as early and as often as possible and the code can't be
all bad (-:
Peter Hansen
>
> >
> > I'll step back from any claim that unit tests really prove anything,
> > but in a very Pythonic and pragmatic fashion they are Good Enough to
> > make them a /necessity/ for anyone trying to deliver reliable software.
>
> my current project is a window manager for X Windows. It is all X lib calls
> and UI events. I can maybe test 5% of it. Seems most of the programs I have
> worked on turn out that way. I write as many tests as I can but usually it
> just doesn't work out for me.
There are approaches that could increase that significantly.
The key pattern is to create a layer which accepts the various calls
you make into the UI and library and turns the information into something
like strings which you can easily test. Take simple code which makes
such calls, test your test code with it, and then freeze the test code.
Now you have a layer which can replace ("mock") the UI or library
and allow you to intercept and check calls that would have been made to it.
Now all your UI stuff is testable at least in terms of whether the right
routines are called at the right times with the right parameters. You
aren't testing whether the right pixel turns on, but then you are using
a library for that: you don't need to test it.
Stepping back even farther: keep the untestable code in a very thin
layer, test it manually and thoroughly, then never change it. You
don't have to test this bit of code that is "so simple it cannot fail",
when the alternative is to skip testing of 95% of your application.
Automated testing of GUIs is a very underdeveloped area of software
development, but it's a frontier that almost nobody is bothering to
explore. If more people check it out I think we'll make better
progress developing it.
-Peter
Roman Suzi
I can agree that tests give practical confidence in the
tested code, but I was speaking about rigourous
verification that code is 'correct'.
And tests can test only some subset of input-output
mapping. Making tests is an art, is not formal act.
I had no intention to say tests are useless, only that
they are not a method of proving the correctness of code
or correctenss of code modifications.
Sincerely yours, Roman Suzi
--
\_ Russia \_ Karelia \_ Petrozavodsk \_ r...@onego.ru \_
\_ Saturday, June 08, 2002 \_ Powered by Linux RedHat 7.2 \_
\_ "Things are getting worse. Please send chocolate." \_
Sean 'Shaleh' Perry
> Automated testing of GUIs is a very underdeveloped area of software
> development, but it's a frontier that almost nobody is bothering to
> explore. If more people check it out I think we'll make better
> progress developing it.
>
the problem is that extra level adds more code, more bloat and is one more
thing for new hackers to learn when they approach the project.
Yes I know tight coupling is usually bad. But sometimes it is the best way to
solve the problem at hand.
and now I have walked this sufficiently off topic to bore most of the readers
of this list.
Steve Holden
> Roman Suzi wrote:
> >
> > On Fri, 7 Jun 2002, Peter Hansen wrote:
> > > You forgot the even more important first thing for a beginner:
> > > get it correct! The only good way to do this is to write unit
> > > tests that prove it. If you don't have tests, you don't have
> > > working code (or how can you prove it?).
> >
> > To prove that code is correct is very difficult (practically
impossible).
> > Unittests are there to prove that you have errors, not the other way
> > around.
> >
> > Of course, they can increase your confidence in that optimized code is
> > equivalent to what you had before.
>
> I disagree, if the tests are written first (and run, proving that
> they will fail when the code is wrong or missing).
>
incorrect test. But a test that succeeds when the code *isn't* demonstrably
wrong doesn't guarantee the code is correct. Most tests aren't complete
enough to guarantee that the tested code runs to specification over all
possible inputs.
Please don't think I'm arguing that test-before-code is bad - I think it's
great, and very helpful for collaborators.
> The tests are executable requirement specifications. Running
> them "proves" (yes, not 100%... nothing could) that the code
> meets the specs. The specs could be wrong of course, but that
> doesn't mean the code is broken...
>
That's pretty much where I fall out with the formal methods crowd (who
invaded the academic world in the 1980's, putting paid to some of *my*
fun...). They don't seem to realise that for practical development projects
all they really do is replace bugs in the implementation with bugs in the
specification, since the specifications become mathematical notations
incomprehensible to the average user. Ultimately it's the users'
requirements that must dictate the specification.
regards
Aahz
Sean 'Shaleh' Perry <shale...@attbi.com> wrote:
>
>and now I have walked this sufficiently off topic to bore most of the
>readers of this list.
I disagree that this is off-topic; comp.lang.python has generally been
tolerant of any topic that *can* be applied to Python programming.
Testing is surely an appropriate topic.
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"I had lots of reasonable theories about children myself, until I
had some." --Michael Rios
Kragen Sitaker
> You forgot the even more important first thing for a beginner:
> get it correct! The only good way to do this is to write unit
> tests that prove it. If you don't have tests, you don't have
> working code (or how can you prove it?).
I agree that correctness is important, although I think it is
sometimes overemphasized; buggy programs can still be useful. (I hate
them, though.) For example, in the ICFP programming contests, any
entry found to contain a bug, however small, is automatically
disqualified.
I also agree that unit tests are very useful for reducing bugginess.
But I do not think that they are either necessary or sufficient for
writing correct code. I think (and extensive experimentation has
shown) that careful reasoning and code reviewing is necessary and
sufficient to write correct code.
It is certainly possible to err too far in the direction of believing
your code correct simply because it passes its test suite, and this is
a common beginner's error. I think it is also possible to err too far
in the direction of writing and running tests instead of desk-checking
code.
Kragen Sitaker
> "gabor" <ga...@realtime.sk> wrote in message
> news:mailman.1023401650...@python.org...
> > for file in directory:
> > file.printsize()
>
> Finally, "printsize" as a method name suggests to me that the loop is more
> likely to be IO-bound than CPU-bound, so attempts to optimise the loop
> overhead may buy very little benefit.
Possible, but not likely --- how much I/O do you think it takes to
read an inode and put a line in an output buffer?
Kragen Sitaker
> Roman Suzi wrote:
> > To prove that code is correct is very difficult (practically impossible).
> > Unittests are there to prove that you have errors, not the other way
> > around.
>
> they will fail when the code is wrong or missing).
I think you are disagreeing because you are ignorant of formal methods.
> The tests are executable requirement specifications. Running
> them "proves" (yes, not 100%... nothing could) that the code
> meets the specs. The specs could be wrong of course, but that
> doesn't mean the code is broken...
Typically specs specify the behavior of the code in an infinite set of
situations. Tests test the behavior of the code in a finite set of
situations, one situation per test.
You said "nothing could", but you are mistaken. Computer programs are
sufficiently amenable to mathematical analysis that it is possible to
prove rigorously that a particular routine meets a particular spec,
100% of the time, if the spec is phrased in sufficiently rigorous
terms.
This has the following drawbacks:
- specs can be wrong (but if they're shorter than the code, they'll
probably have fewer bugs)
- proofs can be wrong (but at least they make you think about the
assumptions you made when writing the code)
- writing the proofs takes a long time, so long that it's practical
only for small programs
- your proof is dependent upon your specification of the program's
environment as well as upon your requirements specification; if your
program depends on a lot of other complicated software, your proof
may not be worth much
This is important not because rigorously proving a program correct is
a worthwhile activity --- maybe it is, but I'm not convinced --- but
because proving a program correct is a more rigorous version of what
good programmers do all the time in writing good code. You should
learn a little bit about it, at least in its lighter-weight forms like
cleanroom programming.
Peter Hansen
>
> Peter Hansen <pe...@engcorp.com> writes:
> > You forgot the even more important first thing for a beginner:
> > get it correct! The only good way to do this is to write unit
> > tests that prove it. If you don't have tests, you don't have
> > working code (or how can you prove it?).
>
> I agree that correctness is important, although I think it is
> sometimes overemphasized; buggy programs can still be useful. (I hate
> them, though.) For example, in the ICFP programming contests, any
> entry found to contain a bug, however small, is automatically
> disqualified.
>
> I also agree that unit tests are very useful for reducing bugginess.
>
> But I do not think that they are either necessary or sufficient for
> writing correct code.
Without tests, how much confidence can you have in the correctness
of your code? It's been said "you don't know it works if you
don't test it".
> I think (and extensive experimentation has
> shown) that careful reasoning and code reviewing is necessary and
> sufficient to write correct code.
Whoa! "extensive experimentation has shown"?? I would suggest that
what extensive experience (I change the word deliberately) has shown
many of us is that careful reasoning and code reviewing, no matter
how extensive, will *always* leave bugs in the code. Have you never
spent a large amount of time carefully designing and coding a wonder
little program, and checked it over very carefully, even with a peer,
and later found a bug in it? Maybe you're just a better programmer
than I, and many others, but I would say this inevitable outcome is
the source of the term "egoless programming" -- if you have a big
ego, you'll be devastated by the repeated proof that you will never,
ever get it right the first time, or even after lots of reviewing.
> It is certainly possible to err too far in the direction of believing
> your code correct simply because it passes its test suite, and this is
> a common beginner's error. I think it is also possible to err too far
> in the direction of writing and running tests instead of desk-checking
> code.
I can only suggest that until you have tried test-first development,
in which the code is written in very, very small steps only to make
small failing tests pass, and discovered the possibilities of
emergent design, and other aspects of this approach, then we will not
find much common ground in this discussion. I simply disagree with
your last sentence above, and I believe beginners _never_ write
unit tests anywhere that I've seen, so I disagree with the first
one too. :)
-Peter
Peter Hansen
>
> Peter Hansen <pe...@engcorp.com> writes:
> > Roman Suzi wrote:
> > > To prove that code is correct is very difficult (practically impossible).
> > > Unittests are there to prove that you have errors, not the other way
> > > around.
> >
> > I disagree, if the tests are written first (and run, proving that
> > they will fail when the code is wrong or missing).
>
> I think you are disagreeing because you are ignorant of formal methods.
No, you're wrong. I'm familiar with formal methods. I was actually
disagreeing because I've been writing these way too late at night
without adequate sleep and my logic centres are deprived and my
language centres are sluggish.
Suffice to say that in another thread I backed away from "tests
can prove anything" -- I was wrong.
What tests do, however, is vastly increase your confidence in the
correctness of the code, and when written properly (in advance
of each small morsel of code that is written, and having been
shown to fail without that code) they are sufficient to deliver
extremely high quality software in which you can have high
confidence. And, I believe, _necessary_ for the same.
-Peter
holger krekel
> Kragen Sitaker wrote:
> > I think (and extensive experimentation has
> > shown) that careful reasoning and code reviewing is necessary and
> > sufficient to write correct code.
>
> Whoa! "extensive experimentation has shown"?? I would suggest that
> what extensive experience (I change the word deliberately) has shown
> many of us is that careful reasoning and code reviewing, no matter
> how extensive, will *always* leave bugs in the code. Have you never
> spent a large amount of time carefully designing and coding a wonder
> little program, and checked it over very carefully, even with a peer,
> and later found a bug in it? Maybe you're just a better programmer
> than I, and many others, but I would say this inevitable outcome is
> the source of the term "egoless programming" -- if you have a big
> ego, you'll be devastated by the repeated proof that you will never,
> ever get it right the first time, or even after lots of reviewing.
I mostly agree and there is even another point that makes tests
important: making sure that later changes do not break
the basic semantics of your program. You would have to do another
code review of the *whole* part of a program to gain confidence, again.
This method doesn't scale well with the number of changes to a program.
Only if you ever touch code once in a year would you affort to gain
confidence by 'closely examining the code'. Most often, people just
assume that their clever modification will not do any harm...
I must admit that writing tests for C++ or Java code really
is a verbose pain. But with python it's almost fun.
just my 2 test cent,
holger
John J. Lee
> Kragen Sitaker wrote:
[...]
> > shown) that careful reasoning and code reviewing is necessary and
> > sufficient to write correct code.
[...]
> ego, you'll be devastated by the repeated proof that you will never,
> ever get it right the first time, or even after lots of reviewing.
Not quite never. It *does* happen, but with the same sort of frequency as
I find dropping from a five club juggle leads to one landing upright on
the floor and staying there: it *has* happened to me, exactly three times
in about a decade of practice.
(what's a club?: http://www.windwizards.com/henrysjuggling.html)
> > It is certainly possible to err too far in the direction of believing
> > your code correct simply because it passes its test suite, and this is
> > a common beginner's error. I think it is also possible to err too far
> > in the direction of writing and running tests instead of desk-checking
> > code.
[...]
Isn't it true that, however much time you're going to spend, the largest
number of bugs are going to get removed by allocating some of the time to
both activities: testing, and staring at the code?
John
John J. Lee
[...]
> What tests do, however, is vastly increase your confidence in the
> correctness of the code, and when written properly (in advance
> of each small morsel of code that is written, and having been
> shown to fail without that code) they are sufficient to deliver
This point -- that tests should fail before they pass -- is easy to forget
about. Thanks for reminding me.
John
Peter Hansen
>
> > Kragen Sitaker wrote:
> > > It is certainly possible to err too far in the direction of believing
> > > your code correct simply because it passes its test suite, and this is
> > > a common beginner's error. I think it is also possible to err too far
> > > in the direction of writing and running tests instead of desk-checking
> > > code.
>
> number of bugs are going to get removed by allocating some of the time to
> both activities: testing, and staring at the code?
Perhaps, assuming you put a lot of bugs in the code, some amount of staring
will be necessary.
I believe staring is of little or no use, however, having tried it many
times with my and others' code which had bugs and not having found those
bugs by staring.
The nastiest bugs were found by lots and lots of work, which usually
included repeated attempts to reproduce the problem, manual runs,
and sometimes using a debugger. Staring helped little. And then,
having found the bug, we look at the code and say "oh yeah, we
should have seen that."
Better not to put the bugs in the code in the first place, however,
so staring or those other things won't be necessary. Writing the
tests and the code together, tests first, in small iterative steps,
tends not to lead to buggy code. The payback is high compared to
staring.
-Peter
Aahz
Peter Hansen <pe...@engcorp.com> wrote:
>
>The nastiest bugs were found by lots and lots of work, which usually
>included repeated attempts to reproduce the problem, manual runs,
>and sometimes using a debugger. Staring helped little. And then,
>having found the bug, we look at the code and say "oh yeah, we
>should have seen that."
Not necessarily.
>Better not to put the bugs in the code in the first place, however,
>so staring or those other things won't be necessary. Writing the
>tests and the code together, tests first, in small iterative steps,
>tends not to lead to buggy code. The payback is high compared to
>staring.
Maybe. The most difficult bug I recall tracking down turned out to
involved a web page created by a program that put a null character at
the end of an URL, like this:
<a href="http://www.example.com/foo/bar.html\x00">
Except, of course, that looking at the page's source in the obvious ways
didn't display the damn thing. I finally used Python to download the
page and display chunks of the pagestring. The problem was compounded
by a bug in MS SQL Server that considered the null-ending and
non-null-ending versions of the string to be either duplicate or unique,
depending on the precise operation.
Kragen Sitaker
> Kragen Sitaker wrote:
> > Peter Hansen <pe...@engcorp.com> writes:
> > I also agree that unit tests are very useful for reducing bugginess.
> >
> > But I do not think that they are either necessary or sufficient for
> > writing correct code.
>
> Without tests, how much confidence can you have in the correctness
> of your code? It's been said "you don't know it works if you
> don't test it".
Tests are one way to increase your confidence in the correctness of
your code, but they are not the only way.
> > I think (and extensive experimentation has
> > shown) that careful reasoning and code reviewing is necessary and
> > sufficient to write correct code.
>
> Whoa! "extensive experimentation has shown"??
Yes. (Well, I guess "correct code" is probably overstating the case.
"Code with many fewer bugs per line than most code in use" is probably
more accurate.) Are you familiar with Cleanroom programming?
> I would suggest that what extensive experience (I change the word
> deliberately) has shown many of us is that careful reasoning and
> code reviewing, no matter how extensive, will *always* leave bugs in
> the code. Have you never spent a large amount of time carefully
> designing and coding a wonder little program, and checked it over
> very carefully, even with a peer, and later found a bug in it?
Of course! People are fallible, it's much easier to do a bad code
review than a good one.
> Maybe you're just a better programmer than I, and many others, but I
> would say this inevitable outcome is the source of the term "egoless
> programming" -- if you have a big ego, you'll be devastated by the
> repeated proof that you will never, ever get it right the first
> time, or even after lots of reviewing.
Yes, one of the beautiful things about programming is how it impresses
upon each of us how fallible we are.
For what it's worth, the term "egoless programming" was invented by
Gerald Weinberg for his book _The Psychology of Computer Programming_
to describe a programming style in which group reviews focus on
finding bugs in code. It comes from Weinberg's observation that code
reviews work very poorly if the author is defensive about bugs, and
much better if the author is trying to find bugs too.
Of course, it's true of testing too; testing works best if you're
trying to find bugs.
> > It is certainly possible to err too far in the direction of believing
> > your code correct simply because it passes its test suite, and this is
> > a common beginner's error. I think it is also possible to err too far
> > in the direction of writing and running tests instead of desk-checking
> > code.
>
> I can only suggest that until you have tried test-first development,
> in which the code is written in very, very small steps only to make
> small failing tests pass, and discovered the possibilities of
> emergent design, and other aspects of this approach, then we will not
> find much common ground in this discussion.
Test-first development, exactly as you describe it above, has been my
preferred development style since mid-2000, at least when working on
programs of more than 100 lines, especially with other people. I've
written a lot of code that way.
I think now we have each committed the same arrogant gaffe in this
discussion: "It is only possible to disagree with me if you do not
know what I am talking about." I was wrong, and so are you.
> I simply disagree with your last sentence above, and I believe
> beginners _never_ write unit tests anywhere that I've seen, so I
> disagree with the first one too. :)
Beginners almost always test their code, but rarely by writing a
formal test suite --- usually by hand.
I think inspection can find bugs that are very difficult to find by
testing, and vice versa. That is the reason for my second statement;
a balance of careful reasoning and testing can produce better code
than the same amount of time spent on either alone.
Delaney, Timothy
It's one of the things that you need to be veryt rigourous about. It's very
tempting, upon receiving a bug report, to go charging into the code, finding
where the problem is, "fixing" it, then realising you need to test it.
Of course, by this time you can't write a test that *fails* with the
previous behaviour unless you go back to a previous version.
Always write a (correct ;) test that fails before "fixing" anything.
Unit tests may include 100% code coverage, but they rarely cover all
possible situations. When a new situation is found, the test should be
written.
Tim Delaney
Peter Hansen
>
> Peter Hansen <pe...@engcorp.com> writes:
> > Have you never spent a large amount of time carefully
> > designing and coding a wonder little program, and checked it over
> > very carefully, even with a peer, and later found a bug in it?
>
> Of course! People are fallible, it's much easier to do a bad code
> review than a good one.
I have to confess I have never seen a successful code review, nor
an environment in which they were used regularly and which had a
high level of code quality. Clearly as a result I'm fairly biased
against them -- I should keep a more open mind on the issue perhaps.
I'm swayed also by the thought that programmers generally rebel
against them, and that therefore they are likely not the best
approach in most shops. Learning XP has taught me that a process
which works in a way programmers find natural is much more likely
to succeed than one which goes against the grain.
> I think now we have each committed the same arrogant gaffe in this
> discussion: "It is only possible to disagree with me if you do not
> know what I am talking about." I was wrong, and so are you.
Point taken; my apologies. (I'm about to go on vacation to NYC for
the rest of the week, so if I suddenly go silent, it's not personal. :)
> I think inspection can find bugs that are very difficult to find by
> testing, and vice versa. That is the reason for my second statement;
> a balance of careful reasoning and testing can produce better code
> than the same amount of time spent on either alone.
For code not written test first, I would strongly concur. For code
written test first, I believe the resulting modularity and cohesion
in the code mostly negates the value of inspection. Again, other
practices from XP are probably polluting my mind against such things --
in this case pair programming. Assuming the formal 'inspection'
advocated by some heavy-weight processes, I think it's of little
use on code pair-programmed. (We could also be more general and
just say that pair programming is on par with inspection and that
either is effective in an appropriate context.)
Cheers,
-Peter
Alexander Garden
> Yes, one of the beautiful things about programming is how it impresses
> upon each of us how fallible we are.
Amen!
Kragen Sitaker
> > review than a good one.
>
> I have to confess I have never seen a successful code review, nor
> an environment in which they were used regularly and which had a
> high level of code quality. Clearly as a result I'm fairly biased
> against them -- I should keep a more open mind on the issue perhaps.
You are experiencing code reviewing whenever you are writing
production code. Do you find that it improves the code quality?
I've found that both pair programming and less-intense kinds of
reviewing are usually extremely helpful.
> I'm swayed also by the thought that programmers generally rebel
> against them, and that therefore they are likely not the best
> approach in most shops.
Yeah, that can be a problem. That problem has made it hard to
introduce pair programming in some places I've worked; it largely
depends on the social dynamics of the place.
> Assuming the formal 'inspection' advocated by some heavy-weight
> processes, I think it's of little use on code pair-programmed. (We
> could also be more general and just say that pair programming is on
> par with inspection and that either is effective in an appropriate
> context.)
I mostly agree that inspection during pair programming finds the bugs
any inspection is likely to find.
Simon Foster
<jame...@btinternet.com> wrote:
>"Peter Hansen" <pe...@engcorp.com> wrote in message
>news:3D00A456...@engcorp.com...
>> Eddie Corns wrote:
>> >
>> > A friend is fond of quoting this to me:
>> >
>> > Rules of Optimization:
>> > Rule 1: Don't do it.
>> > Rule 2 (for experts only): Don't do it yet.
>> > -- M.A. Jackson
>> >
>> You forgot the even more important first thing for a beginner:
>> get it correct!
>
>I favour Kent Beck's aphorism: "Make it work, make it right, make it fast."
>
>James
>
>
>
I prefer this one:
"You can have it right"
"You can have it cheap"
"You can have it now"
Pick any two.
Peter Hansen
>
> On Fri, 7 Jun 2002 21:03:08 +0100, "James Kew"
> <jame...@btinternet.com> wrote:
> >I favour Kent Beck's aphorism: "Make it work, make it right, make it fast."
> >
>
> "You can have it right"
> "You can have it cheap"
> "You can have it now"
>
> Pick any two.
Interestingly, XP provides a way to have all three...
Bo M. Maryniuck
> > I prefer this one:
> >
> > "You can have it right"
> > "You can have it cheap"
> > "You can have it now"
> >
> > Pick any two.
>
> Interestingly, XP provides a way to have all three...
If You mean XP-programming, than I'm not agree: You can't have it NOW.
If You mean WinXP, than I'm agree.reverse() : You can have it bad, expensive
and later (like anything something else from Microsoft, what's newer works
right)...
--
Sincerely yours, Bogdan M. Maryniuck
Who wants to remember that escape-x-alt-control-left shift-b puts you into
super-edit-debug-compile mode?
(Discussion in comp.os.linux.misc on the intuitiveness of commands, especially
Emacs.)
James Ashley
Arguably.
To an extent.
More or less.
Maybe.
XP is interesting, for a particular software environment. Its basic
truths (as I understand them) seem worth researthing. It seems like a
more efficient way to develop software than trying to get the customer
to make any actual decisions before-hand.
But does that discussion belong here? Seems to me that there are plenty
of python-specific theads going on. There seem to be many other places
to examine this forum.
Then again, just because this is c.l.py:
TRUE=-1
FALSE=0
{ right: (TRUE, FALSE),
cheap: (TRUE, FALSE),
quick: (TRUE, FALSE)
}
What's a pythonic way to compare that dict?
Or would it make more sense to make a list of classes?
class right:
pass
class cheap:
pass
class quick:
pass
if isinstance( a, ... ):
...
elif instance( a, ...) :
No, that's not right either.
<BLUSH>
Any suggestions? (Including, of course, those which make me feel
stupid...those are actually preferable)
TIA,
James
Peter Hansen
>
> On Thursday 04 July 2002 06:35, Peter Hansen wrote:
> > > I prefer this one:
> > >
> > > "You can have it right"
> > > "You can have it cheap"
> > > "You can have it now"
> > >
> > > Pick any two.
> >
> > Interestingly, XP provides a way to have all three...
>
> If You mean XP-programming, than I'm not agree: You can't have it NOW.
You're right, of course, but then James Kew's list above was not
quite correct. It's normally "good, cheap, or fast" and although a
substitution of "right" for "good" is okay I don't think "now" is
the same as "fast".
Peter Hansen
>
> In article <3D23D092...@engcorp.com>, Peter Hansen wrote:
> > Simon Foster wrote:
> >>
> >> On Fri, 7 Jun 2002 21:03:08 +0100, "James Kew"
> >> <jame...@btinternet.com> wrote:
> >> "You can have it right"
> >> "You can have it cheap"
> >> "You can have it now"
> >>
> >> Pick any two.
> >
> > Interestingly, XP provides a way to have all three...
> But does that discussion belong here? Seems to me that there are plenty
> of python-specific theads going on. There seem to be many other places
> to examine this forum.
If you have been reading all those threads, you'll have noticed a
number of questions that come up from time to time about developing
large applications with Python, using software patterns with Python,
unit testing with Python, optimizing Python, and so on.
Just because such a thread wanders a little away from Python itself
for a while doesn't make it unsuitable for this newsgroup/mailing list.
Python programmers are by definition not the kind to follow the
rest of the crowd, and they also tend to be interested in practical
advice about how to improve their results and the way they get them.
And you can always stop reading the thread, too. Vote with your feet.
But if you aren't so quick to dismiss it, you may find the thread
comes back to Python in interesting ways, and you might even learn
something.
-Peter
Bo M. Maryniuck
> I don't think "now" is the same as "fast".
--
Sincerely yours, Bogdan M. Maryniuck
As usual, this being a 1.3.x release, I haven't even compiled this
kernel yet. So if it works, you should be doubly impressed.
(Linus Torvalds, announcing kernel 1.3.3 on the linux-kernel mailing list.)
Fernando Pereira
<pe...@engcorp.com> wrote:
> Python programmers are by definition not the kind to follow the
> rest of the crowd
-- F