Why no open(f, "w").write()?
Markus Demleitner
effect about a week ago and it seems it didn't make it across our
news server, and at any rate not to google)
The Jython docs state that
open("some.name", "w").write(stuff)
is bad programming practice (and indeed claim that in Jython,
the above construct leaves some.name empty).
Now, I can't really see what should be wrong with doing something
like that. The temporaray file object generated should have a
refcount of 0 after that line and close the file when being
destroyed, no? I didn't delve too deep into the docs now,
but IIRC refcount-based garbage collection is sort of a guarateed
feature of python, isn't it?
Or am I missing something else?
Markus
Michael P. Soulier
<msde...@tucana.cl.uni-heidelberg.de> wrote:
>
> The Jython docs state that
> open("some.name", "w").write(stuff)
> is bad programming practice (and indeed claim that in Jython,
> the above construct leaves some.name empty).
I can't speak for Jython, but in CPython 2.1, it works fine.
>>> stuff = "stuff to write"
>>> open('stufffile', "w").write(stuff)
>>> open('stufffile', "r").read()
'stuff to write'
Mike
--
Michael P. Soulier, QX41, SKY Tel: 613-765-4699 (ESN: 39-54699)
Optical Networks, Nortel Networks, SDE Pegasus
"...the word HACK is used as a verb to indicate a massive amount
of nerd-like effort." -Harley Hahn, A Student's Guide to Unix
Sean 'Shaleh' Perry
On 29-May-2002 Michael P. Soulier wrote:
> On 24 May 2002 07:53:38 GMT, Markus Demleitner
> <msde...@tucana.cl.uni-heidelberg.de> wrote:
>>
>> The Jython docs state that
>> open("some.name", "w").write(stuff)
>> is bad programming practice (and indeed claim that in Jython,
>> the above construct leaves some.name empty).
>
> I can't speak for Jython, but in CPython 2.1, it works fine.
>
>>>> stuff = "stuff to write"
>>>> open('stufffile', "w").write(stuff)
>>>> open('stufffile', "r").read()
> 'stuff to write'
>
yes, but can you disagree with the statement that it is bad programming
practice. I sure can't.
Donn Cave
| On 24 May 2002 07:53:38 GMT, Markus Demleitner
| <msde...@tucana.cl.uni-heidelberg.de> wrote:
|>
|> The Jython docs state that
|> open("some.name", "w").write(stuff)
|> is bad programming practice (and indeed claim that in Jython,
|> the above construct leaves some.name empty).
|
| I can't speak for Jython, but in CPython 2.1, it works fine.
Yes, of course. The problem is that Java doesn't support the timely
finalization we enjoy in C Python, so for the Java implemention you
have to explicitly invoke the close() that would implicitly be done
during finalization. That causes the buffer to be flushed to the
actual disk file. I assume finalization will eventually occur in
any case, if only at program exit, so this would only be an issue
if the file is needed before then.
I bet this has already been said, if so my apologies for the waste
of bandwidth.
Donn Cave, do...@u.washington.edu
Huaiyu Zhu
For what reason is it bad?
Huaiyu
Gary Herron
> On 29-May-2002 Michael P. Soulier wrote:
> > On 24 May 2002 07:53:38 GMT, Markus Demleitner
> >
> > <msde...@tucana.cl.uni-heidelberg.de> wrote:
> >> The Jython docs state that
> >> open("some.name", "w").write(stuff)
> >> is bad programming practice (and indeed claim that in Jython,
> >> the above construct leaves some.name empty).
> >
> > I can't speak for Jython, but in CPython 2.1, it works fine.
> >
> >>>> stuff = "stuff to write"
> >>>> open('stufffile', "w").write(stuff)
> >>>> open('stufffile', "r").read()
> >
> > 'stuff to write'
>
> yes, but can you disagree with the statement that it is bad programming
> practice. I sure can't.
Here's why its a bad practice:
The question is: When does the file get closed?
The expectation is that the file object will be garbage collected
immediately after the statement is executed (since that's when its ref
count goes to zero), and that this will force the file to be closed.
And in fact the traditional CPython implementation of Python behaves
in exactly this way.
However, Python does not guarantee to garbage collect immediately as
the refcount goes to zero, and in fact Jython does NOT do so,instead
relying on Java's garbage collection.
Thus the "bad" part of this programming practice is that the timing of
the implied close is dependent Python implementation issues (i.e.,
the timing of the garbage collection), and such dependencies are never
a good thing to rely on.
Gary Herron
François Pinard
> Thus the "bad" part of this programming practice is that the timing
> of the implied close is dependent Python implementation issues (i.e.,
> the timing of the garbage collection), and such dependencies are never
> a good thing to rely on.
There might be cases when one moves between Python and Jython, indeed.
When one knows s/he works with Python only, it is good style to rely on
the refcount behaviour, as it yields code which is not only more legible,
but also more elegant and concise. It means that you understand and
accept in advance having to revise your code if you ever want to use other
implementations of Python, like Jython. As someone was pointing to me
very recently, the Python reference tries to describe a common language,
but there is no "C-Python" specific guide. If there was one, the refcount
behaviour would most probably be described as dependable and reliable,
even through future versions, as far as Python programming is concerned.
And besides, it seems that the few implementations of Python do not support
exactly the same language: extensions here may not be available there.
Defining "good style" as the common subset of all Python implementations,
and everything else as "bad style", seems questionable. The only thing
is that you have to be aware of the implications of your choices.
For this "Jython-forces-you-to-explicit-closes" matter, my feeling is that
Jython encourages bad style here, much more than it defines good style.
Surely, there has never been a bad intent from Jython author. We understand
that the limitation comes from the fact Jython relies on the Java system
for collecting garbage. One has to close explicitly in Jython for practical
implementation considerations, this has nothing to do with good style.
--
François Pinard http://www.iro.umontreal.ca/~pinard
Sean 'Shaleh' Perry
> There might be cases when one moves between Python and Jython, indeed.
> When one knows s/he works with Python only, it is good style to rely on
> the refcount behaviour, as it yields code which is not only more legible,
> but also more elegant and concise. It means that you understand and
> accept in advance having to revise your code if you ever want to use other
> implementations of Python, like Jython. As someone was pointing to me
> very recently, the Python reference tries to describe a common language,
> but there is no "C-Python" specific guide. If there was one, the refcount
> behaviour would most probably be described as dependable and reliable,
> even through future versions, as far as Python programming is concerned.
>
depending on implementation is almost never a good idea. Sure CPython behaves
this way now, but it may not next month or next year. More importantly this
style of coding makes the creation of temporaries common and blessed where one
usually does not want an abundance of them.
The arguments then move to pure aesthetic style and we all know that no one can
agree there so there is no need to carry it further.
>
> For this "Jython-forces-you-to-explicit-closes" matter, my feeling is that
> Jython encourages bad style here, much more than it defines good style.
> Surely, there has never been a bad intent from Jython author. We understand
> that the limitation comes from the fact Jython relies on the Java system
> for collecting garbage. One has to close explicitly in Jython for practical
> implementation considerations, this has nothing to do with good style.
>
you can choose to not close() a file in python (either implementation) you just
have to hope the interpreter gets to it soon. In one this happens to be true,
in the other it may not be. As a matter of habit in any long running program I
am always close() what needs to be closed and in other way attempt to help the
garbage collector.
Donn Cave
...
| Thus the "bad" part of this programming practice is that the timing of
| the implied close is dependent Python implementation issues (i.e.,
| the timing of the garbage collection), and such dependencies are never
| a good thing to rely on.
The confusing thing about this practice is that it's both
good and bad.
I wouldn't say open(n, m).write(v) is worth defending per se,
but the notion that the system can tell an object when its
lifetime is over is very valuable in general, in an object
oriented programming model. In more complicated usage, where
the file object (or window, or whatever) persists for some
time and may have several references in use, I'm with M Pinard,
it would seem like a potentially good programming practice.
It's too bad things like good programming practice have to be
subject to the limitations of other programming languages.
Donn Cave, do...@u.washington.edu
Gary Herron
> [Gary Herron]
>
> > Thus the "bad" part of this programming practice is that the timing
> > of the implied close is dependent Python implementation issues (i.e.,
> > the timing of the garbage collection), and such dependencies are never
> > a good thing to rely on.
>
> When one knows s/he works with Python only, it is good style to rely on
> the refcount behaviour, as it yields code which is not only more legible,
> but also more elegant and concise. It means that you understand and
> accept in advance having to revise your code if you ever want to use other
> implementations of Python, like Jython. As someone was pointing to me
> very recently, the Python reference tries to describe a common language,
> but there is no "C-Python" specific guide. If there was one, the refcount
> behaviour would most probably be described as dependable and reliable,
> even through future versions, as far as Python programming is concerned.
>
> exactly the same language: extensions here may not be available there.
> Defining "good style" as the common subset of all Python implementations,
> and everything else as "bad style", seems questionable. The only thing
> is that you have to be aware of the implications of your choices.
>
> Jython encourages bad style here, much more than it defines good style.
> Surely, there has never been a bad intent from Jython author. We
> understand that the limitation comes from the fact Jython relies on the
> Java system for collecting garbage. One has to close explicitly in Jython
> for practical implementation considerations, this has nothing to do with
> good style.
Lots of what you say is true, but not applicable in this case. The example
open(f, 'w').write(stuff)
depends on an accident of current versions of this implementation of
Python. The manual is quite explicit about NOT being able to depend
ANY particualar "features" of garbage collection -- not even on its
existence.
I think we can agree that it must be considered "bad prgramming
practice" to depend on features which the manual claims may not exist,
past, present, or future.
Here's the quote from the Reference Manual:
Objects are never explicitly destroyed; however, when they become
unreachable they may be garbage-collected. An implementation is
allowed to postpone garbage collection or omit it altogether -- it
is a matter of implementation quality how garbage collection is
implemented, as long as no objects are collected that are still
reachable. (Implementation note: the current implementation uses a
reference-counting scheme with (optional) delayed detection of
cyclicly linked garbage, which collects most objects as soon as they
become unreachable, but is not guaranteed to collect garbage
containing circular references. See the Python Library Reference for
information on controlling the collection of cyclic garbage.)
Gary Herron
François Pinard
> depending on implementation is almost never a good idea. Sure CPython
> behaves this way now, but it may not next month or next year.
I've been told that the behaviour is there to stay. We may depend on it.
> As a matter of habit in any long running program I am always close() what
> needs to be closed and in other way attempt to help the garbage collector.
This is surely good to explicitly `close()' when one is done with a file,
but needs to keep a reference on this file for other reasons. The nicest
is not keeping a reference to the file, whenever it can be avoided easily.
Delaney, Timothy
>
> The current (C-)Python implementation is of high quality on
> that respect,
> and after checking with knowledgeable people, I got that the
> dependability
> of refcounts could be considered as cast in stone, exactly
> like if it has
> been documented as such. If I did not get this confirmation
> first, I would
> never have started to use things like "open(FILE,
> 'w').write(CONTENTS)".
> Now, I really see it as part of (unwritten) specifications of
> (C-)Python,
> and perfectly legitimate.
I can come up with a perfectly good example where
open(FILE, 'w').write(CONTENTS)
will fail to close the file in CPython.
There are no guarantees that the reference to the object returned is the
*only* reference to that object. Assume the following (perverse)
implementation of open:
__files = []
def open (path, mode):
f = __open(path, mode)
__files.append(f)
return f
or
class file:
__files = []
def __init__(self, path, mode):
__files.append(self)
Whilst I know this is not the implementation, there is nothing that prevents
it from being like this. So you get back an object that has one more
reference count than expected. It never gets collected.
You should never rely on automatic freeing of *any* resources unless it is
guaranteed. Always explicitly free resources when you have finished with
them.
Tim Delaney
François Pinard
> Here's the quote from the Reference Manual: [...]
Yes, I'm well aware of this quote. :-)
> I think we can agree that it must be considered "bad programming practice"
> to depend on features which the manual claims may not exist, past,
> present, or future.
I heartedly agree that it is bad practice not keeping oneself between the
tracks set by specifications. Without really knowing, I imagine that the
said quote might be a politeness from Guido towards other implementations,
encouraging them, so they could claim being called Python nevertheless.
I wonder if the quote has been there since the beginnings of Python. :-)
> [...] it is a matter of implementation quality how garbage collection
> is implemented, as long as no objects are collected that are still
> reachable.
The current (C-)Python implementation is of high quality on that respect,
and after checking with knowledgeable people, I got that the dependability
of refcounts could be considered as cast in stone, exactly like if it has
been documented as such. If I did not get this confirmation first, I would
never have started to use things like "open(FILE, 'w').write(CONTENTS)".
Now, I really see it as part of (unwritten) specifications of (C-)Python,
and perfectly legitimate.
--
François Pinard http://www.iro.umontreal.ca/~pinard
Kragen Sitaker
> You should never rely on automatic freeing of *any* resources unless it is
> guaranteed. Always explicitly free resources when you have finished with
> them.
You mean, assume you have no garbage collector? If I had to do that,
my Python code would get a *lot* more complex.
I do believe that relying on timely finalization is a mistake, because
every efficient garbage collector fails to perform timely
finalization. I hope that someday in the future, CPython can have an
efficient garbage collector too.
Grant Edwards
>> yes, but can you disagree with the statement that it is bad programming
>> practice. I sure can't.
>
> Here's why its a bad practice:
>
> The question is: When does the file get closed?
Sometime before the program exits.
> The expectation is that the file object will be garbage collected
> immediately after the statement is executed (since that's when its ref
> count goes to zero), and that this will force the file to be closed.
If that's the expectation, then I agree that it's bad practice.
If the expectation is that the file will be closed by the time
the program has terminated, then is it still bad practice?
Probably so...
--
Grant Edwards grante Yow! I'm into SOFTWARE!
at
visi.com
Isaac To
Donn> I assume finalization will eventually occur in any case, if only
Donn> at program exit, so this would only be an issue if the file is
Donn> needed before then.
No. Java explicitly says that finalization of objects are not necessarily
done before program exits. That emulates an object with an infinite
lifetime.
Regards,
Isaac.
Peter Hansen
>
> [Gary Herron]
>
> > Here's the quote from the Reference Manual: [...]
>
> Yes, I'm well aware of this quote. :-)
>
> > I think we can agree that it must be considered "bad programming practice"
> > to depend on features which the manual claims may not exist, past,
> > present, or future.
>
> I heartedly agree that it is bad practice not keeping oneself between the
> tracks set by specifications. Without really knowing, I imagine that the
> said quote might be a politeness from Guido towards other implementations,
> encouraging them, so they could claim being called Python nevertheless.
> I wonder if the quote has been there since the beginnings of Python. :-)
It's been there since at least February 1999, which unfortunately for
debunking or confirming this theory, does not predate JPython...
http://web.archive.org/web/19990203001201/http://www.python.org/doc/ref/objects.html
(Strange, no earlier versions of the docs were cached by the Wayback
Machine...)
-Peter
Isaac To
>> The question is: When does the file get closed?
Grant> Sometime before the program exits.
Wrong. Sometime when the garbage collector is executed, or never if the
garbage collector never choose to collect that.
Regards,
Isaac.
Isaac To
Kragen> "Delaney, Timothy" <tdel...@avaya.com> writes:
>> You should never rely on automatic freeing of *any* resources unless
>> it is guaranteed. Always explicitly free resources when you have
>> finished with them.
Kragen> You mean, assume you have no garbage collector? If I had to do
Kragen> that, my Python code would get a *lot* more complex.
Garbage collection is for collecting memory. So for memory matters, you can
(and actually, must) give up the control to the garbage collector. For
anything else, you *must* do it explicitly. Surely you shouldn't lock a
mutex and remove the reference to the mutex, hoping that the garbage
collector will unlock it quickly enough.
Regards,
Isaac.
Delaney, Timothy
>
> "Delaney, Timothy" <tdel...@avaya.com> writes:
> > You should never rely on automatic freeing of *any*
> resources unless it is
> > guaranteed. Always explicitly free resources when you have
> finished with
> > them.
>
> my Python code would get a *lot* more complex.
Change that to "any resources not controlled by the system itself". Python
does not in fact guarantee a garbage collector, but does not provide any
mechanisms for freeing memory, so an implementation which did not free
memory would not be broken according to the documentation, but would be in
practice. Likewise, Python does not provide any mechanism for explicitly
allocating memory. Memory is controlled by the Python system.
OTOH, Python also makes no guarantees about finalisation, and this is of
course what I was referring to. Files are not a resource controlled by the
Python system. Python makes no guarantees that opened files will be closed.
Likewise sockets, spawned processes, etc.
Tim Delaney
VanL
open(f, 'w').write(stuff)
different than
lines = open(f).readlines()
in terms of bad/good programming practice?
I use that second construction quite frequently.
VanL
Donn Cave
...
| Garbage collection is for collecting memory. So for memory matters, you can
| (and actually, must) give up the control to the garbage collector. For
| anything else, you *must* do it explicitly. Surely you shouldn't lock a
| mutex and remove the reference to the mutex, hoping that the garbage
| collector will unlock it quickly enough.
You're describing a language other than C Python. In C Python, you
*may* close files and rely on other kinds of finalizations through
the reference counting garbage collector, and the result is as timely
as if you did it explicitly. That's finalization as it ought to be,
and it's a shame that other languages compromise on that. Lots of
people don't think it's important, but then the majority of programmers
think it's OK that their programming language requires them to keep
track of memory in the same way.
Donn Cave, do...@drizzle.com
Greg Ewing
>
> Ok, I've read the thread... but how is
>
> open(f, 'w').write(stuff)
>
> different than
>
> lines = open(f).readlines()
In CPython they both work fine. In Jython they could
both leak resources, and when writing there is the
additional possibility of written data not making
it to the file as soon as you'd like.
--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Delaney, Timothy
>
> You're describing a language other than C Python. In C Python, you
> *may* close files and rely on other kinds of finalizations through
> the reference counting garbage collector, and the result is as timely
> as if you did it explicitly. That's finalization as it ought to be,
> and it's a shame that other languages compromise on that. Lots of
> people don't think it's important, but then the majority of
> programmers
> think it's OK that their programming language requires them to keep
> track of memory in the same way.
I would be very happy for this to be the case.
However, CPython is not a language where you can rely on timely
finalisation.
1. The documentation explicitly states that you cannot. Anything else is an
implementation artifact.
2. The thing you expect to be finalised may have another reference
somewhere. Are you 100% sure that no function you've passed it to has held a
reference to it in a cache?
3. Your object may be in a cycle. This will result in a non-timely
finalisation.
4. Your object may be in a cycle where Python explicitly punts and doesn't
run the finalisers.
Tim Delaney
Delaney, Timothy
>
> Ok, I've read the thread... but how is
>
> open(f, 'w').write(stuff)
>
> different than
>
> lines = open(f).readlines()
>
>
> I use that second construction quite frequently.
It suffers from the same problems, but the symptoms aren't as bad.
With the read version, the file may still hang around without being closed.
This may prevent other processes (or the same process) from opening the
file. Usually when the program exists the OS will close the file.
With the write version, the file may hang around without being closed or
flushed. Thus when the program exits, the file may be closed without being
flushed, and thus may not contain the data that you thought you wrote to it.
Tim Delaney
Donn Cave
| > From: Donn Cave [mailto:do...@drizzle.com]
|>
|> You're describing a language other than C Python. In C Python, you
|> *may* close files and rely on other kinds of finalizations through
|> the reference counting garbage collector, and the result is as timely
|> as if you did it explicitly. That's finalization as it ought to be,
|> and it's a shame that other languages compromise on that. Lots of
|> people don't think it's important, but then the majority of
|> programmers
|> think it's OK that their programming language requires them to keep
|> track of memory in the same way.
|
| I would be very happy for this to be the case.
|
| However, CPython is not a language where you can rely on timely
| finalisation.
|
| 1. The documentation explicitly states that you cannot. Anything else is an
| implementation artifact.
The real Python _is_ implementation. We all know that you actually
can rely on timely finalization, and we can guess why the documentation
says what it says.
| 2. The thing you expect to be finalised may have another reference
| somewhere. Are you 100% sure that no function you've passed it to has held a
| reference to it in a cache?
The same applies to memory. If objects were finalized when they still
have references, or if their memory were released, that would be an
error, and I would not be in favor of that.
| 3. Your object may be in a cycle. This will result in a non-timely
| finalisation.
|
| 4. Your object may be in a cycle where Python explicitly punts and doesn't
| run the finalisers.
Same as (2) - it's my responsibility to design my program so that it
is "correct" with respect to object lifetimes, meaning that objects
don't live longer than I want them to. It's axiomatic in Python that
an object with references is still valid - not just a matter of
documented or implemented behavior - and it follows that if I want
an object's lifetime to expire, I have to make sure there are no
references to it. I'm cool with that. Most of the time I will not
care anyway, but to the extent it matters, this is a much better
deal than having to reinvent reference counting while I pretend it
isn't already being done by the interpreter.
Donn Cave, do...@drizzle.com
Donn Cave
| > From: VanL [mailto:ne...@lindbergs.org]
| >
| > Ok, I've read the thread... but how is
| >
| > open(f, 'w').write(stuff)
| >
| > different than
| >
| > lines = open(f).readlines()
| >
| > in terms of bad/good programming practice?
| >
| > I use that second construction quite frequently.
|
| It suffers from the same problems, but the symptoms aren't as bad.
|
| With the read version, the file may still hang around without being closed.
| This may prevent other processes (or the same process) from opening the
| file. Usually when the program exists the OS will close the file.
Certainly on UNIX, there is no such problem - nothing prevents a
file from being opened for read concurrently by many processes or
one process, at least that would be exercised by open(f).readlines()
The only external problem I can think of would be that the file
inode would survive, if something were to delete (unlink) its
directory entry (can you say "reference counting"?) and in some
improbable scenario the filesystem could run out of space.
More likely (though still not terribly likely) you could run out
of file descriptors. That's an integer range, at least 0..255
but probably a lot more, but in a long-running program it could
be something to look out for.
Donn Cave, do...@drizzle.com
Petr Prikryl
"François Pinard" wrote...
> [Gary Herron]
>
> > Thus the "bad" part of this programming practice is that the timing
> > of the implied close is dependent Python implementation issues (i.e.,
> > the timing of the garbage collection), and such dependencies are never
> > a good thing to rely on.
>
> There might be cases when one moves between Python and Jython, indeed.
> When one knows s/he works with Python only, it is good style to rely on
> the refcount behaviour, as it yields code which is not only more legible,
"Explicit is better than implicit." (about the close() here)
[The code was...
>>> stuff = "stuff to write"
>>> open('stufffile', "w").write(stuff)
>>> open('stufffile', "r").read()
]
One day, you may change the code slightly,
write several lines between the two open()
commands, etc. Then it may happen that your
program stops to work and it will be more
difficult to discover what is the problem.
In my opinion, the *short code* does not
automatically mean the *elegant*. The shorter
it is, the more difficult may be to notice the implicit
actions inside (look at the Perl-golf tournaments
fot the example).
> [...] We understand
> that the limitation comes from the fact Jython relies on the Java system
> for collecting garbage. One has to close explicitly in Jython for
practical
> implementation considerations, this has nothing to do with good style.
I would consider this example a bad style in any
language, independently of having or not having
the garbage collector. The main problem is not with
the file object, but with the open file as with
the system resource -- see below.
"François Pinard" wrote in another message...
>
> This is surely good to explicitly `close()' when one is done with a file,
> but needs to keep a reference on this file for other reasons. The nicest
> is not keeping a reference to the file, whenever it can be avoided easily.
Well, but such "laziness" may lead to problems.
What happens when it is not possible to write
into the 'stuffile'. In such case, it is handy to have
access to the file object (the reference to the file
object). Because of the brevity of the code it is also
not so explicitly visible what the write() will do when
the file cannot be opened.
The opened file (when not needed) may cause
problems to other applications that try to do
something with the same file. One should not consider
any open file object as the exclusive property
of the running application. Because of that,
the file should be closed as soon as possible.
But nobody forces you to do it this way.
It is just a good programming practice that help
to avoid possible problems.
--
Petr Prikryl (prikrylp at skil dot cz)
Delaney, Timothy
> Anything else is an
> | implementation artifact.
>
> The real Python _is_ implementation. We all know that you actually
> can rely on timely finalization, and we can guess why the
> documentation says what it says.
Then get it documented to say that it can be relied on, now and forever. If
you manage this, documented for all implementations of Python (obviously,
from a particular version number), I will have no qualms with anyone using
this type of construct on that version and later. Of course, it then
introduces backwards-incompatibility.
Until then I must consider any code using this type of construct as broken
and unsafe. Especially since the documentation currently states that this
cannot be relied on, and there is at least one implementation (Jython) where
this is indeed not the case.
I just find it amazing that so many people here are so careful about so many
things, yet seem to have a complete blind spot on this issue. Why don't we
have
while a = file.readline():
pass
while we're at it? After all, it's the same principle - you're trading off
safety for saving a line or two.
Tim Delaney
François Pinard
> Why don't we have
> while a = file.readline():
> pass
> while we're at it? After all, it's the same principle - you're trading
> off safety for saving a line or two.
The Python language is oriented towards legibility and clarity, and I
would guess that the above has been rejected as lacking legibility, much
more than for safety considerations.
Doing `lines = open(file).readlines()' looks perfectly safe to me, so I'm
not trading any safety. The goal of using such writing is not "saving a
line or two", but writing more legibly, by eliminating unnecessary noise.
Saving lines is not very important. Saving spurious lines is, however.
All those things are quite debatable, there is some aesthetic considerations
associated with legibility; moreover, aesthetics do not alway yield to
legibility. I think Python is nice because it has _one_ artist in charge,
able to impose a common view. If it was "democratic", it would loose much.
Grant Edwards
I know of no OS that doesn't close the file when the process
terminates.
--
Grant Edwards grante Yow! I'm also pre-POURED
at pre-MEDITATED and
visi.com pre-RAPHAELITE!!
Grant Edwards
> In article <7iadqi9...@enark.csis.hku.hk>, Isaac To wrote:
>>>>>>> "Grant" == Grant Edwards <gra...@visi.com> writes:
>>
>> >> The question is: When does the file get closed?
>>
>> Grant> Sometime before the program exits.
>>
>> Wrong. Sometime when the garbage collector is executed, or never if the
>> garbage collector never choose to collect that.
>
> I know of no OS that doesn't close the file when the process
> terminates.
I suppose that doesn't guarantee that data buffered by the
Python implimentation gets flushed...
--
Grant Edwards grante Yow! I hope something GOOD
at came in the mail today so
visi.com I have a REASON to live!!
Steve Holden
> In article <7iadqi9...@enark.csis.hku.hk>, Isaac To wrote:
> >>>>>> "Grant" == Grant Edwards <gra...@visi.com> writes:
> >
> > >> The question is: When does the file get closed?
> >
> > Grant> Sometime before the program exits.
> >
> > Wrong. Sometime when the garbage collector is executed, or never if the
> > garbage collector never choose to collect that.
>
> I know of no OS that doesn't close the file when the process
> terminates.
>
So we're supposed to rely on your ignorance in making implementation
decisions now?
regards
Steve
--
-----------------------------------------------------------------------
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/pwp/
-----------------------------------------------------------------------
François Pinard
[Petr Prikryl]
> "Explicit is better than implicit."
This is another ex-cathedra principle. Do you explicitly `del NAME' for
all your local variables, at the end of a function? The system does it
implicitly for you, and we are all very happy with this.
There is an equilibrium between explicit and implicit, and a sense of good
taste going in that equilibrium. But good taste in Python is not the same
as for Perl or for C, we need to un-dust ourselves a bit when we switch
between languages.
> In my opinion, the *short code* does not automatically mean the *elegant*.
Fully agreed, there is no guarantee there. On the other hand, it sometimes
happens that elegant code is shorter. No reason to feel shy! :-)
> The main problem is not with the file object, but with the open file as
> with the system resource -- see below.
Which open file? It gets closed as soon as expected!
> > The nicest is not keeping a reference to the file, whenever it can be
> > avoided easily.
> Well, but such "laziness" may lead to problems.
This is no `laziness' at all. This is an active seek for legibility.
I want my code to go straight to the essentials, and that it shows those
essentials as clearly as possible. I probably write and rewrite my code
much more patiently than many people do, with that goal always in mind.
> What happens when it is not possible to write into the 'stuffile'.
> [...] Because of the brevity of the code it is also not so explicitly
> visible what the write() will do when the file cannot be opened.
Python will traceback, magically, wonderfully, with adequate information.
> The opened file (when not needed) may cause problems to other applications
> that try to do something with the same file. One should not consider any
> open file object as the exclusive property of the running application.
> Because of that, the file should be closed as soon as possible.
But this is exactly what happens. There is no problem, only fears. Once you
get use to the paradigm, you discover that it is safe and dependable.
François Pinard
> > From: Donn Cave [mailto:do...@drizzle.com]
> > The real Python _is_ implementation. We all know that you actually
> > can rely on timely finalization, and we can guess why the
> > documentation says what it says.
> Then get it documented to say that it can be relied on, now and forever.
That would be ideal, indeed. Despite my current choices, there is always a
tiny, unlikely risk that Python changes his behaviour on this, despite what
I was told, and I would surely prefer not have to revise the code I wrote.
But if I ever have to revise it, for Jython or otherwise, I will be happy
having enjoyed the current legibility and simplicity for a long while!
Once again, I would not have started depending on immediate finalisation
before getting a confirmation of some sort that it is dependable. On one
hand, I've always have been strict, anal, at using only documented features,
in all systems and languages I've used in my life. On the other hand, in
this precise case, the advantages were so appealing that I felt compelled
to ask for clarification about the dependability of immediate finalisation.
Gary Duzan
François Pinard <pin...@iro.umontreal.ca> wrote:
=>[Sean 'Shaleh' Perry]
=>
=>> As a matter of habit in any long running program I am always close() what
=>> needs to be closed and in other way attempt to help the garbage collector.
=>
=>This is surely good to explicitly `close()' when one is done with a file,
=>but needs to keep a reference on this file for other reasons. The nicest
=>is not keeping a reference to the file, whenever it can be avoided easily.
Actually, after a quick read of the spec I don't see any guarantee
that a file object will be closed when it is destroyed, so yes, an
explicit close() should always be used.
Of course, if write() returned the file object instead of nothing,
one could use something like:
open(f, "w").write(data).close()
but that might not be Pythonic enough (or too C++-y) for some.
Gary Duzan
BBN Technologies
A Verizon Company
Gustavo Cordova
> Of course, if write() returned the file object instead of nothing,
> one could use something like:
>
> open(f, "w").write(data).close()
>
> but that might not be Pythonic enough (or too C++-y) for some.
>
> Gary Duzan
> BBN Technologies
> A Verizon Company
>
Well, subclassing is your friend!
-- snip --
class File(file):
def write(self, data):
file.write(self, data)
return self
-- snip --
So now you can have your cake and eat it too. :-)
Or, a "quickfile" kind of thing:
-- snip --
def WriteFile(filename, mode="w", *data):
f = file(filename,mode)
for item in data: f.write(item)
f.close()
-- snip --
ta-daa. Trivial problems demand trivial solutions.
-gustavo
Michael P. Soulier
<hua...@gauss.almadan.ibm.com> wrote:
>>>
>>>>>> stuff = "stuff to write"
>>>>>> open('stufffile', "w").write(stuff)
>>>>>> open('stufffile', "r").read()
>>>
>>
>>yes, but can you disagree with the statement that it is bad programming
>>practice. I sure can't.
>
Well, as one of the credos of Python is that explicit is better than
implicit, why are we relying on the implicit nature of file objects to close
themselves? It's not nearly as easy to read.
Then again, list comprehensions seem somewhat cryptic for Python as well.
Mike
--
Michael P. Soulier, QX41, SKY Tel: 613-765-4699 (ESN: 39-54699)
Optical Networks, Nortel Networks, SDE Pegasus
"...the word HACK is used as a verb to indicate a massive amount
of nerd-like effort." -Harley Hahn, A Student's Guide to Unix
Huaiyu Zhu
>> From: Donn Cave [mailto:do...@drizzle.com]
>>
>> You're describing a language other than C Python. In C Python, you
>> *may* close files and rely on other kinds of finalizations through
>> the reference counting garbage collector, and the result is as timely
>> as if you did it explicitly. That's finalization as it ought to be,
>> and it's a shame that other languages compromise on that. Lots of
>> people don't think it's important, but then the majority of
>> programmers
>> think it's OK that their programming language requires them to keep
>> track of memory in the same way.
>
>I would be very happy for this to be the case.
>
>However, CPython is not a language where you can rely on timely
>finalisation.
>
>1. The documentation explicitly states that you cannot. Anything else is an
>implementation artifact.
There is a fine difference between what you can rely on for generic Python
and what you can rely on for CPython. Should we regard the documentation as
the most authoritative source for CPython in particular? Obviously when it
comes the other way, no one would deny that you can rely on the existence of
certain Java classes in Jython which do not exist in Python documentation.
I've been relying on the timely garbage collection by ref-counting. After
reading the whole thread I think I'd still trust it given what François
wrote about implicit guarantees. These things cannot be broken in future
versions without causing a big uproar.
When porting to Jython there are other issues to consider beside open files.
They are part of the porting cost. To guard against them all in CPython for
any program that might never be ported is likely to cost much more.
The rest of the problems you listed concern situations where ref-count does
not go down to zero, which is a different issue from the one-liners in
consideration.
>
>2. The thing you expect to be finalised may have another reference
>somewhere. Are you 100% sure that no function you've passed it to has held a
>reference to it in a cache?
>
>3. Your object may be in a cycle. This will result in a non-timely
>finalisation.
>
>4. Your object may be in a cycle where Python explicitly punts and doesn't
>run the finalisers.
>
>Tim Delaney
>
>
Huaiyu
John La Rooy
"Michael P. Soulier" <msou...@nortelnetworks.com_.nospam> wrote:
> On Wed, 29 May 2002 21:02:21 +0000 (UTC), Huaiyu Zhu
> <hua...@gauss.almadan.ibm.com> wrote:
> >>>
> >>>>>> stuff = "stuff to write"
> >>>>>> open('stufffile', "w").write(stuff)
> >>>>>> open('stufffile', "r").read()
> >>> 'stuff to write'
> >>>
> >>
> >>yes, but can you disagree with the statement that it is bad programming
> >>practice. I sure can't.
> >
> > For what reason is it bad?
>
> Well, as one of the credos of Python is that explicit is better than
> implicit, why are we relying on the implicit nature of file objects to close
> themselves? It's not nearly as easy to read.
> Then again, list comprehensions seem somewhat cryptic for Python as well.
>
Do you explicitly del all your objects too, or wait for them to disappear when they go out of scope?
John
Isaac To
Donn> You're describing a language other than C Python. In C Python,
Donn> you *may* close files and rely on other kinds of finalizations
Donn> through the reference counting garbage collector, and the result
Donn> is as timely as if you did it explicitly. That's finalization as
Donn> it ought to be, and it's a shame that other languages compromise
Donn> on that. Lots of people don't think it's important, but then the
Donn> majority of programmers think it's OK that their programming
Donn> language requires them to keep track of memory in the same way.
If you don't treat what said by the manual as the contract of the language,
then yes. The manual explicitly says that future versions of the language
might not work that way.
Regards,
Isaac.
Michael P. Soulier
>
> Do you explicitly del all your objects too, or wait for them to disappear
> when they go out of scope?
I only delete them in C++. ;-) Your point is taken, but we must draw the
line somewhere. If you don't care when the file is closed, as long as it is,
then implicit should be fine. If however you do care when it is closed, then
explicit should be better, no?
John La Rooy
pin...@iro.umontreal.ca (François Pinard) wrote:
And certainly if something looks safe, ideally it should be so. Why do we close
files at all ever? Explicitly closing a file *should* be as rare as explicitly
deleting a variable. The reason we do it is because we *know* something about
the underlying implementation i.e. that files are buffered. I hope that CPython
is never going to change that behaviour of closing the file for me when refcount=0
and I think the Jython developers should consider a mechanism for doing it too.
When refcount gets down to 0 and we make an object available for GC can't we ask
is it a file? If so we better flush and close it?
One of the reasons Python is quick to develop in is that we don't have to worry
about memory management much (but we can if we need to). Likewise the less I have
to deal with the fact that some I/O is buffered in my day to day work the better
(as long as I can if I need to).
John
Paul Foley
>> Well, as one of the credos of Python is that explicit is better than
>> implicit, why are we relying on the implicit nature of file objects to close
>> themselves? It's not nearly as easy to read.
> Do you explicitly del all your objects too, or wait for them to disappear when they go out of scope?
Neither. You can't delete objects in Python ("del" doesn't do
anything to objects; it does something to variables), and in theory
they persist forever; GC is just an optimization to allow reuse of
memory used by objects that the program will never access again,
because you don't really have infinite memory.
Correct optimizations don't change the behaviour of correct code.
[Therefore, if the behaviour of your code does change depending on
whether or not the optimization is done, either the optimization is
in error or your code is. Thus, relying on GC to close files (or do
anything else) is Wrong. Python really ought to print a warning when
it GCs an open file]
--
You don't have to agree with me; you can be wrong if you want.
(setq reply-to
(concatenate 'string "Paul Foley " "<mycroft" '(#\@) "actrix.gen.nz>"))
John La Rooy
"Michael P. Soulier" <msou...@nortelnetworks.com_.nospam> wrote:
> On Fri, 31 May 2002 22:55:04 +1200, John La Rooy <lar...@xtar.co.nz> wrote:
> >
> > Do you explicitly del all your objects too, or wait for them to disappear
> > when they go out of scope?
>
> I only delete them in C++. ;-) Your point is taken, but we must draw the
> line somewhere. If you don't care when the file is closed, as long as it is,
> then implicit should be fine. If however you do care when it is closed, then
> explicit should be better, no?
>
> Mike
I've often used the 'open(f, "w").write()' right at the end of a program where
I know the file is about to get closed anyway. From what I've seen here about
Jython, it seems that I can't even count on that behaviour. eeek!
Perhaps the specification will some day change to make the objects returned by
open() to be *smarter* so we can count on them knowing when they should flush
and when they should close the toilet seat.
A few people seem to be getting stuck on the explicit instead of implicit idea
but I think that is mostly because we are used to be told to close files
and we've all had a few late nights because we forgot to. Just as lots of us
learned we should always free() our mallocs() and now we think nothing of
letting python do that housekeeping for us.
cloudcuckoo-land-ly y'rs - John
John La Rooy
Paul Foley <see@below> wrote:
Good point. If I had infinite memory and was allowed an infinite number of
open files, I'd never need to close or flush my file objects, because the
optimisation between my writing to a file and the actual event taking place
will have no effect on the behaviour of my code. But it DOES have an effect
doesn't it? Buffering is an optimisation, isn't it?
John
Isaac To
John> Good point. If I had infinite memory and was allowed an infinite
John> number of open files, I'd never need to close or flush my file
John> objects, because the optimisation between my writing to a file and
John> the actual event taking place will have no effect on the behaviour
John> of my code. But it DOES have an effect doesn't it? Buffering is an
John> optimisation, isn't it?
Is it remotely possible to make buffering to be transparent to the
programmer? I don't think so: at least, when the language needs to define
the concurrency semantics: when two programs open the same file, when is it
guaranteed that the second program will correctly read the things written by
the first program. It is not something that the programmer can say "okay,
I'll let the system to figure that out"... or do you have some new ideas
about it?
Regards,
Isaac.
Isaac To
John> I've often used the 'open(f, "w").write()' right at the end of a
John> program where I know the file is about to get closed anyway. From
John> what I've seen here about Jython, it seems that I can't even count
John> on that behaviour. eeek!
John> Perhaps the specification will some day change to make the objects
John> returned by open() to be *smarter* so we can count on them knowing
John> when they should flush and when they should close the toilet seat.
John> A few people seem to be getting stuck on the explicit instead of
John> implicit idea but I think that is mostly because we are used to be
John> told to close files and we've all had a few late nights because we
John> forgot to. Just as lots of us learned we should always free() our
John> mallocs() and now we think nothing of letting python do that
John> housekeeping for us.
That's possible if you just want to make sure that all files are closed at
the end of execution: just keep a list of opened files objects somewhere
(perhaps using weak references?), and, when the program exits, close all of
them. This is standard OS technique (all process has a list of opened file,
which are closed on exit). It is even easy to implement at library level.
But that won't solve all the problems. In particular, if the output of your
files are read by somebody else, you have to know at some time that the
content is valid for reading.
Regards,
Isaac.
Delaney, Timothy
>
> > Me ...
> >
> > __files = []
> >
> > def open (path, mode):
> > f = __open(path, mode)
> > __files.append(f)
> > return f
> >
> > or
> >
> > class file:
> >
> > __files = []
> >
> > def __init__(self, path, mode):
> > __files.append(self)
> >
> > Whilst I know this is not the implementation, there is nothing that
> > prevents it from being like this. So you get back an object
> that has one
> > more reference count than expected. It never gets collected.
> Well, then the implementation is broken. If you allow the
> implementation
> to collect additional references at random places, you can prove quite
> easily that nothing is safe :)
Not at all. My implementation conforms 100% to the documented behaviour of
the function/type. I should be able to replace __builtin__.file with the
above implementation and expect everything to work (except that I may run
out of memory - but that *also* conforms 100% to documented behaviour).
Indeed, in 2.2.8 File Objects there is in fact no mention that the file
object will be closed when its destructor is called. I searched through much
of the rest of the docs and found no reference to a file object destructor.
Therefore, my above implementation of class file is completely correct in
that it does not have a destructor.
> > You should never rely on automatic freeing of *any*
> resources unless it is
> > guaranteed. Always explicitly free resources when you have
> finished with
> > them.
> Well, then you should del all your names too.
> Oops wait, how can we be sure that del works? What if del
> also increments the
> refcount instead of decrementing?
"""
__del__(self)
Called when the instance is about to be destroyed. This is also
called a destructor If a base class has a __del__() method, the
derived class's __del__() method must explicitly call it to ensure
proper deletion of the base class part of the instance. Note that it
is possible (though not recommended!) for the __del__() method to
postpone destruction of the instance by creating a new reference to
it. It may then be called at a later time when this new reference is
deleted. It is not guaranteed that __del__() methods are called for
objects that still exist when the interpreter exits.
Note: "del x" doesn't directly call x.__del__() -- the former
decrements the reference count for x by one, and the latter is only
called when its reference count reaches zero. Some common situations
that may prevent the reference count of an object to go to zero
include: circular references between objects (e.g., a doubly-linked
list or a tree data structure with parent and child pointers); a
reference to the object on the stack frame of a function that caught
an exception (the traceback stored in sys.exc_traceback keeps the
stack frame alive); or a reference to the object on the stack frame
that raised an unhandled exception in interactive mode (the traceback
stored in sys.last_traceback keeps the stack frame alive). The first
situation can only be remedied by explicitly breaking the cycles; the
latter two situations can be resolved by storing None in
sys.exc_traceback or sys.last_traceback. Circular references which
are garbage are detected when the option cycle detector is enabled
(it's on by default), but can only be cleaned up if there are no
Python-level __del__() methods involved. Refer to the documentation
for the gc module for more information about how __del__() methods
are handled by the cycle detector, particularly the description of
the garbage value.
Warning: Due to the precarious circumstances under which __del__()
methods are invoked, exceptions that occur during their execution are
ignored, and a warning is printed to sys.stderr instead. Also, when
__del__() is invoked in response to a module being deleted (e.g.,
when execution of the program is done), other globals referenced by
the __del__() method may already have been deleted. For this reason,
__del__() methods should do the absolute minimum needed to maintain
external invariants. Python 1.5 guarantees that globals whose name
begins with a single underscore are deleted from their module before
other globals are deleted; if no other references to such globals
exist, this may help in assuring that imported modules are still
available at the time when the __del__() method is called.
"""
An implementation of del which failed to decrement the refcount by one would
not conform to the documented behaviour, and hence would be broken.
Also, in reference to the rest of this thread, note the sentence "It is not
guaranteed that __del__() methods are called for objects that still exist
when the interpreter exits."
> Basically what you have proven is, that if you have a faulty open()
> implementation, the open(f,t).write(d) idiom doesn't work.
So is the Jython implementation faulty? The open(f, t).write(d) idiom does
not work there (it may, if you get lucky).
> In respect to CPython, refcounting behaviour, or more general
> retaining
> references in any gc regime, is a quite important part of the
> interface.
It is explicitly not a documented part of the interface - the documented
part is that it is not safe to rely on this behaviour.
When it is explicitly documented that this behaviour is guaranteed now and
forever, for all implementations of Python after a specified version, I will
*consider* using this behaviour. Until then it is not worthy of
consideration.
Tim Delaney
Kragen Sitaker
> GC is just an optimization to allow reuse of
> memory used by objects that the program will never access again,
> because you don't really have infinite memory.
>
> Correct optimizations don't change the behaviour of correct code.
>
> [Therefore, if the behaviour of your code does change depending on
> whether or not the optimization is done, either the optimization is
> in error or your code is.
So all finalizers should be considered bugs? I think that's a
supremely unpragmatic approach to programming.
John La Rooy
>
> It is explicitly not a documented part of the interface - the documented
> part is that it is not safe to rely on this behaviour.
>
> When it is explicitly documented that this behaviour is guaranteed now and
> forever, for all implementations of Python after a specified version, I will
> *consider* using this behaviour. Until then it is not worthy of
> consideration.
>
> Tim Delaney
>
>
It's unlikely to be explicitly documented. from the docs...
The file() constructor is new in Python 2.2. The previous spelling, open(),
is retained for compatibility, and is an alias for file().
So you shouldn't be using open() in new programs anyway
John
Steve Holden
You might want to consider using it for exactly the reason it's been
retained: backward compatibility to older versions of Python in use by your
potential users.
regards