Autoflushing in Python

閲覧: 3 回
最初の未読メッセージにスキップ

Hrvoje Niksic

未読、
1999/02/06 3:00:001999/02/06
To:
Is there a way to say that you want your file object to be autoflushed
(i.e. flushed after every write())? This would be very useful
occasionally.

I tried turning off the buffering while creating the object, but it
does the wrong thing -- it makes all the writes be one byte long (I'm
not sure if this is a bug or a feature). Even if that were fixed, it
would still not be enough, because I would like to be able to make an
*existing* file object (such as sys.stdout or sys.stderr)
auto-flushing.

Kevin Cazabon

未読、
1999/02/06 3:00:001999/02/06
To:
I think the easiest way would be to define your own write function that
includes the file.flush() at the end...

I'd probably do it something like:

def write_flush(self, data):
self.write(data)
self.flush()

then usage would be:

file=open(filename, 'w')

file.write_flush(data)


or something to that effect...

Kevin Cazabon.


Hrvoje Niksic wrote in message <87lnibz...@pc-hrvoje.srce.hr>...

Kevin Cazabon

未読、
1999/02/06 3:00:001999/02/06
To:
For that attribute to work, you'd of course have to include it in a class of
which your file belonged... if you just want a straight function, use this:

#function

def write_flush(file, data):
file.write(data)
file.flush()
file.close()

#usage

file = open('c:\\temp\\eggs.txt', 'w')
data = 'spam like this'

write_flush(file, data)

I know it's a pain to go replace all your "write"'s, usually a single
file.flush() at the end is sufficient for most usages...

Kevin Cazabon

未読、
1999/02/06 3:00:001999/02/06
To:
Ok.. here's the class version that would work:

#function
class outfile:

def write_flush(self, filename, data):
self = open(filename, 'w')
self.write(data)
self.flush()
self.close()

#usage
file = outfile()

filename = 'c:\\temp\\eggs.txt'

data = 'spam spam, eggs, and spam like this'

file.write_flush(filename, data)


Hrvoje Niksic

未読、
1999/02/07 3:00:001999/02/07
To:
"Kevin Cazabon" <kcaz...@rogers.wave.ca> writes:

> I think the easiest way would be to define your own write function
> that includes the file.flush() at the end...

That may be the easiest way, but that's certainly not what I need
here. For one, I have to change all the instances of "write" to use
the "write_flush" kludge. Second, some built-in things won't honor
the novelty right; for example, the "print" built-in.

> I'd probably do it something like:
>
> def write_flush(self, data):
> self.write(data)
> self.flush()
>
> then usage would be:
>
> file=open(filename, 'w')
>
> file.write_flush(data)

That doesn't work:

>>> file.write_flush("haha")
Traceback (innermost last):
File "<stdin>", line 1, in ?
AttributeError: write_flush

Hrvoje Niksic

未読、
1999/02/07 3:00:001999/02/07
To:
"Kevin Cazabon" <kcaz...@rogers.wave.ca> writes:

> write_flush(file, data)

...which is, as I said, ugly. If there is no way to set up
autoflushing in Python (as seems to be the case), perhaps it is a
feature that should be added?

Paul Prescod

未読、
1999/02/07 3:00:001999/02/07
To:
Kevin Cazabon wrote:
>
> filename = 'c:\\temp\\eggs.txt'
>
> data = 'spam spam, eggs, and spam like this'
>
> file.write_flush(filename, data)

Wouldn't it be better to make an object that actually took a file object
as input and looked like a file object?

class flush_file:
def __init__( self, file ):
self-_file=file
def write( self, data ):
self._file.write( data )
self._file.flush()

sys.stderr = flush_file( sys.stderr )
sys.stdin = flush_file( sys.stdin )

--
Paul Prescod - ISOGEN Consulting Engineer speaking for only himself
http://itrc.uwaterloo.ca/~papresco

"Remember, Ginger Rogers did everything that Fred Astaire did,
but she did it backwards and in high heels."
--Faith Whittlesey

Hans Nowak

未読、
1999/02/07 3:00:001999/02/07
To:
Ce'Nedra took her magical amulet and heard Hrvoje Niksic say:

>Is there a way to say that you want your file object to be autoflushed
>(i.e. flushed after every write())? This would be very useful
>occasionally.
>
>I tried turning off the buffering while creating the object, but it
>does the wrong thing -- it makes all the writes be one byte long (I'm not
>sure if this is a bug or a feature). Even if that were fixed, it would
>still not be enough, because I would like to be able to make an *existing*
>file object (such as sys.stdout or sys.stderr) auto-flushing.

You could redirect sys.stdout... The way to go is to write a file wrapper
class which does support autoflushing, then redirect sys.stdout (or
sys.stderr, or any existing file) to it. Here's an example:

#

import sys, time

class File:
def __init__(self, handle):
self.f = handle
def flush(self):
self.f.flush()
def write(self, str):
self.f.write(str)
self.f.flush()
# add more methods to your liking

oldstdout = sys.stdout
sys.stdout = File(sys.stdout)

for i in range(5):
print "Hello",
time.sleep(0.5)
print

#

This should print "Hello", five times, on the same line, with delays of 0.5
second between each Hello. This won't work without flushing.

You might want to write a File class with more methods so it mimicks a real
file object better. This will do for this purpose, though. And you can always
'turn off' autoflushing by saying

sys.stdout = oldstdout

which restores the original standard output. Simple as that. :^)

+ Hans Nowak (Zephyr Falcon)
+ Homepage (under construction): http://www.cuci.nl/~hnowak/
+ You call me a masterless man. You are wrong. I am my own master.
+ May Chelsea get infected with your underdog!

Hrvoje Niksic

未読、
1999/02/07 3:00:001999/02/07
To:
"Hans Nowak" <ivn...@hvision.nl> writes:

> You could redirect sys.stdout... The way to go is to write a file
> wrapper class which does support autoflushing, then redirect
> sys.stdout (or sys.stderr, or any existing file) to it. Here's an
> example:

[...]

Your solution works for stdout and stderr, but it's not the general
thing I was looking for. I would like the existing objects to be
modifiable with code resembling this:

fileobject.autoflush(1) # Turn on flushing

Assigning to sys.stdout and sys.stderr works as long as everyone is
careful to look them up correctly. In other cases, it is essential to
be able to modify existing file objects to be autoflushing, and
optionally reset them again.

Perl has this in the form of the "->autoflush()" method (which used to
be called $|), and it's a useful thing to have.

Lars Marius Garshol

未読、
1999/02/07 3:00:001999/02/07
To:

* Hrvoje Niksic

|
| ...which is, as I said, ugly. If there is no way to set up
| autoflushing in Python (as seems to be the case), perhaps it is a
| feature that should be added?

I've been thinking the same thing for a while. It is possible to make
Python auto-flush by using the -u startup option, but IMHO that's not
sufficient. There should be a way to do this from inside Python.

One interim solution might be the one proposed by Kevin Cazabon & Paul
Prescod.

--Lars M.

Hrvoje Niksic

未読、
1999/02/07 3:00:001999/02/07
To:
Lars Marius Garshol <lar...@ifi.uio.no> writes:

> * Hrvoje Niksic
> |
> | ...which is, as I said, ugly. If there is no way to set up
> | autoflushing in Python (as seems to be the case), perhaps it is a
> | feature that should be added?
>
> I've been thinking the same thing for a while. It is possible to make
> Python auto-flush by using the -u startup option, but IMHO that's not
> sufficient.

Interesting; I didn't know about this.

> There should be a way to do this from inside Python.

Definitely. And, most importantly, it should be doable on a
per-file-object basis.

Lloyd Zusman

未読、
1999/02/07 3:00:001999/02/07
To:
"Kevin Cazabon" <kcaz...@rogers.wave.ca> writes:

> Ok.. here's the class version that would work:
>
> #function
> class outfile:
>
> def write_flush(self, filename, data):
> self = open(filename, 'w')
> self.write(data)
> self.flush()
> self.close()
>
>
>
> #usage
> file = outfile()
>

> filename = 'c:\\temp\\eggs.txt'
>
> data = 'spam spam, eggs, and spam like this'
>
> file.write_flush(filename, data)

But isn't the self.flush() call redundant here? After all, this
method calls self.close() immediately after the write. This isn't at
all the case that Hrvoje Niksic is talking about. Consider this:

class file:

def open(self, filename, mode):
self.fhandle = open(filename, mode)
self.doflush = None
return self

def autoflush(self, value = 1):
previous = self.doflush
self.doflush = value
return previous

def write(self, data):
self.fhandle.write(data)
if self.doflush:
self.fhandle.flush()

I left out a lot of details, error handling, etc. ... but I believe that
this summarizes the desired behavior.

--
Lloyd Zusman
l...@asfast.com

jeff...@bigfoot.com

未読、
1999/02/08 3:00:001999/02/08
To: hni...@srce.hr
Hrvoje Niksic <hni...@srce.hr> wrote:
> Lars Marius Garshol <lar...@ifi.uio.no> writes:
>
> > * Hrvoje Niksic
> > |
> > | ...which is, as I said, ugly. If there is no way to set up
> > | autoflushing in Python (as seems to be the case), perhaps it is a
> > | feature that should be added?
> >
> > I've been thinking the same thing for a while. It is possible to make
> > Python auto-flush by using the -u startup option, but IMHO that's not
> > sufficient.
>
> Interesting; I didn't know about this.

Setting the environment variable PYTHONUNBUFFERED to any non-null
value achieves the same effect as the '-u' option. This is useful
for CGI programs.

Regards,

Jeff Bauer
Rubicon, Inc.

-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/ Search, Read, Discuss, or Start Your Own

Dieter Maurer

未読、
1999/02/08 3:00:001999/02/08
To:
Lars Marius Garshol <lar...@ifi.uio.no> writes on 07 Feb 1999 20:43:18 +0100:

> * Hrvoje Niksic
> |
> | ...which is, as I said, ugly. If there is no way to set up
> | autoflushing in Python (as seems to be the case), perhaps it is a
> | feature that should be added?
>
> I've been thinking the same thing for a while. It is possible to make
> Python auto-flush by using the -u startup option, but IMHO that's not
> sufficient. There should be a way to do this from inside Python.
>
> One interim solution might be the one proposed by Kevin Cazabon & Paul
> Prescod.
I followed the thread only partially and did not see Kevins and Pauls
proposal. Maybe, I repeat it.

The Python functions that create file objects (i.e.
"open", "fdopen" and "makefile") have an optional argument
"bufsize". "0" means unbuffered (i.e. autoflushing),
"1" means line buffered (i.e. autoflushing after '\n')
and any other positive value means to use a buffer of approx.
that size.

If you want "sys.stdout" to be unbuffered, you may try
"sys.stdout= os.fdopen(os.dup(1),'w',0)" (the "dup" may
not be necessary, it is to prevent finalization of the
original "sys.stdout" to close you file; as far as I know,
this object is at another safe place, and thus probably will
not be finalized).

Dieter

Guido van Rossum

未読、
1999/02/09 3:00:001999/02/09
To:
> Is there a way to say that you want your file object to be autoflushed
> (i.e. flushed after every write())? This would be very useful
> occasionally.
>
> I tried turning off the buffering while creating the object, but it
> does the wrong thing -- it makes all the writes be one byte long (I'm
> not sure if this is a bug or a feature).

Hmm... That must be a bug in the stdio library you're using!
What platform are you using?

Remember (or perhaps you didn't know -- so now you do :-) that Python
file objects are a thin layer around C stdio objects. The obvious way
to do what you ask for is indeed to open the file unbuffered. If the
stdio library doesn't do the right thing, it should be fixed -- I
would think that it is also causing bad performance for other (C)
applications!

> Even if that were fixed, it
> would still not be enough, because I would like to be able to make an
> *existing* file object (such as sys.stdout or sys.stderr)
> auto-flushing.

This has been answered already. (-u or PYTHONUNBUFFERED=1)

To Python's defense: C stdio doesn't provide a way to turn off
buffering after reading or writing to the file has started -- that's
why this has to be done when the file is opened.

In Python 2.0, we may decide to forget about the stdio library and
write a Python-specific I/O library that does the right thing
everywhere...

--Guido van Rossum (home page: http://www.python.org/~guido/)

Christopher G. Petrilli

未読、
1999/02/09 3:00:001999/02/09
To:
Hrvoje Niksic <hni...@srce.hr> wrote:

> If you say it's a libc bug, I have no choice but to trust you.
> Perhaps Python should attempt to work around that particular bug?

Um, I'd prefer Python didn't become even more littered with
"workarounds" for brain-damaged libraries... it's bad enough on a
platform bases (i.e. the difference between Mac/PC/Unix), but to have
"sub-platform" tweeks would just confuse the user.

> True, but autoflushing is a lot simpler than that -- wherever it is
> that Python C code calls "fwrite()", we could check if the autoflush
> behaviour is desired and, if yes, call an additional fflush(). As
> simple as that (at least theoretically; I haven't looked at the code
> yet).

Dear Gods, the overhead :-) Weren't you just complaining about speed in
a previous post, and you want to add a check to EVERY fwrite() call to
see if something that BELONGS i nthe stdio library should be worked
around? This sounds like a real boondoggle... perhaps we should also
check midstream to find out if the processor has SUDDENLY switched from
bigendian to little... yes that would help us!

amazingly-tired-of-the-nitting-without-the-code-to-back-it-up'ly yours,
Chris
--
| Christopher Petrilli
| petr...@amber.org

Chad Netzer

未読、
1999/02/09 3:00:001999/02/09
To:
Hrvoje Niksic wrote:

> Guido van Rossum <gu...@CNRI.Reston.VA.US> writes:
>
> > > I tried turning off the buffering while creating the object, but it
> > > does the wrong thing -- it makes all the writes be one byte long (I'm
> > > not sure if this is a bug or a feature).
> >
> > Hmm... That must be a bug in the stdio library you're using! What
> > platform are you using?
>

> Debian Linux, libc6. I've double-checked, and yes, the bug does
> exists, whoever is responsible. Running this program:
>
> import sys
> x = open ('/etc/passwd', 'r', 0)
> for i in x.readlines ():
> sys.stdout.write (i)
>
> results in the following strace:
>
> open("/etc/passwd", O_RDONLY) = 5
> read(5, "r", 1) = 1
> read(5, "o", 1) = 1
> [... all the file here ...]
> write(1, "root:x:0:0:root:/root:/usr/bin/z"..., 35) = 35
> [...]

But this is the correct behavior, no? Unbuffered reads are one byte, sure,
but writes are the proper length.

Try this example:

import sys
x = open ('/etc/passwd', 'r', 0)
y = open ('passwd_copy', 'w', 0)
for i in x.readlines ():
y.write (i)


On my RedHat 5.2 system, the reads behave as on your program, and the writes
are (properly) one line each. By removing the "0" argument from the y object
open statement, the write gets combined into bigger, multi-line writes.

Chad Netzer
ch...@vision.arc.nasa.gov

Hrvoje Niksic

未読、
1999/02/10 3:00:001999/02/10
To:
Guido van Rossum <gu...@CNRI.Reston.VA.US> writes:

> > Is there a way to say that you want your file object to be autoflushed
> > (i.e. flushed after every write())? This would be very useful
> > occasionally.
> >

> > I tried turning off the buffering while creating the object, but it
> > does the wrong thing -- it makes all the writes be one byte long (I'm
> > not sure if this is a bug or a feature).
>
> Hmm... That must be a bug in the stdio library you're using! What
> platform are you using?

Debian Linux, libc6. I've double-checked, and yes, the bug does
exists, whoever is responsible. Running this program:

import sys
x = open ('/etc/passwd', 'r', 0)
for i in x.readlines ():
sys.stdout.write (i)

results in the following strace:

open("/etc/passwd", O_RDONLY) = 5
read(5, "r", 1) = 1
read(5, "o", 1) = 1
read(5, "o", 1) = 1

read(5, "t", 1) = 1
read(5, ":", 1) = 1
read(5, "x", 1) = 1


[... all the file here ...]
write(1, "root:x:0:0:root:/root:/usr/bin/z"..., 35) = 35
[...]

If you say it's a libc bug, I have no choice but to trust you.


Perhaps Python should attempt to work around that particular bug?

> > Even if that were fixed, it would still not be enough, because I


> > would like to be able to make an *existing* file object (such as
> > sys.stdout or sys.stderr) auto-flushing.
>
> This has been answered already. (-u or PYTHONUNBUFFERED=1)

But this makes *all* the file objects unbuffered, doesn't it?

> To Python's defense: C stdio doesn't provide a way to turn off
> buffering after reading or writing to the file has started

True, but autoflushing is a lot simpler than that -- wherever it is

Tim Peters

未読、
1999/02/10 3:00:001999/02/10
To:
[Hrvoje Niksic]

>>> Is there a way to say that you want your file object to be autoflushed
>>> (i.e. flushed after every write())? This would be very useful
>>> occasionally.
>>>
>>> I tried turning off the buffering while creating the object, but it
>>> does the wrong thing -- it makes all the writes be one byte long (I'm
>>> not sure if this is a bug or a feature).

[Guido]


>> Hmm... That must be a bug in the stdio library you're using! What
>> platform are you using?

Guido, what are you talking about <0.9 wink>?

[Hrvoje]


> Debian Linux, libc6. I've double-checked, and yes, the bug does
> exists, whoever is responsible. Running this program:
>
> import sys
> x = open ('/etc/passwd', 'r', 0)
> for i in x.readlines ():
> sys.stdout.write (i)
>
> results in the following strace:
>
> open("/etc/passwd", O_RDONLY) = 5
> read(5, "r", 1) = 1
> read(5, "o", 1) = 1
> read(5, "o", 1) = 1
> read(5, "t", 1) = 1
> read(5, ":", 1) = 1
> read(5, "x", 1) = 1
> [... all the file here ...]
> write(1, "root:x:0:0:root:/root:/usr/bin/z"..., 35) = 35
> [...]

Well, I'm confused! We started here by wanting writes to autoflush, and
ended with a test case that shows unbuffered reads. What is it that you
would like the read to do here? It doesn't surprise me that an unbuffered
read returns one byte at a time (seems to me that's what unbuffered *means*
<wink>). If you opened via

x = open('/etc/passwd', 'r', 1)

it should read a line at a time instead. But what does that have to do with
writes? Or is the complaint that sys.stdout is line-buffered?

>>> Even if that were fixed, it would still not be enough, because I
>>> would like to be able to make an *existing* file object (such as
>>> sys.stdout or sys.stderr) auto-flushing.

>> This has been answered already. (-u or PYTHONUNBUFFERED=1)

> But this makes *all* the file objects unbuffered, doesn't it?

Good news: no. From the command-line help:

-u : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)

Other files you open yourself so you get to set the buffering you want when
you open them. stdout and stderr are opened automatically by Python, and as
Guido said libc doesn't let Python change its mind later, hence the
command-line option to influence how Python opens those specifically.

> ...


> True, but autoflushing is a lot simpler than that -- wherever it is
> that Python C code calls "fwrite()", we could check if the autoflush
> behaviour is desired and, if yes, call an additional fflush(). As
> simple as that (at least theoretically; I haven't looked at the code
> yet).

I don't think you've made a case that this extra fiddling is useful enough
to justify building it in; if you want line buffering or no buffering the
"open" call supports it directly; and if you want to switch on the fly, it's
no trick to wrap up the suggestion you make here in a simple file-spoofing
class's writexxx() methods. Python doesn't try to cater to all conceivable
behaviors directly, but does try to let you build whatever unusual behaviors
you may want.

python-is-actually-proud-of-not-being-perl<wink>-ly y'rs - tim

Hrvoje Niksic

未読、
1999/02/10 3:00:001999/02/10
To:
Christopher G. Petrilli <petr...@dworkin.amber.org> writes:

> > True, but autoflushing is a lot simpler than that -- wherever it is
> > that Python C code calls "fwrite()", we could check if the autoflush
> > behaviour is desired and, if yes, call an additional fflush(). As
> > simple as that (at least theoretically; I haven't looked at the code
> > yet).
>

> Dear Gods, the overhead :-)

You're not serious, are you? OK, the smiley is there. :-)

> Weren't you just complaining about speed in a previous post, and you
> want to add a check to EVERY fwrite() call to see if something that
> BELONGS i nthe stdio library should be worked around?

Exactly. That is the simplest way to implement "autoflushing", and I
don't think there would be any speed problems with that additional
"if".

But that's not all that important -- the important question is whether
the users *want* such a feature. While I see the value of having it,
others apparently don't, so it won't get in. Oh well.

> This sounds like a real boondoggle... perhaps we should also check
> midstream to find out if the processor has SUDDENLY switched from
> bigendian to little... yes that would help us!

Whatever.

Chad Netzer

未読、
1999/02/10 3:00:001999/02/10
To:
Tim Peters wrote:

> It doesn't surprise me that an unbuffered
> read returns one byte at a time (seems to me that's what unbuffered *means*
> <wink>). If you opened via
>
> x = open('/etc/passwd', 'r', 1)
>
> it should read a line at a time instead.
>

> python-is-actually-proud-of-not-being-perl<wink>-ly y'rs - tim

Just an addendum; when I was testing my script (see my previous post in
this thread) I noticed that using the `1' option for reads did NOT make it line
buffered on my system (RedHat 5.2 w/ glibc2 (libc6)). The buffers were 4096
bytes instead. I'm sure that is a libc implementation detail, as the man pages
on Irix 6.3 and RedHat 5.2 don't specify what line-buffered reads should do
(although they are explicit about line-buffered writes).

Oh well, just an FYI.

Chad Netzer
ch...@vision.arc.nasa.gov

全員に返信
投稿者に返信
転送
新着メール 0 件