Unbuffered stderr in Python 3

[Steven D'Aprano]

In Python 2, stderr is unbuffered.

In most other environments (the shell, C...) stderr is unbuffered.

It is usually considered a bad, bad thing for stderr to be buffered. What
happens if your application is killed before the buffer fills up? The errors
in the buffer will be lost.

So how come Python 3 has line buffered stderr? And more importantly, how can
I turn buffering off?

I don't want to use the -u unbuffered command line switch, because that
effects stdout as well. I'm happy for stdout to remain buffered.


Here is the function I'm using to test this in in the interactive
interpreter:

import sys, time

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):
print(i, file=sys.stderr, end="")
time.sleep(2)
print("", file=sys.stderr)


Running it pauses for 20 seconds, then displays "0123456789" in one go. What
I expect is to see the digits arrive individually.

I tried this:

py> sys.stderr.line_buffering = False
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: readonly attribute


Next I tried this:

py> sys.stderr.buffer.line_buffering = False


which succeeded, but has no effect on print: it still buffers.




--
Steve

[Wolfgang Maier]

On 02.11.2015 08:52, Steven D'Aprano wrote:
> In Python 2, stderr is unbuffered.
>
> In most other environments (the shell, C...) stderr is unbuffered.
>
> It is usually considered a bad, bad thing for stderr to be buffered. What
> happens if your application is killed before the buffer fills up? The errors
> in the buffer will be lost.
>
> So how come Python 3 has line buffered stderr? And more importantly, how can
> I turn buffering off?
>

I cannot comment on your first question, but unbuffered writing to
sys.stderr is possible just like for any other buffered file object. You
just call its flush method after writing.

Since Python3.3, the print function has a flush keyword argument that
accepts a boolean and lets you do just this. Rewrite your example as:

import sys, time

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):

_print(i, file=sys.stderr, end="", flush=True)
time.sleep(2)
print("", file=sys.stderr)

and it should do what you want.


Before Python 3.3, if I needed an unbuffered print function, I used a
wrapper to call flush for me, so instead of:

>
> import sys, time
>
> def test():
> # Simulate a slow calculation that prints status and/or error
> # messages to stderr.
> for i in range(10):
> print(i, file=sys.stderr, end="")
> time.sleep(2)
> print("", file=sys.stderr)
>

I'd use (and still do this if I need compatibility with 3.2):

import sys, time

def _print (*args, **kwargs):
file = kwargs.get('file', sys.stdout)
print(*args, **kwargs)
file.flush()

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):

_print(i, file=sys.stderr, end="")
time.sleep(2)
print("", file=sys.stderr)

Best,
Wolfgang

[Wolfgang Maier]

On 02.11.2015 11:48, Wolfgang Maier wrote:
>
> Since Python3.3, the print function has a flush keyword argument that
> accepts a boolean and lets you do just this. Rewrite your example as:
>

> import sys, time
>
> def test():
> # Simulate a slow calculation that prints status and/or error
> # messages to stderr.
> for i in range(10):

> _print(i, file=sys.stderr, end="", flush=True)
> time.sleep(2)
> print("", file=sys.stderr)
>
> and it should do what you want.

sorry for this mess (copy pasted from the wrong source).
The code should be:

def test():
# Simulate a slow calculation that prints status and/or error
# messages to stderr.
for i in range(10):

[Nobody]

On Mon, 02 Nov 2015 18:52:55 +1100, Steven D'Aprano wrote:

> In Python 2, stderr is unbuffered.
>
> In most other environments (the shell, C...) stderr is unbuffered.
>
> It is usually considered a bad, bad thing for stderr to be buffered. What
> happens if your application is killed before the buffer fills up? The
> errors in the buffer will be lost.
>
> So how come Python 3 has line buffered stderr? And more importantly, how
> can I turn buffering off?

It's probably related to the fact that std{in,out,err} are Unicode
streams.

> type(sys.stderr)
<class '_io.TextIOWrapper'>
> type(sys.stderr.buffer)
<class '_io.BufferedWriter'>
> type(sys.stderr.buffer.raw)
<class '_io.FileIO'>

It appears that you can turn it off with:

sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
or:
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())

This results in a sys.stderr which appears to work and whose
.line_buffering property is False.

[Random832]

Nobody <nob...@nowhere.invalid> writes:

> It's probably related to the fact that std{in,out,err} are Unicode
> streams.

There's no fundamental reason a Unicode stream should have to be line
buffered. If it's "related", it's only in that an oversight was made in
the course of making that change.

[Chris Angelico]

On Wed, Nov 4, 2015 at 2:00 AM, Random832 <rand...@fastmail.com> wrote:
> Nobody <nob...@nowhere.invalid> writes:
>
>> It's probably related to the fact that std{in,out,err} are Unicode
>> streams.
>

> There's no fundamental reason a Unicode stream should have to be line
> buffered. If it's "related", it's only in that an oversight was made in
> the course of making that change.

Yep. Unicode *input* streams need to be buffered, but *output* can
always be insta-flushed. The only significance of Unicode to output is
that a single character may cause multiple bytes to be output; and
since output can block for even a single byte, it should be no
different.

+1 for making sys.stderr unbuffered.

ChrisA

[George Trojan]

On 11/03/2015 05:00 PM, python-li...@python.org wrote:
> On Mon, 02 Nov 2015 18:52:55 +1100, Steven D'Aprano wrote:
>
>> In Python 2, stderr is unbuffered.
>>
>> In most other environments (the shell, C...) stderr is unbuffered.
>>
>> It is usually considered a bad, bad thing for stderr to be buffered. What
>> happens if your application is killed before the buffer fills up? The
>> errors in the buffer will be lost.
>>
>> So how come Python 3 has line buffered stderr? And more importantly, how
>> can I turn buffering off?

> It's probably related to the fact that std{in,out,err} are Unicode
> streams.
>

> > type(sys.stderr)
> <class '_io.TextIOWrapper'>
> > type(sys.stderr.buffer)
> <class '_io.BufferedWriter'>
> > type(sys.stderr.buffer.raw)
> <class '_io.FileIO'>
>
> It appears that you can turn it off with:
>
> sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
> or:
> sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())
>
> This results in a sys.stderr which appears to work and whose
> .line_buffering property is False.
>
>

This does set line buffering, but does not change the behaviour:

(devenv-3.4.1) dilbert@gtrojan> cat x.py
import sys
import time
if sys.version>'3':
import io
sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach())
#sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
print(sys.stderr.line_buffering)
sys.stderr.write('a')
time.sleep(10)

This is python2.7.5. a is printed before ^C.

(devenv-3.4.1) dilbert@gtrojan> /bin/python x.py
a^CTraceback (most recent call last):

Here buffer is flushed on close, after typing ^C.

(devenv-3.4.1) dilbert@gtrojan> python x.py
False
^CaTraceback (most recent call last):

George

[George Trojan]

Here buffer is flushed on close, after typing ^C.//

(devenv-3.4.1) dilbert@gtrojan> python x.py
False
^CaTraceback (most recent call last):

George

Found it. write_through must be set to True.

(devenv-3.4.1) dilbert@gtrojan> cat x.py
import sys
import time
if sys.version>'3':
import io

sys.stderr = io.TextIOWrapper(sys.stderr.detach().detach(),
write_through=True)

#sys.stderr = io.TextIOWrapper(sys.stderr.buffer.raw)
print(sys.stderr.line_buffering)
sys.stderr.write('a')
time.sleep(10)

(devenv-3.4.1) dilbert@gtrojan> python x.py
False

a^CTraceback (most recent call last):

/

/

[Random832]

George Trojan <george...@noaa.gov> writes:
> This does set line buffering, but does not change the behaviour:

The opposite of line buffering is not no buffering, but full
(i.e. block) buffering, that doesn't get flushed until it runs
out of space. TextIOWrapper has its own internal buffer, and its
design apparently doesn't contemplate the possibility of using
it with a raw FileIO object (which may mean that the posted code
isn't guaranteed to work) or disabling buffering.

[Terry Reedy]

On 11/3/2015 10:42 AM, Chris Angelico wrote:
> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <rand...@fastmail.com> wrote:
>> Nobody <nob...@nowhere.invalid> writes:
>>

>>> It's probably related to the fact that std{in,out,err} are Unicode
>>> streams.
>>

>> There's no fundamental reason a Unicode stream should have to be line
>> buffered. If it's "related", it's only in that an oversight was made in
>> the course of making that change.

The current behavior is not an 'oversight'. I was considered, decided,
and revisited in https://bugs.python.org/issue13601. Guido:
"Line-buffering should be good enough since in practice errors messages
are always terminated by a newline." If not, print(part_line,
file=sys.stderr, flush=True) works for the unusual case.

> Yep. Unicode *input* streams need to be buffered, but *output* can
> always be insta-flushed. The only significance of Unicode to output is
> that a single character may cause multiple bytes to be output; and
> since output can block for even a single byte, it should be no
> different.
>
> +1 for making sys.stderr unbuffered.

-
Terry Jan Reedy

[Steven D'Aprano]

On Wednesday 04 November 2015 09:25, Terry Reedy wrote:

> On 11/3/2015 10:42 AM, Chris Angelico wrote:
>> On Wed, Nov 4, 2015 at 2:00 AM, Random832 <rand...@fastmail.com> wrote:
>>> Nobody <nob...@nowhere.invalid> writes:
>>>
>>>> It's probably related to the fact that std{in,out,err} are Unicode
>>>> streams.
>>>
>>> There's no fundamental reason a Unicode stream should have to be line
>>> buffered. If it's "related", it's only in that an oversight was made in
>>> the course of making that change.
>
> The current behavior is not an 'oversight'. I was considered, decided,
> and revisited in https://bugs.python.org/issue13601. Guido:
> "Line-buffering should be good enough since in practice errors messages
> are always terminated by a newline." If not, print(part_line,
> file=sys.stderr, flush=True) works for the unusual case.

This is one of the offending line from our code base:

print('<4>Suspicious answer "{}"!'.format(answer), file=sys.stderr)

So that ought to be terminated by a newline. And yet, the stderr output
doesn't show up until the program exits.




--
Steve

[Wolfgang Maier]

For me, that prints immediately. Where is your output going? Console,
file, ...?

[Steven D'Aprano]

Going to stderr, like the whole thread is about :-)


The output does eventually show up written to the console, but only after
Python exits.



--
Steven

[Wolfgang Maier]

I see :), but seriously, what I meant was: is it going to the console
directly or is your code base redirecting sys.stderr to another buffered
object first?

[Wolfgang Maier]

On 04.11.2015 11:43, Wolfgang Maier wrote:

> I see :), but seriously, what I meant was: is it going to the console
> directly or is your code base redirecting sys.stderr to another buffered
> object first?
>

Standard I/O streams are line-buffered only if interactive (i.e.,
connected to a console), but block-buffered otherwise.
What does sys.stderr.isatty() return?

[Random832]

Wolfgang Maier <wolfgan...@biologie.uni-freiburg.de> writes:
> Standard I/O streams are line-buffered only if interactive (i.e.,
> connected to a console), but block-buffered otherwise.

That's not appropriate for stderr, nor is it justified by the argument
that Terry Reedy cited earlier. I had assumed he was making an implicit
claim that stderr is _always_ line-buffered, _even if_ it is not
interactive.

> What does sys.stderr.isatty() return?

There are many situations in which a nominally interactive process might
not have a "tty" for stderr on MS Windows. In an Emacs M-x shell buffer,
in a cygwin terminal (for a non-cygwin python), etc.

[Dave Farrance]

Random832 <rand...@fastmail.com> wrote:

>The opposite of line buffering is not no buffering, but full
>(i.e. block) buffering, that doesn't get flushed until it runs
>out of space. TextIOWrapper has its own internal buffer, and its
>design apparently doesn't contemplate the possibility of using
>it with a raw FileIO object (which may mean that the posted code
>isn't guaranteed to work) or disabling buffering.

Hmmm. That even seems to cause trouble for sys.stderr.write
(in Python3 with a non-tty e.g. a piped output):

$ python2 -c 'import sys;sys.stderr.write("1\n");print("2")' 2>&1 | tee
1
2
$ python3 -c 'import sys;sys.stderr.write("1\n");print("2")' 2>&1 | tee
2
1

[Terry Reedy]

Or include 'flush=True' if you really want that to be a partial line.

--
Terry Jan Reedy

[srinivas devaki]

On Mon, Nov 2, 2015 at 1:22 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:
>
> So how come Python 3 has line buffered stderr? And more importantly, how can
> I turn buffering off?
>
> I don't want to use the -u unbuffered command line switch, because that
> effects stdout as well. I'm happy for stdout to remain buffered.
>

you can simply turn buffering off for stderr by redefining the print
function or declaring a new print function like


from functools import partial
print = partial(print, flush=True)
# or
from functools import partial
import sys
printerr = partial(print, flush=True, file=sys.stderr)