Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

are final newlines appended if output lacks it?

0 views
Skip to first unread message

Dan Jacobson

unread,
May 3, 2003, 8:53:39 PM5/3/03
to
Doing help() print explains how the blank got in here,
$ python -c "print 'wow',;print 'pow',"|od -c
0000000 w o w p o w \n
0000010
But not the newline. Why?
--
http://jidanni.org/ Taiwan(04)25854780

John Machin

unread,
May 3, 2003, 10:01:48 PM5/3/03
to

Doing help() print also explains how the newline got in there:
"""
A "\n" character is written at the end, unless the print statement
ends
with a comma. This is the only action if the statement contains just
the
keyword print.
"""

John Machin

unread,
May 3, 2003, 10:21:23 PM5/3/03
to
On Sun, 04 May 2003 02:01:48 GMT, sjma...@lexicon.net (John Machin)
wrote:


derrrrr ..... my brain was in neutral. Dan's question/problem is quite
valid, not explained by the documentation, and I have been able to
reproduce it on Windows 2000 with Python 2.2.2.

Doing this give the same effect as what Dan got, except that of course
with Windows the offending extra is '\r\n' instead of just '\n':

python -c "print 'wow',;print 'pow'," >powwow.txt

Doing this:

python -c "import sys;sys.stdout.write('abc');sys.stdout.write('xyz')"
>abcxyz.txt

produces no newline. Looks like there's an undocumented "feature" of
the Python's print statement, nothing to do with the C stdio library
or the OS.

Tim Peters

unread,
May 3, 2003, 11:59:14 PM5/3/03
to
[Dan Jacobson]

> Doing help() print explains how the blank got in here,
>
> $ python -c "print 'wow',;print 'pow',"|od -c
> 0000000 w o w p o w \n
> 0000010
>
> But not the newline. Why?

[John Machin]


> derrrrr ..... my brain was in neutral. Dan's question/problem is quite
> valid, not explained by the documentation, and I have been able to
> reproduce it on Windows 2000 with Python 2.2.2.
>
> Doing this give the same effect as what Dan got, except that of course
> with Windows the offending extra is '\r\n' instead of just '\n':
>
> python -c "print 'wow',;print 'pow'," >powwow.txt
>
> Doing this:
>
> python -c "import sys;sys.stdout.write('abc');sys.stdout.write('xyz')"
> > abcxyz.txt
>
> produces no newline. Looks like there's an undocumented "feature" of
> the Python's print statement, nothing to do with the C stdio library
> or the OS.

OTOH, you'll find that

python -c "print 'abc\n',"

produces 4 (on Linux) or 5 (on Windows) characters. It has to do with
whether sys.stdout.softspace is set when the program is done.
PyRun_SimpleFileExFlags() ends with

if (Py_FlushLine())
PyErr_Clear();
return 0;

and Py_FlushLine() is

int
Py_FlushLine(void)
{
PyObject *f = PySys_GetObject("stdout");
if (f == NULL)
return 0;
if (!PyFile_SoftSpace(f, 0))
return 0;
return PyFile_WriteString("\n", f);
}

So it's specifically sys.stdout (and only sys.stdout) that may get a newline
tacked on, and the relation to print is that normally only print fiddles
with a file's softspace attribute.

Code would break if we changed this, so learn to love it <wink>. I expect
that in Python 3, there won't be any softspace gimmick.


Bengt Richter

unread,
May 4, 2003, 1:02:05 AM5/4/03
to

I wonder if this is related to the softspace bug that I posted about some time ago

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=ah2887%24bpu%240%40216.39.172.122

Regards,
Bengt Richter

Alex Martelli

unread,
May 4, 2003, 2:05:38 AM5/4/03
to
John Machin wrote:
...

> python -c "print 'wow',;print 'pow'," >powwow.txt
>
> Doing this:
>
> python -c "import sys;sys.stdout.write('abc');sys.stdout.write('xyz')"
>>abcxyz.txt
>
> produces no newline. Looks like there's an undocumented "feature" of
> the Python's print statement, nothing to do with the C stdio library
> or the OS.

As Tim mentioned, it's all about softspace. Check:

[alex@lancelot src]$ python -c "print 'wow',;print 'pow'," | wc
1 2 8

8 characters -- 'wow pow\n' -- in one line;

[alex@lancelot src]$ python -c "import
sys;sys.stdout.write('abc');sys.stdout.write('xyz')" | wc
0 1 6

6 characters -- 'abcxyz' -- in zero lines (no \n);

[alex@lancelot src]$ python -c "import
sys;sys.stdout.write('abc');sys.stdout.write('xyz');sys.stdout.softspace=1"
| wc
1 1 7

7 characters -- 'abcxyz\n' -- in one line -- the extra trailing \n thanks
to the softspace setting.


Alex

Bengt Richter

unread,
May 4, 2003, 2:35:49 AM5/4/03
to

So is the goal is to guarantee a final EOL if the last output to sys.stdout
was from a print? (Apparently sys.stdout.write() resets sys.stdout.softspace
unconditionally, so it's allowed to have the last word with no tack-ons,
but a final print with a trailing comma will get its softspace turned to an EOL?

I guess this is just policy, and not related to the apparent (IMO) bug that
confuses interactive echo with program output to stdout, so that interactive
echo of a typed line end erroneously (IMO) resets sys.stdout.softspace. Note:

>>> import sys
>>> class SO:
... def write(self,s): sys.stderr.write(`s`+'\n')
...
>>> sys.stdout = SO()
>>> print sys.stdout.softspace,; print sys.stdout.softspace,
'0'
' '
'1'
'\n'
>>> print sys.stdout.softspace,; print sys.stdout.softspace,
'0'
' '
'1'
'\n'
>>> def foo():
... print sys.stdout.softspace,; print sys.stdout.softspace,
... print sys.stdout.softspace,; print sys.stdout.softspace,
...
>>> foo()
'0'
' '
'1'
' '
'1'
' '
'1'
'\n'

Note that the interactive call to foo() only echoed one EOL, so the
softspace from the first print line survived to influence the second line
inside foo, unlike for the separate invocations preceding.

If the interactive _echo_ is actually going to sys.__stdout__ when normal
sys.stdout is redirected, perhaps sys.__stdout__.softspace should be reset
for the interactive EOL echo instead of sys.stdout.softspace?

Regards,
Bengt Richter

Tim Peters

unread,
May 4, 2003, 1:09:29 PM5/4/03
to
[Bengt Richter]

> So is the goal is to guarantee a final EOL if the last output to
> sys.stdout was from a print?

I don't know, but that appears to be the intent of the code.

> (Apparently sys.stdout.write() resets sys.stdout.softspace
> unconditionally,

Yes.

> so it's allowed to have the last word with no tack-ons,
> but a final print with a trailing comma will get its softspace
> turned to an EOL?

Looks that way, yes.

> I guess this is just policy, and not related to the apparent
> (IMO) bug that confuses interactive echo with program output
> to stdout, so that interactive echo of a typed line end
> erroneously (IMO) resets sys.stdout.softspace.

Heh -- I couldn't parse that. If you think you have a bug, and you care
about it, please open a bug report on SourceForge.

> Note:
>
> >>> import sys
> >>> class SO:
> ... def write(self,s): sys.stderr.write(`s`+'\n')
> ...
> >>> sys.stdout = SO()
> >>> print sys.stdout.softspace,; print sys.stdout.softspace,
> '0'
> ' '
> '1'
> '\n'

Python calls Py_FlushLine() (I showed the source before) after interpreting
each block it reads from the shell. This was one block of input, so
Py_FlushLine was called once after the entire block completed, and because
the block left sys.stdout.softspace true, Py_FlushLine() wrote a newline to
sys.stdout and cleared sys.stdout.softspace.

> >>> print sys.stdout.softspace,; print sys.stdout.softspace
> '0'
> ' '
> '1'
> '\n'

Ditto.

> >>> def foo():
> ... print sys.stdout.softspace,; print sys.stdout.softspace,
> ... print sys.stdout.softspace,; print sys.stdout.softspace,
> ...

This block didn't set sys.stdout.softspace, so Py_FlushLine didn't do
anything when the interpretation of the block completed.

> >>> foo()
> '0'
> ' '
> '1'
> ' '
> '1'
> ' '
> '1'
> '\n'

This block left sys.stdout.softspace set when the block (the call to foo())
completed, so Py_FlushLine did write a newline to sys.stdount (and cleared
sys.stdout.softspace).

> Note that the interactive call to foo() only echoed one EOL, so the
> softspace from the first print line survived to influence the second line
> inside foo, unlike for the separate invocations preceding.

Yes, there's one call to Py_FlushLine per interactive block, after the block
completes executing. The number of Python statements executed per
interactive block makes no difference.

> If the interactive _echo_ is actually going to sys.__stdout__

I'm not sure what "interactive echo" means, but under the only plausible
meanings I can think of, it would be out of Python's hands, controlled by
however your platform C's (and possible GNU readline, if you're using that)
reading from stdin interacts with the console device. Python isn't trying
to echo, or not to echo, anything you type. Python prints interactive
*prompts* ('>>> ' and '... ') to C's idea of what stderr means.

> when normal sys.stdout is redirected, perhaps sys.__stdout__.softspace
> should be reset for the interactive EOL echo instead of
> sys.stdout.softspace?

There is no "interactive EOL echo" in a sense I can grok. Python is
apparently trying to make sure that the output produced by print statements,
to sys.stdout, produced by each interactive block, ends with a newline, and
that's all. Whether the EOLs (or anything else) you type get echoed is a
function of your console device, and don't affect sys.stdout.softspace
either way.


Bengt Richter

unread,
May 4, 2003, 6:21:04 PM5/4/03
to
On Sun, 4 May 2003 13:09:29 -0400, "Tim Peters" <tim...@email.msn.com> wrote:

>[Bengt Richter]
>> So is the goal is to guarantee a final EOL if the last output to
>> sys.stdout was from a print?
>
>I don't know, but that appears to be the intent of the code.
>
>> (Apparently sys.stdout.write() resets sys.stdout.softspace
>> unconditionally,
>
>Yes.
>
>> so it's allowed to have the last word with no tack-ons,
>> but a final print with a trailing comma will get its softspace
>> turned to an EOL?
>
>Looks that way, yes.
>
>> I guess this is just policy, and not related to the apparent
>> (IMO) bug that confuses interactive echo with program output
>> to stdout, so that interactive echo of a typed line end
>> erroneously (IMO) resets sys.stdout.softspace.
>
>Heh -- I couldn't parse that. If you think you have a bug, and you care
>about it, please open a bug report on SourceForge.
>

Well, I think there is a bug, but my original diagnosis was wrong.
What I mean is that during an interactive session there are effectively
several output streams that intertwingle unless there is redirection.
One stream is the echoing of the source you type into the console,
(handled by the console device, as you point out), and others are programmed
outputs to the console via stdout or stderr or sneakily.

Print goes to stdout unless redirected, so normally it is desirable to
flush the output from each chunk of interactive source processed, to
keep things nicely sequenced on the screen. Ok so far.

It is also desirable to have a new line at the end, so the prompt
can be on a fresh line, yet avoiding extra newlines. This probably
led to the temptation of processing softspace in flush, since it
seems similar to what successive prints have to do.

But why should flushing do anything with softspace? Flushing is not logically
for creating new output, it is just making sure all buffered output has gone out.
Softspace is not logically part of that data IMO. To generate a '\n' conditionally
for interactive screen formatting purposes is logically a different thing from
concatenating print statement outputs. Using flush to do the combined function
is mistaken IMO, and it shows up especially when stdout is not going to the screen.

I suspect the flush at the end of a block of input should only flush the buffers,
and not generate any output to stdout, and not change softspace. Instead of using
up softspace to generate a conditional '\n' through stdout, it should output a
conditional '\n' to the prompt stream (stderr), where the condition should be
sys.stdout is sys.__stdout__ and sys.stdout.softspace. IOW, if stdout is not
going to the console, the last interactive input will have echoed a newline, so
there would be no need, and if stdout _is_ going to the console, then a newline
can go to stderr if softspace is set -- without resetting it, however.

Not resetting softspace will mean a possible leading space on a line from
a subsequent print statement. That is good! It shows what was actually pending
in the output sequence.

>> Note:
>>
>> >>> import sys
>> >>> class SO:
>> ... def write(self,s): sys.stderr.write(`s`+'\n')
>> ...
>> >>> sys.stdout = SO()
>> >>> print sys.stdout.softspace,; print sys.stdout.softspace,
>> '0'
>> ' '
>> '1'
>> '\n'
>
>Python calls Py_FlushLine() (I showed the source before) after interpreting
>each block it reads from the shell. This was one block of input, so

As mentioned, ISTM Py_Flushline should not do anything with softspace, nor
create a new output character.

>Py_FlushLine was called once after the entire block completed, and because

One could argue that if stdout is redirected, there is no need to flush it,
unless you think there might be side effects from the output that should be
observed chunk by chunk. This seems like a debugging aid, rather than how
the program would act being run non-interactively. So maybe non-console
stdout flushing should be a command line option?

>the block left sys.stdout.softspace true, Py_FlushLine() wrote a newline to
>sys.stdout and cleared sys.stdout.softspace.

>
>> >>> print sys.stdout.softspace,; print sys.stdout.softspace
>> '0'
>> ' '
>> '1'
>> '\n'
>
>Ditto.
>
>> >>> def foo():
>> ... print sys.stdout.softspace,; print sys.stdout.softspace,
>> ... print sys.stdout.softspace,; print sys.stdout.softspace,
>> ...
>
>This block didn't set sys.stdout.softspace, so Py_FlushLine didn't do
>anything when the interpretation of the block completed.
>
>> >>> foo()
>> '0'
>> ' '
>> '1'
>> ' '
>> '1'
>> ' '
>> '1'
>> '\n'
>
>This block left sys.stdout.softspace set when the block (the call to foo())
>completed, so Py_FlushLine did write a newline to sys.stdount (and cleared
>sys.stdout.softspace).

IMO if you want to flush stdout, softspace should be left pending, and
only the characters _up to_ the as-yet-undetermined space-or-EOL should be
flushed.

>
>> Note that the interactive call to foo() only echoed one EOL, so the
>> softspace from the first print line survived to influence the second line
>> inside foo, unlike for the separate invocations preceding.
>
>Yes, there's one call to Py_FlushLine per interactive block, after the block
>completes executing. The number of Python statements executed per
>interactive block makes no difference.

Ok & ditto.


>
>> If the interactive _echo_ is actually going to sys.__stdout__
>
>I'm not sure what "interactive echo" means, but under the only plausible
>meanings I can think of, it would be out of Python's hands, controlled by
>however your platform C's (and possible GNU readline, if you're using that)
>reading from stdin interacts with the console device. Python isn't trying
>to echo, or not to echo, anything you type. Python prints interactive
>*prompts* ('>>> ' and '... ') to C's idea of what stderr means.
>
>> when normal sys.stdout is redirected, perhaps sys.__stdout__.softspace
>> should be reset for the interactive EOL echo instead of
>> sys.stdout.softspace?
>
>There is no "interactive EOL echo" in a sense I can grok. Python is
>apparently trying to make sure that the output produced by print statements,
>to sys.stdout, produced by each interactive block, ends with a newline, and

ISTM the only reason to force an ending newline is for interactive screen purposes.
That's different from the softspace issue. Faking the current flush softspace processing
with a conditional newline to stderr after flushing stdout _up to_ the softspace would be
ok, for better screen interaction, but only ok if softspace processing were left pending, IMO,
so that a redirected stdout output stream is independent of whether you typed
your print statements separately or e.g. in a block under an "if 1:".

>that's all. Whether the EOLs (or anything else) you type get echoed is a
>function of your console device, and don't affect sys.stdout.softspace
>either way.

Right. I sort of jumped to the right (IMO) conclusion for the wrong reason ;-/
But from the above, do you now see the bug I see?

I don't even think converting softspace to a final EOL would be desirable at
the end of a program (on close) executed as an ordinary script. Would it
break code to change the flush logic? Maybe such code should break ;-)

Isn't softspace logically just about concatenating successive print statement
outputs (with convenience resetting by stdout.write)? And shouldn't the
result be independent of whether (and how chunked) those statements were
executed interactively, if stdout is redirected?

Regards,
Bengt Richter

Tim Peters

unread,
May 4, 2003, 8:46:44 PM5/4/03
to
[Bengt Richter, arguing about softspace]

I don't agree there's a bug here, because the code seems clearly to be
functioning as designed. You can argue that the design is wrong, and then
you're in PEP territory, arguing for a different-- and
incompatible --design. So write a PEP if you feel strongly about this. I
can't make more time for this topic.

> ...


> I suspect the flush at the end of a block of input should only
> flush the buffers,

This business about flushing buffers is repeated several times. Here's the
source of Py_FlushLine() again:

int
Py_FlushLine(void)
{
PyObject *f = PySys_GetObject("stdout");
if (f == NULL)
return 0;
if (!PyFile_SoftSpace(f, 0))
return 0;
return PyFile_WriteString("\n", f);
}

Despite its name, it doesn't flush any buffers, the *only* things it does
are (a) write a newline if sys.stdout.softspace is true; and, (b) clear
sys.stdout.softspace. If sys.stdout happens to be a line-buffered file,
then writing a newline should flush it, of course.

> ...


> Not resetting softspace will mean a possible leading space on a line
> from a subsequent print statement. That is good! It shows what was
> actually pending in the output sequence.

Regardless, it would be incompatible behavior, so won't be changed without
an important use case. I should note that I haven't seen any use cases in
your posts about this, just arguments that "it's wrong". So note that if
you write a PEP, it will need compelling use cases to justify changing
behavior.

> ...


> Would it break code to change the flush logic?

Of course.

> Maybe such code should break ;-)

Not unless there are compelling advantages to be gained by breaking it.

> Isn't softspace logically just about concatenating successive
> print statement outputs (with convenience resetting by stdout.write)?
> And shouldn't the result be independent of whether (and how chunked)
> those statements were executed interactively, if stdout is redirected?

I don't know -- I didn't design it and can't recall ever caring about it.
If I did care and disliked the current behavior, I'd stay away from "print",
which-- as noted before --is normally the only thing that ever sets
softspace. If you want total WYSIWYG behavior, stick to sys.stdout.write().

It's certainly the case that GUIs like IDLE rely on the current behavior to
keep their (redirected!) version of sys.stdout looking nice; e.g., look at
code.py's InteractiveInterpreter.runcode(), and you'll see that it emulates
the same kind of newline-appending logic, via code.py's more-aptly (than
Py_FlushLine) named softspace() function.


Bengt Richter

unread,
May 5, 2003, 3:13:46 AM5/5/03
to
On Sun, 04 May 2003 20:46:44 -0400, Tim Peters <tim...@comcast.net> wrote:

>[Bengt Richter, arguing about softspace]
>
>I don't agree there's a bug here, because the code seems clearly to be
>functioning as designed. You can argue that the design is wrong, and then

Ok, if the following is as designed, then I'll argue it's wrong ;-)

[22:20] C:\pywk\clp>python >z
Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)
Type "help", "copyright", "credits" or "license" for more inf
>>> print 123
>>> print 123
>>> print 123,
>>> print 123,
>>> if 1:
... print 456,
... print 456,
... print 456,
...
>>> print 'xxx'
>>> ^Z


[22:35] C:\pywk\clp>type z
123
123
123
123
456 456 456
xxx

[22:35] C:\pywk\clp>cat |python >z2
print 123
print 123
print 123,
print 123,
if 1:
print 456,
print 456,
print 456,

print 'xxx'
^Z

[22:37] C:\pywk\clp>type z2
123
123
123 123 456 456 456 xxx

IMO z and z2 should have been equivalent outputs.
The screen echo for the input is the same either way,
and ISTM z2 is the output you'd expect from reading
either of the (identically echoed) inputs logs.

Making nice on the screen when the output is interlaced
should IMO not sacrifice the integrity of the programmed
output. ISTM it's just a matter of knowing when a prompt
needs to be preceded by a newline (but as part of the prompt,
not part of the stdout data, which per se should not be
interfered with).

>you're in PEP territory, arguing for a different-- and
>incompatible --design. So write a PEP if you feel strongly about this. I
>can't make more time for this topic.
>

Sorry. I wouldn't normally have gotten into this either, except my gut feel
is that z and z2 being different above is wrong. (Plus my original take
on the problem was wrong ;-)

[ ... Py_FlushLine code ...]


>Despite its name, it doesn't flush any buffers, the *only* things it does
>are (a) write a newline if sys.stdout.softspace is true; and, (b) clear
>sys.stdout.softspace. If sys.stdout happens to be a line-buffered file,
>then writing a newline should flush it, of course.
>
>> ...
>> Not resetting softspace will mean a possible leading space on a line
>> from a subsequent print statement. That is good! It shows what was
>> actually pending in the output sequence.
>
>Regardless, it would be incompatible behavior, so won't be changed without
>an important use case. I should note that I haven't seen any use cases in
>your posts about this, just arguments that "it's wrong". So note that if
>you write a PEP, it will need compelling use cases to justify changing
>behavior.

Nothing compelling, just the example above. It's only compelling in rare
aesthetically charged moods ;-) It's not like I can't work around it ;-)
[...]


>> Isn't softspace logically just about concatenating successive
>> print statement outputs (with convenience resetting by stdout.write)?
>> And shouldn't the result be independent of whether (and how chunked)
>> those statements were executed interactively, if stdout is redirected?
>
>I don't know -- I didn't design it and can't recall ever caring about it.
>If I did care and disliked the current behavior, I'd stay away from "print",
>which-- as noted before --is normally the only thing that ever sets
>softspace. If you want total WYSIWYG behavior, stick to sys.stdout.write().
>

I think the print softspace logic is ok, it's interfering with it in the stdout
output stream that's not ;-) Look what happens if we use stderr and its softspace:

>>> import sys
>>> print >> sys.stderr, 123,
123>>> print >> sys.stderr, 123,
123>>> print >> sys.stderr, 123,
123>>> print >> sys.stderr, 123,
123>>> print >> sys.stderr, 123
123
>>> print >> sys.stderr, 123
123
>>> print >> sys.stderr, 123
123

The stderr output stream is intact, because Py_Flushline is hardcoded to stdout,
but for that reason there's also no prettifying extra newlines to keep the prompts
at the left (which could be done without clobbering the softspace logic).


>It's certainly the case that GUIs like IDLE rely on the current behavior to
>keep their (redirected!) version of sys.stdout looking nice; e.g., look at

Maybe someone will want to make stderr intertwingle nicely too, and the change
can be worked in.

>code.py's InteractiveInterpreter.runcode(), and you'll see that it emulates
>the same kind of newline-appending logic, via code.py's more-aptly (than
>Py_FlushLine) named softspace() function.
>

I'm not arguing against inserting extra newlines for the console, just not
by clobbering softspace logic ;-)

(BTW, if both stdout and stderr are going to the console, I guess there
might be more complications in deciding whether a prompt needs a prefixed newline ;-)

Regards,
Bengt Richter

Peter Hansen

unread,
May 5, 2003, 10:13:28 AM5/5/03
to
Bengt Richter wrote:

>
> Tim Peters wrote:
> >Regardless, it would be incompatible behavior, so won't be changed without
> >an important use case.

> Nothing compelling, just the example above. It's only compelling in rare


> aesthetically charged moods ;-) It's not like I can't work around it ;-)

Well, looks like that closes the discussion then. ;-)

Tim Peters

unread,
May 6, 2003, 11:19:36 PM5/6/03
to
[Tim]

> Regardless, it would be incompatible behavior, so won't be
> changed without an important use case.

[Bengt Richter]


> Nothing compelling, just the example above. It's only compelling in rare
> aesthetically charged moods ;-) It's not like I can't work around it ;-)

[Peter Hansen]


> Well, looks like that closes the discussion then. ;-)

It was closed anyway <wink>. I think it did a fine job of illustrating why
no form of file.softspace is likely to exist in Python 3, though.

magic-only-works-so-long-as-you-don't-question-it-ly y'rs - tim


0 new messages