Question about propagating universal_newlines through subprocess.Popen to io.TextIOWrapper

[Bill Deegan]

Greetings,

I was surprised to see that if I set encoding in my call to
subprocess.Popen() as follows:

p = Popen(cmd,
stdin=stdin,
stdout=subprocess.PIPE,
stderr=stderr_value,
env=os.environ,
universal_newlines=False, #universal_newlines,
encoding='utf-8')

That universal_newlines value is discarded due to:

text_mode = encoding or errors or universal_newlines

...


if text_mode:
self.stdout = io.TextIOWrapper(self.stdout,
encoding=encoding, errors=errors)


There doesn't seem to be a way to set encoding without forcing
univeral_newlines.

This seems like a bug?

-Bill

[eryk sun]

On Mon, Jun 26, 2017 at 5:23 PM, Bill Deegan <bi...@baddogconsulting.com> wrote:
>
> That universal_newlines value is discarded due to:
>
> text_mode = encoding or errors or universal_newlines
>
> ...
>
> if text_mode:
> self.stdout = io.TextIOWrapper(self.stdout,
> encoding=encoding, errors=errors)
>
> There doesn't seem to be a way to set encoding without forcing
> univeral_newlines.
>
> This seems like a bug?

The behavior is documented:

If encoding or errors are specified, or universal_newlines is true,
the file objects stdin, stdout and stderr will be opened in text
mode using the encoding and errors specified in the call or the
defaults for io.TextIOWrapper.

For stdin, line ending characters '\n' in the input will be
converted to the default line separator os.linesep. For stdout and
stderr, all line endings in the output will be converted to '\n'.
For more information see the documentation of the io.TextIOWrapper
class when the newline argument to its constructor is None.

Prior to 3.6, the way to get text streams was to enable
universal_newlines. Maybe for 3.7 the default can change to None, with
the addition of the following code:

if universal_newlines is None or universal_newlines:
newline = None
else:
newline = ''

if text_mode:
self.stdin = io.TextIOWrapper(self.stdin, write_through=True,
line_buffering=(bufsize == 1),
encoding=encoding, errors=errors, newline=newline)

[Bill Deegan]

Ideally (for my use case) it would be something which propagated
universal_newlines to io.TextIOWrapper().. rather than discards it.
In my case I want the stdout to be encoded utf-8, but I do not want \r's
changed to \n's as my test system is capturing the output of a progress
indicator which uses \r to return to beginning of line and overwrite the
previous output.

-Bill

[eryk sun]

On Mon, Jun 26, 2017 at 8:59 PM, Bill Deegan <bi...@baddogconsulting.com> wrote:
>
> Ideally (for my use case) it would be something which propagated
> universal_newlines to io.TextIOWrapper().. rather than discards it.
> In my case I want the stdout to be encoded utf-8, but I do not want \r's
> changed to \n's as my test system is capturing the output of a progress
> indicator which uses \r to return to beginning of line and overwrite the
> previous output.

An enhancement issue can be created for 3.7, but this is new behavior
that certainly won't be backported to 3.6. For now you'll have to do
it yourself. Leave out the encoding argument, and manually wrap
p.stdout with an io.TextIOWrapper that sets the newline argument to an
empty string.

[Bill Deegan]

O.k. I'll file an enhancement. Seems like propagating the values would be
better functionality.

Thanks!
-Bill