Problems with "Tarfile.close()"

Dr Rainer Woitok

unread,

Dec 20, 2019, 10:24:50 AM12/20/19

to

Greetings,

One of my Python scripts basically does the following:

source = tarfile.open(name=tar_archive , mode='r|*')
dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt)

.
.
.

source.close()
dest.close()

In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I
ran into the problem that under Python 3.6 the call to "dest.close()"
fails:

Traceback (most recent call last):
File ".../tar_archive.copy", line 137, in <module>
dest.close()
File "/usr/lib64/python3.6/tarfile.py", line 1742, in close
self.fileobj.close()
File "/usr/lib64/python3.6/tarfile.py", line 467, in close
self.fileobj.write(self.buf)
TypeError: write() argument must be str, not bytes

What am I doing wrong? By the way: since on some hosts this script is
running on the transition from Python 2.7 to Python 3.x will not happen
immediately, I need a solution which works with both versions.

Sincerely,
Rainer

Chris Angelico

unread,

Dec 20, 2019, 10:34:54 AM12/20/19

to

On Sat, Dec 21, 2019 at 2:29 AM Dr Rainer Woitok
<rainer...@gmail.com> wrote:
>
> Greetings,
>
> One of my Python scripts basically does the following:
>
> source = tarfile.open(name=tar_archive , mode='r|*')
> dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt)
>
> .
> .
> .
>
> source.close()
> dest.close()
>
> In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I
> ran into the problem that under Python 3.6 the call to "dest.close()"
> fails:

(I think the fact that it fails when you close the file is a red
herring; it would fail at some point, and it happens to hold things
over until it closes.)

> Traceback (most recent call last):
> File ".../tar_archive.copy", line 137, in <module>
> dest.close()
> File "/usr/lib64/python3.6/tarfile.py", line 1742, in close
> self.fileobj.close()
> File "/usr/lib64/python3.6/tarfile.py", line 467, in close
> self.fileobj.write(self.buf)
> TypeError: write() argument must be str, not bytes
>
> What am I doing wrong? By the way: since on some hosts this script is
> running on the transition from Python 2.7 to Python 3.x will not happen
> immediately, I need a solution which works with both versions.
>

Possibly the easiest fix would be to open all files in binary mode. I
think your content is binary anyway by the look of it. Just add the
letter "b" to the end of your open file modes, and on Py2, it'll
ensure that no newline conversion happens (mainly an issue on Windows,
which is why you probably don't feel the need to do this), but on Py3,
it means that it expects bytestrings everywhere.

If that *doesn't* work, then you may need to mark some of your strings
as binary, if you have unadorned strings containing ASCII data.

ChrisA

Ethan Furman

unread,

Dec 20, 2019, 10:42:12 AM12/20/19

to

On 12/20/2019 04:19 AM, Dr Rainer Woitok wrote:

> One of my Python scripts basically does the following:
>
> source = tarfile.open(name=tar_archive , mode='r|*')
> dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt)
>
> .
> .
> .
>
> source.close()
> dest.close()
>
> In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I
> ran into the problem that under Python 3.6 the call to "dest.close()"
> fails:
>

> Traceback (most recent call last):
> File ".../tar_archive.copy", line 137, in <module>
> dest.close()
> File "/usr/lib64/python3.6/tarfile.py", line 1742, in close
> self.fileobj.close()
> File "/usr/lib64/python3.6/tarfile.py", line 467, in close
> self.fileobj.write(self.buf)
> TypeError: write() argument must be str, not bytes

In Python 3 `sys.stdout` is a character interface, not bytes.

There are a couple solutions to the Python 3 aspect of the problem here:

https://stackoverflow.com/q/908331/208880

If those answers do not work on Python 2 you'll need to detect which version you are on and act appropriately, perhaps hiding that bit of complexity in a function or class.

--
~Ethan~

Dr Rainer Woitok

unread,

Dec 20, 2019, 12:20:32 PM12/20/19

to

Ethan,

On Friday, 2019-12-20 07:41:51 -0800, you wrote:

> ...

> In Python 3 `sys.stdout` is a character interface, not bytes.

Does that mean that with Python 3 "Tarfile" is no longer able to write
the "tar" file to a pipe? Or is there now another way to write to a
pipe? And if that new way also worked with Python 2, it would be even
better ... :-)

> There are a couple solutions to the Python 3 aspect of the problem here:
>
> https://stackoverflow.com/q/908331/208880

Using "sys.stdout.buffer" seems to work in Python 3 (at least with my
current rather trivial test case) but does not work in Python 2. Quest-
ion: what is the cheapest way to retrieve the Python version the script
is executing in?

Sincerely,
Rainer

Peter Otten

unread,

Dec 20, 2019, 2:12:35 PM12/20/19

to

While I didn't look into the stackoverflow page an easy way to get something
that accepts bytes may be

# untested
stdout = sys.stdout
try:
stdout = stdout.buffer
except AttributeError:
pass

tarfile.open(fileobj=stdout, ...)

Chris Angelico

unread,

Dec 20, 2019, 2:45:51 PM12/20/19

to

This construct can be simplified down to:

stdout = getattr(sys.stdout, "buffer", sys.stdout)

ChrisA

Dr Rainer Woitok

unread,

Dec 21, 2019, 3:16:13 AM12/21/19

to

Chris,

On Saturday, 2019-12-21 06:45:24 +1100, you wrote:

> ...

> This construct can be simplified down to:
>
> stdout = getattr(sys.stdout, "buffer", sys.stdout)

Nice! And with my admittedly trivial test case it correctly works under
both, Python 2 and Python 3.

Now I'll do some more serious testing ...

Sincerely,
Rainer

Barry

unread,

Dec 21, 2019, 4:59:07 AM12/21/19

to

> On 20 Dec 2019, at 15:31, Dr Rainer Woitok <rainer...@gmail.com> wrote:
>
> Greetings,
>

> One of my Python scripts basically does the following:
>
> source = tarfile.open(name=tar_archive , mode='r|*')
> dest = tarfile.open(fileobj=sys.stdout, mode='w|', format=fmt)
>
> .
> .
> .
>
> source.close()
> dest.close()
>
> In an attempt to move my Python scripts from Python 2.7 to Python 3.6 I
> ran into the problem that under Python 3.6 the call to "dest.close()"
> fails:
>
> Traceback (most recent call last):
> File ".../tar_archive.copy", line 137, in <module>
> dest.close()
> File "/usr/lib64/python3.6/tarfile.py", line 1742, in close
> self.fileobj.close()
> File "/usr/lib64/python3.6/tarfile.py", line 467, in close
> self.fileobj.write(self.buf)
> TypeError: write() argument must be str, not bytes
>

> What am I doing wrong? By the way: since on some hosts this script is
> running on the transition from Python 2.7 to Python 3.x will not happen
> immediately, I need a solution which works with both versions.

Stdout on python 3 is a text stream. Tarfiles need a binary stream.

I searched for “python 3 binary stdout” and found this
stack over flow question that has suggested fixes.

https://stackoverflow.com/questions/908331/how-to-write-binary-data-to-stdout-in-python-3#908440

Barry

>
> Sincerely,
> Rainer
> --
> https://mail.python.org/mailman/listinfo/python-list
>