[Python-Dev] NamedTemporaryFile and context managers

[Ethan Furman]

In issue14243 [1] there are two issues being tracked:

- the difference in opening shared files between posix and Windows
- the behavior of closing the underlying file in the middle of
NamedTemporaryFile's context management

I'd like to address and get feedback on the context management issue.

```python
from tempfile import NamedTemporaryFile

with NamedTemporaryFile() as fp:
fp.write(b'some data')
fp = open(fp.name())
data = fp.read()

assert data == 'some_data'
```

Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that
cannot open a temp file for reading while it is still open for writing). This would look like:

```python
from tempfile import NamedTemporaryFile

with NamedTemporaryFile() as fp:
fp.write(b'some data')
fp.close() # Windows workaround
fp.open()
data = fp.read()

assert data == 'some_data'
```

The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file
[2]. To handle this scenario, my proposal is two-fold:

1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close
2) add `.open()` to NamedTemporaryFile

A possible side effect of (1) is that temp files may accumulate if the interpreter crashes, but given the
file-management abilities in today's software that seems like a minor annoyance at most.

The backwards compatibility issue of (1) is that the file is no longer deleted after a manual `close()` -- but why one
would call close() and then stay inside the CM, outside of testing, I cannot fathom. [3]

So, opinions on modifying NamedTemporaryFile to not delete on close() if inside a CM, and add open() ?

--
~Ethan~


[1] https://bugs.python.org/issue14243
[2] plus, the `.open()` doesn't yet exist
[3] feel free to educate me :-)
_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/DZNEIRHZ7LVQNQRQ2JL5B4AOC3HH3B6O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Antoine Pitrou]

On Thu, 8 Apr 2021 13:31:26 -0700
Ethan Furman <et...@stoneleaf.us> wrote:
>
> ```python
> from tempfile import NamedTemporaryFile
>
> with NamedTemporaryFile() as fp:
> fp.write(b'some data')
> fp.close() # Windows workaround
> fp.open()
> data = fp.read()
>
> assert data == 'some_data'
> ```
>
> The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file
> [2]. To handle this scenario, my proposal is two-fold:
>
> 1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close
> 2) add `.open()` to NamedTemporaryFile

Instead, you could add a dedicated `.reopen()`?

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/HUGSC2FB6VW3VFKGDYKUYUKNJEWNPF5T/

[Ronald Oussoren via Python-Dev]

On 8 Apr 2021, at 22:31, Ethan Furman <et...@stoneleaf.us> wrote:

In issue14243 [1] there are two issues being tracked:

- the difference in opening shared files between posix and Windows
- the behavior of closing the underlying file in the middle of
 NamedTemporaryFile's context management

I'd like to address and get feedback on the context management issue.

```python
from tempfile import NamedTemporaryFile

with NamedTemporaryFile() as fp:
   fp.write(b'some data’)

      fp.flush(). # needed to ensure data is actually in the file ;-)

   fp = open(fp.name())
   data = fp.read()

assert data == 'some_data'
```

I generally use a slightly different pattern for this:

with NamedTemporaryFile() as fp:

   fp.write(b'some data’)

   fp.flush()

   fp.seek(0)

   data = fp.read()

That is, reuse the same “fp” and just reset the stream.  An advantage of this approach is that you don’t need a named temporary file for this (and could even use a spooled one).   That said, I at times use this pattern with a named temporary file with a quick self-test for the file contents before handing of the file name to an external proces.

Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that cannot open a temp file for reading while it is still open for writing).  This would look like:

```python
from tempfile import NamedTemporaryFile

with NamedTemporaryFile() as fp:
   fp.write(b'some data')
   fp.close()  # Windows workaround
   fp.open()
   data = fp.read()

assert data == 'some_data'
```

The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file [2].  To handle this scenario, my proposal is two-fold:

1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close
2) add `.open()` to NamedTemporaryFile

I’ve never had the need for such an API, but must say that I barely use Windows and hence have not run into the “cannot open file for reading what it is still open for writing” issue.

A possible side effect of (1) is that temp files may accumulate if the interpreter crashes, but given the file-management abilities in today's software that seems like a minor annoyance at most.

The backwards compatibility issue of (1) is that the file is no longer deleted after a manual `close()` -- but why one would call close() and then stay inside the CM, outside of testing, I cannot fathom.  [3]

So, opinions on modifying NamedTemporaryFile to not delete on close() if inside a CM, and add open() ?

--
~Ethan~

Ronald

—

Twitter / micro.blog: @ronaldoussoren

Blog: https://blog.ronaldoussoren.net/

[Ethan Furman]

On 4/8/21 1:43 PM, Antoine Pitrou wrote:
> On Thu, 8 Apr 2021 13:31:26 -0700
> Ethan Furman <et...@stoneleaf.us> wrote:
>>
>> ```python
>> from tempfile import NamedTemporaryFile
>>
>> with NamedTemporaryFile() as fp:
>> fp.write(b'some data')
>> fp.close() # Windows workaround
>> fp.open()
>> data = fp.read()
>>
>> assert data == 'some_data'
>> ```
>>
>> The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file
>> [2]. To handle this scenario, my proposal is two-fold:
>>
>> 1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close
>> 2) add `.open()` to NamedTemporaryFile
>
> Instead, you could add a dedicated `.reopen()`?

The main hurdle is that on Windows we let the OS manage the lifetime of the file, which means that it is deleted as soon
as it is closed. We would need to remove that branch and treat all NamedTemporaryFiles the same.

We could add reopen(), but since close() is already there... although I do like the name of `reopen`.

--
~Ethan~

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2DGIFUS6SU56MP6OWOEJPSWEDR2PL5CS/

[Rob Cliffe via Python-Dev]

Well this works:

from tempfile import NamedTemporaryFile
import os

with NamedTemporaryFile(delete=False) as fp:

      fp.write(b'some data')
      fp.close()

      with open(fp.name, 'rb') as fp2:
            data = fp2.read()
      os.remove(fp.name)
assert data == b'some data'

Of course it is theoretically possible that another app will do
something to the temporary file between the "fp.close()" and the "open"
on the next line, or between closing fp2 and the "os.remove".  Is this a
concern?
Rob Cliffe

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/5E5PTFW2BXU33V7LGVKMRD5I2425OT6G/

[Eric V. Smith]

On 4/8/2021 4:43 PM, Antoine Pitrou wrote:
> On Thu, 8 Apr 2021 13:31:26 -0700
> Ethan Furman <et...@stoneleaf.us> wrote:
>> ```python
>> from tempfile import NamedTemporaryFile
>>
>> with NamedTemporaryFile() as fp:
>> fp.write(b'some data')
>> fp.close() # Windows workaround
>> fp.open()
>> data = fp.read()
>>
>> assert data == 'some_data'
>> ```
>>
>> The problem is that, even though `fp.open()` is still inside the context manager, the `close()` call deletes the file
>> [2]. To handle this scenario, my proposal is two-fold:
>>
>> 1) stop using the TEMPFILE OS attribute so the OS doesn't delete the file on close
>> 2) add `.open()` to NamedTemporaryFile
> Instead, you could add a dedicated `.reopen()`?

I think capturing the intent is important, rather than just putting
.close() followed by .open(). Maybe a name that embodies "reopen for
reading without deleting" would make the intent clear? Maybe .reopen()
already captures that. Anyway, +1 for a method that combines the close/open.

Eric

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/4SZVHI6FEHAWBMGI5KUFYUOTTGQQACN5/

[Ivan Pozdeev via Python-Dev]

On 08.04.2021 23:31, Ethan Furman wrote:
> In issue14243 [1] there are two issues being tracked:
>
> - the difference in opening shared files between posix and Windows
> - the behavior of closing the underlying file in the middle of
>  NamedTemporaryFile's context management
>
> I'd like to address and get feedback on the context management issue.
>
> ```python
> from tempfile import NamedTemporaryFile
>
> with NamedTemporaryFile() as fp:
>    fp.write(b'some data')
>    fp = open(fp.name())
>    data = fp.read()
>
> assert data == 'some_data'
> ```
>
> Occasionally, it is desirable to close and reopen the temporary file in order to read the contents (there are OSes that cannot open a temp
> file for reading while it is still open for writing). This would look like:
>

What's the problem with `NamedTemporaryFile(mode='w+b')`? (it's actually the default!)

You can both read and write without reopening.
If you don't want to seek() between reading and writing, you can dup() the descriptor and wrap it with another file object:
`fp2=open(os.dup(fp.file.fileno()))`

--
Regards,
Ivan

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/SW4LRGEXYER6HDE3WBIKSQHENQRZLMRP/

[Serhiy Storchaka]

08.04.21 23:31, Ethan Furman пише:

> In issue14243 [1] there are two issues being tracked:
>
> - the difference in opening shared files between posix and Windows
> - the behavior of closing the underlying file in the middle of
>   NamedTemporaryFile's context management
>
> I'd like to address and get feedback on the context management issue.
>
> ```python
> from tempfile import NamedTemporaryFile
>
> with NamedTemporaryFile() as fp:
>     fp.write(b'some data')
>     fp = open(fp.name())
>     data = fp.read()
>
> assert data == 'some_data'
> ```

These issues are usually solved by using TemporaryDirectory:

with TemporaryDirectory() as td:
filename = os.path.join(td, 'tempfile')
with open(filename, 'wb') as fp:
fp.write(b'some data')
with open(filename, 'rb') as fp:
data = fp.read()

What if make NamedTemporaryFile a wrapper around TemporaryDirectory and
always create a new temporary directory for file?

@contextmanager
def NamedTemporaryFile():
with TemporaryDirectory() as td:
filename = os.path.join(td, 'tempfile')
with open(filename, 'wb') as fp:
yield fp

_______________________________________________
Python-Dev mailing list -- pytho...@python.org
To unsubscribe send an email to python-d...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at https://mail.python.org/archives/list/pytho...@python.org/message/2H2PIMPZ3VE2JDBUA7WVX25ULIC3C4SM/