Temporary files problems

[Andrey Novoseltsev]

Hello!

I've run into this while testing upgrades to Ubuntu 22.04 from 20.04, but it seems that it comes from how temporary files are handled in Sage 9.7:

TMP_DIR_FILENAME_BASE=tempfile.TemporaryDirectory()

this temporary directory is created and then used ever after. But what if it gets deleted? For example, what if someone uses tmpreaper or something similar, when old files are removed from /tmp? Sage does not notice it, but things break in a weird way, e.g. plots are not displayed and the error message is not transparent.

I believe this is exactly what is happening and I would appreciate some attention to this matter ;-)

Thank you!

Andrey

[John H Palmieri]

See also https://ask.sagemath.org/question/64192/temporary-html-files-location-in-sage-97/ — some browsers can't access local files in /tmp from the notebook, so plots do not appear. Perhaps TMP_DIR_FILENAME_BASE could be customizable. Other options?

[Andrey Novoseltsev]

I think it must be customizable, and also its use should be checked for errors - if it does not exist anymore, it should be recreated... Public SageCell servers probably see enough traffic to keep it alive, but people with private servers are very likely to run into problems, e.g. when their students are sleeping or pub crawling, all at once ;-)

[Martin R]

I also have this problem when doing

sage: qu = findstat("Permutations", compute_my_brand_new_permutation_statistic); qu
St000000: a new statistic on Permutations
sage: qu.submit()

which writes the information for findstat into a temporary file, and then opens a webbrowser to display it.

[Dima Pasechnik]

On Tue, Sep 27, 2022 at 3:58 AM Andrey Novoseltsev <novo...@gmail.com> wrote:
>
> Hello!
>
> I've run into this while testing upgrades to Ubuntu 22.04 from 20.04, but it seems that it comes from how temporary files are handled in Sage 9.7:
>
> TMP_DIR_FILENAME_BASE=tempfile.TemporaryDirectory()
>
> this temporary directory is created and then used ever after. But what if it gets deleted?

I think it's meant to be used in a context manager, i.e. with with
statement, e.g., citing python docs:

>>> with tempfile.TemporaryDirectory() as tmpdirname:
... print('created temporary directory', tmpdirname)
>>>
# directory and contents have been removed

Or with a callback (from Sage source)

TMP_DIR_FILENAME_BASE=tempfile.TemporaryDirectory()
atexit.register(lambda: TMP_DIR_FILENAME_BASE.cleanup())


Needless to say, one can interfere with it in a bad way messing around
with ~/.sage/ - but so it the case for any file-based process...

For example, what if someone uses tmpreaper or something similar,
when old files are removed from /tmp? Sage does not notice it, but
things break in a weird way, e.g. plots are not displayed and the
error message is not transparent.
>
> I believe this is exactly what is happening and I would appreciate some attention to this matter ;-)
>
> Thank you!
> Andrey
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/5ee63ec7-26b2-4c32-a5a7-45fb63a11327n%40googlegroups.com.

[John H Palmieri]

One of the issues is that code in Sage is using this temporary directory in the background; the user is not doing it and has no control over it. Regardless of the context manager approach, apparently some browsers refuse to open local files that are not in the user's home directory, so this location just won't work for them.

[Dima Pasechnik]

On Tue, 27 Sep 2022, 15:45 John H Palmieri, <jhpalm...@gmail.com> wrote:

One of the issues is that code in Sage is using this temporary directory in the background; the user is not doing it and has no control over it. Regardless of the context manager approach, apparently some browsers refuse to open local files that are not in the user's home directory, so this location just won't work for them.

tempfile.TemporaryDirectory(suffix=None, prefix=None, dir=None, ignore_cleanup_errors=False)

dir=  may be used to set a directory to place

the created one in.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/bcfa5eaa-600b-4752-8b52-189422564a49n%40googlegroups.com.

[Andrey Novoseltsev]

On Tuesday, 27 September 2022 at 03:22:47 UTC-6 dim...@gmail.com wrote:

On Tue, Sep 27, 2022 at 3:58 AM Andrey Novoseltsev <novo...@gmail.com> wrote:
> this temporary directory is created and then used ever after. But what if it gets deleted?
I think it's meant to be used in a context manager, i.e. with with
statement, e.g., citing python docs:

>>> with tempfile.TemporaryDirectory() as tmpdirname:
... print('created temporary directory', tmpdirname)
>>>
# directory and contents have been removed

Or with a callback (from Sage source)

TMP_DIR_FILENAME_BASE=tempfile.TemporaryDirectory()
atexit.register(lambda: TMP_DIR_FILENAME_BASE.cleanup())


Needless to say, one can interfere with it in a bad way messing around
with ~/.sage/ - but so it the case for any file-based process...

It seems to me like a pretty common and sensible practice, that /tmp directories get cleaned up, especially on systems with lots of activity. I would not consider it as "a bad way messing around". But creating a temporary directory and then counting on it being there hours later for another quick temporary operation may be a bit too optimistic. That callback is designed to clean up after Sage quits, which is a good thing to do, but it is not related to being able to use the directory. 

[Matthias Koeppe]

Previous discussion -- in the ticket that made these changes to SAGE_TMP --- https://trac.sagemath.org/ticket/33213#comment:11

[Dima Pasechnik]

Basically, we should deprecate and remove tmp_dir() and tmp_filename()
from Sage.
Does Sagecell use them? It should not, Python3 has perfectly good
replacements...

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8400b398-f51a-4a1b-a911-32f60ef8b8fdn%40googlegroups.com.

[John H Palmieri]

On Tuesday, September 27, 2022 at 8:33:01 AM UTC-7 dim...@gmail.com wrote:

On Tue, 27 Sep 2022, 15:45 John H Palmieri, <jhpalm...@gmail.com> wrote:

One of the issues is that code in Sage is using this temporary directory in the background; the user is not doing it and has no control over it. Regardless of the context manager approach, apparently some browsers refuse to open local files that are not in the user's home directory, so this location just won't work for them.

tempfile.TemporaryDirectory(suffix=None, prefix=None, dir=None, ignore_cleanup_errors=False)

dir=  may be used to set a directory to place

the created one in.

Exactly, and this is what we should allow users to customize, but they currently can't (without modifying the Sage source code).

[John H Palmieri]

On Tuesday, September 27, 2022 at 10:11:00 AM UTC-7 dim...@gmail.com wrote:

Basically, we should deprecate and remove tmp_dir() and tmp_filename()
from Sage.
Does Sagecell use them? It should not, Python3 has perfectly good
replacements...

As long as we test everything on a variety of platforms and browsers, because it seems that things are broken on some platforms now. https://ask.sagemath.org/question/64192/temporary-html-files-location-in-sage-97/

[Andrey Novoseltsev]

On Tuesday, 27 September 2022 at 11:11:00 UTC-6 dim...@gmail.com wrote:

Basically, we should deprecate and remove tmp_dir() and tmp_filename()
from Sage.
Does Sagecell use them? It should not, Python3 has perfectly good
replacements...

The problems that surfaces so far that I believe these tools may not address on their own:

1) tmp directory is used for "too long" i.e. while Sage process is running,  which may be a problem if it gets automatically cleaned

2) tmp stuff is deleted "too fast" e.g. as soon as it was generated, while some interfaces may rely on access to generated content a bit after Sage is done with it, e.g. to show plots in a browser

3) default location may not work with our interfaces as well

[Matthias Koeppe]

On Tuesday, September 27, 2022 at 10:21:15 AM UTC-7 John H Palmieri wrote:

On Tuesday, September 27, 2022 at 8:33:01 AM UTC-7 dim...@gmail.com wrote:

tempfile.TemporaryDirectory(suffix=None, prefix=None, dir=None, ignore_cleanup_errors=False)

dir=  may be used to set a directory to place

the created one in.

Exactly, and this is what we should allow users to customize, but they currently can't (without modifying the Sage source code).

Users can set the TMPDIR environment variable -- see

https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir

[Dima Pasechnik]

this sounds like a bad design

3) default location may not work with our interfaces as well

dir= parameter in TemporaryDirectory etc

can be used to set a better location.

This should take care of 1) too.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/faca7769-4469-4c98-aace-04ab902a370dn%40googlegroups.com.

[Matthias Koeppe]

https://manpages.ubuntu.com/manpages/bionic/man8/tmpreaper.8.html

suggests "As long as there are files present inside a subdirectory, it won't get removed. You can use a non-writable, self-owned file, perhaps named ".tmpreaper", or, if you are su, a file that has the ext2fs immutable attribute set, to keep a subdirectory from being deleted."

So perhaps we should just create such a file in TMP_DIR_FILENAME_BASE after creating it.

On Monday, September 26, 2022 at 7:58:01 PM UTC-7 novo...@gmail.com wrote:

[Michael Orlitzky]

On Tue, 2022-09-27 at 18:10 +0100, Dima Pasechnik wrote:
> Basically, we should deprecate and remove tmp_dir() and tmp_filename()
> from Sage.
> Does Sagecell use them? It should not, Python3 has perfectly good
> replacements...
>

That was always the plan. From #33213:

> Afterward, the custom functions tmp_dir() and tmp_filename() can be
> deprecated in favor of tempfile.TemporaryDirectory() and 
> tempfile.NamedTemporaryFile().

That solves the problem.

[John H Palmieri]

I created https://trac.sagemath.org/ticket/34593: we should document how to change the location of the temporary directory, and maybe we should create a `.tmpreaper` file.