Moving SAGE_TMP to the system location

[Michael Orlitzky]

tl;dr I think we should forego SAGE_TMP and use whatever the system's
temporary location is.

The first-order benefits of that are,

1. No need to maintain SAGE_TMP code ourselves.

2. The system temporary location is often optimized for temporary 
access (e.g. is mounted in RAM), and is automatically cleaned 
occasionally. Compare to SAGE_TMP which defaults to a directory 
under $HOME.

The second-order benefit is that it would eliminate the need for our
custom tmp_filename() and tmp_dir() functions. Both of those are thin
wrappers around what python provides, differing mainly in that they use
SAGE_TMP. But moreover,

1. tmp_filename() opens a file, closes it, and then gives you back
the name. This is in contrast to python's NamedTemporaryFile(),
which gives you the open file without closing it. Since in most 
cases you actually want to use the file, tmp_filename() is 
slightly but pointlessly inefficient.

2. tmp_dir() returns only the name of the newly-created directory,
while python's TemporaryDirectory() gives you a file-like object.
If you need to clean up, the file-like object can cleanup(), 
while the name returned from tmp_dir() requires you to call 
something like shutil.rmtree() on it.

3. Neither of them remove the file or directory when you're done 
with it, requiring sage-cleaner to deal with them at a later
point (if all goes well). The python tempfile functions give
you the option to auto-delete.

These are small annoyances. Does SAGE_TMP provide any benefits to
justify them?

If not, I'm proposing we,

1. Replace all direct uses of SAGE_TMP in library/doctest code with 
python's tempfile module.
2. Drop SAGE_TMP from tmp_filename() and tmp_dir(); this will revert
to whatever directory the OS uses.
3. Slowly migrate away from tmp_filename() and tmp_dir() to the 
functions that python provides.

[William Stein]

On Sun, Feb 13, 2022 at 6:57 AM Michael Orlitzky <mic...@orlitzky.com> wrote:
>
> tl;dr I think we should forego SAGE_TMP and use whatever the system's
> temporary location is.

+1.

I'm pretty sure I "created" this SAGE_TMP thing in the first place,
and that my reasons were:

a. ignorance about the built in Python solution (which in 2004 might
have been immature),

b. concerns about security, which are no longer of primary importance
for Sage. For example,
on a multiuser Linux system, admins need to make sure that files
written to /tmp are by default only
readable by one user; also, it's annoying if users accidentally write
a lot of data to /tmp, causing
trouble for everybody else.


Regarding these, of course (a) is not a good argument, and (b) should
no longer be either.
In addition, for users of CoCalc, the home directory is a network
mounted ZFS filesystem, with snapshots
and compression, whereas /tmp is a ramdisk dedicated to just that one
project that is orders of magnitude
faster than $HOME. Thus for CoCalc, it would be *massively* better
for Sage to use /tmp rather
than the current SAGE_TMP defaulting to a directory under $HOME.

-- William

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/84da47e539b342c270760e7e6200272a820fa5d1.camel%40orlitzky.com.



--
William (http://wstein.org)

[Volker Braun]

Are we talking about /tmp or /var/tmp? 

https://systemd.io/TEMPORARY_DIRECTORIES/ says:

* /tmp/ should be used for smaller, size-bounded files only; /var/tmp/ should be used for everything else.
* Data that shall survive a boot cycle shouldn’t be placed in /tmp/.

While its often the case that /var/tmp has some time-based cleanup, this is not universally so.

[Michael Orlitzky]

On Sun, 2022-02-13 at 09:18 -0800, Volker Braun wrote:
> Are we talking about /tmp or /var/tmp?

Short answer: /tmp.

Long answer: The python tempfile functions use a directory that,

"is chosen from a platform-dependent list, but the user of the 
application can control the directory location by setting the 
TMPDIR, TEMP or TMP environment variables."

So ultimately, it's up to python, or the user. But TMPDIR is typically
associated with /tmp on UNIX (both are POSIX), and I don't think
there's cross-platform consensus on a /var/tmp equivalent. So my
money's on /tmp where there's a choice.

[François Bissey]

> On 14/02/2022, at 03:57, Michael Orlitzky <mic...@orlitzky.com> wrote:
>
> If not, I'm proposing we,
>
> 1. Replace all direct uses of SAGE_TMP in library/doctest code with
> python's tempfile module.
> 2. Drop SAGE_TMP from tmp_filename() and tmp_dir(); this will revert
> to whatever directory the OS uses.
> 3. Slowly migrate away from tmp_filename() and tmp_dir() to the
> functions that python provides.

+1

[Michael Orlitzky]

On Sun, 2022-02-13 at 09:57 -0500, Michael Orlitzky wrote:
>
> If not, I'm proposing we,
>
> 1. Replace all direct uses of SAGE_TMP in library/doctest code with 
> python's tempfile module.
> 2. Drop SAGE_TMP from tmp_filename() and tmp_dir(); this will revert
> to whatever directory the OS uses.

I'm working on a branch at,

https://trac.sagemath.org/ticket/33213

to implement these two. If anyone has a Cygwin installation handy and
can answer my comment there, I might be able to further streamline the
doctests.