Running doctests in group writable directory

[scma...@gmail.com]

After our sysadmin updated the computer we use for a major project in Sage from v5.3 to v5.12, our automated doctests stopped working. It appears to be because the directory that contains our code is group writable (which makes sense because there are multiple users in our group who need to use this directory). The directory is not world writable. Is there a way to allow the doctests to run in a group writable directory?

Here is the output when trying to run a doctest via sage -t FILENAME:

Traceback (most recent call last):
File "/Applications/Sage-5.12-OSX-64bit-10.6.app/Contents/Resources/sage/local/bin/sage-runtests", line 87, in <module>
err = DC.run()
File "/Applications/Sage-5.12-OSX-64bit-10.6.app/Contents/Resources/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 882, in run
self.test_safe_directory()
File "/Applications/Sage-5.12-OSX-64bit-10.6.app/Contents/Resources/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 391, in test_safe_directory
.format(os.getcwd()))
RuntimeError: refusing to run doctests from the current directory '/CURRENT/DIRECTORY' since untrusted users could put files in this directory, making it unsafe to run Sage code from

[Volker Braun]

See http://trac.sagemath.org/ticket/13579#comment:40 and following...

[scma...@gmail.com]

I read through that ticket before posting, but I didn't (and still don't) see a solution to my problem. Admittedly I don't understand all of the issues talked about on that ticket. I created a test script in the same group writable directory I mentioned previously with the following content:

def spam():
"""
Run this function.

EXAMPLE::

sage: spam()
"Here's your spam."

"""
print "Here's your spam."

if __name__ == "__main__":
spam()


When I run this with python or sage, it runs fine. However, when I try to run the doctest, I get the same RuntimeError as before. I'm not sure how this fits into the discussion you referenced on trac, but it doesn't seem like the right behavior, unless I'm missing something.

[Nils Bruin]

On Thursday, November 7, 2013 7:13:39 AM UTC-8, scma...@gmail.com wrote:

I read through that ticket before posting, but I didn't (and still don't) see a solution to my problem. Admittedly I don't understand all of the issues talked about on that ticket. I created a test script in the same group writable directory I mentioned previously with the following content:

...

When I run this with python or sage, it runs fine. However, when I try to run the doctest, I get the same RuntimeError as before. I'm not sure how this fits into the discussion you referenced on trac, but it doesn't seem like the right behavior, unless I'm missing something.

Your problem arises from the fact that sage's python is patched to be a little more picky about  permissions on paths. For reference, this is from sys_path_security.patch

+    if (given_arg[0] != '\0' && stat(given_arg, &arg_stat) == 0) {
+        /* Only keep group bits if the group is the same as the
+         * group of "parent" (otherwise the group is considered unsafe). */
+        if (arg_stat.st_gid != parent_stat.st_gid)
+            arg_stat.st_mode &= 0707;
+        /* If parent does *not* have the sticky bit set, "arg" is at
+         * least as writable as "parent".  This obviously only applies
+         * if "arg" is an existing file/directory inside "parent", which
+         * is the case here. */
+        if (!(parent_stat.st_mode & S_ISVTX))
+            arg_stat.st_mode |= parent_stat.st_mode;

So I suspect your group-writable directory sits in a directory with a different group ID (that would be the normal setup for, say, a group writeable directory in /home). Your use case shows perhaps that this is not such a great heuristic. On the other side, from a security point of view it's better than nothing.

I see several solutions:
 - Change group ownership of the parent directory (that might need help from your sysadmin and it's very likely he'd have good reasons to object)
 - Nest everything one level deeper: make a directory INSIDE your group-owned-and-writeable directory and put everything in there. I think that might be enough to circumvent the newly-devised test.

[scma...@gmail.com]

On Thursday, November 7, 2013 10:16:45 AM UTC-7, Nils Bruin wrote:

Your problem arises from the fact that sage's python is patched to be a little more picky about  permissions on paths.

How come this only comes into play for doctesting and not for just running a script with sage? Using the example I posted before in the file example_script.py, I get

$ sage example_script.py

Here's your spam.

$ sage -t example_script.py 

Traceback (most recent call last):

...

RuntimeError: refusing to run doctests from the current directory '/DIR1/DIR2' since untrusted users could put files in this directory, making it unsafe to run Sage code from

 

So I suspect your group-writable directory sits in a directory with a different group ID (that would be the normal setup for, say, a group writeable directory in /home). Your use case shows perhaps that this is not such a great heuristic. On the other side, from a security point of view it's better than nothing.

I see several solutions:
 - Change group ownership of the parent directory (that might need help from your sysadmin and it's very likely he'd have good reasons to object)
 - Nest everything one level deeper: make a directory INSIDE your group-owned-and-writeable directory and put everything in there. I think that might be enough to circumvent the newly-devised test.

 

I thought about that too. However, the path for example_script.py looks like /DIR1/DIR2/example_script.py, where DIR1 and DIR2 are both group writable and belong to the same group. However, they do have different owners, so I tried nesting the script deeper, so the path is /DIR1/DIR2/example_dir/example_dir2/example_script.py, where both example_dir and example_dir2 have the same owner and group and are group writable (but not world writable). The results were the same as before.

I also tried running it in a group writable directory in my home directory, which also failed. It seems to me that the documented behavior does not match the actual behavior.

[Nils Bruin]

On Thursday, November 7, 2013 10:01:00 AM UTC-8, scma...@gmail.com wrote:

How come this only comes into play for doctesting and not for just running a script with sage? Using the example I posted before in the file example_script.py, I get

 

It looks like sage silences the python message. If you run "sage --python" you get sage's python straight and if you run it where you want to run your doctest, you'll probably see a warning message printed. Sage's python is only patched to print a warning (and change the path setup). It still runs.

The doctest script checks whether python prints a warning message and, if so, refuses to run. The warning messages from "sage --python" about paths are pretty specific, though, so perhaps running it gives you some hints. I thought about that too. However, the path for example_script.py looks like /DIR1/DIR2/example_script.py, where DIR1 and DIR2 are both group writable and belong to the same group. However, they do have different owners, so I tried nesting the script deeper, so the path is /DIR1/DIR2/example_dir/example_dir2/example_script.py, where both example_dir and example_dir2 have the same owner and group and are group writable (but not world writable). The results were the same as before.

I also tried running it in a group writable directory in my home directory, which also failed. It seems to me that the documented behavior does not match the actual behavior.

I can confirm that I also am not able to get "sage --python" to run without printing a warning in any situation I tried where the current directory is group writeable. I think the intention was that this should in principle be possible so perhaps this is a bug.

[Jeroen Demeyer]

On 2013-11-07 19:37, Nils Bruin wrote:
> I can confirm that I also am not able to get "sage --python" to run
> without printing a warning in any situation I tried where the current
> directory is group writeable.

You need either your umask to allow group-writing or you need to run
python on a group-writable script or you need Python itself to be
group-writable. But the scenario of the original poster of a "trusted"
group indeed isn't supported, since there is no way for Python to know
that that group is trusted.

[Nils Bruin]

Ah thanks! Translating this into commands:

$ umask 002
$ sage -t ...

should work. By setting your shell umask, you're informing python that you consider groups safe. Do we have this documented anywhere? I wouldn't have thought of this.

[Jeroen Demeyer]

On 2013-11-08 09:29, Nils Bruin wrote:
> Do we have this documented anywhere?

No. The place to document this would of course by Python. I personally
find it very unfortunate that upstream CPython seems to ignore this
issue. Perhaps my fix isn't perfect (as shown by this thread), but not
doing anything is even worse.

[Dima Pasechnik]

It might be that they want a platform-agnostic fix.
Yours is Unix-only, right?

[Jeroen Demeyer]

On 2013-11-08 12:37, Dima Pasechnik wrote:
> It might be that they want a platform-agnostic fix.

That's not the issue at all.

[scma...@gmail.com]

That makes sense, but it didn't work for me:

$ umask 002

$ umask

0002

$ sage -t example_script.py 

Traceback (most recent call last):

...

RuntimeError: refusing to run doctests from the current directory '/DIR1/DIR2' since untrusted users could put files in this directory, making it unsafe to run Sage code from

[Nils Bruin]

On Friday, November 8, 2013 6:56:54 AM UTC-8, scma...@gmail.com wrote:

That makes sense, but it didn't work for me:

$ umask 002

$ umask

0002

$ sage -t example_script.py 

Traceback (most recent call last):

...

RuntimeError: refusing to run doctests from the current directory '/DIR1/DIR2' since untrusted users could put files in this directory, making it unsafe to run Sage code from

Hm, would you mind posting the results of:

$pwd

and then the permissions of all components, e.g.: if it's /home/user/sage

$ ls -dl /home
$ ls -dl /home/user
$ ls -dl /home/user/sage
$ ls -dl /home/user/sage/example_script.py
$ ls -dl `which sage`

You can apply some bijective map to the UIDs, GIDs, and directory names if you think that's required.

You might also want to see what kind of file system is mounted for this. I think there are network file systems that keep track of permissions via ACL and return garbage for unix permissions. That would definitely throw off Jeroen's heuristic (and problems like this is why the python devs are so reluctant to include a change like this)

[scma...@gmail.com]

On Friday, November 8, 2013 9:53:06 AM UTC-7, Nils Bruin wrote:

Hm, would you mind posting the results of:

$pwd

and then the permissions of all components, e.g.: if it's /home/user/sage

$ ls -dl /home
$ ls -dl /home/user
$ ls -dl /home/user/sage
$ ls -dl /home/user/sage/example_script.py
$ ls -dl `which sage`

You can apply some bijective map to the UIDs, GIDs, and directory names if you think that's required.

$ pwd

/GROUP_DIR/CODE_DIR/example_dir/example_dir2

$ ls -dl example_script.py 

-rwxrwxr-x  1 scott  RESEARCH_GROUP  196 Nov  7 10:53 example_script.py

$ dir=$(pwd); while [ ! -z "$dir" ]; do ls -ld "$dir"; dir=${dir%/*}; done; ls -ld /

drwxrwxr-x  3 scott  RESEARCH_GROUP  102 Nov  8 13:20 /GROUP_DIR/CODE_DIR/example_dir/example_dir2

drwxrwxr-x  3 scott  RESEARCH_GROUP  102 Nov  7 10:43 /GROUP_DIR/CODE_DIR/example_dir

drwxrwx--x  82 tyler  RESEARCH_GROUP  2788 Nov  8 13:20 /GROUP_DIR/CODE_DIR

drwxrwxr-x  32 amanda  RESEARCH_GROUP  1088 Nov 30  2012 /GROUP_DIR

drwxrwxr-x  44 root  _www  1564 Sep 21 22:35 /

$ ls -dl `which sage`

lrwxr-xr-x  1 root  wheel  71 Nov  4 16:23 /usr/bin/sage -> /Applications/Sage-5.12-OSX-64bit-10.6.app/Contents/Resources/sage/sage

 

You might also want to see what kind of file system is mounted for this. I think there are network file systems that keep track of permissions via ACL and return garbage for unix permissions. That would definitely throw off Jeroen's heuristic (and problems like this is why the python devs are so reluctant to include a change like this)

I'm not positive what all of that means, but the computer it's on is a Mac OS X server running OS X 10.6, if that helps. 

[Nils Bruin]

OK, I've tried to replicate your scenario and for me it works (on linux), so it might be something about OSX. I think what you are experiencing can be characterized as a "bug". Hopefully someone can fix it or find a work-around. For reference, this is what I get:

$ pwd
/home/nbruin/U
$ ls -dl example_script.py
-rwxrwxr-x 1 nbruin nbruin 194 2013-11-08 12:54 example_script.py

$ dir=$(pwd); while [ ! -z "$dir" ]; do ls -ld "$dir"; dir=${dir%/*}; done; ls -ld /

drwxrwxr-x 2 nbruin nbruin 4096 2013-11-08 12:54 /home/nbruin/U
drwxr-xr-x 22 nbruin nbruin 4096 2013-11-08 12:54 /home/nbruin
drwxr-xr-x 361 root root 12288 2013-10-29 15:06 /home
drwxr-xr-x 28 root root 4096 2013-09-26 09:14 /

$ ls -dl `which sage`

-rwxr-xr-x 1 root root 4995 2012-05-17 23:06 /usr/local/bin/sage
$ umask
0022

$ sage -t example_script.py
Traceback (most recent call last):

  File "/usr/local/share/sage-5.8/local/bin/sage-test", line 53, in <module>
    .format(os.getcwd()))
RuntimeError: refusing to run doctests from the current directory '/home/nbruin/U' since untrusted users could put files in this directory, making it unsafe to run Sage code from

$ umask 002
$ umask
0002
$ sage -t example_script.py

sage -t  "example_script.py"                               
     [3.9 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 4.0 seconds

There is one workaround possible: build sage's python with the relevant patch removed. It's "sys_path_security.patch". Your system administrator might be able to help.

[Nils Bruin]

On Friday, November 8, 2013 1:24:33 PM UTC-8, Nils Bruin wrote:

I think what you are experiencing can be characterized as a "bug". Hopefully someone can fix it or find a work-around.

In fact, I've just tried the same scenario on bsd.math.washington.edu, which runs Darwin (so I guess OSX). (note that /usr/local/bin/sage on that machine is version 5.2, which doesn't have the patch in yet. Use /Users/buildbot/build/sage/bsd-1/bsd_binary/build/sage-5.13.beta1/sage instead)

$ cd /tmp
$ mkdir U
$ chmod 770 U
$ cd U
$ touch test.py
$ umask 022
$ sage -t test.py #this fails as expected
$ umask 002
$ sage -t test.py #this fails on bsd but succeeds on boxen

so something with retrieving umask doesn't work properly on OSX. With a concise way of reproducing the problematic behaviour, this is now http://trac.sagemath.org/ticket/15387#