tests related to papers and books

[William Stein]

Hi,

I'm at a reproducible research workshop, and that reminded me that we
have a directory in Sage full of tests related to published works:

https://github.com/sagemath/sage/tree/master/src/sage/tests

But it's really not much at all! So if you're going to write (or
wrote) a paper with Sage examples, try to remember to add a file to
the above directory, so your examples work forever, etc.

William


--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Anne Schilling]

Hi William,

We actually added tests for our k-Schur function book. But a lot of time, when syntax in
Sage changes, these tests just get replaced by others without checking with the authors
of the book/paper. So the published Sage examples can still break. How should this
be handled?

Anne

[William Stein]

On Thu, May 8, 2014 at 9:26 PM, Anne Schilling
<anne1.s...@gmail.com> wrote:
> Hi William,
>
> We actually added tests for our k-Schur function book. But a lot of time,
> when syntax in
> Sage changes, these tests just get replaced by others without checking with
> the authors
> of the book/paper. So the published Sage examples can still break. How
> should this
> be handled?

Changes to Sage that break published works should require a 1-year
deprecation period (during which the author is notified
and can post an errata), and only after that year is the change
actually made. This is an official policy on all nontrivial
backward incompatible changes, which we decided on a few years ago.
There's a deprecated decorator for this, etc.

Of course, when one is actually "in the trenches" coding, it can be
very frustrating to have to wait an entire year for your important
change (that breaks something) to actually be officially released.
And it is those people who are in the trenches coding that are the
reason Sage exists and is getting better.

William

>
> Anne
>
>
> On Thursday, May 8, 2014 3:38:10 PM UTC-7, William wrote:
>>
>> Hi,
>>
>> I'm at a reproducible research workshop, and that reminded me that we
>> have a directory in Sage full of tests related to published works:
>>
>> https://github.com/sagemath/sage/tree/master/src/sage/tests
>>
>> But it's really not much at all! So if you're going to write (or
>> wrote) a paper with Sage examples, try to remember to add a file to
>> the above directory, so your examples work forever, etc.
>>
>> William
>>
>>
>> --
>> William Stein
>> Professor of Mathematics
>> University of Washington
>> http://wstein.org
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[leif]

Anne Schilling wrote:
> We actually added tests for our k-Schur function book. But a lot of
> time, when syntax in
> Sage changes, these tests just get replaced by others without checking
> with the authors
> of the book/paper.

git log src/sage/tests/book_schilling_zabrocki_kschur_primer.py | grep
Author ;-)


I actually thought there was some README or WARNING file in that folder,
explaining the convention for modifying tests there, but apparently
there isn't.

Probably each relevant file should contain a CAPSLOCK header with
information on how to contact the author(s) etc.


-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Volker Braun]

On Friday, May 9, 2014 1:40:51 PM UTC+2, leif wrote:

Probably each relevant file should contain a CAPSLOCK header with
information on how to contact the author(s) etc.

DEAR SIR WE HAVE RECENTLY COME INTO POSSESSION OF THIS EXTREMELY VALUABLE DOCTEST FOR WHICH WE WOULD LIKE TO ENLIST YOUR HELP TO ENSURE ITS SAFE PASSAGE. 

[David Loeffler]

It's happened a few times that people have put tests in the sage/tests
directory which explicitly require that certain inputs return
NotImplementedError, or something similar. Surely we shouldn't have to
go through the rigmarole of contacting the original author and waiting
a year in order to add a previously un-implemented feature?

I'd say that, alongside the policy of not changing the output of tests
in sage/tests without a year's deprecation period and consultation
with the original author, we should have a parallel policy that all
tests added to sage/tests should be "forward-compatible", in the sense
that they shouldn't be broken by the addition of reasonably
foreseeable new features to Sage.

David

[Jeroen Demeyer]

On 2014-05-09 14:08, David Loeffler wrote:
> It's happened a few times that people have put tests in the sage/tests
> directory which explicitly require that certain inputs return
> NotImplementedError, or something similar. Surely we shouldn't have to
> go through the rigmarole of contacting the original author and waiting
> a year in order to add a previously un-implemented feature?
>
> I'd say that, alongside the policy of not changing the output of tests
> in sage/tests without a year's deprecation period and consultation
> with the original author, we should have a parallel policy that all
> tests added to sage/tests should be "forward-compatible", in the sense
> that they shouldn't be broken by the addition of reasonably
> foreseeable new features to Sage.

Also, sometimes floating-point accuracy of some function improves,
"breaking" a doctest.

[leif]

David Loeffler wrote:
> It's happened a few times that people have put tests in the sage/tests
> directory which explicitly require that certain inputs return
> NotImplementedError, or something similar. Surely we shouldn't have to
> go through the rigmarole of contacting the original author and waiting
> a year in order to add a previously un-implemented feature?
>
> I'd say that, alongside the policy of not changing the output of tests
> in sage/tests without a year's deprecation period and consultation
> with the original author, we should have a parallel policy that all
> tests added to sage/tests should be "forward-compatible", in the sense
> that they shouldn't be broken by the addition of reasonably
> foreseeable new features to Sage.

Well, the point is that (useful!) *examples* from published books should
work for a reasonable amount of time.

I don't think a reader will be upset if Sage suddenly implements
something which didn't work to the time a book was published; we should
still notify the authors in such cases though.

[leif]

Same for deprecation warnings, but see my other reply. The files there
don't have to be "read-only", but changes should be reviewed carefully,
and the authors should get notified when appropriate (which I think is
not the case for floating-point noise).

[Anne Schilling]

On 5/9/14 4:40 AM, leif wrote:
> Anne Schilling wrote:
>> We actually added tests for our k-Schur function book. But a lot of
>> time, when syntax in
>> Sage changes, these tests just get replaced by others without checking
>> with the authors
>> of the book/paper.
>
> git log src/sage/tests/book_schilling_zabrocki_kschur_primer.py | grep
> Author ;-)

Well, this tells you the authors who made changes to the test files
(and they are not necessarily the authors of the book)!

> I actually thought there was some README or WARNING file in that folder,
> explaining the convention for modifying tests there, but apparently
> there isn't.
>
> Probably each relevant file should contain a CAPSLOCK header with
> information on how to contact the author(s) etc.

Ok, this is now

http://trac.sagemath.org/ticket/16315

Anne

[leif]

Anne Schilling wrote:
> On 5/9/14 4:40 AM, leif wrote:
>> Anne Schilling wrote:
>>> We actually added tests for our k-Schur function book. But a lot of
>>> time, when syntax in
>>> Sage changes, these tests just get replaced by others without checking
>>> with the authors
>>> of the book/paper.
>>
>> git log src/sage/tests/book_schilling_zabrocki_kschur_primer.py | grep
>> Author ;-)
>
> Well, this tells you the authors who made changes to the test files
> (and they are not necessarily the authors of the book)!

Which exactly was my point ("[some of] these tests just get replaced by
$OTHERS without checking with the authors").

[Nathann Cohen]

Hello everybody !

I thought a bit about this ongoing conversation, and I wondered if the best to do wouldn't be to give Sage users some *automatic* service to *help them maintain their code*.

My point is that I would not like us to take upon ourselves to maintain everybody's code, given that sometimes, let's face it, the code written by mathematicians is just very very bad, and sometimes buggy. And that fixing somebody else's bug in a code that we do not use ourselves is not a very pleasant task.

What would you think of a service like that :

1) Authors can upload worksheets or doctests on some website/machine

2) We swear that we will test the file each time a new version of Sage is released

3) If some doctest ever breaks, we automatically send them an email telling them so, so that they can update their code *if they like*

Advantages :

1) It would help them make sure that their code works, and they would be aware of which modifications have to be made

2) We do not have to maintain their code ourselves, possibly having to fix their own blunder

3) It is really a public service and can grow as large as needed without requiring the slightest effort from our part

What do you think ?

Nathann

[leif]

Well, essentially that's already possible:

Create a trac ticket with your file(s) in a branch, and let the
patchbots regularly test it.

(Although that's kind of abusing Sage's trac, since such tickets will
presumably never get merged, i.e., closed.)


We could also set up a patchbot that tests remote branches other than
those on Sage's git server with automatic mail feedback.

[Nathann Cohen]

> Well, essentially that's already possible:
>
> Create a trac ticket with your file(s) in a branch, and let the patchbots
> regularly test it.
>
> (Although that's kind of abusing Sage's trac, since such tickets will
> presumably never get merged, i.e., closed.)
>
>
> We could also set up a patchbot that tests remote branches other than those
> on Sage's git server with automatic mail feedback.

HMmmm... Perhaps this could also be merged with the cloud ? It would
make it easier for everybody to "add a tracked Sage file" to a
repository than create a git branch ?

By the way I would be glad if the patchbot was willing to tun on my machine T_T

Nathann

[rjf]

It seems to me that the "reproducibility" should be with respect to the same conditions as the original publication.  That is, someone who says  "I'm telling the truth because yada yada Sage version  x.y.z on machine q.p....  should provide not only the commands, but version x.y.z  and machine q.p on which it was run.  (Provide to whom? )

[Nicolas M. Thiery]

I was pondering about something along those lines lately :-)

4) We could get some statistics:
- Is feature A used a lot?
- How may notebooks out there suddenly break when I change A
- ...
which could help take informed decisions.

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Dr. David Kirkby]

On 10 May 2014 17:37, "rjf" <fat...@gmail.com> wrote:
>
>
>
> It seems to me that the "reproducibility" should be with respect to the same conditions as the original publication.  That is, someone who says  "I'm telling the truth because yada yada Sage version  x.y.z on machine q.p....  should provide not only the commands, but version x.y.z  and machine q.p on which it was run.  (Provide to whom? )

IF I am understanding you correctly,  that would be extreamly unusual and inappropriate.  I admit that I may have may have misunderstood you.

I believe that you are saying to allow others to verify the results you need to supply both the source code and computer hardware to others.

Why should research with Sage be l performed in a way different from how research in any research area I can think of? I have worked on medical physics,  RF engineering,  optics and never worked that way and only seen a limited subset of it.

It would be economically impossible to do this. How much money would you need to put into a research grant application if one of the objectives was to provide computer hardware to one or more people that might want it?

The nearest I have seen to this sort of thing is where a research group measures something (the device under test or DUT)  then passes the DUT to other research groups who measure the same DUT. When all groups taking part of the study have measured the DUT using different equipment in different laboratories with different people,  the results are compared.

It is just not practical to distribute the same computer hardware,  and in any case it might cause inaccurate conclusions to be reached. Remember the Pentium bug, where an Intel CPU gave an error computing certain floating point operations?

Making the source code available is both practical and useful,  especially if it can be compiled on multiple hardware platforms with different compilers.  That would reduce the chances of erroneous results from CPU bugs, or similar.

Dave

[John Cremona]

I'm sure that rtf did not mean to provide the physical machine, just the information about the machine (hardware and os, say)!

John

--

[William Stein]

On Mon, May 12, 2014 at 8:51 AM, Dr. David Kirkby <drki...@gmail.com> wrote:
>
> On 10 May 2014 17:37, "rjf" <fat...@gmail.com> wrote:
>>
>>
>>
>> It seems to me that the "reproducibility" should be with respect to the
>> same conditions as the original publication. That is, someone who says
>> "I'm telling the truth because yada yada Sage version x.y.z on machine
>> q.p.... should provide not only the commands, but version x.y.z and
>> machine q.p on which it was run. (Provide to whom? )
>
> IF I am understanding you correctly, that would be extreamly unusual and
> inappropriate. I admit that I may have may have misunderstood you.

Hi,

There is a rather large Moore/Sloane funded project jointly between
Univ of Washington (where I am), Berkeley (where RJF is), and
NYU on "reproducible research". For example, I spent all day last
Thursday at this workshop

http://escience.washington.edu/event/first-reproducibility-workshop

as one of the speakers (as was Fernando Perez of IPython). I also
co-organized a week-long workshop on "Reproducible Research"
at ICERM a bit over a year ago, with academic people along with people
from Google, Github, etc.,

As far as I can tell, there are no two active researchers that agree
on what the phrase "reproducible research" means. Frequently people
think the meaning is clear until they talk to somebody else in a
different area about it. More or less -- in traditional pure math,
"reproducible research" means "research", i.e., you publish a proof.
But in each area of the sciences, industry, government labs, journal
publication, etc., it is very subtle and different what it means or
_should_ mean (based on various arguments and tradeoffs). And as soon
as you think you have your head around it, somebody points out some
subtle aspect of their own situation that forces you to reconsider.

Anyway, I just wanted to point out that there is a lot going on at the
moment that is motivated by "reproducible research".

William

>
> I believe that you are saying to allow others to verify the results you need
> to supply both the source code and computer hardware to others.
>
> Why should research with Sage be l performed in a way different from how
> research in any research area I can think of? I have worked on medical
> physics, RF engineering, optics and never worked that way and only seen a
> limited subset of it.
>
> It would be economically impossible to do this. How much money would you
> need to put into a research grant application if one of the objectives was
> to provide computer hardware to one or more people that might want it?
>
> The nearest I have seen to this sort of thing is where a research group
> measures something (the device under test or DUT) then passes the DUT to
> other research groups who measure the same DUT. When all groups taking part
> of the study have measured the DUT using different equipment in different
> laboratories with different people, the results are compared.
>
> It is just not practical to distribute the same computer hardware, and in
> any case it might cause inaccurate conclusions to be reached. Remember the
> Pentium bug, where an Intel CPU gave an error computing certain floating
> point operations?
>
> Making the source code available is both practical and useful, especially
> if it can be compiled on multiple hardware platforms with different
> compilers. That would reduce the chances of erroneous results from CPU
> bugs, or similar.
>
> Dave
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Dr. David Kirkby]

On 12 May 2014 16:58, "John Cremona" <john.c...@gmail.com> wrote:
>
> I'm sure that rtf did not mean to provide the physical machine, just the information about the machine (hardware and os, say)!
>
> John

..k
"(Provide to whom? )"