Automatically test Maple, Magma [...] when running the tests

[Nathann Cohen]

Hello everybody,

Nowadays when running 'sage -t', one automatically runs all tests
flagged with '#optional - X' for all installed packages X which are
[optional+new_style].

But many doctests are flagged with '#optional - X' where X is not even
a package, e.g. when X is a proprietary software that we are not
allowed to redistribute as a package.

What packages of this kind should we test? How can we detect if it is
locally available?

I created ticket #18904 [1] to auto-detect CPLEX and Gurobi
(proprietary LP solvers), but if you have more in mind please add a
commit so that they are added to the list.

THaaaaaaaaaaanks,

Nathann

[1] http://trac.sagemath.org/ticket/18904

[John Cremona]

On 15 July 2015 at 10:43, Nathann Cohen <nathan...@gmail.com> wrote:

Hello everybody,

Nowadays when running 'sage -t', one automatically runs all tests
flagged with '#optional - X' for all installed packages X which are
[optional+new_style].

But many doctests are flagged with '#optional - X' where X is not even
a package, e.g. when X is a proprietary software that we are not
allowed to redistribute as a package.

What packages of this kind should we test? How can we detect if it is
locally available?

I created ticket #18904 [1] to auto-detect CPLEX and Gurobi
(proprietary LP solvers), but if you have more in mind please add a
commit so that they are added to the list.

Thanks for doing this.  Magma should be in this list too.   The command line

echo 'quit;' | magma -b -n

will exit with no output if magma exists on the system, while if not then it gives

magma: command not found

and I expect you can write this into a test.

John

 

THaaaaaaaaaaanks,

Nathann

[1] http://trac.sagemath.org/ticket/18904

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Nathann Cohen]

Hello,

> echo 'quit;' | magma -b -n

Actually, the test will appear in a Python script (and we can use any
of Sage's functions). We can of course run this external command, but
wouldn't there be an easier way to test this? To illustrate, the test
for Cplex is the following:

try:
MixedIntegerLinearProgram(solver='cplex')
<add cplex to the list>
except:
pass

It also checks for possible problems if the license is misinstalled of
expired. I just tried to run magma on my computer and it raised an
exception. Do you think that we could 'detect' magma better by running
something like that?

try:
magma(1)
<add cplex to the list>
except:
pass

Nathann

[Dima Pasechnik]

yes, this looks about right. Also, for  more of this:

sage: mathematica(10)
10
sage: matlab(10)
    10
sage: magma(10)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[...]

Nathann

[Nathann Cohen]

> yes, this looks about right. Also, for more of this:

I added matlab, maple, macaulay2, octave and matlab with the same test
pattern. I did not add mathematica: on my machine, ' mathematica(1)'
hangs forever (I don't have it).

The code is getting uglier, so if you know how it could be rewritten?... :-/

Nathann

[John Cremona]

On 15 July 2015 at 14:19, Nathann Cohen <nathan...@gmail.com> wrote:

> yes, this looks about right. Also, for  more of this:

I added matlab, maple, macaulay2, octave and matlab with the same test
pattern. I did not add mathematica: on my machine, ' mathematica(1)'
hangs forever (I don't have it).

Same here.  I killed it with Ctrl-C, then did mathematica?? and was surprised to get the message:

Mathematica crashed -- automatically restarting.

so perhaps the mathametica interface is stuck in a loop endlessly re-trying.

John

 

The code is getting uglier, so if you know how it could be rewritten?... :-/

Nathann

[Sébastien Labbé]

It does work for me (with mathematica installed):

$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath Version 6.8.beta8, Release Date: 2015-07-10               │
│ Type "notebook()" for the browser-based notebook interface.        │
│ Type "help()" for help.                                            │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: mathematica(35)
35
sage: mathematica_console()
Mathematica 10.0 for Linux x86 (64-bit)
Copyright 1988-2014 Wolfram Research, Inc.

In[1]:=

[Dima Pasechnik]

On Wednesday, 15 July 2015 14:27:02 UTC+1, John Cremona wrote:

On 15 July 2015 at 14:19, Nathann Cohen <nathan...@gmail.com> wrote:

> yes, this looks about right. Also, for  more of this:

I added matlab, maple, macaulay2, octave and matlab with the same test
pattern. I did not add mathematica: on my machine, ' mathematica(1)'
hangs forever (I don't have it).

Same here.  I killed it with Ctrl-C, then did mathematica?? and was surprised to get the message:

Mathematica crashed -- automatically restarting.

so perhaps the mathametica interface is stuck in a loop endlessly re-trying.

see http://trac.sagemath.org/ticket/18908

[Jeroen Demeyer]

Hello Nathann,

I do not agree with testing all external packages. The fact that I
happen to have a magma/maple/whatever installation on my computer does
not mean that I care about Sage doctest failures. Maybe I have a version
of the external package which is not fully compatible with Sage but
which works for me? Or maybe I have the external package but I don't
care about the Sage interface with that package?

By automatically running those tests, you are forcing people to care
about stuff that they didn't want to care about.

The fact that proprietary packages will probably never be automatically
tested on the buildbot for license reasons is also a good reason to not
run those tests automatically.

Jeroen.

[Sébastien Labbé]

By automatically running those tests, you are forcing people to care
about stuff that they didn't want to care about.

I don't really care myself about the mathematica interface. But it happens that mathematica 10.0 is installed on my machine in my lab and that most of the people in my lab use mathematica rather than sage. They won't test the mathematica doctests in the sage library since they are not sage developpers. But, in the next year, I might have a research discussion with them where they will be using mathematica and I will be using sage. And it is possible that I will need to use the interface. And if the doctests are broken, then, I won't be happy. And I won't be able to convince them about using Sage or that interface.

Personnaly, I want the mathematica optional tests to be run on my machine when I run "make ptestlong" if mathematica is available on my machine.

I we don't agree with this, then, I would like a command like the one suggested by dimpase [1]:

make ptest OPTIONAL+=mathematica

[1] http://trac.sagemath.org/ticket/18904#comment:27

Sébastien

[Nathann Cohen]

I also believe that we should test them. The very fact that there are
broken doctests that we did not fix only because we never run them
proves it (see comment 7 of #18908).

Also, there are two things that you seem to take for granted:
- "[...] does not mean that I care about Sage doctest failures":
Why should there be failures in the first place? Doctests must pass,
and we will fix those failures. That's the purpose of doctests: detect
failures and fix them.

- "Maybe I have a version of the external package which is not fully

compatible with Sage but which works for me?"

Why wouldn't we be able to detect that? What is somebody has such an
install and *wants* to use it with Sage? We must be able to raise a
proper exception if something does not work, so that (s)he can figure
out how to fix his install.

"The fact that proprietary packages will probably never be
automatically tested on the buildbot for license reasons is also a
good reason to not run those tests automatically."

"For license reasons"? What's wrong with having Maple installed on a
patchbot? We can already have patchbot running non-GPL-compatible
softwares like CBC, as it is a new-style optional package.

Finally, regardless of whether the patchbots will run the tests, you
know as well as I that something which is not tested gets broken. And
that's the current state of our interface.* modules, unless we do
something about it.

Nathann

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/sage-devel/M98nD7833KM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Jeroen Demeyer]

On 2015-07-16 12:51, Sébastien Labbé wrote:
> I don't really care myself about the mathematica interface. But it
> happens that mathematica 10.0 is installed on my machine in my lab and
> that most of the people in my lab use mathematica rather than sage. They
> won't test the mathematica doctests in the sage library since they are
> not sage developpers.

Let's assume the story ends here. You use Sage only and other people use
Mathematica only. If you run "make ptestlong", would you like to see
failures related to Mathematica? I know I would be bothered by seeing
those failures.

Jeroen.

[Nathann Cohen]

> Let's assume the story ends here. You use Sage only and other people use
> Mathematica only. If you run "make ptestlong", would you like to see
> failures related to Mathematica? I know I would be bothered by seeing those
> failures.

Well, perhaps we could .... fix the problems that the doctests report,
so that Sage gets better?

[Jeroen Demeyer]

On 2015-07-16 13:03, Nathann Cohen wrote:
> What is somebody has such an
> install and *wants* to use it with Sage?

This points to the core problem: we simply have no way to guess whether
people want to do that or not.

Here is a possible compromise proposal:
* keep default optional package as they are now
* add a new flag --optional=external (or change --optional=all to do
this) to test all auto-detected-but-not-part-of-Sage packages like
mathematica.
* add a corresponding Makefile target running tests with this flag.

[Jeroen Demeyer]

On 2015-07-16 13:13, Nathann Cohen wrote:
> Well, perhaps we could .... fix the problems that the doctests report,
> so that Sage gets better?

Of course we should fix the failures, but that's really not the point of
this discussion.

[Nathann Cohen]

> This points to the core problem: we simply have no way to guess whether
> people want to do that or not.

We do not have to care. Doctests must pass, whatever the install is.
If the local install is broken then we should have a way to detect it,
and if we detect it we must not run the doctests automatically. But if
the install is not broken and the doctests break, then we must know
it.

> Here is a possible compromise proposal:
> * keep default optional package as they are now
> * add a new flag --optional=external (or change --optional=all to do this)
> to test all auto-detected-but-not-part-of-Sage packages like mathematica.
> * add a corresponding Makefile target running tests with this flag.

You know that nobody will ever run those things. The same way that we
do not do it today, even though we have a "--optional=mathematica"
flag available. That's just code that nobody will run, and our
interface will remain as broken as they are now because nobody tests
them.

Anyway. Like I said on the ticket, I'm tired of arguing with you when
you don't want to hear something. Cplex and Gurobi will be tested, and
those who care about having a working interface with Maple, Matlab,
Octave, Mathematica, Scilab, Macaulay2 and others will have to defend
their interests against you. Too bad you fight for a way which will
bring no improvement to our software.

Nathann

[Jeroen Demeyer]

On 2015-07-16 13:22, Nathann Cohen wrote:
> You know that nobody will ever run those things.

If it's running "make ptestextlong" instead of "make ptestlong" maybe
they will. It's not really more complicated.

[Sébastien Labbé]

Let's assume the story ends here. You use Sage only and other people use
Mathematica only. If you run "make ptestlong", would you like to see
failures related to Mathematica? I know I would be bothered by seeing
those failures.

The doctests in src/sage/interfaces/*
are the same as
the doctests in src/sage/finance/*
and the same as
the doctests in src/sage/numerical/*
and the same as
etc.
and I would be bothered by seeing failure in any of those modules.

[Jeroen Demeyer]

On 2015-07-16 13:22, Nathann Cohen wrote:

>> This points to the core problem: we simply have no way to guess whether
>> people want to do that or not.
>
> We do not have to care. Doctests must pass, whatever the install is.
> If the local install is broken then we should have a way to detect it,
> and if we detect it we must not run the doctests automatically. But if
> the install is not broken and the doctests break, then we must know
> it.

I agree with this whole paragraph you wrote.

However, it's only because you used "we" where I assume you mean "Sage
developers in general". If you replace "we" by any person in particular,
I no longer agree.

Jeroen.

[Nathann Cohen]

> However, it's only because you used "we" where I assume you mean "Sage
> developers in general". If you replace "we" by any person in particular, I
> no longer agree.

So there are two kinds of people who run the doctests:
- Those who run the doctests and want to see the errors
- Those who run the doctests and cannot be bothered with the errors

We could have a .txt file somewhere in Sage that contains the output
of 'ptestlong' when it works. If a non-developer runs the doctest we
can just `cat output.txt` (all tests pass) so that he feels better.
That would save them a lot of time.

Nathann

[Sébastien Labbé]

Nice if a target for make allows to run test for src/sage/interfaces/* that are usable.

Another possibility is to add "make ptestnoextlong" for people who do not want to run tests in src/sage/interfaces/*  :P

[Jeroen Demeyer]

Sorry, I didn't phrase my question clearly enough, let me try again. I'm
honestly really interested in the answer, both from Sébastien and from
Nathann (and please do not change the question!)

> I don't really care myself about the mathematica interface. But it
> happens that mathematica 10.0 is installed on my machine in my lab and
> that most of the people in my lab use mathematica rather than sage. They
> won't test the mathematica doctests in the sage library since they are
> not sage developpers.

Let's assume the story ends here. You use Sage only and other people use

Mathematica only, on the same lab computer. You have never used
Mathematica and in fact do not care about it at all. As it happens, the
version of Mathematica installed is a very old outdated version which is
not compatible with the newer Sage interface.

Now every time you run "make ptestlong" you will see doctest failures
related to Mathematica. Those failures cannot really be fixed, since the
features that the doctests need are not available in the old version of
Mathematica on your machine.

Would you be bothered by the fact that you will always see those doctest
failures that you cannot get rid of, just because somebody installed an
outdated version of Mathematica on that machine?

Jeroen.

[Nathann Cohen]

> Now every time you run "make ptestlong" you will see doctest failures
> related to Mathematica. Those failures cannot really be fixed, since the
> features that the doctests need are not available in the old version of
> Mathematica on your machine.

If your problem is with an ancient version of Mathematica which we
know is not supported by Sage, then we must detect that and not run
the optional mathematica tests (because it is not supported).

And once this is done, there will be two nice outcomes:
- Whoever runs the doctests with an ancient mathematica version will
not automatically run those tests (because the version is too old)
- Whoever runs 'mathematica(1)' with an unsupported version of
mathematica will see an exception saying "Your version of mathematica
is not supported".

> Would you be bothered by the fact that you will always see those doctest
> failures that you cannot get rid of, just because somebody installed an
> outdated version of Mathematica on that machine?

So that this does not apply anymore.

Nathann

[Sébastien Labbé]

Would you be bothered by the fact that you will always see those doctest
failures that you cannot get rid of, just because somebody installed an
outdated version of Mathematica on that machine?

Jeroen.

My answer is : Yes I will be bothered.

So maybe the system has to know with which version the doctests were written... Ok, well, maybe mathematica changes so much at every release that it is too complicated to support the doctests for it...

Sébastien

[Jeroen Demeyer]

What if mathematica(1) or mathematica('1+1') works on the old version
but something more complicated does not work?

[Nathann Cohen]

> What if mathematica(1) or mathematica('1+1') works on the old version but
> something more complicated does not work?

How challenging can it be to figure out the version of mathematica?

mathematica(1) is just the way I used to detect whether the software
is installed. If we need something more reliable, we can have a
'is_mathematica_available()' function somewhere that checks it
properly.

Nathann

[Dima Pasechnik]

On Thursday, 16 July 2015 15:20:18 UTC+1, Nathann Cohen wrote:

> What if mathematica(1) or mathematica('1+1') works on the old version but
> something more complicated does not work?

How challenging can it be to figure out the version of mathematica?

sage: mathematica('$VersionNumber').sage()
9.0
 

[Sébastien Labbé]

On Thursday, July 16, 2015 at 4:17:10 PM UTC+2, Jeroen Demeyer wrote:

What if mathematica(1) or mathematica('1+1') works on the old version
but something more complicated does not work?

Sage supports only one version of open source and free optional sage packages, since users can obtain easily that version freely and this one is declared officially in the file /build/pkgs/maxima/package-version.txt for example. For external packages that are not free like mathematica, it not so easy to ask the user to upgrade to the most recent or to the one is used by Sage. So, we have to make a choice. Either we try to support all mathematica versions. Or, we support only one. But if we chose to support one version. Then it should be declared that the interface is for mathematica v9.0 for instance maybe in some package-version.txt file. And if I have mathematica v 10.0 on my machine, then "make ptestlong" should not run the mathematica optional since the version do not match. But if I have the good version, then, why not "make ptestlong" run those tests?

[Nathann Cohen]

Small update: I am done working on this patch. Reviewers tend to abuse their position by imposing their ideas and force the author to implement them, taking the ticket as a hostage.

Whoever wants this automatically testing to work can continue the work on the ticket:

http://trac.sagemath.org/ticket/18904#comment:58

Have fun,

Nathann

On Wednesday, July 15, 2015 at 11:43:45 AM UTC+2, Nathann Cohen wrote:

Hello everybody,

Nowadays when running 'sage -t', one automatically runs all tests
flagged with '#optional - X' for all installed packages X which are
[optional+new_style].

But many doctests are flagged with '#optional - X' where X is not even
a package, e.g. when X is a proprietary software that we are not
allowed to redistribute as a package.

What packages of this kind should we test? How can we detect if it is
locally available?

I created ticket #18904 [1] to auto-detect CPLEX and Gurobi
(proprietary LP solvers), but if you have more in mind please add a
commit so that they are added to the list.

THaaaaaaaaaaanks,

Nathann

[1] http://trac.sagemath.org/ticket/18904

[Julien Puydt]

Hi,

Here is another idea : don't put the interface to the external package
in sage-the-library, but in an optional package.

That way, doctests are only run for people who care. That, and it would
be possible to have an optional package for foo17 and another for
foo21... hence supporting incompatible versions.

Snark on #sagemath

[Dima Pasechnik]

On Friday, 17 July 2015 11:40:04 UTC+1, Snark wrote:

Hi,

Le 16/07/2015 13:16, Jeroen Demeyer a écrit :
> On 2015-07-16 13:03, Nathann Cohen wrote:
>> What is somebody has such an
>> install and *wants* to use it with Sage?
> This points to the core problem: we simply have no way to guess whether
> people want to do that or not.
>
> Here is a possible compromise proposal:
> * keep default optional package as they are now
> * add a new flag --optional=external (or change --optional=all to do
> this) to test all auto-detected-but-not-part-of-Sage packages like
> mathematica.
> * add a corresponding Makefile target running tests with this flag.
>

Here is another idea : don't put the interface to the external package
in sage-the-library, but in an optional package.

this presently means that such packages become upstream w.r.t. Sage.
An extra burden to maintain etc...

[Julien Puydt]

Hi,

Le 17/07/2015 12:59, Dima Pasechnik a écrit :
>
> On Friday, 17 July 2015 11:40:04 UTC+1, Snark wrote:
>
> Hi,
>
> Le 16/07/2015 13:16, Jeroen Demeyer a écrit :
> > On 2015-07-16 13:03, Nathann Cohen wrote:
> >> What is somebody has such an
> >> install and *wants* to use it with Sage?
> > This points to the core problem: we simply have no way to guess
> whether
> > people want to do that or not.
> >
> > Here is a possible compromise proposal:
> > * keep default optional package as they are now
> > * add a new flag --optional=external (or change --optional=all to do
> > this) to test all auto-detected-but-not-part-of-Sage packages like
> > mathematica.
> > * add a corresponding Makefile target running tests with this flag.
> >
>
> Here is another idea : don't put the interface to the external package
> in sage-the-library, but in an optional package.
>
>
> this presently means that such packages become upstream w.r.t. Sage.
> An extra burden to maintain etc...

I'm not sure I understand how it can be an extra burden, and I have no
clue about the caetera...

Snark on #sagemath

[Jeroen Demeyer]

On 2015-07-17 13:15, Julien Puydt wrote:
> I'm not sure I understand how it can be an extra burden

The problem is not so much with additional packages, it's with
additional packages which only exist for Sage. If there is an active
upstream which would exist anyway without Sage, that's healthy.

However, "upstream" packages which are really part of Sage are a
maintainance burden. I think that SageNB is a very good example of this:
part of the reason that so little SageNB development happened recently
is because it is a separate package. (I know there are other reasons
too, but that's not the point here)

Jeroen.

[Nathann Cohen]

> I'm not sure I understand how it can be an extra burden, and I have no clue
> about the caetera...

Writing the interface code in an optional package means:
- The interface code is no longer part of 'standard sage'. It is
better then to host is somewhere, on some publicly available
repository (with some extra code administration to do there)
- The interface not being part of standard sage, its documentation is
not part of our online "official" reference manual, so people will not
find it easily
- Every single update of the interface code results in an update of
the package we ship
- The history of changes to the interface does not appear in our git history

Nathann