Optional and experimental packages

[John H Palmieri]

With a Python 3 build of Sage on OS X 10.14.6, I decided to install as many optional and experimental packages as I could. The results:

Optional:

- the following packages failed to build, and the reason wasn't completely obvious:

awali
buckygen
cbc
gambit
gdb

mpi4py

- the following packages failed because they (or their installation scripts) are not compatible with Python 3:

beautifulsoup
brian
guppy
mercurial
p_group_cohomology (but work is in progress)
pyx
scons
trac

- the following packages failed at first, but built after installing some prerequisities:

deformation — requires installation of mpir
dot2tex — requires Graphviz
rst2ipynb — requires pandoc

- I skipped the following packages:

atlas (installation is skipped on OS X)

python2 (I wanted to use a pure Python 3 build)

- Every other optional package built.

Question/Proposal: do we demote the failed packages to experimental? (Not deformation, dot2tex, or rst2ipynb, also not p_group_cohomology because it is just about ready for py3, but the others.) I plan to do this unless there are objections.

Experimental:

- the following failed to build:

autotools
cocoalib
libtheora
polymake
qepcad
scipoptsuite
surf
valgrind

- the following succeeded:

gap3
lie
modular_decomposition
perl_term_readline_gnu

polymake succeeded but only after installing the perl JSON package. (Why is jupymake optional and polymake experimental? jupymake has polymake as a dependency, and optional packages should not depend on experimental packages.)

- I skipped the following:

compilerwrapper — when I install this, it breaks the Sage library: after touching any .pyx file, "sage -b" or "make" fails with an error about ld.

Doctesting:

Then I ran `make ptestalllong`. There were lots of "internet" failures. Otherwise:

- failures in one file because of rst2ipynb

- failures in two files because of dot2tex

- failures in one file because of latex

- failures in one file because of gap_packages (reported by Emmanuel Charpentier on sage-release many times)

Other than the internet problems, not too bad.

--

John

[John H Palmieri]

Oops, I spoke too soon. Two more files had failures:

- sage -t --long src/sage/databases/stein_watkins.py  # 20 doctests failed
- sage -t --long src/sage/interfaces/frobby.py  # 12 doctests failed

[Samuel Lelievre]

Mon 2019-10-28 22:02:23 UTC, John H Palmieri:

With a Python 3 build of Sage on OS X 10.14.6, I decided to install

as many optional and experimental packages as I could. The results:

Thanks John for trying this and reporting!

 

Optional:

- the following packages failed to build, and the reason wasn't completely obvious:

awali
buckygen
cbc
gambit
gdb

mpi4py

- the following packages failed because they (or their installation scripts) are not compatible with Python 3:

beautifulsoup

I think we should either remove BeautifulSoup

or replace it by beautifulsoup4:

https://pypi.org/project/beautifulsoup4/

 

brian
guppy
mercurial

Sage-wise, I believe Mercurial is a remnant of the time

before we switched to Git. I think we should remove it.

I think the dependency on the perl json package

should be better documented if needed,

and polymake should be made optional.

[Thierry]

Hi,

On Mon, Oct 28, 2019 at 04:37:53PM -0700, Samuel Lelievre wrote:
[...]

> >
> > - the following packages failed because they (or their installation
> > scripts) are not compatible with Python 3:
> >
> > beautifulsoup
> >
>
> I think we should either remove BeautifulSoup
> or replace it by beautifulsoup4:
> https://pypi.org/project/beautifulsoup4/
>
>
> > brian
> > guppy
> > mercurial
> >
>
> Sage-wise, I believe Mercurial is a remnant of the time
> before we switched to Git. I think we should remove it.

Perhaps should we simply remove the pseudo-packages whose type is "pip"
(since there is the "sage -pip install" command), that is:

beautifulsoup
biopython
brian
guppy
mercurial
mpi4py
nibabel
pybtex
pyflakes
sqlalchemy
trac

Ciao,
Thierry

[Nico Van Cleemput]

buckygen is a pure C package, so I doubt that this has anything to do with the switch to Python 3. Do you have any more information about the fail build, because here it built fine.

Nico

Op ma 28 okt. 2019 om 23:02 schreef John H Palmieri <jhpalm...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/150bd508-b9b2-4953-aa50-cb27cc840b78%40googlegroups.com.

[Dima Pasechnik]

clang is unhappy about C standards violations. E.g. this is what I get
with clang 7:

cc -O4 buckygen.c -o buckygen
cc: warning: -O4 is equivalent to -O3 [-Wdeprecated]
In file included from buckygen.c:272:
./splay.c:139:6: warning: implicit declaration of function
'outputnode' is invalid in C99 [-Wimplicit-function-declaration]
ACTION(p);
^
buckygen.c:254:19: note: expanded from macro 'ACTION'
#define ACTION(p) outputnode(p)
^
In file included from buckygen.c:272:
./splay.c:263:15: warning: implicit declaration of function
'comparenodes' is invalid in C99 [-Wimplicit-function-declaration]
cmp = COMPARE(p);
^
buckygen.c:260:20: note: expanded from macro 'COMPARE'
#define COMPARE(p) comparenodes(canong, codelength, type, p)
^
In file included from buckygen.c:272:
./splay.c:328:15: warning: implicit declaration of function
'comparenodes' is invalid in C99 [-Wimplicit-function-declaration]
cmp = COMPARE(p);
^
buckygen.c:260:20: note: expanded from macro 'COMPARE'
#define COMPARE(p) comparenodes(canong, codelength, type, p)
^
In file included from buckygen.c:272:
./splay.c:352:20: error: non-void function 'splay_delete' should
return a value [-Wreturn-type]
if (p == NULL) return;
^
./splay.c:366:2: error: non-void function 'splay_delete' should return
a value [-Wreturn-type]
return;
^
./splay.c:376:9: error: non-void function 'splay_delete' should return
a value [-Wreturn-type]
return;
^
buckygen.c:943:1: warning: type specifier missing, defaults to 'int'
[-Wimplicit-int]
outputnode(SPLAYNODE *liste)
^
4 warnings and 3 errors generated.
make: *** [makefile:12: buckygen] Error 1

Easy to fix, I'd say - can this be done upstream?

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CADXCEk9nCeroEgL2aTP%3D-sBL%3DG6GGAiuwRjyiZDiOY_5oA3iHw%40mail.gmail.com.

[Nico Van Cleemput]

Well, upstream sits at another desk in my office, so I will ask when he gets in.

Nico

Op di 29 okt. 2019 09:29 schreef Dima Pasechnik <dim...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq1s%2BQPD6yim8c2FNo2xd72-A_zM5AE_V2PBeKA_YEJR4w%40mail.gmail.com.

[Nico Van Cleemput]

OK, the answer is below:

The file splay.c is actually taken from Nauty, so that issue should be resolved there. The most obvious solution is to not compile this as C99, but as C89 since that is what buckygen was written in.

So, I can mail this to Brendan McKay to ask to solve this, but since Nauty is already in Sage the issue should already be resolved in that code, so we could also just take splay.c from there.

Nico

Op di 29 okt. 2019 om 09:31 schreef Nico Van Cleemput <nico.van...@gmail.com>:

[kcrisman]

>
> Sage-wise, I believe Mercurial is a remnant of the time
> before we switched to Git. I think we should remove it.

I'm surprised it hadn't been long before.  Even I would approve that.

 

Perhaps should we simply remove the pseudo-packages whose type is "pip"
(since there is the "sage -pip install" command), that is:

beautifulsoup
biopython
brian
guppy
mercurial
mpi4py
nibabel
pybtex
pyflakes
sqlalchemy
trac

As long as we can test that this works properly with "our" Python and make sure to update any documentation, that would be okay.  brian for instance probably doesn't need too much doc change.  beautifulsoup was (is?) used for sagenb -> rst conversion, if I recall correctly, and if it isn't used elsewhere would probably be reasonable to deprecate, much as one might mourn the loss of sagenb in the near future.  (Though there may still be some places in some tutorials where this is mentioned, beautifulsoup I mean.) 

[Samuel Lelièvre]

Tue 2019-10-29 11:57 UTC, kcrisman:

>
>> Perhaps should we simply remove the pseudo-packages whose type is "pip"
>> (since there is the "sage -pip install" command), that is:
>>
>> beautifulsoup
>> biopython
>> brian
>> guppy
>> mercurial
>> mpi4py
>> nibabel
>> pybtex
>> pyflakes
>> sqlalchemy
>> trac
>
>
> As long as we can test that this works properly with "our" Python
> and make sure to update any documentation, that would be okay.
> brian for instance probably doesn't need too much doc change.

> beautifulsoup was (is?) used for sagenb -> rst conversion,
> if I recall correctly, and if it isn't used elsewhere would probably be
> reasonable to deprecate, much as one might mourn the loss of
> sagenb in the near future. (Though there may still be some places
> in some tutorials where this is mentioned, beautifulsoup I mean.)

We should keep the ability to export sagenb ".sws" worksheets
to rst and to ipynb for a long time. There are many, really many
people still using SageNB, and many, really many sws worksheets
out there.

[kcrisman]

That is very good news (to me, though probably not to most) but then we may need to keep the older beautifulsoup package unless someone can test robustly whether the new one has similar enough of an API to work with this.

[Dima Pasechnik]

I think the Sage tutorial mentioning beautifulsoup actually says to install it as

sage --pip install beatifulsoup4

which would install the package from the net rather than Sage's one 

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/161d6918-5dd0-44de-996c-95eb0111c709%40googlegroups.com.

[Samuel Lelièvre]

Le jeu. 31 oct. 2019 à 08:09, kcrisman:

>
> On Wednesday, October 30, 2019 at 1:47:48 AM UTC-4, Samuel Lelievre wrote:
>>
>> We should keep the ability to export sagenb ".sws" worksheets
>> to rst and to ipynb for a long time. There are many, really many
>> people still using SageNB, and many, really many sws worksheets
>> out there.
>
> That is very good news (to me, though probably not to most)

I realise my phrasing was ambiguous. In what I wrote, the sentence
"We should keep the ability to ... for a long time" was meant as
"My suggestion would be to keep the ability to ... for a long time".
Reading your answer, I realise it instead may have sounded as
"Don't worry, very likely we will keep the ability to ... for a long time."

[kcrisman]

No, I interpreted it more as "sagenb was a successful product". 

[kcrisman]

On Thursday, October 31, 2019 at 8:16:55 AM UTC-4, Dima Pasechnik wrote:

I think the Sage tutorial mentioning beautifulsoup actually says to install it as

sage --pip install beatifulsoup4

which would install the package from the net rather than Sage's one 

Great, perhaps that was updated at some point. 

[jplab]

Hi,

Le lundi 28 octobre 2019 23:02:23 UTC+1, John H Palmieri a écrit :

Experimental:

- the following failed to build:

autotools
cocoalib
libtheora
polymake
qepcad
scipoptsuite
surf
valgrind

- the following succeeded:

gap3
lie
modular_decomposition
perl_term_readline_gnu

polymake succeeded but only after installing the perl JSON package. (Why is jupymake optional and polymake experimental? jupymake has polymake as a dependency, and optional packages should not depend on experimental packages.)

This problem should be addressed here: https://trac.sagemath.org/ticket/27763 .

I do not know how jupymake ended up being optional while it depends on an experimental package. Nevertheless, I know that there was the idea of making polymake optional now that it has a more stable installation and portability.

See the ticket: https://trac.sagemath.org/ticket/22710 for more information about the development of polymake package and interface.

[John H Palmieri]

Note that "sage --sws2rst ..." relies on sagenb, and therefore will only work with Sage built with Python 2.

In any case, https://trac.sagemath.org/ticket/28685 upgrades to beautifulsoup4, which is compatible with Python 3.

[John H Palmieri]

Two more tickets:

https://trac.sagemath.org/ticket/28686: change a bunch of packages from optional to experimental, delete a few pip packages (as Samuel suggested). I didn't delete all of the pip packages, just the ones that are not compatible with Python 3.

https://trac.sagemath.org/ticket/28687: change scons from optional to experimental. I thought briefly of trying to upgrade it instead, but I don't really care about the package, so someone else who is actually interested will have to do that.

  John

[David Coudert]

File splay.c has changed in nauty (at least in 26r12). The main difference is 

- static SPLAYNODE* 

- SPLAY_DELETE(SPLAYNODE **to_root, SPLAYNODE *p)

+ void

+ SPLAY_DELETE(SPLAYNODE **to_root, SPLAYNODE *p)

With this change, I can do gcc -O3 -std=c89 buckygen.c -o buckygen

So upstream must be updated.

Le mardi 29 octobre 2019 11:56:12 UTC+1, nvcleemp a écrit :

OK, the answer is below:

The file splay.c is actually taken from Nauty, so that issue should be resolved there. The most obvious solution is to not compile this as C99, but as C89 since that is what buckygen was written in.

So, I can mail this to Brendan McKay to ask to solve this, but since Nauty is already in Sage the issue should already be resolved in that code, so we could also just take splay.c from there.

Nico

Op di 29 okt. 2019 om 09:31 schreef Nico Van Cleemput <nico.va...@gmail.com>:

>> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

>> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/150bd508-b9b2-4953-aa50-cb27cc840b78%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CADXCEk9nCeroEgL2aTP%3D-sBL%3DG6GGAiuwRjyiZDiOY_5oA3iHw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Nico Van Cleemput]

Yes, I was actually also looking at it just this morning. It's been a while since I did any Sage development, but I'll try to fix this. Will first have to reread how the packages work these days.

Nico

Op za 2 nov. 2019 om 10:50 schreef David Coudert <David....@inria.fr>:

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/2c5288ea-4b93-4652-a935-27cbece55d94%40googlegroups.com.

[Dima Pasechnik]

I have opened #28688 to fix buckygen.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CADXCEk-A8kojWXLBM98CrBeAkYh-uQF-4G%2Bjf5SE6LKOGQKAog%40mail.gmail.com.

[Dima Pasechnik]

has upstream updated the package?

On Sat, 2 Nov 2019, 11:58 Nico Van Cleemput, <nico.van...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CADXCEk-A8kojWXLBM98CrBeAkYh-uQF-4G%2Bjf5SE6LKOGQKAog%40mail.gmail.com.