Some work on Minimal Sage in Gentoo
Mark
I've wanted to make a "minimal sage" for some time. I finally got
motivated to do some work on it. Don't ask what I _should_ have been
doing.
I'm working with gentoo linux ... it has nice package management and
many of the things I needed were included already. I'll make as many
comments as I can to help other reconstruct what I did.
Essentially, I did these steps:
1) Extracted sage-2.3 from the overall sage distribution.
tar xf sage-2.3 # the top level distro
cd sage-2.3/spkg/standard
tar xf sage-2.3 # the specific sage code
mv sage-2.3 workdir
2) Use a different setup.py and a edit one small thing
cd workdir
# add marksetup.py to workdir
cp other/location/marksetup.py ./
# fix one path problem
sed -e 's:../../../:../../:' sage/libs/cf/cf.pyxe > tmpfile
mv tmpfile sage/libs/cf/cf.pyxe
3) Build the csage stuff by hand and make it available
<<this library build isn't quite right>>
cd sage-2.3/spkg/standard
tar xf sage_c_lib-2.3.spkg
cd sage_c_lib-2.3/src
gcc -fPIC -g -c *.c -I/usr/include/python2.4
gcc -shared -Wl,-soname,libcsage.so -o libcsage.so *.o
ln -s libcsage.so csage.so
cp *.so workdir/mylibs/
cp *.h workdir/myinc/
4) Make sure the following programs are installed via portage
(gentoo's package manager):
<< you might need more ... i already had a bunch of stuff installed>>
<< you might have to unmask some of these as well (a gentoo thing)>>
maxima 5.9.3
gsl 1.8
ntl 5.4-r1
singular 3.0.2.1 (i used my own ebuild, but the built in should be ok)
pexpect .999
blas-atlas-3.7.24
mpfr-2.2.1_p1
numpy-1.0.1-r1
5) use my shoddy ebuilds for these:
<< you will need to fish out singular-3.0.2-gentoo.diff for factory
to use the ebuild I have here>>
pari 2.3.1 (simply a version bump from 2.3.0 in portage)
factory 3.0.2.1 (from singular)
iml-1.0.1
mpfi-1.3.4
mwrank-061003
6) Build it:
python marksetup.py build
7) Figure out everthing that built, but has run time errors!
<<to be determined>>
I haven't tested the produced code AT ALL. It may well be complete
junk.
I just wanted to get what I've done up so others can (maybe) take
advantage of it
and/or make progress on it.
Files (ebuilds and marksetup.py) can be found at:
http://www.cs.pitt.edu/~fenner/work/projects/minimal-sage/minimal-sage.tgz
Regards,
Mark
William Stein
> I've wanted to make a "minimal sage" for some time. I finally got
> motivated to do some work on it. Don't ask what I _should_ have been
> doing.
> I'm working with gentoo linux ... it has nice package management and
> many of the things I needed were included already. I'll make as many
> comments as I can to help other reconstruct what I did.
>
> Essentially, I did these steps:
>
> 1) Extracted sage-2.3 from the overall sage distribution.
>
> tar xf sage-2.3 # the top level distro
> cd sage-2.3/spkg/standard
> tar xf sage-2.3 # the specific sage code
> mv sage-2.3 workdir
>
> 2) Use a different setup.py and a edit one small thing
>
> cd workdir
> # add marksetup.py to workdir
> cp other/location/marksetup.py ./
> # fix one path problem
> sed -e 's:../../../:../../:' sage/libs/cf/cf.pyxe > tmpfile
> mv tmpfile sage/libs/cf/cf.pyxe
>
> 3) Build the csage stuff by hand and make it available
> <<this library build isn't quite right>>
I'm curious -- Is there something wrong with SAGE's csage build process?
> 4) Make sure the following programs are installed via portage
> (gentoo's package manager):
> << you might need more ... i already had a bunch of stuff installed>>
> << you might have to unmask some of these as well (a gentoo thing)>>
I just want to emphasize concern here. SAGE is not likely to work correctly
if you replace the components it uses by ones with different versions. In
many cases we program around or patch issues in specific versions of
the dependent programs, and in other cases we use new features that
were recently added. For example...
> maxima 5.9.3
In maxima, I think version 5.10, a lot of the special functions stuff was
reorganized -- there's no way SAGE will past its test suite with maxima
5.9.3. Moreover, SAGE currently ships with maxima 5.11 and assumes
it is available.
> gsl 1.8
SAGE ships with gsl 1.9. I don't remember if we had to change anything
going from 1.8 to 1.9 -- I think probably not, though there were some important
bug fixes in 1.9.
> ntl 5.4-r1
SAGE will work fine with that.
> singular 3.0.2.1 (i used my own ebuild, but the built in should be ok)
Probably this is fine...
> pexpect .999
SAGE uses pexpect version 2.0. The current official version of pexpect
is version 2.1, but SAGE will not work with pexpect 2.1 at all. I have no
idea if SAGE will work at all with pexpect .999.
> blas-atlas-3.7.24
Should be fine.
> mpfr-2.2.1_p1
Should be fine.
> numpy-1.0.1-r1
Should be fine.
> 5) use my shoddy ebuilds for these:
> << you will need to fish out singular-3.0.2-gentoo.diff for factory
> to use the ebuild I have here>>
>
> pari 2.3.1 (simply a version bump from 2.3.0 in portage)
SAGE should not work at all with an officially released PARI.
For SAGE we do some nontrivial patching to the library, e.g,.
replacing calls to exit() with calls to abort().
> factory 3.0.2.1 (from singular)
I think Martin A might have fixed a bug in factory that needed
to be fixed in order to use it from SAGE.
> iml-1.0.1
I made some minor changes to IML for using in SAGE,
but nothing that affects the interface. The changes were
to remove the atlas dependency. However, Martin A. implemented
arbitrary precision p-adic system solving for the IML included
in SAGE, whereas the official release version doesn't have that.
> mpfi-1.3.4
Carl Witty said he fixed several bugs in mpfi that were uncovered
in the course of wrapping it for use in SAGE.
> mwrank-061003
The official release should be fine.
> 6) Build it:
> python marksetup.py build
>
> 7) Figure out everthing that built, but has run time errors!
> <<to be determined>>
> I haven't tested the produced code AT ALL. It may well be complete
> junk.
> I just wanted to get what I've done up so others can (maybe) take
> advantage of it
> and/or make progress on it.
>
>
> Files (ebuilds and marksetup.py) can be found at:
> http://www.cs.pitt.edu/~fenner/work/projects/minimal-sage/minimal-sage.tgz
Just out of curiosity, how difficult would it be to create a non-minimal normal
SAGE ebuild? This would likely actually work perfectly, and hence would
get a lot of use. Bobby Moretti did this with a Debian package at one point,
and it was pretty impressive and would have been really useful had somebody
volunteered to maintain it.
William
Mark
Well, I had a nice reply typed and ... and I lost it. Typical. I'll
try again ... but it won't be nearly as witty.
Gentoo has the following higher versions available (then what I used):
pexpect-2.1 (not sure why I only read .999)
maxima-5.11.0
gsl-1.9
I forgot to mention that there is also a pyrexembed ebuild in my file
tarball.
On the issue of a batteries included sage for gentoo: this is
definitely doable. In fact, I could the following roadmap (note,
portage is the gentoo package build and management system):
1) Simple ebuild that just runs the sage build process and install
all of the sage stuff out of the way (/opt/sage). The difficult
step would be divorcing the build time location from the run time
location (b/c portage does a build to a dummy location and then
copies it to the live filesystem).
2) Individual ebuilds for each sage specific package that install
into
(/opt/sage). The ebuilds would live in their own space (not the
gentoo main tree). They would also be renamed to prevent
conflicts with "live" packages.
Now the fun stuff:
3) Use as many "live" ebuilds as possible and as few custom
ebuilds/sources as possible. Install into the main tree when
possible but if we need a customized package for sage, we could
put it somewhere out of the way. At some point, we'd also add
ebuilds
for those packages that aren't in portage at all.
4) Use only live ebuilds. Have separate ebuilds for customized
packages (treat these as unique packages unrelated to the
mainstream version). Portage can install multiple versions of
packages but I'm not sure how general it is. In the worst case,
we could have separate subdirectories for different versions if
need be (/usr/lib/ntl-123 and /usr/lib/ntl-sage/ or so). Making
sure that there aren't conflicts between mainstream and customized
versions would be a pain.
Anyway, just some thoughts.
Regards,
Mark
Mark
Sorry. About the csage build process: I built by hand b/c the
automated process has the "whole nine yards" of autoconf tools with
it. I couldn't find the location for specifying Python.h and I
noticed there were only two source files and two headers. So, I
compiled by hand. The problem I referred to was in my hand building
process .... I don't think I built it 100% correctly.
Mark
William Stein
> pexpect-2.1 (not sure why I only read .999)
Do not use pexpect-2.1. It is very broken.
> maxima-5.11.0
> gsl-1.9
Good.
> On the issue of a batteries included sage for gentoo: this is
> definitely doable. In fact, I could the following roadmap (note,
> portage is the gentoo package build and management system):
>
> 1) Simple ebuild that just runs the sage build process and install
> all of the sage stuff out of the way (/opt/sage). The difficult
> step would be divorcing the build time location from the run time
> location (b/c portage does a build to a dummy location and then
> copies it to the live filesystem).
One option is to just copy the build directory over to the run-time
location at the end of the build, i.e., as make install. This should work
if you run SAGE once from the target location. Let me know when
you succeed at doing this. It would be a significant step forward
for SAGE.
> 2) Individual ebuilds for each sage specific package that install
> into
> (/opt/sage). The ebuilds would live in their own space (not the
> gentoo main tree). They would also be renamed to prevent
> conflicts with "live" packages.
OK.
> Now the fun stuff:
>
> 3) Use as many "live" ebuilds as possible and as few custom
> ebuilds/sources as possible. Install into the main tree when
> possible but if we need a customized package for sage, we could
> put it somewhere out of the way. At some point, we'd also add
> ebuilds
> for those packages that aren't in portage at all.
Makes sense. Note that you really _must_ look in the SAGE.txt
files and spkg-install files in each .spkg to see if any patches were
needed for SAGE, and the same should be done when a package
gets upgraded.
Mark
**********************************************************************************
> 1) Simple ebuild that just runs the sage build process and
install
> all of the sage stuff out of the way (/opt/sage). The
difficult
> step would be divorcing the build time location from the run
time
> location (b/c portage does a build to a dummy location and
then
> copies it to the live filesystem).
One option is to just copy the build directory over to the run-time
location at the end of the build, i.e., as make install. This should
work
if you run SAGE once from the target location. Let me know when
you succeed at doing this. It would be a significant step forward
for SAGE.
**********************************************************************************
So, I'm not sure how easy this is ... for example, when you build
sage, you set SAGE_ROOT and this is used throughout the build process
and the runtime usage (right? I could well be wrong). This coupling
of the build time and run time environments is my biggest concern.
Now, in gentoo, the builds are all done in /var/tmp/portage/
packagename/work. Then, make install is pointed to (with either --
prefix and/or DESTDIR) /var/tmp/portage/packagename/image. Under
image, there might be image/usr image/share/doc image/include and so
forth. For what I'm talking about using /opt for sage, we'd have
image/opt/sage/<all the sage stuff>. This is done in a "sandbox" that
prevents writing anywhere else in the system. So, any hardcoded paths
will pose a problem (everything has to be relative to --prefix and/or
DESTDIR). Finally, this gets copied into the live filesystem: for
us, this would be into /opt/sage/. And this point, we have to make
sure all the references in sage are relative to /opt/sage ... NOT to /
var/tmp/portage/sage/image/opt/sage.
Regards,
Mark
James McCaw
I have done a similar effort for debian some time back (greater than 6
months). An alioth pkg-sage project
(http://alioth.debian.org/projects/pkg-sage/) was even set up to begin
a real attempt at packaging up sage for debian. Unfortunately, the
alioth build machine crashed and the small amount of work that had
been done was lost.
Anyway, I've attached my relatively detailed listing of how to go
about preparing sage for debianization - at the moment, it is a true
hack. The main packaging issue is similar to Mark's comment just above
- the linking of build directory to run time directory. This is most
problematic for .spkg's that install python files. The solution I had
was to patch the spkg-install files to change "python setup.py
install" to ""python setup.py install --install-lib <appropriate
relative path for debian installation>"
where for me that appropriate path was
"../../debian/python-sage/usr/lib/python2.4/site-packages"
I believe it would be easy enough to modify the spkg-install script to
have this appropriate relative directory passed in from the main
install script. William, would that be possible?
Anyway, any successful build of sage on debian will also have the
problems with un-sage-patched versions of some software so runtime
issues will/may occur. But, if sage was to ever be incorporated into
debian proper (rather than existing as a package installed from
outside of debian) it would need to rely mainly on existing debian
packages. Maybe only those that are truly forked from upstream (pari
is a good example) would remain included with sage. All generic stuff
(bzip, readline etc) would be pulled from the sage build process and
package.
If anyone is interested in signing up to the debian sage-package
project just register at the alioth link above. I'll make people
administrators so that the alioth webpage can be modified, files
added, and progress made!
Great work on the gentoo packaging effort Mark. I suspect that any
future efforts from either gentoo or debian packaging efforts will be
of mutual benefit to each other and sage upstream.
Cheers
James McCaw
--
James McCaw
james...@gmail.com
William Stein
> > 1) Simple ebuild that just runs the sage build process and
> install
> > all of the sage stuff out of the way (/opt/sage). The
> difficult
> > step would be divorcing the build time location from the run
> time
> > location (b/c portage does a build to a dummy location and
> then
> > copies it to the live filesystem).
>
> One option is to just copy the build directory over to the run-time
> location at the end of the build, i.e., as make install. This should
> work
> if you run SAGE once from the target location. Let me know when
> you succeed at doing this. It would be a significant step forward
> for SAGE.
> ***************************************************************************
>*******
There's also a "make install" option, after you complete the build.
You have to set the environment variable DESTDIR to the target directory,
then type "make install". We needed this for the Debian package.
> So, I'm not sure how easy this is ... for example, when you build
> sage, you set SAGE_ROOT and this is used throughout the build process
> and the runtime usage (right? I could well be wrong). This coupling
> of the build time and run time environments is my biggest concern.
It's not coupled so much. The only coupling is that the file
SAGE_ROOT/sage has the location sage is installed in listed near
the top. That file then sets the environment variable SAGE_ROOT,
which is then used by the rest of SAGE.
> Now, in gentoo, the builds are all done in /var/tmp/portage/
> packagename/work. Then, make install is pointed to (with either --
> prefix and/or DESTDIR) /var/tmp/portage/packagename/image. Under
> image, there might be image/usr image/share/doc image/include and so
> forth. For what I'm talking about using /opt for sage, we'd have
> image/opt/sage/<all the sage stuff>. This is done in a "sandbox" that
> prevents writing anywhere else in the system. So, any hardcoded paths
> will pose a problem (everything has to be relative to --prefix and/or
> DESTDIR). Finally, this gets copied into the live filesystem: for
> us, this would be into /opt/sage/. And this point, we have to make
> sure all the references in sage are relative to /opt/sage ... NOT to /
> var/tmp/portage/sage/image/opt/sage.
One can build SAGE anywhere, move it anywhere else, and run it once,
and it will adapt to being in the new location. (Spaces in the path
are dangerous though...)
-- William
Mark
> There's also a "make install" option, after you complete the build.
> You have to set the environment variable DESTDIR to the target directory,
> then type "make install". We needed this for the Debian package.
>
> It's not coupled so much. The only coupling is that the file
> SAGE_ROOT/sage has the location sage is installed in listed near
> the top. That file then sets the environment variable SAGE_ROOT,
> which is then used by the rest of SAGE.
>
> One can build SAGE anywhere, move it anywhere else, and run it once,
> and it will adapt to being in the new location. (Spaces in the path
> are dangerous though...)
>
> -- William
William,
This is all great news. I didn't realize sage had been modified to
respect DESTDIR. If that is the case, all is good. I'm doing a
compile on my machine to check on a few things, but essentially, we
should be able to make a simple ebuild that will:
1) download sage source distro
2) run make
3) run make install to portages image directory
4) portage will copy sage to /opt/sage
5) in the post install section of the ebuild, we will call sage once
to start it up and let it fix its current path
Presto! We have sage installed and managed by portage (of course
we're not using much of the management). Once I verify it compiles on
my system (which it should), I'll put that ebuild together and give it
a whirl.
Regards,
Mark
Mark
compliance with gentoo quality control standards, etc. etc. It puts
sage under /opt/sage/ as listed in the top-level sage makefile.
Gentoo complains of a number of "fast and loose" things going on in
some of the compiled code. Some of these things might be necessary
(for sage) even if gentoo complains about them. I'll put the log up
later (my dept. server seems to be done ATM).
******************************************************************************************************
# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
DESCRIPTION="Software for Algebra and Geometry Experimentation"
HOMEPAGE="http://modular.math.washington.edu/sage/"
SRC_URI="http://modular.math.washington.edu/sage/dist/src/${P}.tar"
LICENSE=""
SLOT="0"
KEYWORDS="x86"
IUSE=""
DEPEND=""
RDEPEND="${DEPEND}"
#S="${WORKDIR}/${P}"
src_compile() {
emake || die "emake failed"
sed -e 's:\(^SAGE_ROOT=\).*:\1/opt/sage:' sage > tmpfile
mv tmpfile sage
}
src_install() {
emake DESTDIR="${D}"/opt/sage install || die "emake install
failed"
}