sage & gentoo prefix

[Burcin Erocal]

Hi Francois,

let's move the discussion to sage-devel.

For reference: We have been talking about replacing the sage build
system with the gentoo package manager and ebuilds via gentoo prefix:

https://github.com/cschwan/sage-on-gentoo/issues/52#issuecomment-982600

Francois Bissey wrote:
> We shouldn't continue the discussion here but at least its documented. Yes
> you can relocate a prefix on some platform (I am not sure you can easily on
> aix but we are not too concerned by that).
>
> But I think if we want to replace the spkg system by ebuilds we could go some
> other ways.
>
> - using host gcc with a set of flags in a special sage profile that would
> mimic some of the set up in numerous spkg. - use Volker's gcc wrapper to use
> the host gcc and create a relative prefix instead of fixed one. I think we
> have a fixed prefix because a relative one would be hard to set up on some
> platforms.

When I saw Volker's message, I also started thinking of using these wrappers
instead of gcc. Is there a virtual/compiler etc. we can provide in a gccwrapper
ebuild to make this work seemlessly? Or do we still need to add gcc to
packages.provided?

> The main difficulty becomes the bootstraping we need to get a working
> prefixed python in a seamless fashion.

+1. I will try to make a tarball which handles bootstrapping a sage-prefix. :)

I don't have much experience with gentoo-prefix yet, so it might take a while
before I get a handle on things.

> There are other issues. The current overlay provides an alternate way of
> building sage already but it is not as lightweight as you'd like.

This is something that needs to be changed "upstream", in Sage, anyway. There
were comments on sage-devel about changing the way the library is built so
that it works without a separate build directory.

> Aside from that it is not as developer friendly as vanilla sage. You cannot
> apply a patch and do "sage -b". Effectively to apply a patch I change the
> ebuild and do the equivalent of "sage -ba". On the other hand upgrade always
> works :)

Can't we tell emerge to "keepwork" and allow people to "make install" from the
work copy? I guess this means overriding the sandbox setup somehow.

If we place the working copy in a more visible place, like ${EPREFIX}/devel,
this would match the current development model. This would also allow people
to easily develop other packages in Sage.

Cheers,
Burcin

[Keshav Kini]

Wow, this sounds really interesting. IMHO replacing our homegrown SPKG system with a more general and commonly used system would be great. Reusing code, and all that. It would also allow us to use the power of `emerge -C` to, say, uninstall SPKGs (though I'm not sure how exactly it works - looking at the Gentoo documentation about ebuilds, it doesn't seem like the ebuild file specifies exactly how to uninstall... does `emerge` just guess based on what the install script put into the system, what to take remove from it?) From your discussion on github, though, I take it we'd actually be forced to ship our own compilers and everything, because gentoo-prefix just works that way. Is that right? (After all, gentoo-prefix is supposed to basically be the install system for an OS distribution, not just a group of packages...) That doesn't seem so great. I think most people would agree that Sage is huge enough as it is...

Anyway, do keep us posted!

-Keshav

[cschwan]

On Apr 12, 4:24 pm, Keshav Kini <keshav.k...@gmail.com> wrote:
> Wow, this sounds really interesting. IMHO replacing our homegrown SPKG
> system with a more general and commonly used system would be great. Reusing
> code, and all that. It would also allow us to use the power of `emerge -C`
> to, say, uninstall SPKGs (though I'm not sure how exactly it works - looking
> at the Gentoo documentation about ebuilds, it doesn't seem like the ebuild
> file specifies exactly how to uninstall... does `emerge` just guess based on
> what the install script put into the system, what to take remove from it?)

Right! For example, I installed sage with:

emerge -va sage

which displays a list of packages (in that case sage and its
dependencies) that are about to install if I press enter ("-a" = ask),
how much will be downloaded (-v = verbose) and which options are used
(in Gentoo-speak: USE-flags, e.g. sage can be installed with/without
latex, examples and the testsuite).

Unistallation happens with "emerge -C sage" which removes only the
package specified - "emerge --depclean" can be used to safely remove
unused dependencies.

> From your discussion on github, though, I take it we'd actually be forced to
> ship our own compilers and everything, because gentoo-prefix just works that
> way. Is that right? (After all, gentoo-prefix is supposed to basically be
> the install system for an OS distribution, not just a group of packages...)

Yes, without work you are forced to install things such gcc, perl and
autotools and so on. But there are ways to circumvent this - I am
curious if this is feasible.

[Volker Braun]

Having a working Gentoo-prefix that one could use if the host system's toolchain is broken already would be very interesting. 

But when I played around with it I ran into some library problems. For example, gmp/mpir would be installed twice (host and gentoo prefix), potentially mutually incompatible versions. Sometimes configure scripts look in the wrong place, for example. Its no insurmountable problem, but I never managed to recompile the whole prefix with itself.

[Francois Bissey]

That would definitely be bugs. If you don't care about submitting bugs you can
always forward me your problems and I'll work on a solution and get it
reported in the appropriate place. prefix is definitely a work in progress and
there are rather nasty corners, especially when you stray off linux to other
platforms.

But you raised the gmp/mpir issue that I think should be mentioned.
In sage-on-gentoo we use gmp rather than mpir. It's a technical choice.
gmp and mpfr (and now mpc) are system packages required to build gcc.
Almost no packages in sage have ported to mpir, rather sage relies on mpir
installing compatibility (or fake) headers and libraries. This is a problem on
gentoo unless we ask the toolchain team to switch to mpir. Somehow I don't
see that happening soon unless upstream gcc itself makes it an option.
Porting all the stack of software to pure mpir is possible but then we'd also
need a mpir enabled mpfr separate from the system one... in short the
KISS solution is to stick to gmp.

I could provide a "small" tree that would let you build the system with mpir,
that could be an interesting experiment.

On the other hand if we go anywhere with the experiment of using a reduced
portage with gcc wrappers there would be no problems of collision between gmp
and mpir.

Francois

This email may be confidential and subject to legal privilege, it may
not reflect the views of the University of Canterbury, and it is not
guaranteed to be virus free. If you are not an intended recipient,
please notify the sender immediately and erase all copies of the message
and any attachments.

Please refer to http://www.canterbury.ac.nz/emaildisclaimer for more
information.

[Francois Bissey]

> Hi Francois,

> > Aside from that it is not as developer friendly as vanilla sage. You
> > cannot apply a patch and do "sage -b". Effectively to apply a patch I
> > change the ebuild and do the equivalent of "sage -ba". On the other hand
> > upgrade always works :)
>
> Can't we tell emerge to "keepwork" and allow people to "make install" from
> the work copy? I guess this means overriding the sandbox setup somehow.
>
> If we place the working copy in a more visible place, like
> ${EPREFIX}/devel, this would match the current development model. This
> would also allow people to easily develop other packages in Sage.
>

Hi Burcin,

it would be complicated and requires that we change the current setup.
We currently don't ship the hg tree at all. We disable -b -ba and -upgrade.
Reenabling any of these will negate the benefit of having a package manager.
That is you will get files that are not tracked anymore which may wreck
upgrades. Or you would have to be careful to always do your patching in a
clone remembering that the clone will be untracked from the portage point of
view.
Note that we already have a hg-less devel tree so that we can run the test
suite (USE=testsuite) but unlike normal sage it is not linked to python/site-
packages/sage.

Of course everything is a matter of choice between various trade-off so you
could have to give up something one way or another.

Francois

[Georg S. Weber]

> Hi Burcin,
>
> it would be complicated and requires that we change the current setup.
> We currently don't ship the hg tree at all. We disable -b -ba and -upgrade.
> Reenabling any of these will negate the benefit of having a package manager.
> That is you will get files that are not tracked anymore which may wreck
> upgrades. Or you would have to be careful to always do your patching in a
> clone remembering that the clone will be untracked from the portage point of
> view.
> Note that we already have a hg-less devel tree so that we can run the test
> suite (USE=testsuite) but unlike normal sage it is not linked to python/site-
> packages/sage.
>
> Of course everything is a matter of choice between various trade-off so you
> could have to give up something one way or another.
>
> Francois
>

Hi all,

thanks for this highly motivating thread! It seems there are a handful
people interested, and I'm happy to join in. My goal might seem at
first pretty moderate: Identify the mutual benefits of Sage and Gentoo
(especially Gentoo Prefix), and write this down somewhere at an easily
accessible place (e.g. the Sage Wiki, maybe as some "SEP" --- Sage
Enhancement Proposal). We should take great care that we do write down
and communicate the discussions, results, proposals agreed (at least
among the participants) upon, sample code/installations, links, and
such.

Although Gentoo Prefix is by far more mature than Sage (and so the
benefits for Sage are surely much greater, than possible synergies for
Gentoo Prefix), I think I can identify some points where Gentoo Prefix
might eventually profit, too. See e.g. the Gentoo (Alt) Prefix mailing
thread containing the following message (Fabian Groffen does know what
he talks about, but it would be good to have at least the hard facts
clearly documented, and easily accessible):

http://archives.gentoo.org/gentoo-alt/msg_3e0aa31ec42c917484e6b0f86ffa5f2a.xml

or the current lack of Gentoo Prefix "binary distributions" resp. the
need of external support to do the "very first start up" (since
bootstrapping ain't really for the faint-hearted --- I did that myself
successfully on Maemo5, OS X 10.4 and OS X 10.6, but I'm a developer,
not a "mere user"), see e.g.:

http://sourceforge.net/projects/prefix-launcher/

One last thing about "sage -clone, -b, -ba, br, ..." --- from my
memory about several discussions, the sage community regards this as a
"killer feature", i.e. that there is not some "sage" and then some
other, different "sage-dev" package/distribution, but only one. So the
hurdle for any user to gradually become a developer is made as low as
possible. I have some rough ideas how to make this possible "inside" a
standard distribution with also an eye on reliable package management,
i.e. not restricted to Gentoo/Gentoo Prefix, but also Debian and so
on. But these ideas would need thorough discussion.


Cheers,
Georg

[Burcin Erocal]

Hi,

After sleeping on it for quite a long time, I finally took the time to
prepare a prototype tarball:

http://sage.math.washington.edu/home/burcin/sage-prefix.tar

Despite the name, ATM it only installs Singular with it's dependencies.
Only linux platforms that already have python > 2.6 are expected to
work. This doesn't solve any of the problems mentioned before about
using gentoo-prefix for the Sage build system. It was merely an
exercise for me to see the scope of the problems. :)

Details of how it works are at the end of this message.

On Wed, 13 Apr 2011 05:34:48 -0700 (PDT)
"Georg S. Weber" <georg...@googlemail.com> wrote:

> Hi all,
>
> thanks for this highly motivating thread! It seems there are a handful
> people interested, and I'm happy to join in. My goal might seem at
> first pretty moderate: Identify the mutual benefits of Sage and Gentoo
> (especially Gentoo Prefix), and write this down somewhere at an easily
> accessible place (e.g. the Sage Wiki, maybe as some "SEP" --- Sage
> Enhancement Proposal). We should take great care that we do write down
> and communicate the discussions, results, proposals agreed (at least
> among the participants) upon, sample code/installations, links, and
> such.

Here is a list of pros & cons from my (Sage centered) point of view:

Pros:

- flexible, very configurable source based package manager, extends the
current system with:
- uninstall packages
- binary packages
- separation of build time and run time dependencies
- properly handles CFLAGS, etc.
- separates source, metadata, and build instructions so it is easier
to update packages

- many packages already available
- in the gentoo portage tree
[http://gentoo-portage.com/sci-mathematics]
- sage-on-gentoo overlay [https://github.com/cschwan/sage-on-gentoo]
- gentoo-science overlay
[http://git.overlays.gentoo.org/gitweb/?p=proj/sci.git;a=tree]

- makes it easier to create custom distributions, which include
different sets of default packages

- reduces optional/experimental package maintenance

- it is possible to specify what is already available on different
platforms (we don't need to distribute gfortran, ATLAS, etc. if not
needed)
See for example:
http://sage.math.washington.edu/home/burcin/portage/profiles/sage/linux/package.provided

Cons:

- portage has a lot of dependencies
- python, sed, bash, etc.
This makes bootstrapping nontrivial and makes the distribution bigger

- package creation process involves learning portage conventions
[http://devmanual.gentoo.org/index.html]

- still a lot of open ends
- easy development
- relocating an install
- automating maintenance of prefix portage tree

> Although Gentoo Prefix is by far more mature than Sage (and so the
> benefits for Sage are surely much greater, than possible synergies for
> Gentoo Prefix), I think I can identify some points where Gentoo Prefix
> might eventually profit, too. See e.g. the Gentoo (Alt) Prefix mailing
> thread containing the following message (Fabian Groffen does know what
> he talks about, but it would be good to have at least the hard facts
> clearly documented, and easily accessible):
>
> http://archives.gentoo.org/gentoo-alt/msg_3e0aa31ec42c917484e6b0f86ffa5f2a.xml

Since we are not aiming to provide a complete system with a separate
compiler tool chain and libraries, I don't think much of this applies
to us. Sage already manages to relocate a binary install, so we know
that it is not impossible to do this in our restricted use case. :)

> or the current lack of Gentoo Prefix "binary distributions" resp. the
> need of external support to do the "very first start up" (since
> bootstrapping ain't really for the faint-hearted --- I did that myself
> successfully on Maemo5, OS X 10.4 and OS X 10.6, but I'm a developer,
> not a "mere user"), see e.g.:
>
> http://sourceforge.net/projects/prefix-launcher/

IMHO, bootstrapping is still the biggest problem. Though at least we
don't have to worry about building a tool chain.

> One last thing about "sage -clone, -b, -ba, br, ..." --- from my
> memory about several discussions, the sage community regards this as a
> "killer feature", i.e. that there is not some "sage" and then some
> other, different "sage-dev" package/distribution, but only one. So the
> hurdle for any user to gradually become a developer is made as low as
> possible. I have some rough ideas how to make this possible "inside" a
> standard distribution with also an eye on reliable package management,
> i.e. not restricted to Gentoo/Gentoo Prefix, but also Debian and so
> on. But these ideas would need thorough discussion.

I agree that this is a killer feature. The prototype tarball does
nothing about this problem. As I said earlier, I think we can
reconfigure portage to use "devel/*" as the build directory, and hook
up some internal portage commands to handle "make install" etc. Though
I haven't tried any of this and I'd appreciate hearing any other ideas.

-------------------------

Notes about the prefix tarball:

- To work around the bootstrapping problem, I assume that python and
the basic tools like bash (maybe also sed) are already installed on
the system. Of course this works perfectly well on my machines which
already run gentoo and portage (the package manager) is installed on
the base system. :)

IIRC, there were messages on this list about having python as a
dependency already. What are the dependencies of Sage besides tar,
gcc and gfortran? Does Sage depend on bash?

Is python available on the default install of all the systems
supported by Sage at the moment? Which version?

Unfortunately portage expects python-2.6 or newer. Using pkgcore or
paludis instead might solve this problem though.

pkgcore: http://www.pkgcore.org/
paludis: http://paludis.pioto.org/

Files in the tarball:

- install.sh
script to handle bootstrapping (= installing portage) then call
portage to install singular

- local/
this is the real prefix (in the gentoo sense), the root of the
install, named this way to replicate the directory structure of a
Sage tree. The symlink 'usr' in this directory also serves this
purporse.

- dist/
directory which includes package management related stuff, modeled
after the 'spkg' directory.

- portage/
the ebuild database for portage,
there is a version checked into mercurial in my home directory
on sage.math:
http://sage.math.washington.edu/home/burcin/portage/
Though I only started using mercurial at the end, after
realizing this is much more complicated than I expected. :)

- scripts/
contains the scripts used for the prefix tarball
- install.sh is the one you see at the root of the tarball
- bootstrap-prefix.sh
script borrowed from gentoo-prefix, commented out most
lines that actually run code, functions from this are
called in bootstrap-sage.sh
- bootstrap-sage.sh
redefines the paths and necessary functions from
bootstrap-prefix.sh to conform to our layout

Two important functions:

- bootstrap_tree()
creates the directory tree found in the tarball
- bootstrap_setup()
This is called from bootstrap_portage while installing
portage in the first (and only) step of bootstrapping
sets a default profile for portage based on the detected
platform. I kept only the linux platforms from the ones
defined in bootstrap-prefix

- profiles/
"profile" definitions, mostly copied from the gentoo-prefix tree
The only new part is the sage/ directory:
http://sage.math.washington.edu/home/burcin/portage/profiles/sage/
This adds two new profiles that inherit from the corresponding
prefix ones (see the files named 'parent'), These profiles define
architecture keywords. The package manager uses these to decide
which packages should be considered stable for a given platform.

The packages which should be considered already installed are
listed in the file packages.provided:

http://sage.math.washington.edu/home/burcin/portage/profiles/sage/linux/package.provided

How does it work?

- The install script
- sets the PATH and EPREFIX variables
EPREFIX is used to indicate the root directory for the prefix
install in the gentoo-prefix project
- installs portage in this EPREFIX "by hand" without using a package
manager. This is done by the bootstrap_portage() function in
bootstrap-prefix.sh.

- during the portage install, it detects the "profile" needed, by
looking a the CHOST variable. The profile, as explained above, lets
the package manager know the architecture etc.

- install.sh also puts a configuration file for portage in the prefix.
gentoo-prefix distributes a modified the portage tarball with
special default configuration. I didn't want to do that again,
or patch portage during the install.

When the package manager is installed properly below, the global
configuration is patched as necessary. See:

http://sage.math.washington.edu/home/burcin/portage/sys-apps/portage/files/make.globals.patch

So the configuration file is overwritten after that step.

- installs Volker's compiler wrapper using portage
AFAICT, this wasn't absolutely necessary since I didn't try
relocating the tree after the build. However, I can see that the
singular built in the prefix uses the ntl libraries installed in
the prefix instead of the ones on my system. This is an improvement
over the current Sage build I have.

- calls portage to install itself properly
This pulls in:
app-shells/bash
sys-libs/zlib
app-admin/eselect
virtual/libintl
virtual/libffi
app-admin/eselect-python
dev-lang/python
sys-apps/portage
app-admin/python-updater

- calls portage to install baselayout-prefix
This is the package that provides files for the environment
variables, etc. in the prefix. this pulls in:
sys-libs/readline
sys-apps/baselayout-prefix

I don't know why readline is built here and not as a dependency of
python. Perhaps there is a missing dependency in the gentoo ebuilds.

- calls portage to install singular
This pulls in:
dev-libs/gmp
dev-libs/gf2x
dev-libs/ntl
sci-mathematics/singular

- puts in a startprefix script when it's done
This starts a new shell with the PATH, etc. set.

To try it out, you just need to unpack the tar and run install.sh. Then
the startprefix script will get you a shell inside the prefix, where
you can run Singular.

Comments?

Cheers,
Burcin

[Maarten Derickx]

On May 3, 7:23 pm, Burcin Erocal <bur...@erocal.org> wrote:
>
>   IIRC, there were messages on this list about having python as a
>   dependency already. What are the dependencies of Sage besides tar,
>   gcc and gfortran? Does Sage depend on bash?
>

It definitly depends on bash, there are a lot of bash scripts it uses
(even for starting sage). To get an idea of which scripts all use bash
do for example.

grep -r $SAGE_ROOT/local/bin bash

[Volker Braun]

Looks great! It doesn't quite compile on Fedora 14. NTL can't find its generated header gf2x.h:

http://pastebin.com/WsfQcc2A

Not sure if its a problem with the prefix or just an NTL bug?

Volker

[Francois Bissey]

The version of NTL in portage uses gf2x not a NTL generated one.
So it should have been included. sage-on-gentoo has been using gf2x
almost forever :)

Francois

[Francois Bissey]

> > Looks great! It doesn't quite compile on Fedora 14. NTL can't find its
> > generated header gf2x.h:
> >
> > http://pastebin.com/WsfQcc2A
> >
> > Not sure if its a problem with the prefix or just an NTL bug?
>
> The version of NTL in portage uses gf2x not a NTL generated one.
> So it should have been included. sage-on-gentoo has been using gf2x
> almost forever :)
>

Now I had time to look at your pastebin and a bit at the tarball - I will need
to find a "non gentoo" linux system to test that one properly.
It looks like gf2x hasn't been installed before ntl which it should. Can you
check this? Of Burcin may have missed it because it was in his system.
That kind of stuff happens on a regular basis.

Francois

[François]

On May 4, 7:58 am, Maarten Derickx <m.derickx.stud...@gmail.com>
wrote:

I think we were discussing python as a dependency when talking about
including
patch. There was some motion to use hg instead but that introduced
circular
dependencies on python. To find out what's really needed apart from
bash
we have to look at prereq and what it is checking for.
config.ac in prereq is checking for compilers (c,c++ and fortran),
YACC and LEX (or functional equivalents), ar, m4, ranlib, ld,
perl-5.8.0+ (for building R).
A lot of these will be brought in by the prefix unless we require them
and
put them in package.provided.

Francois

[François]

On May 4, 5:23 am, Burcin Erocal <bur...@erocal.org> wrote:
> Hi,

>
> > One last thing about "sage -clone, -b, -ba, br, ..." --- from my
> > memory about several discussions, the sage community regards this as a
> > "killer feature", i.e. that there is not some "sage" and then some
> > other, different "sage-dev" package/distribution, but only one. So the
> > hurdle for any user to gradually become a developer is made as low as
> > possible. I have some rough ideas how to make this possible "inside" a
> > standard distribution with also an eye on reliable package management,
> > i.e. not restricted to Gentoo/Gentoo Prefix, but also Debian and so
> > on. But these ideas would need thorough discussion.
>
> I agree that this is a killer feature. The prototype tarball does
> nothing about this problem. As I said earlier, I think we can
> reconfigure portage to use "devel/*" as the build directory, and hook
> up some internal portage commands to handle "make install" etc. Though
> I haven't tried any of this and I'd appreciate hearing any other ideas.
>

I don't know what Georg has in mind. The current sage-on-gentoo
strip the hg repo so the first step will be to include that back.
Next I would like to keep the way it is installed now as it insure
integrity of the package for smooth upgrade. But once we put back
the hg repo we can create clones. In the developer guide it is
strongly
suggested to create a clone before doing whatever you want and then
you can
switch between clones or the main branch. I am basically suggesting
that
we enforce cloning for development.

>   - installs Volker's compiler wrapper using portage
>     AFAICT, this wasn't absolutely necessary since I didn't try
>     relocating the tree after the build. However, I can see that the
>     singular built in the prefix uses the ntl libraries installed in
>     the prefix instead of the ones on my system. This is an improvement
>     over the current Sage build I have.
>

Relocation aside I think the wrapper is still needed. Remember how
you first filed issues on github about stuff not found in the prefix?
The wrapper can help with that.
https://github.com/cschwan/sage-on-gentoo/issues/53
https://github.com/cschwan/sage-on-gentoo/issues/52

>   - calls portage to install baselayout-prefix
>     This is the package that provides files for the environment
>     variables, etc. in the prefix. this pulls in:
>         sys-libs/readline
>         sys-apps/baselayout-prefix
>
>     I don't know why readline is built here and not as a dependency of
>     python. Perhaps there is a missing dependency in the gentoo ebuilds.

Not sure.

>
>   - calls portage to install singular
>     This pulls in:
>         dev-libs/gmp
>         dev-libs/gf2x
>         dev-libs/ntl
>         sci-mathematics/singular
>

It would be be better to call "emerge -D singular" rather just
"emerge singular" may be that would take care of Volker's
problem. Actually Volker may very well suffer from the problem
you had above.

> Comments?

Well done so far. We need a repo. While we are putting this together
I'd like it if you added the following to etc/make.conf:
PORT_LOGDIR="${SBASE}"/var/log/portage
PORTAGE_ELOG_CLASSES="warn error info log qa"
PORTAGE_ELOG_SYSTEM="save"
FEATURES="split-log"

That way we have a log record similar to the one produced by
sage when building spkg. I have a feeling we may need it.
The "split-log" means that the log will be sorted in folders
according
to categories (app-shell and so on) given the clutter generated that's
fairly useful.
I still build sage on a machine that doesn't have sse2 you know!

Francois

[dagss]

I don't really have a say in this, but I've given this a lot of thought since I decided to drop Sage as my scientific Python distribution a year ago and have been searching for a new one ever since.

My problem with Gentoo in general is that it solves the problems that SPKGs have by adding complexity (in the same way any Linux distro does).

I much prefer something like Nix (http://nixos.org), where Eelco Dolstra instead had a new, brilliant idea which just makes simple things powerful. I wrote these notes up on using Nix for a scientific Python distribution (including Sage):

https://github.com/dagss/scidist/blob/master/ideas.rst

(I may be interested in putting in work in this direction...)

But of course, Gentoo has a scientific community etc. etc. which Nix sort of lacks, so I can definitely see Gentoo making more sense for you.

Dag Sverre Seljebotn

[Burcin Erocal]

Perhaps we can add a function in ebuilds which sets up a development
environment for the source. This would be similar to the way you can
call post install configuration from ebuilds any time after it's
merged. IIRC, the portage command for this was:

ebuild <ebuild_file_name> config

It would be best if this can avoid rebuilding the source.

At some point we thought about adding an option to Sage to do this.
Since setting this up for pynac, the notebook, etc. was getting to be
an FAQ.

> >   - installs Volker's compiler wrapper using portage
> >     AFAICT, this wasn't absolutely necessary since I didn't try
> >     relocating the tree after the build. However, I can see that the
> >     singular built in the prefix uses the ntl libraries installed in
> >     the prefix instead of the ones on my system. This is an
> > improvement over the current Sage build I have.
> >
>
> Relocation aside I think the wrapper is still needed. Remember how
> you first filed issues on github about stuff not found in the prefix?
> The wrapper can help with that.
> https://github.com/cschwan/sage-on-gentoo/issues/53
> https://github.com/cschwan/sage-on-gentoo/issues/52

If make.conf sets CFLAGS, etc. properly, then we can get away without
the wrapper to build the packages in the prefix. I ran into the
problem in those tickets because I was setting CFLAGS myself in
make.conf, without explicitly specifying the prefix include and library
paths. I somehow expected that to be handled automatically.

I spent time including the wrapper, because I want the prefix to work as
a development environment for mathematical software as well. If you
work in the shell started by the startprefix script, things should
work seamlessly and the compiler should look in the right places for
header files and libraries. The wrapper solves this problem perfectly.

> >   - calls portage to install baselayout-prefix
> >     This is the package that provides files for the environment
> >     variables, etc. in the prefix. this pulls in:
> >         sys-libs/readline
> >         sys-apps/baselayout-prefix
> >
> >     I don't know why readline is built here and not as a dependency
> > of python. Perhaps there is a missing dependency in the gentoo
> > ebuilds.
>
> Not sure.
>
> >
> >   - calls portage to install singular
> >     This pulls in:
> >         dev-libs/gmp
> >         dev-libs/gf2x
> >         dev-libs/ntl
> >         sci-mathematics/singular
> >
>
> It would be be better to call "emerge -D singular" rather just
> "emerge singular" may be that would take care of Volker's
> problem. Actually Volker may very well suffer from the problem
> you had above.

Good point. It's been a while since I used emerge/portage actually. I
switched to paludis long ago. I should have used "-D" while preparing
the packages.provided file as well. I will have a go at doing this
later.

> > Comments?
>
> Well done so far. We need a repo. While we are putting this together
> I'd like it if you added the following to etc/make.conf:
> PORT_LOGDIR="${SBASE}"/var/log/portage
> PORTAGE_ELOG_CLASSES="warn error info log qa"
> PORTAGE_ELOG_SYSTEM="save"
> FEATURES="split-log"
>
> That way we have a log record similar to the one produced by
> sage when building spkg. I have a feeling we may need it.
> The "split-log" means that the log will be sorted in folders
> according
> to categories (app-shell and so on) given the clutter generated that's
> fairly useful.
> I still build sage on a machine that doesn't have sse2 you know!

It might be cleaner to add these to the make.defaults file for portage.
Then we won't confuse the user with the extra lines in make.conf. I
will revise the patch for make.defaults.

There is a mercurial repository which you can pull from in

http://sage.math.washington.edu/home/burcin/portage

I will upload this to bitbucket today.

Thanks for the feedback.

Cheers,
Burcin

[Burcin Erocal]

I didn't know perl was a requirement. Anybody want to add to this? :)

I think they should be in package.provided to keep in line with Sage.

Does anyone have trouble installing these on their system before Sage?
I suppose this is not the right place to ask this question, since most
developers would have gone past that step long ago. :)

The python dependency also came up when discussing its use in
spkg-install scripts, for example in ATLAS. ATM, ATLAS spkg depends on
the Python spkg for this reason.

Cheers,
Burcin

[Burcin Erocal]

Hi Dag,

On Wed, 4 May 2011 00:51:56 -0700 (PDT)
dagss <d.s.se...@astro.uio.no> wrote:

> I don't really have a say in this, but I've given this a lot of
> thought since I decided to drop Sage as my scientific Python
> distribution a year ago and have been searching for a new one ever
> since.
>
> My problem with Gentoo in general is that it solves the problems that
> SPKGs have by adding complexity (in the same way any Linux distro
> does).
>
> I much prefer something like Nix (http://nixos.org), where Eelco
> Dolstra instead had a new, brilliant idea which just makes simple
> things powerful. I wrote these notes up on using Nix for a scientific
> Python distribution (including Sage):
>
> https://github.com/dagss/scidist/blob/master/ideas.rst
>
> (I may be interested in putting in work in this direction...)

Robert posted a link to your notes here at some point. I also saw the
thread on the gentoo-alt mailing list where you asked about using
prefix for a scientific software distribution. Especially after reading
your notes, I seriously considered using Nix as well.

Nix is based on a beautiful idea, but it doesn't really reduce the
complexity. We want more features from the build system, this will
naturally add complexity. When I first started out using Gentoo, I
thought it was a beautiful and simple extension of the BSD ports system.

I admit that I am biased since I am already very familiar with Gentoo.
But I like the fact that build scripts are written in bash. Knowing
that it has been field tested and most of the quirks that might come up
in package management are already worked out also makes me lean toward
Gentoo. I am willing to spend time on this, but I don't want to spend
time fixing a package manager.

The items you list in the "Needed work" section of your notes already
look like writing a package manager from scratch. The last ticket
(#6) about soft dependencies shows that the Nix approach of installing
every package in a separate directory and linking directly to the files
in that directory is not really useful either.

BTW, you could use the gentoo package manager to replicate a previous
version of your tree with using binary packages (you can make these
even after you are done installing the ebuild). There seems to be code
for something like this already:

http://en.gentoo-wiki.com/wiki/Demerge

> But of course, Gentoo has a scientific community etc. etc. which Nix
> sort of lacks, so I can definitely see Gentoo making more sense for
> you.

This is the strongest argument that makes Gentoo attractive. It
justifies the effort to switch to their package format. Since then, we
won't have to worry about manually updating spkg's ourselves.

Francois is always reporting how a more recent version of a package
works perfectly with Sage. This is simply because Gentoo gives him
those packages for free on his sage-on-gentoo install.

I would be very happy to see a discussion about the pros and cons of
this approach. I know I sort of skipped over the SEP step Georg
wanted. :)

Thank you for the feedback.

Burcin

[dagss]

What I mean is that you still have to do things like 1) tell make to install to temporary location, 2) scan for list of files, 3) record list of files for future uninstall, 4) move the files to final location.

With Nix this sort of thing just goes away.

Of course, it's already thoroughly debugged, so complexity may not matter.

 

BTW, you could use the gentoo package manager to replicate a previous

version of your tree with using binary packages (you can make these
even after you are done installing the ebuild). There seems to be code
for something like this already:

http://en.gentoo-wiki.com/wiki/Demerge

Yes, but AFAICT we're still talking at least half a minute for a rollback, and some hundred megabytes for creating a new branch. With Nix you can do both in less than a second. Very useful during development.

And I really like the Nix idea that several people can completely safely share a nix store (and so share disk space + build time CPU).

Think about the consequences for Sage development:

 1) William gets a patch from Trac and tests it against a package on sage.math

 2) Without knowing about this, you do the same, also on sage.math

...then Nix figures out that this derivation was already built and just uses William's build results.

With Portage, do get this behaviour, William would at the very least have to register somewhere that he created a binary package corresponding to this trac ticket, and offer it for download. That requires you to create tools around that.

With Nix, you just keep your existing workflow (use "patch" on your sources), and the cache kicks in more transparently.

> But of course, Gentoo has a scientific community etc. etc. which Nix
> sort of lacks, so I can definitely see Gentoo making more sense for
> you.

This is the strongest argument that makes Gentoo attractive. It
justifies the effort to switch to their package format. Since then, we
won't have to worry about manually updating spkg's ourselves.

Francois is always reporting how a more recent version of a package
works perfectly with Sage. This is simply because Gentoo gives him
those packages for free on his sage-on-gentoo install.

Well, if he's running Gentoo as his Linux distro, that *greatly* reduces the problem in the first place.

The main problem one must focus on here is that Gentoo packages (and nixpkgs for that matter) has strong assumptions in them that you use the whole system. If you want to use the host system toolchain, you get all kinds of problems, expected or unexpected.

For instance, Gentoo prefix requires a more modern version of Bash than what is available on my Uni's computers. So Sage would need to (perhaps optionally after probing) build its own Bash, or it becomes impossible to install out of the box at my Uni. And Gentoo prefix patches their gcc as well -- although I did make a little bit of headway simply telling it to use the one of the system, that's still around where I stopped using Gentoo Prefix. Also I remember that Gentoo packages in general seems to like living on the bleeding edge of autotools, so probably that must be built as well.

I remember seeing discussions on this list about whether "patch" should be shipped with Sage or not. Discussing going with Gentoo Prefix *really* puts that in perspective :-)

Of course, if you either

 a) decide to turn Sage into a mini-OS, or

 b) only use Portage and don't care about compatability Gentoo packages

you don't have this problem. Perhaps some compromise can be reached, but not, is my hunch, without added complexity and quirks.

I'm sure you already know this, and also my information may be outdated as I last checked out Gentoo prefix a year ago.

Anyway, this is the main reason I ended up proposing just "doing my/our own thing" (which is when you need simple&elegant technology). Anything coming out of the distro communities seem to have the assumptions that you should always use a distro for all your software. E.g., the Nix mailing list they explicitly told me to check out 0install or GUB for my purposes instead...

IF you can get something working on Gentoo Prefix without lots and lots of work, I agree 100% that's the way forward at the time being, simply because there's no mature alternative at the moment. But if lots and lots of work is needed anyway, that's when you start to look at technical advantages/disadvantages.

Dag Sverre 

[Volker Braun]

FWIW, I do have a SAGE_PREFIX/local/include/gf2x.h. The failing compilation

x86_64-pc-linux-gnu-g++ -I../include -I.  -O2 -pipe -c -fPIC  -o GF2X.lo GF2X.c

GF2X.c:13:18: fatal error: gf2x.h: No such file or directory

is executed in some faraway path, so it doesn't find the header unless it is in one of the standard system locations. That is,

# cpp -v

...

#include "..." search starts here:

#include <...> search starts here:

 /usr/local/include

 /usr/lib/gcc/x86_64-redhat-linux/4.5.1/include

 /usr/include

End of search list.

[Francois Bissey]

Yes that's a big problem with the current tarball of Burcin.
It doesn't fail for him or me because we already have the stuff in the system.
I am wondering if the wrapper is even used.
Adding "-I$SAGE_PREFIX/include" should work but I think there are issues
with the script as it is. I would be more comfortable if the startprefix script
was installed before we try to emerge singular. That way we can go back
start the prefix and try to emerge again, if it works it means portage is not
set up properly while the script is running.

Francois

[Francois Bissey]

We could go on about package theory for a while here. Just to make sure
I got my point across, sage -b(xx) would work on the clone.

> > >   - installs Volker's compiler wrapper using portage
> > >     AFAICT, this wasn't absolutely necessary since I didn't try
> > >     relocating the tree after the build. However, I can see that the
> > >     singular built in the prefix uses the ntl libraries installed in
> > >     the prefix instead of the ones on my system. This is an
> > > improvement over the current Sage build I have.
> >
> > Relocation aside I think the wrapper is still needed. Remember how
> > you first filed issues on github about stuff not found in the prefix?
> > The wrapper can help with that.
> > https://github.com/cschwan/sage-on-gentoo/issues/53
> > https://github.com/cschwan/sage-on-gentoo/issues/52
>
> If make.conf sets CFLAGS, etc. properly, then we can get away without
> the wrapper to build the packages in the prefix. I ran into the
> problem in those tickets because I was setting CFLAGS myself in
> make.conf, without explicitly specifying the prefix include and library
> paths. I somehow expected that to be handled automatically.
>
> I spent time including the wrapper, because I want the prefix to work as
> a development environment for mathematical software as well. If you
> work in the shell started by the startprefix script, things should
> work seamlessly and the compiler should look in the right places for
> header files and libraries. The wrapper solves this problem perfectly.
>

OK.

> > Well done so far. We need a repo. While we are putting this together
> > I'd like it if you added the following to etc/make.conf:
> > PORT_LOGDIR="${SBASE}"/var/log/portage
> > PORTAGE_ELOG_CLASSES="warn error info log qa"
> > PORTAGE_ELOG_SYSTEM="save"
> > FEATURES="split-log"
> >
> > That way we have a log record similar to the one produced by
> > sage when building spkg. I have a feeling we may need it.
> > The "split-log" means that the log will be sorted in folders
> > according
> > to categories (app-shell and so on) given the clutter generated that's
> > fairly useful.
> > I still build sage on a machine that doesn't have sse2 you know!
>
> It might be cleaner to add these to the make.defaults file for portage.
> Then we won't confuse the user with the extra lines in make.conf. I
> will revise the patch for make.defaults.
>

OK.

> There is a mercurial repository which you can pull from in
>
> http://sage.math.washington.edu/home/burcin/portage
>

If we are seriously working on this I'll be more concerned about
finding somewhere to host a clone and push [google code comes to mind]

Francois

[Volker Braun]

On Wednesday, May 4, 2011 1:27:27 PM UTC+1, François wrote:

Yes that's a big problem with the current tarball of Burcin.
It doesn't fail for him or me because we already have the stuff in the system.

We could use strace to make sure that no process in the compilation accesses headers or libraries outside of the prefix.

Volker

[Burcin Erocal]

Hi Francois,

On Thu, 05 May 2011 00:33:57 +1200
Francois Bissey <francoi...@canterbury.ac.nz> wrote:

> > > Well done so far. We need a repo. While we are putting this
> > > together I'd like it if you added the following to etc/make.conf:
> > > PORT_LOGDIR="${SBASE}"/var/log/portage
> > > PORTAGE_ELOG_CLASSES="warn error info log qa"
> > > PORTAGE_ELOG_SYSTEM="save"
> > > FEATURES="split-log"
> > >
> > > That way we have a log record similar to the one produced by
> > > sage when building spkg. I have a feeling we may need it.
> > > The "split-log" means that the log will be sorted in folders
> > > according
> > > to categories (app-shell and so on) given the clutter generated
> > > that's fairly useful.
> > > I still build sage on a machine that doesn't have sse2 you know!
> >
> > It might be cleaner to add these to the make.defaults file for
> > portage. Then we won't confuse the user with the extra lines in
> > make.conf. I will revise the patch for make.defaults.
> >
>
> OK.
>
> > There is a mercurial repository which you can pull from in
> >
> > http://sage.math.washington.edu/home/burcin/portage
> >
>
> If we are seriously working on this I'll be more concerned about
> finding somewhere to host a clone and push [google code comes to mind]

I added support for logging and put the repository on bitbucket:

https://bitbucket.org/burcin/sage-prefix

I will be heading home now. No time to make a new tarball
unfortunately.

Tomorrow, I will try to look into Volker's problem with NTL. You are
probably right and it is caused by the wrapper not working as expected.

Removing the sse2 use flag makes the gf2x build crash. Should NTL
depend on gf2x conditionally?

Cheers,
Burcin

[Volker Braun]

I finished compilerwrapper-1.1 which now allows you to transform the wrapper as well as the wrapped gcc binary names by means of configure options. For example,

./configure --with-gcc-transform-name='s/^/x86_64-/' --program-suffix='-4'

will call the compiler wrapper binaries gcc-4, c++-4, ... and call the underlying gcc binaries x86_64-gcc, x86_64-c++, ... The linker is always ld.

I think this is the most flexible option. 

Volker

[Ondrej Certik]

On Wed, May 4, 2011 at 12:51 AM, dagss <d.s.se...@astro.uio.no> wrote:
> I don't really have a say in this, but I've given this a lot of thought
> since I decided to drop Sage as my scientific Python distribution a year ago
> and have been searching for a new one ever since.

Just curious --- why did you drop it? Is it described at ideas.rst below?

> My problem with Gentoo in general is that it solves the problems that SPKGs
> have by adding complexity (in the same way any Linux distro does).
> I much prefer something like Nix (http://nixos.org), where Eelco Dolstra
> instead had a new, brilliant idea which just makes simple things powerful. I
> wrote these notes up on using Nix for a scientific Python distribution
> (including Sage):
> https://github.com/dagss/scidist/blob/master/ideas.rst
>
> (I may be interested in putting in work in this direction...)
> But of course, Gentoo has a scientific community etc. etc. which Nix sort of
> lacks, so I can definitely see Gentoo making more sense for you.

Btw, I have created a Sage compatible distribution:

http://qsnake.com/

all source code is hosted at github, and this 800 lines script:

https://github.com/qsnake/qsnake/blob/master/spkg/base/qsnake_run.py

is the whole "qsnake" program (just like the "sage" program) with all
options, package management (with build dependencies etc.) and so on.

Since I switched to use Fortran mainly, from C/C++ recently, I'll make
sure that the Fortran support works well (it is so so at the moment
---- but it uses the default Fortran on linux, no wrapper).

In any case, each package is just a regular git repo, and so patches
to it can be submitted upstream quite easily. However, given my
limited time, and the size of the problem, I can see that it doesn't
solve all problems. I'll study the ideas.rst in more details, as I can
see, lots of very clever people have thought about this already.

For my usage I use Qsnake and it works excellent for me (I use it
every day). I use it in the following way:

certik1@pike:~/repos/dftatom(master)$ qsnake --shell
Type CTRL-D to exit the Qsnake shell.
Qsnake: certik1@pike:~/repos/dftatom(master)$

and then with the "Qsnake" prefix on the command line, I can use all
the program installed in there (cmake, python, scipy, ....). No need
to be root.

Ondrej

[dagss]

On Thursday, May 5, 2011 3:42:59 AM UTC+2, Ondrej Certik wrote:

On Wed, May 4, 2011 at 12:51 AM, dagss <d.s.se...@astro.uio.no> wrote:
> I don't really have a say in this, but I've given this a lot of thought
> since I decided to drop Sage as my scientific Python distribution a year ago
> and have been searching for a new one ever since.

Just curious --- why did you drop it? Is it described at ideas.rst below?

Well, I figured that making SPKG's wasted too much of my time, and that I didn't any of the benefits I'd expect (like clean upgrading etc.). I guess I just discovered Sage's premise: The point is to simplify things for end-users, not the developers (and the distinction between the two is too high). As I don't care too much about distributing my software to others (I don't have any peer cosmologists using Python), the conclusion was I should stop writing SPKG's and rather just easy_install what I missed.

Then I realized that EPD had a lot of packages that I missed (and had to manually install) in Sage, so I switched to patching my EPD install instead of my Sage install.

I'm not happy about the situation, but the benefits of current systems are too small to warrant the investment. Hence scidist/ideas.rst, which is what I'd like to have in a system to make it more than worth the investment of starting to use it + lowering the investment necessary.

 

> My problem with Gentoo in general is that it solves the problems that SPKGs
> have by adding complexity (in the same way any Linux distro does).
> I much prefer something like Nix (http://nixos.org), where Eelco Dolstra
> instead had a new, brilliant idea which just makes simple things powerful. I
> wrote these notes up on using Nix for a scientific Python distribution
> (including Sage):
> https://github.com/dagss/scidist/blob/master/ideas.rst
>
> (I may be interested in putting in work in this direction...)
> But of course, Gentoo has a scientific community etc. etc. which Nix sort of
> lacks, so I can definitely see Gentoo making more sense for you.

Btw, I have created a Sage compatible distribution:

http://qsnake.com/

all source code is hosted at github, and this 800 lines script:

https://github.com/qsnake/qsnake/blob/master/spkg/base/qsnake_run.py

is the whole "qsnake" program (just like the "sage" program) with all
options, package management (with build dependencies etc.) and so on.

Well, for me EPD would work better as my stopgap solution because of existing package set. Long-term I think the Nix idea of cached build derivations is too good to give up (which doesn't exclude being Sage-compatible).

Dag sverre

[Burcin Erocal]

Hi Volker,

This is exactly what we needed. Thank you.

Can you tag the release with "hg tag"? This will let bitbucket generate
a tarball with a human readable name.

Cheers,
Burcin

[Ondrej Certik]

On Thu, May 5, 2011 at 12:49 AM, dagss <d.s.se...@astro.uio.no> wrote:
>
> On Thursday, May 5, 2011 3:42:59 AM UTC+2, Ondrej Certik wrote:
>>
>> On Wed, May 4, 2011 at 12:51 AM, dagss <d.s.se...@astro.uio.no> wrote:
>> > I don't really have a say in this, but I've given this a lot of thought
>> > since I decided to drop Sage as my scientific Python distribution a year
>> > ago
>> > and have been searching for a new one ever since.
>>
>> Just curious --- why did you drop it? Is it described at ideas.rst below?
>
> Well, I figured that making SPKG's wasted too much of my time, and that I
> didn't any of the benefits I'd expect (like clean upgrading etc.). I guess I
> just discovered Sage's premise: The point is to simplify things for
> end-users, not the developers (and the distinction between the two is too
> high). As I don't care too much about distributing my software to others (I
> don't have any peer cosmologists using Python), the conclusion was I should
> stop writing SPKG's and rather just easy_install what I missed.
> Then I realized that EPD had a lot of packages that I missed (and had to
> manually install) in Sage, so I switched to patching my EPD install instead
> of my Sage install.
> I'm not happy about the situation, but the benefits of current systems are
> too small to warrant the investment. Hence scidist/ideas.rst, which is what
> I'd like to have in a system to make it more than worth the investment of
> starting to use it + lowering the investment necessary.

Right. The same reasons why I started Qsnake --- I was spending too
much time creating SPKG, updating it and so on. In Qsnake, all I have
to do is to push a new change to the package to github, and do

qsnake -d
qsnake -f install package

And that's it. It automatically figures out that something has
changed, creates the spkg on the fly and saves it to spkg/standard/,
just like in Sage.

If you are the only one using it --- do you need binary packages? I
simply decided to stick with the Sage model to just use source Spkg
packages, and possibly create one binary of the whole thing. This is a
model, that one person can manage to do.

Ondrej

[Volker Braun]

Ooh didn't know bitbucket can do that :-)

https://bitbucket.org/vbraun/compilerwrapper/get/compilerwrapper-1.1.tar.bz2

[Francois Bissey]

That's not normal. Could you forward me the failing log? As for conditional
dependency it used to be that way but I don't see a reason to change it.

I have started populating your tree - it is not complete yet. I think I
haven't put any autotools which will be an extra requirement compared to
vanilla sage. I will also create a virtual/gmp so we can switch between mpir
and gmp. I am not sure how complete this is yet. I'll have to test from the
mac :)

Francois

[Burcin Erocal]

On Sat, 07 May 2011 21:15:11 +1200
Francois Bissey <francoi...@canterbury.ac.nz> wrote:

> > On Thu, 05 May 2011 00:33:57 +1200
> >

> > I added support for logging and put the repository on bitbucket:
> >
> > https://bitbucket.org/burcin/sage-prefix
> >
> > I will be heading home now. No time to make a new tarball
> > unfortunately.
> >
> > Tomorrow, I will try to look into Volker's problem with NTL. You are
> > probably right and it is caused by the wrapper not working as
> > expected.
> >
> > Removing the sse2 use flag makes the gf2x build crash. Should NTL
> > depend on gf2x conditionally?
> >
> That's not normal. Could you forward me the failing log? As for
> conditional dependency it used to be that way but I don't see a
> reason to change it.

I'm building without the sse2 use flag now. I'll put the logs up
somewhere if/when it fails.

> I have started populating your tree - it is not complete yet. I think
> I haven't put any autotools which will be an extra requirement
> compared to vanilla sage. I will also create a virtual/gmp so we can
> switch between mpir and gmp. I am not sure how complete this is yet.
> I'll have to test from the mac :)

I just pushed some more changes to bitbucket. The latest version builds
on a 32-bit debian squeeze machine. I will also try it on other linux
variants I find.

http://sage.math.washington.edu/home/burcin/sage-prefix-20110507.tar

I am hoping to get away without installing autotools, at least on
platforms Sage doesn't need them. Using the gentoo wrappers for
auto{conf,make} were enough for debian. It seems the debian autoconf
wrapper chooses the older version by default whereas gentoo picks the newer one. :)

This tarball should also solve Volker's problem where ntl couldn't find
gf2x. I realized that I completely misunderstood what exactly the
compilerwrapper does. After all it was necessary to add the include and
library paths to CFLAGS. I plan to patch compilerwrapper in the future
to do this itself.

Cheers,
Burcin

[Volker Braun]

On Saturday, May 7, 2011 2:29:27 PM UTC+1, Burcin Erocal wrote:

This tarball should also solve Volker's problem where ntl couldn't find
gf2x. I realized that I completely misunderstood what exactly the
compilerwrapper does. After all it was necessary to add the include and
library paths to CFLAGS. I plan to patch compilerwrapper in the future
to do this itself.

Since there is already the standard CFLAGS variable I didn't implement an extra WRAPPER_CFLAGS. Could be done, of course, but CFLAGS is generally well-supported so why reinvent the wheel.

A more interesting option might be to use the compilerwrapper to check that every package respects CFLAGS. That is, set CFLAGS='CFLAGS_TAG' and then remove it from the gcc command line in the compilerwrapper (or abort if it is missing).

[Burcin Erocal]

On Sat, 07 May 2011 21:15:11 +1200
Francois Bissey <francoi...@canterbury.ac.nz> wrote:

> > Removing the sse2 use flag makes the gf2x build crash. Should NTL
> > depend on gf2x conditionally?
> >
> That's not normal. Could you forward me the failing log? As for
> conditional dependency it used to be that way but I don't see a
> reason to change it.

Here is the log:

http://sage.math.washington.edu/home/burcin/gf2x-0.9.5:20110507-133059.log

All the logs from that build are here:

http://sage.math.washington.edu/home/burcin/gf2x_sse2_logs.tar.bz2

This should be reproducible by just removing any mention of sse2 from
install.sh.

Thank you.

Burcin

[Volker Braun]

You shouldn't need autotools to build anything, only the autogenerated scripts that are in the source tarball. But it seems like some patches are re-running autoconf. On my Fedora 14 install it breaks early on:

>>> Emerging (5 of 8) app-admin/eselect-python-20100321

 * eselect-python-20100321.tar.bz2 RMD160 SHA1 SHA256 size ;-) ...                                       [ ok ]

 * Package:    app-admin/eselect-python-20100321

 * Repository: gentoo_prefix

 * Maintainer: python

 * USE:        amd64 elibc_glibc kernel_linux prefix userland_GNU

 * FEATURES:   preserve-libs

>>> Unpacking source...

>>> Unpacking eselect-python-20100321.tar.bz2 to /home/vbraun/opt/sage-prefix-20110507/dist/tmp/portage/app-admin/eselect-python-20100321/work

 * Applying eselect-python-20100321-prefix.patch ...                                                     [ ok ]

ac-wrapper: Unable to locate any usuable version of autoconf.

  I tried these versions: 2.50:2.5 2.99:2.5 2.98:2.5 2.97:2.5 2.96:2.5 2.95:2.5 2.94:2.5 2.93:2.5 2.92:2.5 2.91:2.5 2.90:2.5 2.89:2.5 2.88:2.5 2.87:2.5 2.86:2.5 2.85:2.5 2.84:2.5 2.83:2.5 2.82:2.5 2.81:2.5 2.80:2.5 2.79:2.5 2.78:2.5 2.77:2.5 2.76:2.5 2.75:2.5 2.74:2.5 2.73:2.5 2.72:2.5 2.71:2.5 2.70:2.5 2.69:2.5 2.68:2.5 2.67:2.5 2.66:2.5 2.65:2.5 2.64:2.5 2.63:2.5 2.62:2.5 2.61:2.5 2.60:2.5 2.59:2.5  2.13:2.1

  With a base name of '/home/vbraun/opt/sage-prefix-20110507/local/usr/bin/autoheader'.

 * ERROR: app-admin/eselect-python-20100321 failed (unpack phase):

 *   autogen.sh failed

 * 

 * Call stack:

 *     ebuild.sh, line  62:  Called src_unpack

 *   environment, line 2623:  Called die

 * The specific snippet of code:

 *       ./autogen.sh || die "autogen.sh failed"

 * 

 * If you need support, post the output of 'emerge --info =app-admin/eselect-python-20100321',

 * the complete build log and the output of 'emerge -pqv =app-admin/eselect-python-20100321'.

 * The complete build log is located at '/home/vbraun/opt/sage-prefix-20110507/dist/log/build/app-admin/eselect-python-20100321:20110507-142824.log'.

 * The ebuild environment file is located at '/home/vbraun/opt/sage-prefix-20110507/dist/tmp/portage/app-admin/eselect-python-20100321/temp/environment'.

 * S: '/home/vbraun/opt/sage-prefix-20110507/dist/tmp/portage/app-admin/eselect-python-20100321/work/eselect-python-20100321'

>>> Failed to emerge app-admin/eselect-python-20100321, Log file:

>>>  '/home/vbraun/opt/sage-prefix-20110507/dist/log/build/app-admin/eselect-python-20100321:20110507-142824.log'

[Burcin Erocal]

On Sat, 7 May 2011 07:30:52 -0700 (PDT)
Volker Braun <vbrau...@gmail.com> wrote:

> You shouldn't need autotools to build anything, only the
> autogenerated scripts that are in the source tarball. But it seems
> like some patches are re-running autoconf. On my Fedora 14 install it
> breaks early on:

<snip error log>

Unfortunately gentoo ebuilds assume that autotools are available.
Initially, it failed in the same place while trying to build on debian.
If you already have autoconf and autoheader, it is merely a problem of
the wrapper picking the right executable. I will see if I can get
access to a fedora 14 box to fix this.

Thanks.

Burcin

[Burcin Erocal]

On Sat, 7 May 2011 06:42:10 -0700 (PDT)
Volker Braun <vbrau...@gmail.com> wrote:

> On Saturday, May 7, 2011 2:29:27 PM UTC+1, Burcin Erocal wrote:
> >
> > This tarball should also solve Volker's problem where ntl couldn't
> > find gf2x. I realized that I completely misunderstood what exactly
> > the compilerwrapper does. After all it was necessary to add the
> > include and library paths to CFLAGS. I plan to patch
> > compilerwrapper in the future to do this itself.
> >
>
> Since there is already the standard CFLAGS variable I didn't
> implement an extra WRAPPER_CFLAGS. Could be done, of course, but
> CFLAGS is generally well-supported so why reinvent the wheel.

I want the compiler wrapper to handle building programs within the
prefix without the user having to keep track of the prefix root, etc.
The prefix should be easy to use as a development environment for math
software, where all the core libraries are already present.

> A more interesting option might be to use the compilerwrapper to
> check that every package respects CFLAGS. That is, set
> CFLAGS='CFLAGS_TAG' and then remove it from the gcc command line in
> the compilerwrapper (or abort if it is missing).

This might be an issue for Sage, but I don't think it is needed if we
take gentoo ebuilds. At least I hope their QA process makes sure these
work as intended.

Cheers,
Burcin

[Volker Braun]

On Saturday, May 7, 2011 3:57:02 PM UTC+1, Burcin Erocal wrote:

> Since there is already the standard CFLAGS variable I didn't
> implement an extra WRAPPER_CFLAGS.

I want the compiler wrapper to handle building programs within the

prefix without the user having to keep track of the prefix root, etc.
The prefix should be easy to use as a development environment for math
software, where all the core libraries are already present.

Then why not use CPATH / C_INCLUDE_PATH / CPLUS_INCLUDE_PATH?

http://gcc.gnu.org/onlinedocs/gcc-4.6.0/gcc/Environment-Variables.html

Volker

[Francois Bissey]

> On Sat, 07 May 2011 21:15:11 +1200
>
>

> I just pushed some more changes to bitbucket. The latest version builds
> on a 32-bit debian squeeze machine. I will also try it on other linux
> variants I find.
>
> http://sage.math.washington.edu/home/burcin/sage-prefix-20110507.tar
>
> I am hoping to get away without installing autotools, at least on
> platforms Sage doesn't need them. Using the gentoo wrappers for
> auto{conf,make} were enough for debian. It seems the debian autoconf
> wrapper chooses the older version by default whereas gentoo picks the newer
> one. :)
>

That would be nice but we are adding to the pre-requirements to build sage.
We need them to patch any configure/makefile.am/libtool problems.

Francois

[Francois Bissey]

> On Sat, 07 May 2011 21:15:11 +1200
>
> Francois Bissey <francoi...@canterbury.ac.nz> wrote:
> > > Removing the sse2 use flag makes the gf2x build crash. Should NTL
> > > depend on gf2x conditionally?
> >
> > That's not normal. Could you forward me the failing log? As for
> > conditional dependency it used to be that way but I don't see a
> > reason to change it.
>
> Here is the log:
>
> http://sage.math.washington.edu/home/burcin/gf2x-0.9.5:20110507-133059.log
>

Access forbidden for that one.

> All the logs from that build are here:
>
> http://sage.math.washington.edu/home/burcin/gf2x_sse2_logs.tar.bz2
>

But I get that one.

I think I see the kind of problem in configure. It will be a problem if we
want to create binaries for older machines. It is definitely a bug.
That raise the question of useflags do we give people some choices before
they start?

Francois

[Burcin Erocal]

On Sun, 08 May 2011 07:06:03 +1200
Francois Bissey <francoi...@canterbury.ac.nz> wrote:

> > On Sat, 07 May 2011 21:15:11 +1200
> >
> > Francois Bissey <francoi...@canterbury.ac.nz> wrote:
> > > > Removing the sse2 use flag makes the gf2x build crash. Should
> > > > NTL depend on gf2x conditionally?

<snip log exchange>

> I see the kind of problem in configure. It will be a problem
> if we want to create binaries for older machines. It is definitely a
> bug. That raise the question of useflags do we give people some
> choices before they start?

We should make sure the packages build with different use flags. They
provide great flexibility. In this case, they could just replace the
SAGE_FAT_BINARY environment variable.

I don't know how/when to let the user configure use flags. I only set
sse2 by default because gf2x didn't build otherwise. :) I would expect
this to be done by some autodetect script in the future, but for now,
we can make the install script work in two stages. First sets up
portage proper, the next stage installs the rest which would include
packages that might be affected by the optimization flags.

I will introduce functions for these stages in install.sh and allow
them to be run individually with command line arguments.

Thanks.

Burcin

[Burcin Erocal]

Because I didn't know about them. :)

I will change the baselayout-prefix ebuild to add these as
default environment variables for the prefix. This saves me having to
patch compilerwrapper. Thanks.

Are these GCC specific? Are there equivalent parameters for Intel/Sun
compilers?

Cheers,
Burcin

[Volker Braun]

On Tuesday, May 10, 2011 10:17:40 AM UTC+1, Burcin Erocal wrote:

> > Then why not use CPATH / C_INCLUDE_PATH / CPLUS_INCLUDE_PATH?

> Are these GCC specific? Are there equivalent parameters for Intel/Sun

I don't know. But they seem useful enough that every compiler vendor should have something like that :-)

 

[Francois Bissey]

OK, I am sorry I haven't provided you with a fix yet. Things are a bit hectic
again for me at the moment. Hopefully I'll have a nice quiet spot on Thursday.

Francois

[Burcin Erocal]

On Wed, 11 May 2011 00:10:29 +1200
Francois Bissey <francoi...@canterbury.ac.nz> wrote:

No worries. We also haven't faced any of the bigger outstanding issues,
like relocating the tree or setting up packages in an "easy to
patch/developer friendly" way. There really is no hurry to fix this. :)

Cheers,
Burcin

[François]

gf2x is a little bugger. If you are on x86_64 or a few other select platforms (any of core2|opteron|x86_64|nocona|k10 )

it automatically add files tuned for sse2 regardless of the other configurations checks. That will be quite messy to untangle 

properly.

I may have to force sse2 on x86_64, the assumption that x86_64 always has sse2 is probably true so

it wouldn't be a problem for 64bits binaries. The only problem is what happens on core2 iMac running

10.5.8 in 32bits mode.

Francois