compile sage from github

[Ralf Hemmecke]

Hello!

I've cloned from git://github.com/sagemath/sage.git
(094712a Merge in bug fixes for the dev scripts)
and typed "make" on a (relatively minimal) Debian 7 Installation.

I didn't know whether such a step would actually succeed, since I
haven't yet been able to find a description that such a simple "make"
would work. So I didn't expect success.

But I am somewhat surprised that the build fails at building git (see
below). :-(

My Debian system says:
git version 1.7.10.4
and Sage wants to build 1.7.12.2.

Is there really any reason why Sage wouldn't be satisfied with a smaller
version number. What super features are used when building/developing
sage that justify the need for 1.7.12.2?

What actually surprises me is that Sage must build git at all. Wouldn't
it make sense to restrict build and development scripts to use features
of a git version number that is usually satisfied by current major
stable distributions and build git only when this is not fuflilled for
some very old distribution. Note that git must have been installed,
since I have (successfully) cloned sage from github.

Ralf

=================================================================

Manifying blib/man3/Git.3pm
make[5]: Leaving directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src/perl'
make[4]: Leaving directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src/perl'
make[4]: Entering directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src/git_remote_helpers'
make[4]: Leaving directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src/git_remote_helpers'
make[4]: Entering directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src/templates'
make[4]: Leaving directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src/templates'
MSGFMT po/build/locale/da/LC_MESSAGES/git.mo
/bin/sh: 1: msgfmt: not found
make[3]: *** [po/build/locale/da/LC_MESSAGES/git.mo] Error 127
make[3]: Leaving directory
`/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0/src'
Error building git.

real 1m20.508s
user 1m2.884s
sys 0m7.028s
************************************************************************
Error installing package git-1.7.12.2.p0
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
/home/me/v/git/sage/logs/pkgs/git-1.7.12.2.p0.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0 and type
'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
(cd '/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0' &&
'/home/me/v/git/sage/sage' --sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************
make[2]: ***
[/home/me/v/git/sage/local/var/lib/sage/installed/git-1.7.12.2.p0] Error 1
make[2]: Leaving directory `/home/me/v/git/sage/build'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/me/v/git/sage/build'

real 159m35.587s
user 120m8.078s
sys 8m16.735s
***************************************************************
Error building Sage.

The following package(s) may have failed to build:

package: git-1.7.12.2.p0
log file: /home/me/v/git/sage/logs/pkgs/git-1.7.12.2.p0.log
build directory:
/home/me/v/git/sage/local/var/tmp/sage/build/git-1.7.12.2.p0

The build directory may contain configuration files and other potentially
helpful information. WARNING: if you now run 'make' again, the build
directory will, by default, be deleted. Set the environment variable
SAGE_KEEP_BUILT_SPKGS to 'yes' to prevent this.

make: *** [build] Error 1

[Vincent Delecroix]

Hi,

The part of the log you paste is not long enough. I guess you need to
install msgfmt which is not part of standard debain. The name of the
package is gettext.

Does anybody know why git is a standard spkg ? It is likely that
anybody has his/her own git...

Vincent


2013/11/10, Ralf Hemmecke <hemm...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/groups/opt_out.
>

[Felix Salfelder]

On Sun, Nov 10, 2013 at 04:56:58PM +0100, Ralf Hemmecke wrote:
> What actually surprises me is that Sage must build git at all. Wouldn't
> it make sense to restrict build and development scripts to use features
> of a git version number that is usually satisfied by current major
> stable distributions and build git only when this is not fuflilled for
> some very old distribution. Note that git must have been installed,
> since I have (successfully) cloned sage from github.

Hi Ralf.

I have addressed this problem within my gsoc project. the solution is a
configurable toplevel build system that allows to choose between "build
stuff shipped with sage (the distribution)" and "fallback to already
installed software" on a per-package basis. see [1] for instructions on
how to build a demo sage (the distribution) on top of debian.

if this is of any use, you may want to help to integrate a proper build
system for sagelib [2], and eventually finish the toplevel build system
described above [3].

good luck
felix

[1] https://wiki.debian.org/DebianScience/Sage
[2] http://trac.sagemath.org/ticket/14807
[3] http://trac.sagemath.org/ticket/14796

[Ralf Hemmecke]

> The part of the log you paste is not long enough. I guess you need to
> install msgfmt which is not part of standard debain. The name of the
> package is gettext.

Thanks. I'd probably found that myself. But of course, that was not my
point.

> Does anybody know why git is a standard spkg ? It is likely that
> anybody has his/her own git...

That comes closer. Wouldn't it make sense to let (sage-)configure test
whether the features of an installed git are already sufficient? I
personally would get confused if sage used its own git.

Ralf

[Vincent Delecroix]

2013/11/10, Felix Salfelder <fe...@salfelder.org>:

Hi Felix,

I guess that it is one solution but not necessarily the solution ;-) I
am also convinced that it is the best one. I would be happy to be part
of that project but I am wondering if it is worth it. As it is
mentioned on the debian wiki, some of the sage packages are patched.
This implies that you can not really use the system versions. If there
is no obligation of having non-patched spkg I do not see how this
project can work. I do really think that this is the best of what can
happen for Sage but I also think that we need an agreement of all Sage
developers that spkg can not be patched...

All best,
Vincent

[Felix Salfelder]

On Sun, Nov 10, 2013 at 01:02:30PM -0500, Vincent Delecroix wrote:

> I do really think that this is the best of what can
> happen for Sage but I also think that we need an agreement of all Sage
> developers that spkg can not be patched...

actually, it's the other way around. if there were no patched/bleeding
edge packages within sage-the-distribution, there was no need for that
toplevel build system -- you could just run standalone sagelib on top of
anything you like.

the idea is to run sage (the distribution) on top of (e.g.) debian. and
only compile packages that deviate from upstream or the (e.g.) debian
package. particularly, this enables you to *develop* sage within
sage-the-distrition on top of any other software distribution without
unnecessarily compiling hundreds of spkgs (like git).

regards
felix

[Volker Braun]

On Sunday, November 10, 2013 10:14:30 AM UTC-8, Felix Salfelder wrote:

the idea is to run sage (the distribution) on top of debian. and
only compile packages that deviate from upstream or the debian
package.

Fixed that for you.

 

[Felix Salfelder]

that's great. would you be so kind and tell me/us where to find it and
how to make use of it?

thanks
felix

[Volker Braun]

On Monday, November 11, 2013 7:11:20 AM UTC-8, Felix Salfelder wrote:

> Fixed that for you.

that's great. would you be so kind and tell me/us where to find it and
how to make use of it? 

 FTFY (Often used sarcastically) - not to fix an honest mistake, but to sarcastically disagree with someone.

[Julien Puydt]

Le 11/11/2013 16:25, Volker Braun a �crit :

Can we get the un-sarcastic direct criticism to discuss on technical merit?

Snark on #sagemath

[Volker Braun]

On Monday, November 11, 2013 7:51:23 AM UTC-8, Snark wrote:

Can we get the un-sarcastic direct criticism to discuss on technical merit?

Its not a technical problem - I think Felix is technically very competent. But he didn't listen to our concerns nor did he engage in discussions. He continues to be focused on Debian whereas Sage has a much larger target. In the end, a Debian fork was created which does absolutely nothing for other platforms.

[Ralf Hemmecke]

On 11/11/2013 06:29 PM, Volker Braun wrote:

> Its not a technical problem - I think Felix is technically very competent.
> But he didn't listen to our concerns nor did he engage in discussions. He
> continues to be focused on Debian whereas Sage has a much larger target. In
> the end, a Debian fork was created which does absolutely nothing for other
> platforms.

Since I did not follow the GSOC discussion, can you point me (or better
summarize) what the better alternative would be to install sage on
default packages that are available a given distribution? Of course, I
mean that a sage installation should never install anything that is
already available in a standard distribution.

Obviously your (and probably others) opinion is that a sage installation
as approached by Felix for Debian cannot easily be lifted to other
non-debian distributions. I haven't actually looked at any code (so I'm
rather uninformed). Still, can you summarize what the problems with
Felix' approach is?

Thank you
Ralf

[Volker Braun]

On Monday, November 11, 2013 9:39:28 AM UTC-8, Ralf Hemmecke wrote:

Of course, I
mean that a sage installation should never install anything that is
already available in a standard distribution.

Even if that means that the Sage testsuite fails because distribution packages have a bug that isn't fixed upstream yet? Thats an important policy question and we never had that discussion....

can you summarize what the problems

Build it on all supported platforms and then report back. This ought to be the most important output of the project but I never saw it.

[Ralf Hemmecke]

>> mean that a sage installation should never install anything that is
>> already available in a standard distribution.

> Even if that means that the Sage testsuite fails because distribution
> packages have a bug that isn't fixed upstream yet? Thats an important
> policy question and we never had that discussion....

Come on, it's clear that a decent system that wants to build on existing
software has to specify the least requirement for that existing
software. If the underlying distribution does not fulfil that
requirement, something else has to be done.

My initial question was "What requirement for git does Sage have?"
Where can I find a list of version number for all the packages that are
available in some distribution and *could* be used by Sage.
In other words why did my sage build want to install git on Debian 7
although I already have it installed. Does that mean the git that comes
with Debian 7 has a bug that sage cannot live with? I guess not.

> can you summarize what the problems

> Build it on all supported platforms and then report back. This ought to be
> the most important output of the project but I never saw it.

I don't have the feeling that this answers my question. :-(

Ralf

[Volker Braun]

On Monday, November 11, 2013 10:00:21 AM UTC-8, Ralf Hemmecke wrote:

My initial question was "What requirement for git does Sage have?"

It should have none, you found a bug. It would be nice if you could answer Vincent's question about the full build log so it can be fixed.

 

[Ralf Hemmecke]

>> My initial question was "What requirement for git does Sage have?"

> It should have none, you found a bug. It would be nice if you could answer
> Vincent's question about the full build log so it can be fixed.

OK. I can only dig that out later.

But I think even if I send that log, the bug is rather that Sage should
check "git --version" and then don't start trying to build git at all,
no? My log wouldn't help for that, since all the information about git
versions was already in my initial mai.

Ralf

[William Stein]

On Mon, Nov 11, 2013 at 10:09 AM, Ralf Hemmecke <hemm...@gmail.com> wrote:
>>> My initial question was "What requirement for git does Sage have?"
>
>> It should have none, you found a bug. It would be nice if you could answer
>> Vincent's question about the full build log so it can be fixed.
>
> OK. I can only dig that out later.
>
> But I think even if I send that log, the bug is rather that Sage should
> check "git --version" and then don't start trying to build git at all,
> no?

If this (and any other similar) change is made, it needs to be made in
a way that doesn't mess up the binaries we build. The binaries that
are built for generic redistribution should have everything built by
default. Build machines will likely have git installed, but a random
user installing Sage from a binary could easily not have git on their
target system.

-- william



>
> Ralf

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/groups/opt_out.

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Volker Braun]

On Monday, November 11, 2013 10:09:54 AM UTC-8, Ralf Hemmecke wrote:

But I think even if I send that log, the bug is rather that Sage should
check "git --version" and then don't start trying to build git at all,
no?

That is really a feature request, not a bug. Of course it would be nice. If you want to work on a configure-style script that queries the system for build- and runtime dependencies then please do! Ideally it'll know about package management in different distributions to make reasonable suggestions about what to install before trying to build from source. And is easily extensible. 

[Felix Salfelder]

On Mon, Nov 11, 2013 at 06:39:28PM +0100, Ralf Hemmecke wrote:
> Obviously your (and probably others) opinion is that a sage installation
> as approached by Felix for Debian cannot easily be lifted to other
> non-debian distributions. I haven't actually looked at any code (so I'm
> rather uninformed). Still, can you summarize what the problems with
> Felix' approach is?

Hi there.

nothing i proposed to merge into sage is debian specific^*. i merely
made it possible to optionally disable packages, and make sure the rest
still works (independent of this choice). finally it worked with either
all packages enabled on a random platform, or with most packages
disabled (manually) and running on debian (carefully crafted with recent
packages).

and to make it clear: the main focus has never been debian. that's why i
was not done after unhardwiring the run time paths in sagelib (=src/*),
it was pretty clear that nobody cares. Furthermore, i didn't even dare
to propose automatic choice of whether a package should be disabled. you
guess that i am sure it would work out for most packages.

Volker (welcome back from sarcasm mode!) may be right that some doctests
were failing in my demo tarball, but that was about the same number that
were failing in the "build-system" branch, where I started off. once the
dust from the git transition has settled, it should be straightforward to
review/merge/rebase the patches/commits/branches i have lined up. but i
didnt receive any serious feedback yet -- particularly not during the
project period. googlegroups won't forget.

have fun
felix

^*Volker might have misconceived my failed attempt to fix libgap. when
it turned out that OSX doesnt like symbol name mangling, i proposed to
patch the (not yet existing, not yet planned, not yet reachable)
sagelib package for debian, instead of unnecessarilty packaging
libgap for debian.

[Julien Puydt]

Le 11/11/2013 19:31, Volker Braun a �crit :

This is what Felix' GSOC was about... make it possible to let sage build
everything ; or be more reasonable and sensible and only build the
necessary parts.

Let me repeat: Felix' GSOC doesn't break the build-everything sage (at
least not more than it is in the main branch!).

Snark on #sagemath

[Felix Salfelder]

On Mon, Nov 11, 2013 at 07:09:54PM +0100, Ralf Hemmecke wrote:
> But I think even if I send that log, the bug is rather that Sage should
> check "git --version" and then don't start trying to build git at all,
> no?

"checking whether <package> is necessary" is one part of the problem.
according to some sage developers it must also be impossible to
accidentally run into <package> provided by the (surrounding) system, if
<package> has not been explicitly marked unnecessary. so unless you
change the rules, you will have to resolve #12627 (my proposed toplevel
build system does).

regards
felix

[Robert Bradshaw]

It's much better to accidentally build a git that we don't need than
the other way around (especially if version mismatches manifest as
subtle bugs). Also, the git shipped with Sage, if not patched, at
least is well tested in context. Working binary downloads are very
important too, not everyone clones and builds.

The biggest bug is that clone + make didn't just work; once we solve
that we can optimize the process.

- Robert

[Vincent Delecroix]

Hi,

I definitely think that aim of Felix is good and needed. For example,
it is hard to incorporate python module into Sage distribution (I
recently went into troubles with pygtk). I guess that having Sage
working with the builtin Python would solve the problem. Could
someone confirm ?

It is hard to me to understand how sage currently deals with libraries
and programs. And it is even harder to understand Felix solution.
Could someone try to explain where should I look ?

Thanks,
Vincent

2013/11/11, Julien Puydt <julien...@laposte.net>:

[Ralf Hemmecke]

> It would be nice if you could answer Vincent's question about the
> full build log so it can be fixed.

As promised...

Oh, interesting, I just looked through it. One can find there something like

/bin/sh: 1: curl-config: not found
/bin/sh: 4: autoconf: not found

/bin/sh: 1: msgfmt: not found

but only the last one seems to matter.

Ralf

[Volker Braun]

This has been reported before at https://groups.google.com/d/msg/sage-git/4JtJ9w-kmTw/tWdXPChx4d4J

[Ralf Hemmecke]

>> check "git --version" and then don't start trying to build git at all,

> That is really a feature request, not a bug. Of course it would be nice. If
> you want to work on a configure-style script that queries the system for
> build- and runtime dependencies then please do!

Interesting! You can check the archive that I tried to do this some
years ago and nobody was interested, quite the contrary. Time seemed to
have changed that, but my coding time now mainly goes to other projects.
Sorry.

Ralf

[Volker Braun]

This is now http://trac.sagemath.org/15401 (needs review)

[Ralf Hemmecke]

On 11/11/2013 07:22 PM, William Stein wrote:
>> check "git --version" and then don't start trying to build git at all,

> If this (and any other similar) change is made, it needs to be made in
> a way that doesn't mess up the binaries we build. The binaries that
> are built for generic redistribution should have everything built by
> default.

I guess, it should be standard that a "make" started from a github
checkout tests the system and builds git (or anything else that is
already available in a reasonable version on the system) only if its
non-existing or existing in an incompatible form.

I would assume that for creating the source release one would (instead)
have to type "make dist" [1] and for a binary distribution perhaps "make
bindist". The *result* would then contain/build everything as required.

It's perhaps also reasonable that the Makefile that is produced for the
distribution is slightly different than the one on github or that one
simply adds an additional file to the distribution tarball that enables
"make" to distinguish between github checkout and source/binary
distribution.

Given the many software packages on which Sage builds, I wouldn't expect
that I get exactly the same sage (as in the official release) when I
type "make" from a github checkout. The only thing I would expect is
that I am *able* to build exactly the same distribution files and thus
can do everything localy a release manager usually does. For that, it
is, of course, necessary, to list somewhere the *exact* versions of all
the subpackages that go into a sage release (i.e. to make it reproducible).

I think sage already does the latter with the spkgs, but why not also
have another list for *minimal version requirements* of other software
like git, maxima, ecl, etc.?

Just an idea.

Ralf

[1] http://www.gnu.org/software/make/manual/make.html#Standard-Targets

[Julien Puydt]

Le 11/11/2013 20:45, Robert Bradshaw a �crit :

This is maddening.

All other projects would do like this:
(1) check if git is available ;
(2) check if the available git has a recent enough version (for other
things than git, a full equality on the version might make sense) ;
(3) in case it's not a good git, either error out (hard dep) or build a
good one (embedded source dep).

Only sage won't look around for what is there, and will actively spread
doctests around to check the embedded version was really used (there are
places where exact versions are checked just for that!) ; and will also
on some occasions *patch* the embedded version and use the variation
somewhere else for no other reason than making using upstream break
(example: disabling threading in ecl was done in the spkg instead of
calling the api to disable it in ecl.pyx).

"Beware of the subtle bugs!" say the sage developers. The spkg-ed
versions have subtle bugs, so using them doesn't magically solve
anything. The rest of the world knows about the subtle bugs, and deals
with them. They even deal with them better precisely because having
people use the code in more various settings makes them easier to
understand and fix.

To make a comparison which should go directly to the heart of quite a
few people around here: when you want to use a theorem, you check the
hypotheses then call it by its name; you don't fall down to ZFC and do
the full climb upwards!

Snark on #sagemath

[Volker Braun]

On Monday, November 11, 2013 11:55:56 PM UTC-8, Snark wrote:

example: disabling threading in ecl

Perhaps the scope of the Sage project is different from that of an embedded Lisp interpreter?

[Robert Bradshaw]

On Mon, Nov 11, 2013 at 11:55 PM, Julien Puydt <julien...@laposte.net> wrote:

> Le 11/11/2013 20:45, Robert Bradshaw a écrit :
>> On Mon, Nov 11, 2013 at 10:31 AM, Volker Braun <vbrau...@gmail.com> wrote:
>>> On Monday, November 11, 2013 10:09:54 AM UTC-8, Ralf Hemmecke wrote:
>>>>
>>>> But I think even if I send that log, the bug is rather that Sage should
>>>> check "git --version" and then don't start trying to build git at all,
>>>> no?
>>>
>>>
>>> That is really a feature request, not a bug. Of course it would be nice. If
>>> you want to work on a configure-style script that queries the system for
>>> build- and runtime dependencies then please do! Ideally it'll know about
>>> package management in different distributions to make reasonable suggestions
>>> about what to install before trying to build from source. And is easily
>>> extensible.
>>
>> It's much better to accidentally build a git that we don't need than
>> the other way around (especially if version mismatches manifest as
>> subtle bugs). Also, the git shipped with Sage, if not patched, at
>> least is well tested in context. Working binary downloads are very
>> important too, not everyone clones and builds.
>
> This is maddening.

Why is this so maddening? Are you out of room on your hard drive? It
took too long to build? It offends your sense of aesthetics?

I'm not arguing that it's not a nice goal to have, but is it worth the
cost? In particular, right now where we're finally trying to get a
functioning Sage out the door with a more modern package layout and
revision control system in place?

Personally, my sweet spot would be something like nix, where you
rebuild everything unless the versions are *exactly* the same, for
total reproducibility, and this is handled behind the scenes for you
with a well-founded package manager.

> All other projects would do like this:
> (1) check if git is available ;
> (2) check if the available git has a recent enough version (for other
> things than git, a full equality on the version might make sense) ;
> (3) in case it's not a good git, either error out (hard dep) or build a
> good one (embedded source dep).
>
> Only sage won't look around for what is there, and will actively spread
> doctests around to check the embedded version was really used (there are
> places where exact versions are checked just for that!) ; and will also
> on some occasions *patch* the embedded version and use the variation
> somewhere else for no other reason than making using upstream break
> (example: disabling threading in ecl was done in the spkg instead of
> calling the api to disable it in ecl.pyx).
>
> "Beware of the subtle bugs!" say the sage developers. The spkg-ed
> versions have subtle bugs, so using them doesn't magically solve
> anything.

The "subtle bugs" are sometimes in the interface, not the
(necessarily) package. I know the sage-5.12 sources are well tested
against cython 0.19.1, a specifically patched version of maxima
5.19.1, and pari 2.5.4. It cuts out debugging the whole "you got an
error/different answer than me because you were using version X
instead of version Y..."

While I think it'd be ideal to have a distribution that uses the
system packages when available, I think (1) that's a much harder
problem that shouldn't be solved while moving to git and (2) then we
have to deal with determining what possible sets of dependencies are
compatible. (Existing package mechanisms make this easier, we will be
leveraging them.) But even then it'd be hard to approach the ease of
use of shipping everything that "just works" together; a huge part of
Sage's value is that it takes many obscure math packages and actually
makes them build and run and be tested on a variety of different
platforms. It's often been cited that Sage is the easiest way to
get/use X, where X is some arcane piece of Math software that is
awesome at what it does but is unlikely to be packaged up and
maintained like gmp or git with good versioning and any promise of
backwards compatibility. Even some of the (more) mainstream packages
like maxima that are not intended to be used as a library make it
difficult to swap out one version for the next (slight changes in the
UI are easy for humans to cope with, less so for computers) or need
hacks to safely return error messages rather than calling abort() when
things go wrong.

> The rest of the world knows about the subtle bugs, and deals
> with them. They even deal with them better precisely because having
> people use the code in more various settings makes them easier to
> understand and fix.
>
> To make a comparison which should go directly to the heart of quite a
> few people around here: when you want to use a theorem, you check the
> hypotheses then call it by its name; you don't fall down to ZFC and do
> the full climb upwards!

If theorems had version numbers, we would likely use them. (Actually,
some of them do, e.g. "strong" vs "weak" and they are often, though
not always, referred to as such when required.)

- Robert

[Michael Orlitzky]

On 11/12/2013 12:29 PM, Robert Bradshaw wrote:
>> This is maddening.
>
> Why is this so maddening? Are you out of room on your hard drive? It
> took too long to build? It offends your sense of aesthetics?

Yes, yes, and yes. But there's more:

* Bundling libraries are terrible for security. I should be able to
open a PNG in code that I myself wrote, without worrying about a
buffer overflow. My distro patches these holes immediately. Sage
doesn't, there's just not enough manpower.

* It's wasted thousands of hours of developer time. Look at the
ChangeLog for a major release -- how many of the tickets are math?
How many are a result of trying to bootstrap the universe?

* It doesn't work. At the boundary between stuff-we-build and stuff-
the-system-provides, things break. Sage uses the dvipng built
against my system's gd, but runs it against the one bundled with
sage. *crash* Guess I can't plot anything today.

* Distributions won't touch it. Linux users should be our bread and
butter, but they can't install it. I should be able to tell people
"just type 'yum install sage'", but instead, I have to teach them
how to initiate a big bang the result of which, after millions of
seconds, might look like sage. Then they run into the aforementioned
problems anyway.

[Ralf Hemmecke]

On 11/12/2013 07:00 PM, Michael Orlitzky wrote:
> Yes, yes, and yes. But there's more:

Oh... I only wondered why Sage from github wants to install its own git.
But this thread seems to develop into a discussion of some people that
like a monolithic we-redo-all-ourselves sage and some that like more if
sage builds on underlying software that comes with the distribution.

That nearly sounds like a good reason to fork. ;-)

Ralf

PS: Maybe I shouldn't have started typing make on a system without
gettext. :-(

[Abdó Roig-Maranges]

Hi

mic...@orlitzky.com writes:

> * Bundling libraries are terrible for security. I should be able to
> open a PNG in code that I myself wrote, without worrying about a
> buffer overflow. My distro patches these holes immediately. Sage
> doesn't, there's just not enough manpower.
>
> * It's wasted thousands of hours of developer time. Look at the
> ChangeLog for a major release -- how many of the tickets are math?
> How many are a result of trying to bootstrap the universe?
>
> * It doesn't work. At the boundary between stuff-we-build and stuff-
> the-system-provides, things break. Sage uses the dvipng built
> against my system's gd, but runs it against the one bundled with
> sage. *crash* Guess I can't plot anything today.
>
> * Distributions won't touch it. Linux users should be our bread and
> butter, but they can't install it. I should be able to tell people
> "just type 'yum install sage'", but instead, I have to teach them
> how to initiate a big bang the result of which, after millions of
> seconds, might look like sage. Then they run into the aforementioned
> problems anyway.

And some more points:

* If sage started to use what is available on the system, instead of
duplicating effort, probably more linux distros would start packaging it.

* Depending on things instead of bundling them is the standard practice. I'd
even say that the "batteries included" policy is kind of arrogant. Sage can
pull this off only because it is virtually the only software package doing
it. If every software depending on git, or zlib or python shipped their own
version, things would get unbearable pretty fast. In this sense sage is not
being a good neighbour in the system.

* System git and sage git share resources, like the main config file at
~/.gitconfig. What if it becomes incompatible between system and sage's git
versions? The same goes for any other lib or standard tool sage
uses. Embedding the tools within sage looks like shifting away the "different
versions problem" from developer hands to the user.


I realize sage might have kind of unique needs to patch specific software, and
might have hard dependencies on specific versions of certain scientific
libraries. But I believe this policy of bundling everything should not be the
rule, but an exception. Even more so for stable standard stuff like git, zlib,
etc.

If sage finds a bug to a package, it should first try fixing it upstream, then
try to work-around it, and as a last resort, bundle a patched version. That is
the usual way open-source communities operate.

An other issue is having a "batteries included" tarball around. That may be nice
for some, I understand that, but it shouldn't be the only option.

Also, I think sage gets the focus wrong on this issue. Sage should aim at making
the code base as adaptable as possible to different environments, and software
versions, instead of trying to carefully control the environment. In the long
run that should ease sage's own development! for instance when upgrading bundled
python. It is a matter of being aware of what is backwards compatible and what
isn't when writing code. For instance, always use '--porcelain' versions of git
commands, etc.

As I said, it may not be possible to be that adaptable for some libs (so then
bundle them!). But for a non-negligible amount of software sage depends on, I'm
certain it is possible and mostly painless in the long run to be more
adaptable. Autotools is a mature and portable piece of software which aims to do
just that, to adapt the source tree to the particularities of the build
environment before building. A proof that this can be achieved is that virtually
every single open source software except sage manages to do it! Sage is an
awesome piece of software, but is not that special to be different from
everybody else in that respect.

What seems kind of scary is the amount of work needed to do that, but it seems
Felix has already walked a long way in that direction, isn't it?

Abdó Roig.

[William Stein]

This discussion should be on sage-flame.

https://groups.google.com/forum/#!forum/sage-flame

[Julien Puydt]

Le 12/11/2013 12:33, Volker Braun a �crit :

My point was:

(1) sage uses ECL for some things ;
(2) sage doesn't want ECL with threading enabled ;
(3) instead of using the api call to disable threading (a one-liner!),
the spkg had a patch to disable threading in the sage-compiled ecl
(using the past since I think that's now done correctly... or did I
still not find the time to push it? Sigh...).

It's not clear to me what your point about the scope of sage vs an
embedded lisp interpreter was.

Snark

[Robert Bradshaw]

On Tue, Nov 12, 2013 at 11:20 AM, Abdó Roig-Maranges
<abdo...@gmail.com> wrote:
>

> Hi
>
> mic...@orlitzky.com writes:
>
>> * Bundling libraries are terrible for security. I should be able to
>> open a PNG in code that I myself wrote, without worrying about a
>> buffer overflow. My distro patches these holes immediately. Sage
>> doesn't, there's just not enough manpower.
>>
>> * It's wasted thousands of hours of developer time. Look at the
>> ChangeLog for a major release -- how many of the tickets are math?
>> How many are a result of trying to bootstrap the universe?

I think many (most) of these issues are inherent to updating
dependencies and would have occurred whether or not Sage is shipping
the sources themselves.

Writing code that works against, for example, multiple versions of
Python takes extra care, as does testing it. Expand this to many
different libraries and one has a full cross product of possible
failure modes. Couple that with the (like it or not) common pattern of
learning correctness (or to code at all) by "what works on my machine"
and there's value to having a standard environment. Transferring that
to "works on your machine too" *is* harder the more underlying
libraries can vary, and also negatively impacts reproducibility (which
has practical applications for development and academic value).

It's trading one set of issues for another.

> As I said, it may not be possible to be that adaptable for some libs (so then
> bundle them!). But for a non-negligible amount of software sage depends on, I'm
> certain it is possible and mostly painless in the long run to be more
> adaptable. Autotools is a mature and portable piece of software which aims to do
> just that, to adapt the source tree to the particularities of the build
> environment before building. A proof that this can be achieved is that virtually
> every single open source software except sage manages to do it! Sage is an
> awesome piece of software, but is not that special to be different from
> everybody else in that respect.

I think Sage is unique in the number of dependencies that are not (or
likely never will be) maintained by professional developers and
packaged in mainstream distributions (e.g. cliquer, ecm, flint,
genus2reduction, gfx2, gfan, givaro, iml, lcalc, fpll, m4ri, ...) I
agree that there are also a huge number of libraries that are almost
silly to build ourselves (zlib, sqlite, patch, ...) but at least
shipping them has value over making a hard dependency.

> What seems kind of scary is the amount of work needed to do that, but it seems
> Felix has already walked a long way in that direction, isn't it?

To clarify my position, I am fully supportive of such efforts (e.g.
lmonade). Once we move over to git, this can even be a branch with
easy sharing of code back and forth (compared to the way it is now). I
think the two modes can coexist, and am hopeful that Sage can
eventually move more and more this direction. Unless we require all
dependencies to be upstream (that'll never happen) Sage will have to
deal with packaging itself (hopefully via a standard package manager
that, well, it will probably have to ship). Also, I am convinced we
need an incremental system rather than trying to replace the whole
system at once. Most importantly, this should not be a blocker for
moving to git.

- Robert

[Julien Puydt]

Le 12/11/2013 22:12, Robert Bradshaw a �crit :

> Writing code that works against, for example, multiple versions of
> Python takes extra care, as does testing it. Expand this to many
> different libraries and one has a full cross product of possible
> failure modes. Couple that with the (like it or not) common pattern of
> learning correctness (or to code at all) by "what works on my machine"
> and there's value to having a standard environment. Transferring that
> to "works on your machine too" *is* harder the more underlying
> libraries can vary, and also negatively impacts reproducibility (which
> has practical applications for development and academic value).

Write all your code against single versions of
python/boost/cython/whatever, during months. Try to update any of your
so-specific versions, and see everything break at once. That is how
current sage works, and why upgrading core things is hairy.

Instead let people use various versions of everything. At the slightest
unconsidered change, people report immediately that something is amiss,
so you have a wealth of different reports which show if your small code
change is faulty, or if you just hit a bug in some of your deps, and you
have a clue in which of the deps and which precise version is having the
issue.

It's like evolution in species: if your code gets too adapted to a
single environment, it will break at the slightest change.

The sooner it breaks, the better (less to debug). And the wider range of
variants, the better (easier to debug).

Snark on #debian-science