Sage "and" Anaconda

[kcrisman]

I don't know if this is "really" using Sage in Anaconda, but anyway a poster on ask.sagemath has come up with something that might be of interest to those who have talked about this distribution in the past:

http://ask.sagemath.org/question/32741/integrate-sage-jupyter-notebook-in-anaconda-python-distribution/?answer=32757#post-id-32757    

[Erik Bray]

No, I don't think it really has anything to do with Anaconda
specifically. In fact, by copying the kernel spec file into their
local share directory, really their solution will apply to whatever
Python distribution they're running the Jupyter notebook out of.
Their solution isn't a bad one actually. It does mean they have two
different Pythons running--the one in Anaconda and the one installed
with Sage--but that's not a big deal. If I understand how Jupyter
kernels work now, there would be two Python processes running anyways.
At worst, you're duplicating the number of Python module sources that
are cached in memory by using two separate Python distributions.

There's also no reason they need to use Anaconda just to get Jupyter.
As you point out it comes with sage now. But even if it didn't they
could `pip install jupyter notebook` into $SAGE_LOCAL (using the pip
that comes with sage) and run that notebook installation all the same.

It should also technically be possible (I haven't tried it yet, but I
intend to) to install sage into a conda environment by setting
$SAGE_LOCAL to the path to that environment. My bet is that there are
unknown bumps in the path, but that would be the general idea.

I feel like it may be poorly understood that sage provides its own
Python distribution.

Erik

[kcrisman]

I feel like it may be poorly understood that sage provides its own
Python distribution.

You've got that right! 

[William Stein]

My impression is that Anaconda is basically meant to solve the same
problem as "Sage the distribution", except that (1) it was started
later, (2) has way more funding, (3) is focused on scientific
software, (4) they are fully committed to it being a package manager,
whereas often Sage developers will say our package manager is not a
package manager, as an excuse for it not being very good in some ways.

Thus: it would be really cool to investigate further integration with
or leveraging of Anaconda for Sage! It would be cool if you or
anybody else did so.

-- William

>
> I feel like it may be poorly understood that sage provides its own
> Python distribution.
>
> Erik
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

[Erik Bray]

Hi William,

On Thu, Mar 10, 2016 at 4:43 PM, William Stein <wst...@gmail.com> wrote:
> On Thu, Mar 10, 2016 at 7:34 AM, Erik Bray <erik....@gmail.com> wrote:
>> It should also technically be possible (I haven't tried it yet, but I
>> intend to) to install sage into a conda environment by setting
>> $SAGE_LOCAL to the path to that environment. My bet is that there are
>> unknown bumps in the path, but that would be the general idea.
>
> My impression is that Anaconda is basically meant to solve the same
> problem as "Sage the distribution", except that (1) it was started
> later, (2) has way more funding, (3) is focused on scientific
> software, (4) they are fully committed to it being a package manager,
> whereas often Sage developers will say our package manager is not a
> package manager, as an excuse for it not being very good in some ways.

You are absolutely correct on points (1) through (3). (4) is half
right and half opinion, though I don't disagree with the opinion part
:) (granted it's not a fair comparison either but it is what it is).

> Thus: it would be really cool to investigate further integration with
> or leveraging of Anaconda for Sage! It would be cool if you or
> anybody else did so.

Agreed--I think in principle it would be the easiest way to get Sage
on Windows and OSX. The major roadblock would be getting all of
Sage's existing dependencies packaged for conda*. Many of them are
already there, many aren't. And of course that says nothing for those
dependencies that don't (yet) work natively on the platforms that
conda supports.

Erik


* A point of clarification for those who are unfamiliar with
conda/Anaconda. Anaconda itself is a scientific Python distribution,
and didn't necessarily start as a package management system either.
However, the way Anaconda was built required some kind of package
management anyways, much akin to Sage the distribution--that is--it
was necessary to manage the dependencies between software included in
Anaconda, how to build and install all that software, and the
occasional patches for platform support and build issues.

The Anaconda project quickly split out the "package management" aspect
into a separate project called just "conda", which is both the package
manager and the name of the command one uses to control the package
manager. Since then "conda" has become the core of Anaconda, and is
in many senses more fundamental. Now, when one installs "Anaconda",
they are installing "conda" along with a conda meta-package called
"Anaconda" that consists of Python (2 or 3) and the core scientific
Python software that originally made up the Anaconda distribution.
But one can package just about any other code for conda and install it
in addition to the "Anaconda" core software.

P.P.S. This is probably something Sage-the-distribution should have
done long ago, but I know I'm far from the first to say that, and
totally understand that the resources haven't been there. It's not a
trivial undertaking.

It might almost be worth rebuilding Sage's packaging on top of the
conda platform, so that we can eliminate a lot of the overhead of
developing a packaging system alongside sage. But the devil is of
course in the details, and I could also understand any wariness over
depending on a third-party system, over which we don't have full
control, for Sage-the-distribution's packaging.

[Jeroen Demeyer]

On 2016-03-10 17:12, Erik Bray wrote:
> It might almost be worth rebuilding Sage's packaging on top of the
> conda platform, so that we can eliminate a lot of the overhead of
> developing a packaging system alongside sage. But the devil is of
> course in the details, and I could also understand any wariness over
> depending on a third-party system, over which we don't have full
> control, for Sage-the-distribution's packaging.

Never having used conda, some quick questions in case you know the answer:
1) Does conda support the same operating systems as Sage?
2) Does conda deal with non-Python packages too?
3) Does conda support building packages in parallel?

Do you consider conda superior than the Sage package system? Do you know
of problems that conda has but Sage doesn't?


Jeroen.

[Erik Bray]

On Thu, Mar 10, 2016 at 5:19 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-03-10 17:12, Erik Bray wrote:
>>
>> It might almost be worth rebuilding Sage's packaging on top of the
>> conda platform, so that we can eliminate a lot of the overhead of
>> developing a packaging system alongside sage. But the devil is of
>> course in the details, and I could also understand any wariness over
>> depending on a third-party system, over which we don't have full
>> control, for Sage-the-distribution's packaging.
>
>
> Never having used conda, some quick questions in case you know the answer:
> 1) Does conda support the same operating systems as Sage?

No--this is one of the "details" that would count out sole reliance on
something like conda right now. Most significantly out of the Fully
Supported and Expected to Work columns in [1] being the Solaris
variants. A question I have is for how long does Sage intend to fully
support those platforms? There's nothing particular about conda that
would preclude supporting Solaris that I'm aware of (it doesn't rely
deeply on kernel-specific features or anything like that). Its lack
of support for Solaris is probably mostly due to lack of demand.

Anaconda (and subsequently conda) has been intended mostly for
individuals working on their personal / work computers, as installing
a full scientific Python stack was otherwise difficult for those
users. So it focused primarily on the OS's the majority of
researchers are using for their day-to-day work, less so on servers
managed by IT professionals.

> 2) Does conda deal with non-Python packages too?

Yes, that was one of the primary motivations behind developing their
own packaging system, rather than just relying on pip-installing into
virtualenvs. It also predated the standardization of the "wheel"
format, just barely. But it's well known that the wheel format is
still insufficient for certain types of binary dependencies.

> 3) Does conda support building packages in parallel?

That I don't know. The main challenge to building packages in
parallel is just ensuring that all of a package's dependencies are
built first, right? I honestly don't know how Continuum builds their
full stack. Worth asking though.

> Do you consider conda superior than the Sage package system? Do you know of
> problems that conda has but Sage doesn't?

I don't have deep enough experience with either to have a strong
opinion on the first question. I'll suggest a few points though:
Perhaps the single greatest advantage I can think of is the support
for multiple, simultaneous installation environments a la virtualenv.
If I had to guess this should be possible with Sage too by changing
around $SAGE_LOCAL and rebuilding, but I haven't tried that. You can
probably answer for me how simple that is. The advantage to conda
environments though is nothing needs to be rebuilt to be installed in
multiple environments--the existing binary packages can just be
unpacked to different locations, and there are explicit command-line
programs for managing environments.

You can see more about the package format itself here:
http://conda.pydata.org/docs/build_tutorials/pkgs2.html#about-conda-build-recipe
As one would expect, it's not terribly different from what goes into
an spkg (there are only so many obvious ways to do this). I suppose
one advantage it does have (which you've seen me agitate for) is
slightly more structured metadata (it uses yaml, which means a yaml
parser needs to be bootstrapped before any other package can be
built). But this is a mostly superficial point, I think.

I don't know what specific problems conda has, much less that sage
doesn't. I haven't built any large collections of packages for conda
so I haven't personally tested its limits. I know other people who
are happy with it though.

To me the biggest reason for bringing up the idea of using it is less
whether one is better than the other, and more how much we can save
time and effort by leveraging an existing project. To be clear, I'm
not saying the existing effort has been misspent by any means. conda
didn't even exist until a couple years ago. It also doesn't have as
wide platform support.

Best,
Erik

[1] https://wiki.sagemath.org/SupportedPlatforms

[William Stein]

On Thu, Mar 10, 2016 at 8:44 AM, Erik Bray <erik....@gmail.com> wrote:
> On Thu, Mar 10, 2016 at 5:19 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> On 2016-03-10 17:12, Erik Bray wrote:
>>>
>>> It might almost be worth rebuilding Sage's packaging on top of the
>>> conda platform, so that we can eliminate a lot of the overhead of
>>> developing a packaging system alongside sage. But the devil is of
>>> course in the details, and I could also understand any wariness over
>>> depending on a third-party system, over which we don't have full
>>> control, for Sage-the-distribution's packaging.
>>
>>
>> Never having used conda, some quick questions in case you know the answer:
>> 1) Does conda support the same operating systems as Sage?
>
> No--this is one of the "details" that would count out sole reliance on
> something like conda right now. Most significantly out of the Fully
> Supported and Expected to Work columns in [1] being the Solaris
> variants. A question I have is for how long does Sage intend to fully
> support those platforms? There's nothing particular about conda that
> would preclude supporting Solaris that I'm aware of (it doesn't rely
> deeply on kernel-specific features or anything like that). Its lack
> of support for Solaris is probably mostly due to lack of demand.

We supported Solaris because a customer specifically paid us to
provide such support. Those days are long past. We could drop
Solaris support and I think one person in the world (David Kirkby)
would notice.

> Anaconda (and subsequently conda) has been intended mostly for
> individuals working on their personal / work computers, as installing
> a full scientific Python stack was otherwise difficult for those
> users. So it focused primarily on the OS's the majority of
> researchers are using for their day-to-day work, less so on servers
> managed by IT professionals.

Yes, that's a difference with Sage -- the Solaris support did come
from requirements of IT professionals...

>
>> 2) Does conda deal with non-Python packages too?
>
> Yes, that was one of the primary motivations behind developing their
> own packaging system, rather than just relying on pip-installing into
> virtualenvs. It also predated the standardization of the "wheel"
> format, just barely. But it's well known that the wheel format is
> still insufficient for certain types of binary dependencies.

Their motivation (for conda) was pretty similar to my motivation for
Sage's packaging system.

>
>> 3) Does conda support building packages in parallel?
>
> That I don't know. The main challenge to building packages in
> parallel is just ensuring that all of a package's dependencies are
> built first, right? I honestly don't know how Continuum builds their
> full stack. Worth asking though.

(I can't imagine that they would be against adding such functionality if it
isn't there.)

>> Do you consider conda superior than the Sage package system? Do you know of
>> problems that conda has but Sage doesn't?
>
> I don't have deep enough experience with either to have a strong
> opinion on the first question. I'll suggest a few points though:
> Perhaps the single greatest advantage I can think of is the support
> for multiple, simultaneous installation environments a la virtualenv.
> If I had to guess this should be possible with Sage too by changing
> around $SAGE_LOCAL and rebuilding, but I haven't tried that. You can
> probably answer for me how simple that is. The advantage to conda
> environments though is nothing needs to be rebuilt to be installed in
> multiple environments--the existing binary packages can just be
> unpacked to different locations, and there are explicit command-line
> programs for managing environments.

Another relevant thing: It is supported by a company that got over
$20m in funding just in the last year
(https://mattermark.com/companies/continuum.io).

>
> You can see more about the package format itself here:
> http://conda.pydata.org/docs/build_tutorials/pkgs2.html#about-conda-build-recipe
> As one would expect, it's not terribly different from what goes into
> an spkg (there are only so many obvious ways to do this). I suppose
> one advantage it does have (which you've seen me agitate for) is
> slightly more structured metadata (it uses yaml, which means a yaml
> parser needs to be bootstrapped before any other package can be
> built). But this is a mostly superficial point, I think.
>
> I don't know what specific problems conda has, much less that sage
> doesn't. I haven't built any large collections of packages for conda
> so I haven't personally tested its limits. I know other people who
> are happy with it though.
>
> To me the biggest reason for bringing up the idea of using it is less
> whether one is better than the other, and more how much we can save
> time and effort by leveraging an existing project.

+1

> To be clear, I'm
> not saying the existing effort has been misspent by any means. conda
> didn't even exist until a couple years ago. It also doesn't have as
> wide platform support.
>
> Best,
> Erik
>
> [1] https://wiki.sagemath.org/SupportedPlatforms
>

[Erik Bray]

On Thu, Mar 10, 2016 at 5:52 PM, William Stein <wst...@gmail.com> wrote:
> On Thu, Mar 10, 2016 at 8:44 AM, Erik Bray <erik....@gmail.com> wrote:
>> On Thu, Mar 10, 2016 at 5:19 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>>> On 2016-03-10 17:12, Erik Bray wrote:
>>> 3) Does conda support building packages in parallel?
>>
>> That I don't know. The main challenge to building packages in
>> parallel is just ensuring that all of a package's dependencies are
>> built first, right? I honestly don't know how Continuum builds their
>> full stack. Worth asking though.
>
> (I can't imagine that they would be against adding such functionality if it
> isn't there.)

One other note on this question-- Sage already leverages the existing
parallel build capability of make to handle this. I don't think
there's anything deep to it, though Jeroen can correct me if there are
details I'm missing.

Even if there weren't some existing parallel build capability for
conda packages, it would at most a couple hours to write (at least a
prototype of) something that parses out the dependency tree for a
given conda package, and writes it out to a Makefile template with the
appropriate rules set up.

Erik

[Jeroen Demeyer]

After a quick look at conda, one major difference is that its focus is
really on binary packages. It might not be easy to support the Sage
"build from source" model in conda.

Of course, we could build binary conda packages for Sage, but that's not
the same as using conda as package manager within Sage.

Jeroen.

[Jeroen Demeyer]

On 2016-03-10 18:08, Erik Bray wrote:
> One other note on this question-- Sage already leverages the existing
> parallel build capability of make to handle this. I don't think
> there's anything deep to it, though Jeroen can correct me if there are
> details I'm missing.

It's not just a matter of running make in parallel, it's also a matter
of conda supporting parallel installation. For example, setuptools
breaks when installing multiple packages at the same time. We had to
patch setuptools in Sage to implement a lock to serialize it.

[Erik Bray]

On Thu, Mar 10, 2016 at 6:10 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> After a quick look at conda, one major difference is that its focus is
> really on binary packages. It might not be easy to support the Sage "build
> from source" model in conda.

Right...

> Of course, we could build binary conda packages for Sage, but that's not the
> same as using conda as package manager within Sage.

Could you or someone point me to some more resources as to how this is
used in practice? As I see it the main use cases are somewhat limited
(though important!) such as rebuilding your BLAS library and anything
that depends on it. Although this is mostly a developer activity it's
not solely a developer activity either.

When one rebuilds a package in the Sage source tree it still gets
installed into $SAGE_LOCAL--is that all that different then? The only
difference that I see is skipping a step of packing into a tarball and
compressing. It's a fairly trivial intermediate step, and one that
could probably be cut out entirely if the install target is specified
at build time.

I guess more generally what I'm interested to learn more about is what
the typical development workflow is of people who work regularly on
Sage...

[William Stein]

On Thu, Mar 10, 2016 at 9:10 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> After a quick look at conda, one major difference is that its focus is
> really on binary packages. It might not be easy to support the Sage "build
> from source" model in conda.

This change in focus could be a big plus. For example, do you prefer
Debian Linux or Gentoo Linux?

Sage's packaging was only build-from-source originally, since that was
all I could figure out how to do. Binary was just too hard. I tried.

William


>
> Jeroen.

[Volker Braun]

On Thursday, March 10, 2016 at 5:19:39 PM UTC+1, Jeroen Demeyer wrote:

Never having used conda, some quick questions in case you know the answer:
1) Does conda support the same operating systems as Sage?

I'd say yes, plus Windows (though not all conda packages support windows, or can compile the bundled windows binary)

Not sure about Solaris but then we haven't tested compiling Sage on Solaris in a long time, so its not going to work either regardless of what you find on some wiki page.

2) Does conda deal with non-Python packages too?

Yes

 

3) Does conda support building packages in parallel?

Its a real package manager, so you always only compile a single package (given its prerequisites). What you do in the background is up to you.

The package builder isn't really that central, the package format is just tarred-up binaries with instructions for how to patch paths to shorter paths.

Do you consider conda superior than the Sage package system?

Pretty much everything is superior to the Sage package system. If you can't uninstall then its not a package system in my book, for starters.

Do you know
of problems that conda has but Sage doesn't? 

Afaik conda binaries don't distinguish different linux distributions beyond 32/64 bit, so you need a special build system with an ancient glibc. Which introduces various performance and security issues if you want to have a single Sage conda binary repo, so I'm not super excited about that especially if its a service that you potentially open up to the internet.

But just using the conda package manager to manage Sage-the-distribution is definitely something that we should think about. Our current build system sucks so badly precisely because we don't have any distinction between build and install. Regardless of how we build our binaries, we should look into using conda for installing and managing the built binaries. In particular, going back in the git history should just uninstall the current version of packages and reinstall cached binaries. Instead of recompiling everything from scratch.

[Erik Bray]

From the look of that patch, that's an artifact of using easy_install
to install Python packages. We should definitely switch over to using
pip to install Python packages, but that's a separate issue.

conda could in theory have a similar sort of issue if there are any
files it needs to write to when a package is installed into an
environment. A question worth looking more into--thanks for
clarifying.

[William Stein]

One thing it is not right now is: "setup a virtualenv with just the
Sage library" and go at it. I very much wish this was supported,
since a lot of people use SageMathCloud and want to do sage dev, but
it's way too heavy weight. If it were as simple as a virtualenv, it
would be a lot easier.

So please consider both what the workflow is now, and what it could be
in the future if we were able to expand our developer base.

-- William


--
William (http://wstein.org)

[Erik Bray]

On Thu, Mar 10, 2016 at 6:20 PM, Volker Braun <vbrau...@gmail.com> wrote:
> On Thursday, March 10, 2016 at 5:19:39 PM UTC+1, Jeroen Demeyer wrote:
>>
>> Never having used conda, some quick questions in case you know the answer:
>> 1) Does conda support the same operating systems as Sage?
>
>
> I'd say yes, plus Windows (though not all conda packages support windows, or
> can compile the bundled windows binary)
>
> Not sure about Solaris but then we haven't tested compiling Sage on Solaris
> in a long time, so its not going to work either regardless of what you find
> on some wiki page.

What, you mean everything I read on a wiki isn't currently accurate? ;)

Erik

[Erik Bray]

Same. If all of Sage-the-library's dependencies are provided by the
system (a BIG if) then this is all that's really needed to hack on the
sage Python package in principle. It would certainly be my preference
but that's just because it's what I'm used to, not out of any value
judgment (though it's also what most Python developers in general are
used to).

> So please consider both what the workflow is now, and what it could be
> in the future if we were able to expand our developer base.

+1 though I wouldn't want to step on too many toes either. I've been
on both sides of that before in other projects so I know how it
goes... :)

[Jeroen Demeyer]

On 2016-03-10 18:19, William Stein wrote:
> For example, do you prefer
> Debian Linux or Gentoo Linux?

I absolutely hate Debian and their "we only package software from 5
years ago" philosophy.

I use Gentoo myself.

[Michael Orlitzky]

On 03/10/2016 11:19 AM, Jeroen Demeyer wrote:
>
> Never having used conda, some quick questions in case you know the answer:
> 1) Does conda support the same operating systems as Sage?
> 2) Does conda deal with non-Python packages too?
> 3) Does conda support building packages in parallel?
>
> Do you consider conda superior than the Sage package system? Do you know
> of problems that conda has but Sage doesn't?
>

Someone recently pointed me at Conda thanks to an article I wrote. I
researched it for a few hours so that I could compare it to Gentoo
prefix (for the guy who pointed it out to me).

1. Conda itself is cross-platform, but some packages don't support
some platforms. No difference there.

2. Conda can handle packages written in any language. For binaries,
that's easy -- they just give you a binary and you run it. The
story for source packages is a little more dubious.

3. The build scripts for source installs are *very* rudimentary, so
you can make individual packages build in parallel, but as far as
I know it can't build two different packages simultaneously.

There are four main reasons a "package manager" will fail:

a. It doesn't support building from source.

b. It bundles all of its dependencies.

c. It only works with one language.

d. It's tied to one platform.

Conda more or less passes all four, and there are only two other systems
(Nix and Gentoo prefix) that do. Its main issue is that all of the build
scripts are incredibly naive. For example, the source-based install
routing for PostgreSQL is,

https://github.com/conda/conda-recipes/blob/master/postgresql-9.3/build.sh

It also has few packages compared to Nix or Gentoo. Building packages
from source is complicated, and there are a ton of corner cases and
weird features that you need to support. It also quickly becomes
necessary to have some way of sharing code.

A lot of very smart people have put a lot of time into thinking about
these things and https://dev.gentoo.org/~ulm/pms/head/pms.html is the
result. If you try to start a new source-based package manager, you're
only going to be making mistakes that somebody else already made ten
years ago.

So the tl;dr is, Conda looks promising because they haven't made any of
the stupid mistakes (a) - (d) that immediately doom a package manager.
But they're still 15 years behind. Why start over?

[Jeroen Demeyer]

On 2016-03-10 19:23, Michael Orlitzky wrote:
> Its main issue is that all of the build
> scripts are incredibly naive. For example, the source-based install
> routing for PostgreSQL is,
>
> https://github.com/conda/conda-recipes/blob/master/postgresql-9.3/build.sh

I consider the fact that build scripts are simple a feature, not a bug.
That's much better than all the "cruft" that is going into the Sage
spkg-install scripts.

Jeroen.

[Jeroen Demeyer]

On 2016-03-10 18:19, William Stein wrote:

> On Thu, Mar 10, 2016 at 9:10 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> After a quick look at conda, one major difference is that its focus is
>> really on binary packages. It might not be easy to support the Sage "build
>> from source" model in conda.
>
> This change in focus could be a big plus.

We really should not change this focus, Sage should always support
building from source. Reasons:
1) easily allow development on Sage packages
2) performance (binaries don't know which SSE/AVX instructions your
processor has)
3) portability (with binaries, you are always limited to the platforms
that you built binaries for)

[Michael Orlitzky]

I'm not arguing that complex is better than simple, but most of their
build scripts are so simple that they don't work. Error handling is
nonexistent, options are hard-coded, paths are random, the environment
is not sanitized, installed files are not tracked, ABI changes go
unnoticed, CFLAGS are ignored, LDFLAGS are ignored,... all things we've
already figured out.

The Sage spkg build scripts have gotten crazy because there's no
abstraction. We could almost copy the standard ebuild functions
(installing files, patch application, etc.) into Sage and eliminate a
ton of boilerplate if the whole thing weren't so damned Quixotic.

[William Stein]

On Thu, Mar 10, 2016 at 11:10 AM, Michael Orlitzky <mic...@orlitzky.com> wrote:
> On 03/10/2016 01:47 PM, Jeroen Demeyer wrote:
>> On 2016-03-10 19:23, Michael Orlitzky wrote:
>>> Its main issue is that all of the build
>>> scripts are incredibly naive. For example, the source-based install
>>> routing for PostgreSQL is,
>>>
>>> https://github.com/conda/conda-recipes/blob/master/postgresql-9.3/build.sh
>>>
>>
>> I consider the fact that build scripts are simple a feature, not a bug.
>> That's much better than all the "cruft" that is going into the Sage
>> spkg-install scripts.
>>
>
> I'm not arguing that complex is better than simple, but most of their
> build scripts are so simple that they don't work. Error handling is
> nonexistent, options are hard-coded, paths are random, the environment
> is not sanitized, installed files are not tracked, ABI changes go
> unnoticed, CFLAGS are ignored, LDFLAGS are ignored,... all things we've
> already figured out.

Is "we" = "gentoo", "sage", or packaging experts in general?

> The Sage spkg build scripts have gotten crazy because there's no
> abstraction. We could almost copy the standard ebuild functions
> (installing files, patch application, etc.) into Sage and eliminate a
> ton of boilerplate if the whole thing weren't so damned Quixotic.

Interesting.

--
William (http://wstein.org)

[François Bissey]

conda is married to the x86(_64) architecture. No support for power for
you.

There is probably no reason why it is so apart from it being the
dominant market. The installer is x86 only and I haven't seen the
sources anywhere _I_ looked.

Francois

[William Stein]

On Thu, Mar 10, 2016 at 11:58 AM, François Bissey
<francoi...@canterbury.ac.nz> wrote:
> On 03/11/16 07:54, Jeroen Demeyer wrote:
>>
>> On 2016-03-10 18:19, William Stein wrote:
>>>
>>> On Thu, Mar 10, 2016 at 9:10 AM, Jeroen Demeyer
>>> <jdem...@cage.ugent.be> wrote:
>>>>
>>>> After a quick look at conda, one major difference is that its focus is
>>>> really on binary packages. It might not be easy to support the Sage
>>>> "build
>>>> from source" model in conda.
>>>
>>>
>>> This change in focus could be a big plus.
>>
>>
>> We really should not change this focus, Sage should always support
>> building from source. Reasons:
>> 1) easily allow development on Sage packages
>> 2) performance (binaries don't know which SSE/AVX instructions your
>> processor has)
>> 3) portability (with binaries, you are always limited to the platforms
>> that you built binaries for)
>>
>
> conda is married to the x86(_64) architecture. No support for power for
> you.

or ARM? If so, that's a serious drawback.

> There is probably no reason why it is so apart from it being the dominant
> market. The installer is x86 only and I haven't seen the
> sources anywhere _I_ looked.

Are these the sources?

https://github.com/conda

> Francois

[Jeroen Demeyer]

On 2016-03-10 20:58, François Bissey wrote:
> conda is married to the x86(_64) architecture. No support for power for
> you.

Are you saying that conda won't work at all on anything besides x86? Or
just that there are only official binaries for x86?

[Francois Bissey]

There are only binaries for x86. But now Jeroen has pointed me to the sources
so it may be able to support more.
I remember looking for the sources some time ago with the idea that may be I
could use conda on a power7 cluster to manage the installed python packages.
My search at the time turned nothing.
And I didn’t look very hard this morning before posting.

François

[Michael Orlitzky]

On 03/10/2016 02:37 PM, William Stein wrote:
>>
>> I'm not arguing that complex is better than simple, but most of their
>> build scripts are so simple that they don't work. Error handling is
>> nonexistent, options are hard-coded, paths are random, the environment
>> is not sanitized, installed files are not tracked, ABI changes go
>> unnoticed, CFLAGS are ignored, LDFLAGS are ignored,... all things we've
>> already figured out.
>
> Is "we" = "gentoo", "sage", or packaging experts in general?
>

Gentoo "and friends." The package manager specification was originally
written by someone who parted ways with Gentoo. Having a spec made it
possible to create another source-based package manager that could share
the same package repository. There are now three separate
implementations (Portage, Paludis, and Pkgcore) of that specification,
and several distributions using them. So I would include Exherbo,
Funtoo, and even Google's Chrome OS (based on Gentoo) with the people
who are mostly getting things right.

[William Stein]

Could you say more? (I mean about what you might imagine Sage could
do to improve -- both incrementally or dramatically?) I'm sure
everybody really appreciates your experience, and speaking up about
this here.

[Michael Orlitzky]

On 03/10/2016 03:23 PM, William Stein wrote:
>
> Could you say more? (I mean about what you might imagine Sage could
> do to improve -- both incrementally or dramatically?) I'm sure
> everybody really appreciates your experience, and speaking up about
> this here.
>

Oops, I wrote a book:

Dramatically, I really like Burcin's lmonade project. It keeps the "one
big download, everything just works" model, but instead of using a
Sage-only package manager with Sage-only packages, it reuses the
existing Portage package manager and the Gentoo package repository. You
can install it without admin rights, and upgrades are easy. And most
importantly, Sage developers don't have to worry about building and
installing GCC and all of the other necessary evils -- Gentoo developers
do it.

But incremental improvements have more to offer. I think the single best
thing you could do for Sage's popularity is make it installable with
"emerge sage", "apt-get sage", "yum install sage", or "brew install
sage". Users might get a version that's a month old, and they might wind
up with a minor version of gap that you have not personally tested, but
wow is that better than having to try to explain how to build Sage to a
seventy-five-year-old guy from another continent.

On Fedora, Debian, Gentoo, and derivatives that should be possible, and
it is happening slowly. We're no longer patching everything that we use.
We have a ./configure script. Work is being done to pick up system
packages where possible (http://trac.sagemath.org/ticket/20137). If
configure.ac evolves to have a complete list of dependencies and to
search the system for them, that will be a huge hurdle cleared. With it
gone, the Linux distributions will finally be able to package Sage, and
I think most of them will. You already have Debian/Ubuntu, Fedora, and
Gentoo people interested.

For OSX, I hear homebrew is the best option. Slowly we can create
homebrew packages for all of Sage's dependencies, and finally for Sage
itself. (Unfortunately, OSX access is needed to do most of that work).
But in the meantime, the existing build system can remain as a fallback.
Once everything has been packaged, "brew install sage" should work
flawlessly for OSX users as well. That covers basically everyone who is
building Sage right now. And I can even suggest it to people that I
don't want to spend hours helping.

After that work has been completed, the spkgs and sage-install could go.
Updating a Sage depencency at that point will involve a one-line change
to configure.ac, and maybe an update to a homebrew package (depending on
how much of that burden we take on). Windows users get a VM, same as
always, unless someone knows how to package things for Cygwin.

We can still offer big binary packages -- that doesn't have to change --
but building from source should be as easy as building the Sage python
library. You can get all of the dependencies installed easily by
installing Sage with your package manager. If you want to modify one of
Sage's dependencies, that should be possible -- just make your change in
the upstream project and `make install` it to /usr/local which will then
be picked up when you rebuild the Sage library.

What about the failing tests due to outdated dependencies? If there's a
bug in Sage due to a bug in an upstream project, I think we should add
the test case to the upstream project instead of our doctests. That way
if we upgrade the Sage library and not libfoo, the doctests don't start
failing. This isn't critical because most end users won't be running the
tests (we make it possible on Gentoo, though).

Anyway, the best laid plans and all that. Some or all or none of that
could go awry. The first step is to make it possible to use system
libraries instead of building another copy of everything I already have
installed. Once that's done... life finds a way.

[Volker Braun]

On Thursday, March 10, 2016 at 6:23:51 PM UTC+1, William wrote:

One thing it is not right now is: "setup a virtualenv with just the
Sage library" and go at it.    I very much wish this was supported,
since a lot of people use SageMathCloud and want to do sage dev, but
it's way too heavy weight.  If it were as simple as a virtualenv, it
would be a lot easier. 

That sounds like a low-hanging fruit, you just set SAGE_LOCAL and PATH to an existing Sage binary and then install src/setup.py in your virtualenv. I tried and it doesn't work but thats probably just because of some simple path issues. 

In the just-released 7.1.rc0 we make some steps to using pkg-config for library detection (in particular, blas/lapack). Ideally you shouldn't have to set SAGE_LOCAL, all library dependencies should be detected via pkg-config. Then you'd only need to set PATH for the binary dependencies and for a suitable pkg-config. 

[Erik Bray]

On Thu, Mar 10, 2016 at 7:23 PM, Michael Orlitzky <mic...@orlitzky.com> wrote:
> Conda more or less passes all four, and there are only two other systems
> (Nix and Gentoo prefix) that do. Its main issue is that all of the build
> scripts are incredibly naive. For example, the source-based install
> routing for PostgreSQL is,
>
> https://github.com/conda/conda-recipes/blob/master/postgresql-9.3/build.sh

I would note that the build scripts on the conda-recipes repository
are "community supplied" and, it is pointed out, "not necessarily the
same scripts used to build the packages in Anaconda". That is, it's
possible--likely even--that for some packages Continuum has their own
build recipes. That said, I also know that they often start with, or
just use outright recipes from conda-recipes for adding new packages.

> It also has few packages compared to Nix or Gentoo. Building packages
> from source is complicated, and there are a ton of corner cases and
> weird features that you need to support. It also quickly becomes
> necessary to have some way of sharing code.

I don't the the number of packages is a fair comparison. It's not an
operating system--it started out for bundling scientific Python
software and nothing more. But since much of that software had
non-Python dependencies anyways...

> So the tl;dr is, Conda looks promising because they haven't made any of
> the stupid mistakes (a) - (d) that immediately doom a package manager.
> But they're still 15 years behind. Why start over?

Start over with what? It's an apples to oranges comparison.

[Erik Bray]

On Thu, Mar 10, 2016 at 10:45 PM, Michael Orlitzky <mic...@orlitzky.com> wrote:
> On 03/10/2016 03:23 PM, William Stein wrote:
>>
>> Could you say more? (I mean about what you might imagine Sage could
>> do to improve -- both incrementally or dramatically?) I'm sure
>> everybody really appreciates your experience, and speaking up about
>> this here.
>>

> After that work has been completed, the spkgs and sage-install could go.
> Updating a Sage depencency at that point will involve a one-line change
> to configure.ac, and maybe an update to a homebrew package (depending on
> how much of that burden we take on). Windows users get a VM, same as
> always, unless someone knows how to package things for Cygwin.

I don't think this was intentional, but the degree to which this
entire post ignores / dismisses Windows comes off as a bit flippant.
Maybe you didn't know this but I'm working on getting sage and its
dependencies working on Windows. The VM approach has never been a
real viable solution.

Eventually, packaging for Cygwin is certainly possible. There's also
now Chocolatey which is sort of like a "homebrew for Windows".
Finally, and again, the advantage I see with conda is that it works on
all 3 OS's (that's nothing to do with the packages themselves
working--just the system).

That said I agree with everything else you wrote, and the sort of work
that would make it possible to build and use conda packages for sage
is by and large the same work that would make any other packaging
system usable for sage and its dependencies--that is--the work of
decoupling the sage distribution from the packaging / build system it
uses.

> We can still offer big binary packages -- that doesn't have to change --
> but building from source should be as easy as building the Sage python
> library. You can get all of the dependencies installed easily by
> installing Sage with your package manager. If you want to modify one of
> Sage's dependencies, that should be possible -- just make your change in
> the upstream project and `make install` it to /usr/local which will then
> be picked up when you rebuild the Sage library.

I agree. But if I understand correctly William's comment about
"change in focus" is that "build everything from source" is not an
ideal default for the vast majority of people, including developers.

I am new to this project and spent a good amount of time last week
trying to better understand what exactly is happening when Sage is
sitting there taking hours to build on my laptop, and what exactly I'm
building. The example that struck me--that I keep coming back to--is
that everything kicked off by building GNU patch. And I kept thinking
to myself--why the heck are you building patch for me? I already have
a perfectly good patch program--the exact same version in fact.
Looking at the SPKG.txt I can understand why--patch isn't the same on
all systems. But really (especially these days) that should be a
minority case, and Sage should have just used the `patch` I have.

It's a minor example that only takes a tiny bit of time to compile,
but from there I came to understand the large number of other such
dependencies.

*Even if* Sage absolutely required me to have a special version of
patch to build packages from source (and indeed, even assuming I want
to build from source at all), it would have been much quicker to
download, unpack, and install a binary package that has already been
built for me. Yes, for linear algebra libraries and like I may want
to recompile them and their dependencies for my platform. But that
doesn't mean I gain anything from building *everything* (or even most
things) from source. That's why I didn't use gentoo for very long
last time I tried it--I don't have time for that. Neither do most
users.

So anyways, I agree building from source is good to have. But it
doesn't need to be the default.

[Erik Bray]

On Thu, Mar 10, 2016 at 10:57 PM, Volker Braun <vbrau...@gmail.com> wrote:
> On Thursday, March 10, 2016 at 6:23:51 PM UTC+1, William wrote:
>>
>> One thing it is not right now is: "setup a virtualenv with just the
>> Sage library" and go at it. I very much wish this was supported,
>> since a lot of people use SageMathCloud and want to do sage dev, but
>> it's way too heavy weight. If it were as simple as a virtualenv, it
>> would be a lot easier.
>
>
> That sounds like a low-hanging fruit, you just set SAGE_LOCAL and PATH to an
> existing Sage binary and then install src/setup.py in your virtualenv. I
> tried and it doesn't work but thats probably just because of some simple
> path issues.

I will work on this. Is there a Trac issue for this already I can
assign myself to?

[Dima Pasechnik]

On Friday, March 11, 2016 at 11:03:13 AM UTC, Erik Bray wrote:

On Thu, Mar 10, 2016 at 10:45 PM, Michael Orlitzky <mic...@orlitzky.com> wrote:
> On 03/10/2016 03:23 PM, William Stein wrote:
>>
>> Could you say more?  (I mean about what you might imagine Sage could
>> do to improve -- both incrementally or dramatically?)    I'm sure
>> everybody really appreciates your experience, and speaking up about
>> this here.
>>
> After that work has been completed, the spkgs and sage-install could go.
> Updating a Sage depencency at that point will involve a one-line change
> to configure.ac, and maybe an update to a homebrew package (depending on
> how much of that burden we take on). Windows users get a VM, same as
> always, unless someone knows how to package things for Cygwin.

I don't think this was intentional, but the degree to which this
entire post ignores / dismisses Windows comes off as a bit flippant.
Maybe you didn't know this but I'm working on getting sage and its
dependencies working on Windows.  The VM approach has never been a
real viable solution.

how would you deal with GAP/libGAP? (in particular, given that there are basically two

branches now, HPC-GAP and "classic" GAP)...

  

Eventually, packaging for Cygwin is certainly possible.

Cygwin packaging seems to be harder than for MSYS2

(for Cygwin installer is a pain in the neck, IMHO)

well, yes, there is one Sage package that is "dual" in the way you would like 

patch to be, it is gcc.

So this is doable, just needs a fix...

 

It's a minor example that only takes a tiny bit of time to compile,
but from there I came to understand the large number of other such
dependencies.

*Even if* Sage absolutely required me to have a special version of
patch to build packages from source (and indeed, even assuming I want
to build from source at all), it would have been much quicker to
download, unpack, and install a binary package that has already been
built for me.  Yes, for linear algebra libraries and like I may want
to recompile them and their dependencies for my platform.  But that
doesn't mean I gain anything from building *everything* (or even most
things) from source.  That's why I didn't use gentoo for very long
last time I tried it--I don't have time for that.  Neither do most
users.

there is a special mechanics to avoid buildling Atlas when you build  Sage, 

see the installation manual.

Although I imagine it ought to be more the way gcc package is.

[Jeroen Demeyer]

On 2016-03-11 12:03, Erik Bray wrote:
> But really (especially these days) that should be a
> minority case, and Sage should have just used the `patch` I have.

Since recently, I started adding some general mechanism to actually do
this. I.e. make the special case that we currently have for GCC the
general case. So, be patient and it will happen.

We still need to think about how we determine whether the system version
of a package is good enough. For GCC, this is actually about 200 lines
in configure.ac.

For executables, we could probably just look at the output of "PROGRAM
--version" but then we need to decide which versions are good: for
example, does GNU patch 2.6.1 work with Sage?

In general, this is not an easy problem...


Jeroen.

[Michael Orlitzky]

On 03/11/2016 06:03 AM, Erik Bray wrote:
>
> I don't think this was intentional, but the degree to which this
> entire post ignores / dismisses Windows comes off as a bit flippant.
> Maybe you didn't know this but I'm working on getting sage and its
> dependencies working on Windows. The VM approach has never been a
> real viable solution.

Not intentional, I didn't realize anyone was working on it. We do have
to support Windows somehow, and I agree that a VM is silly.

I'm not suggesting that we throw everything out and then start fixing
Linux/OSX while we ignore Windows. The existing build system will always
be there, so nothing will get /worse/. But I can imagine making steady
improvements on Linux/OSX so that eventually sage-install is obsolete
there. I mostly left out Windows because I don't know how to make steady
improvements there.

[Michael Orlitzky]

On 03/11/2016 05:45 AM, Erik Bray wrote:
>
>> It also has few packages compared to Nix or Gentoo. Building packages
>> from source is complicated, and there are a ton of corner cases and
>> weird features that you need to support. It also quickly becomes
>> necessary to have some way of sharing code.
>
> I don't the the number of packages is a fair comparison. It's not an
> operating system--it started out for bundling scientific Python
> software and nothing more. But since much of that software had
> non-Python dependencies anyways...
>
>> So the tl;dr is, Conda looks promising because they haven't made any of
>> the stupid mistakes (a) - (d) that immediately doom a package manager.
>> But they're still 15 years behind. Why start over?
>
> Start over with what? It's an apples to oranges comparison.
>

Start over with "let's figure out how to do cross-platform source-based
package management." Why isn't it a fair comparison? Gentoo prefix, Nix,
and Conda all do the same thing.

[Erik Bray]

Ah, I didn't know about Gentoo Prefix at all. Thanks, that's worth
looking into as well.

I think it will be great to get Sage's dependencies available on as
many downstream packaging systems as possible.

[Travis Scrimshaw]

FWIW, I was able to build from source on Cygwin with 7.0.beta1 + http://trac.sagemath.org/ticket/19894 + http://trac.sagemath.org/ticket/19868. Since #19894 has been merged in, it should just need the latest beta + #19868, but I haven't tried it recently.

Best,
Travis

[Francesco Biscani]

On 11 March 2016 at 14:50, Michael Orlitzky <mic...@orlitzky.com> wrote:

Start over with "let's figure out how to do cross-platform source-based
package management." Why isn't it a fair comparison? Gentoo prefix, Nix,
and Conda all do the same thing.

I have been a Gentoo user for more than 12 years now, and I wholeheartedly agree that - at least on paper - Gentoo prefix seems like a very good fit for what the Sage distribution aims to be. Gentoo devs have been dealing for a long time with all sorts of issues related to compilation from source, including dependency management, library versioning, ABI breakages, major version updates (Python 2 vs 3, Perl, etc.), and so on. IMO, they have done a fantastic job at it.

As a piece of anecdotal evidence, my main workstation at home was built in 2009, and the Gentoo system on it has been updated in a continuous (rolling) fashion ever since. Among others, it has survived:

- various GCC upgrades (from 4.5 -> 5.3, including various libstdc++ ABI changes),

- various breaking Python/Perl/Ruby/etc. updates, including my current setup in which Python 2 and 3 are installed at the same time (all python packages are installed automatically in their Python 2 and 3 versions),

- continuous kernel upgrades,

- various major switches of low-level system components, including ALSA -> pulseaudio, systemV -> openrc-> systemd, and more,

- simultaneous multilib installation of 32-bit and 64-bit packages (in my current setup, most packages are compiled both in 32-bit and 64-bit mode, in order to provide dependencies for binary-only 32-bit packages such as Skype).

A fantastic feature is that whenever an ABI-breaking changes happens in a library, the package manager keeps around the previous version's .so files until all packages depending on it have been updated (then the old files get automatically removed). Reverse dependency checking provides a way to eliminate cruft once the dependencies of a software change. And so on and on. To me, the portage package system is an amazing piece of engineering. Even if the Sage project decides not to use it, I think it would be really beneficial to study how things are done in portage. All other package manager I have used after portage (pip, yum, hashstack, even apt-get) have frankly felt like a child's toy in comparison.

With that said, my experiences with the prefix version of gentoo on OSX have been a bit of a mixed bag. It felt to me like it was very close to be generally viable, but a lot of manual tweaking was still needed. I have no experiences with it on Windows platforms, but I think it does have some support for cygwin.

Cheers,

  Francesco.

[Francois Bissey]

In early 2011 we had a working version of sage on gentoo-prefix on OS X via sage-on-gentoo.
Then the situation degraded as apple “support” for gcc vanished.
At the moment we have a working toolchain based on clang on gentoo prefix but
no working fortran compiler as far as I can see.
I tried to work on bringing a pre made gfortran but definitely lack the time to do so.
So at the moment sage-on-gentoo-prefix on OS X is indeed stopped.

François

[Volker Braun]

As another data point, I always had trouble running gentoo-prefix / lemona.de on Fedora. More often than not it caughed up some error about file ownership/permissions and errored out. 

Fundamentally, I think a user-space package manager like conda or hashdist is sufficiently different from a system-level package manager. E.g. for the former it is perfectly fine to blow away the entire managed file tree and reinstall, which doesn't really work on the system level. The latter has lots of other duties (like file ownership/permissions) which require admin access.

As another example, rpm can also act as user-space package manager. Its just that everybody has long given up on building relocatable rpms and it has long gotten used to having admin access, so its a theoretical possibliity at best. 

[William Stein]

Volker,

Any chance you have a moment to make some remarks about hashdist in
this thread? https://github.com/hashdist/hashdist

-- William

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William (http://wstein.org)

[Volker Braun]

I use hashdist outside of Sage on a daily basis, and its promise of "never compile anything twice" really works. It does bulletproof incremental builds, and the dependency tracking works very well in practice. 

If it weren't for distributing binary builds of Sage we should be using hashdist right now. In hashdist, the "installed package" is a bunch of symlinks back to the build store under ~/.hashdist/. This is why reinstallation is basically instantaneous, if the build is cached only a couple of symlinks are recreated. But you can't just tar up the install directory since its just a bunch of symlinks. You can of course copy files from the build store but they will keep their hardcoded paths and not work on a different computer.

One solution would be to export the hashdist build stores as conda packages to solve binary distribution. It just means that, internally, the build store would use long directory names as padding. There is a writeup at https://github.com/hashdist/hashdist/wiki/HDEP-1:-Crating-System-for-Redistributing-Packages, but its not implemented yet.

[Erik Bray]

On Fri, Mar 11, 2016 at 10:32 PM, Volker Braun <vbrau...@gmail.com> wrote:
> As another data point, I always had trouble running gentoo-prefix /
> lemona.de on Fedora. More often than not it caughed up some error about file
> ownership/permissions and errored out.
>
> Fundamentally, I think a user-space package manager like conda or hashdist
> is sufficiently different from a system-level package manager. E.g. for the
> former it is perfectly fine to blow away the entire managed file tree and
> reinstall, which doesn't really work on the system level. The latter has
> lots of other duties (like file ownership/permissions) which require admin
> access.

I was under the impression from the few minutes of research I did that
you could in principle set a prefix to /home/user/<wherever> and
install into that. But if it's not intended for user-space
installation then forget it.

It's also unclear to me whether or not it supports binary packages? I
don't think I can stress enough that build-from-source cannot and
should not be forced on most users for most applications.

> As another example, rpm can also act as user-space package manager. Its just
> that everybody has long given up on building relocatable rpms and it has
> long gotten used to having admin access, so its a theoretical possibliity at
> best.

Let me tell you all the times I've had to manually unpack an rpm and
fix up RPATHs... :(

> On Friday, March 11, 2016 at 8:14:13 PM UTC+1, François wrote:
>>
>> In early 2011 we had a working version of sage on gentoo-prefix on OS X
>> via sage-on-gentoo.
>> Then the situation degraded as apple “support” for gcc vanished.
>> At the moment we have a working toolchain based on clang on gentoo prefix
>> but
>> no working fortran compiler as far as I can see.
>> I tried to work on bringing a pre made gfortran but definitely lack the
>> time to do so.
>> So at the moment sage-on-gentoo-prefix on OS X is indeed stopped.
>>
>> François
>>
>> > On 12/03/2016, at 04:01, Francesco Biscani <blues...@gmail.com> wrote:
>> >
>> > With that said, my experiences with the prefix version of gentoo on OSX
>> > have been a bit of a mixed bag. It felt to me like it was very close to be
>> > generally viable, but a lot of manual tweaking was still needed. I have no
>> > experiences with it on Windows platforms, but I think it does have some
>> > support for cygwin.
>>

[Volker Braun]

On Tuesday, March 15, 2016 at 10:47:42 AM UTC+1, Erik Bray wrote:

I was under the impression from the few minutes of research I did that
you could in principle set a prefix to /home/user/<wherever> and
install into that.

As I said, gentoo prefix / lmona.de can do that in principle. In practice it often doesn't work though because the "gentoo prefix" == user-space package manager receives much less testing than the gentoo == system-level package manager. Or the wrong kind of testing; gentoo prefix surely works great on top of gentoo but coverage on other distros/OSX/... is bad in my experience.