RFC: New Build/Packaging System

[Volker Braun]

This is a RFC for new packaging system for "sage-the-distribution". I've already talked about this with a few of you at the last sage days, but finally it managed to do something about it. The goal is to be:

* Git-aware: use SHA1 hashes instead of timestamps for dependency calculations

* Unified machine-readable package configuration using YAML

* Dependency handling also for optional packages

* Distinction between different types of dependencies: build time, runtime hard/soft, testing.

* Modular, allowing for easy experimentation with per-package backends

* written in pure Python without any dependencies

* doctested for Python 2.6, 2.7, 3.3, and 3.4.

Having all package configuration data accessible in machine-readable form is IMHO the key to managing complexity. There are various possibilities for the file format, but in the end I think YAML is the best choice. It integrates very well with Python, is easy to read/write for humans, and if things go wrong the parser can pin-point the error very well. Downside is that it is not in the Python standard library, but then I don't think we should that let us dictate the file format. We'll just include the Python-only part of PyYAML as a fallback if the OS Python does not have it. 

The entire package configuration will be in a file SAGE_ROOT/build/pkgs/<name>/package.yaml, for example

----------------------------------------

name:

    matplotlib

category:

    standard

source:

    version:  

        1.3.1

    tarball:

        name:     matplotlib-{source.version}.tar.gz

        sha1:     f340378c43c4c3f6219ef9fd84af4ebbe18f8feb

builder:

    type:            SpkgInstallScript

    install_script:  spkg-install

depends:

    build:

        - pkgconf

        - setuptools

    hard:

        - python

        - numpy

        - freetype

        - libpng

        - gdmodule

        - dateutil

        - pyparsing

----------------------------------------

I've been working on an implementation, which you can find at http://trac.sagemath.org/ticket/16483. It is not feature-complete, but can already build the standard packages in dependency order. 

The package manager lives in SAGE_ROOT/build/manager. Eventually, all build-related sage command line switches should just call it:

$ ./build/manager/sage-pkg help

usage: sage-pkg [-h] [--config CONFIG] [--log LOG]

                

                {info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}

                ...

The Sage Package Manager

positional arguments:

  {info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}

    info                Print information about package

    shell               IPython shell

    list                List all packages

    pkg-upgrade-v1      Upgrade package descriptions

    help                Get help

    download            Build package up to the "download" step

    unpack              Build package up to the "unpack" step

    prepare             Build package up to the "prepare" step

    configure           Build package up to the "configure" step

    compile             Build package up to the "compile" step

    check               Build package up to the "check" step

    install             Build package and install

    build               Build everything

    get                 Download tarball/spkg/file

optional arguments:

  -h, --help            show this help message and exit

  --config CONFIG       Builder configuration file

  --log LOG             One of [DEBUG, INFO, ERROR, WARNING, CRITICAL]

$ ./build/manager/sage-pkg info matplotlib

Configuration:

- config.builder.install_script = spkg-install

- config.builder.type = SpkgInstallScript

- config.category = standard

- config.depends.build = 'pkgconf', 'setuptools']

- config.depends.hard = ['python', 'numpy', 'freetype', 'libpng', 'gdmodule', 'dateutil', 'pyparsing']

- config.name = matplotlib

- config.source.tarball.name = matplotlib-1.3.1.tar.gz

- config.source.tarball.sha1 = f340378c43c4c3f6219ef9fd84af4ebbe18f8feb

- config.source.version = 1.3.1

Status: up to date

Installed when: 2 hours ago

[Julien Puydt]

Hi,

Le 15/06/2014 22:59, Volker Braun a écrit :
> This is a RFC for new packaging system for "sage-the-distribution". I've
> already talked about this with a few of you at the last sage days, but
> finally it managed to do something about it. The goal is to be:
>
> * Git-aware: use SHA1 hashes instead of timestamps for dependency
> calculations
> * Unified machine-readable package configuration using YAML
> * Dependency handling also for optional packages
> * Distinction between different types of dependencies: build time, runtime
> hard/soft, testing.
> * Modular, allowing for easy experimentation with per-package backends
> * written in pure Python without any dependencies
> * doctested for Python 2.6, 2.7, 3.3, and 3.4.
>

I don't have much to comment on yet (I'll need to dig further), but I
already have a few questions:

(1) what is the relation between your work and Felix Salfelder's from
last summer, which still hasn't been fully included as far as I know?

(2) does it allow uninstalling packages?

(3) does it allow using system libs/binaries when they're available and
suitable?

Snark on #sagemath

[Volker Braun]

On Sunday, June 15, 2014 10:14:59 PM UTC+1, Snark wrote:

(1) what is the relation between your work and Felix Salfelder's

None.

 

which still hasn't been fully included as far as I know?

Because it doesn't work. There is always trac if you want to know the details.

(2) does it allow uninstalling packages?

Not yet, but it provides at least a framework for separating the package build steps. 

 

(3) does it allow using system libs/binaries when they're available and
suitable?

No, but it'll provide a configuration mechanism that can be used for that.

[François Bissey]

On Sun, 15 Jun 2014 14:40:13 Volker Braun wrote:
> On Sunday, June 15, 2014 10:14:59 PM UTC+1, Snark wrote:
> > (1) what is the relation between your work and Felix Salfelder's
>
> None.
>

No surprise if a bit sad.

> > which still hasn't been fully included as far as I know?
>
> Because it doesn't work. There is always trac if you want to know the
> details.
>
> (2) does it allow uninstalling packages?
>
>
> Not yet, but it provides at least a framework for separating the package
> build steps.
>

So I guess you don't record files installed. Is it at least feasible?

> > (3) does it allow using system libs/binaries when they're available and
> > suitable?
>
> No, but it'll provide a configuration mechanism that can be used for that.

Excellent.

Francois

[Volker Braun]

On Sunday, June 15, 2014 10:42:55 PM UTC+1, François wrote:

So I guess you don't record files installed. Is it at least feasible?

Its definitely something that I want. But in the interest of keeping manageable milestones I'm not going to change the spgk-install scripts for now. Recording the files installed can be added later. This is a RFC, not a RFFR (request for feature requests)

[leif]

Volker Braun wrote:
> This is a RFC for new packaging system for "sage-the-distribution". I've
> already talked about this with a few of you at the last sage days, but
> finally it managed to do something about it. The goal is to be:
>
> * Git-aware: use SHA1 hashes instead of timestamps for dependency
> calculations

? Hashs of what exactly? Modification / installation time of course
matters...

> * Unified machine-readable package configuration using YAML
> * Dependency handling also for optional packages
> * Distinction between different types of dependencies: build time,
> runtime hard/soft, testing.
> * Modular, allowing for easy experimentation with per-package backends
> * written in pure Python without any dependencies
> * doctested for Python 2.6, 2.7, 3.3, and 3.4.

So we're finally making (some) Python a prerequisite -- fine, provided
all (or at least most) packages get built with the system's Python,
i.e., do not (artificially) depend on Sage's Python.

Not sure what "hard" dependencies are supposed to mean -- above you talk
about hard/soft *runtime* dependencies, while at least some of the
"hard" dependencies listed above are definitely *build* time
dependencies (Python, freetype, libpng...).

What about versions in dependencies (i.e., prerequisites)?

(Patch level) version of the Sage package / YAML file?

I'd also add a couple of things from SPKG.txt:

- perhaps at least short (one-line) description
- copyright / license information
- URL of upstream tarball if appropriate
- upstream contact and website
- author(s)
- perhaps (Sage) package maintainers
- list of patches?

> I've been working on an implementation, which you can find at
> http://trac.sagemath.org/ticket/16483. It is not feature-complete, but
> can already build the standard packages in dependency order.
>
>
> The package manager lives in SAGE_ROOT/build/manager. Eventually, all
> build-related sage command line switches should just call it:
>
> $ ./build/manager/sage-pkg help
> usage: sage-pkg [-h] [--config CONFIG] [--log LOG]
>
> {info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
> ...
>
> The Sage Package Manager
>
> positional arguments:
>
> {info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
> info Print information about package
> shell IPython shell

:-) Another (useless?) dependency?

> list List all packages
> pkg-upgrade-v1 Upgrade package descriptions
> help Get help
> download Build package up to the "download" step
> unpack Build package up to the "unpack" step
> prepare Build package up to the "prepare" step
> configure Build package up to the "configure" step
> compile Build package up to the "compile" step
> check Build package up to the "check" step
> install Build package and install
> build Build everything
> get Download tarball/spkg/file
>
> optional arguments:
> -h, --help show this help message and exit
> --config CONFIG Builder configuration file
> --log LOG One of [DEBUG, INFO, ERROR, WARNING, CRITICAL]

Hmmm, 'pkg-upgrade-v1'?

Not immediately clear what the difference between 'get' and 'download'
is, nor what 'build' here is supposed to mean.

> $ ./build/manager/sage-pkg info matplotlib
> Configuration:
> - config.builder.install_script = spkg-install
> - config.builder.type = SpkgInstallScript
> - config.category = standard
> - config.depends.build = 'pkgconf', 'setuptools']
> - config.depends.hard = ['python', 'numpy', 'freetype', 'libpng',
> 'gdmodule', 'dateutil', 'pyparsing']
> - config.name = matplotlib
> - config.source.tarball.name = matplotlib-1.3.1.tar.gz
> - config.source.tarball.sha1 = f340378c43c4c3f6219ef9fd84af4ebbe18f8feb
> - config.source.version = 1.3.1
> Status: up to date
> Installed when: 2 hours ago

Curious what the output is when the installed version differs from the
(officially) available one...

Do we / can we keep package info files for various versions?


-leif

--
() The ASCII Ribbon Campaign
/\ Help Cure HTML E-Mail

[Volker Braun]

On Sunday, June 15, 2014 10:59:57 PM UTC+1, leif wrote:

> * Git-aware: use SHA1 hashes instead of timestamps for dependency
> calculations

?  Hashs of what exactly?  Modification / installation time of course
matters...

No, modification times precisely does not matter. Git does not record timestamps, and switching branches therefore causes needless recompilation even if you switch back to your the branch!

 

Not sure what "hard" dependencies are supposed to mean -- above you talk
about hard/soft *runtime* dependencies, while at least some of the
"hard" dependencies listed above are definitely *build* time
dependencies (Python, freetype, libpng...).

Runtime dependencies are of course required to build. But not the other way round, you don't need to recompile everything because you updated "patch".

What about versions in dependencies (i.e., prerequisites)?

The version is whatever version is in Sage

(Patch level) version of the Sage package / YAML file?

There is no need for patch levels. The dependency computation uses the SHA1 of the git tree object that holds the package dir as the true version. The tarball version is just for the UI.

 

I'd also add a couple of things from SPKG.txt:

Yes, but that can wait for a followup.

 >      shell               IPython shell 

:-)  Another (useless?) dependency?

Not a dependency, just an optional command line interface if IPython is installed in the OS Python.

Hmmm, 'pkg-upgrade-v1'?

Debug, ignore.

Do we / can we keep package info files for various versions? 

There is always "git checkout"

[François Bissey]

On Sun, 15 Jun 2014 15:29:58 Volker Braun wrote:
> On Sunday, June 15, 2014 10:59:57 PM UTC+1, leif wrote:
> > > * Git-aware: use SHA1 hashes instead of timestamps for dependency
> > > calculations
> >
> > ? Hashs of what exactly? Modification / installation time of course
> > matters...
>
> No, modification times precisely does not matter. Git does not record
> timestamps, and switching branches therefore causes needless recompilation
> even if you switch back to your the branch!
>
> > Not sure what "hard" dependencies are supposed to mean -- above you talk
> > about hard/soft *runtime* dependencies, while at least some of the
> > "hard" dependencies listed above are definitely *build* time
> > dependencies (Python, freetype, libpng...).
>
> Runtime dependencies are of course required to build. But not the other way
> round, you don't need to recompile everything because you updated "patch".
>

No. One example: scipy is a runtime dependency of sage but not a build time.
You can build sage without scipy being installed. Same with maxima.

[Volker Braun]

So how about naming them compile/hard/soft/test, is that clearer?

I don't think we need separate (only) run-time dependencies...

[Chris Kees]

HI Volker,

If you haven't already looked at it, you might be interested  in the hashdist project: https://github.com/hashdist/hashdist.   I've included a few more comments below. I will be at sage days this week.

On Sun, Jun 15, 2014 at 3:59 PM, Volker Braun <vbrau...@gmail.com> wrote:

This is a RFC for new packaging system for "sage-the-distribution". I've already talked about this with a few of you at the last sage days, but finally it managed to do something about it. The goal is to be:

* Git-aware: use SHA1 hashes instead of timestamps for dependency calculations

hashdist can use git commit ID's and pull source from github. It also uses SHA1 hashes for other types of source archives. hashdist also hashes and caches the builds following the approach described by E. Dolstra (Nix package  manager).

* Unified machine-readable package configuration using YAML

package and stack configuration files are in yaml

 

* Dependency handling also for optional packages

* Distinction between different types of dependencies: build time, runtime hard/soft, testing.

here's and example of the matplotlib package spec:

https://github.com/hashdist/hashstack/blob/master/pkgs/matplotlib/matplotlib.yaml

 

* Modular, allowing for easy experimentation with per-package backends

not sure I understand this point

 

* written in pure Python without any dependencies

written in pure Python with all depedencies bundled.

 

* doctested for Python 2.6, 2.7, 3.3, and 3.4.

I don't think there are any major issues with Python 3, but most of the stacks we are building are Python 2.7.

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Volker Braun]

On Monday, June 16, 2014 5:53:36 AM UTC+1, Chris Kees wrote:

Nix package  manager

I'm aware of that. Going the Nix way means a lot of modifications to how packages are built, much of it at odds with how normal distributions package software. I wasn't the only one who thought that would be a bad idea.

[Volker Braun]

Some questions from playing around with hashdist a bit:

* How do I build packages in parallel?

* I can't build Python on Fedora 20? What are the actual hashdist dependencies?

 
On Monday, June 16, 2014 5:53:36 AM UTC+1, Chris Kees wrote:

* Modular, allowing for easy experimentation with per-package backends

not sure I understand this point

By which I mean: You don't have to drink the cool-aid all the way if you don't want to.. Packages can define alternative builders, (a Python class named by the builder.type key) that handles the package. For example there is a:

--------------------------

name:

    sage_c_library

category:

    standard

source:

    - src/c_lib

        

builder:

    type:     ShellCommands

    workdir:    src/c_lib

    commands:  

        - source ../bin/sage-env && scons -Q install

depends:

    compile:

        - scons

    hard:

        - mpir

        - ntl

        - pari

        - python

--------------------------

[Volker Braun]

On Monday, June 16, 2014 1:44:43 PM UTC+1, Volker Braun wrote:

* How do I build packages in parallel?

Just to clarify, the question is how can I build different packages at the same time. Not: run make -j 10 for each package.

[Aron Ahmadia]

Hi Volker,

Chris pointed me at this thread.  Thanks for taking a look at hashdist.  In response to your questions:

* How do I build packages in parallel?

This isn't currently enabled, but it would not be hard to make an option.  Right now we build up a list of packages that we would like to install, then execute parallel installation on each package.  Since the dependency graphs tend to be fairly narrow for many of our longest environment builds, I suspect it's actually faster to do it this way (but have not measured).  

* I can't build Python on Fedora 20? What are the actual hashdist dependencies?

 What profile are you using and what error are you getting?  The example Python profile we use assumes a few things may already be on the host system to simplify builds (and also verifies that a few of Python's optional components are installed).  This is admittedly not great practice, and in the future, we'll explicitly enumerate the Python package dependencies we assume may be on the host.  One thing I *don't* like doing is building my own openssl in hashdist, since crypto software should really be handled by the operating system, not an overlay package manager.  

I am assuming you don't have sqlite or bzip2 installed on your core Fedora 20 system.  Try this install profile:

  python:

    host: false

    build_with: |

      bzip2, sqlite

Regarding modular packages, hashdist defines the concept of a profile (think of it like a pip requirements.txt) as well as a package (which describes how to build a given package).  Profiles can define parameters, which then modify how a package is built.  Profiles can also specify alternative package providers, such as a BLAS from the system or built by the hoser.

Happy to answer more questions on this mailing list if I'm not derailing your RFC.  From your initial proposal, it does look like you're interested in many of the same features.

-A

[Volker Braun]

For the record, Sage build a lot slower to build if you build packages one after the other.

So hashdist packages can sort of change how they are built by inheriting ("extends:"). But can we override / extend:

* source download (to use the Sage mirror network)

* the hash computation for the source? What if I want to install files/directories directly from the repository?

Is there a way to skip the Nix versioned directories stuff and instead just install directly into the destination tree? Because thats what we do now, and changing the build system and 50 package build scripts at the same time is not really feasible. 

[Aron Ahmadia]

Hi Volker,

Inline-repies this time:

On Mon, Jun 16, 2014 at 1:49 PM, Volker Braun <vbrau...@gmail.com> wrote:

For the record, Sage build a lot slower to build if you build packages one after the other.

That's good to know.  We're doing our best to get hashdist in shape for generating relocatable binary builds.  This should have a much bigger impact on build times than the level of parallelism offered in compiles, since users could specify what packages they want and then just have them downloaded for their given operating system (assuming somebody has taken the time to generate and cache their build dependencies somewhere public).

 

So hashdist packages can sort of change how they are built by inheriting ("extends:"). But can we override / extend:

* source download (to use the Sage mirror network)

Yes.  There's a concept of a source cache where all git repositories and tarballs are stored.  You can have as many mirrors as you like.  I don't have a public one that I can show you right now.  It follows the same directory structure as the local directory source cache (which lives by default in ~/.hashdist/src) and can be simply rsycned up to create.

 

* the hash computation for the source? What if I want to install files/directories directly from the repository?

I don't understand this question.    

Is there a way to skip the Nix versioned directories stuff and instead just install directly into the destination tree? Because thats what we do now, and changing the build system and 50 package build scripts at the same time is not really feasible. 

The 'hit develop' command will copy built trees over into the target profile directory.  I suppose we could modify the develop command so that it allows users to skip the build cache, but it is very convenient to have the individual builds in their own directories for a number of reasons, and we haven't had a request for this particular feature yet.

A

[Ondřej Čertík]

Hi Volker,

I was also pointed to this thread. Aron answered pretty much
everything, so just a few comments:

On Mon, Jun 16, 2014 at 11:49 AM, Volker Braun <vbrau...@gmail.com> wrote:
> For the record, Sage build a lot slower to build if you build packages one
> after the other.
>
> So hashdist packages can sort of change how they are built by inheriting
> ("extends:"). But can we override / extend:
> * source download (to use the Sage mirror network)

Yes. You just modify the url field to your own mirror. Assuming you
didn't modify the tarball,
you don't need to modify the hash. If you modified the tarball, then
you need to update the hash.

> * the hash computation for the source?

There is hash computation to verify the tarball, e.g.:

sources:
- url: http://archive.ipython.org/release/2.0.0/ipython-2.0.0.tar.gz
key: tar.gz:j2b4bd4nn7iorbemnd6a5mxzwbkptn2c


But the hash that comes into the build is computed from a json input
script, that contains
this source "key", then the script that builds the package, then any
dependencies etc.

> What if I want to install
> files/directories directly from the repository?

Could you clarify this question a bit? I am not sure what exactly you
are asking.

> Is there a way to skip the Nix versioned directories stuff and instead just
> install directly into the destination tree? Because thats what we do now,

So we create a profile from these versioned directories, which you can
view as a destination tree.
They contain hardlinks into versioned directories. You use the
destination tree just like a Sage tree.

> and changing the build system and 50 package build scripts at the same time
> is not really feasible.

We plan to eventually port packages that Sage uses. We've already
ported quite a few relevant packages,
e.g.:

https://github.com/hashdist/hashstack/blob/master/pkgs/linbox.yaml
https://github.com/hashdist/hashstack/blob/master/pkgs/ntl.yaml
https://github.com/hashdist/hashstack/blob/master/pkgs/flint.yaml

and more will come in the future. Essentially anyone can just send a PR:

https://github.com/hashdist/hashstack/pulls

Ondrej

P.S. We now also have a more fancy web page: https://hashdist.github.io/

[Volker Braun]

On Monday, June 16, 2014 7:04:51 PM UTC+1, Ondřej Čertík wrote:

Yes. You just modify the url field to your own mirror.

No. I know that I want to download xyz.tar.gz, and I have a list of sage mirrors ranked by speed, and I want to download from the fastest one.

> What if I want to install
> files/directories directly from the repository?  

Could you clarify this question a bit? I am not sure what exactly you
are asking.

I want to install part of Sage (say, under src/c_lib) as a "package" of the build process. How do I do that without creating a tarball? I have a unique hash, namely the git tree blob SHA1 for src/c_lib.

[Ondřej Čertík]

On Mon, Jun 16, 2014 at 12:20 PM, Volker Braun <vbrau...@gmail.com> wrote:
> On Monday, June 16, 2014 7:04:51 PM UTC+1, Ondřej Čertík wrote:
>>
>> Yes. You just modify the url field to your own mirror.
>
>
> No. I know that I want to download xyz.tar.gz, and I have a list of sage
> mirrors ranked by speed, and I want to download from the fastest one.

I don't think we support this yet. Essentially you would have some
path postfix, but then
you want to change the mirror at one place, and all the packages would
be downloaded
from this mirror?

>
>> > What if I want to install
>> > files/directories directly from the repository?
>>
>>
>> Could you clarify this question a bit? I am not sure what exactly you
>> are asking.
>
>
> I want to install part of Sage (say, under src/c_lib) as a "package" of the
> build process. How do I do that without creating a tarball? I have a unique
> hash, namely the git tree blob SHA1 for src/c_lib.

Ah, from a git repository? That's easy, you just use the git hash, see e.g.:

https://github.com/hashdist/hashstack/blob/master/pkgs/ginac.yaml

sources:
- url: git://www.ginac.de/ginac.git
key: git:edfa67d26bac695b5ef9911f3cda3ff50232e35a

And you can use local path as well, e.g.:

sources:
- url: /nh/nest/u/ondrej/repos/ginac
key: git:edfa67d26bac695b5ef9911f3cda3ff50232e35a


Ondrej

[Volker Braun]

On Monday, June 16, 2014 7:27:54 PM UTC+1, Ondřej Čertík wrote:

Ah, from a git repository? That's easy, you just use the git hash, see e.g.:

sources:
- url: /nh/nest/u/ondrej/repos/ginac
  key: git:edfa67d26bac695b5ef9911f3cda3ff50232e35a

I don't want to use the SHA1 of the entire repo, I only want to use a sub-directory of the repository. In other words, it would be nice to have a finer-grained control on recompilation than the entire Sage repository.

[Aron Ahmadia]

Hi Volker,

We have implemented mirror support that does what you want: https://github.com/hashdist/hashdist/blob/master/hashdist/core/source_cache.py#L740

Mirrors are specified in the hashdist config file, not profiles, but I don't think it would be hard to add support for mirrors at the profile-level.  I don't see a reason to specify mirrors at the package level, which is where we tend to put the canonical source.

We don't support sub-commit level Git objects, currently.  At this point, you'd need to depend on a commit that contained your git tree.  The full commit would be unpacked in the build, and you could do a sub-install from there.  I (and the rest of the developers) would probably need to think a bit about the advantages/risks of allowing sub-commit objects to be referenced in package installs.  On my first impression, it seems a little low-level for HashDist's layer of responsibility.  Creating such a subtree commit object is not hard for an advanced Git developer, however...

A

[Volker Braun]

On Monday, June 16, 2014 7:32:06 PM UTC+1, Aron Ahmadia wrote:

At this point, you'd need to depend on a commit that contained your git tree.  The full commit would be unpacked in the build, and you could do a sub-install from there.

In particular its not possible to build from the already-existing git repo? I don't want to have to specify the version of sage-the-library, I just want to build it out of the source tree. The version is the git tree sha1 if the tree is clean, and random / file checksum if not clean. At the same time there we have at least one package that depends on Sage being able to start, so we need to integrate sage in our package management.

[Aron Ahmadia]

On Mon, Jun 16, 2014 at 3:07 PM, Volker Braun <vbrau...@gmail.com> wrote:

In particular its not possible to build from the already-existing git repo? I don't want to have to specify the version of sage-the-library, I just want to build it out of the source tree. The version is the git tree sha1 if the tree is clean, and random / file checksum if not clean. At the same time there we have at least one package that depends on Sage being able to start, so we need to integrate sage in our package management.

I'm working on a feature similar to what you're asking for this week.  I'm programmatically using the functionality in `git stash` to create a commit of the current working tree before launching the build.  It's not implemented on anything besides napkins yet, though :)

A

[Ondřej Čertík]

The actual command is "git stash create", which will not modify any of
your source files,
i.e. if you have uncommitted changes, but returns a hash of a "stash"
commit, that can then
be used in hashdist.

Volker, my understanding is that this would be useful for developing a
package, to be able to quickly
run a build, without committing. But for the end user, you always want
to build from some commit.

I.e. if I am developing my own library, I would first put it into
hashdist (with url to github), and then when developing
a new version, I change the url to my local checkout, and I want be
able to quickly check that it installs and works,
so then I would use this feature.

Ondrej

[Volker Braun]

On Monday, June 16, 2014 8:40:34 PM UTC+1, Ondřej Čertík wrote:

Volker, my understanding is that this would be useful for developing a
package, to be able to quickly
run a build, without committing. But for the end user, you always want
to build from some commit.

Yes, I agree. But our primary aim here is reduce development turnaround time.

[Ondřej Čertík]

Definitely. I just wanted to clarify this. I've wanted this feature
for a long time too, when developing.

Ondrej

[Chris Kees]

Thanks Volker, Aron, and Ondrej. I think I understand most of Volker's questions about HashDist  a lot better now. 

One thing I wanted to clarify is that HashDist uses the abstract idea of functional package management from Nix, but HashDist is not built on the Nix package manager or a rewrite/refactorization of it. In particular, the build recipes for packages are highly customizable and flexible.  We erred on the side of pragmatism in several areas of the design due  to our experience building complex  stacks on HPC platforms.

Another point I wanted  to underscore is that some of the functionality like parallel builds across  packages and integrated, efficient development workflows  have been part of the design since the early stages of HashDist development.   There's just a limit to how fast we can implement features because we all have primary research objectives  in other areas.  -Chris 

[Volker Braun]

I've spent some time looking at hashdist which is probably the closest to what we need, but I don't think its the way to go for us right now. First, Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is fixed its hard to do anything with a real package management system. At the same time, hashdist isn't ready. Its version 0.2, which is incompatible to 0.1, there is no documentation for writing builder plugins / API. And much of the source is, let's say, lightly documented with doctests being almost completely absent. And no parallel build support, ... But in the future we should definitely reconsider that, when these issues have been addressed on both sides.

What I don't like in hashdist is that the packages don't have a clear backing as Python classes in the code, which is related to not being able to customize the heck out of it. There is the school of thought that says you shouldn't be allowed to do anything that doesn't lead to provably correct hash->build maps, and everything must be stateless functional. Thats nice if you want a package manager for end users only, but that is IMHO not good enough for development purposes. I'd rather have to do an occasional "make clean" than an extra 10k file ops every time I test a change. Fundamentally, what does a package define? Its how to get the source code, the version of the source code, and how to turn the source into a binary. Right now, hashdist packages only have a hook into the third part. Why can't we override how hashing works, or where to get the source from? Yes that might allow you to break the promise of provably correct builds, but we are all adults here. As long as you build stuff in my home directory I ought to be able to break anything I want.

Not only is package == builder Python object a clear API to override all aspects of the package build, you then can't help yourself but load them all into an IPython session. I found that to be a really nice mechanism to debug and explore the whole packaging:

 

$ ./build/manager/sage-pkg shell

Python 2.7.5 (default, Feb 19 2014, 13:47:28) 

Type "copyright", "credits" or "license" for more information.

IPython 0.13.2 -- An enhanced Interactive Python.

?         -> Introduction and overview of IPython's features.

%quickref -> Quick reference.

help      -> Python's own help system.

object?   -> Details about 'object', use 'object??' for extra details.

Loaded packages: atlas, autotools, boehm_gc, boost_cropped, bzip2, cddlib, cephes, cliquer, configure, conway_polynomials, cryptominisat, cvxopt, cython, database_cremona_ellcurve, database_gap, database_stein_watkins, database_stein_watkins_mini, database_symbolic_data, dateutil, docutils, dot2tex, ecl, eclib, ecm, elliptic_curves, fflas_ffpack, flint, flintqs, freetype, gap, gap_packages, gcc, gd, gdmodule, genus2reduction, gf2x, gfan, git, git_trac, givaro, glpk, gmp, graphs, gsl, iconv, iml, ipython, jinja2, jmol, lcalc, libfplll, libgap, libm4ri, libm4rie, libogg, libpng, libtheora, linbox, lrcalc, matplotlib, maxima, mcqd, mercurial, mpc, mpfi, mpfr, mpir, mpmath, ncurses, networkx, ntl, numpy, openssl, palp, pari, pari_galdata, pari_seadata_small, patch, pexpect, pillow, pkgconf, pkgconfig, polybori, polytopes_db, ppl, pycrypto, pygments, pynac, pyparsing, python, r, ratpoints, readline, rpy2, rubiks, sage_c_library, sage_mode, sage_noarch, sage_python_library, sagenb, sagetex, scipy, scons, setuptools, singular, six, sphinx, sqlalchemy, sqlite, symmetrica, sympow, sympy, tachyon, termcap, topcom, tornado, zlib, zn_poly

In [1]: ppl.config

Out[1]: 

Configuration:

- config.builder.check_script = spkg-check

- config.builder.install_script = spkg-install

- config.builder.type = SpkgInstallScript

- config.category = standard

- config.depends.hard = ['mpir', 'glpk']

- config.name = ppl

- config.source.tarball.name = ppl-1.1.tar.bz2

- config.source.tarball.sha1 = f76fbc2d374170771fed030b79a5ffac08d907bf

- config.source.version = 1.1

In [2]: type(ppl)

Out[2]: sage_pkg.package.spkg_install.SpkgInstallScript

In [3]: type(ppl).mro()

Out[3]: 

[sage_pkg.package.spkg_install.SpkgInstallScript,

 sage_pkg.package.sage_environment_mixin.SageEnvironmentMixin,

 sage_pkg.package.sage_mirror_mixin.SageMirrorMixin,

 sage_pkg.package.base.PackageBase,

 object]

In [4]: ppl.version_stamp

Out[4]: 'c2b217a0d2c918132a0af6189d5f9a74bce4c41f'

In [5]: ppl.get_all_dependencies()

Out[5]: (mpir, glpk)

In [6]: ppl.get_all_dependencies()[0] is mpir

Out[6]: True

In [7]: ppl.download()

http://www.sagemath.org/packages/upstream/ppl/ppl-1.1.tar.bz2

[............................................................]

In [8]: ppl.unpack()

In [10]: ppl.prepare()

In [11]: ppl.install()

====================================================

ppl

****************************************************

Host system:

Linux localhost.localdomain 3.14.4-200.fc20.x86_64 #1 SMP Tue May 13 13:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

****************************************************

C compiler: gcc

C compiler version:

...

make[1]: Leaving directory `/home/vbraun/Code/sage/local/var/tmp/sage-build/ppl/ppl-1.1/m4'

Successfully installed ppl

Finished installing ppl.spkg

In [12]: 

[Ondřej Čertík]

Hi Volker,

Thanks for considering hashdist. Few comments:

On Tue, Jun 17, 2014 at 8:33 AM, Volker Braun <vbrau...@gmail.com> wrote:
> I've spent some time looking at hashdist which is probably the closest to
> what we need, but I don't think its the way to go for us right now. First,
> Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is
> fixed its hard to do anything with a real package management system.

We currently use RPATH, and plan to move to $ORIGIN, so that the build
is relocatable.

https://github.com/hashdist/hashdist/issues/13

But $ORIGIN is not implemented yet. This is another show stopper, as
Sage needs to be relocatable.

> At the
> same time, hashdist isn't ready. Its version 0.2, which is incompatible to
> 0.1,

Yes, it is important to get the design right, and sometimes it is not
backwards compatible. That will obviously improve.

> there is no documentation for writing builder plugins / API.

I created an issue for it:

https://github.com/hashdist/hashdist/issues/247

But can you please clarify what you mean by "builder plugin"?

> And much
> of the source is, let's say, lightly documented with doctests being almost
> completely absent.

https://github.com/hashdist/hashdist/issues/248

> And no parallel build support,

https://github.com/hashdist/hashdist/issues/246

> ... But in the future we
> should definitely reconsider that, when these issues have been addressed on
> both sides.

Thank you for giving us feedback. I have marked all these issues with "sage",
so that you can conveniently view them at one place:

https://github.com/hashdist/hashdist/issues?labels=sage

Is there anything else missing?

>
> What I don't like in hashdist is that the packages don't have a clear
> backing as Python classes in the code, which is related to not being able to
> customize the heck out of it. There is the school of thought that says you
> shouldn't be allowed to do anything that doesn't lead to provably correct
> hash->build maps, and everything must be stateless functional. Thats nice if
> you want a package manager for end users only, but that is IMHO not good
> enough for development purposes. I'd rather have to do an occasional "make
> clean" than an extra 10k file ops every time I test a change. Fundamentally,
> what does a package define? Its how to get the source code, the version of
> the source code, and how to turn the source into a binary. Right now,
> hashdist packages only have a hook into the third part. Why can't we
> override how hashing works, or where to get the source from? Yes that might
> allow you to break the promise of provably correct builds, but we are all
> adults here. As long as you build stuff in my home directory I ought to be
> able to break anything I want.

So what you are missing is that you want to modify one line in openssl,
and you know that it won't change anything in the other 50+ packages
that depend on it
(e.g. Python, numpy, scipy, ...), so you want to be able to simply
build openssl,
but keep the hash, so that it does not trigger a rebuild of the rest
of the stack?

Or also something else?

That's a nice feature.

In general I agree with you, it's always better to write something
that exactly solves the problem that you have at hand, than using a
more general package. That's what Sage did, and that's what all the
HPC distributions did etc. It fixes the particular problem that the
team was facing, but usually it's not very applicable for other
people. With hashdist, we are trying to not only fix this for us, but
also make it so that it can be useful by other people. Essentially,
there is a lot of common things, that all of us need, and then there
are always project specific features, so we want to make hashdist work
in a way, so that you can take it, and build something specific on top
of it. So we appreciate all the feedback, that you gave us.

If you remember anything else for which I didn't create an issue,
please let us know.

Ondrej

[dagss]

On Tuesday, June 17, 2014 4:33:24 PM UTC+2, Volker Braun wrote:

I've spent some time looking at hashdist which is probably the closest to what we need, but I don't think its the way to go for us right now. First, Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is fixed its hard to do anything with a real package management system. At the same time, hashdist isn't ready. Its version 0.2, which is incompatible to 0.1, there is no documentation for writing builder plugins / API. And much of the source is, let's say, lightly documented with doctests being almost completely absent. And no parallel build support, ... But in the future we should definitely reconsider that, when these issues have been addressed on both sides.

There's quite a lot of nosetests and everything was definitely developed by unit tests...I'd argue that doctests vs. nosetests depends a little bit on taste and a lot of what kind of code (and how complicated test fixtures) you need.

I won't contest that code documentation is light. That's mainly my fault, and mainly because Hashdist was (perhaps still is) an experiment -- it doesn't make so much sense to document things thoroughly if they may change at any moment, and the number of coders touching the code are only a couple. This must definitely change.

 

What I don't like in hashdist is that the packages don't have a clear backing as Python classes in the code, which is related to not being able to customize the heck out of it. There is the school of thought that says you shouldn't be allowed to do anything that doesn't lead to provably correct hash->build maps, and everything must be stateless functional. Thats nice if you want a package manager for end users only, but that is IMHO not good 

enough for development purposes. I'd rather have to do an occasional "make clean" than an extra 10k file ops every time I test a change. Fundamentally, what does a package define? Its how to get the source code, the version of the source code, and how to turn the source into a binary. Right now, hashdist packages only have a hook into the third part. Why can't we override how hashing works, or where to get the source from? Yes that might allow you to break the promise of provably correct builds, but we are all adults here. As long as you build stuff in my home directory I ought to be able to break anything I want.

One of the reasons Hashdist came about was actually because "we're all adults" and to remove the strictness of Nix. Perhaps it's still too strict, but at least we said the exact same thing about Nix at one point :-)

A lot of what you say is fully correct, and is simply an effect of Hashdist still being in development and still being developed. Many of the things you asked for are almost there (e.g., package builds already happen breadth-first and turning that into parallel is a mostly trivial matter). Others may simply be that we didn't think about them, and I think you'll find we're very open to somebody coming in and change how things are done. We are definitely not purists. You can definitely build a stack relying solely on LD_LIBRARY_PATH. And so on.

We very much have the same goal here. What you have to ask yourself is: Will starting from scratch get you where you want faster than building on and improving Hashdist. Either way, there's work to be done.

If you are interested in cooperating perhaps the quickest way to move forward is a Google Hangout session; I think a lot of this is really about what software architecture would fit the package manager of Sage, and that all the smaller issues discussed so far in the thread is more on the background noise level (as they are things easily changed about Hashdist), and it may be difficult to kick the discussion to the level we need in an email thread.

Dag Sverre 

[Aron Ahmadia]

Just a general +1 to Dag and Ondrej's comments.

I should take the blame (and responsibility) for places where documentation or testing is weak.  hashdist itself has quite good coverage in its core, and we've been running nightly builds of the Proteus stack on a number of operating systems with it for the last 6 months (self-issued certificate, sorry): https://proteus.usace.army.mil/buildbot/waterfall

A

[Volker Braun]

Good idea, whats a good time? 

I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT.

https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g

[Volker Braun]

Since nobody showed up, maybe you can suggest when? Do you have time during the week?

cc to hashdist list in case somebody hasn't heard of this thread yet ;-)

[François Bissey]

I was available 55mn in but just the two of us wouldn't have done much without
someone from hashdist.
I would have liked to be present because I several interests in this.
Next time tomorrow is not possible for me as I will be in a meeting.
But generally that kind of time means morning in my time zone.
Of course if you put it on Friday it will be Saturday morning for me and that
will suck big time.

Francois

[William Stein]

On Tue, Jun 17, 2014 at 4:33 PM, François Bissey
<francoi...@canterbury.ac.nz> wrote:
> I was available 55mn in but just the two of us wouldn't have done much without
> someone from hashdist.
> I would have liked to be present because I several interests in this.
> Next time tomorrow is not possible for me as I will be in a meeting.
> But generally that kind of time means morning in my time zone.
> Of course if you put it on Friday it will be Saturday morning for me and that
> will suck big time.
>

Chris Kees gave a 45 minute talk on hashdist today at Sage Days, which
I video'd. I uploaded the video to youtube here:

http://youtu.be/no4MyXn4Uik

It's still processing right now.

> Francois
>
> On Tue, 17 Jun 2014 16:25:08 Volker Braun wrote:
>> Since nobody showed up, maybe you can suggest when? Do you have time during
>> the week?
>>
>> cc to hashdist list in case somebody hasn't heard of this thread yet ;-)
>>
>> On Tuesday, June 17, 2014 5:25:23 PM UTC+1, Volker Braun wrote:
>> > Good idea, whats a good time?
>> >
>> > I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT.
>> >
>> > https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g
>> >
>> > On Tuesday, June 17, 2014 4:30:18 PM UTC+1, dagss wrote:
>> >> If you are interested in cooperating perhaps the quickest way to move
>> >> forward is a Google Hangout session; I think a lot of this is really
>> >> about
>> >> what software architecture would fit the package manager of Sage, and
>> >> that
>> >> all the smaller issues discussed so far in the thread is more on the
>> >> background noise level (as they are things easily changed about
>> >> Hashdist),
>> >> and it may be difficult to kick the discussion to the level we need in an
>> >> email thread.
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Chris Kees]

I'm mostly available tomorrow (Wednesday) and Thursday pretty much anytime after 7AM pacific. -Chris

--
You received this message because you are subscribed to the Google Groups "hashdist" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+u...@googlegroups.com.

[Volker Braun]

How about Thursday, then? I'd suggest again 23:00 BST = 15:00 PDT = 22:00 CEDT. That might give the kiwis a chance, too.

[Aron Ahmadia]

I can do Thursday at 1500.  Sorry for not declining the original invite. 

A

[Volker Braun]

Ok, lets go for Thursday 22:00 UTC!

Hangout event: https://plus.google.com/events/c7mko935v5tijbld59ig7kgl1n4

[Dag Sverre Seljebotn]

Can we do two hours earlier? I'm on CEST (Oslo) and it appears this is
midnight for me.

I'm basically available rest of today and tomorrow, 08-23 CEST.

Dag Sverre

On 06/18/2014 01:44 PM, Volker Braun wrote:
> How about Thursday, then? I'd suggest again 23:00 BST = 15:00 PDT =
> 22:00 CEDT. That might give the kiwis a chance, too.
>
>
>
> On Wednesday, June 18, 2014 2:19:08 AM UTC+1, Chris Kees wrote:
>
> I'm mostly available tomorrow (Wednesday) and Thursday pretty much
> anytime after 7AM pacific. -Chris
>
>
> On Tue, Jun 17, 2014 at 4:25 PM, Volker Braun <vbrau...@gmail.com
> <javascript:>> wrote:
>
> Since nobody showed up, maybe you can suggest when? Do you have
> time during the week?
>
> cc to hashdist list in case somebody hasn't heard of this thread
> yet ;-)
>
>
> On Tuesday, June 17, 2014 5:25:23 PM UTC+1, Volker Braun wrote:
>
> Good idea, whats a good time?
>
> I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT.
>

> https://plus.google.com/__events/__cff6ldm40v1skgq5eeslmhg4j5g

> <https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g>
>
>
> On Tuesday, June 17, 2014 4:30:18 PM UTC+1, dagss wrote:
>
> If you are interested in cooperating perhaps the
> quickest way to move forward is a Google Hangout
> session; I think a lot of this is really about what
> software architecture would fit the package manager of
> Sage, and that all the smaller issues discussed so far
> in the thread is more on the background noise level (as
> they are things easily changed about Hashdist), and it
> may be difficult to kick the discussion to the level we
> need in an email thread.
>
>
> --
> You received this message because you are subscribed to the
> Google Groups "hashdist" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to hashdist+u...@googlegroups.com <javascript:>.

>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "hashdist" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to hashdist+u...@googlegroups.com

> <mailto:hashdist+u...@googlegroups.com>.

[Aron Ahmadia]

I've got a standing commitment at 1700 EDT / 2100 UTC, but

I believe Dag is proposing 1300 PDT / 1600 EDT / 2000 UTC / 2200 CEST

Does 20:00 UTC work for everyone?

A

an email to hashdist+unsubscribe@googlegroups.com
<mailto:hashdist+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "hashdist" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hashdist+unsubscribe@googlegroups.com.

[Volker Braun]

20:00 UTC is fine with me!

I've updated the event: https://plus.google.com/events/c7mko935v5tijbld59ig7kgl1n4

[François Bissey]

On Wed, 18 Jun 2014 10:21:40 Volker Braun wrote:
> 20:00 UTC is fine with me!
>
> I've updated the event:
> https://plus.google.com/events/c7mko935v5tijbld59ig7kgl1n4
>

I may do a quick appearance at the beginning, it is 8am for
me and I have to prepare the kids for school and go to work
myself.

Francois

[Volker Braun]

I've made a proof-of-concept conversion of the Sage packaging system to hashdist:

https://github.com/vbraun/sagestack

Obviously its not working, but you can build third-party packages and maybe play around with hashdist.

[Volker Braun]

PS: I just remembered that this is using file globs, so you need hashdist from this branch: https://github.com/vbraun/hashdist/tree/files_glob. Though it should be merged upstream soon.

[Aron Ahmadia]

Wow, this is really impressive work!

A

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/52VPaADq0pU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.

To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.

[Aron Ahmadia]

Also, I just merged Volker's pull request in to <#> master.  You shouldn't need to use his hashdist branch.

A