modularize packages
Ondrej Certik
I am using Qsnake daily, we did 2 releases and I plan to do another one soon.
I think that one big thing, that is still not optimal is handling user
packages. Here are some of the use cases:
1) I want to install Trilinos (nontrivial) + some FEM package +
umfpack, and allow my colleague to run the FEM package and see some
result
2) Install my FEM research codes + all dependencies, some of the
dependencies are highly nontrivial to install (petsc for example)
3) Henrik has some research packages, that he wants to use and install
on computers very easily
Sage solves this by simply including all the packages in Sage. In
Qsnake, so far I also included some packages in the default
installation. But I don't think that's the way, because each of us
will have his own set of packages, and I don't think it scales well to
have everything in the main distribution.
So far I have my custom packages on my github, and then I check them
out, and do:
qsnake install .
which installs using the local spkg-install file. This works
excellent. So one obvious addition is to simply do:
qsnake install https://github.com/certik/package.git
What about dependencies? I think that to make something working now, I
would only allow dependencies for packages from
https://github.com/qsnake, most of them would be optional, but qsnake
would know how to install them (with dependencies).
I would also like to implement to only download packages when they are
needed. Currently everything is downloaded with "qsnake -d" all the
time, and it is wasteful (for example to download VTK+mayavi, if you
don't install it).
The release of Qsnake, or the "qsnake -d"/"qsnake -b" combination
would then contain some "reasonable" set of packages for scientific
computing (some equivalent of ETS + some subset of Sage), something
that most people would want, and also so that it installs quite fast.
Maybe however, we can really just distribute a small qsnake (just
python+git installed), and the user would install everything else on
the fly.
Another thing is with binary packages --- I know how to do it --- I
would make a snapshot of all files in $SPKG_LOCAL, then install a
package, make a snapshot and get all files that are new. Then store
them in a binary package, and the binary package would be stored in
$QSNAKE_ROOT/spkg/binary (for example). For the time being, I don't
know how portable such binaries would be, so I want to concentrate on
good "source installation" first.
To make things usable, I think I really want people to create simple
binaries for their set of packages. So that I can create a binary of
Qsnake + my research codes, for couple architectures, and then people
can just unpack + run it. Example use case: my fiance has a Mac (no
gcc installed) and I would love to just download my code and run it as
binary (using Python).
So given all this, and our manpower, I think that a solid goal could be to:
1) make "qsnake install something" both download and install it (thus
eliminating "qsnake -d" completely)
2) implement "qsnake install qsnake-base" meta-package that will
install some basic configuration, for example numpy+scipy, cmake,
ipython+deps, git, sympy, matplotlib
3) make other meta-packages, like "qsnake install qsnake-sage" (in
principle), or "qsnake install qsnake-fem" (FEM packages) or "qsnake
install qsnake-electronic-structure" (to install electronic structure
packages), or "qsnake install qsnake-ets" (to install ETS packages)
All of the packages about would be hosted at
https://github.com/qsnake/, and we would make sure that things work
well together. We would only accept packages there, that people are
willing to maintain. It would work with dependencies too. Besides
that, to allow to install people's packages:
4) implement "qsnake install
https://github.com/certik/my_custom_package.git", this would work
without dependencies
5) implement "qsnake install
https://github.com/certik/my_custom_package.git branch", this would
install a particular branch, so that you can send people some code to
test, for example when I am trying to make Trilinos work, but I am not
sure if it installs on the Mac, so I just create a git branch
"for_min", and he can test on the Mac using:
qsnake install cmake umfpack gfortran-osx #standard qsnake packages
qsnake install https://github.com/certik/trilinos.git for_min
after we nail all issues with Min, then I create a pull request to
https://github.com/qsnake/trilinos.git, and it will become part of
standard Qsnake. As to my own research code, that might never be part
of Qsnake, it will just be on my certik github.
Let me know if you have any thoughts about this. Do you think it would
fix all the issues, so that people can start using Qsnake for their
research?
Ondrej
Henrik Johansson
Hi all,
For me who only use the base packages of qsnake this sounds like a really good idea.
The only limitation see is that the dependencies and versions still are stored in the monolithic packages.json in qsnake.git. Maybe one could allow external meta-packages that could be installed like in 4) which only contain a packages.json file.
/Henrik
Ondrej Certik
Yes, I think that's the right approach.
>
> The only limitation see is that the dependencies and versions still are
> stored in the monolithic packages.json in qsnake.git. Maybe one could allow
If the package is hosted at github, then there is no version stored in
the packages.json.
> external meta-packages that could be installed like in 4) which only contain
> a packages.json file.
So it would override the spkg/base/packages.json? Or install some
additions to it, something like:
spkg/base/packages.d/your_list.json
(my_list.json, some_otherlist.json) and qsnake would just collect all
these files. This should probably be installed into $SPKG_LOCAL/etc.
Then you can just create a simple meta-package, that installs this
file, and that's it.
Pretty much, packages.json should contain the name of the package,
dependencies, and the location. And this should be completely
independent of qsnake, so that all of us can create our own. Something
like a custom Ubuntu repository with packages.
The builtin packages.json would only contain the standard distribution
packages, that we do maintain inside qsnake (which will be a lot of
packages, just like Ubuntu has pretty much everything, except your own
research code).
Ondrej
henrik johansson
On Wednesday 08 June 2011 03:40:08 Ondrej Certik wrote:
> > The only limitation see is that the dependencies and versions still are
> > stored in the monolithic packages.json in qsnake.git. Maybe one could
> > allow
>
> If the package is hosted at github, then there is no version stored in
> the packages.json.
True, I guess as we discussed over email that it could still be useful to put SHA1 versions in gitted packages when making releases. Then you can rebuild any version of a distribution at any time.
>
> > external meta-packages that could be installed like in 4) which only
> > contain a packages.json file.
>
> So it would override the spkg/base/packages.json? Or install some
> additions to it, something like:
>
> spkg/base/packages.d/your_list.json
> (my_list.json, some_otherlist.json) and qsnake would just collect all
> these files. This should probably be installed into $SPKG_LOCAL/etc.
> Then you can just create a simple meta-package, that installs this
> file, and that's it.
First I was thinking that each spkg package may contain an own packages.json that would be merged with the one in spkg/base but it would probably be better to save them separately as you suggest. Maybe the file could still called packages.json in the spkg-file and then be copied to spkg/base/packages.d/<package name>.json.
> Pretty much, packages.json should contain the name of the package,
> dependencies, and the location. And this should be completely
> independent of qsnake, so that all of us can create our own. Something
> like a custom Ubuntu repository with packages.
> The builtin packages.json would only contain the standard distribution
> packages, that we do maintain inside qsnake (which will be a lot of
> packages, just like Ubuntu has pretty much everything, except your own
> research code).
Yes, exactly. So a qsnake would work like a combination of apt and dpkg in Debian. I hope we are not reinventing something that is already solved before :).
Have a nice weekend!
/Henrik