How does Pkg interact with other software-distribution methods?

[Steven G. Johnson]

Say that I'm a Linux distro. I want to install Julia packages with my
own packaging system (.deb, .rpm, ...) into a system-wide directory
under /usr (e.g. /usr/lib/julia/...). I want versioning and upgrading
of those packages to be handled by the same mechanism as the rest of my
system updates (e.g. apt-get upgrade), *not* by a separate git mechanism.

Say I'm a software distributor of a larger (non-Julia) program like GSL,
which is already installed from a tarball via "./configure && make &&
make install". I want to bundle a Julia interface that is installed
along with my C libraries etcetera, and I want it to be installed by my
'make install' under /usr/local (say /usr/local/lib/julia...)
(installation from source should not touch /usr/lib since that is
controlled by the distro). I don't want a separate git repository or
whatever for the Julia interface, because the Julia interface is tied to
a specific version of my library.

How does this interact with the Pkg system?

A simple possibility would be to make sure Julia searches a documented
standard FHS-compliant path in addition to whereever it looks for
Pkg-controlled stuff. Say $HOME/.julia/...something... then
/usr/local/lib/julia/...something... then
/usr/lib/julia/...something... in that order.

Please don't get so creative with Pkg that you make life difficult for
existing software-distribution schemes.

--SGJ

PS. And have a way to modify this path on installation, in case the
sysadmin wants a different path, e.g. /opt/julia or /usr/packages/julia
or whatever, and provide a way to query julia for the correct place to
install "local" packages.

PPS. And in the longer run, one could contribute some patches to GNU
automake etcetera so that people already using autotools can easily
install julia components of a larger package using the same tool
(similar to how Python is handled now).

[John Myles White]

I may be totally crazy for saying this, but I pretty strongly feel that it should be impossible to install Julia programs using something like apt-get. I've had endless nightmares with this in the past when apt-get style mechanisms and RubyGems fought to the death for control of my system.

-- John

> --
>
>
>

[Alessandro "Jake" Andrioni]

On 4 December 2012 12:07, John Myles White <johnmyl...@gmail.com> wrote:
> I may be totally crazy for saying this, but I pretty strongly feel that it should be impossible to install Julia programs using something like apt-get. I've had endless nightmares with this in the past when apt-get style mechanisms and RubyGems fought to the death for control of my system.

This. I had the same problem with Python and pip/easy_install on
Ubuntu. Versions of packages from six months ago conflicting and
without updates, the horror, the horror.

[Steven G. Johnson]

Steven G. Johnson wrote:
> PS. And have a way to modify this path on installation, in case the
> sysadmin wants a different path, e.g. /opt/julia or /usr/packages/julia
> or whatever, and provide a way to query julia for the correct place to
> install "local" packages.

Note also that, the long run, it is important to allow different
versions of Julia to co-exist on the same system. This means that
search paths must be versioned (e.g. /usr/lib/julia/v1.0), including the
Pkg search path, either by default or by allowing the sysadmin the
option of renaming the path on installation.

(I notice that LOAD_PATH currently uses /usr/share/julia. /usr/share
rather than /usr/lib makes sense since Julia files are currently
arch-independent, but it is important to have a /usr/local/share/julia
in the default search path as well. Also to support versioning and
renaming of the path, and to document a place for user-installed
packages to go, e.g. /usr/share/julia/ or /usr/share/julia/extras/ or ...?)

[Steven G. Johnson]

This is why you need to document a clean way for Pkg to interact with
system (or makefile) installed packages. (Python is not a good model
here!)

Probably the Pkg-controlled stuff should live in its own directory
(under /usr/local somewhere, since /usr is controlled by the distro),
and should take precedence the load path (so that you can install the
bleeding-edge git Pkg in preference to the older package included by the
distro).

Distros will always be slightly out of date (or very out of date if you
use Debian stable). But refusing to work well with distros, which are
the main software-distribution mechanism in the GNU/Linux world, seems
like a bad choice strategically.

--SGJ

[Tim Holy]

Is there a way to get apt-get to make use of Julia's Pkg() system to do the
updates?

Having come from the wild west of Matlab, where you find shiny (or not-so-
shiny) things scattered hither and yon, I don't have much experience with
package management. However, it occurs to me that Steven's point is a good one
when you think about the possibility of "system" applications being written in
Julia. The user may have no idea that their email client was actually written
in Julia (giggle now, but it could happen some day), but s/he might
desperately need an important security update.

--Tim

[Steven G. Johnson]

Tim Holy wrote:
> On Tuesday, December 04, 2012 12:18:57 PM Alessandro Jake Andrioni wrote:
>> On 4 December 2012 12:07, John Myles White <johnmyl...@gmail.com> wrote:
>>> I may be totally crazy for saying this, but I pretty strongly feel that it
>>> should be impossible to install Julia programs using something like
>>> apt-get. I've had endless nightmares with this in the past when apt-get
>>> style mechanisms and RubyGems fought to the death for control of my
>>> system.
>> This. I had the same problem with Python and pip/easy_install on
>> Ubuntu. Versions of packages from six months ago conflicting and
>> without updates, the horror, the horror.
>
> Is there a way to get apt-get to make use of Julia's Pkg() system to do the
> updates?

Many things are possible technically, but I doubt that Debian (or
Fedora, or Ubuntu, or ...) will implement any technique that causes them
to lose control of the versioning, which would defeat the whole point of
a distro.

It doesn't have to be a mess as long as you plan for interoperability in
advance. You just need at least six places in the default load path,
searched in order:

1) user-installed Pkg stuff in $HOME (e.g. $HOME/.julia/<version>/packages)
2) user-installed non-Pkg stuff (e.g. from a Makefile or just copying
files) in $HOME (e.g. $HOME/.julia/<version>/unpackaged)
3) system-wide installed Pkg stuff not controlled by the distro and
hence in /usr/local (e.g. /usr/local/share/julia/<version>/packages)
4) system-wide installed non-Pkg stuff (e.g. from Makefiles of larger
programs) (e.g. /usr/local/share/julia/<version>/unpackaged)
5) distro-installed non-Pkg stuff (e.g.
/usr/share/julia/<version>/unpackaged
6) stuff bundled with Julia (e.g. /usr/share/julia/<version>)

--SGJ

[Tim Holy]

Is 6) for things that are official Julia packages but which have been installed
via the distro? Or do you need to add an item 4.5) for such a category? As
long as the distro packaging goes through Pkg, presumably we could use the
package manager to solve conflicts: as needed, new versions will be added to 1)
and will take precedence.

If it's really that simple, we should definitely do it. I just don't have the
expertise to comment usefully.

--Tim

[Steven G. Johnson]

Tim Holy wrote:
> Is 6) for things that are official Julia packages but which have been installed
> via the distro? Or do you need to add an item 4.5) for such a category? As
> long as the distro packaging goes through Pkg, presumably we could use the
> package manager to solve conflicts: as needed, new versions will be added to 1)
> and will take precedence.

(6) is for things bundled with Julia, e.g. what you already install now
in /usr/share/julia/base etcetera. (In a distro where julia is packaged
by the distro, this would be installed and versioned by the distro in
/usr. If the sysadmin installs julia from source, it should go in
/usr/local/bin and /usr/local/share/julia etcetera instead, by default.)

I'm not sure what you mean by "official" Julia packages. There will be
functions bundled with the julia source code (like elementary matrix
operations), third-party add-ons packaged with your Pkg, and third-party
add-ons not-packaged with Pkg (e.g. by distros or as part of the
makefiles of existing software).

From the sound of the Pkg design, it is unlikely that distros use it
because it basically is an alternative to their existing versioning and
upgrading systems. There is nothing wrong with this as long as long as
you provide a way for Pkg and non-Pkg third-party code to co-exist, e.g.
by living in different load paths.

--SGJ

[Tom Short]

I'm not sure that plays well with git reproducibility. I really like where Stefan's package system is going with respect to reproducibility, and he gets that by using one git directory. I also like how I can change any package in my .julia and develop right in .julia.

That seems at odds with having packages spread all over. Maybe the system packages can get pulled into the user's ~/.julia.

I even like having Julia itself locally installed, so I can mess with that.

- Tom

--SGJ

--

[Steven G. Johnson]

Tom Short wrote:
> I'm not sure that plays well with git reproducibility. I really like
> where Stefan's package system is going with respect to reproducibility,
> and he gets that by using one git directory. I also like how I can
> change any package in my .julia and develop right in .julia.

With any numerical software, if you change to a different version you
cannot in general expect bit-exact reproducibility even on the same
machine with the same compilers. Users understand this, I think, and
every software distribution scheme will include information about the
versions that are being installed.

If you want to tell someone, "To get exactly the same results as me,
don't use your distro's version, use julia version YYY and use Pkg to
install commit XXX of this third-party code," you are still free to do
so. And since my suggested load path priorities Pkg-installed code over
distro-installed code, this will do what you want even if they
previously installed the distro version of the package.

(For that matter, since the Pkg system only controls Julia code and not
external C libraries, even using Pkg will not guarantee
bit-reproducibility if e.g. the user has a different version of FFTW or
OpenBLAS installed.)

> That seems at odds with having packages spread all over. Maybe the
> system packages can get pulled into the user's ~/.julia.

That's not how system-wide software installation works. If you want to
install stuff in your home directory, great, but that is orthogonal to
deciding what to do for software that the distro or sysadmin wants to
install system-wide. (And that is why $HOME/.julia should take
precedence in the load path over system-wide paths: so that users can
override the system-wide stuff with their own preferred versions of
packages.)

I understand the attraction of dictating your own software universe with
its own rules, independent of everyone else's. But interoperability
with the rest of the universe seems like a better strategy for a
language looking to attract developers. And there are a lot of people
out there already attached to other software-distribution schemes.

[Tim Holy]

On Tuesday, December 04, 2012 10:33:40 AM Steven G. Johnson wrote:
> From the sound of the Pkg design, it is unlikely that distros use it
> because it basically is an alternative to their existing versioning and
> upgrading systems. There is nothing wrong with this as long as long as
> you provide a way for Pkg and non-Pkg third-party code to co-exist, e.g.
> by living in different load paths.

Suppose I do this:

apt-cache search email
....

Hmm, that "julia-email-client" looks interesting.

apt-get -u install julia-email-client
<this downloads 14 dependencies, which are other Julia packages, as well>

Test it out. Hmm, it has a few bugs. Oh, look, it's written in Julia, and I
see there has been a lot of development since my distro was released! I wonder
whether they're fixed those bugs?

julia> Pkg.add("email.jl")
julia> Pkg.update()
<only installs newer versions of the 3 dependencies required to run the latest
email.jl; the other 11 dependencies will use apt's version.>

[Jared Kofron]

The only experience I have that is similar to this is homebrew, which will direct you to
install things in the supported 3rd party way:

<snip>

[galois] ~/Downloads $ brew install texlive
Error: No available formula for texlive
Installing TeX from source is weird and gross, requires a lot of patches,
and only builds 32-bit (and thus can't use Homebrew deps on Snow Leopard.)

We recommend using a MacTeX distribution: http://www.tug.org/mactex/

</snip>

I've always appreciated that they just don't even try to get it right, but also let you know that
there is a way to do it and let you know. I'm not sure what the mechanism is for something
like apt (portage can do it with einfo) is for dumping additional post-install info, but that's one
way to get it done.

> --
>
>
>

[Aron Ahmadia]

Steven, et al,

The Python community is well aware of the packaging/distro interaction problem for scientific software packages, and there is a new initiative called hashdist that is trying to make progress on this front.

Your input would definitely be welcome: http://hashdist.readthedocs.org/en/latest/

A

--

[Stefan Karpinski]

On Tue, Dec 4, 2012 at 9:07 AM, John Myles White <johnmyl...@gmail.com> wrote:

I may be totally crazy for saying this, but I pretty strongly feel that it should be impossible to install Julia programs using something like apt-get. I've had endless nightmares with this in the past when apt-get style mechanisms and RubyGems fought to the death for control of my system.

Having system-managed language-level packages is a broken design that someone cooked up because they were confused by the fact that both things have the same name. This should never be done.

I considered this when first starting out on the package project. Jeff very reasonably wanted to know why we couldn't just use apt or yum or something like that. Took me a while to figure out why. The issue is this: os-level packaging systems are os-specific (yes, some are shared, but each distro chooses their own, and OSes almost never share). If we used an os-level package manger, we'd have to repackage every single Julia package for every single os-level package manager. Which is a ton of work and completely unnecessary. If there were a single OS-level package management system that everyone on every OS used, then yes, by all means, we should just hook into that. But there isn't. Instead there are a dozen of them at least.

The language level is actually a very natural breaking point where the os-level package manager stops and a language-specific packaging system begins. Languages have their own conventions about how to lay out files and how to load them. The system manager doesn't know or care about that (or shouldn't). By having a language-level package manager, you only need to package things once instead of once per OS that someone might want to use your code on. This is why every single language has its own system for this: Perl, Java, Python, Ruby, R, etc. This is not an accident or something epically dumb that every single language has independently screwed up.

So my position is that there will be no os-managed Julia packages because it's a terrible idea. The OS can install Julia, but Julia installs Julia packages.

[Stefan Karpinski]

On the other hand, it seems perfectly sane for the Julia package manager to have hooks to use OS-level package managers to install external dependencies like C libraries. The Tk package comes to mind. It would be great if doing Pkg.add("Tk") also hooked into apt/yum/port/homebrew and installed the tk libraries for you.

[Stefan Karpinski]

On Tue, Dec 4, 2012 at 10:48 AM, Steven G. Johnson <ste...@alum.mit.edu> wrote:

Tom Short wrote:

I'm not sure that plays well with git reproducibility. I really like where Stefan's package system is going with respect to reproducibility, and he gets that by using one git directory. I also like how I can change any package in my .julia and develop right in .julia.

With any numerical software, if you change to a different version you cannot in general expect bit-exact reproducibility even on the same machine with the same compilers.  Users understand this, I think, and every software distribution scheme will include information about the versions that are being installed.

If you want to tell someone, "To get exactly the same results as me, don't use your distro's version, use julia version YYY and use Pkg to install commit XXX of this third-party code," you are still free to do so.  And since my suggested load path priorities Pkg-installed code over distro-installed code, this will do what you want even if they previously installed the distro version of the package.

(For that matter, since the Pkg system only controls Julia code and not external C libraries, even using Pkg will not guarantee bit-reproducibility if e.g. the user has a different version of FFTW or OpenBLAS installed.)

Just because getting bit-level reproducibility isn't really practical doesn't mean that we can't try to at least get algorithm-level reproducibility. This argument is a bit like saying "Hey, we'll never get these photons to do the same exact thing again, so why bother documenting our experimental setup at all?" I know the analogy isn't perfect, but it's not too far off.

[Aside: Since Julia knows what versions of libraries it is loading into memory when it runs and their paths, we could conceivably snapshot those and include them in some sort of record of what a setup looked like when an experiment was run.]

That seems at odds with having packages spread all over. Maybe the system packages can get pulled into the user's ~/.julia.

That's not how system-wide software installation works.   If you want to install stuff in your home directory, great, but that is orthogonal to deciding what to do for software that the distro or sysadmin wants to install system-wide.   (And that is why $HOME/.julia should take precedence in the load path over system-wide paths: so that users can override the system-wide stuff with their own preferred versions of packages.)

I'm not convinced that multiple levels of packages are actually a good idea. Which version of that package am I using again? Am I loading the system one or the user one or the one that's in this project? They're all different versions! Every time I've tried to work with this, it ends up being kind of a nightmare. Instead, I always install to system on machines where I have root and I always install to the user on systems where I don't. If I ever have packages installed for the system and the user, then that means at some point I messed up. And now it's just dumb luck that they happen to work together. And sometimes they don't. The end result of that situation is usually me tearing everything out and installing from scratch, so that everything either at the system or the user level, but not both, and I know I actually have a set of packages that work together.

If it's easy enough to manage what packages are installed for a project, I feel like not sharing or layering is by far the best approach. This is a bit like the OS shared library issue. OS X, which is not known for its lack of usability just makes each application have a self-contained set of its own libraries inside the application folder. Does it use a bit more disk space? Sure. But it works without headaches.

[Steven G. Johnson]

Tim Holy wrote:=

> <only installs newer versions of the 3 dependencies required to run the latest
> email.jl; the other 11 dependencies will use apt's version.>

Would be nice, but I don't think it would be terrible to simply install
newer versions of all the dependencies. This might be a lot simpler to
implement since then Pkg doesn't need to look at the unpackaged stuff at
all or try to figure out what Pkg versions they correspond to (if any!
--- distros often apply their own bugfix patches, for better or for worse).

The key point is for Pkg and non-Pkg code to go in different
standardized directories so that they don't fight for control over the
same files, with the Pkg versions taking precedence if both are present.
Also to respect the Unix/FHS/GNU hierarchy (particularly /usr vs.
/usr/local).

--SGJ

[Stefan Karpinski]

On Tue, Dec 4, 2012 at 12:44 PM, Steven G. Johnson <ste...@alum.mit.edu> wrote:

The key point is for Pkg and non-Pkg code to go in different standardized directories so that they don't fight for control over the same files, with the Pkg versions taking precedence if both are present.  Also to respect the Unix/FHS/GNU hierarchy (particularly /usr vs. /usr/local).

100% agree with this.

[Steven G. Johnson]

Stefan Karpinski wrote:
> I'm not convinced that multiple levels of packages are actually a good
> idea. Which version of that package am I using again? Am I loading the
> system one or the user one or the one that's in this project? They're
> all different versions! Every time I've tried to work with this, it ends
> up being kind of a nightmare. Instead, I always install to system on
> machines where I have root and I always install to the user on systems
> where I don't.

Yes, this a sensible policy. But to achieve this you still have to
support both schemes and let the user and sysadmin decide which is
appropriate.

Imagine a multi-user machine where a sysadmin has installed some
packages system-wide, but an individual user wants to experiment with a
new package so she needs to install it in $HOME while still using the
system-wide packages.

> If it's easy enough to manage what packages are installed for a project,
> I feel like not sharing or layering is by far the best approach. This is
> a bit like the OS shared library issue. OS X, which is not known for its
> lack of usability just makes each application have a self-contained set
> of its own libraries inside the application folder. Does it use a bit
> more disk space? Sure. But it works without headaches.

GNU/Linux distros have never and probably will never adopt this policy.
If Julia gets packaged by a distro, it will use the shared version of
FFTW, BLAS, etcetera packaged by the distro and not a Julia-specific
version.

When in Rome, you need to dress like the Romans. Or at least allow the
Romans to enter your house without requiring that they change their
clothes first, to stretch the metaphor.

[Stefan Karpinski]

On Tue, Dec 4, 2012 at 12:53 PM, Steven G. Johnson <ste...@alum.mit.edu> wrote:

Stefan Karpinski wrote:

I'm not convinced that multiple levels of packages are actually a good idea. Which version of that package am I using again? Am I loading the system one or the user one or the one that's in this project? They're all different versions! Every time I've tried to work with this, it ends up being kind of a nightmare. Instead, I always install to system on machines where I have root and I always install to the user on systems where I don't.

Yes, this a sensible policy.  But to achieve this you still have to support both schemes and let the user and sysadmin decide which is appropriate.

Imagine a multi-user machine where a sysadmin has installed some packages system-wide, but an individual user wants to experiment with a new package so she needs to install it in $HOME while still using the system-wide packages.

I would argue that the best approach is actually to clone and modify the system package setup. Git makes cloning a repo locally fast and cheap (it can even share inodes using hard links). If there are compiled resources that would take a while to recompile, some scheme for sharing those is needed too.

This approach would allow users with no special need to just use the system versions but then when they start having "special needs", they get a full copy and are no longer subject to the whims of the sysadmin. If they want to pull upstream changes from the system, that's a simple Pkg.pull operation away.

If it's easy enough to manage what packages are installed for a project, I feel like not sharing or layering is by far the best approach. This is a bit like the OS shared library issue. OS X, which is not known for its lack of usability just makes each application have a self-contained set of its own libraries inside the application folder. Does it use a bit more disk space? Sure. But it works without headaches.

GNU/Linux distros have never and probably will never adopt this policy.  If Julia gets packaged by a distro, it will use the shared version of FFTW, BLAS, etcetera packaged by the distro and not a Julia-specific version.

When in Rome, you need to dress like the Romans.  Or at least allow the Romans to enter your house without requiring that they change their clothes first, to stretch the metaphor.

Agreed. Things like FFTW and BLAS should definitely be installed via the system package manager. Our current approach of downloading and compiling our own versions from source is really a stop gap until we can work with enough different distros well enough to do things that way. To some extent, distros also need to have enough flexibility to install various numerical libraries configured in a way that works, so it's a bit of a meeting in the middle situation.

My thought currently is that each external library that we rely on should have a Julia packages that wraps it – like the FFTW module, except a package. That package would then be responsible for ensuring that the external library exists and is appropriately configured and installed. This could be done via apt or yum or whatever, and as a fallback, through source.

[Steven G. Johnson]

Stefan Karpinski wrote:
> I considered this when first starting out on the package project. Jeff
> very reasonably wanted to know why we couldn't just use apt or yum or
> something like that. Took me a while to figure out why. The issue is
> this: os-level packaging systems are os-specific (yes, some are shared,
> but each distro chooses their own, and OSes almost never share). If we
> used an os-level package manger, we'd have to repackage every single
> Julia package for every single os-level package manager.

False. *You* don't need to repackage anything. You just need to say to
the distros (and other existing software that already uses 'make'):
"Hey, if you want to install your own stuff and not use Pkg, here is
where you should put it."

There is a big distinction between *allowing* people to use some other
software distribution mechanism (and standardizing the path to do so),
and using it yourself.

> Languages have their own conventions about how to lay out files
> and how to load them.

Then why install in /usr/share/julia at all? Why not just install in
/julia? After all, the OS filesystem hierarchy has nothing to do with you.

> So my position is that there will be no os-managed Julia packages
> because it's a terrible idea. The OS can install Julia, but Julia
> installs Julia packages.

I'm not sure what you think you are accomplishing by flipping your
middle finger at distros. Or at existing software packages that are
already built with 'make' and might want to install a Julia interface
when they do 'make install'. Or at MacOS software developers who want
to use the App Store or other existing installation tools. Or at
Windows software developers who want to use the Windows installer. Or
at....

Is this really the battle you want to fight with Julia? When all it
would take would be a few additions to LOAD_PATH?

[Steven G. Johnson]

Stefan Karpinski wrote:
> On the other hand, it seems perfectly sane for the Julia package manager
> to have hooks to use OS-level package managers to install external
> dependencies like C libraries. The Tk package comes to mind. It would be
> great if doing Pkg.add("Tk") also hooked into apt/yum/port/homebrew and
> installed the tk libraries for you.

So the distro manages the tk version, but Pkg manages the julia
interface version? Sounds like a recipe for breakage.

Not to mention that you are then in the business of adding interfaces to
apt/yum/port/homebrew/julia/appstore/microsoft/ rather than letting
distros/packagers do it for you if that is what they want.

[Steven G. Johnson]

Stefan Karpinski wrote:
> This approach would allow users with no special need to just use the
> system versions but then when they start having "special needs", they
> get a full copy and are no longer subject to the whims of the sysadmin.

Requiring every user to recompile their own version of Julia just to use
one package that isn't installed systemwide? Seriously?

Imagine if every language did this. "Oh, you want to use my C library
and it is not installed? First go recompile your own version of gcc.
And use the bleeding-edge git version, by the way. Oh, that version
is not compatible with the old Debian system you are running on? Then
it is simple, just first go compile your own operating system image and
run it in a virtual machine...."

[Stefan Karpinski]

I'm not flipping my middle finger at anyone. People are free to modify LOAD_PATH however they want. However, I think that all those packages I see in port/yum/whatever that are prefixed with p5- or p5.8- or p5.12, py27-, or py32- and so on (there are literally thousands of these on my system) are a bad idea and installing perl or python packages that way seems to generally result in tears.

There's a natural breaking point between language packages and OS packages. The language itself belongs on the OS side while the language's packages belong on the language side. If distros need to install certain language-level packages, e.g., because some external program requires them, then that's ok, but they should probably do it through the language package manager.

--

[Stefan Karpinski]

On Tue, Dec 4, 2012 at 1:13 PM, Steven G. Johnson <ste...@alum.mit.edu> wrote:

Stefan Karpinski wrote:

This approach would allow users with no special need to just use the system versions but then when they start having "special needs", they get a full copy and are no longer subject to the whims of the sysadmin.

Requiring every user to recompile their own version of Julia just to use one package that isn't installed systemwide?  Seriously?

Why would they need to recompile julia? We're talking about cloning Julia packages, not recompiling julia itself. The julia program is fine to install via the OS package manager.

 

Imagine if every language did this.  "Oh, you want to use my C library and it is not installed?   First go recompile your own version of gcc.  And use the bleeding-edge git version, by the way.  Oh, that version is not compatible with the old Debian system you are running on?   Then it is simple, just first go compile your own operating system image and run it in a virtual machine...."

Yes, that would be silly.

[Jameson Nash]

We aren't the first to abandon system package managers for local language code. So rather than just giving my own thoughts here, I thought it would be best to quote the guys over at Node.JS which is used to *gasp* write entire webservers (of which the user has no knowledge :) that need to be kept stable and up-to-date with security too. (In case it is doubt, I prefer that dependancies are not installed by the system). Just my 2 cents.

Why can't npm just put everything in one place, like other package managers?

Not every change is an improvement, but every improvement is a change. This would be like asking git to do network IO for every commit. It's not going to happen, because it's a terrible idea that causes more problems than it solves.

It is much harder to avoid dependency conflicts without nesting dependencies. This is fundamental to the way that npm works, and has proven to be an extremely successful approach. See folders(1) for more details.

If you want a package to be installed in one place, and have all your programs reference the same copy of it, then use the npm link command. That's what it's for. Install it globally, then link it into each program that uses it.

Whatever, I really want the old style 'everything global' style.

Write your own package manager, then. It's not that hard.

npm will not help you do something that is known to be a bad idea.

https://npmjs.org/doc/faq.html

--

[Stefan Karpinski]

On Tue, Dec 4, 2012 at 1:08 PM, Steven G. Johnson <ste...@alum.mit.edu> wrote:

Stefan Karpinski wrote:

On the other hand, it seems perfectly sane for the Julia package manager to have hooks to use OS-level package managers to install external dependencies like C libraries. The Tk package comes to mind. It would be great if doing Pkg.add("Tk") also hooked into apt/yum/port/homebrew and installed the tk libraries for you.

So the distro manages the tk version, but Pkg manages the julia interface version?  Sounds like a recipe for breakage.

Definitely needs some finessing in terms of which direction the dependencies go. My guess would be that it's best to let the OS installed-version dictate what interface version gets installed and use that to determine viable versions for other packages. Not sure about this though.

Not to mention that you are then in the business of adding interfaces to apt/yum/port/homebrew/julia/appstore/microsoft/ rather than letting distros/packagers do it for you if that is what they want.

Calling `apt-get install foo` isn't particularly difficult.

[Steven G. Johnson]

Stefan Karpinski wrote:
> I'm not flipping my middle finger at anyone. People are free to modify
> LOAD_PATH however they want. However, I think that all those packages I
> see in port/yum/whatever that are prefixed with p5- or p5.8- or p5.12,
> py27-, or py32- and so on (there are literally thousands of these on my
> system) are a bad idea and installing perl or python packages that way
> seems to generally result in tears.

If you don't standardize the LOAD_PATH, then it will end in tears
because every distro will pick its own load path to install julia
packages into, and may even just install into the same directory
controlled by Pkg and end up with conflicts.

And history of other source-loading languages shows that you need to
have a language version in the load path (/usr/share/julia/<version>, as
otherwise you make it unnecessarily difficult for people to have
multiple versions of Julia installed. When in the future you release
Julia 2.0 with backward-incompatible syntax changes, people will want to
have both it and Julia 1.0 installed on the same system (the latter so
that they can continue to use programs incompatible with the new
syntax). Again, people can modify the LOAD_PATH themselves, but it is
much better for everyone if this is standardized.

[Stefan Karpinski]

Thanks. NPM has a rather similar design to Pkg. I would add that Julia's LOAD_PATH mechanism isn't and won't be tied to the package manager. If you want to setup Julia packages by hand, or using a different arrangement, then that's completely fine. After all, that's what we've been doing for a while now.

[Stefan Karpinski]

The version number thing might be a good idea. Certainly having some plan for that eventuality is appropriate.

--

[Steven G. Johnson]

Stefan Karpinski wrote:
> Why would they need to recompile julia? We're talking about cloning
> Julia packages, not recompiling julia itself. The julia program is fine
> to install via the OS package manager.

If you just want to clone a package, that is already handled by my
suggestion: $HOME/.julia (or whatever) should indeed take precedence
over /usr/local/share/julia and /usr/share/julia, precisely so that
users can easily overwrite system-installed packages with their own
versions.

[Steven G. Johnson]

Stefan Karpinski wrote:
> So the distro manages the tk version, but Pkg manages the julia
> interface version? Sounds like a recipe for breakage.
>
> Definitely needs some finessing in terms of which direction the
> dependencies go.

"Some finessing" is an understatement, I think.

> Not to mention that you are then in the business of adding

> interfaces to apt/yum/port/homebrew/julia/__appstore/microsoft/

> rather than letting distros/packagers do it for you if that is what
> they want.
>
>
> Calling `apt-get install foo` isn't particularly difficult.

How many dozen of these do you want to add? What if "foo" has a
different name in different packaging systems (e.g. fftw3-dev vs.
fftw3-devel vs. libfftw3-dev vs ....)

[Steven G. Johnson]

Stefan Karpinski wrote:
> Thanks. NPM has a rather similar design to Pkg. I would add that Julia's
> LOAD_PATH mechanism isn't and won't be tied to the package manager. If
> you want to setup Julia packages by hand, or using a different
> arrangement, then that's completely fine. After all, that's what we've
> been doing for a while now.

Past experience with Julia is probably a terrible guide to the future
here. If Julia is successful, the userbase will expand far beyond
people on the mailing list who compile Julia from git.

All I'm suggesting is (a) put standard place(s) in the LOAD_PATH for
non-Pkg code, (b) the load-path should respect the FHS when on Unix
(/usr vs. /usr/local vs. $HOME should all co-exist), and (c) standardize
a language version identifier in the load paths.

The longer you wait to take these simple steps, the more painful it will be.

[John Myles White]

I do think all of those specific things are good changes to make.

-- John

> --
>
>
>

[Jeff Bezanson]

Those changes sound good to me too. Though presumably respecting the
FHS will not aid windows app store integration much :)

> --
>
>
>

[Steven G. Johnson]

Jeff Bezanson wrote:
> Those changes sound good to me too. Though presumably respecting the
> FHS will not aid windows app store integration much :)

Presumably the default path should be different on Windows. Whatever
the path, however, having a designated location to install non-Pkg stuff
will help integration with any other software-distribution scheme.

[Jameson Nash]

I assume default path on windows should probably be derived from %HOME% = %APPDATA%/Julia (where %APPDATA% = 'C:\Users\username\AppData\Roaming', typically) ?

Currently I have %HOME% set to the julia installation directory, which is generally sub-optimal since it may not be writable.

> --
>
>
>

[Sébastien Villemot]

Le mardi 04 décembre 2012 à 13:39 -0500, Steven G. Johnson a écrit :
> Stefan Karpinski wrote:
> > Thanks. NPM has a rather similar design to Pkg. I would add that Julia's
> > LOAD_PATH mechanism isn't and won't be tied to the package manager. If
> > you want to setup Julia packages by hand, or using a different
> > arrangement, then that's completely fine. After all, that's what we've
> > been doing for a while now.
>
> Past experience with Julia is probably a terrible guide to the future
> here. If Julia is successful, the userbase will expand far beyond
> people on the mailing list who compile Julia from git.

Thanks Steven for raising these points concerning the distribution of
Julia packages through GNU/Linux package managers. As a Debian
Developer, and as the packager of Julia in Debian, I very much agree
with what you previously said.

Having the possibility of distributing packages through
apt/yum/pacman/... should not be thought as antagonistic to the Pkg+git
system, but rather as complementary. I think the use cases are
different: developers and power users will want the latest packages and
therefore use Git, while other users will probably prefer their
distribution package manager for stability and convenience.

I actually opened an issue on that very topic (before reading the
present thread), and there I give other reasons for playing nice with
the distribution package manager:

https://github.com/JuliaLang/julia/issues/1692

> All I'm suggesting is (a) put standard place(s) in the LOAD_PATH for
> non-Pkg code, (b) the load-path should respect the FHS when on Unix
> (/usr vs. /usr/local vs. $HOME should all co-exist), and (c) standardize
> a language version identifier in the load paths.

This seems a very sensible plan to me.

--
.''`. Sébastien Villemot
: :' : Debian Developer
`. `' http://www.dynare.org/sebastien
`- GPG Key: 4096R/381A7594

[Stefan Karpinski]

On Thu, Dec 6, 2012 at 12:27 PM, Sébastien Villemot <seba...@debian.org> wrote:

Having the possibility of distributing packages through
apt/yum/pacman/... should not be thought as antagonistic to the Pkg+git
system, but rather as complementary. I think the use cases are
different: developers and power users will want the latest packages and
therefore use Git, while other users will probably prefer their
distribution package manager for stability and convenience.

That makes sense to me but I do worry about the transitional user experience. Someone is using the system-default Julia setup, discovers that there is *one* package that isn't available from the system, switches over to using the package manager and suddenly needs to install all the packages that already worked. Or maybe if you're using the package manager, the default package are just there for the using with no guarantees of compatibility. I mean that is what every other language-level package manager does, although it seems a bit broken. To switch fully over to the package manager, you'd have to make sure that Julia is ignoring system-installed packages.

I actually opened an issue on that very topic (before reading the
present thread), and there I give other reasons for playing nice with
the distribution package manager:

 https://github.com/JuliaLang/julia/issues/1692

> All I'm suggesting is (a) put standard place(s) in the LOAD_PATH for
> non-Pkg code, (b) the load-path should respect the FHS when on Unix
> (/usr vs. /usr/local vs. $HOME should all co-exist), and (c) standardize
> a language version identifier in the load paths.

This seems a very sensible plan to me.

Seems ok to me too. One question. Does the (b) bit about FHS mean that Julia package files get scattered all around the file system. If so, we may need to keep that in mind when looking for things because architecture-dependent and independent files (typically .so vs .jl) will end up in different places, but we need to be able to find both of them.

[Sébastien Villemot]

Le jeudi 06 décembre 2012 à 12:40 -0500, Stefan Karpinski a écrit :
> On Thu, Dec 6, 2012 at 12:27 PM, Sébastien Villemot
> <seba...@debian.org> wrote:
>
> Having the possibility of distributing packages through
> apt/yum/pacman/... should not be thought as antagonistic to
> the Pkg+git
> system, but rather as complementary. I think the use cases are
> different: developers and power users will want the latest
> packages and
> therefore use Git, while other users will probably prefer
> their
> distribution package manager for stability and convenience.

> That makes sense to me but I do worry about the transitional user
> experience. Someone is using the system-default Julia setup, discovers
> that there is *one* package that isn't available from the system,
> switches over to using the package manager and suddenly needs to
> install all the packages that already worked. Or maybe if you're using
> the package manager, the default package are just there for the using
> with no guarantees of compatibility. I mean that is what every other
> language-level package manager does, although it seems a bit broken.
> To switch fully over to the package manager, you'd have to make sure
> that Julia is ignoring system-installed packages.

The ideal solution to this is to have the Julia pkg manager understand
both system-wide and per-user locations, so that thanks to dependency
resolving it will only install the missing package(s) locally instead of
all packages.

I understand that this may be more complex to implement. If the solution
is instead to have Julia pkg manager only understand ~/.julia, while
system-wide packages are found with a simple load-path, I agree that
there may be a potential for breakage (but maybe not, if the system is
designed to give precedence to files under ~/.julia). In the worst case
you will have either to uninstall the system-wide packages (not possible
when you are not root), or implement a switch for ignoring system-wide
stuff as you suggest.

Note that in Debian we have such a setup for Octave: most (but not all)
Octave Forge packages are available through apt and will install
under /usr. But you can also install packages locally in your home with
Octave's pkg system. This has been working fine for years, and we got no
user complaint about this. My understanding is that most users are happy
with the packages provided through apt, while some install packages
locally.

> > All I'm suggesting is (a) put standard place(s) in the
> LOAD_PATH for
> > non-Pkg code, (b) the load-path should respect the FHS when
> on Unix
> > (/usr vs. /usr/local vs. $HOME should all co-exist), and (c)
> standardize
> > a language version identifier in the load paths.

> This seems a very sensible plan to me.

> Seems ok to me too. One question. Does the (b) bit about FHS mean that
> Julia package files get scattered all around the file system. If so,
> we may need to keep that in mind when looking for things because
> architecture-dependent and independent files (typically .so vs .jl)
> will end up in different places, but we need to be able to find both
> of them.

Ideally yes. For example, Octave Forge packages are split in two, with
M-files under /usr/share and (compiled) Oct-files under /usr/lib.

[Toivo Henningsson]

Here's a thought:

If e.g. a distro wants to provide julia packages, it could do so by providing a METADATA repo and a set of package repos,
both as current as is suitable. These will effectively work as a database of available package versions, and the package manager can do its thing with it, to find a proper combination of packages. If the user wants to use more recent packages, he switches to a current METADATA (but hopefully the old git repos could still be used as a database)

Too crazy?

 / Toivo

[Stefan Karpinski]

I'm fine with that, but I suspect that system package managers are not going to be into that scheme.

 / Toivo

--
 
 
 

[Toivo Henningsson]

Ok, then how about this spin on it:

• A distro can provide a monolithic set of prepackaged julia packages, picked by the julia package manager at some point to be mutually consistent. The packages' git repos might be included in the pack.
• If the user adds/modifies his requirements, he/she has to use the package manager,
  but the package manager will try not to download new packages unless it has to. (modify the cost function?)
  
• Those packages which are left at the version supplied by the distro can continue to use the original files.
  The others might go in the user's home directory. (at least if he/she does not have root access)

It seems like it could be possible to provide a more gradual transition from prepackaged to Pkg-installed packages this way,
while still keeping Pkg in charge to decide which package versions may be used together. Hopefully most inter-package requirements will be of the "recent enough" type, so that many of the prepackaged packages might still be useful.

[Stefan Karpinski]

My thought was that people can have both system-installed stuff and have a Pkg-managed .julia directory and the package manger knows nothing about the system-installed stuff, but LOAD_PATH will, by default, look for packages in both places. If those don't mesh well, that's unfortunate, but that's what happens when you mix packaging systems. If you want Pkg-style version resolution *only*, then you have to tell Julia to ignore the system-installed stuff (or simply not have any).

The other approach I can think of that makes some sense is that upon startup, if ~/.julia exists, it will *only* use that, and otherwise use system-installed stuff. That means that the mixed system/Pkg situation would have to be forced by adding both to LOAD_PATH. On the other hand, it means that as soon as you install anything using Pkg, you stop having the system-installed things available by default.

--
 
 
 

[Harlan Harris]

I sorta like the latter of these two suggestions. When you do "Pkg.init()", it would be easy for the package manager to identify the packages that were part of the system install, and copy those packages into .julia (or re-pull them from git, perhaps).

--
 
 
 

[Stefan Karpinski]

If the system itself uses Pkg, then it could automatically do a clone of the system repo for you. I'm not sure how realistic that is, however.

--
 
 
 

[Tim Holy]

On Thursday, December 06, 2012 04:48:13 PM Stefan Karpinski wrote:
> My thought was that people can have both system-installed stuff and have a
> Pkg-managed .julia directory and the package manger knows nothing about the
> system-installed stuff, but LOAD_PATH will, by default, look for packages
> in both places. If those don't mesh well, that's unfortunate, but that's
> what happens when you mix packaging systems. If you want Pkg-style version
> resolution *only*, then you have to tell Julia to ignore the
> system-installed stuff (or simply not have any).

If LOAD_PATH knows about the system packages, but Pkg does not, then when you
say Pkg.add("xyz") I'm guessing it will install all the dependencies. Assuming
that .julia takes precedence in LOAD_PATH, won't it then find a self-consistent
set of files? And default to the system packages only for things that are
independent of anything you've installed via Pkg?

I guess this assumes that the dependencies are comprehensive. But presumably
we'll need that to be true for people who don't have system packages at all.

--Tim

[Stefan Karpinski]

Yes, that was my thought. The system versions would sort of provide a back-drop of stuff you have available, e.g. just for interactive use. There's a chance that you might use a system-installed package and that it depends on a package that has been installed by Pkg at an incompatible version. The solution then is to use Pkg to install another copy of the system package and ensure that the versions are compatible.

It might be nice to make sure that requirements can only be satisfied from "up" the load path. I.e. a project can satisfy its requirements from anywhere, while .julia can only satisfy requirements from .julia or the system but not the project (it *should* only ever find those requirements in .julia, since Pkg ensures dependencies are satisfies); the system can only satisfy its requirement from the system, not anywhere else. That seems like a generally sane strategy and would prevent a system package from using an incompatible .julia package. It would also prevent a naming conflict in a project from clashing with a required package, which might be a good thing.

Seems complicated though.