NPM style app/script distribution and installation

[Ben Ward]

Hi,

I'd just like to draw anyone who is interested to this BioJulia issue: https://github.com/BioJulia/Bio.jl/issues/267

Essentially we are wondering if apps could be distributed using julia and Pkg, by having packages include bin scripts that could be symlinked into the path (much like NPM does https://docs.npmjs.com/files/package.json#bin) during install of the package.

At the moment the suggestion is perhaps to just use deps/build.jl to do this. But a specific hook in the Pkg install process could also be cool.

Advice or suggestions welcome here or the issue, I'm not 100% familiar with the Pkg internals, I have only read the Pkg.update function source.

Many Thanks,

Ben.

[Stefan Karpinski]

Seems like a plausible thing to do, although we'd have to have a discussion about the security implications. I'll keep it in mind for the Pkg3 revamp of the package manager.

[Ben Ward]

Thanks Stefan,

From our brief discussion in the issue, we considered maybe there would be some hook that gets run during Pkg.add and Pkg.rm, bit how the build.jl script of each package gets run during install.

In fact you could use a build.jl script to achieve the kind of effect:

import Glob
for script in Glob.glob("bin/*.jl")
    symlink(script, joinpath("/usr/local/bin", splitdir(script)[2]))
end

But of course without a corresponding Pkg.rm operation to undo that, you'll have broke links all up in your /usr/local/bin :/

I tried to read the Pkg. source today to see if I could see where in the add process for example the package repository is downloaded, and build.jl run and so on.

I got as far as the REQUIRE stuff getting resolved and edited by Entry.add, which executes Entry.edit, but then Entry.edit calls resolve, and there's graph stuff and at that point I got lost so couldn't really determine where in the entire process something like a symlinking of "app scripts" in a package would occur (somewhere just after the package is downloaded, and build.jl and so on is run for the package is my guess where this fits). 

I know a lot of people I work with, and bioinformaticians in general would really like this: we very often create libraries which can be loaded in scripts and other future projects, but we also need that code quickly runnable in day to day pipelines and jobs, so often have small apps or scripts very thinly wrapping some of the functions in those libraries or packages.

[Chris Foster]

It might also be worth looking at the python "entry points" mechanism
as a source of inspiration. Here's a summary as I understand it:

The idea is to associate a script name (which will go in the path)
with a function inside your module. At install time, the python setup
system then generates a tiny thunk script with the given name and puts
it in the install path for the python environment. The thunk deals
with using the correct python interpreter path to import the correct
module and call the desired function.

This can be nice as any platform-specific annoyances regarding setting
up paths and other environment details can go into the autogenerated
script and the package author doesn't need to deal with them. It also
allows for a layer of renaming to abstract the details of what is
considered callable from the shell on different systems. Eg, a batch
script could be generated on windows (I don't know if this is what
actually happens), while on linux you'd want the executable flag set
and avoid a filename extension. It also avoids the need for a script
per entry point which might be a good or bad thing depending on your
point of view.

Downsides as I see them: 1) the install directory can't be trivially
copied to a different path - this can be bad if you want to package an
entire environment 2) Calling the entry points during package
development is different from after they've been installed, and this
is annoying.

Is there any documentation or code for the proposed Pkg3 design other
than the juliacon talk?

[Stefan Karpinski]

On Sat, Aug 20, 2016 at 9:20 PM, Chris Foster <chri...@gmail.com> wrote:

Is there any documentation or code for the proposed Pkg3 design other
than the juliacon talk?

 Not yet but there will be a Julep write up in a bit, at which point we'll have a public discussion.

[Chris Foster]

Great, I'm looking forward to this. At the moment we use a hacked up
combination
of normal Pkg, DeclarativePackages.jl and custom shell scripts for
managing projects
and packaging up a julia environment for moving to a different
machine. This works ok
for pure julia code, but binary dependencies are a mess, especially
when the system
package manager is involved.