Pkg.add refactoring proposal

365 views
Skip to first unread message

Rory Finnegan

unread,
Apr 19, 2016, 12:21:14 PM4/19/16
to julia-dev
Hi there,

I'm working at a company that needs to add some features to julia's pkg manager. Specifically, we want to be able to 1) install packages from an arbitrary REQUIRE file and 2) include git URIs in REQUIRE files. Rather than just taking on these features we thought we might do some refactoring to make the API a little cleaner.

For point 1 we currently have a method called `addrequire` that adds the entries from a REQUIRE file to the global REQUIRE file and calls resolve. While the approach seems reasonable I personally don't like the name `addrequire`, so here is my refactoring proposal to address that.

  1. `Pkg.add` should just add a requirement to the global REQUIRE file without doing a resolve. This method should dispatch on a String (package name), URI (remote git uri) and a Path type
    • If the type is a String we check that it is a registered package name and add it to the global REQUIRE file.
    • If the type is a URI we check it and also write it to the global REQUIRE file.
    • if the type is a Path and is a REQUIRE file (`isfile(...)`) add all entries in REQUIRE file, and if the path is a directory (`isdir(...)`) search for a REQUIRE file.
  2. `Pkg.install` should be a wrapper around `Pkg.add` and `Pkg.resolve` and it should also dispatch on the above types and behave in the same way.

Pros:
  • Better utilization of multiple dispatch to help with code & API clarity. Might also help with modularity of `Pkg` in base, but I'd have to dig into the code more.
  • Layered API given that `install` is just a wrapper to `add` and `resolve`
  • Allows installing packages from arbitrary files not just pkg REQUIRE files.
Cons:
  • API change (the existing `Pkg.add` is essentially renamed to `Pkg.install`)
  • Currently would require a URI type and a Path type, which are currently in external packages.

Does this seem reasonable? Do people have other suggestions or ideas?

Thanks,
Rory

Yichao Yu

unread,
Apr 19, 2016, 8:06:08 PM4/19/16
to Julia Dev
I think package names and url/filepath are reasonably distinguished so
you don't need to use types to separate them.

Adding a REQUIRE file sounds a little bit arbitrary (it's effectively
a set of packages, maybe with version constraint) so it might make
more sense to allow parsing a REQUIRE file.

Tony Kelman

unread,
Apr 19, 2016, 8:45:41 PM4/19/16
to julia-dev
I said this on the PR, but having an API in Pkg that's equivalent to concatenating the contents of a file to the global REQUIRE file seems a bit much. Maybe it could be added to PkgDev, but I'm not understanding the use case for it.

Rory Finnegan

unread,
Apr 19, 2016, 9:34:33 PM4/19/16
to julia-dev
@Yichao Yu I agree that you could probably make the above refactoring work where URIs and Paths are still just string, and maybe that is the better short term option. On the other hand, I see a lot of benefits to having them be distinct types anyways and this seems like a good place to utilize multiple dispatch to allow more modularity and flexibility.

To clarify, our use case is that we often have julia code that isn't in the form of a package (like a script or notebook), but we still want to distribute a REQUIRE file for it. Right now I usually just include a requirements.jl script that installs the dependencies or just include the dependency list in my instructions on how to run it. I'm open to suggestions if you have a better approach to achieve the same goal.

Jeffrey Sarnoff

unread,
Apr 20, 2016, 12:25:03 AM4/20/16
to julia-dev
Changing the use/syntax/scope of Julia's REQUIRE files to meet a specific need seems to be a `sticky` approach that may make future work not tidy.  

One way may be to continue using your requirements.jl script but put in PKG.jl/deps/ alongside a simple build.jl file that incorporates or calls it (as appropriate).  That way, your unpackaged dependencies should be fetched and placed/processed as you need when the PKG is add_ed.  And, whenever there is an upgrade to one of the unpackaged dependencies, running Pkg.bulid("PKG") will refetch and reput/reprocess all of them. I don't know what is in the requirements.jl file .. if it is not too much, the contents may better live in the build.jl file directly. Take a look at BinDeps.jl for more possibilities (remember to put BinDeps in the usual REQUIRE file if you end up using it).

Rory Finnegan

unread,
Apr 20, 2016, 9:40:38 AM4/20/16
to julia-dev
@Jeffrey Sarnoff I'm not distributing a pkg, just a single script or notebook. Making an entire package for one script and a REQUIRE file seems a bit much.

Tom Breloff

unread,
Apr 20, 2016, 9:44:55 AM4/20/16
to juli...@googlegroups.com
Rory: have you had a chance to review https://github.com/tshort/Maker.jl?  It seems like this might be the right sort of application of that package.  (disclaimer: I haven't used it yet)

Jeffrey Sarnoff

unread,
Apr 20, 2016, 9:50:59 AM4/20/16
to julia-dev
Making a package that allows you to distribute any single script or notebook that you may need to distribute, assuming this may change with time, might work well for you -- alternatively, if this is about a couple of notebooks, take a look at NBInclude.jl:

"NBInclude is a package for the Julia language which allows you to include and execute IJulia (Julia-language Jupyter) notebook files just as you would include an ordinary Julia file. "

wil...@gmail.com

unread,
May 1, 2016, 11:20:35 AM5/1/16
to julia-dev
Rory, I think the issue is much bigger than modification of the package manager installation routine. You wish a project dependency management, but julia does not provide one. I could see a  simple solution - reorder package loading to prioritize LOAD_PATH over Pkg.dir(). As soon as it's changed, LOAD_PATH can be injected with some local directory in your project that would hold all dependencies. So, then you clone or copy dependencies and local project is ready to run. All these steps can be automated and included in the julia's package manager.

Chris Foster

unread,
May 1, 2016, 11:54:59 PM5/1/16
to juli...@googlegroups.com
Rory - did you check out DeclarativePackages.jl? We're using this for
managing a subset of dependencies outside the main METADATA.jl
(including prerelease testing of unreleased packages, specified only
with a git url). If you care about reproducible builds it's also
pretty great!
Reply all
Reply to author
Forward
0 new messages