Deploying Julia libraries

[Sebastian Good]

With the release of v0.4 imminent, I thought I'd see what the latest thinking was on deploying Julia libraries (or even executables) to foreign machines. I see several steps required to run a Julia program:

1. Install Julia. This is easy enough to do in a system image.

2. Add/clone packages. This can be scripted, but is at least time consuming an more commonly dangerous: a production machine reaching out to the internet is frowned upon.

3. Precompile. This can be done on demand but also takes time and introduces another step that could theoretically fail. 

Is it possible to pre-compile sets of packages into a portable binary, or at least one that assumes only a base installation of Julia?

Thanks

[Steven G. Johnson]

Just install the packages into some directory, and add that directory to the LOAD_PATH.   You can also precompile them and put the .ji files in some directory that you add to the 

LOAD_CACHE_PATH path.   That way your users will get a fixed set of packages at a known version, won't need Internet access, and won't need to precompile.

[Sebastian Good]

Setting JULIA_PKGPATH lets me put everything in a more amenable spot, but transporting everything over to another identical machine results in inconsistent behavior with precompiled binaries. They are always “stale”. Is there a good resource to consult to understand binary staleness?

Thanks

[Steven G. Johnson]

On Wednesday, September 30, 2015 at 9:27:29 PM UTC-4, Sebastian Good wrote:

Setting JULIA_PKGPATH lets me put everything in a more amenable spot, but transporting everything over to another identical machine results in inconsistent behavior with precompiled binaries. They are always “stale”. Is there a good resource to consult to understand binary staleness?

Staleness is determined by this function:

https://github.com/JuliaLang/julia/blob/e54c38d27028a0c79a10140c8271d484e182b295/base/loading.jl#L421-L449

In particular, it is considered stale if the absolute path changes, or if the modification time of a dependency (which is checked via the absolute path) does not match the modification time of the file when the cache was created.

So, a precompiled image will be considered stale if you change the absolute path of the cache or source directories.   It might be nice to make these relocatable without invalidating the cache.

However, in the meantime I don't think there is much downside to just having the user re-precompile the packages.  Precompilation is very different than building the package; it doesn't require internet access or other external resources, and re-running precompilation can basically only fail if importing the package fails, in which case you probably have bigger problems.

[Peter Simon]

I tried modifying LOAD_CACHE_PATH, but it is a non-exported variable of Base.  Even after importing it, I am not allowed to modify it.  This capability would be useful to me since I've written some Julia command-line utilities to called by numerous non-Julia-proficient users.  These utilities make use of standard packages (such as PyPlot) that require precompilation, and it is undesirable that each new user must go through the precompilation delay on their first use of the utility. It would be much nicer if LOAD_CACHE_PATH could be pointed to some common directory that is readable and writable by everyone so that the modules would only have to be precompiled once upon first use.   Is this a reasonable topic for filing as a Julia issue?

[Steven G. Johnson]

On Saturday, October 3, 2015 at 5:14:57 PM UTC-4, Peter Simon wrote:

I tried modifying LOAD_CACHE_PATH, but it is a non-exported variable of Base.  Even after importing it, I am not allowed to modify it.  

It is an array, so it is mutable.  You can modify its elements by push! and similar functions.  You just can't assign it to a completely new array because it is "const" (i.e. it is effectively a constant pointer to an array, not a pointer to a constant array, in C parlance).

[Peter Simon]

Thanks!