Downloading all packages in JuliaLang

[Scott Jones]

I'd like to be able to search the Julia code in the registered packages, just to see if

changes that I'm thinking of making (like `isvalid(string, index)` -> `haskey(string, index)`, which somebody else proposed)

would be likely to cause breakage.

Would the best way of doing this be to try to clone all of the packages? Any idea how much space that would take?  I can use the shallow-clone feature that somebody brought up recently.

If so, is there an easy way to do that?

Thanks, Scott

[Elliot Saba]

I believe Iain does this with his PackageEvaluator script, he'd have to weight in on how much diskspace is used, but I would guess that the vast majority of his diskspace is spent satisfying binary dependencies for packages, not on Julia source code itself.  Therefore, if you're not interested in ever actually calling `Pkg.build()`, your disk space usage should be much less.

-E

[Steven G. Johnson]

Here is a much shorter (shell) script that grabs and/or updates the latest version of all the packages:

#!/bin/sh

for f in ~/.julia/v0.4/METADATA/*/url; do
    p=$(basename $(dirname $f))
    url=`cat $f`
    echo "updating $p from $url..."
    if test ! -d $p; then
        git clone $url $p
    else
        (cd $p; git pull origin master)
    fi
done

I put it in an "allpackages" subdirectory and run it occasionally, precisely for this reason (checking out the impact of changes to Julia or its standard library).  It takes about 1.5G of space.

[Scott Jones]

Thanks very much, kind sir!  That did the trick perfectly... I actually ran it twice, first exactly as above (1.43GB), and then with the --depth=1 flag (1.05GB), which was better for my laptop...