git protocol packages

[Blake Johnson]

Is there a way to still support git protocol (as opposed to https) packages with the new libgit2 based package system? I have a fair number of private packages on a local server, and it sure would be nice to be able to fetch those with SSH key authentication.

[Isaiah Norton]

I'm not sure if this is supposed to be officially supported yet, but I was able to get ssh:// to work on OS X:

1. `brew install libssh2`

2. from julia root dir: `cd deps && make configure-libgit2 VERBOSE=1`

3. copy the cmake command printed by above, and re-run it manually. For some reason PKG_CONFIG_MODULE didn't detect libgit2 the first time (discovered by trial-and-error, verified by `make distclean-libgit2` and doing the process again).

  it should look something like:

     `cmake {HOME}/git/jl71/deps/srccache/libgit2/ -DCMAKE_INSTALL_PREFIX:PATH={HOME}/git/jl71/usr -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_C_COMPILER="clang" -DCMAKE_C_COMPILER_ARG1="-m64 " -DCMAKE_CXX_COMPILER="clang++" -DCMAKE_CXX_COMPILER_ARG1="-m64 " -DTHREADSAFE=ON -DCMAKE_BUILD_TYPE=Release`

4. in julia root directory: `make clean && make`

5. start ssh-agent. in bash: "$ eval `ssh-agent`"

6. run something that causes ssh-agent to unlock the key, for example regular command line git clone'ing a repository via ssh.

After those steps, the following works:

julia> Pkg.clone("ssh://g...@github.com/johnmyleswhite/ASCIIPlots.jl.git")

If I neglect step 6, then the callback ("credentials_cb") gets called indefinitely (noted via print debugging), so it seems that we are missing some step to make ssh-agent unlock the key pair (which happens via system prompt on OS X).

So: it looks like this is almost-supported, but we need to fix build issues and teach the libgit2 wrapper to set up ssh-agent credentials correctly on its own (at least on OS X). Presumably the situation is the same or better on Linux. On Windows, building against libssh2 is explicitly disabled by our Makefile.

[Erik Schnetter]

I tried installing libssh2 automatically
<https://github.com/eschnett/julia/tree/eschnett/libssh2>, but failed
due to "use of undeclared identifier 'LIBSSH2_KNOWNHOST_KEY_UNKNOWN'".
Apparently, the build process picks up a system include directory that
has another, older libssh2 installed, while detecting where my OpenSSL
library is installed. This looks messy, so I gave up for the time
being. Maybe the solution is to distribute OpenSSL as well. Or to
disable OpenSSL, and use MbedTLS instead (that we also distribute).

If you are using a key chain, then the ssh agent should start
automatically when you log in. This works out of the box for me on OS
X, presumably using the OS X keychain.

Thanks for the pointers.

-erik

--
Erik Schnetter <schn...@gmail.com>
http://www.perimeterinstitute.ca/personal/eschnetter/

[Isaiah Norton]

The problem is likely that libgit2 detects libssh2 with CMake's PKG_CHECK_MODULE. This uses `pkg-config`, so it finds whatever is already on your system rather than the built-from-source version. There may be some CMake magic to override that behavior, but probably the easiest way around is to pass the required variables to CMake directly when configuring libgit2 from the Julia Makefile. For the variables, see:

https://github.com/libgit2/libgit2/blob/2f0450f4d635358f6da5d174c128b9ed1059bbf8/CMakeLists.txt#L344-L360

If LIBSSH2_FOUND is set before PKG_CHECK_MODULES, then that macro will short-circuit without setting any variables.

[Tony Kelman]

libgit2 can use the osx native tls library on mac, so we shouldn't need openssl there. Not sure why openssh is getting confused. I'm pretty sure there are https keychain helpers out there if it's entering your password for private repos that you're worried about.

[Rob J. Goedman]

Thanks for the pointers! Having also struggled with private packages on 0.5 for a while now, I tried below steps (a few times). No such luck.

Also, using https: for a private package on 0.5 on my system hangs Pkg.update(). After ^C it states it’s updating TP but that is not the case.

Could that have to do with:

If I neglect step 6, then the callback ("credentials_cb") gets called indefinitely (noted via print debugging), so it seems that we are missing some step to make ssh-agent unlock the key pair (which happens via system prompt on OS X).

Pkg.clone() works.

Regards,

Rob

               _

   _       _ _(_)_     |  A fresh approach to technical computing

  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org

   _ _   _| |_  __ _   |  Type "?help" for help.

  | | | | | | |/ _` |  |

  | | |_| | | | (_| |  |  Version 0.5.0-dev+3344 (2016-03-31 06:13 UTC)

 _/ |\__'_|_|_|\__'_|  |  master/c7f5926 (fork: 57 commits, 4 days)

|__/                   |  x86_64-apple-darwin15.4.0

julia> Pkg.update()

INFO: Updating METADATA...

INFO: Updating Unitful master...

INFO: Updating CSoM master...

Username for 'https://github.com/goedman/TP.jl.git':goedman

Password for 'goedman@https://github.com/goedman/TP.jl.git':

^CWARNING: fetch: InterruptException()

INFO: Updating TP master...

INFO: Updating Benchmarks master...

INFO: Updating NMfE master...

INFO: Updating Jags master...

INFO: Updating ASCIIPlots master...

INFO: Computing changes...

INFO: No packages to install, update or remove

[Erik Schnetter]

I see the same symptoms.

-erik

[Eric Forgy]

I have the same issue with private repos and SSH on Windows. It was working fine before the switch.

[Eric Forgy]

PS: This also means Travis and Appveyor no longer work for me when the package I'm testing depends on other private repos. I really hope a solution to this problem can be found.

[Rob J. Goedman]

Just upgraded 0.5 and now it seems to work using https:// !

Not ideal, but way better!