Pkg.init() problems on 0.5-rc0

[Daniel O'Malley]

I'm having trouble getting the package system initialized on 0.5-rc0. I am on a Mac and behind a proxy (with http_proxy and https_proxy set appropriately). When I do Pkg.init(), I get

julia> Pkg.init()
INFO: Initializing package repository /homedir/.julia/v0.5
INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR: GitError(Code:ERROR, Class:OS, Failed to connect to github.com: Operation timed out)
 in macro expansion at ./libgit2/error.jl:98 [inlined]
 in clone(::String, ::String, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:191
 in #clone#105(::String,
 ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload},
::Function, ::String, ::String) at ./libgit2/libgit2.jl:308
 in (::Base.LibGit2.#kw##clone)(::Array{Any,1}, ::Base.LibGit2.#clone, ::String, ::String) at ./<missing>:0
 in (::Base.Pkg.Dir.##4#6{String,String})() at ./pkg/dir.jl:49
 in cd(::Base.Pkg.Dir.##4#6{String,String}, ::String) at ./file.jl:59
 in init(::String, ::String) at ./pkg/dir.jl:47
 in init() at ./pkg/pkg.jl:70

Running

git clone https://github.com/JuliaLang/METADATA.jl


from the command line succeeds, and the package system works fine for me on julia 0.4. There has been some discussion of this issue for 0.5 (https://github.com/JuliaLang/julia/issues/14167 and the issues referenced therein), but as far as I can tell the workaround is to modify some julia build settings and rebuild. Is there a workaround where the prebuilt binaries can be used?

Any help would be much appreciated.

Thanks,
Dan

[Tony Kelman]

I believe you need to build libgit2 against libcurl in order for proxies to work on linux and mac. We do not currently have it set up in our build system to do that in a self contained distributable way for binaries, but it might not be too hard to write the necessary makefile to make it work.

We were prioritizing getting ssh remotes to work again for the first rc. Do ssh remotes work over proxies by any chance? I don't currently have immediate access to an environment behind a proxy to test this, but we had looked into setting one up that we could revisit.

[Tony Kelman]

Another thing you could try is just rebuilding libgit2 locally on your system (instead of all of julia), or maybe using the build from homebrew if they link against libcurl, and replacing the copy of libgit2 included in julia with your rebuilt version (either overwrite the dylib, or move the old one out of the way and use the right DYLD_LIBRARY_PATH setting to load a different one).

[Daniel O'Malley]

Tony, thanks for the quick response and all the work you put into julia. It would be great to have the package system working for Mac/Linux from behind a proxy without needing to rebuild anything. If you need someone to help test from behind a proxy, please let me know. I'd be happy to do it.

When you say "ssh remotes", do you mean something like addprocs(["machine1", "machine2"])? That does work for me, but I haven't tried addprocs'ing to a machine outside our network.

[Tony Kelman]

No, I meant ssh:// or git:// url's for packages.

[Andreas Lobinger]

So do i understand correctly that the current win64 build doesn't include git/ssh via proxy? Or is just the setting in (where?) missing? Is there a counterpart to the https-insteadof-git setting available?
I know it's not helpful, but i get (behind a company http/https proxy) the same error with Pkg.init() (which is in comparision to other RC0 issues a showstopper).

[Tony Kelman]

You can use Pkg.setprotocol!("https"). On Windows libgit2 and libssh2 should be linking against the built-in native winhttp backend and they might not need libcurl for proxy support there, but I don't know how to configure them for proxy support. This is going to require some testing, investigation, and figuring out what does and does not work. See https://github.com/JuliaLang/julia/issues/13472#issuecomment-202568599 where at least one person reports winhttp "should be fine."

[Daniel O'Malley]

Oh, sorry for the mixup. Doing a

git clone ssh://g...@github.com:/JuliaLang/METADATA.jl.git

from a shell succeeds. I haven't had success trying to change the protocol with Pkg.setprotocol! though. Whether I do Pkg.setprotocol!("ssh"), Pkg.setprotocol!("git"), or Pkg.setprotocol!("notaprotocol"), Pkg.init() seems to always try to clone via https.

[Tony Kelman]

That could be considered a bug, Pkg.init should probably respect the Pkg.setprotocol! setting, at least when using the DEFAULT_META value. You should also be able to do Pkg.init("ssh://g...@github.com:/

JuliaLang/METADATA.jl.git") as a workaround to avoid having to go through command-line git.

[Daniel O'Malley]

I tried the Pkg.init("ssh://...") a couple different ways, but neither worked.

Pkg.init("ssh://g...@github.com:/JuliaLang/METADATA.jl.git")

gave GitError(Code:EINVALIDSPEC, Class:Net, Malformed URL 'ssh://g...@github.com:/JuliaLang/METADATA.jl.git').

Pkg.init("ssh://g...@github.com/JuliaLang/METADATA.jl.git")

asked me about my ssh keys then resulted in an authentication failure. If I were to get Pkg.init() working in this way, Pkg.add and other Pkg functions would still have problems though, right?

On Monday, August 1, 2016 at 9:09:22 AM UTC-6, Tony Kelman wrote:

That could be considered a bug, Pkg.init should probably respect the Pkg.setprotocol! setting, at least when using the DEFAULT_META value. You should also be able to do Pkg.init("ssh://git@github.com:/

[Tony Kelman]

I think Pkg.add should obey setprotocol!, it's likely a bug that Pkg.init doesn't. The extra colon after .com probably is malformed so isn't the first error correct? The second issue might be fixed by some authentication changes that Keno just merged a few minutes ago, there should be binaries that include that fix built in a few hours.

On Monday, August 1, 2016 at 8:31:44 AM UTC-7, Daniel O'Malley wrote:

I tried the Pkg.init("ssh://...") a couple different ways, but neither worked.

Pkg.init("ssh://git@github.com:/JuliaLang/METADATA.jl.git")

gave GitError(Code:EINVALIDSPEC, Class:Net, Malformed URL 'ssh://g...@github.com:/JuliaLang/METADATA.jl.git').

Pkg.init("ssh://git@github.com/JuliaLang/METADATA.jl.git")

asked me about my ssh keys then resulted in an authentication failure. If I were to get Pkg.init() working in this way, Pkg.add and other Pkg functions would still have problems though, right?

[Daniel O'Malley]

OK, I tried again with the latest nightlies. Now Pkg.init("ssh://g...@github.com/JuliaLang/METADATA.jl.git") succeeds. However, I'm still not able to add packages. Pkg.setprotocol!("ssh") does seem to work with Pkg.add though:

julia> Pkg.setprotocol!("ssh"); Pkg.add("ACME")

INFO: Cloning cache of ACME from ssh://github.com/HSU-ANT/ACME.jl.git

WARNING: The explicitly provided credentials were incompatible with the server's supported authentication methods

ERROR: Cannot clone ACME from ssh://github.com/HSU-ANT/ACME.jl.git. No errors

 in prefetch(::String, ::String, ::Array{String,1}) at ./pkg/cache.jl:56

 in resolve(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{String,Tuple{VersionNumber,Bool}}, ::Dict{String,Base.Pkg.Types.Fixed}, ::Dict{String,VersionNumber}, ::Set{String}) at ./pkg/entry.jl:512

 in resolve(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{String,Tuple{VersionNumber,Bool}}, ::Dict{String,Base.Pkg.Types.Fixed}) at ./pkg/entry.jl:476

 in edit(::Function, ::String, ::Base.Pkg.Types.VersionSet, ::Vararg{Base.Pkg.Types.VersionSet,N}) at ./pkg/entry.jl:30

 in (::Base.Pkg.Entry.##2#5{String,Base.Pkg.Types.VersionSet})() at ./task.jl:309

 in sync_end() at ./task.jl:275

 in macro expansion at ./task.jl:284 [inlined]

 in add(::String, ::Base.Pkg.Types.VersionSet) at ./pkg/entry.jl:51

 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at ./pkg/dir.jl:31

 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at ./file.jl:59

 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31

 in add(::String) at ./pkg/pkg.jl:100

[Tony Kelman]

This sounds like the issue people have been reporting at https://github.com/JuliaLang/PkgDev.jl/issues/48, though it's more likely a problem with base Julia.

On Tuesday, August 2, 2016 at 9:21:43 AM UTC-7, Daniel O'Malley wrote:

OK, I tried again with the latest nightlies. Now Pkg.init("ssh://git@github.com/JuliaLang/METADATA.jl.git") succeeds. However, I'm still not able to add packages. Pkg.setprotocol!("ssh") does seem to work with Pkg.add though:

[Daniel O'Malley]

I wonder if the problem is that "ssh://g...@github.com" is needed for ssh remotes rather than just "ssh://github.com". The former works with Pkg.clone, but the latter does not. For example,

julia> Pkg.clone("ssh://github.com/HSU-ANT/ACME.jl.git")
INFO: Cloning ACME from ssh://github.com/HSU-ANT/ACME.jl.git

WARNING: The explicitly provided credentials were incompatible with the server's supported authentication methods

ERROR: GitError(Code:EAUTH, Class:None, No errors)

 in macro expansion at ./libgit2/error.jl:98 [inlined]

 in clone(::String, ::SubString{String}, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:191
 in #clone#107(::String, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload}, ::Function, ::String, ::SubString{String}) at ./libgit2/libgit2.jl:308
 in clone(::String, ::SubString{String}) at ./pkg/entry.jl:195
 in clone(::String) at ./pkg/entry.jl:221
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}}, ::String) at ./file.jl:59

 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31

 in clone(::String) at ./pkg/pkg.jl:151

produces the same warning as Pkg.add("ACME"), but

julia> Pkg.clone("ssh://g...@github.com/HSU-ANT/ACME.jl.git")
INFO: Cloning ACME from ssh://g...@github.com/HSU-ANT/ACME.jl.git
INFO: Computing changes...
INFO: Installing Compat v0.8.6
INFO: Installing ProgressMeter v0.3.2
INFO: Package database updated

succeeds.