Pkg.init() problems on 0.5-rc0

333 views
Skip to first unread message

Daniel O'Malley

unread,
Jul 31, 2016, 7:41:34 PM7/31/16
to julia-users
I'm having trouble getting the package system initialized on 0.5-rc0. I am on a Mac and behind a proxy (with http_proxy and https_proxy set appropriately). When I do Pkg.init(), I get

julia> Pkg.init()
INFO
: Initializing package repository /homedir/.julia/v0.5
INFO
: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl
ERROR
: GitError(Code:ERROR, Class:OS, Failed to connect to github.com: Operation timed out)
 
in macro expansion at ./libgit2/error.jl:98 [inlined]
 
in clone(::String, ::String, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:191
 
in #clone#105(::String,
 
::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload},
::Function, ::String, ::String) at ./libgit2/libgit2.jl:308
 
in (::Base.LibGit2.#kw##clone)(::Array{Any,1}, ::Base.LibGit2.#clone, ::String, ::String) at ./<missing>:0
 
in (::Base.Pkg.Dir.##4#6{String,String})() at ./pkg/dir.jl:49
 
in cd(::Base.Pkg.Dir.##4#6{String,String}, ::String) at ./file.jl:59
 
in init(::String, ::String) at ./pkg/dir.jl:47
 
in init() at ./pkg/pkg.jl:70

Running


from the command line succeeds, and the package system works fine for me on julia 0.4. There has been some discussion of this issue for 0.5 (https://github.com/JuliaLang/julia/issues/14167 and the issues referenced therein), but as far as I can tell the workaround is to modify some julia build settings and rebuild. Is there a workaround where the prebuilt binaries can be used?

Any help would be much appreciated.

Thanks,
Dan

Tony Kelman

unread,
Jul 31, 2016, 8:12:12 PM7/31/16
to julia-users
I believe you need to build libgit2 against libcurl in order for proxies to work on linux and mac. We do not currently have it set up in our build system to do that in a self contained distributable way for binaries, but it might not be too hard to write the necessary makefile to make it work.

We were prioritizing getting ssh remotes to work again for the first rc. Do ssh remotes work over proxies by any chance? I don't currently have immediate access to an environment behind a proxy to test this, but we had looked into setting one up that we could revisit.

Tony Kelman

unread,
Jul 31, 2016, 8:16:34 PM7/31/16
to julia-users
Another thing you could try is just rebuilding libgit2 locally on your system (instead of all of julia), or maybe using the build from homebrew if they link against libcurl, and replacing the copy of libgit2 included in julia with your rebuilt version (either overwrite the dylib, or move the old one out of the way and use the right DYLD_LIBRARY_PATH setting to load a different one).

Daniel O'Malley

unread,
Jul 31, 2016, 10:02:15 PM7/31/16
to julia-users
Tony, thanks for the quick response and all the work you put into julia. It would be great to have the package system working for Mac/Linux from behind a proxy without needing to rebuild anything. If you need someone to help test from behind a proxy, please let me know. I'd be happy to do it.

When you say "ssh remotes", do you mean something like addprocs(["machine1", "machine2"])? That does work for me, but I haven't tried addprocs'ing to a machine outside our network.

Tony Kelman

unread,
Aug 1, 2016, 12:31:38 AM8/1/16
to julia-users
No, I meant ssh:// or git:// url's for packages.

Andreas Lobinger

unread,
Aug 1, 2016, 4:35:28 AM8/1/16
to julia-users
So do i understand correctly that the current win64 build doesn't include git/ssh via proxy? Or is just the setting in (where?) missing? Is there a counterpart to the https-insteadof-git setting available?
I know it's not helpful, but i get (behind a company http/https proxy) the same error with Pkg.init() (which is in comparision to other RC0 issues a showstopper).

Tony Kelman

unread,
Aug 1, 2016, 7:23:14 AM8/1/16
to julia-users
You can use Pkg.setprotocol!("https"). On Windows libgit2 and libssh2 should be linking against the built-in native winhttp backend and they might not need libcurl for proxy support there, but I don't know how to configure them for proxy support. This is going to require some testing, investigation, and figuring out what does and does not work. See https://github.com/JuliaLang/julia/issues/13472#issuecomment-202568599 where at least one person reports winhttp "should be fine."

Daniel O'Malley

unread,
Aug 1, 2016, 10:01:51 AM8/1/16
to julia-users
Oh, sorry for the mixup. Doing a

git clone ssh://g...@github.com:/JuliaLang/METADATA.jl.git

from a shell succeeds. I haven't had success trying to change the protocol with Pkg.setprotocol! though. Whether I do Pkg.setprotocol!("ssh"), Pkg.setprotocol!("git"), or Pkg.setprotocol!("notaprotocol"), Pkg.init() seems to always try to clone via https.

Tony Kelman

unread,
Aug 1, 2016, 11:09:22 AM8/1/16
to julia-users
That could be considered a bug, Pkg.init should probably respect the Pkg.setprotocol! setting, at least when using the DEFAULT_META value. You should also be able to do Pkg.init("ssh://g...@github.com:/
JuliaLang/METADATA.jl.git") as a workaround to avoid having to go through command-line git.

Daniel O'Malley

unread,
Aug 1, 2016, 11:31:44 AM8/1/16
to julia-users
I tried the Pkg.init("ssh://...") a couple different ways, but neither worked.

Pkg.init("ssh://g...@github.com:/JuliaLang/METADATA.jl.git")

gave GitError(Code:EINVALIDSPEC, Class:Net, Malformed URL 'ssh://g...@github.com:/JuliaLang/METADATA.jl.git').


asked me about my ssh keys then resulted in an authentication failure. If I were to get Pkg.init() working in this way, Pkg.add and other Pkg functions would still have problems though, right?

On Monday, August 1, 2016 at 9:09:22 AM UTC-6, Tony Kelman wrote:
That could be considered a bug, Pkg.init should probably respect the Pkg.setprotocol! setting, at least when using the DEFAULT_META value. You should also be able to do Pkg.init("ssh://git@github.com:/

Tony Kelman

unread,
Aug 1, 2016, 1:24:51 PM8/1/16
to julia-users
I think Pkg.add should obey setprotocol!, it's likely a bug that Pkg.init doesn't. The extra colon after .com probably is malformed so isn't the first error correct? The second issue might be fixed by some authentication changes that Keno just merged a few minutes ago, there should be binaries that include that fix built in a few hours.



On Monday, August 1, 2016 at 8:31:44 AM UTC-7, Daniel O'Malley wrote:
I tried the Pkg.init("ssh://...") a couple different ways, but neither worked.

Pkg.init("ssh://git@github.com:/JuliaLang/METADATA.jl.git")

gave GitError(Code:EINVALIDSPEC, Class:Net, Malformed URL 'ssh://g...@github.com:/JuliaLang/METADATA.jl.git').


asked me about my ssh keys then resulted in an authentication failure. If I were to get Pkg.init() working in this way, Pkg.add and other Pkg functions would still have problems though, right?

Daniel O'Malley

unread,
Aug 2, 2016, 12:21:43 PM8/2/16
to julia-users
OK, I tried again with the latest nightlies. Now Pkg.init("ssh://g...@github.com/JuliaLang/METADATA.jl.git") succeeds. However, I'm still not able to add packages. Pkg.setprotocol!("ssh") does seem to work with Pkg.add though:

julia> Pkg.setprotocol!("ssh"); Pkg.add("ACME")
INFO: Cloning cache of ACME from ssh://github.com/HSU-ANT/ACME.jl.git
WARNING: The explicitly provided credentials were incompatible with the server's supported authentication methods
ERROR: Cannot clone ACME from ssh://github.com/HSU-ANT/ACME.jl.git. No errors
 in prefetch(::String, ::String, ::Array{String,1}) at ./pkg/cache.jl:56
 in resolve(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{String,Tuple{VersionNumber,Bool}}, ::Dict{String,Base.Pkg.Types.Fixed}, ::Dict{String,VersionNumber}, ::Set{String}) at ./pkg/entry.jl:512
 in resolve(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{String,Tuple{VersionNumber,Bool}}, ::Dict{String,Base.Pkg.Types.Fixed}) at ./pkg/entry.jl:476
 in edit(::Function, ::String, ::Base.Pkg.Types.VersionSet, ::Vararg{Base.Pkg.Types.VersionSet,N}) at ./pkg/entry.jl:30
 in (::Base.Pkg.Entry.##2#5{String,Base.Pkg.Types.VersionSet})() at ./task.jl:309
 in sync_end() at ./task.jl:275
 in macro expansion at ./task.jl:284 [inlined]
 in add(::String, ::Base.Pkg.Types.VersionSet) at ./pkg/entry.jl:51
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at ./file.jl:59
 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in add(::String) at ./pkg/pkg.jl:100

Tony Kelman

unread,
Aug 2, 2016, 8:32:13 PM8/2/16
to julia-users
This sounds like the issue people have been reporting at https://github.com/JuliaLang/PkgDev.jl/issues/48, though it's more likely a problem with base Julia.

On Tuesday, August 2, 2016 at 9:21:43 AM UTC-7, Daniel O'Malley wrote:
OK, I tried again with the latest nightlies. Now Pkg.init("ssh://git@github.com/JuliaLang/METADATA.jl.git") succeeds. However, I'm still not able to add packages. Pkg.setprotocol!("ssh") does seem to work with Pkg.add though:

Daniel O'Malley

unread,
Aug 3, 2016, 10:37:13 AM8/3/16
to julia-users
I wonder if the problem is that "ssh://g...@github.com" is needed for ssh remotes rather than just "ssh://github.com". The former works with Pkg.clone, but the latter does not. For example,

julia> Pkg.clone("ssh://github.com/HSU-ANT/ACME.jl.git")
INFO: Cloning ACME from ssh://github.com/HSU-ANT/ACME.jl.git

WARNING: The explicitly provided credentials were incompatible with the server's supported authentication methods
ERROR: GitError(Code:EAUTH, Class:None, No errors)

 in macro expansion at ./libgit2/error.jl:98 [inlined]
 in clone(::String, ::SubString{String}, ::Base.LibGit2.CloneOptions) at ./libgit2/repository.jl:191
 in #clone#107(::String, ::Bool, ::Ptr{Void}, ::Nullable{Base.LibGit2.AbstractPayload}, ::Function, ::String, ::SubString{String}) at ./libgit2/libgit2.jl:308
 in clone(::String, ::SubString{String}) at ./pkg/entry.jl:195
 in clone(::String) at ./pkg/entry.jl:221
 in (::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}})() at ./pkg/dir.jl:31
 in cd(::Base.Pkg.Dir.##2#3{Array{Any,1},Base.Pkg.Entry.#clone,Tuple{String}}, ::String) at ./file.jl:59

 in #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{Any,N}) at ./pkg/dir.jl:31
 in clone(::String) at ./pkg/pkg.jl:151

produces the same warning as Pkg.add("ACME"), but

julia> Pkg.clone("ssh://g...@github.com/HSU-ANT/ACME.jl.git")
INFO: Cloning ACME from ssh://g...@github.com/HSU-ANT/ACME.jl.git
INFO: Computing changes...
INFO: Installing Compat v0.8.6
INFO: Installing ProgressMeter v0.3.2
INFO: Package database updated


succeeds.
Reply all
Reply to author
Forward
0 new messages