Sometimes 'go get'ing vanity domains is very slow

[Caleb Spare]

I'm on a fast internet connection and typically go getting packages (if they're on GitHub and have few dependencies) takes a few seconds.

Sometimes when the packages have a canonical import path at a certain vanity domain, even when the code itself is hosted on GitHub, go get takes

​ several minutes.

As an example,

​I ran 'go get ​robpike.io/ivy' and it took 4.5 minutes. I ran with -v and annotated every output line with a timestamp; see the bottom of this email. As you can see, go get tries to fetch the meta tag https://vanity.dom/pkg?go-get=1, times out after 30 seconds, and then tries http instead of https. It repeats this, with 30s timeout, for every subpackage (all in the same repo).

I believe that this is the same issue I described here:

https://github.com/azul3d/issues/issues/24

​The website at the vanity domain may be configured to drop, rather than reject, https connections. For instance you can see that running this command hangs:

$ ​

curl 'https://robpike.io/ivy?go-get=1'​

(​Wild speculation: are both of these vanity domains hosted on GAE? ​Does GAE drop https connections when it's not configured to handle them?)

Have other people run into this? Am I correct in asserting that it's essentially a misconfiguration on the vanity domain endpoint with respect to handling https? Is there something go get can do to work around this or discourage this kind of misconfiguration?

Perhaps after https://github.com/golang/go/issues/9637 goes through it will be 'fixed' by forcing everyone with a vanity domain to make https work (unless they want everyone to use an -insecure flag to download their packages).

-Caleb

$ rm -rf $GOPATH/src/robpike.io
$ go get -v -u robpike.io/ivy 2>&1 | tsprefix
Mar  8 16:10:52 Fetching https://robpike.io/ivy?go-get=1
Mar  8 16:11:22 https fetch failed.
Mar  8 16:11:22 Fetching http://robpike.io/ivy?go-get=1
Mar  8 16:11:22 Parsing meta tags from http://robpike.io/ivy?go-get=1 (status code 200)
Mar  8 16:11:22 get "robpike.io/ivy": found meta tag main.metaImport{Prefix:"robpike.io/ivy", VCS:"git", RepoRoot:"https://github.com/robpike/ivy.git"} at http://robpike.io/ivy?go-get=1
Mar  8 16:11:22 robpike.io/ivy (download)
Mar  8 16:11:23 Fetching https://robpike.io/ivy/config?go-get=1
Mar  8 16:11:53 https fetch failed.
Mar  8 16:11:53 Fetching http://robpike.io/ivy/config?go-get=1
Mar  8 16:11:53 Parsing meta tags from http://robpike.io/ivy/config?go-get=1 (status code 200)
Mar  8 16:11:53 get "robpike.io/ivy/config": found meta tag main.metaImport{Prefix:"robpike.io/ivy", VCS:"git", RepoRoot:"https://github.com/robpike/ivy.git"} at http://robpike.io/ivy/config?go-get=1
Mar  8 16:11:53 get "robpike.io/ivy/config": verifying non-authoritative meta tag
Mar  8 16:11:53 Fetching https://robpike.io/ivy?go-get=1
Mar  8 16:12:23 https fetch failed.
Mar  8 16:12:23 Fetching http://robpike.io/ivy?go-get=1
Mar  8 16:12:23 Parsing meta tags from http://robpike.io/ivy?go-get=1 (status code 200)
Mar  8 16:12:23 Fetching https://robpike.io/ivy/parse?go-get=1
Mar  8 16:12:53 https fetch failed.
Mar  8 16:12:53 Fetching http://robpike.io/ivy/parse?go-get=1
Mar  8 16:12:53 Parsing meta tags from http://robpike.io/ivy/parse?go-get=1 (status code 200)
Mar  8 16:12:53 get "robpike.io/ivy/parse": found meta tag main.metaImport{Prefix:"robpike.io/ivy", VCS:"git", RepoRoot:"https://github.com/robpike/ivy.git"} at http://robpike.io/ivy/parse?go-get=1
Mar  8 16:12:53 get "robpike.io/ivy/parse": verifying non-authoritative meta tag
Mar  8 16:12:53 Fetching https://robpike.io/ivy?go-get=1
Mar  8 16:13:23 https fetch failed.
Mar  8 16:13:23 Fetching http://robpike.io/ivy?go-get=1
Mar  8 16:13:23 Parsing meta tags from http://robpike.io/ivy?go-get=1 (status code 200)
Mar  8 16:13:23 Fetching https://robpike.io/ivy/scan?go-get=1
Mar  8 16:13:53 https fetch failed.
Mar  8 16:13:53 Fetching http://robpike.io/ivy/scan?go-get=1
Mar  8 16:13:53 Parsing meta tags from http://robpike.io/ivy/scan?go-get=1 (status code 200)
Mar  8 16:13:53 get "robpike.io/ivy/scan": found meta tag main.metaImport{Prefix:"robpike.io/ivy", VCS:"git", RepoRoot:"https://github.com/robpike/ivy.git"} at http://robpike.io/ivy/scan?go-get=1
Mar  8 16:13:53 get "robpike.io/ivy/scan": verifying non-authoritative meta tag
Mar  8 16:13:53 Fetching https://robpike.io/ivy?go-get=1
Mar  8 16:14:23 https fetch failed.
Mar  8 16:14:23 Fetching http://robpike.io/ivy?go-get=1
Mar  8 16:14:23 Parsing meta tags from http://robpike.io/ivy?go-get=1 (status code 200)
Mar  8 16:14:23 Fetching https://robpike.io/ivy/value?go-get=1
Mar  8 16:14:53 https fetch failed.
Mar  8 16:14:53 Fetching http://robpike.io/ivy/value?go-get=1
Mar  8 16:14:53 Parsing meta tags from http://robpike.io/ivy/value?go-get=1 (status code 200)
Mar  8 16:14:53 get "robpike.io/ivy/value": found meta tag main.metaImport{Prefix:"robpike.io/ivy", VCS:"git", RepoRoot:"https://github.com/robpike/ivy.git"} at http://robpike.io/ivy/value?go-get=1
Mar  8 16:14:53 get "robpike.io/ivy/value": verifying non-authoritative meta tag
Mar  8 16:14:53 Fetching https://robpike.io/ivy?go-get=1
Mar  8 16:15:23 https fetch failed.
Mar  8 16:15:23 Fetching http://robpike.io/ivy?go-get=1
Mar  8 16:15:23 Parsing meta tags from http://robpike.io/ivy?go-get=1 (status code 200)
Mar  8 16:15:24 robpike.io/ivy/config
Mar  8 16:15:24 robpike.io/ivy/scan
Mar  8 16:15:24 robpike.io/ivy/value
Mar  8 16:15:24 robpike.io/ivy/parse
Mar  8 16:15:24 robpike.io/ivy

[David Symonds]

That sounds like pretty poor behaviour on the server's side, but also
from `go get`. Perhaps the latter should kick off the second fallback
request after waiting for 5s instead of waiting for the whole 30s
timeout. Could you file an issue for this?

[Caleb Spare]

​Thanks David. Filed here: https://github.com/golang/go/issues/10120​