CERTFICATE_VERIFY_FAILED contacting dl.bintray.com

[Eric Zundel Ayers]

Garrett and I are both seeing an issue bootstrapping pants. I filed an issue and proposed fix https://github.com/pantsbuild/pants/issues/2718 . Can someone verify that this is or isn’t a problem outside of Square?

rm -rf ~/.cache/pants/bin/go
./pants  test.go contrib/go/examples:: -ldebug

In the open source version of pants, this fails for me with :

INFO] Attempting to fetch go.tar.gz binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/go/mac/10.11/1.5.2/go.tar.gz ...
FAILURE: Failed to fetch binary bin/go/mac/10.11/1.5.2/go.tar.gz from any source: (Failed to fetch binary from https://dl.bintray.com/pantsbuild/bin/build-support/bin/go/mac/10.11/1.5.2/go.tar.gz: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>)

I’m suspecting some kind of problem local to our network. Although I’m having trouble downloading from pants bootstrapping, Chrome doesn’t have a problem downloading https://dl.bintray.com/pantsbuild/bin/build-support/bin/go/mac/10.11/1.5.2/go.tar.gz

​

[Patrick Lawson]

What version of Python is being used during this run?  I fuzzily recall an issue with python, urlopen, and OpenSSL that presented like this.

If it's network-local: does your network proxy bintray?

[Eric Zundel Ayers]

I am using Python 2.7.10 from brew on our mac laptops. I'm not sure about Garrett.   I have a check to make sure that everyone uses 2.7.9 or higher  when running pants, but don't have exact control over which version gets installed on everyone's laptop.

[John Sirois]

No such issue for me.  My details:

```

jsirois@gill ~/dev-pants (master) $ git log --oneline -1

7fddcc5 Allow specification of extra env vars for junit_tests runs.

jsirois@gill ~/dev-pants (master) $ rm -rf ~/.cache/pants/bin/go

jsirois@gill ~/dev-pants (master) $ python2 --version

Python 2.7.11

jsirois@gill ~/dev-pants (master) $ ./pants test contrib/go/tests/::

...

14:42:42 00:01   [resolve]

14:42:42 00:01     [ivy]

14:42:42 00:01     [go]INFO] Attempting to fetch go.tar.gz binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/go/linux/x86_64/1.5.2/go.tar.gz ...

INFO] Fetched go.tar.gz binary from: https://dl.bintray.com/pantsbuild/bin/build-support/bin/go/linux/x86_64/1.5.2/go.tar.gz .

14:48:56 06:15       [github.com/AdRoll/goamz/dynamodb]

14:48:56 06:15       [github.com/bitly/go-simplejson]

...

14:48:57 06:16       [github.com/robertkrimen/otto/underscore]

14:48:57 06:16     [node]

14:48:57 06:16   [resources]

...

                     ========== 104 passed in 44.25 seconds ===========

                     

14:50:51 00:48     [junit]

14:50:51 00:48     [go]

14:50:51 00:48     [node]

14:50:51 00:48   [complete]

               SUCCESS

```

The binary fetching code does not use requests which carries its own cert store (which has the root cert that signed bintray's - below - see build-support/pants_dev_deps.venv/lib/python2.7/site-packages/requests/cacert.pem), but perhaps you can check if your openssl cert store has it?

# Issuer: CN=GeoTrust Global CA O=GeoTrust Inc.

# Subject: CN=GeoTrust Global CA O=GeoTrust Inc.

# Label: "GeoTrust Global CA"

# Serial: 144470

# MD5 Fingerprint: f7:75:ab:29:fb:51:4e:b7:77:5e:ff:05:3c:99:8e:f5

# SHA1 Fingerprint: de:28:f4:a4:ff:e5:b9:2f:a3:c5:03:d1:a3:49:a7:f9:96:2a:82:12

# SHA256 Fingerprint: ff:85:6a:2d:25:1d:cd:88:d3:66:56:f4:50:12:67:98:cf:ab:aa:de:40:79:9c:72:2d:e4:d2:b5:db:36:a7:3a

[Eric Zundel Ayers]

+pants-devel

We don't have to make a decision right away, but I'd like for things to work for Pants 1.0 users out of the box.

I'm working around this locally with either my local patch or and/or uploading the go binary to my local binary repo (we use go 1.5.0 and 1.5.1 locally anyway)

On Thu, Dec 17, 2015 at 5:27 PM, Eric Zundel Ayers <zun...@squareup.com> wrote:

I checked the Keychain Access for our macs which do have a certificate for GeoTrust Global CA and the signature matches. 

We aren’t the only environment that has run into this: https://github.com/Homebrew/homebrew/issues/38226 which implies a ‘modern’ brew installation with openssl breaks.

We are using openssl 1.0.2e

brew upgrade openssl


Error: openssl 1.0.2e already installed

Even if we were to get to the bottom of this, I can’t fix everyone’s laptop at Square that has this problem, but I could stick a value to not check certs in pants.ini.

I have this patch which works around it by just not verifying certs. Right now, the patch changes pants behavior to not check by default

I’d prefer to just flip the cert verification off by default. If that’s too dangerous for your taste then I will propose that I flip the logic so that the behavior stays the same by default, but you can flip off cert verification.

​

[Eric Zundel Ayers]

Just to follow up on this, I resolved the problem in  https://rbcommons.com/s/twitter/r/3275/ which I just pushed to master.

The solution I went with was to use the same Fetcher library that ivy bootstrapper does that uses the requests library.  Not only does this just "work" for me out of the box, it makes the HTTP based downloading work consistently and we already have a troubleshooting section for how to deal with certificate problems using the requests library.