SSL: CERTIFICATE_VERIFY_FAILED with wunderfixer

[graham.hagen]

When I run wunderfixer I get the following error...

Unable to open Weather Underground station KWAOLYMP77  or  <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)> 

It was working just fine a few days ago, but now I just get this error. 

Any help would be greatly appreciated.

[Thomas Keffer]

Please post more of the log than a single line. We need to see if this is in the http GET or POST.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gjr80]

FWIW I am seeing the same thing:

gary@stormbird:/usr/share/weewx$ ./wunderfixer --date=2016-05-04 --log=weewx
Using configuration file /etc/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_mysql'
Unable to open Weather Underground station XXXXXXXXXXX  or  <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
Could not get Weather Underground data. Exiting.

and the same thing in log speak:

May  5 20:24:02 stormbird weewx[12779]: wunderfixer: Using weewx configuration file /etc/weewx/weewx.conf.
May  5 20:24:02 stormbird weewx[12779]: wunderfixer: Using database binding 'wx_binding', which is bound to database 'archive_mysql'
May  5 20:24:03 stormbird weewx[12779]: wunderfixer: Unable to open Weather Underground station XXXXXXXXXXX or <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
May  5 20:24:03 stormbird weewx[12779]: wunderfixer: Could not get Weather Underground data. Exiting.

My obfuscation of the station ID. This occurred on a 3.5.0 deb install that was installed yesterday on Debian Jessie (am now trying to catch up on about 30 hours of outage). Was previously running 3.3.1 on the same machine but with Debian Wheezy and it ran quite happily at 2358 each night with no errors. Unfortunately I don't have the time just now to look into it further.

Gary

[Thomas Keffer]

So we know it's on the GET.

FWIW, I have been unable to reproduce this error.

I don't know why it's using a connection requiring SSL. All URLs use a simple http connection.

My suspicions are the problems are on the WeatherUnderground end, but don't have any proof of that.

-tk

--

[vince]

=On Thursday, May 5, 2016 at 5:30:17 AM UTC-7, Tom Keffer wrote:

I don't know why it's using a connection requiring SSL. All URLs use a simple http connection.

Tom did WU switch their setup to quietly forcing requests on 80 to use 443 like google did for many things ?

I just looked up my site at http://www.wunderground.com/personal-weather-station/dashboard?ID=KWAFEDER15 and it switched to https on me when the page resolved....

[Thomas Keffer]

Vince, I think you're on to something.

Starting with Python 2.7.9, the default is to require certificate verification (PEP 476). Could those of you who are having this problem please check your version number?

python -V

-tk

--

[Andrew Milner]

Tom

FYI - Mine, that works, on RPi Wheezy is 2.7.3

On my new RPi3 - with Jessie - that I've not used with weewx yet - is 2.7.9

Andrew

[Steve]

I didn't know I had this problem. 

Every night at 23:57 I have cron run /home/weewx/bin/wunderfixer and no errors appear in any of my logs.

I just ran it manually and got the error:

root@rpi3:~# /home/weewx/bin/wunderfixer
Using configuration file /home/weewx/weewx.conf.

Using database binding 'wx_binding', which is bound to database 'archive_mysql'

Unable to open Weather Underground station IVICTORI391  or  <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

Could not get Weather Underground data. Exiting.

root@rpi3:~#

Raspberry Pi3 running Jessie,

root@rpi3:~# python -V
Python 2.7.9
root@rpi3:~#

[vince]

On Thursday, May 5, 2016 at 8:25:12 AM UTC-7, Steve wrote:

I didn't know I had this problem. 

Every night at 23:57 I have cron run /home/weewx/bin/wunderfixer and no errors appear in any of my logs.

Try something like the following (untested):

   57 23 * * * /home/weewx/bin/wunderfixer 2>>/var/tmp/wunderfixer.errors

Basically call it specifying stderr 'appends' to a file in some scratch location.....

[Steve]

Thanks for that, I'll give it a shot. I'll probably send it to /var/log/cron.log on my machine.

It works when I do it manually, I'll see how it goes tonight. 

Regards,

Steve.

[gjr80]

Didn't realise it myself until last night but using --log=weewx on the wunderfixer command line logs wunderfixer output to whatever log file weewx uses.

Gary

[Liz]

On Thu, 5 May 2016 07:14:54 -0700
Thomas Keffer <tke...@gmail.com> wrote:

> Vince, I think you're on to something.
>
> Starting with Python 2.7.9, the default is to require certificate

> verification (PEP 476 <https://www.python.org/dev/peps/pep-0476/>).

> Could those of you who are having this problem please check your
> version number?
>
> python -V

I can confirm that I am using Python 2.7.9

$ wunderfixer

Using configuration file /etc/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database

'archive_mysql' Unable to open Weather Underground station INSWBEEL2

[Thomas Keffer]

So, we think we understand what's going on. 

Frustrating: if the Weatherunderground is going to silently switch http requests to https requests, you'd think they'd get their certificates right. 

It really should be fixed on their end. To change things on our end, we'd have to do https requests, then override the certificate verification, as per PEP 476. That seems perverse.

-tk

[gjr80]

Frustrating: if the Weatherunderground is going to silently switch http requests to https requests, you'd think they'd get their certificates right. 

Seems to be a lot of coincidences going on here, WU changing to https, Python changing the handling of https sites in 2.7.9 (which appears to have happened back in 2014), lots of people upgrading to Debian Jessie based dists (seemingly) resulting in them moving to Python 2.7.9. Also interesting that the same URL used by wunderfixer to get station history works without issue in my browser, but I understand, again from googling, that browsers can pull a few tricks to get around a troublesome https site.

I was worried that some other code I use with weewx to pull in WU almanac, forecast and conditions data via their API might have similar issues. It doesn't the API still uses http (no s). So I guess this means the forecast extension is safe as well.

It really should be fixed on their end. To change things on our end, we'd have to do https requests, then override the certificate verification, as per PEP 476. That seems perverse.

Interesting while I was googling this issue I came across the site https://www.ssllabs.com/ssltest/index.html which purportedly does a 'deep analysis of the configuration of any SSL web server on the public Internet'. It gave WU a thumbs up with an 'A'. Whilst WU does some things well I get the feeling they are not too responsive to (some, many, most ?) customer issues, is there any real prospect of them chnaging/fixing the issue? As perverse as the case by case work around in PEP 476 is do we have any other practical choice? The funy thing will be that if a weewx user is using an earlier version of python everything will work fine for them, they will effectively have the PEP 476 workaround.

Gary

[vince]

On Friday, May 6, 2016 at 3:47:46 PM UTC-7, gjr80 wrote:

is there any real prospect of them chnaging/fixing the issue? As perverse as the case by case work around in PEP 476 is do we have any other practical choice? 

I ran into this a couple weeks ago trying to get a windows server to download puppet modules.  The upstream vendor's site was signed with a certificate that Windows doesn't trust (yet Linux does - odd).  The fix was for me to automate downloading and installing  a cert that 'one' cert for that 'one' upstream provider from their site.   Ugly.

Has anybody looked at the WU trust chain enough to figure out how to make the client (wunderfixer) os trust their cert ?  That's the only real solution that makes sense to me long-term.  The odds of getting WU to do anything are pretty sure to be zero-ish.

[vince]

On Friday, May 6, 2016 at 3:47:46 PM UTC-7, gjr80 wrote:

 Also interesting that the same URL used by wunderfixer to get station history works without issue in my browser, but I understand, again from googling, that browsers can pull a few tricks to get around a troublesome https site.

can you get the same data from python ?

that would be really interesting.  Hope it fails just like wunderfixer, right ? 

[gjr80]

Has anybody looked at the WU trust chain enough to figure out how to make the client (wunderfixer) os trust their cert ?  That's the only real solution that makes sense to me long-term.  The odds of getting WU to do anything are pretty sure to be zero-ish.

No SSL expert but did some more googling and came across some interesting info on a few stackoverflow threads. This one was probably of most value http://stackoverflow.com/questions/27804710/python-urllib2-ssl-error/27826829#27826829

I did some playing around with some python and certificates and I could not get it to verify. The openssl bug mentioned was fixed at 1.0.2b but I am using Debian and my up to date openssl is only 1.0.1k. I think I will just patch my wunderfixer with the PEP 476 workaround and come back to it later, too many other more important things to do.

Gary

[Chris Mattingly]

I was only able to break this on my CentOS 6.7 / python 3.5.1 setup by removing /both/ of the following CAs from the OS default CA trust store (/etc/pki/tls/certs/ca-bundle.crt):

CN=GTE CyberTrust Global Root
CN=Baltimore CyberTrust Root

With either, or both, of these in the CA trust store, python3.5 was happy as it could be connecting to https://api.wunderground.com/.  But once both were removed, my test line promptly provided the CERTIFICATE_VERIFY_FAILED error.

Check your OS trust store and make sure both of these certs are in there and I'd expect this to rectify the problem.

Lastly, not to defend WU, but, they do have their certificates right.

-Chris

--

[Liz]

On Fri, 6 May 2016 23:20:08 -0400
Chris Mattingly <ch...@camattin.com> wrote:

> I was only able to break this on my CentOS 6.7 / python 3.5.1 setup by
> removing /both/ of the following CAs from the OS default CA trust
> store (/etc/pki/tls/certs/ca-bundle.crt):
>
> CN=GTE CyberTrust Global Root
> CN=Baltimore CyberTrust Root
>
> With either, or both, of these in the CA trust store, python3.5 was
> happy as it could be connecting to https://api.wunderground.com/.
> But once both were removed, my test line promptly provided the
> CERTIFICATE_VERIFY_FAILED error.
>
> Check your OS trust store and make sure both of these certs are in
> there and I'd expect this to rectify the problem.
>
> Lastly, not to defend WU, but, they do have their certificates right.
>
> -Chris

Thanks for the tip, Chris.
I'm on Debian stretch/sid.
I checked that I had the latest openssl installed. /etc/ssl/certs
contains both of

Baltimore_CyberTrust_Root.pem

Cybertrust_Global_Root.pem

and I have a non-working "wunderfixer".

Liz

[vince]

I gave up trying to get the cert stuff working, so here's the code to disable strict checking.

Add the following to wunderfixer at about line 59, right above the 'usagestr' block......

Tests ok for me on 2.7.9 (a pi3) and 2.7.3 (old debian)

#---- start patch to disable ssl verification ----

#

# wunderground.com quietly redirects http=>https and

# python 2.7.9 turns on strict certificate verification

# which breaks the original wunderfixer 0.5.2

#

# long discussion of the internals is at

#     http://stackoverflow.com/questions/27804710/python-urllib2-ssl-error/27826829#27826829

#

# the following code is taken verbatim from

#     https://dnaeon.github.io/disable-python-ssl-verification/

import ssl

try:

    _create_unverified_https_context = ssl._create_unverified_context

except AttributeError:

    # Legacy Python that doesn't verify HTTPS certificates by default

    pass

else:

    # Handle target environment that doesn't support HTTPS verification

    ssl._create_default_https_context = _create_unverified_https_context

#---- end patch to disable ssl verification ----

[Steve]

Thanks for that Vince, your mod worked for me too. 

It was easy to find where to put it, but in my editor it was about line 65 (for wunderfixer 1.0.0 8/16/15 Published version).

Regards,

Steve.

[vince]

On Saturday, May 7, 2016 at 8:37:05 AM UTC-7, Steve wrote:

Thanks for that Vince, your mod worked for me too. 

It was easy to find where to put it, but in my editor it was about line 65 (for wunderfixer 1.0.0 8/16/15 Published version).

yeah - I couldn't find the current one.

I got pointed to sourceforge which must have been an old version. 

[vince]

Tom/Matthew - I opened a PR for this patch.......

       disable ssl verification     

[gjr80]

Well, I think I had some success with getting the certificate to verify under Debian Jessie. On a VM running Debian Jessie 8.3 installing the GTE CyberTrust Global Root certificate to my CA (Baltimore CyberTrust Root was already there) seems to have done the trick and my test program now does a 'wunderfixer like urllib2.urlopen' without the SSL verification error.

Unfortunately I am out of time just now to trial this on my on-line system, will do that later tonite or tomorrow. Only a handful of commands and some copying and pasting so hopefully an easy long term fix.

Gary

[Liz]

I've installed GTE CyberTrust Global Root certificate to my CA,
disabled the patch which disables the SSL checking
and wunderfixer fails as before
Baltimore CyberTrust Root was installed.

Liz

[gjr80]

Interesting, I just had time to sit down and try things out on 2 systems, one an RP1 model B running Rapsbian Jessie and the other an Odroid U3 running Debian Jessie. Same situation/results, both had the Baltimore certificate but not the GTE certificate. Both gave the certificate verification error when running my https test script (which does a WU urllib2.urlopen()), the Odroid gave the same error when running wunderfixer unpatched. Installed the GTE certificate on both and both now run my https test script without error and the Odroid runs wunderfixer unpatched without error. Looks like the solution for me.

FWIW the steps I took were:

1. Inspected contents of /etc/ssl/certs and confirmed presence or absence of GTE CyberTrust Global Root or Baltimore CyberTrust Root certificates (in my case Baltimore was there but GTE was not)
2. Ran wunderfixer unpatched and/or my https test script and confirmed https certificate verification error occurred.
3. Created GTE_CyberTrust_Global_Root.crt file in /usr/local/share/ca-certificates using the command:

    $ sudo nano /usr/local/share/ca-certificates/GTE_CyberTrust_Global_Root.crt

4. Copied GTE certificate from this link (Googled it). Copied line starting -----BEGIN CERTIFICATE----- through -----END CERTIFICATE----- inclusive
5. Pasted the copied certificate into /usr/local/share/ca-certificates/GTE_CyberTrust_Global_Root.crt
6. Saved /usr/local/share/ca-certificates/GTE_CyberTrust_Global_Root.crt
7. Installed the new certificate using:

    $ sudo update-ca-certificates

8. Ran wunderfixer unpatched and/or my https test script and confirmed https certificate verification error no longer occurred.
9. Removed GTE certificate by:

    $ sudo rm  /usr/local/share/ca-certificates/GTE_CyberTrust_Global_Root.crt
    $ sudo update-ca-certificates

10. Ran wunderfixer unpatched and/or my https test script and confirmed https certificate verification error had returned.

My https test script is simply:

#!/usr/bin/python

import urllib2

wu_data = urllib2.urlopen("https://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IQUEENSL336&month=5&day=4&year=2016&format=1")
for row in wu_data:
    print row

If this works for anyone else I will write it up in the wiki.

Gary

[Liz]

On Sun, 8 May 2016 02:37:08 -0700 (PDT)
gjr80 <gjrod...@gmail.com> wrote:

> 4. Copied GTE certificate from this link

> <https://www.tbs-certificates.co.uk/FAQ/en/31.html> (Googled it).

> Copied line starting -----BEGIN CERTIFICATE----- through -----END
> CERTIFICATE----- inclusive
> 5. Pasted the copied certificate into
> /usr/local/share/ca-certificates/GTE_CyberTrust_Global_Root.crt
> 6.
> Saved /usr/local/share/ca-certificates/GTE_CyberTrust_Global_Root.crt

I did sudo mkdir /usr/local/share/ca-certificates/
cd /usr/local/share/ca-certificates/
sudo wget https://www.digicert.com/CACerts/GTECyberTrustGlobalRoot.crt

(from the link on this page
https://www.digicert.com/digicert-root-certificates.htm)


however, adding this cert to my testing/unstable Debian with Python
2.7.9 did not allow the unpatched wunderfixer to work.

You are on Jessie. This suggests that your fix works in Jessie, and
will be broken again in Stretch.

Liz

[gjr80]

If I download the certificate at the your wget statement and then import it with sudo update-ca-certificates unpatched wunderfixer and my test script both fail. I think I will worry about Stretch if and when we come to that hurdle.

Gary

[gjr80]

OK, curiosity got the better of me. Downloaded the latest Stretch i386 iso cd build (2 May) and ran it up on a VM. My https test script worked out of the box as did unpatched wunderfixer. Baltimore CyberTrust Root was in /etc/ssl/certs but not GTE CyberTrust Global Root. In this case I don't think Stretch wont be such a big deal.

Gary

[Thomas Keffer]

I was finally able to reproduce the error by using a Docker Jessie image running Python 2.7.9.

Then I was able to fix the error by doing the following

cd /usr/local/share/ca-certificates

sudo wget  http://www.tbs-x509.com/GTECyberTrustGlobalRoot2018.crt

sudo update-ca-certificates

I then tried Stretch, and wunderfixer worked fine out of the box.

I then added the above certificate to Stretch, and wunderfixer continued to work fine.

Personally, I feel better about asking users to fix their certificates, rather than patching wunderfixer to ignore certificate validation.

-tk

--

[Liz]

On Sun, 8 May 2016 09:15:24 -0700
Thomas Keffer <tke...@gmail.com> wrote:

> I was finally able to reproduce the error by using a Docker Jessie
> image running Python 2.7.9.
>
> Then I was able to fix the error by doing the following
>
> cd /usr/local/share/ca-certificates
> sudo wget http://www.tbs-x509.com/GTECyberTrustGlobalRoot2018.crt
> sudo update-ca-certificates
>
>
> I then tried Stretch, and wunderfixer worked fine out of the box.
>
> I then added the above certificate to Stretch, and wunderfixer
> continued to work fine.
>
> Personally, I feel better about asking users to fix their
> certificates, rather than patching wunderfixer to ignore certificate
> validation.
>
> -tk
>

I've just imported the certificate for GTE CyberTrust as described by
Gary. this time wunderfixer works in its original form.
Comparing the two downloads they are quite different, but Tom got the
crt from digicert to work, and I didn't.

I thought it was better security wise to get the certificate from
digicert rather than the third party tbs, but this didn't work for me.

It would be better overall if the Debian certificates were more
complete. I found other evidence on the net of other projects having to
import additional certificates into Debian jessie and above.

Liz

[Thomas Keffer]

Liz, I was wondering about that. In your earlier email you reported downloading GTE CyberTrust certificate from digicert, but you didn't report rebuilding the certificate cache using update-ca-certificates. Did you just forget to mention it?

-tk

Liz

[Liz]

On Sun, 8 May 2016 16:28:21 -0700
Thomas Keffer <tke...@gmail.com> wrote:

> Liz, I was wondering about that. In your earlier email you reported
> downloading GTE CyberTrust certificate from digicert, but you didn't
> report rebuilding the certificate cache using update-ca-certificates.
> Did you just forget to mention it?
>
> -tk

I certainly rebuilt the certificate store with sudo
update-ca-certificates.

I didn't mention it because I was just commenting on what seemed to be
an easier fix (wget) instead of using an editor and copy paste.

Liz