rtupdate.wunderground.com certificate error

[Brice Ruth]

Just a heads up (I've reported this to wunderground) -

It looks like the SSL/TLS certificate for `rtupdate.wunderground.com` is incorrect or the API has been compromised in some way. Accessing `https://rtupdate.wunderground.com` in a browser such as Chrome yields a security warning: NET::ERR_CERT_COMMON_NAME_INVALID and the `weewx` software used to upload data to wunderground reports `hostname 'rtupdate.wunderground.com' doesn't match either of '*.
prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.u
s-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'` - maybe the DNS was switched to a new API endpoint but the SSL/TLS certificate that matches the DNS name wasn't configured correctly?

 

This is blocking all weather station updates to `rtupdate.wunderground.com` that are using secure transport (which should be required because security credentials are passed in the requests).

[Denny Page]

I also started seeing this. You can also see the error here: https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php

The certificate that is up there is a 90 day Let's Encrypt certificate, with no alternate names at all. Essentially a bogus certificate.

I reported it to Wundergound support around 17:00 yesterday. Still no response...

[Kevin Rowett]

About 22:00 UTC Jan 29, I started getting this error:

restx: Wunderground-PWS: Unexpected exception of type <class 'ssl.CertificateError'>

weewx[598]: restx: Wunderground-PWS: Thread exiting. Reason: hostname 'weatherstation.wunderground.com' doesn't match either of '*.prod-pw

s-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.cont

ainers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

KR

[Dave McCreath]

Exactly the same here https://www.wunderground.com/dashboard/pws/ICAMBSLE2

Jan 30 15:15:15 raspberrypi weewx[18320]: restx: Wunderground-PWS: Unexpected exception of type <class 'ssl.CertificateError'>

Jan 30 15:15:15 raspberrypi weewx[18320]: restx: Wunderground-PWS: Thread exiting. Reason: hostname 'weatherstation.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

Also posted the issue on wxforum.net, potential short term solution attached but I'll probably wait until WU updates the certificate (as I've no idea how to switch the upload to HTTP instead of HTTPS)

.

[Dave McCreath]

Looks like WU are onto this, see attached from apicommunity.wunderground.com/.

[J B]

Same problem here. Thanks for posting the response from WU.

[Denny Page]

I received an email from Wunderground "Yes, there was a problem that has been resolved."

While the certificate error has been resolved, it appears that the https head still is not accepting updates. The http head appears fully functional. I've reported the secondary problem to them.

[Dave McCreath]

Snap! 

SSL Certificate issue resolved but data still not being accepted via https.

Webcam updates are getting through.

[J B]

I just tried running the request that Weewx attempts manually via curl and it still gives me a certificate error:

curl: (60) SSL certificate problem: unable to get local issuer certificate

Maybe there's some caching involved?

[Kevin Rowett]

I no longer get a ssl cert error. 

Now get this error:

restx: Wunderground-PWS: Failed to publish record 2020-01-30 20:08:00 PST (1580443680): Failed upload after 3 tries

KR

[Dale Cav]

I have the same issue

restx: Wunderground-PWS: Failed to publish record 2020-01-31 11:05:00 UTC (1580468700): Failed upload after 3 tries

[Dave McCreath]

Spoke too soon.

SSL Certificate error went away last night, still failed to upload (on third error), now back to SSL Certificate error again!

Jan 31 10:50:27 raspberrypi weewx[545]: restx: Wunderground-PWS: Thread exiting. Reason: hostname 'weatherstation.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

Bummer.

[Brice Ruth]

Yeah, I'm still seeing errors:

Jan 31 10:18:28 raspberrypi weewx[9240]: restx: Wunderground-RF: Failed to publish record 2020-01-31 10:18:22 CST (1580487502): Failed upload after 1 tries

Brice Ruth, FCD
Software Engineer, Madison WI

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/730af4c4-6fea-446f-ac0a-3ef112d65739%40googlegroups.com.

[Brice Ruth]

Eventually it exits with the SSL error again ...

Jan 31 10:20:44 raspberrypi weewx[9240]: restx: Wunderground-RF: Thread exiting. Reason: hostname 'rtupdate.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

 

Brice Ruth, FCD
Software Engineer, Madison WI

[Denny Page]

For those of you still experiencing an issue, I posted this on the Wunderground apicommunity thread:

So, the certificate isn't verifiable due to authority trust. Easily identifiable with wget or curl. It would appear that the certificate authority ("DigiCert SHA2 Secure Server CA") is not trusted with at least some of the Linux distributions. It is trusted in current MacOs.

-----

denny ~ $ wget https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php
--2020-01-31 08:28:41--  https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php
Resolving rtupdate.wunderground.com... 169.60.133.170, 169.47.111.58, 52.116.188.166
Connecting to rtupdate.wunderground.com|169.60.133.170|:443... connected.
ERROR: cannot verify rtupdate.wunderground.com's certificate, issued by 'CN=DigiCert SHA2 Secure Server CA,O=DigiCert Inc,C=US':
  Unable to locally verify the issuer's authority.
To connect to rtupdate.wunderground.com insecurely, use `--no-check-certificate'.
denny ~ $ curl https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
denny ~ $ 

[Brice Ruth]

Yeah - it's the intermediate certificate. DIgicert's tool even says what's wrong with it - SSL works on a "chain of trust" - so, most distributions trust the "root" certificates - then they issue "intermediate" certificates and then sign customer's certificates with that. This requires customers to publish both the server & intermediate certificate(s) to allow the client to "chain" back to the root certificate that it trusts.

If it helps anyone - I've modified my weewx install to use http temporarily ... I added this in weewx/restx.py around line 597

        use_http_to_post = to_bool(_ambient_dict.pop('use_http', False))
        if use_http_to_post:
            StdWunderground.pws_url = re.sub("https","http",StdWunderground.pws_url)
            StdWunderground.rf_url = re.sub("https","http",StdWunderground.rf_url)

Then a new config flag in weewx.conf `use_http` will let you toggle on/off

Brice Ruth, FCD
Software Engineer, Madison WI

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6f2decf9-4c1a-48f6-8a88-6eac8f15c62e%40googlegroups.com.

[Denny Page]

Wunderground just posted a note about the intermediate certificate issue. Hopefully they will fix it shortly.

[Travis Bully]

For what it's worth, I just updated my keystore with their "bad" digicert intermediate to fix my issue.  I've attached the cert here if you need it.

[Brice Ruth]

Yeah, that's not a bad way to fix the problem, honestly. Better than sending credentials over cleartext, I just hate messing with cacerts stores that the system manages. Pain and suffering often follow :(

Brice Ruth, FCD
Software Engineer, Madison WI

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/42a3d359-85a4-4f52-9e57-c66da178ee9e%40googlegroups.com.

[Travis Bully]

Ubuntu makes it pretty straightforward.  Copy the file to /usr/share/ca-certificates and run update-ca-certificates.  Better than the old days, anyway.

This DID fix my problem but it does appear that they continue to mess with their services.  restx will sometimes get the below error which doesn't look like the same intermediate digicert problem.  Afterwards, Weewx needs a restart and is then "fixed" for awhile.  

Jan 31 13:07:01 homeauto03 weewx[3186]: *** CertificateError: hostname 'rtupdate.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'
Jan 31 13:07:01 homeauto03 weewx[3186]: restx: Wunderground-RF: Thread exiting. Reason: hostname 'rtupdate.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/mYhw6CSKHHg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAFbExW5sQSEDwyWHpgnx5WEwzg3K0FaJ6BAJeSnnmittUyEwYg%40mail.gmail.com.

[Glenn Godden]

In the weewx /usr/share/weewx/weewx/restx.py file if you change from "https" to "http" it will resolve the weatherunderground issue:

"https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php" to

 

"http://weatherstation.wunderground.com/weatherstation/updateweatherstation.php"

Your may need to change file permissions to read+write to edit the file:

sudo chmod a+w /usr/share/weewx/weewx/restx.py

You will need to restart weewx post the change

: Glenn

On Thursday, January 30, 2020 at 7:57:14 AM UTC-8, Brice Ruth wrote:

[Kevin Rowett]

Be advised...by changing from https protocol to http, your WX station password/API key will be sent in plain text, and if your ISP is comcast...

KR

[Glenn Godden]

Concur, an informed decision to be made by the station owner until weatherunderground resolves their SSL issues.

: Glenn

[Kevin Key]

Thanks Glen for the detailed instructions. This fixed the problem. And Kevin R. and all, yeah I plan on changing it back to the SSL link and changing my password once Weather Underground gets their act together and fixes this issue.

On Friday, January 31, 2020 at 10:24:26 AM UTC-8, Glenn Godden wrote:

[Denny Page]

It's fixed now.

[Travis Bully]

Agreed.  Seems like I still get one cert error every now and then.  Likely still propagating changes through their infrastructure?

Jan 31 14:29:25 homeauto03 weewx[3784]: restx: Wunderground-RF: Published record 2020-01-31 14:29:24 EST (1580498964)
Jan 31 14:29:26 homeauto03 weewx[3784]: restx: Wunderground-RF: Failed upload attempt 1: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>
Jan 31 14:29:31 homeauto03 weewx[3784]: restx: Wunderground-RF: Failed to publish record 2020-01-31 14:29:26 EST (1580498966): Failed upload after 1 tries
Jan 31 14:29:31 homeauto03 weewx[3784]: restx: Wunderground-RF: Published record 2020-01-31 14:29:30 EST (1580498970)

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/mYhw6CSKHHg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f5e5741a-e5f8-419b-9c0d-df1972fe5ced%40googlegroups.com.

[Brice Ruth]

Looking like it's working to me, too.

Brice Ruth, FCD
Software Engineer, Madison WI

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAF_%2Bh57S88%3DZOX1mXK8oZBx-_KEHvDPFP%3Dvtd4u3_1qg7994yg%40mail.gmail.com.

[J B]

Working here too. I had to restart Weewx.

On Friday, January 31, 2020 at 11:34:19 AM UTC-8, Brice Ruth wrote:

Looking like it's working to me, too.

Brice Ruth, FCD
Software Engineer, Madison WI

On Fri, Jan 31, 2020 at 1:31 PM Travis Bully <tbu...@gmail.com> wrote:

Agreed.  Seems like I still get one cert error every now and then.  Likely still propagating changes through their infrastructure?

Jan 31 14:29:25 homeauto03 weewx[3784]: restx: Wunderground-RF: Published record 2020-01-31 14:29:24 EST (1580498964)
Jan 31 14:29:26 homeauto03 weewx[3784]: restx: Wunderground-RF: Failed upload attempt 1: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>
Jan 31 14:29:31 homeauto03 weewx[3784]: restx: Wunderground-RF: Failed to publish record 2020-01-31 14:29:26 EST (1580498966): Failed upload after 1 tries
Jan 31 14:29:31 homeauto03 weewx[3784]: restx: Wunderground-RF: Published record 2020-01-31 14:29:30 EST (1580498970)

On Fri, Jan 31, 2020 at 2:30 PM Denny Page <splo...@gmail.com> wrote:

It's fixed now.

On Friday, January 31, 2020 at 8:54:02 AM UTC-8, Denny Page wrote:

Wunderground just posted a note about the intermediate certificate issue. Hopefully they will fix it shortly.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/mYhw6CSKHHg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to weewx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f5e5741a-e5f8-419b-9c0d-df1972fe5ced%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[Travis Bully]

Not here yet.  It'll work for awhile and then will get the random cert error I sent earlier.  A restart of weewx will get it going again.  I wish restx would just try again after xx seconds.

[Mark Branch]

Everything works for me now.  No restart was required.

[Thomas Keffer]

Travis: what is the error? If it's a certificate error, Version 4 will do a retry after an hour. Unfortunately, Version 3 does not.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/eaa42f60-98fc-4974-8d25-915a4c7e0d80%40googlegroups.com.

[Travis Bully]

Hey Thomas - 

Thanks for asking.  Error string is below.  I'm running 3.9.2 for now.  It looks like I run in to a bad cert every 4-5 minutes.  Looking at their DNS, they appear to only have three servers running and they all have the same certificate string from "Let's Encrypt".  Odd.

Jan 31 16:11:04 homeauto03 weewx[8713]: restx: Wunderground-RF: Unexpected exception of type <class 'ssl.CertificateError'>

Jan 31 16:11:04 homeauto03 weewx[8713]: *** Traceback (most recent call last):

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/share/weewx/weewx/restx.py", line 354, in run_loop

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self.process_record(_record, dbmanager)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/share/weewx/weewx/restx.py", line 417, in process_record

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self.post_with_retries(_request, data)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/share/weewx/weewx/restx.py", line 442, in post_with_retries

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     _response = self.post_request(request, data)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/share/weewx/weewx/restx.py", line 509, in post_request

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     _response = urllib2.urlopen(request, data=data, timeout=self.timeout)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     return opener.open(url, data, timeout)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/urllib2.py", line 429, in open

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     response = self._open(req, data)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/urllib2.py", line 447, in _open

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     '_open', req)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     result = func(*args)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     context=self._context)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/urllib2.py", line 1195, in do_open

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     h.request(req.get_method(), req.get_selector(), req.data, headers)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/httplib.py", line 1069, in request

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self._send_request(method, url, body, headers)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/httplib.py", line 1109, in _send_request

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self.endheaders(body)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/httplib.py", line 1065, in endheaders

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self._send_output(message_body)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/httplib.py", line 892, in _send_output

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self.send(msg)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/httplib.py", line 854, in send

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self.connect()

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/httplib.py", line 1290, in connect

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     server_hostname=server_hostname)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     _context=self)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/ssl.py", line 599, in __init__

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     self.do_handshake()

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/ssl.py", line 836, in do_handshake

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     match_hostname(self.getpeercert(), self.server_hostname)

Jan 31 16:11:04 homeauto03 weewx[8713]: ***   File "/usr/lib/python2.7/ssl.py", line 288, in match_hostname

Jan 31 16:11:04 homeauto03 weewx[8713]: ***     % (hostname, ', '.join(map(repr, dnsnames))))

Jan 31 16:11:04 homeauto03 weewx[8713]: *** CertificateError: hostname 'rtupdate.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

Jan 31 16:11:04 homeauto03 weewx[8713]: restx: Wunderground-RF: Thread exiting. Reason: hostname 'rtupdate.wunderground.com' doesn't match either of '*.prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567-997b58a668d15d562a6bed58ea7c5f9e-0000.us-south.containers.appdomain.cloud', 'prod-pws-ng-546567.us-south.containers.appdomain.cloud'

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/eaa42f60-98fc-4974-8d25-915a4c7e0d80%40googlegroups.com.

[Leon Shaner]

Hey, WeeWx'ers.

I took a different approach to working around the WU certificate issue -- keep SSL but don't verify the CERT.  Scroll for that "solution."

But meanwhile, using Wunderground-RF (rapid fire) I am getting an HTTP Error 404:  Not Found.

Any ideas on that?   Did they change the URL, or is it just down?

I noticed that even their own WunderStation App on iOS/iPadOS is subject to the certificate issue and reports all stations as "not reporting" -- even ones that are, like mine, which I can verify via the WU website has the data and has reported within minutes ago.

From the logs, re: RapidFire HTTP 404:

Feb  1 12:31:51 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Feb  1 12:31:51 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed to publish record 2020-02-01 12:31:51 EST (1580578311): Failed upload after 1 tries

Feb  1 12:31:56 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Feb  1 12:31:56 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed to publish record 2020-02-01 12:31:56 EST (1580578316): Failed upload after 1 tries

Feb  1 12:32:04 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Feb  1 12:32:04 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed to publish record 2020-02-01 12:32:04 EST (1580578324): Failed upload after 1 tries

Feb  1 12:32:05 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Feb  1 12:32:05 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 tries

Feb  1 12:32:06 nixie weewx[13184] DEBUG weewx.restx: Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Feb  1 12:32:06 nixie weewx[13184] ERROR weewx.restx: Wunderground-RF: Failed to publish record 2020-02-01 12:32:05 EST (1580578325): Failed upload after 1 tries

Here is the diff for my SSL CERT workaround:

$ diff restx.py restx.py.20200201.1

110,115d109

< # SSL certificate hack

< global WUssl

< WUssl = ssl.create_default_context();

< WUssl.check_hostname=False

< WUssl.verify_mode=ssl.CERT_NONE

<

454d447

<         _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)

543,544c536

< #        _response = urllib.request.urlopen(request, data=data_bytes, timeout=self.timeout)

<         _response = urllib.request.urlopen(request, data=data_bytes, timeout=self.timeout, context=WUssl)

---

>         _response = urllib.request.urlopen(request, data=data_bytes, timeout=self.timeout)

1051,1052c1043

< #            _response = urllib.request.urlopen(request, timeout=self.timeout)

<             _response = urllib.request.urlopen(request, timeout=self.timeout, context=WUssl)

---

>             _response = urllib.request.urlopen(request, timeout=self.timeout)

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

On Jan 31, 2020, at 4:16 PM, Thomas Keffer <tke...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECD1_-w%3D8V5eb5-qYsgMBPcvnNunAOe%2BQbnqpKsOsyGUg%40mail.gmail.com.

[Leon Shaner]

Hey, Tom,

The reason I get 204 back from wunderfixer on date 2020-01-30 is I really did have 0 records uploaded to WU (due to a full day of certificate issues).

Under the new API an HTTP 204 is returned for a valid request for which there is no data.

Here is my workaround:

$ diff wunderfixer wunderfixer_tk

407,409c407

<                 # Valid response, it's just that WU has no records for the requested date

<                 # Return an empty list of TimeStamps (as in WU has no records)

<                 return {}

---

>                 raise IOError("Probably a bad station ID or invalid date")

Or if the line numbers don't match yours, it looks like this in context:

        if hasattr(response, 'code') and response.code != 200:

            if response.code == 204:

                # Valid response, it's just that WU has no records for the requested date

                # Return an empty list of TimeStamps (as in WU has no records)

                return {}

            else:

                raise IOError("Bad response code returned: %d" % response.code)

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

On Feb 1, 2020, at 12:40 PM, Leon Shaner <le...@isylum.org> wrote:

Hey, WeeWx'ers.

[John]

This edit to wunderfixer beta worked for me. I had about a week of missing data and was having trouble catching up.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/eaa42f60-98fc-4974-8d25-915a4c7e0d80%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zECD1_-w%3D8V5eb5-qYsgMBPcvnNunAOe%2BQbnqpKsOsyGUg%40mail.gmail.com.

[Brice Ruth]

Is anyone else still seeing errors? It works for a bit, then fails enough for weewx to fail out and I have to restart...

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f134fb8c-e276-4539-a75a-3c0d5a1260d1%40googlegroups.com.

--

[Brice Ruth]

I have 3.9.2-1 (latest available on my Pi 4).

[Cameron D]

I am seeing the occasional error in the logs, but running v4 beta 10 and it keeps  running and succeeds each time after the error.

Errors have been:

•  self-signed cert (3 times)
• cert for wrong hostname (using base server name instead of wunderground's cname) (twice)
• connection closed prematurely. (twice)

It looks to me as if they are fiddling around with their servers.  The server DNS provides 3 IPv4 addrs so maybe we occasionally pick one they are working on and they didn't give enough time for DNS changes to filter through.

[Brice Ruth]

I wonder if they forgot to update one...

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/321a1732-f7bd-4588-bc39-67e8440d80fd%40googlegroups.com.

[Brice Ruth]

So, is there a way to get 4.x on the you? Guessing I need to grab the source if it’s not published yet? 

[Brice Ruth]

on the *Pi

Stupid auto correct

[J B]

Like some of you I'm still getting intermittent errors that stop weewx 3.9.2 from uploading until I restart. Someone over at https://apicommunity.wunderground.com/ posted this creative workaround that seems to be working so far. It lowers the security of python but it's better than sending everything over http.

tmarschner

• 2 Posts
 
•  0 Reply Likes

Hi there,

Here's an alternative workaround that doesn't require you to send your Wunderground credentials in clear text. It turns off certificate validation for python. I've tested this on weewx 3.9.2 on a Raspberry Pi (v. 9 Stretch) running python 2.7.13.

Modify the weewx startup script /etc/init.d/weewx.

Find the start-stop-daemon line that starts up weewx (the do_start routine) and insert "/usr/bin/env PYTHONHTTPSVERIFY=0" into the --exec parameter as shown below:

start-stop-daemon --start --chuid $WEEWX_USER --pidfile $PIDFILE --exec /usr/bin/env PYTHONHTTPSVERIFY=0 $DAEMON -- $DAEMON_ARGS || return 2

Cheers,
Tom

[J B]

I should mention that I'm assuming that this workaround doesn't turn off certificate validation system-wide but I'm not well-versed enough in how this works to know for sure. Since the daemons are managed by root I guess it's possible.

[Dave McCreath]

JB

Many thanks for pointing me in the direction of the fix, applied (at line 61) and it appears to be working fine after a reboot. 

A quick query on Verion 4.xx of weewx for those Linux savvy bodies amongst us, how easy is it to install on an RPi and is an undated version of Wunderfixer included (that works)?

Regards

Dave

[Brice Ruth]

Thanks for this JB - to you or anyone else following the community threads - has anyone from WU piped in to say they're re-engaged to fix the remaining issues? I haven't seen any posts from them recently ...

Brice Ruth, FCD
Software Engineer, Madison WI

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/98106b14-546a-481b-a49c-c8cf3ec19d7c%40googlegroups.com.

[Leon Shaner]

Hey, WeeWx'ers,

I am still getting 404 errors from the rtupdate URL site.

There is a reply over here about it.

I'm now looking into what the headers need to be.

Does anyone know?

https://apicommunity.wunderground.com/weatherapi/topics/is-the-rtupdate-wunderground-com-address-no-longer-valid

For requests to route properly, the
host header must be set correctly
path must be correct

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

On Feb 1, 2020, at 12:39 PM, Leon Shaner <le...@isylum.org> wrote:

Hey, WeeWx'ers.

[Brice Ruth]

header would be something like `Host: rtupdate.wunderground.com` ... basically, whatever you're using as the hostname in the URL, needs to be sent as the `Host` header ... that allows load-balancers to route the request to the correct backend.

Brice Ruth, FCD
Software Engineer, Madison WI

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CE8C91F5-D61F-4618-A317-7E97B9ECE912%40isylum.org.

[Leon Shaner]

Brice,

Thanks!  That's interesting...  I just added an explicit Host header to the rtupdate request and put debug output there.
This was unexpected:

Feb  3 14:16:34 nixie weewx[8584] DEBUG weewx.restx: Wunderground-RF: Added Header: 'Host: weatherstation.wunderground.com'

That isn't the host that should be used for rapidfire, but it's taken from the URL being used.  :-(
We have:

$ grep wunderground.com restx.py
    rf_url = "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php"
    pws_url = "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php"

It will take me some time to figure out why restx.py is using the wrong URL.
Must have been doing this for a long time, but seems that due to changes on the WU side, they no longer accept rapidfire updates to the weatherstation.wunderground.com host.

Regards,
\Leon

le...@isylum.org - Dearborn, Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAFbExW60hmrJRd1yWJAK8f0H%2BXnZsnYe8%3DxMpvKe9n2ws4sq9w%40mail.gmail.com.

[Leon Shaner]

Tom,

Looking at the code, under the do_rapidfire_post, we clearly see it uses rf_url.  In my debug output I referenced protocol_name and it's showing Wunderground-RF as the protocol, but the hostname is from pws_url.
I have temporarily set pws_url to the rtupdate URL and my 404's have stopped.
Looks like the rtupdate URL will take regular posts, but the weatherstation URL won't take rf posts.  :-/

My debug with pws_url replaced with the rf_url value now shows:

Feb  3 14:36:30 nixie weewx[8747] DEBUG weewx.restx: Wunderground-RF: Added Header: 'Host: rtupdate.wunderground.com'

That's what it should have shown if rf_url was getting used for do_rapidfire_post.
I'm stumped.  :-(

Regards,
\Leon

le...@isylum.org - Dearborn, Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6438f7e3-b03f-4ea3-e324-d89640c88e09%40isylum.org.

[Leon Shaner]

Hey, WeeWx'ers.

Even my workaround of setting pws_url to the value of the rf_url is still getting 404's, even if I explicitly set a Host header.   :-(((((

This is what I have tried...  I guess WU is just a steaming pile as per usual.  :-/

Here is my attempted fix, which is part hack.
The "fix" is adding the header.   The "hack" is forcing the code to use the rtupdate URL (because I couldn't spot why the PWS URL was being used for RF).

$ diff restx.py restx.py_getting404
100d99
< from urllib.parse import urlparse
454,458c453
<         _parseduri = urlparse(url)
<         _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
<         _request.add_header("Host", "%s" % _parsedhost)
< #        log.debug("%s: Added Header: 'Host: %s'"
< #                  % (self.protocol_name, _parsedhost))
---

>         _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)

610,611c605
<     #pws_url = "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php"
<     pws_url = "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php"
---
>     pws_url = "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php"

Or in case your line numbers don't match mine (likely), here it is in context:

...
# Python 2/3 compatiblity shims
import six
from six.moves import http_client
from six.moves import queue
from six.moves import urllib
from urllib.parse import urlparse

...

    def get_request(self, url):
        """Get a request object. This can be overridden to add any special headers."""
        _request = urllib.request.Request(url)
        _parseduri = urlparse(url)
        _parsedhost = '{uri.netloc}'.format(uri=_parseduri)
        _request.add_header("Host", "%s" % _parsedhost)
#        log.debug("%s: Added Header: 'Host: %s'"
#                  % (self.protocol_name, _parsedhost))

        _request.add_header("User-Agent", "weewx/%s" % weewx.__version__)

        return _request


...

    # the rapidfire URL:
    rf_url = "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php"
    # the personal weather station URL:
    #pws_url = "https://weatherstation.wunderground.com/weatherstation/updateweatherstation.php"
    pws_url = "https://rtupdate.wunderground.com/weatherstation/updateweatherstation.php"

le...@isylum.org - Dearborn, Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c017d408-7abb-695c-8968-9058e22b3389%40isylum.org.

[Brice Ruth]

So, I'm a little stumped on why you're getting 404s unless you happen to be hitting a region that's totally horked - where are you physically located? I'm currently using 3.9.2-1 on my Pi 4 and I had temporarily disabled `https`, but now have `https` re-enabled, but with the recent addition of `/usr/bin/env PYTHONHTTPSVERIFY=0` to stop the weewx thread from failing out after awhile. I've used rapidfire from the start and I don't see anywhere that it's using the `weatherstation` URL - do you have `archive_post` enabled as well, maybe? My errors were always indicating failures in talking to `rtupdate...` as well, so I'm pretty sure on this side, it hasn't been talking to anything but the rapidfire URL ... just not sure why your setup seems to be doing that?

Brice Ruth, FCD
Software Engineer, Madison WI

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/45f04b3b-ae67-aaeb-9d25-625ea42b111b%40isylum.org.

[Leon Shaner]

Tom, All,

On that WU forum, an official rep said the rtupdate and weatherstation URL's go the same place.

He said it shouldn't matter which one.  So we can converge on a single URL.

This is down to same old same old "WU is a steaming pile" being the reason for the 404's.

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

[Thomas Keffer]

I agree with your conclusions. I learned a long time ago not to waste too much energy chasing WU flaws. There are many, and they come and go. Good way to frustrate yourself...😞

[Leon Shaner]

FWIW, I haven't seen any 404's since around these times from last night:

Feb  4 20:43:52 Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Feb  4 20:43:52 Wunderground-RF: Failed to publish record 2020-02-04 20:43:53 EST (1580867033): Failed upload after 1 tries

Feb  4 20:49:10 Wunderground-RF: Failed upload attempt 1: <urlopen error _ssl.c:1039: The handshake operation timed out>

Feb  4 20:49:10 Wunderground-RF: Failed to publish record 2020-02-04 20:48:49 EST (1580867329): Failed upload after 1 tries

Feb  4 20:49:45 Wunderground-RF: Failed upload attempt 1: <urlopen error timed out>

Feb  4 20:49:45 Wunderground-RF: Failed to publish record 2020-02-04 20:49:15 EST (1580867355): Failed upload after 1 tries

Feb  4 20:50:19 Wunderground-RF: Failed upload attempt 1: <urlopen error timed out>

Feb  4 20:50:19 Wunderground-RF: Failed to publish record 2020-02-04 20:49:49 EST (1580867389): Failed upload after 1 tries

Feb  4 21:04:41 Wunderground-RF: Failed upload attempt 1: <urlopen error [Errno 104] Connection reset by peer>

Feb  4 21:04:41 Wunderground-RF: Failed to publish record 2020-02-04 21:04:41 EST (1580868281): Failed upload after 1 tries

Feb  4 21:04:44 Wunderground-RF: Failed upload attempt 1: HTTP Error 404: Not Found

Regards,

\Leon

--

Leon Shaner :: Dearborn, Michigan (iPad)

On Feb 4, 2020, at 9:15 PM, Thomas Keffer <tke...@gmail.com> wrote:



[Dave McCreath]

Leon

The text below was posted by Tim Roche (WU IT) on the WeatherUndergound FaceBook page 17 hours ago:

Quick update: We have some answers for why many of the stations are not sending data. It appears that many of the embedded device controllers don't behave very well with host names, and are sending data to us in a way that is not expected.

We love that enthusiasts building cool stuff send data to Weather Underground, so while we asses how to fix these issues, we have temporarily rolled back to our legacy system.

We will be testing over the next week to see how to fix these problems, and will ask for help from some of you.

I would like to personally apologize for this issue, we felt our testing was robust, but obviously there were gaps. We're still working on a plan and I will try to keep you updated here.

Tim is a regular poster on the site and it's a good source of what's going on.

I think the time coincides with everything correcting itself on your system

Regards

Dave

[Brice Ruth]

Alrighty. Thanks for that!

Brice Ruth, FCD
Software Engineer, Madison WI

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/77bd16db-0eed-4598-94ea-f7ca3a0dc945%40googlegroups.com.

[J B]

Thanks for the update.

I'm not really sure what he means about embedded device controllers...most of us are running RaspberryPis that are sending data using Python. I'm pretty sure it can handle host names fine.

[Thomas Keffer]

Yeah, I thought that statement was a bit of a cop-out. 

They messed up, pure and simple.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/b2f3448a-dc34-4fa6-a938-2264197c25af%40googlegroups.com.

[Denny Page]

There are a few people in that thread that are using Arduino, and complaining about 404 errors. I believe that is what they were referring to by “embedded device controllers"

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/mYhw6CSKHHg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEAN71P5ypM9OT0hvtjUxact9UL9m%2BQG_H7MF5xw%2BPwMvw%40mail.gmail.com.

[Brice Ruth]

Yeah, agreed. That was my take as well. I think there's ... other issues, too.

Brice Ruth, FCD
Software Engineer, Madison WI

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/17C85818-7A6B-40F6-B28E-E72E034A5B2B%40me.com.