Weather Underground can't handle zero records from weewx

810 views
Skip to first unread message

Luc

unread,
Nov 26, 2013, 5:37:05 AM11/26/13
to weewx...@googlegroups.com
Hi Matthew or Tom,

I got the following message from Weather Underground:

The Weather Underground PWS network has detected that your personal weather station (ID #IPARAMAR5) has failed on our QC system.
The thermometer sensor on your weather station has reported values which are likely impossible. We only accept temperature values between -60 F (-51 C) and 130 F (54 C). Your station reported a temperature value outside of this range.

The minimum value of dewpoint show -73.3 Celsius (-100.0 Fahrenheit). 

 Huidig:Hoog:Laag:Gemiddelde:
Temperatuur:25.3 °C25.7 °C24.2 °C20.2 °C
Dauwpunt:23.7 °C23.7 °C-73.3 °C12.6 °C
Vochtigheid:91%91%87%80%
Windsnelheid:0.0km/u /0.0km/u--
Windstoot:1.6km/u /1.6km/u--
Wind:ONO--zuid
Druk:1009.0hPa1010.4hPa1008.4hPa-
Neerslag:0.0mm
 

This value is not shown in the records sent, It looks like the 'zero' records are the cause of this behaviour.

I'm wondering what data is in weewx's 'zero' record sent to Wunderground. I have set debug to 1 and the logging shows no details of the Wunderground data.

Cheers, Luc

Tabular Weather History for november 26, 2013
TijdTemp.DauwpuntDrukWindWindsnelheidWindstootVochtigheidHoeveelheidneerslag (per uur)
00:0025.7 °C23.3°C1010.4hPaCalm 0.0km/u87%0.0mm / 0.0mm total
00:0525.7 °C23.3°C1010.0hPaCalm 0.0km/u87%0.0mm / 0.0mm total
00:1525.7 °C23.3°C1010.0hPaCalm 0.0km/u87%0.0mm / 0.0mm total
00:2025.6 °C23.4°C1010.0hPaCalm 0.0km/u88%0.0mm / 0.0mm total
00:2525.6 °C23.4°C1010.0hPaCalm 0.0km/u88%0.0mm / 0.0mm total
00:3525.5 °C23.3°C1010.0hPaCalm 0.0km/u88%0.0mm / 0.0mm total
00:45---Calm --%0.0mm / 0.0mm total
00:50---Calm --%0.0mm / 0.0mm total
00:5525.3 °C23.2°C1010.0hPaCalm 0.0km/u88%0.0mm / 0.0mm total
01:0025.3 °C23.2°C1010.0hPaCalm 0.0km/u88%0.0mm / 0.0mm total
01:0525.3 °C23.2°C1010.0hPaCalm 0.0km/u88%0.0mm / 0.0mm total
01:1025.3 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
01:1525.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
01:2525.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
01:3025.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
01:4025.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
01:45---Calm --%0.0mm / 0.0mm total
01:5025.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
01:5525.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:0025.2 °C23.3°C1009.7hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:0525.2 °C23.3°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:1025.2 °C23.2°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:1525.2 °C23.2°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:2025.1 °C23.2°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:2525.1 °C23.2°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:3025.1 °C23.2°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:3525.1 °C23.1°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:40---Calm --%0.0mm / 0.0mm total
02:4524.9 °C22.9°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:5024.9 °C22.9°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
02:5524.9 °C22.9°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
03:0024.9 °C22.9°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
03:0524.8 °C22.9°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
03:1024.8 °C22.9°C1009.4hPaCalm 0.0km/u89%0.0mm / 0.0mm total
03:1524.8 °C23.1°C1009.0hPaCalm 0.0km/u90%0.0mm / 0.0mm total
03:20---Calm --%0.0mm / 0.0mm total
03:25---Calm --%0.0mm / 0.0mm total
03:3024.7 °C23.0°C1009.0hPaCalm 0.0km/u90%0.0mm / 0.0mm total
03:3524.7 °C22.9°C1009.0hPaCalm 0.0km/u90%0.0mm / 0.0mm total
03:4524.8 °C23.0°C1009.0hPaCalm 0.0km/u90%0.0mm / 0.0mm total
03:5024.8 °C23.1°C1009.0hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:0024.7 °C23.0°C1009.0hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:0524.7 °C22.9°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:1024.7 °C22.9°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:1524.7 °C22.9°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:2024.7 °C23.0°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:3024.7 °C22.9°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:4024.6 °C22.8°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:5024.6 °C22.9°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
04:5524.6 °C22.9°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:0024.6 °C22.9°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:0524.6 °C22.8°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:1024.5 °C22.8°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:2024.4 °C22.7°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:3024.4 °C22.7°C1008.4hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:4024.4 °C22.7°C1008.7hPaCalm 0.0km/u90%0.0mm / 0.0mm total
05:4524.4 °C22.8°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
05:5024.4 °C22.8°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
05:5524.3 °C22.8°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:0524.3 °C22.7°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:1024.3 °C22.7°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:1524.3 °C22.7°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:2024.3 °C22.8°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:2524.3 °C22.8°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:30---Calm --%0.0mm / 0.0mm total
06:35---Calm --%0.0mm / 0.0mm total
06:4024.2 °C22.6°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
06:5024.2 °C22.7°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
07:0024.4 °C22.8°C1008.4hPaCalm 0.0km/u91%0.0mm / 0.0mm total
07:0524.4 °C22.9°C1008.7hPaCalm 0.0km/u91%0.0mm / 0.0mm total
07:1024.6 °C23.1°C1008.7hPaCalm 0.0km/u91%0.0mm / 0.0mm total
07:1524.7 °C23.2°C1008.7hPaCalm 0.0km/u91%0.0mm / 0.0mm total


mwall

unread,
Nov 26, 2013, 7:54:17 AM11/26/13
to weewx...@googlegroups.com
On Tuesday, November 26, 2013 5:37:05 AM UTC-5, Luc wrote:
This value is not shown in the records sent, It looks like the 'zero' records are the cause of this behaviour.

I'm wondering what data is in weewx's 'zero' record sent to Wunderground. I have set debug to 1 and the logging shows no details of the Wunderground data.

luc,

assuming you are running weewx 2.5, in bin/weewx/restful.py function Ambient.getURL (around line 226), add one line to change this:

        # This will be the complete URL for the HTTP GET:                      
        _url=self.http_prefix + _urlquery
        return _url

to this:

        # This will be the complete URL for the HTTP GET:                      
        _url=self.http_prefix + _urlquery
        syslog.syslog(syslog.LOG_INFO, "URL: %s" % _url)
        return _url

then you will be able see what is posted to wunderground.

m

Luc

unread,
Nov 26, 2013, 8:16:22 AM11/26/13
to weewx...@googlegroups.com

Burns

unread,
Nov 26, 2013, 4:06:44 PM11/26/13
to weewx...@googlegroups.com
I'm interested in this too.  I did not get an automatic email from Wunderground, but I did see on the web site that they think I gave them bad data.  OTOH, they seemed to imply I had given it before I switched to weewx, so it may actually be messed up even with good data.

Luc

unread,
Nov 26, 2013, 4:17:20 PM11/26/13
to weewx...@googlegroups.com
Hi Burns,

Be patient. I have seen periods of 10 days not having an 'zero' records. 

Cheers, Luc

Guilhem M

unread,
Dec 3, 2013, 2:58:51 AM12/3/13
to weewx...@googlegroups.com
I have the same problem with the 0 value. But I am not sure, weewx transmit "0" but "" (blank character)... isn't it ?

David Rose

unread,
Dec 3, 2013, 1:46:07 PM12/3/13
to weewx...@googlegroups.com
Recently I experienced the issue with randomly failing the Weather
Underground QC system during the night. I added the the logging code
from mwall and found that when the upload would fail the QC system the
following fields were not present in the WU post:

dewptf=
humidity=
tempf=

My issue was due to a low battery in the field device. New batteries
and all is good again.

David.

On Tue, Dec 3, 2013 at 1:58 AM, Guilhem M <guilh...@gmail.com> wrote:
> I have the same problem with the 0 value. But I am not sure, weewx transmit
> "0" but "" (blank character)... isn't it ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Weewx user's group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to weewx-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

Thomas Keffer

unread,
Dec 3, 2013, 1:49:09 PM12/3/13
to weewx-user
There's something I don't understand about this. If a datum is missing (value None), then it should be skipped completely in the post. That is, the fields

dewptf
humidity
tempf

should not be in the URL GET.

What version are you using? Is this V2.5, or the alpha version of 2.6?

-tk

David Rose

unread,
Dec 3, 2013, 2:10:35 PM12/3/13
to weewx...@googlegroups.com
Tom,

When the WU QC would fail the 3 fields were not included in the post to WU. 

Version 2.5
WMR200 driver: $Revision: 1635 $    $Date: 2013-11-06 16:41:53 -0800 (Wed, 06 Nov 2013) $

David. 

Thomas Keffer

unread,
Dec 3, 2013, 4:29:29 PM12/3/13
to weewx-user
Now I'm confuzed. If the three fields were not included in the post, then why would WU complain?

-tk

Luc

unread,
Dec 3, 2013, 5:28:36 PM12/3/13
to weewx...@googlegroups.com
Hi folks,

These are the messages to wunderground in weewx version 2.5.0:


With a null data record wunderground gets confused.
The null values are shown normal as in the example below of 06:50::
Tijd    Temp.     Dauwpunt   Druk       Wind   Windsnelheid   Windstoot   Vochtigheid   Hoeveelheidneerslag (per uur) 
06:40   24.6 °C   23.2°C     1008.7hPa  Calm                  0.0km/u     92%           0.0mm / 0.0mm total 
06:50   -         -          -          Calm   -              -           %             0.0mm / 0.0mm total

But the daily minimum of dewpoint (Dauwpunt) is set to -73,3 degrees Celsius (-100.0 degrees Fahrenheit) as seen below:
                   Huidig:      Hoog:      Laag:      Gemiddelde:
Temperatuur:    26.1 °C      26.1 °C    24.6 °C    23.4 °C
Dauwpunt:       24.1 °C      24.2 °C   -73.3 °C    19.9 °C
Vochtigheid:    89%          93%        89%        88%
Windsnelheid:   3.2km/u /    4If.8km/u    -          -
Windstoot:      6.4km/u /    8.0km/u    -          -
Wind:           ZZO          -          -          OZO
Druk:           1010.4hPa    1010.4hPa  1007.7hPa  -
Neerslag:       0.0mm

I have written a message to he...@wunderground.com on 29th of November, but so far I didn't receive a reply.

Cheers, Luc




Thomas Keffer

unread,
Dec 3, 2013, 5:54:31 PM12/3/13
to weewx-user
OK, now I get it. Clearly this is a WU problem. 

-tk


Luc

unread,
Dec 3, 2013, 6:08:16 PM12/3/13
to weewx...@googlegroups.com
Tom,

Wunderground interprets a null data record as follows:

<observation_time>Last Updated on December 3, 2:10 AM SRT</observation_time>

<observation_time_rfc822>Tue, 03 Dec 2013 05:10:00 GMT</observation_time_rfc822>

<weather/>

<temperature_string>-999.0 F ( C)</temperature_string>

<temp_f>-999.0</temp_f>

<temp_c>-572.8</temp_c>

<relative_humidity>-999</relative_humidity>

<wind_string>Calm</wind_string>

<wind_dir>North</wind_dir>

<wind_degrees>-999</wind_degrees>

<wind_mph>-999.0</wind_mph>

<wind_gust_mph>-999.0</wind_gust_mph>

<pressure_string>-100.00" (-3386.0 mb)</pressure_string>

<pressure_mb>-3386.0</pressure_mb>

<pressure_in>-100.00</pressure_in>

<dewpoint_string>-99.9 F (-73.3 C)</dewpoint_string>

<dewpoint_f>-99.9</dewpoint_f>

<dewpoint_c>-73.3</dewpoint_c>

<heat_index_string/>

<heat_index_f/>

<heat_index_c/>

<windchill_string/>

<windchill_f/>

<windchill_c/>

<solar_radiation/>

<UV/>

<precip_1hr_string>0.00 in (0.0 mm)</precip_1hr_string>

<precip_1hr_in>0.00</precip_1hr_in>

<precip_1hr_metric>0.0</precip_1hr_metric>

<precip_today_string>0.00 in (0.0 mm)</precip_today_string>

<precip_today_in>0.00</precip_today_in>

<precip_today_metric>0.0 mm</precip_today_metric>

<history_url>

http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=IPARAMAR5

</history_url>

<ob_url>

http://www.wunderground.com/cgi-bin/findweather/getForecast?query=5.82173014,-55.21914673

</ob_url>

</current_observation>

<current_observation>

<credit>Weather Underground Personal Weather Station</credit>

<credit_URL>http://wunderground.com/weatherstation/</credit_URL>

<image>

<url>

http://icons.wunderground.com/graphics/bh-wui_logo.gif

</url>

<title>Weather Underground</title>

<link>http://wunderground.com/weatherstation/</link>

</image>

<location>

<full>Tammenga, Paramaribo, PARAMARIBO, PARAMARIBO</full>

<neighborhood>Tammenga</neighborhood>

<city>Paramaribo</city>

<state>PARAMARIBO</state>

<zip>0000A</zip>

<latitude>5.82173014</latitude>

<longitude>-55.21914673</longitude>

<elevation>7 ft</elevation>

</location>

<station_id>IPARAMAR5</station_id>

<station_type/> 

Perhaps you should contact them and agree about the format of null data.

Cheers, Luc

David Rose

unread,
Dec 3, 2013, 6:17:21 PM12/3/13
to weewx...@googlegroups.com
I agree that this is an issue with WU. I believe WU QC system complained due to the fact that 'the temperature weewx didn't send' didn't agree with the surrounding stations temperature. 


David.

Thomas Keffer

unread,
Dec 3, 2013, 6:21:23 PM12/3/13
to weewx-user
Well, they say that the only required fields are the action (updateraw), station id, password, and dateutc. 

http://wiki.wunderground.com/index.php/PWS_-_Upload_Protocol

If they are not honoring that, then I'm not quite sure what to do. I guess send those funky values when a datum is null?

-tk



--

Thomas Keffer

unread,
Dec 3, 2013, 6:24:23 PM12/3/13
to weewx-user
I guess they don't believe in the old adage, "Better to remain silent and be thought a fool than to speak and to remove all doubt."

-tk

Luc

unread,
Dec 3, 2013, 6:34:13 PM12/3/13
to weewx...@googlegroups.com
OK Tom,

Clear. Let's wait for a reply of wunderground..

Cheers, Luc

Guilhem M

unread,
Dec 4, 2013, 2:07:21 AM12/4/13
to weewx...@googlegroups.com
I agree with you, no doubt, problem is coming from WU.
A solution (until they correct it) is that weewx takes a record from archive with no NULL value... If archive is saved each one minute, there is not a lot of impact because, in my case, not a lot of raw have a NULL value. What do u yhink about ?

Henk van Rooyen

unread,
Sep 5, 2014, 12:00:54 PM9/5/14
to weewx...@googlegroups.com
Apologies for resurrecting this old thread, but can someone please advise:
Brand new install of weewx on raspberry pi, reporting wmr200 data to wunderground. Exact same problem as above - the occasional missed value from the weather station results in a NULL value being sent to wunderground, where they interpret it as some weird value like -572.8C thus causing my station to fall foul of their quality control. Is there any setting I am missing in the config file that can prevent the sending of NULL values as described below, or will I need knowledge of Python to fix this?

Thanks
H

Thomas Keffer

unread,
Sep 5, 2014, 12:23:58 PM9/5/14
to weewx-user

Hello Henk

Weewx does not send null values. It sends no value at all. Unfortunately, in the absence if a value, WU often substitutes some random value. The problem seems to be entirely on their side. At least, we haven't found any solution on our side.

-tk

Fat-fingered from my Android

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Henk van Rooyen

unread,
Sep 5, 2014, 12:26:39 PM9/5/14
to weewx...@googlegroups.com

Oh well...
Thanks
H

Thomas Keffer

unread,
Sep 5, 2014, 12:29:18 PM9/5/14
to weewx-user

Luc, did you ever hear anything from the WU?

-tk

Fat-fingered from my Android

Luc Heijst

unread,
Sep 5, 2014, 1:02:56 PM9/5/14
to weewx...@googlegroups.com
Tom, I wrote them twice. The second time also complaining they never gave an answer.
Still waiting for an answer :-(

Thomas Keffer

unread,
Sep 5, 2014, 1:51:57 PM9/5/14
to weewx-user
Did you tell them you were representing a vast network of weather enthusiasts who can make or break them? :-)

-tk


--

Luc Heijst

unread,
Sep 5, 2014, 2:14:53 PM9/5/14
to weewx...@googlegroups.com
On Friday, 5 September 2014 14:51:57 UTC-3, Tom Keffer wrote:
Did you tell them you were representing a vast network of weather enthusiasts who can make or break them? :-)

-tk

Tom, I'm afraid I was less subtle in my message. I wonder if they read messages of customers anyhow or have a 'complain' filter build in which directs such messages directly to the waste paper basket.

Despite all that said I like many features of Weather Display.

BTW Yesterday I got a message from Kari Strenfel about a good looking iPad app "WunderStation" which presents weather data of all WU stations on your iPad in a very promising way. Unfortunately the app crashed on my iPad 2 after 2 seconds. 
I was thinking about sending them a message about this but decided not as you may understand...

Luc

Thomas Keffer

unread,
Sep 5, 2014, 2:20:18 PM9/5/14
to weewx-user
Back up a second. What's the connection to WeatherDisplay?

-tk


--

L.J.M. Heijst

unread,
Sep 5, 2014, 2:26:03 PM9/5/14
to Thomas Keffer
Sorry Tom,
Typo!   I meant to say Weather Underground.
Luc
.
-- 
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nfyCbNWL36o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Luc Heijst

unread,
Sep 5, 2014, 3:33:18 PM9/5/14
to weewx...@googlegroups.com
On Friday, 5 September 2014 13:29:18 UTC-3, Tom Keffer wrote:

Luc, did you ever hear anything from the WU?

-tk

Tom,
Just wrote a mail to Alan Sterremberg -CEO of Weather Underground- about this subject. Maybe it helps.
Luc 

Luc Heijst

unread,
Sep 5, 2014, 7:34:49 PM9/5/14
to weewx...@googlegroups.com
Bad luck, Tom,

William of the suport desk of Weather Underground wrote me:
"The null entry for data needs work on our end, as we have a lot of legacy systems and each of them handles it differently. We hope to unify this soon, and make it easier for developers and partners to use our data."
and:
"Alan Steremberg is no longer with the company."
and:
"I completely understand your interest in speaking to the highest level of the company, but if I were to pass every request to him, he wouldn't get much work done. I also wouldn't be doing my job. :-)"

Luc

 

Thomas Keffer

unread,
Sep 5, 2014, 8:15:26 PM9/5/14
to weewx-user

Thanks for pursuing this, Luc.

Do you think he understands that we are not posting "nulls," but rather, the data are missing completely from the URL string?

-tk

Fat-fingered from my Android

On Friday, 5 September 2014 16:33:18 UTC-3, Luc Heijst wrote:

--

Luc Heijst

unread,
Sep 5, 2014, 8:30:53 PM9/5/14
to weewx...@googlegroups.com
On Friday, 5 September 2014 21:15:26 UTC-3, Tom Keffer wrote:

Thanks for pursuing this, Luc.

Do you think he understands that we are not posting "nulls," but rather, the data are missing completely from the URL string?

-tk

Tom, 

Not sure what WU thinks; this is what I wrote "Right now we omit non-existing data in our messages to Weather Underground," 
Don't tell anyone further, but I plan to do some tests with a "weewx" data message to WU with "null" data in it to see how the WU software reacts. 
I guess it can't be worse than the values of -573 degrees Celsius, -999 percent humidity and -33857 hPa barometric pressure I got now.

Luc

Thomas Keffer

unread,
Sep 5, 2014, 8:32:57 PM9/5/14
to weewx-user
Your message to them sounds pretty clear. 

I find it hard to believe that other weather software are not running into this problem. What are they doing differently? Is there some magic null value (-999?) that can be included in the URL to signal missing data?

I'll be interested in the results of your tests.

-tk


--

Luc Heijst

unread,
Sep 5, 2014, 9:13:20 PM9/5/14
to weewx...@googlegroups.com
On Friday, 5 September 2014 21:32:57 UTC-3, Tom Keffer wrote:
Your message to them sounds pretty clear. 

I find it hard to believe that other weather software are not running into this problem. What are they doing differently? Is there some magic null value (-999?) that can be included in the URL to signal missing data?

I'll be interested in the results of your tests.

-tk

Weather Underground gives the user the possibillity to delete "wrong" data in their database.
When all data within a day is deleted you get the silly numbers I mentioned in my previous post.
So I think apart from WU themselves there is nothing we can do about it, but just out of curiosity I will do the test.

Luc  

Andrew Milner

unread,
Sep 5, 2014, 9:46:48 PM9/5/14
to weewx...@googlegroups.com
I discovered I had got some negative winds during 2013 recently!!!

I was actually trying to see my barometer ranges for a normal year to try and get Zambretti forecasting set up more suited to my location ... but then I noticed that when viewing the WU graphs for the year the wind wa actually going negative on their graphs!!!  Digging deeper this appeared to be coming from days when there were holes in the wind data (eg when the console loses the remote sensor readings) - and these were sufficient to create the ridiculous phenomenon of negative wind!!!!

I was about to write and complain unitl I spotted this thread.  But yes, it is very deeprooted within WU setup.  I really cannot understand why they do not make at least a simple change at their end to handle their own -- creations correctly.

Luc Heijst

unread,
Sep 8, 2014, 10:42:36 AM9/8/14
to weewx...@googlegroups.com
On Friday, 5 September 2014 21:32:57 UTC-3, Tom Keffer wrote:
I'll be interested in the results of your tests.
-tk

These are the results of the tests with sending data to Weather Underground with one or more parameters missing or having testdata value.
Note the weewx version subnumber is changed deliberately  to check if the sent data is received correctly in WU.

09:19 data of 09:07:00 with windgustmph=; 
2.6.6&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A07%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=&rainin=0.00
result: 2014-09-08 09:07:00,28.1,23.9,1015.1,oost,96,11.3,0.0,78,0.0,,,0.0,weewx-2.6.6,2014-09-08 12:07:00, 

09:24 data of 09:08:00 with windgustmph=none; 
2.6.7&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A08%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=none&rainin=0.00
result: 2014-09-08 09:08:00,28.1,23.9,1015.1,oost,96,11.3,0.0,78,0.0,,,0.0,weewx-2.6.7,2014-09-08 12:08:00, 

09:26 data of 09:09:00 with windgustmph=nill; 

2.6.8&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A09%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=nill&rainin=0.00
result: 2014-09-08 09:09:00,28.1,23.9,1015.1,oost,96,11.3,0.0,78,0.0,,,0.0,weewx-2.6.8,2014-09-08 12:09:00,

09:27 data of 09:11:00 with windgustmph=-999; 
2.6.9&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A11%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=-999&rainin=0.00
result: 2014-09-08 09:11:00,28.1,23.9,1015.1,oost,96,11.3,-1607.4,78,0.0,,,0.0,weewx-2.6.9,2014-09-08 12:11:00,

09:30 data of 09:12:00 with windspeedmph=-1; 
2.6.10&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A12%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=-1&windgustmph=014&rainin=0.00
result: 2014-09-08 09:12:00,28.1,23.9,1015.1,oost,96,-1.6,22.5,78,0.0,,,0.0,weewx-2.6.10,2014-09-08 12:12:00, 

09:33 data of 09:13:00 with windspeedmph=-999; 
2.6.11&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A13%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=-999&windgustmph=014&rainin=0.00
result: 2014-09-08 09:13:00,28.1,23.9,1015.1,oost,96,-1607.4,22.5,78,0.0,,,0.0,weewx-2.6.11,2014-09-08 12:13:00, 

09:35 data of 09:14:00 with windspeedmph n/a; 
2.6.12&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A14%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windgustmph=014&rainin=0.00
result: 2014-09-08 09:14:00,28.1,23.9,1015.1,oost,96,-1607.4,22.5,78,0.0,,,0.0,weewx-2.6.12,2014-09-08 12:14:00,

09:37 data of 09:16:00 with all data n/a; 
08+12%3A16%3A00
result: 2014-09-08 09:16:00,-572.8,-73.3,-3386.0,noord,-999,-1607.4,-1607.4,-999,-2539.7,,,-2539.7,weewx-2.6.13,2014-09-08 12:16:00, 

09:40 data of 09:17:00 with all data n/a except rainin; 
08+12%3A17%3A00&rainin=0.00
result: 2014-09-08 09:17:00,-572.8,-73.3,-3386.0,noord,-999,-1607.4,-1607.4,-999,0.0,,,-2539.7,weewx-2.6.14,2014-09-08 12:17:00,

09:46 data of 09:18:00 with all values except baromin and dewptf
09-08+12%3A18%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:18:00,28.1,-73.3,-3386.0,oost,96,11.3,22.5,78,0.0,,,0.0,weewx-2.6.13,2014-09-08 12:18:00, 

09:50 data of 09:19:00 with all values except baromin=-999 and dewptf=-999
&dewptf=-999&humidity=078&dateutc=2014-09-08+12%3A19%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:19:00,28.1,-73.3,-3386.0,oost,96,11.3,22.5,78,0.0,,,0.0,weewx-2.6.14,2014-09-08 12:19:00,

09:54 data of 09:21:00 with all values except dewptf=""
2.6.15&baromin=29.979&dewptf=&humidity=078&dateutc=2014-09-08+12%3A21%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:21:00,28.1,-17.8,1015.1,oost,96,11.3,22.5,78,0.0,,,0.0,weewx-2.6.15,2014-09-08 12:21:00, 

09:55 data of 09:22:00 with all values except dewptf=n/a
2.6.16&baromin=29.979&humidity=078&dateutc=2014-09-08+12%3A22%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:22:00,28.1,-73.3,1015.1,oost,96,11.3,22.5,78,0.0,,,0.0,weewx-2.6.16,2014-09-08 12:22:00, 

09:56 data of 09:23:00 with all values except humidity=n/a
2.6.17&baromin=29.979&dewptf=75.0&dateutc=2014-09-08+12%3A23%3A00&dailyrainin=0.00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:23:00,28.1,23.9,1015.1,oost,96,11.3,22.5,-999,0.0,,,0.0,weewx-2.6.17,2014-09-08 12:23:00,

10:08 data of 09:24:00 with all values except dailyrain=n/a
2.6.18&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A24%3A00&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:24:00,28.1,23.9,1015.1,oost,96,11.3,22.5,78,0.0,,,-2539.7,weewx-2.6.18,2014-09-08 12:24:00, 

10:10 data of 09:26:00 with all values except dailyrain=-999
2.6.19&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A26%3A00&dailyrainin=-999&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:26:00,28.1,23.9,1015.1,oost,96,11.3,22.5,78,0.0,,,-2539.7,weewx-2.6.19,2014-09-08 12:26:00,  

10:13 data of 09:27:00 with all values except dailyrain=""
2.6.20&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A27%3A00&dailyrainin=&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014&rainin=0.00
result: 2014-09-08 09:27:00,28.1,23.9,1015.1,oost,96,11.3,22.5,78,0.0,,,0.0,weewx-2.6.20,2014-09-08 12:27:00, 

10:17 data of 09:28:00 with all values except rainin=n/a
2.6.21&baromin=29.979&dewptf=75.0&humidity=078&dateutc=2014-09-08+12%3A28%3A00&dailyrainin=&winddir=096&tempf=82.6&windspeedmph=007&windgustmph=014
result: 2014-09-08 09:28:00,28.1,23.9,1015.1,oost,96,11.3,22.5,78,-2539.7,,,0.0,weewx-2.6.21,2014-09-08 12:28:00,

parameter sent parameter in WU value sent value in WU value sent value in WU
n/a Time
baromin PressurehPa "normal" "normal" left out -3386,0
dewptf DewpointC "normal" "normal" left out -73,3
humidity Humidity "normal" "normal" left out -999
dailyrainin dailyrainMM "normal" "normal" left out -2539,7
winddir WindDirection "normal" "normal" left out noord (=north)
n/a WindDirectionDegrees -999
tempf TemperatureC "normal" "normal" left out -572,8
windspeedmph WindSpeedKMH "normal" "normal" left out -1607,4
gustspeedmph WindSpeedGustKMH "normal" "normal" left out -1607,4
rainin HourlyPrecipMM "normal" "normal" left out -2539,7
n/a Conditions n/a n/a left out n/a
n/a Clouds n/a n/a left out n/a
softwaretype SoftwareType "normal" "normal" weewx-2.6.4 weewx-2.6.4
dateutc DateUTC "normal" "normal" "normal" "normal"

Conclusion:
Leaving data out in the string to Weather Underground results in default "bogus" values.
Sometimes this has no consequences, sometimes it does. 
a. When dewptf is left out, one get's messages mailed about dewpoint values of -73,3 degrees C which are outside ranges.
b. Sometimes when values within a day are missing the daymaximum, dayminimum and dayaverage values are calculated inclusive the "bogus" values. See attached file.

Recommendation:
When none of the values are present in an archive message, skip sending the datastring to Weather Underground.

Luc
Bogus values Weather Underground.jpg

Andrew Milner

unread,
Sep 8, 2014, 11:37:57 AM9/8/14
to weewx...@googlegroups.com
The problem with WU is that the bogus values appear to get included in all reports and graphs - hence my negative winds and I also found it absolutely impossible to get a good picture of anything - a single omission on a day can create total havoc.

Perhaps weewx users should take a stand and almost insist that WU stop creating and even worse - using - bogus values, or perhaps we could find a means to publicise the errors which their data now contains - rendering it pretty useless!!  WU needs to be able to cope with missing readings and not just create a value.

I assume the problem occurs with all PWS systems and not just WU.  Out of curiosity - what do other reporting systems do??  Maybe the answer is just to stop submitting to WU but to somewhere else instead!!  Anyone got any ideas of a better place than WU which does not corrupt the data when some is omitted??


Thomas Keffer

unread,
Sep 8, 2014, 1:16:02 PM9/8/14
to weewx-user

Very interesting, and very definitive. Thank you, Luc, for your great research.

Pretty amazing that WU doesn't see this.

Your recommendation looks reasonable, perhaps with a configuration option to turn it on or off, should WU ever get it together and fix this.

Thanks again.

-tk

Fat-fingered from my Android

--

Luc Heijst

unread,
Sep 8, 2014, 4:33:26 PM9/8/14
to weewx...@googlegroups.com
On Monday, 8 September 2014 14:16:02 UTC-3, Tom Keffer wrote:

Pretty amazing that WU doesn't see this.

I think the development team has to do too many tasks with too little people.
Their priority seems to be the presentation layer.
May be one day they have time to debug the data layer, that is: If there are any users left that time...

Look what happened with WordPerfect: how fast a good product left the word processing stage...

Luc

Andrew Milner

unread,
Sep 9, 2014, 1:53:10 AM9/9/14
to weewx...@googlegroups.com
Well if they regard the data layer as less important than the presentation layer maybe it is time to find a better alternative to WU which handles missing data correctly rather than causing corruption and invalidity?  I am not sure if dropping an entire record is better or worse than substituting last known value when a reading is missing - at least one then gets the new fresh data posted whilst dropping the entire record is a bit like a sledge hammer to crack a nut.  Shifting to another aggregator may be the only real solution - and help to drive the dissatisfaction point home.

Anyone with an alternative global weather site that we can post to?  I know there are many in the US - hence the global requirement.  PWS is just WU under a different branding... any more?

Regards WP - yes, I can well remember the days when it was so much better and easier to use than Microsoft Word .......oh for bygone times .....

David Hathaway

unread,
Sep 9, 2014, 9:28:04 AM9/9/14
to weewx...@googlegroups.com
First of all, thank you Luc for the in depth testing.  Very appreciated!

Secondly, what do other software uploaders do?  Presumably, all the software reading weather stations have to deal with the same missing data as Weewx does... what do they send?  Is it handled differently by WU?

Lastly, my solution has been to use the last known value when one comes up missing (up to a defined maximum age).  The reason being that in my case, WU would not report results if data was missing.  I have noticed that on a three-minute interval, changes from one report to the next are not that huge, and I can live with the minor (if any) inaccuracy.  As Andrew noted, I have the view that " at least one then gets the new fresh data posted whilst dropping the entire record is a bit like a sledge hammer to crack a nut".

I might maybe switch to a different aggregator, but I already have my home system accessible via the web page, and the WU Android app is so very nice.  I like looking on my phone and seeing my home conditions in the notification area.

Andrew Milner

unread,
Sep 9, 2014, 10:09:23 AM9/9/14
to weewx...@googlegroups.com
David

How have you managed to substitute last known data for missing data within the WU uploaded records?  Are you using rapid fire or just the standard archival updating to WU?

I use standard weewx reporting on archive intervals - and would love to be able to replace None with last known to cover the occasions when my WH1080 loses contact with the outdoor sensors.

The more I think about it the less I find myself supporting Tom's 'drop the record' approach even though in my situation I would only be left with a pressure reading (as all other values come from the external sensors) - although the situation of a faulty device, stuck anemometer etc should not, in my opinion, result in no data at all being submitted.



--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nfyCbNWL36o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

Thomas Keffer

unread,
Sep 9, 2014, 10:13:29 AM9/9/14
to weewx-user

I'll look into using the last good value if a datum is missing. Unfortunately, coding it is not as simple as it sounds. For every None value, now you have to go to the database to see whether a good value has appeared recently. The alternative is a stateful design that retains the value, but, so far, I've avoided that.

In any case, it will be a while. Matthew and I are doing a boat delivery --- our first ever face-to-face meeting!

-tk

Fat-fingered from my Android

--

David Hathaway

unread,
Sep 9, 2014, 10:53:28 AM9/9/14
to weewx...@googlegroups.com
Tom,

Have fun in the face-to-face.  3-D meetings can be rewarding!

Andrew,

I am attaching my "backfill" version of the driver (which lives in the /usr/share/weewx/user directory).  Please understand that I don't know Python worth a lick, but this works well.

The basic approach is to look through the range of sensors.  If values are missing, then it backfills if from the database. 

There are a couple special cases.  For example, if windSpeed is 0, the windDir can be expected to be missing. WU handles that case fine.  Then there are the sensors that my system doesn't have (radiation and UV), which are always missing.  I delete some sensors during initialization, but the special processing remains in case I forget to miss them.

The heart of the service is the get_backfill function, which searches the database:

def get_backfill(self, key):
       try:
          # look back to fill records, but only 10 minutes max
          timeNow = time.time()
          oldest = timeNow - 60*self.maxAge
          the_sql = 'SELECT %s FROM archive WHERE %s IS NOT NULL AND dateTime >= %d AND dateTime <= %d ORDER BY dateTime desc LIMIT 1' % (key, key, oldest, timeNow)
          syslog.syslog(syslog.LOG_DEBUG, "backfill: getURL backfilling key %s sql='%s'" % (key, the_sql))
          return self.archive.getSql(the_sql)
       except Exception, e:
          # Error raised
          syslog.syslog(syslog.LOG_DEBUG, "backfill: get_backfill: Exception raised %s" % (e))
       return None

The comment says "10 minutes max", but I later made it user definable.  In weewx.conf:

[[Backfill]]
        # This section is for configuring posts to the Weather Underground using backfill
        # If you wish to do this, uncomment the following station and password
        # lines and fill them with your station and password:
        station = KTXSUGAR19
        password = wouldntyouliketoknow 

        # Set the following to True to have weewx use the WU "Rapidfire"
        # protocol
        rapidfire = False
        
        # Set the maximum age you will backfill from, in minutes
        maxage = 11 

        # list of sensors we will format
        # sensors = dateTime, barometer, outTemp, outHumidity, windSpeed, windDir, windGust, dewpoint, hourRain, dayRain, radiation, UV
        # sensors = dateTime, barometer, outTemp, outHumidity, windSpeed, windDir, windGust, dewpoint, hourRain, dayRain
        sensorskip = radiation, UV

Also, I modified the logwatch definition to look for and report "backfill" statistics.

--------------------- weewx Begin ------------------------ 
counts:
  archive: records added         195
  backfill: skipped windDir      173
  filegenerator: files generated 2730
  genimages: images generated    2911
  restx: records published       195
  wxengine: garbage collected    3

errors:

---------------------- weewx End -------------------------


Anyways, this was my approach, and maybe it will give some ideas to those who know what they are doing.

Dave

backfill.py

Andrew Milner

unread,
Sep 9, 2014, 11:16:17 AM9/9/14
to weewx...@googlegroups.com
David - aha - you are attaking the problem in the driver - not really what I had foreseen.  I had envisaged correcting the problem in the restful WU upload service since it is a WU problem.   It is correct that the weewx database contains none when no reading is available.

Tom - as a thought - one could retain the last valid datafields on a continual basis and either use the retained data or update the retained valid data when building the WU update record.  Using this approach would not require database searches as the 'last valid' data would just be retained within the restful service.  At startup there may be a need to search for the valid data in order to initialise the retained data - or it could be held in the archive database with a unique timestamp eg 0000000001 to remove the need for searching.

David Hathaway

unread,
Sep 9, 2014, 11:35:05 AM9/9/14
to weewx...@googlegroups.com
Andrew,

I really should have not tried responding while working.  I already lack concentration....

The backfill service is a derivative of the StdResftul upload service.

[StdRESTful]
       [[Backfill]]

and also,

class StdBackfill(StdRESTbase):
    """Specialized version of the Ambient protocol for the Weather Underground, backfilled.
    """

class BackfillThread(RESTThread):
    """Concrete class for threads posting from the archive queue,
       using the Ambient PWS protocol."""
    
I did mention that I am not a Python programmer??

All the processing happens in the "format_url" function, which creates the HTML string sent to Wunderground.

Apologies for being confusing,

Dave

Thomas Keffer

unread,
Sep 9, 2014, 12:35:00 PM9/9/14
to weewx-user


On Sep 9, 2014 8:16 AM, "Andrew Milner" <andrew.s...@gmail.com> wrote:
>
> Tom - as a thought - one could retain the last valid datafields on a continual basis and either use the retained data or update the retained valid data when building the WU update record.  Using this approach would not require database searches as the 'last valid' data would just be retained within the restful service.  At startup there may be a need to search for the valid data in order to initialise the retained data - or it could be held in the archive database with a unique timestamp eg 0000000001 to remove the need for searching.

Yes, this is the "stateful" approach. Weewx generally avoids doing this because it requires management and knowledge of that state. I much prefer stateless solutions. Simpler, and more robust to external changes.

-tk

Andrew Milner

unread,
Sep 9, 2014, 1:16:09 PM9/9/14
to weewx...@googlegroups.com
Whilst I may be inclined to go along with your purist philosophy I feel that WU's inability to cope with the real world possibly requires a bending of the rules.  I am not convinced this becomes stateful though in that a possible translation of my approach is " If field is missing get it from archive record whose timestamp is 000000001.  If field is present update the field in archive record whose timestamp is 00000000001".  The remainder of restx/restful should remain as is.

-tk

--

Henk van Rooyen

unread,
Sep 11, 2014, 11:21:19 AM9/11/14
to weewx...@googlegroups.com

Hi:
I made contact with Wunderground as well. Please see correspondence below and attached. Not knowing the inner workings of weewx,  I don't know if this adds anything or if I just misunderstood the underlying problem...

Henk
---------- Forwarded message ----------
From: "Kari" <he...@wunderground.com>
Date: 11 Sep 2014 4:59 PM
Subject: Re: "Impossible values" from PWS
To: "HvRooyen" <he...@vanrooyen.co.za>
Cc:

> Hi there, 
>
> The "-572" you are seeing in C, converts to -999F.  This means the data is coming into us as "-999".  The problem is not on our end.  The problem is the weather station software is sending the data as "-999."  I have attached a screen shot in hopes to prove to you how the data is coming into Weather Underground.  Again, we display the data exactly as it comes in.  Can you try to use a different software?  What weather station do you have? I'm sure there is another software available to send data from your station to Weather Underground. 
>
> More information on available software here:
>http://www.wunderground.com/weatherstation/setup.asp
>
>
> Thanks and let me know if you have any other questions,
> Kari
>
> Helpful? Click to give Kari thanks!
>
> On Sep 11, 2014 at 1:25 PM HvRooyen <he...@vanrooyen.co.za> wrote:
>>
>> Kari:
>> I appreciate that gaps may exist, but to see - 572 degr C (see attached)
>> is obviously an error. The link I sent in my original query (a discussion
>> forum for weewx... ) describes the problem
>> in some detail, demonstrates it very elegantly, and also includes info
>> from WU seemingly confirming that the problem is on your side. Please also
>> note that I have already replaced my wmr200 with a brand new one, yet the
>> problem persists.
>> Your advice will be appreciated.
>> Regards
>> Henk van Rooyen
>>
>> On Sep 9, 2014 at 9:16 PM Kari <he...@wunderground.com> wrote:
>>>
>>> Hi there, 
>>>
>>> Unfortunately, we display the data exactly as it comes into us.  If there is a gap in the data, you should see a gap in the data. 
>>>
>>> Thanks and let me know if you have any other questions,
>>> Kari
>>>
>>> On Sep 9, 2014 at 11:33 AM HvRooyen <he...@vanrooyen.co.za> wrote:
>>>>
>>>> New PWU (IFREESTA40). Wireless connection sometimes drops, leading to gaps in data. This is apparently not unknown with the OS range. However, the "impossible values" reported by WU are apparently entirely created on your side. Please seehttps://groups.google.com/forum/#!msg/weewx-user/nfyCbNWL36o/DXMLnMW6LGoJfor a description, analysis and proposal regarding the problem.

Screen_Shot_2014-09-11_at_7.58.14_AM-b72e9d26.png

Andrew Milner

unread,
Sep 11, 2014, 11:39:25 AM9/11/14
to weewx...@googlegroups.com
No, it is WU who are still refusing to accept that when a field is missing they are substituting -999 for missing data - it is NOT being supplied to them - it is a bug in their software which processes the submitted data.

Marc's elegant test quite clearly shows the data submitted and the data as displayed by WU.  Perhaps you could reply and include a cut and paste of Marc's test - excluding any references to him of course!!

I really cannot believe they are being so adamant about blaming the submission, Marc's test shows the submitted string.

Andrew Milner

unread,
Sep 11, 2014, 11:40:11 AM9/11/14
to weewx...@googlegroups.com
Sorry - not Marc, should have been Luc.  My apologies for name confusion.


On Thursday, 11 September 2014 18:21:19 UTC+3, Henk van Rooyen wrote:

Thomas Keffer

unread,
Sep 11, 2014, 7:24:11 PM9/11/14
to weewx-user
Agreed.

Indeed a grep of the weewx code base shows no instances of the number 999, positive or negative.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Charles Sowers

unread,
Sep 18, 2014, 2:06:35 AM9/18/14
to weewx...@googlegroups.com
A quick look at 50 or so stations near me finds 10  that show precip rate or accum at -999.99. How do they not see this?

Charles

Matthew Sargent

unread,
Oct 1, 2014, 7:34:48 AM10/1/14
to weewx...@googlegroups.com
Is there any value we can use would indicate to WU that there is no data for that sensor?

Matt

Andrew Milner

unread,
Oct 1, 2014, 8:02:27 AM10/1/14
to weewx...@googlegroups.com
WU KNOW there is no value ... if you look at the tabular detailed data you will see they have put -- where the data is missing.  The problem is that they then replace their -- with these stupid values when they are creating say a daily average or any other aggregation.  It is totally within their grasp but they seem to just refuse to acknowledge this.
Whatever value you were to introduce would just make things worse - the best one can do is replace with the last submitted reading to at least prevent WU creating nonsensical values.

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nfyCbNWL36o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

Andrew Milner

unread,
Oct 1, 2014, 8:12:16 AM10/1/14
to weewx...@googlegroups.com
I will have to take back my previous comment as it seems that WU may possibly have made some changes at their end.  I can find days with missing readings in the tabular data (ie --) but cannot find where this data has produced odd values in weekly, monthly values etc - and in fact max and min values for days with gaps appear to be correct from the readings which were supplied.  Can someone else check and see if the problem has actually disappeared now?  I hope it has!!
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.

Luc Heijst

unread,
Oct 1, 2014, 12:44:47 PM10/1/14
to weewx...@googlegroups.com
On Wednesday, 1 October 2014 09:12:16 UTC-3, Andrew Milner wrote:
 Can someone else check and see if the problem has actually disappeared now?  I hope it has!!

I still have problems, I'm afraid. See attached WU year report.
The graphs of temperature/dewpoint and wind/gust speeds are in fact useless due to the values of -573 degrees Celsius and the range of 100 to 400 km/h negative wind/gust speeds.
Luc
WU year report.jpg

Andrew Milner

unread,
Oct 1, 2014, 1:09:34 PM10/1/14
to weewx...@googlegroups.com
They seem to be doing something.  I am ok from March or June (Not sure which) 2013.  However I can only get data for 2013 or 2014 under the yearly options - makes me think they are working through the old data slowly and corrcting something!!

Luc Heijst

unread,
Oct 4, 2014, 2:33:15 PM10/4/14
to weewx...@googlegroups.com
On Monday, 8 September 2014 12:37:57 UTC-3, Andrew Milner wrote:
Maybe the answer is just to stop submitting to WU but to somewhere else instead!!  Anyone got any ideas of a better place than WU which does not corrupt the data when some is omitted??

Weathercloud is new and fast growing. Maybe this might be in the near future an alternative for Wearher Underground. 
Kenneth Baker announced in the weewx development forum he is working on a weewx restful service for Weathercloud. 

Luc

Luc Heijst

unread,
Oct 8, 2014, 5:56:30 PM10/8/14
to weewx...@googlegroups.com
On Friday, 5 September 2014 13:29:18 UTC-3, Tom Keffer wrote:

Luc, did you ever hear anything from the WU?

-tk

Yesterday I wrote an email to Jeff Masters -co-founder of Weather Underground- in which I expressed my concerns about the problems with the WU data and WU's non-supportive help desk.
Today I received a mail from Kari Strenfel -meteorologist- who offered help to sort out the problems. I have confidence that the real communication with WU now has started.

Luc
 

Thomas Keffer

unread,
Oct 8, 2014, 6:05:09 PM10/8/14
to weewx-user
That's good news. If you need to name drop the 3,000+ weewx users, do it!

-tk

--

Luc Heijst

unread,
Oct 14, 2014, 4:25:47 PM10/14/14
to weewx...@googlegroups.com
Hello folks,

I had mail contact with Kari Strenfel of Weather Underground.

We discussed among others the problems with uploading of None Available data (N/A) to Weather Underground and how Weather Underground handles that data.
The protocol weewx uses right now for N/A data (omitting the N/A data and its parameter name in the message) is the right one. 
We discussed the possibility to send a "null" value instead, but that won't give us any advantages and is not implemented yet.
Kari wrote: "I will see if the developers can add a "null" value that will be accepted on our side.  For now, if the data is not available , -999 is the standard.  Any values of -999 or -99 will be assumed and displayed on our side as missing or N/A."

Note: Weather Underground uses the following values when dta is N/A.:
<relative_humidity> -999 percent
<temp_f> -999,0 Fahrenheit
<wind_degrees> -999 degrees
<wind_dir> North
<wind_mph> -999,0 mph
<wind_gust_mph> -999,0 mph
<pressure_in> -100,0 inhg
<dewpoint_f> -99,9 Fahrenheit
<precip_1hr_in> 0,00 inch

The N/A values in the Weather underground database should not presented as the numeric values above.
Kari wrote: "In programing languages and in meteorology, any -999 or -99 value means missing data or data is not available.  We are working on displaying this on our site as "NA" instead of a numeric value." 

Also these N/A data may not be part of any Weather Underground graphs or summary data.
I wrote to Kari: "Please realise you should also skip these "N/A" values in summary data calculations, like High / High  average / Mean / Low Average / Low - values and in graphs.".

Of course we are curious when all this will be realised.Kari couldn't give us a datae: "No , I cannot give you a date.  The developers are currently swamped.  I apologize.".

Luc

NB. I also mentioned the problems with deleting some custom data from the Weather Underground database and uploading history data to Weather Underground, but received no answer for these problems so far.

Luc Heijst

unread,
Oct 14, 2014, 6:14:39 PM10/14/14
to weewx...@googlegroups.com
The N/A value of <pressure_in> is in fact -99.999 and not -100,0 as I wrote in my previous message.
The translation to metric units should be -3386.354802785165 hpa, but WU presents this as -3386.0 hpa.

Thomas Keffer

unread,
Oct 14, 2014, 8:07:53 PM10/14/14
to weewx-user
​Presently, weewx just omits the fields for any bad or missing values, rather than set them to -999. Did you ask Kari about this?

-tk

--

L.J.M. Heijst

unread,
Oct 14, 2014, 8:13:39 PM10/14/14
to Thomas Keffer
Tom,
This is currently the right way to do this.
Luc
.
-- 
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nfyCbNWL36o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Luc Heijst

unread,
Oct 14, 2014, 8:20:28 PM10/14/14
to weewx...@googlegroups.com
On Tuesday, 14 October 2014 21:13:39 UTC-3, Luc Heijst wrote:
Tom,
This is currently the right way to do this.
Luc
(Omit the fields for any bad or missng valuaes in the message, I mean) 

f5...@free.fr

unread,
Jun 17, 2016, 11:05:24 AM6/17/16
to weewx-user
Hello

I am Phil from France.
I am using VP2 + Weewx 2.6.2 (Raspberry).
I saw this post but I don't understand if the problem is finished now or not.

I am running Wunderstation on Ipad2. All seems ok, but when there is no data for Temperature, Dew point or pressure, the dayly graph on wunstation show a bad value (very low around -15°C or 0 hpa)
My ID is IBRETAGN147.

I tried to delete the "no value" but the problem is still here.
I tried to delete the complete line, but the problem is still here.

I bought a SD card including WeeWx 2.6.2 with a configuration to export to special servers (CreaWeather, Meteo Bretagne and Info-climat).
For the moment, the manager of this card does not want to upgrade to latest versions. So I have to stay with this version.

I would like to know how you find a solution please.
Is it an update of WeeWx ?

Many thanks.

Sorry for my bad english.

F5PHW Phil

Thomas Keffer

unread,
Jun 17, 2016, 11:33:46 AM6/17/16
to weewx-user
Hello, Phil

Unfortunately, this is a problem with the Weather Underground servers. If they do not receive a value for an observation, they substitute a random value.

Despite several complaints from us, they do not seem to be fixing the problem.

Luc tested many different strategies that weewx could take to get around this problem, but none of them seem to work. So, there does not appear to be anything we can do on our end.

Sorry

-tk


--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

f5...@free.fr

unread,
Jun 17, 2016, 12:00:21 PM6/17/16
to weewx...@googlegroups.com
Hello Thomas

Thanks for quick answer.
Good news for my poor english language, I understood well the problem and it is not a weewx problem.
I sent mail to WU (Raki) and I am waiting for his answer.

Many thanks.

Phil


Web page on http://f5phw.free.fr
F5PHW on FACEBOOK : https://www.facebook.com/phil.berger.75
WX in Nostang on http://www.nostang.meteoamikuze.com/index.html

----- Mail original -----
De: "Thomas Keffer" <tke...@gmail.com>
À: "weewx-user" <weewx...@googlegroups.com>
Envoyé: Vendredi 17 Juin 2016 17:33:14
Objet: Re: [weewx-user] Re: Weather Underground can't handle zero records from weewx
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/nfyCbNWL36o/unsubscribe .
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com .
Reply all
Reply to author
Forward
0 new messages