Weewx -- wunderground problem when weather-station to base communication drops out

[Ralph Peters]

This is a new problem for me.  I have been running an acurite pro weather station with weewx for over 2 years (see https://www.wunderground.com/personal-weather-station/dashboard?ID=KNMALBUQ335)

When communication drops out between the weather station and the base, weewx says there is stale data

......

Jun 12 15:49:35 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:49:35 raspberrypi weewx[969]: acurite: R1: ignoring stale data (rssi indicates no communication from sensors): 01 0e 7a 71 02 62 00 3e 00 00
Jun 12 15:49:35 raspberrypi weewx[969]: acurite: next read in 18 seconds
Jun 12 15:49:53 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:49:53 raspberrypi weewx[969]: acurite: R1: ignoring stale data (rssi indicates no communication from sensors): 01 0e 7a 71 02 62 00 3e 00 00
Jun 12 15:49:53 raspberrypi weewx[969]: acurite: next read in 18 seconds

If I look at the weather data from my local computer, I see there is a weather data gap starting about 15:00 gap and signal quality is zero starting about 15:00.  See the first attachment.

But wunderground shows bad/weird data.  See the second attachment 15:00 where we see an outside temperature of zero!  And other weirdness also.

How can I fix it so that the wunderground site is NOT reporting bad data???

If I look at weewx.conf, I see that I have settings to assure "quality".

from weewx.conf:

[StdQC]
   
    [[MinMax]]
        barometer = 26, 32.5, inHg
        outTemp = -40, 120, degree_F
        inTemp = 10, 120, degree_F
        outHumidity = 0, 100
        inHumidity = 0, 100
        windSpeed = 0, 120, mile_per_hour
        pressure = 24, 34.5, inHg

Any bright ideas about how to fix this??  It is a recent problem -- seen in the last few weeks.  I am using weewx 3.8.0.1 from 11/2017.

Thanks!

[Ralph Peters]

Additional info:

It appears weewx may be posting to wunderground, even though the data is bad -- see entry:

Jun 12 15:50:18 raspberrypi weewx[969]: restx: Wunderground-PWS: Published record 2018-06-12 15:50:00 MDT (1528840200)

taken from the log:

Jun 12 15:50:11 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:50:11 raspberrypi weewx[969]: acurite: R1: ignoring stale data (rssi indicates no communication from sensors): 01 0e 7a 71 02 62 00 3e 00 00
Jun 12 15:50:11 raspberrypi weewx[969]: acurite: next read in 6 seconds
Jun 12 15:50:17 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:50:17 raspberrypi weewx[969]: manager: Added record 2018-06-12 15:50:00 MDT (1528840200) to database 'weewx.sdb'
Jun 12 15:50:17 raspberrypi weewx[969]: manager: Added record 2018-06-12 15:50:00 MDT (1528840200) to daily summary in 'weewx.sdb'
Jun 12 15:50:17 raspberrypi weewx[969]: restx: StationRegistry: wait interval (324000 < 604800) has not passed for record 2018-06-12 15:50:00 MDT (1528840200)
Jun 12 15:50:17 raspberrypi weewx[969]: reportengine: Running reports for latest time in the database.
Jun 12 15:50:17 raspberrypi weewx[969]: reportengine: Running report StandardReport
Jun 12 15:50:17 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:50:17 raspberrypi weewx[969]: reportengine: Found configuration file /etc/weewx/skins/Standard/skin.conf for report StandardReport
Jun 12 15:50:17 raspberrypi weewx[969]: cheetahgenerator: using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']
Jun 12 15:50:17 raspberrypi weewx[969]: acurite: next read in 12 seconds
Jun 12 15:50:17 raspberrypi weewx[969]: manager: Daily summary version is 1.0
Jun 12 15:50:18 raspberrypi weewx[969]: restx: Wunderground-PWS: Published record 2018-06-12 15:50:00 MDT (1528840200)
Jun 12 15:50:20 raspberrypi weewx[969]: cheetahgenerator: Generated 14 files for report StandardReport in 2.75 seconds
Jun 12 15:50:20 raspberrypi weewx[969]: manager: Daily summary version is 1.0
Jun 12 15:50:21 raspberrypi weewx[969]: imagegenerator: Generated 10 images for StandardReport in 1.15 seconds
Jun 12 15:50:21 raspberrypi weewx[969]: copygenerator: copied 0 files to /var/www/weewx

[Thomas Keffer]

I'm not sure what is going on here, but most likely the WU is substituting a value of zero for a missing temperature value. That would not surprise me.

The other possibility is that the Acurite driver is misinterpreting a bad value as 0, but that seems unlikely: the driver expects values in celsius, so I would expect to see 32F, not 0F.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ralph Peters]

Hi,

I agree that it is most likely that WU is getting weird data and doing some sort of "fix/substitution/....  I guess my question, now, is

Can I tell Weewx to "NOT PUBLISH" to WU if the data is know to be stale?  It seems silly to connect with WU if the data is bad...

Ralph

[gjr80]

Hi Ralph,

You might want to try running weeWX with debug=2, that will result in the exact data being set to WU being record in the log. Assuming the issue is ongoing you can then check what is being sent to WU against what you are recording in your archive against what WU is displaying. Will be clear then whether the issue is WU or weeWX. I think I know where I would put my money, but if you want to know for sure...

To the best of my knowledge the only 'control' over what data is sent to WU (apart from enabling/disabling posting to WU in weewx.conf) is that any obs that is None is omitted by weeWX from the uploaded data. There is no ability to not post if the data is stale.

Gary

[Ralph Peters]

Thanks.  I will give debug=2 a try.  I think that I can test by putting up a barrier around the base (aluminum foil?)  to break communication.  Thanks for the other info also. 

Ralph

[Anthony Fisher]

I can confirm this is a problem for me too. I am using weewx to post data from an Oregon LW301 set up with temp, baro, rain and wind sensors and have noticed several times per day the temperature and barometer readings dropping to 0f (-17.8C as I'm using metric) for temp and 0 for pressure on Wunderground.

It seems to be when weewx does not send data if a sensor is not reporting temporarily.

I tested today by taking the outdoor temp sensor offline for a few minutes and weewx didn't send data to Wunderground for a few minutes - during this time Wunderground plots the temp in tables and graphs at 0f. Surely they should record nothing/null/NA when they don't get a value?

Screenshots of the weewx debug log and results on Wunderground attached.

Cheers,

Anthony

[Thomas Keffer]

I tested today by taking the outdoor temp sensor offline for a few minutes and weewx didn't send data to Wunderground for a few minutes - during this time Wunderground plots the temp in tables and graphs at 0f. Surely they should record nothing/null/NA when they don't get a value?

​You would think so. 

Just to make a fine point: WeeWX did indeed send data to the WU, it just didn't send any missing data, in this case, temperature.

I learned a long time ago not to chase bugs on Wunderground. There is too many of them, they come and go, and they never get fixed. We just have to live with them.

-tk

​

 

[Ralph Peters]

Hii

It would be nice if we had an option ON WEEWX, that said:

"DO NOT SEND ANY DATA OR CONNECT IF THE DATA IS STALE OR BAD".

Then the individual could decide what to do.  Yes, that would put a hole in the data stream and WU might complain, but that might not be a bad thing. 

Where can we make a suggestion to the keeper of weewx?

Ralph

[Thomas Keffer]

You just did.

This is unlikely to be a priority, but if you want to submit a Pull Request...

A more general facility would work for all RESTful services, and would allow you to specify which observation types are required.

-tk

[Andrew Milner]

How do you define stale data and/or bad data - especially when some of the station types using weewx do not output all data in every oop packet - as they emit partial packets and some packets only appear every x minutes - meaning that there has earlier been a gut of requests for the restx services to do data caching so that they upload stae data to wu to cover the occasions when they do not have any fresh data from a particular sensor.  

The best option is to not rely on WU at all - but to et weewx take care of things in the way it does best - sticking to its principles of not inventing data or filling in gaps - but reporting exactly what I has seen.

[Ralph Peters]

Hi,

In my case we see:

Jun 12 15:49:35 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:49:35 raspberrypi weewx[969]: acurite: R1: ignoring stale data (rssi indicates no communication from sensors): 01 0e 7a 71 02 62 00 3e 00 00
Jun 12 15:49:35 raspberrypi weewx[969]: acurite: next read in 18 seconds
Jun 12 15:49:53 raspberrypi weewx[969]: acurite: Found station at bus=001 device=004
Jun 12 15:49:53 raspberrypi weewx[969]: acurite: R1: ignoring stale data (rssi indicates no communication from sensors): 01 0e 7a 71 02 62 00 3e 00 00
Jun 12 15:49:53 raspberrypi weewx[969]: acurite: next read in 18 seconds

.

The "ignoring stale data" comment would seem to be an indication that sending NO DATA at the next "send to WU" could be a reasonable thing to do, particularly if no "non-stale" data is available for the last time period.  YES, there are occasions when it is difficult to decide to send or not send.  It seems that we cannot always assume that WU will do the "right thing".

WEEWX developers, think about adding an option to not-send if ONLY stale data is available!!

Thanks for all the comments,

Ralph

[Andrew Milner]

I don't even think all drivers have a sense of 'stale data'.  The accurite driver does - agreed - but it is also the driver that has the hardest problem coping with partial packets and partial transmissions.  the restx option is independent of the drivers however.  I suspect you're going to end up giving Tom his pull request if you want the issue resolving!!

Even in your example it is not clear if ALL data is stale, or just one sensor which is not transmitting - the rest being good data, since accurite outputs partial packets - and I was under the impression that restx has recently been updated to perform caching in order to handle partial packets better for WU uploads.

[Thomas Keffer]

This might be doable. The goal is to send no data at all, if certain select observation types are missing. 

There are many "partial packet" stations out there, where observation types are missing on nearly every packet. Right now, we use a cache in restx.py to fill them back in before posting to WU. This cache will only hold old data up to a certain age, then it throws them out.

So, once the cache has done its job, we could examine the resultant post and see if certain "essential" observation types are missing. If so, we pass on doing the post.

This introduces some complexity to get around what is a pure Weather Underground problem. 

-tk

[Anthony Fisher]

I scaled back the weewx archive interval to 5 mins to make it more likely to pick up valid values from all sensors (previously was sending every minute). Worked well for half the day but have had another couple of drop outs the last hour that have made it through to the Wunderground data.

That solution sounds good to me (from a purely selfish point of view), even if it is correcting for a Wunderground problem. It doesn't sound like it will get better on their end overnight if they have a history of such things, and I'd much rather a brief problem doesn't throw the figures out completely - kind of defeats then purpose of using them.

Unless someone can recommend a different service with equivalent usability and charting that weewx posts to? I started posting to WOW as well, but a brief play with their web interface left a bit to be desired in comparison. Frustrating that WU can't get the data side right.

Side note - I can't get WU to delete the bogus data records either using Safari - you use their button to delete a highlighted record, their site says it is deleting the record, it doesn't show momentarily, then when you refresh the page it is back! I tried in Safari and then Chrome. Nothing to do with weewx of course, but exacerbates the problem somewhat!

[Andrew Milner]

Although I upload to WU I never actually use their site to view the data - instead I run a webserver on the Raspberry Pi that runs weewx and just display the pages generated by weewx (with bootstrap skin).  I don't find WU very useable or necessary, but use it more as a cloud storage area for possible use in disaster recovery 

[Anthony Fisher]

I'll have to invest some time doing that. I was probably trying to have something working quickly the lazy way. :)