V4.6.0 beta 3

[Tom Keffer]

The Seasons skin has been parameterized. It should now work out of the box for any combination of types that appear in the extended wview schema. See PR #702.

http://weewx.com/downloads/development_versions/

[t...@tom.org]

Docker image updated for b3 at mitct02/weewx:4.6.0b3 if anyone wants to test/play without affecting your system. I am happy to support people who want to use it or try it. It works for me, but I have not worked closely with many people on supporting their use cases.

[Doug Jenkins]

Tom:

I will give your docker image a try and let you know. Been interested in moving my main WeeWx setup to Docker. Will report any issues and any suggestions back to you.

Doug

[Vince Skahan]

If you don't mind building your own, I have a repo up at https://github.com/vinceskahan/weewx-docker that takes a little different tact in building a docker image of weewx for a bunch of os variants all off those mainstream base os.    If you build the ubuntu2004 'setup' variant then it's only 168 MB in size plus 126 MB for the nginx image.

You'd have to do a little light editing of the Dockerfile to pull the beta version of your choice.  Should be obvious.

[Doug Jenkins]

Vince:

Actually I was emailing Tom M. yesterday about his docker image. I have it "kind of" working with mqtt and belchertown extensions. my problem is that I use docker compose to setup a weewx "stack" of containers to support my solution (niginx container, mqtt container, mariadb container). Docker Compose will not allow you to directly map a file (eg weewx.conf) from your host to a target file inside a container.

the other problem is device mapping. I am using Accurite 5n1 weather station that its display is plugged in via USB to a machine. I am trying to map the /dev/bus folder via volumes to the container, but WeeWX fails its LOOP generation as the USB is not readable.

I will take a look at your repo and Tom's repo to see what can be used together. 

I am researching the device mapping as I do not want to run my container in privileged mode unless there is no option.

[Vince Skahan]

On Thursday, October 7, 2021 at 11:40:06 AM UTC-7 do...@dougjenkins.com wrote:

 Docker Compose will not allow you to directly map a file (eg weewx.conf) from your host to a target file inside a container.

It does for me.  I run my ecowitt weewx instance in docker that way.

services:
    weewx:
        volumes:

          - weewx-ecowitt-html:/home/weewx/public_html
          - weewx-ecowitt-archive:/home/weewx/archive
          - weewx-ecowitt-ssh:/root/.ssh
          - weewx-ecowitt-logs:/var/log
          - ${PWD}/configs/weewx.conf:/home/weewx/weewx.conf

 

the other problem is device mapping. I am using Accurite 5n1 weather station that its display is plugged in via USB to a machine. I am trying to map the /dev/bus folder via volumes to the container, but WeeWX fails its LOOP generation as the USB is not readable.

I've done this with the zWave dongle device that I use for Home Assistant with a specified device, but not with the whole folder.

services:

   homeassistant:

     devices:
       - /dev/ttyACM0:/dev/ttyACM0

I am researching the device mapping as I do not want to run my container in privileged mode unless there is no option.

That I can't help you with other than to suggest using podman, which did seem to work unprivileged when I tried it in early 2020, but it wasn't a fully functional docker replacement for me on x86_64 at that time and it was a bit odd in the hoops you needed to do to get it to run unprivileged.  I haven't kept up with it since then so maybe today's story is more promising.

 

[Bill Richter]

I was running V4.6.0. beta 2.  I upgraded.  Lots of new readings for soil* leaf* with values for sensors my Davis Vantage doesn't have.  Also the transmitter value went from OK to 0.00000.  I thought these soil and leaf values (along with others that were new) were defined in the seasons skin.  No luck in removing them.  The lack a valid transmitter value was also a deal breaker.  I've reverted from beta3 to beta2.  I neglected to try the Standard skin while on beta3.  No issues upgrading from beta1 to beta2.

No I don't have a 'test' environment for this, so further diagnostics are not easy.  I think I might just wait until V4.6.0 and upgrade then. This is the package I used, python3-weewx_4.6.0b3-1_all.deb

[Tom Keffer]

Thanks, Bill.

The transmitter value was fixed in commit 378cd18.

By "Lots of new readings", do you mean in the templates? Or, plots? 

Sorry for the trouble!

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/4dec109b-7d06-439c-953a-46fdfcc0b3abn%40googlegroups.com.

[Tom Keffer]

Couple other questions: are the values in your database for soil* and leaf* null? Or, zero? Have you used those database columns for something else?

[Vince Skahan]

The beta looks good on my Ecowitt setup with inside gateway/sensor, outside sensor, 3 outside sensors, and one soil sensor.

I do see blank wind+rain graphs and a zero for rain rate in the data table that probably shouldn't be there, considering that I have no such sensors, but it did automagically pick up the extra t+h and soil sensor, which was pretty cool.

[Bill Richter]

By "Lots of new readings", do you mean in the templates? Or, plots? 

 

I mean new readings, leaf* and soil* in the left hand summary high/low. 

 

I didn't think to check the database values (sqlite), since nothing shows up in beta2 and I have nothing writing to those things like soilMoist1.  But imagine my surprise to see, yes there is non 0 data in there.

 

OK, so I don't think I want to delete the tables.  Should the values be set to 0 or NULL?  It looks like the value should be set to NULL. The puzzle for me is the difference between beta2 and beta3.  As well as why I can't edit the skin to remove the generation of soilMoist1.  On the other hand, it would work 'normally' if the soilMoist1 was 0 or NULL.  No idea how those values would get in the database.

[Tom Keffer]

Beta 3 uses the presence of data in the database to decide whether or not to produce a plot and/or include the data in the left-hand column.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/199909aa-2fe7-4829-bd84-2f4e0a6dfd8dn%40googlegroups.com.

[Bill Richter]

Yes, I was thinking that based on the changes and the new value for checking data first.  Just not clear why the database has values, unexpected.  I'll set the values to NULL for the sensors that don't exist but have data in the database.   I feel lucky so, I'll try beta3 again.

[Tom Keffer]

Bill, another question. Normally, when you upgrade between versions, the installer does not touch your skins. Are you running your old pre V4.6 Seasons skin? Or, the V4.6 skin?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/48f63cae-debe-4ab8-908a-23e91e8f507cn%40googlegroups.com.

[Bill Richter]

The first time I did it, I just did a apt install python3-weewx_4.6.0b3-1_all.deb.  I've not noticed a change log or I don't recall any instructions for upgrading from 4.5.1 to 4.6.0.beta1.

So when I downgraded I did a apt remove weewx, then installed beta2. Just now I did an install to get to beta3 and added the battery patch. That patch fixes the battery reading.

First,

sqlite> update archive set leafWet2=NULL;
sqlite> select leafWet2 from archive where leafWet2 IS NOT NULL;
sqlite>

(time passes, at least 5M)

sqlite> select leafWet2 from archive where leafWet2 IS NOT NULL;
8.0
sqlite> 

And now I have data in leafWet2!

[Bill Richter]

I also set a limit in weewx.conf for leafTemp1.   leafTemp1 = 0, 0, degree_F

Oct 12 19:00:14 weewx weewx[25634] WARNING weewx.qc: 2021-10-12 18:55:00 PDT (1634090100) Archive value 'leafTemp1' -26.0 outside limits (0.0, 0.0)

Suggesting again, something is putting incorrect data into weewx.sdb for certain fields that have no real sensor.

So to make sure I had the new Seasons  skins, I apt remove weewx, then apt install python3-weewx_4.6.0b3-1_all.deb

weewx# ls -l
total 16
drwxr-xr-x 2 root root 4096 Oct 12 18:54 font
drwxr-xr-x 2 root root 4096 Oct 12 18:54 lang
drwxr-xr-x 2 root root 4096 Oct 12 18:54 NOAA
-rw-r--r-- 1 root root 2732 Oct 12 18:52 sensors.inc

[gjr80]

First step in looking for the culprit is some config info. Running wee_debug and posting the output would be good. Just remember if posting wee_debug output to check it for sensitive info first (eg user names, passwords, API keys etc), wee_debug should obfuscate such info but it's not perfect.

Gary

[gjr80]

On more thing, re Seasons template versioning. Tom has started versioning the Seasons template so if you have a look in skins/Seasons/index.html.tmpl you should find a version number in the first few lines, eg:

## Copyright 2009-2018 Tom Keffer, Matthew Wall
## Distributed under terms of GPLv3. See LICENSE.txt for your rights.
## Version: 4.6.0b4

#errorCatcher Echo

Gary

On Wednesday, 13 October 2021 at 12:15:35 UTC+10 bill.g...@gmail.com wrote:

[Bill Richter]

and if I can read your mind, here is a LOOP packet.

    # The type of LOOP packet to request: 1 = LOOP1; 2 = LOOP2; 3 = both
    loop_request = 1

LOOP:   2021-10-12 21:01:21 PDT (1634097681) altimeter: 30.08348929681147, appTemp: 40.85484139638294, barometer: 30.103, cloudbase: 8025.412856671072, consBatteryVoltage: 4.7, dateTime: 1634097681, dayET: 0.0, dayRain: 0.0, dewpoint: 21.564183430647283, ET: None, extraAlarm1: 0, extraAlarm2: 0, extraAlarm3: 0, extraAlarm4: 0, extraAlarm5: 0, extraAlarm6: 0, extraAlarm7: 0, extraAlarm8: 0, forecastIcon: 6, forecastRule: 44, heatindex: 41.75600000000001, humidex: 45.7, inDewpoint: 38.953359270593296, inHumidity: 33.0, insideAlarm: 0, inTemp: 69.3, leafWet1: 0, leafWet4: 0.0, maxSolarRad: 0.0, monthET: 0.0, monthRain: 0.2, outHumidity: 38.0, outsideAlarm1: 0, outsideAlarm2: 0, outTemp: 45.7, pressure: 27.423019470139963, rain: 0.0, rainAlarm: 0, rainRate: 0.0, soilLeafAlarm1: 0, soilLeafAlarm2: 0, soilLeafAlarm3: 0, soilLeafAlarm4: 0, soilMoist1: 0, soilTemp1: 0, stormRain: 0.0, sunrise: 1634047860, sunset: 1634088480, txBatteryStatus: 0, usUnits: 1, windchill: 45.7, windDir: None, windGust: 0.0, windGustDir: None, windrun: None, windSpeed: 0.0, windSpeed10: 0.0, yearET: 0.0, yearRain: 0.2

[Bill Richter]

## Version: 4.6.0b4

I ponder wee_debug.

This is a change between beta2 and beta3, I didn't make any adjustments to weewx.conf, just the upgrade.

I observed  apt remove weewx left things behind, I can see it leaving things that have been modified but not if the same code matches the file in the deb archive.  So, I'm perplexed as well that on install the Seasons files were not restored.  I copies them from git to get them back in place and confirm the are the most recent dev versions.

So the code change to the Seasons skin seems to be working, but I'm a bit stuck now on tracing why  data is being loaded into weewx.sdb from a value that is not output by Vantage.  IE in the LOOP packet leafWet1 = 0.

I'd expect the test to be showvalue if not NULL or 0.

Since I set the values to NULL  and the LOOP packet is saying 0 and that's what's being put into weewx.sdb, I'm not expecting the label or graph to be generated via if $year.leafWet1.has_data or $year.leafWet2.has_data.

sqlite> select leafWet1 from archive where leafWet1 IS NOT NULL;
0.0
0.0
0.0
0.0

sqlite>

So I struggle here,

1. LOOP data shows 0, but the label and graph is displayed.

2. LOOP data for other values doesn't appear to exist yet it's in the database now. 

3. Just to  eliminate database issues, I've set leafWet2 and leafWet2 to NULL so the graph and label shouldn't print.

4. but this is new behavior from beta2 to beta3.

My brain hurts.

[gjr80]

Bill,

The (one) change from b2 to b3 is that Seasons will now display whatever data it can find in any of the extended schema files in your database. Previously, Seasons displayed a fixed subset of the extended schema. b3/b4 Seasons is not creating the bogus data, it is merely displaying it - something else is creating the bogus data. That could be any of a myriad of sources; the driver, a core WeeWX service or some third party service. It is impossible to say what the source could be without knowing how WeeWX is configured, ie what driver, what services etc are loaded and how are they configured. That is what weewx.conf/wee_debug will tell us.

Since you seem to have installed the deb package you should be able to run wee_debug by simply typing:

$ wee_debug —info

to display the wee_debug output on the console. You can add the —output option to redirect the output to a file if you wish. Alternatively, if you are not comfortable using wee_debug you can just take copy of your weewx.conf, sanitise it by obliterating any sensitive info (user names, passwords, API keys etc) and post the sanitised file.

Gary

[Bill Richter]

Always, just after I hit send...here is a LOOP packet with data for sensors that don't exist on the Vantage Vue. I can see now then since the skin change to show if data is there, this will be displayed.  But here is another LOOP packet snippet.

leafTemp1: -90.0, leafTemp2: -81.0, leafWet1: 0, leafWet2: 8.0

So I can guess Davis isn't going to be helpful.  It would appear that's the source of the bad data.

So,  weewx needs to know these values should be forced to NULL or 0, which would prevent display.

[Bill Richter]

I'm not sure the wee_debug will help, but I attached it.

So,  yes, the data is coming in from the driver in a LOOP packet for sensors that don't exist and getting put into the database.   The problem is down at the driver.  So there are no configuration options on the driver to force values to NULL or 0.   Assuming either value is valid, I didn't look at the source, forcing the field to NULL should prevent display.

So I guess what you'd suggest is:

stop weewxd.

set the leafWet1 to NULL

run wee_reports

check the web page to see if leafWet1 is gone.

[gjr80]

Unfortunately I don’t see any wee_debug output attached? I’d suggest we find the source and fix the problem at source. The driver is most likely the culprit but when you are displaying LOOP packets on the console you are not looking at the data straight out of the driver, rather you are looking at the LOOP packet after every service has had its turn processing the LOOP packet. So it doesn’t pin down the source.

Gary

[Bill Richter]

No I forgot it, attached here. I was looking on github weewx, but I'm not finding the drivers directory.  So I went looking at the vantage.py source /usr/share/weewx/weewx/drivers/vantage.py.  Python is not my strong suit.  So I wondered if

   'leafWet1'        : lambda p, k: float(p[k]) if p[k] != 0xff else None,

could be changed to 'leafWet1'   :None

and that would force the driver to always return NULL.

Then I thought, how about monitoring the serial port to see what was coming in.  Another challenge, I think wireshark might help trace ttyUSB0 data. I tried screen, I'll have to try that again, thinking that would accept the LOOP command and I'd see raw data from the Vantage console.

[gjr80]

OK, nothing of consequence in weewx.conf. You can run the vantage driver directly and have it output loop packets direct to the console, this takes WeeWX out of the equation. You can run the vantage driver directly by:

1. stopping WeeWX

2. running the following command:

$ PYTHONPATH=/usr/share/weewx python3 -m weewx.drivers.vantage

You should see continuous loop packets on the console. Are any bogus fields showing? If not let it run for a while. You can post the output here. Control-C will shut that down. Would also be useful to know the driver version:

$ PYTHONPATH=/usr/share/weewx python3 -m weewx.drivers.vantage —version

Gary 

[Bill Richter]

DRIVER_NAME = 'Vantage'
DRIVER_VERSION = '3.2.3'

output looks 'normal'.  I let it run for a while, looking for leafTemp1, nothing found, but the database is getting a new record every 5 minutes.  After setting to 0.0 and confirming no none 0.0 records.

sqlite> select leafTemp1 from archive where leafTemp1 != 0.0;
-26.0
-26.0
-26.0
sqlite>

{'dateTime': 1634110148, 'usUnits': 1, 'barometer': 30.104, 'inTemp': 70.3, 'inHumidity': 31.0, 'outTemp': 42.2, 'windSpeed': 0.0, 'windSpeed10': 0.0, 'windDir': 342.0, 'outHumidity': 47.0, 'rainRate': 0.0, 'stormRain': 0.0, 'dayRain': 0.0, 'monthRain': 0.2, 'yearRain': 0.2, 'dayET': 0.0, 'monthET': 0.0, 'yearET': 0.0, 'leafWet4': 0.0, 'insideAlarm': 0, 'rainAlarm': 0, 'outsideAlarm1': 0, 'outsideAlarm2': 0, 'extraAlarm1': 0, 'extraAlarm2': 0, 'extraAlarm3': 0, 'extraAlarm4': 0, 'extraAlarm5': 0, 'extraAlarm6': 0, 'extraAlarm7': 0, 'extraAlarm8': 0, 'soilLeafAlarm1': 0, 'soilLeafAlarm2': 0, 'soilLeafAlarm3': 0, 'soilLeafAlarm4': 0, 'txBatteryStatus': 0, 'consBatteryVoltage': 4.7, 'forecastIcon': 6, 'forecastRule': 44, 'sunrise': 1634134260, 'sunset': 1634174820, 'rain': 0.0}

The plot thickens.

[Greg Troxel]

I just started having a lot of messages from weewx-devel land in spam.
It took a bit to track it down, but apparently somehow this got
rewritten (I think perhaps by Bill's client) from

> The transmitter value was fixed in commit 378cd18

> <https://github.com/weewx/weewx/commit/378cd18b42366391dba818dcd1d1581a509d46de>

to

The transmitter value was fixed in commit=C2=A0<a=
href=3D"https://github.com/weewx/weewx/commit/378cd18b42366391dba818dcd1d1=
581a509d46de" target=3D"_blank" rel=3D"nofollow" data-saferedirecturl=3D"ht=
tps://www.google.com/url?hl=3Den&amp;q=3Dhttps://github.com/weewx/weewx/com=
mit/378cd18b42366391dba818dcd1d1581a509d46de&amp;source=3Dgmail&amp;ust=3D1=
634150760731000&amp;usg=3DAFQjCNE0_G29uab2hCQGkhg03Op_EJRSlA">378cd18</a>

which seems to wrap it in a google lookup with tracking IDs, and
triggers

* 5.0 GOOG_MALWARE_DNLD File download via Google - Malware?

in spamassassin. I was getting ready to report a false positive rule
when I figured out that the rule was not firing incorrectly.

Of course I realize there's no malware, but probably (non-weewx!) people
use this technique to sneak malware links past incoming mail security
scanners.

Probably there is nothing to be done, but I thought I'd explain in case
others are wondering why half of their weewx messages are ending up in
the spam folder.

Greg

[Bill Richter]

gmail irritates me to no end. Along with a lot of other spam filters who fail to check a domains email reputation stopping once they see a dynamic IP.  As best I can tell, all I did on that link was cut and paste that change to apply  to my version of beta3.  I did try to reply to a message from weewx-devel, but it bounced due to missing PTR record on my IPv6, so I finally fixed that.  I also added an attachment via the weewx-devel conversation  but that doesn't appear to be the cause.  No more ranting. Sorry.

[Vince Skahan]

Tom - here's a small patch to Seasons index.html.tmpl that only shows the wind+rain graphs if there is data.  My ecowitt setup doesn't have those sensors...I didn't see an easy way to not show rain rate in the tabular settings so that always shows 0.00 of course.

diff index.html.tmpl /etc/weewx/skins/Seasons/index.html.tmpl  -u

--- index.html.tmpl 2021-10-05 15:01:07.130512500 -0700

+++ /etc/weewx/skins/Seasons/index.html.tmpl 2021-10-13 09:48:45.397240373 -0700

@@ -51,10 +51,14 @@

             <img src="${x}tempdew.png"   alt="$obs.label.outTemp" />

             <img src="${x}tempfeel.png"  alt="$obs.label.feel" />

             <img src="${x}hum.png"       alt="$obs.label.outHumidity" />

+            #if $year.windSpeed.has_data

             <img src="${x}wind.png"      alt="$obs.label.windSpeed" />

             <img src="${x}winddir.png"   alt="$obs.label.windDir" />

             <img src="${x}windvec.png"   alt="$obs.label.windvec" />

+            #end if

+            #if $year.rain.has_data

             <img src="${x}rain.png"      alt="$obs.label.rain" />

+            #end if

             #if $year.ET.has_data

             <img src="${x}ET.png"        alt="$obs.label.ET"/>

             #end if

[Greg Troxel]

Bill Richter <bill.g...@gmail.com> writes:

> gmail irritates me to no end. Along with a lot of other spam filters who
> fail to check a domains email reputation stopping once they see a dynamic
> IP. As best I can tell, all I did on that link was cut and paste that
> change to apply to my version of beta3. I did try to reply to a message
> from weewx-devel, but it bounced due to missing PTR record on my IPv6, so I
> finally fixed that. I also added an attachment via the weewx-devel
> conversation but that doesn't appear to be the cause. No more ranting.
> Sorry.

Thanks for commenting. I didn't mean to give you a hard time personally
and would have guessed that you did something reasonable and google
decided to be extra helpful in an odd way. I was just expecting that a
bunch of other people were having spam filtering issues and thought I
might save them some cycles in debugging it. (I'm on the spamassasin
users list and report buggy rules, a number of which have been
adjusted.)

[gjr80]

On Wednesday, 13 October 2021 at 17:37:33 UTC+10 bill.g...@gmail.com wrote:

DRIVER_NAME = 'Vantage'
DRIVER_VERSION = '3.2.3'

Good, current version of the driver.

 

output looks 'normal'.  I let it run for a while, looking for leafTemp1, nothing found,

So the output you refer - I am guessing that is from running the driver directly as I mentioned previously? How long did you let it run for, at least 10 minutes (twice your archive interval)? It only takes one loop packet during an archive interval to include leafTemp1 and leafTemp1 will appear in your database.

 

but the database is getting a new record every 5 minutes. 

Hang on, were these records saved when running the driver directly? If so that shouldn't happen, running the driver directly just outputs loop packets to the console, nothing is sent to WeeWX and hence nothing written to database. Did you stop WeeWX before running the driver directly?

 

After setting to 0.0 and confirming no none 0.0 records.

Not sure what you mean by this.

sqlite> select leafTemp1 from archive where leafTemp1 != 0.0;
-26.0
-26.0
-26.0
sqlite>

When querying the database you might find it useful (and will help us) if you use a query similar to this:

SELECT dateTime,datetime(dateTime, 'unixepoch','localtime'),leafTemp1 FROM archive WHERE leafTemp1 != 0.0;

This will give something like the following:

 

1487015700|2017-02-14 05:55:00|2.0

where you can see the timestamp of the record.

{'dateTime': 1634110148, 'usUnits': 1, 'barometer': 30.104, 'inTemp': 70.3, 'inHumidity': 31.0, 'outTemp': 42.2, 'windSpeed': 0.0, 'windSpeed10': 0.0, 'windDir': 342.0, 'outHumidity': 47.0, 'rainRate': 0.0, 'stormRain': 0.0, 'dayRain': 0.0, 'monthRain': 0.2, 'yearRain': 0.2, 'dayET': 0.0, 'monthET': 0.0, 'yearET': 0.0, 'leafWet4': 0.0, 'insideAlarm': 0, 'rainAlarm': 0, 'outsideAlarm1': 0, 'outsideAlarm2': 0, 'extraAlarm1': 0, 'extraAlarm2': 0, 'extraAlarm3': 0, 'extraAlarm4': 0, 'extraAlarm5': 0, 'extraAlarm6': 0, 'extraAlarm7': 0, 'extraAlarm8': 0, 'soilLeafAlarm1': 0, 'soilLeafAlarm2': 0, 'soilLeafAlarm3': 0, 'soilLeafAlarm4': 0, 'txBatteryStatus': 0, 'consBatteryVoltage': 4.7, 'forecastIcon': 6, 'forecastRule': 44, 'sunrise': 1634134260, 'sunset': 1634174820, 'rain': 0.0}

 OK, this looks like a loop packet straight off the driver/station, all looks good.

The plot thickens.

Yes. A little background on how your station works with WeeWX. The vantage driver obtains loop packets from your station every 2.5 odd seconds. Various WeeWX services process the incoming loop packets; notable here are StdCalibrate which applies corrections to the loop data and StdWXCalculate which adds derived obs to the loop packet (eg apparent temperature). WeeWX then accumulates the loop packet data and the process repeats. At the end of the archive interval (every five minutes in your case) WeeWX asks the driver/station for an archive record. In your case the station logger generates that archive record and sends it to the driver which sends it to WeeWX. Again those WeeWX services process the archive record. One important difference to how loop packets are processed is that WeeWX extracts whatever additional information it can from those accumulated loop packets and adds it to the archive record if it is not already in the archive record. WeeWX then saves this archive record to your database. This is a greatly simplified outline of what goes on and it can vary from user to user depending on settings.

So what? There are a few mechanisms that come to mind that could see leafTemp1 data saved in your database. For that to happen leafTemp1 must appear in the archive record that WeeWX saves to database. The following possibilities come to mind:

1. leafTemp1 is appearing regularly or randomly in loop packets, WeeWX accumulates this data and when the archive record is being processed before saving to database WeeWX is extracting the accumulated leafTemp1 data and adding it to the archive record which is then saved to database. This appears to be unlikely given you did not see any leafTemp1 data in the loop packets when you ran the driver directly. Though this depends on how often the issue shows itself, if it's every record then close monitoring of the driver loop packets for one or two archive intervals should either prove or dispel this as the problem. If it's a 'once in a while' problem then this is harder to prove one way or another.

2. The logger in your console (that generates the archive record and sends it to the driver) is for some unknown reason adding leafTemp1 data to the archive record. WeeWX then processes the archive record (but doesn't add leafTemp1 - it's already there) and saves it to database. This is consistent with not seeing leafTemp1 in any loop packets. Possible I guess, seems an unusual failure though. Might benefit from clearing the logger using wee_device --clear-memory to clear the logger memory. There is no way of accessing the archive record as emitted by the driver without patching the either the driver code or WeeWX code.

3. Some other service is setting leafTemp1. Given your config this is pretty unlikely plus there are no other non-core WeeWX services being run that could be misbehaving.

What to do. First up I would set your leafTemp1 data in your database to None rather than 0.0. 0.0 is considered valid data, eg a temperature of 0 degrees is still valid. In WeeWX parlance, missing or no data is usually indicated by the python value None. To do this:

1. stop WeeWX

2. use an SQLite update query to set leafTemp1 to NULL:

$ sqlite3 /var/lib/weewx/weewx.sdb

sqlite> UPDATE archive SET leafTemp1=NULL WHERE leafTemp1 IS NOT NULL;

sqlite> .q

3. rebuild the daily summary tables:

$ wee_database --rebuild-daily

Then I would remove the [StdCalibrate] [[Corrections]] setting you have for leafTemp1.

Then run the driver directly for at least 10 minutes and verify that leafTemp1 does not appear in any loop packets. Post the output her if you wish. You can check the database if you like (but use LeafTemp1 IS NOT NULL rather than LeafTemp1 != 0.0 in your query) but there should be nothing in there.

Now run WeeWX directly, this will display loop packets (lines starting with LOOP:) and archive records (lines starting with REC:) on the console. Let WeeWX run for at least 10 minutes and see it leafTemp1 appears anywhere. If it appears does it appear in LOOP: or REC: lines or both? Post the output here. Again you can check your database if you wish

Gary

[Bill Richter]

I did almost all of what you suggested, but didn't run for 10 minutes. Rather than try to remember , what I did exactly, I'll follow your very detailed instructions.

weewx# sqlite3 /var/db/weewx.sdb
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.

sqlite> UPDATE archive SET leafTemp1=NULL WHERE leafTemp1 IS NOT NULL;
sqlite> .q

weewx#

weewx# wee_database --rebuild-daily
Using configuration file /etc/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
All daily summaries will be rebuilt.
Proceed (y/n)? y
Rebuilding daily summaries in database 'weewx.sdb'

#   This section can adjust data using calibration expressions.

[StdCalibrate]

    [[Corrections]]
        # For each type, an arbitrary calibration expression can be given.
        # It should be in the units defined in the StdConvert section.

##############################################################################

weewx# date
Wed Oct 13 20:16:55 PDT 2021
weewx# PYTHONPATH=/usr/share/weewx python3 -m weewx.drivers.vantage >/tmp/vantage.dump
^C
  KeyboardInterrupt
weewx# date
Wed Oct 13 20:31:09 PDT 2021

weewx# grep leafTemp1 /tmp/vantage.dump
weewx# wc -l /tmp/vantage.dump
424 /tmp/vantage.dump
weewx#

weewx# sqlite3 /var/db/weewx.sdb
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select leafTemp1 from archive where leafTemp1 IS NOT NULL;
sqlite> .q
weewx#

weewx# weewxd --config /etc/weewx/weewx.conf > /tmp/weewx.log
^C
KeyboardInterrupt

weewx#
weewx# date
Wed Oct 13 20:46:38 PDT 2021
weewx#

weewx# sqlite3 /var/db/weewx.sdb
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> select leafTemp1 from archive where leafTemp1 IS NOT NULL;
-26.0
-26.0
-26.0
-26.0
-26.0
-26.0
-26.0
sqlite> SELECT dateTime,datetime(dateTime, 'unixepoch','localtime'),leafTemp1 FROM archive WHERE leafTemp1 IS NOT NULL;
1634181300|2021-10-13 20:15:00|-26.0
1634181600|2021-10-13 20:20:00|-26.0
1634181900|2021-10-13 20:25:00|-26.0
1634182200|2021-10-13 20:30:00|-26.0
1634182500|2021-10-13 20:35:00|-26.0
1634182800|2021-10-13 20:40:00|-26.0
1634183100|2021-10-13 20:45:00|-26.0
sqlite>

weewx# grep leafTemp1 /tmp/vantage.dump
weewx# wc -l /tmp/vantage.dump
424 /tmp/vantage.dump

weewx#

weewx# wee_device --clear-memory
Using configuration file /etc/weewx/weewx.conf
Using Vantage driver version 3.2.3 (weewx.drivers.vantage)
Proceeding will erase all archive records.
Are you sure you wish to proceed (y/n)? y
Erasing all archive records ...
Archive records erased.

This confirms my earlier testing, it's not coming from  the console, something is touching the LOOP packet, and it's the same value all the time.  This is not a 'new" issue with 4.5.0.beta*.

SELECT dateTime,datetime(dateTime, 'unixepoch','localtime'),soilMoist4 FROM archive WHERE soilMoist4 IS NOT 0.0;

1487341800|2017-02-17 06:30:00|8.0

1487342100|2017-02-17 06:35:00|8.0
1487342400|2017-02-17 06:40:00|8.0
1487342700|2017-02-17 06:45:00|8.0
1487343000|2017-02-17 06:50:00|8.0

[Bill Richter]

I meant 4.6.0.beta*

So weewx has been running for a bit and that bad data is present.

leafTemp1       -26.0°F
leafTemp2       -82.0°F
leafWet1           64
leafWet2          8
soilTemp1       -26.0°F
soilTemp2       -82.0°F
soilTemp3       -58.0°F
soilTemp4       -90.0°F
soilMoist1      2 cb
soilMoist2      8 cb
soilMoist3      2 cb
soilMoist4      0 cb

[gjr80]

This confirms my earlier testing, it's not coming from  the console, something is touching the LOOP packet, and it's the same value all the time.

Well we don't know that for sure, certainly I've not seen a loop packet emitted directly by the driver/station that contains bogus data, but remember the driver also gets archive records from the station and it is possible that archive record from the station has the bogus data.

 

This is not a 'new" issue with 4.5.0.beta*

I meant 4.6.0.beta*

Agree, this has been there for a while, it is just the 4.6.0b3 changes to the Seasons skin have visually highlighted the problem.

 

sqlite> SELECT dateTime,datetime(dateTime, 'unixepoch','localtime'),leafTemp1 FROM archive WHERE leafTemp1 IS NOT NULL;
1634181300|2021-10-13 20:15:00|-26.0
1634181600|2021-10-13 20:20:00|-26.0
1634181900|2021-10-13 20:25:00|-26.0
1634182200|2021-10-13 20:30:00|-26.0
1634182500|2021-10-13 20:35:00|-26.0
1634182800|2021-10-13 20:40:00|-26.0
1634183100|2021-10-13 20:45:00|-26.0

So it seems fairly safe to say it occurs regularly. Can you try one more thing:

1. Stop WeeWX

2. Edit weewx.conf and set debug = 1, save weewx.conf

3. Run WeeWX directly and let WeeWX run for at least 10 minutes.

4. There will be a lot of output to the console; 10 minutes of loop packets and at least two archive records. Post the entire console output showing the 10 minutes of loop packets/archive records. If you console buffer is not big enough you can pipe to a file and post the file.

5. Take a WeeWX log extract showing the entire WeeWX startup and the 10 minutes WeeWX was running directly. Don't edit the log extract and make sure you get the full WeeWX start. Post the log extract back here.

Gary

[Bill Richter]

Here you go.

Oct 14 19:34:58 weewx weewx[20997] INFO __main__: Received signal TERM (15).
Oct 14 19:34:58 weewx weewx[20997] INFO weewx.engine: Main loop exiting. Shutting engine down.
Oct 14 19:34:58 weewx weewx[20997] INFO weewx.engine: Shutting down StdReport thread
Oct 14 19:34:58 weewx weewx[20997] INFO __main__: Terminating weewx version 4.6.0b3
Oct 14 19:34:59 weewx weewx[29686]: Stopping weewx weather system: weewx.
Oct 14 19:35:24 weewx weewx[29782] INFO __main__: Initializing weewx version 4.6.0b3
Oct 14 19:35:24 weewx weewx[29782] INFO __main__: Using Python 3.7.3 (default, Jan 22 2021, 20:04:44) #012[GCC 8.3.0]
Oct 14 19:35:24 weewx weewx[29782] INFO __main__: Platform Linux-5.10.63-v7+-armv7l-with-debian-10.11
Oct 14 19:35:24 weewx weewx[29782] INFO __main__: Locale is 'LC_CTYPE=en_US.UTF-8;LC_NUMERIC=en_US.UTF-8;LC_TIME=C.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=en_US.UTF-8;LC_ADDRESS=en_US.UTF-8;LC_TELEPHONE=en_US.UTF-8;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=en_US.UTF-8'
Oct 14 19:35:24 weewx weewx[29782] INFO __main__: Using configuration file /etc/weewx/weewx.conf
Oct 14 19:35:24 weewx weewx[29782] INFO __main__: Debug is 1
Oct 14 19:35:24 weewx weewx[29782] DEBUG __main__: Initializing engine
Oct 14 19:35:24 weewx weewx[29782] INFO weewx.engine: Loading station type Vantage (weewx.drivers.vantage)
Oct 14 19:35:24 weewx weewx[29782] DEBUG weewx.drivers.vantage: Driver version is 3.2.3
Oct 14 19:35:24 weewx weewx[29782] DEBUG weewx.drivers.vantage: Option loop_request=1
Oct 14 19:35:24 weewx weewx[29782] DEBUG weewx.drivers.vantage: Opened up serial port /dev/ttyUSB0; baud 19200; timeout 5.00
Oct 14 19:35:24 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:35:24 weewx weewx[29782] DEBUG weewx.drivers.vantage: Hardware type is 17
Oct 14 19:35:24 weewx weewx[29782] DEBUG weewx.drivers.vantage: ISS ID is 1
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.drivers.vantage: Hardware name: Vantage Vue
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdTimeSynch
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdTimeSynch
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdConvert
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.engine: StdConvert target unit is 0x1
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdConvert
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdCalibrate
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdCalibrate
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdQC
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdQC
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.wxservices.StdWXCalculate
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.wxservices.StdWXCalculate
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdWXXTypes
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdWXXTypes
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdPressureCooker
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdPressureCooker
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdRainRater
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdRainRater
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdDelta
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdDelta
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdArchive
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.engine: Archive will use data binding wx_binding
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.engine: Record generation will be attempted in 'hardware'
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.engine: Using archive interval of 300 seconds (specified by hardware)
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Use LOOP data in hi/low calculations: 1
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdArchive
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.restx.StdStationRegistry
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.restx: StationRegistry: Station will be registered.
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.restx.StdStationRegistry
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.restx.StdWunderground
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.restx: WU essentials: {}
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.restx: Wunderground-PWS: Data for station KCAGEORG10 will be posted
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.restx.StdWunderground
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.restx.StdPWSweather
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.restx: PWSWeather: Data for station KCAGEORG10 will be posted
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.restx.StdPWSweather
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.restx.StdCWOP
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.restx: CWOP: Data for station EW6476 will be posted
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.restx.StdCWOP
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service user.windy.Windy
Oct 14 19:35:25 weewx weewx[29782] INFO user.windy: version is 0.7
Oct 14 19:35:25 weewx weewx[29782] INFO user.windy: Data will be uploaded to https://stations.windy.com/pws/update
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service user.windy.Windy
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdPrint
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdPrint
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Loading service weewx.engine.StdReport
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.engine: Finished loading service weewx.engine.StdReport
Oct 14 19:35:25 weewx weewx[29782] INFO __main__: Starting up weewx version 4.6.0b3
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.engine: Clock error is 1.49 seconds (positive is fast)
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx.sdb'
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.manager: Starting backfill of daily summaries
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:35:25 weewx weewx[29782] INFO weewx.manager: Daily summaries up to date
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.drivers.vantage: Getting archive packets since 2021-10-14 19:30:00 PDT (1634265000)
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.drivers.vantage: Retrieving 1 page(s); starting index= 2
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:35:25 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:35:26 weewx weewx[29782] DEBUG weewx.drivers.vantage: Empty record page 0; index 3
Oct 14 19:35:26 weewx weewx[29782] INFO weewx.engine: Starting main packet loop.
Oct 14 19:35:26 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:35:26 weewx weewx[29782] DEBUG weewx.drivers.vantage: Requesting 200 LOOP packets.
Oct 14 19:35:26 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:35:26 weewx weewx[29782] DEBUG weewx.restx: CWOP: Connected to server cwop.aprs.net:14580
Oct 14 19:40:15 weewx weewx[29782] DEBUG weewx.drivers.vantage: Getting archive packets since 2021-10-14 19:35:00 PDT (1634265300)
Oct 14 19:40:15 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:40:15 weewx weewx[29782] DEBUG weewx.drivers.vantage: Retrieving 1 page(s); starting index= 3
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.drivers.vantage: Empty record page 0; index 4
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.restx: StationRegistry: wait interval (300 < 604800) has not passed for record 2021-10-14 19:40:00 PDT (1634265600)
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.restx: CWOP: wait interval (300 < 600) has not passed for record 2021-10-14 19:40:00 PDT (1634265600)
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.reportengine: Running reports for latest time in the database.
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.drivers.vantage: Requesting 200 LOOP packets.
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.reportengine: Running report 'SeasonsReport'
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:40:16 weewx weewx[29782] DEBUG weewx.reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Oct 14 19:40:17 weewx weewx[29782] DEBUG weewx.cheetahgenerator: Using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'weewx.cheetahgenerator.JSONHelpers', 'weewx.cheetahgenerator.Gettext']
Oct 14 19:40:17 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:40:31 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:41:06 weewx weewx[29782] DEBUG weewx.reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Oct 14 19:41:06 weewx weewx[29782] DEBUG weewx.reportengine: Report 'MobileReport' not enabled. Skipping.
Oct 14 19:41:06 weewx weewx[29782] DEBUG weewx.reportengine: Report 'StandardReport' not enabled. Skipping.
Oct 14 19:41:06 weewx weewx[29782] DEBUG weewx.reportengine: Report 'FTP' not enabled. Skipping.
Oct 14 19:41:06 weewx weewx[29782] DEBUG weewx.reportengine: Report 'RSYNC' not enabled. Skipping.
Oct 14 19:45:01 weewx CRON[31005]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 14 19:45:15 weewx weewx[29782] DEBUG weewx.drivers.vantage: Getting archive packets since 2021-10-14 19:40:00 PDT (1634265600)
Oct 14 19:45:15 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:45:15 weewx weewx[29782] DEBUG weewx.drivers.vantage: Retrieving 1 page(s); starting index= 4
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.restx: StationRegistry: wait interval (600 < 604800) has not passed for record 2021-10-14 19:45:00 PDT (1634265900)
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.reportengine: Running reports for latest time in the database.
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.reportengine: Running report 'SeasonsReport'
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.drivers.vantage: Requesting 200 LOOP packets.
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.drivers.vantage: Gentle wake up of console successful
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'SeasonsReport'
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.restx: CWOP: Connected to server cwop.aprs.net:14580
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.cheetahgenerator: Using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras', 'weewx.cheetahgenerator.JSONHelpers', 'weewx.cheetahgenerator.Gettext']
Oct 14 19:45:16 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:45:25 weewx weewx[29782] DEBUG weewx.manager: Daily summary version is 4.0
Oct 14 19:45:59 weewx weewx[29782] DEBUG weewx.reportengine: Report 'SmartphoneReport' not enabled. Skipping.
Oct 14 19:45:59 weewx weewx[29782] DEBUG weewx.reportengine: Report 'MobileReport' not enabled. Skipping.
Oct 14 19:45:59 weewx weewx[29782] DEBUG weewx.reportengine: Report 'StandardReport' not enabled. Skipping.
Oct 14 19:45:59 weewx weewx[29782] DEBUG weewx.reportengine: Report 'FTP' not enabled. Skipping.
Oct 14 19:45:59 weewx weewx[29782] DEBUG weewx.reportengine: Report 'RSYNC' not enabled. Skipping.
Oct 14 19:47:04 weewx weewx[29782] INFO weewx.engine: Main loop exiting. Shutting engine down.
Oct 14 19:47:04 weewx weewx[29782] INFO weewx.engine: Shutting down StdReport thread
Oct 14 19:47:04 weewx weewx[29782] DEBUG weewx.engine: StdReport thread has been terminated
Oct 14 19:47:04 weewx weewx[29782] DEBUG weewx.restx: Shut down Windy thread.
Oct 14 19:47:04 weewx weewx[29782] DEBUG weewx.restx: Shut down CWOP thread.
Oct 14 19:47:04 weewx weewx[29782] DEBUG weewx.restx: Shut down PWSWeather thread.
Oct 14 19:47:04 weewx weewx[29782] DEBUG weewx.restx: Shut down Wunderground-PWS thread.
Oct 14 19:47:04 weewx weewx[29782] DEBUG weewx.restx: Shut down StationRegistry thread.
Oct 14 19:47:04 weewx weewx[29782] CRITICAL __main__: Keyboard interrupt.

[gjr80]

Thanks, that seems pretty conclusive. There are two archive records in the console output you attached that contain leafTemp1 and friends, we have all of the loop packets for each of those archive intervals and none of the loop packets contain leafTemp1 or friends. This is consistent with the loop packets we were seeing direct from the driver yesterday; none of them contained leafTemp1 or friends either. The implication is that leafTemp1 is not appearing through the loop packets. So it must be coming in through the hardware archive record the driver obtains from the logger or some other code must be adding it. Within the 4.6.0b3 code base leafTempx only appears in the vantage driver, weewx/drivers/vantage.py. The only non-core WeeWX code you are running is the Windy uploader, the Windy uploader has no concept of leafTemp so it is not the source. One of the WeeWX services such as StdWXCalculate could be calculating leafTemp1, but if that was the case leafTemp1 would need to appear in your weewx.conf; it does not.

So we are left with the hardware archive record. I strongly suspect that when interrogated for a hardware archive record your logger is returning data in a number of fields for non-existent sensors, this suggests the logger is faulty. The driver is then decoding this data and leafTemp1 and friends appear in the archive record. Interesting, we have only seen two archive records but each field seems to always have the same value. I've not heard of such a fault before, but I am no Davis expert, perhaps when Tom is back he may have something to add. Actually, I do seem to remember a Davis user some time ago getting bogus UV or radiation values when they did not have either UV or solar sensors. Cannot remember any further details though.

So what to do. There are a couple of things you can do within WeeWX. First you can force all of the bogus data fields to None using [StdCalibrate] [[Corrections]] in weewx.conf, eg:

[StdCalibrate]

    [[Corrections]]

        leafTemp1 = None

        leafTemp2 = None

        etc

This will be applied to both loop packets and archive records before they are saved to database. Setting the field to None (rather than 0.0) effectively sees WeeWX ignore them and nothing will be written to database in those fields.

Second, you could change record generation from hardware to software. With hardware generation the driver obtains the archive record from the hardware, with software record generation WeeWX synthesises an archive record from the accumulated loop data received during an archive interval. You are presently using hardware record generation so you would change to software record generation. To do this, in weewx.conf under [StdArchive] set record_generation = software and restart WeeWX. There are no disadvantages to software record generation that I am aware of, it was the case that you lost the ability to backfill archive records if WeeWX is off line for some reason, but that has not been the case for some time now. If your station does backfill if WeeWX goes offline hardware archive records will be obtained from the logger which could contain the bogus fields again. So I would make sure the corrections mentioned previously are in place.

Non-WeeWX options wise you could replace the logger, though it's only supposition on my part that the logger is the problem. Perhaps an expensive test for perhaps little gain. You used wee_device --clear-memory to clear the logger memory, you could take that a step further and remove the logger from the console, remove all power from the console (including batteries), wait for ten minutes then re-seat the logger and re-power the console, do another wee_device --clear-memory and see if it comes good - this is real clutching at straws stuff though.

Finally, whatever solution you do you once you have implemented it you are going to have to clear the bogus data from your archive again using an SQLite update query on each field concerned (setting each to NULL) and then you will need to rebuild the daily summaries again.

Gary

[Bill Richter]

Thank you so much for your exhaustive and informative analysis. I like using StdCalibrate/corrections method. I'll set that up now and clear the database etc. 

I think one final suggestion would be to make an update to the vantage driver documenation, suggesting the same fix for sensors not enabled.

I do have an SDR USB device, I was thinking of reading data directly from the ISS, which is the third viable option. I've not been able to get 'real time' updates working with any of the current projects and I won't migrate to Cumulus (no Windows apps fan).

Standby for a update after I make those changes to weewx.conf.

[Bill Richter]

And another option, replace the ISS.  But the SDR method might be some interesting fun.  I have a lot of money invested in the Davis console and their premium logger accessory.;

I followed your recommendation. Corrections set to none for those values that are bogus.

I'm puzzled why this is happening, would like to trace it down and fix it, but I also don't want to know that much about python or the internals of weewx.  I can fiddle with this given some direction though.

So, the fix works. I'd be wondering about Tom's thoughts about the final solution.  But I think an update to the Vantage driver might be helpful to people as 4.6.0 and the Seasons skins rolls out.  A short, here is the weewx.conf update, and how to clear the database and rebuild the daily summaries.  Which did require deleting and regenerating the summaries.

[gjr80]

On Saturday, 16 October 2021 at 07:07:31 UTC+10 bill.g...@gmail.com wrote:

And another option, replace the ISS. 

Yes, but given that ISS data is clearly getting to the console/logger (the loop packets are fine) and since the logger is solely responsible for constructing the hardware archive record I would see replacing the ISS as a long shot.

 

But the SDR method might be some interesting fun. 

Remember the two issues with using SDR are (1) as the pressure sensor is in the console you will lose all pressure information (and any WeeWX derived obs that utilise pressure) unless you provide pressure data separately and (2) you lose the ability to catchup if WeeWX is down for any period of time. The other complicating factor with using a SDR with a Vantage ISS is that you have to contend with with the frequency hopping used between ISS and console, still doable but not with the standard WeeWX SDR driver.

I'm puzzled why this is happening, would like to trace it down and fix it, but I also don't want to know that much about python or the internals of weewx.  I can fiddle with this given some direction though.

So, the fix works. I'd be wondering about Tom's thoughts about the final solution.  But I think an update to the Vantage driver might be helpful to people as 4.6.0 and the Seasons skins rolls out.  A short, here is the weewx.conf update, and how to clear the database and rebuild the daily summaries.  Which did require deleting and regenerating the summaries.

I'm not sure there is a final solution short of replacing the faulty hardware. I don't see modification on the vantage driver being required, it is doing its job and decoding the data provided by the console/logger. There is no way for the driver to distinguish between good data (ie from a station that has valid leafTempx data) and bogus data. In the main the driver cannot tell what sensors exist/do not exist other than by the data included in the packet/record. Arguably you could look at what is (or is not in the loop packet) and if something is in the archive record but not in the loop packet you discard it but this will only cover a limited suite of fields (the loop packets are pretty sparse in some regards) and this adds a level of complexity that addresses a very rare corner case that I think Tom will say is not warranted.

 

Gary

[Bill Richter]

Yes, I think  I'd replace the Davis completely, the next thing will go bad is the super cap in the ISS so another decision time.  I already have a spare Accurite on the shelf.  All good points about the SDR implementation.

I'm not sure about faulty hardware. I don't recall every seeing the values in the loop packet direct from the driver.  Anyway another simple cause would be not initalizing the values to NULL and then using random values from memory.

But still I think the easy fix is simply use the Corrections option and a note in the Vantage driver for 4.6.X and Seasons skin.  I didn't try any other skins.

I also think Tom has a Vantage (not sure if it's a Vue) so he might have seen it already.  But certainly as 4.6.X rolls out to weewx Vantage users, we'll have been there first!.

[Tom Keffer]

What triggered all this is that V4.6 figures out what to display on the basis of what's in the database. It's quite possible lots of Vantage stations are emitting bogus data for missing sensors, but no one ever bothered to plot it. I'll have to check my own database when I get home.

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6c393968-99d3-4e54-880f-64e1ce80721dn%40googlegroups.com.

[Vince Skahan]

On Friday, October 15, 2021 at 7:46:37 PM UTC-7 Tom Keffer wrote:

What triggered all this is that V4.6 figures out what to display on the basis of what's in the database. It's quite possible lots of Vantage stations are emitting bogus data for missing sensors, but no one ever bothered to plot it. I'll have to check my own database when I get home.

Here's mine - wow is there ever some bogus data in there, but I've done things like run weewx-wd for a while (then turned it off) and had a Lacrosse station before switching to a VP2 ages ago, but I see thousands of bogus measurements over the years when I had no soil/leaf/solar sensors ever.

I can rerun this limiting it to the last year if needed.

=========================================

My db field counts sorted by record count

=========================================

dateTime 1437009

barometer 1436996

outTemp 1434802

dewpoint 1434574

outHumidity 1434479

windGust 1423973

windSpeed 1421912

ET 1342089

inHumidity 1342089

inTemp 1342089

rain 1342089

rainRate 1342089

heatindex 1339861

windchill 1339859

rxCheckPercent 1318752

txBatteryStatus 1154921

consBatteryVoltage 1154717

windGustDir 1112577

pressure 1069647

altimeter 1051853

windDir 962745

=======================

(elements with lesser counts)

I never had leaf or soil sensors on this station

=======================

inDewpoint 170470

maxSolarRad 164396

windrun 164394

cloudbase 164388

humidex 164388

appTemp 164386

extraTemp1 98776

extraTemp2 98776

soilMoist1 88430

soilMoist2 88430

soilMoist3 88430

soilTemp1 65396

soilTemp2 65396

soilTemp3 65396

extraTemp3 3638

extraHumid1 2231

extraHumid2 2231

hail 2231

hailRate 2231

heatingTemp 2231

heatingVoltage 2231

inTempBatteryStatus 2231

leafTemp1 2231

leafTemp2 2231

leafWet1 2231

leafWet2 2231

outTempBatteryStatus 2231

rainBatteryStatus 2231

referenceVoltage 2231

soilMoist4 2231

soilTemp4 2231

supplyVoltage 2231

windBatteryStatus 2231

I did this by making a file called 'elements' from the schema of the archive table with the element names one-per-line, then generated the output by:

    for e in `cat elements`; do echo -n "$e "; echo "select COUNT(*) from archive where ${e} is not NULL" | sqlite3 weewx.sdb ;  done | grep -v " 0$" | sort +1nr > counts

[pa...@pauland.net]

Hi Bill,

I've been following the mystery of where the soil and leaf data is coming from. Gary and Tom have given you excellent help in trying to track down the source. I'm hesitant to mention this , because I don't want to confuse the issue, however if you have done some customization in the past it's possible that you have added something to your extensions.py file that's now having undesired effect . Might be worth checking the contents depending on your WeeWX installation type:

/home/weewx/bin/user/extensions.py

or

/usr/share/weewx/user/extensions.py

Thanks,

Paul 

[Tom Keffer]

I’ll have to check my database when I get home, but if this is widespread, we may have to rethink our strategy of checking for data before generating a plot.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6bf017e0-c84c-4185-97bb-23cebbf3062an%40googlegroups.com.

--

-tk

[Bill Richter]

No, the only customization was to add the windy extension and the fix for the battery status. I did try the real time steel skins but that was removed.  I also tried the envoy extension, I see that is still there, I'll remove it and recheck.

Removing the un configured, installed envoy extension made no difference, the fake data is still generated.

Part of me wants to keep looking, the other part of me says, the Corrections fix works why look further.

[Vince Skahan]

Tom - I added a check for the last 12 months and it looks good here for me.  No extra fields in my VP2 db in the last year.

# this checks a year ago as of this writing

for e in `cat elements`; do echo -n "$e "; echo "select COUNT(*) from archive where ${e} is not NULL and (dateTime BETWEEN 1602864397 AND datetime('now', 'localtime') )" | sqlite3 weewx.sdb ;  done | grep -v " 0$" | sort +1nr

[Joel Bion]

Here's the output from my system (VP2) using a script similar in spirit to that provided by Vince. It's odd that some field's i'd expect, like wind direction, appear to not show up about half the time, but that is probably on my end. But it doesn't appear I am getting any spurious leaf or such readings.

dateTime           930586
ET                 930586
rain               930586
rainRate           930586
rxCheckPercent     930586
usUnits            930586
windGust           930586
barometer          930582
inHumidity         930582
inTemp             930582
windSpeed          929949
outTemp            929879
radiation          929878
UV                 929876
outHumidity        929874
dewpoint           929850
heatindex          929850
windchill          929231
altimeter          889512
pressure           889512
consBatteryVoltage 764664
txBatteryStatus    764664
windGustDir        609688
windDir            474362
extraHumid1             0
extraHumid2             0
extraTemp1              0
extraTemp2              0
extraTemp3              0
hail                    0
hailRate                0
heatingTemp             0
heatingVoltage          0
inTempBatteryStatus     0
leafTemp1               0
leafTemp2               0
leafWet1                0

leafWet2                0

outTempBatteryStatus    0
rainBatteryStatus       0
referenceVoltage        0
soilMoist1              0
soilMoist2              0
soilMoist3              0
soilMoist4              0
soilTemp1               0
soilTemp2               0
soilTemp3               0
soilTemp4               0
supplyVoltage           0
windBatteryStatus       0

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1842d45c-8104-4205-8d6c-f20aaa27712bn%40googlegroups.com.

[Vince Skahan]

I think the missing wind direction readings is weewx not reporting that item when the wind speed is zero.

[Andy]

 

Battery status displayed as numbers

Clean setup.py install of WeeWX version 4.6.0b3

Battery Status

Outside Temperature Battery 1.000000

Inside Temperature Battery 0.000000

Wind Battery 0.000000

[Andy]

No rain plot

[Tom Keffer]

The battery indicator problem was fixed in commit 378cd18

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/143c2a72-5999-4adb-9f38-cdc7d7d1eaa2n%40googlegroups.com.

[Tom Keffer]

Not sure about this 'rain' problem, but there have been so many changes in this area since beta-3, that I'm hesitant to draw much of a conclusion.

If you're comfortable with git, you could clone the repository and run directly from the development branch. If not, no worries. I'm planning on sending b-6 out real soon now.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/ad6b3976-2560-4c5e-b0f5-8a95f8717c87n%40googlegroups.com.

[Andy]

Yes, that fixed the battery indicator. Running the master branch gets a rain plot, with development no rain plot. There is no rain in the new database since it has not rained today.

The rtl_433 is a bit of a moving target, but looks to be working. I will create some rain tomorrow.

out:['{"time" : "2021-11-03 02:33:04", "model" : "Acurite-5n1", "message_type" : 49, "id" : 2662, "channel" : "A", "sequence_num" : 0, "battery_ok" : 1, "wind_avg_km_h" : 0.000, "wind_dir_deg" : 180.000, "rain_in" : 24.820, "mic" : "CHECKSUM"}\n', '{"time" : "2021-11-03 02:33:04", "model" : "Acurite-5n1", "message_type" : 49, "id" : 2662, "channel" : "A", "sequence_num" : 1, "battery_ok" : 1, "wind_avg_km_h" : 0.000, "wind_dir_deg" : 180.000, "rain_in" : 24.820, "mic" : "CHECKSUM"}\n',

{'dateTime': 1635906784, 'usUnits': 1, 'protocol.0A66.Acurite5n1PacketV2': None, 'model.0A66.Acurite5n1PacketV2': 'Acurite-5n1', 'channel.0A66.Acurite5n1PacketV2': 'A', 'sequence_num.0A66.Acurite5n1PacketV2': 0, 'battery.0A66.Acurite5n1PacketV2': 0, 'mod.0A66.Acurite5n1PacketV2': None, 'freq.0A66.Acurite5n1PacketV2': None, 'rssi.0A66.Acurite5n1PacketV2': None, 'snr.0A66.Acurite5n1PacketV2': None, 'noise.0A66.Acurite5n1PacketV2': None, 'msg_type.0A66.Acurite5n1PacketV2': 49, 'wind_speed.0A66.Acurite5n1PacketV2': 0.0, 'wind_dir.0A66.Acurite5n1PacketV2': 180.0, 'rain_total.0A66.Acurite5n1PacketV2': 24.82}

Nov  2 19:46:22 debian weewx[14328] DEBUG user.sdr: packet={'windDir': 202.5, 'windSpeed': 0.0, 'rain_total': 24.82, 'windBatteryStatus': 0, 'dateTime': 1635907576, 'usUnits': 1}

rain_total = rain_total.0A66.Acurite5n1PacketV2

[Tom Keffer]

The driver should emit 'rain' in the LOOP packet. It looks like yours is just emitting 'rain_total'. The SDR driver should be calculating 'rain' from 'rain_total'. See the discussion on genLoopPackets in the Customizing Guide. 

Not sure what the difference would be between master and development. 

1. Are you using the same configuration file (weewx.conf) for both?

2. Which version of the skin is development using? The new version? 

3. Set debug=1, then rerun, then check the log. It may be trying to tell you something.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/9ce16043-d06f-4215-ad5f-5b43ce9fc671n%40googlegroups.com.