Wrong monthly average temperature

[Berny Cl]

Hi everybody,

since the last update to version 4.20, i have noticed an incorrect value for the monthly average temperature at the history table and also in the monthly NOAA table.

I use the niculskin and my station is a FineOffset (WS 1080).

See at: http://haus-volmering.de/history.html

(Durchschnittstemperatur = Average Temperature for Nov is obviously incorrect)

How can I fix that?

[Tom Keffer]

Most likely it's some bad data. Let's check the database directly. 

First find the database. If you did a package install, it's most likely at /var/lib/weewx/weewx.sdb. If you did a setup.py install, it's at /home/weewx/archive/weewx.sdb. Let's assume the former.

Then, run two queries:

sqlite /var/lib/weewx/weewx.sdb

sqlite> select avg(outTemp) from archive where strftime("%Y-%m", dateTime,'unixepoch','localtime')=='2020-11';

sqlite> select sum(sum)/sum(count) from archive_day_outTemp where strftime("%Y-%m",dateTime,'unixepoch','localtime')=='2020-11';

The first method calculates the average temperature for Nov 2020 by using the main archive table. The second by using the daily summaries. The two numbers should be very close. See what you get and we'll take it from there.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/277ec811-21b4-41e9-8fc1-cda38d87014dn%40googlegroups.com.

[Berny Cl]

sqlite> select avg(outTemp) from archive where strftime("%Y-%m", dateTime,'unixepoch','localtime')=='2020-11';

51.117818676717

sqlite> select sum(sum)/sum(count) from archive_day_outTemp where strftime("%Y-%m",dateTime,'unixepoch','localtime')=='2020-11';

51.114923603352

Thank you!

OK, I did that. The two numbers are very close. I think they are correct (in Fahrenheit) but in my history table the temperature ist too high (in degree Celsius).

[Tom Keffer]

1. That looks reasonable. One other query to try:

sqlite> select sum(wsum)/sum(sumtime) from archive_day_outTemp where strftime("%Y-%m",dateTime,'unixepoch','localtime')=='2020-11';

2. If that doesn't reveal anything, I will send you an instrumented version of xtypes.py that will log the calculation.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/87243ab4-47a4-4c85-b004-530f84e77673n%40googlegroups.com.

[Berny Cl]

sqlite> select sum(wsum)/sum(sumtime) from archive_day_outTemp where strftime("%Y-%m",dateTime,'unixepoch','localtime')=='2020-11';

60.1308595259353

Ok, that looks the same like the value in my history table.

[Tom Keffer]

So, for some reason, the weighted sum (field 'wsum') has too high a value, or the sum of observation time (field 'sumtime') has too low a value.

The easiest fix is to just rebuild the daily summaries using the wee_database utility.

Stop weewxd. then,

wee_database --drop-daily

wee_database --rebuild-daily

Restart weewxd

For a database of your size, it shouldn't take more than a minute or two.

It could take some time for the NOAA and html files to get corrected. You can speed things up by deleting them and allowing weewx to regenerate them.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f5a4768c-27f7-49c6-807e-496296d0c380n%40googlegroups.com.

[Berny Cl]

Yeah, everything looks great again.

Thank you Tom for that excellent support.

Greetings from Suedlohn (Germany).

Berny

[michael.k...@gmx.at]

I am observing the same situation, as well as other WeeWX users near me. The average is clearly off since the 4.2.0 update. It also affects yearly average since then. So I guess this is something that happened with the 4.2.0 Version. I don't want to rebuild my daily values, how to find the bad values and correct them? Probably in the archive_daily table of the day I made the update?

I found something: It's a change with "sum":

1-13 has "new" sumtime, 14 a mix (the day I updated) and old sumtime are 15 and and older:

Isn't there a config that sets how this is calculated?

[michael.k...@gmx.at]

1606431600 -2.9 1606461229 6.5 1606483508 90.7493145743143 287 90.7493145743143 287

1606345200 -1.4 1606376580 5.8 1606396159 208.754783549784 288 208.754783549784 288

1606258800 -2.5 1606290557 5.3 1606310759 71.2823196248196 288 71.2823196248196 288

1606172400 -2.1 1606194900 9.9 1606224600 752.777460317461 286 752.777460317461 286

1606086000 0.29997 1606171800 7.6 1606136074 951.861259018759 288 951.861259018759 288

1605999600 -0.1001 1606073204 10.2 1606050999 884.716731601731 288 884.716731601731 288

1605913200 -1.6 1605940146 7.7 1605966081 442.098484848485 288 442.098484848485 288

1605826800 0.29997 1605913105 6.2 1605872440 1254.14992424242 287 1254.14992424242 287

1605740400 4.7 1605740431 13.9 1605793485 2373.86111111111 288 2373.86111111111 288

1605654000 3.1 1605733623 16.0 1605706154 2105.63747835498 288 2105.63747835498 288

1605567600 4.8 1605652944 13.6 1605615932 2268.75462121212 288 2268.75462121212 288

1605481200 1.4 1605493171 8.9 1605535929 1811.52088383838 288 1811.52088383838 288

1605394800 2.1 1605473931 16.1 1605445486 2346.52735930736 288 2346.52735930736 288

1605308400 5.5 1605328978 17.3 1605360608 2590.47363636364 288 650228.309292929 69058

1605222000 4.0 1605238302 14.9 1605272440 2255.38090909091 288 676614.272727273 86400

1605135600 2.7 1605160230 12.0 1605186834 1942.27053030303 288 582681.159090909 86400

1605049200 3.5 1605078802 11.3 1605095884 1937.71213564214 285 581313.640692641 85500

1604962800 1.2 1604988775 9.4 1605013839 1337.86883477633 287 401360.650466329 86100

1604876400 -0.1001 1604903428 9.3 1604925619 1065.95282828283 288 319785.848484848 86400

So I guess if you just set sumtime = count and wsum = sum for all archive_day tables it should work?

[michael.k...@gmx.at]

Here is what I did and it worked for me :)

[gjr80]

I’m not sure I see the problem you are trying to fix, I don’t see anything ‘obviously off’. The ‘weighting’ of archive records was reworked in WeeWX v3.7.0 to properly support multiple different archive intervals in a database. Your data appears to have a constant five minute archive interval, and in such a case your wsum and sumtime values are simply the sum and count fields multiplied by your (constant) archive interval of 300 seconds. Your average as returned by .avg (wsum/sumtime) will be identical to sum/count. If you are seeing the same situation as the OP I doubt the change in wsum/sumtime is the issue.

Have you worked through the queries that Tom asked the OP to work through? Some solid numbers or other evidence of an error would help.

Gary

[michael.k...@gmx.at]

I guess the Problem is with the day of the version upgrade only. This day produces values that are off. since the yearly average 2020 and the monthly average 11/2020 is affected, both of these averages are wrong. 12/2020 should have worked.

11/01 til 11/13 worked also, so do 11/15 up until today 11/28. Only 11/14 is wrong. So I guess fixing only all 11/14 values in all tables would have fixed this for my particular case also. But hey, changing all the values to the way weewx produces them currently won't hurt, I hope :)

[gjr80]

Ok, I see the issue now.

Whilst changing wsum/sumtime worked in your case it only worked because you have a single constant archive interval in your database. Changing wsum/sumtime to sum/count in a database that has multiple different archive intervals would introduce errors. The correct fix (for all users) would have been to use wee_database with the  —update option as per the WeeWX 3.7.0 upgrade instructions when you upgraded to WeeWX v3.7.0 or beyond. 

Also, to clarify your last point, you haven’t actually changed the values ‘to the way WeeWX calculates them’. Since WeeWX 3.7.0 the way WeeWX calculates wsum/sumtime is to weight them by the archive interval, your values are unweighted.

Gary

[michael.k...@gmx.at]

I got the point, with the constant archive interval. But why did it change from weighted wsum and sumtime values to unweighted when upgrading 4.1.1 => 4.2.0? Did I unintentionally change a config portion when merging the configs? I guess many users aren't aware that this happened to them also. 3.7.0 was released in March 2017, my friend who is having the same issue started with 3.8.2 in Nov. 2018.

[Tom Keffer]

Michael,

Can you check what the version number is for your daily summaries?

select * from archive_day__metadata;

should do it.

-tk

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/73ded510-7efd-49b4-b2b5-00d6cf9ff806n%40googlegroups.com.

[michael.k...@gmx.at]

Version 2.0

[Tom Keffer]

Thanks.

Somehow, your daily summary shifted from V1.0 to V2.0 behavior starting with the 1605308400 timestamp. 

Gary: the only thing I can think of that changed in V4.2 is this commit, which was in response to this thread. Offhand, I can't see how the commit would cause Michael's symptoms.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ad81bb6f-896e-4236-ac9d-49ed688ad04dn%40googlegroups.com.

[Tom Keffer]

Michael, I have no idea what happened, but if you want to modify your daily summaries, it should be done right. If I understand you correctly, you patched them with SQL commands like

update archive_day_ET set wsum=sum, sumtime=count;

update archive_day_UV set wsum=sum, sumtime=count;
update archive_day_altimeter set wsum=sum, sumtime=count;

etc.

This will leave your summaries in an inconsistent state: the numbers are what would be expected for a Version 1 summary, yet your metadata will say Version 2. Furthermore, as new days are added, Version 2 numbers will be used, and you'll be back where you started: with half Version 1 and half Version 2. 

I would suggest making it totally Version 2. Do an update, except weight them with the archive interval. This will work if your archive interval is a constant 5 minutes (300 seconds):

1. Stop weewx.

2. Make a backup of weewx.sdb

3. Use updates that look like:

update archive_day_ET set wsum=sum*300, sumtime=count*300;

update archive_day_UV set wsum=sum*300, sumtime=count*300;
update archive_day_altimeter set wsum=sum*300, sumtime=count*300;

etc.

4. Restart weewx

On Sun, Nov 29, 2020 at 11:36 AM michael.k...@gmx.at <michael.k...@gmx.at> wrote:

https://imgur.com/a/yZUPoTq

michael.k...@gmx.at schrieb am Samstag, 28. November 2020 um 09:27:44 UTC+1:

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c720db5a-0470-43e8-9044-f73db6ccd307n%40googlegroups.com.

[Tom Keffer]

I think I found the problem. The version number was inadvertently being truncated from '2.0' to '2'. Later, the weighting factor is determined as:

        weight = 60.0 * record['interval'] if self.version >= '2.0' else 1.0

Because '2' collates ahead of '2.0', the weight was set to 1.0.

Fixed in commit 9510f38.

Michael, if you want to apply the fix to your copy, go into weewx/manager.py. Around line 827 change this

        v = self._read_metadata('Version')
        self.version = v[0] if v is not None else "1.0"

to this

        self.version = self._read_metadata('Version')
        if self.version is None:
            self.version = '1.0'

Thanks, Michael, for your diligence in drawing this to my attention.

I don't know why so many of your messages keep going to spam. I keep telling Google Groups to allow all messages from you to pass, but it doesn't always pay attention.

-tk

On Mon, Nov 30, 2020 at 11:06 AM michael.k...@gmx.at <michael.k...@gmx.at> wrote:

Hello again,

What happens now is: metadata says "Version 2.0", weewx is currently producing "Version 1.0" numbers (4.2.0). So I am afraid, when I follow your advice and and recalculate "Version 2.0" values, tomorrow I'll end up having mixed values again.

I checked a database snapshot BEFORE I updated t0 4.2.0:

Version 2.0

It didn't change during the update.

Here  are some entries from archive_day_outTemp AFTER the update to 4.2.0, but BEFORE I manually changed to Version 1.0 Nuumbers: (dateTime DESC)

1605481200 1.4 1605493171 8.9 1605535929 1811.52088383838 288 1811.52088383838 288

1605394800 2.1 1605473931 16.1 1605445486 2346.52735930736 288 2346.52735930736 288

1605308400 5.5 1605328978 17.3 1605360608 2590.47363636364 288 650228.309292929 69058

1605222000 4.0 1605238302 14.9 1605272440 2255.38090909091 288 676614.272727273 86400

1605135600 2.7 1605160230 12.0 1605186834 1942.27053030303 288 582681.159090909 86400

Weewx just "decided" to produce Version 1.0 Number after the update an mixing both oh the particular day.

Now something also very interesting, the following lines are from a friend's database, two days before and after the update: (dateTime DESC)

1604098800 5.1 1604184954 15.8 1604153156 3087.89794733045 288 3087.89794733045 288

1604012400 6.9 1604022829 12.6 1604059984 2970.48686868687 288 2970.48686868687 288

1603926000 5.6 1603926667 9.7 1603970504 2352.86239898999 288 235090.302070707 30188

1603839600 -1.2 1603859547 11.9 1603896779 1507.83905483406 288 452351.716450217 86400

1603753200 0.29997 1603839591 9.9 1603808400 1324.44158008658 288 397332.474025974 86400

The same happened, he ales has Version 2.0 in metadata, Version 2.0 numbers before the update, mixed on the day of the update and Version 1.0 after the update. But his average calculation still works.

So somehow something happened. @all Readers: can you check your database before and after the upgrade? Is anybody seeing the same behavior? We, my friend an me, have very similar installations, maybe the same quirksy config? Or maybe this is more widespread, but most of the users don't recognise the effects?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a4c40640-77f5-49b7-9fea-7668b19979c8n%40googlegroups.com.

[michael.k...@gmx.at]

Thank you again and again, especially the support you are providing! I'm more than happy, when my inputs lead to improvements.

I stopped weewx

I ran the 

update archive_day_XXX set wsum=sum*300, sumtime=count*300;

queries,

I edited manager.py

I started weewx again.

Log looks normal.

Most recent archive_day_outTemp line before:

1606690800 -3.8 1606716001 5.6 1606739368 66.9377561327561 246 20081.3268398268 73800

Most recent archive_day_outTemp line after two values were added to archive :

1606690800 -3.8 1606716001 5.6 1606739368 61.3091847041846 248 18392.7554112554 74400

I've done the math and everything looks correct to me :)