Max results - where there they come from?

[kac...@gmail.com]

I had some problems with connections and also physical shocks on sensors. 
So, I stopped the Weewx service, copy the file /var/lib/weewx/weewx.sdb to my PC. I opened it with SQLiteDatabaseBrowser and change some extreme (unreal) results. For example wind and also rainRate which was caused by the son's ball :-)
After that, I copy the file back onto my server and start the Weewx service again. 
Now some changes are properly shown (for example in graphs), but the max results are old (unreal). I don't understand how this is calculated. I even tried "xstats" and it also shows old (unreal) max results. I checked again in the database and there are no old results there. 

Any idea how to get rid of "unreal" results which should not apear in the database (and AFAIK they are not there anymore)?

Here is the monthly graph where (for example) where we can see wind (VETER) max on the 18th and 19th of July. It was 65 km/h:
https://izo.amebis.si/belchertown/graphs/?graph=month

Here is a summary where the max wind for July 2023 is 50 km/h:
https://izo.amebis.si/belchertown/

It is similar to the rain rate. Max is 13107.0 mm/hr (unreal and I can't find this result in the database (I sorted the rain rate column, tried to check for the particular time ... there is no such data anymore). 

Regards.

[michael.k...@gmx.at]

The values are in the "archive_day_*" tables in the database. These tables contain a dateTime values,  min and max values as well as their exact time (depending on the source), sum and weighted sum values and a count. When you change a value in the archive table by hand, it won't show up there, thus not show up in any statistics in the front end, as long as you don't fix this in the proper way. 

See https://weewx.com/docs/latest/utilities.htm#wee_database_utility for a description of a tool that does it the proper way. Be aware, that some actions might lead to loss in precision. For instance, the "exact time" I mentioned before is then changed to the dateTime of the archive value, e.g. when you maximum gust occurred at 8:33pm using a 5 min archive_interval, it might happen, that after running the tool you maximum gust will be shown at 8:35pm. Or, if your maximum temp was at 4:32pm showing 30,1°C with a single loop value of 30,1°C, and all other loop values were 30,0°C in that archive_interval, after running the tool, it might be, that your maximum will read as 30,0°C at 1:15pm, if that was the first interval that day, that had 30,0°C.

[kac...@gmail.com]

How I hate Google Groups. It looks like I answer only to you Michael. 
To be short - your explanation helps me a lot. Thank you. 

I have one more question. Here is how I do it now when I want to change something manually (for example the rain rate - which is quite often a problem). 

1. Stop service on Raspberry PI. 
2. Copy WEEWX.SBD to my PC and open it with DB Browser for SQLite. 
3. Using the website https://www.epochconverter.com/ find the approximate dateTime to find the position in the database and change the number in the table "archive". 
4. After your explanation from yesterday change the number in the table "archive_day_rainRate". 
5. Write the changes to the file and copy it back to Raspberry PI. 
6. Start the Weewx service. 

Is there any better way?

[michael.k...@gmx.at]

1. Stop service on Raspberry PI. 

2. Copy WEEWX.SBD to your PC and open it with DB Browser for SQLite. 

3. Using the website https://www.epochconverter.com/ find the approximate dateTime to find the position in the database and change the number in the table "archive". 

4. Adjust the maximum value and time in the table "archive_day_xx". 

5. Write the changes to the file and copy it back to Raspberry PI.

6. Run sudo wee_database --reweight --date=YYYY-MM-DD (this affects only the particular day you changed values for)
7. Start the Weewx service.

The "reweight" option doesn't touch the min/max values and their timestamp, you took care of that yourself in (4.)

[kac...@gmail.com]

Thank you for your time. Everything is clear now. 

[gjr80]

Just a few points here. I'm sure it's just a typo that has perpetuated itself through the thread, but the default WeeWX database is weewx.sdb not weewx.sbd. Also, case matters here.

The recommended approach for correcting spurious data in the database is to locate and fix the spurious data in the archive table and then rebuild the daily summaries for the period concerned using wee_database. This procedure is covered in the Cleaning up old 'bad' data wiki page (note this page details the use of sqlite3 to manipulate the archive, these steps can equally be completed using other tools such as DB Browser for SQLite etc, also the page currently does not mention the use of --from, --to or --date to limit the period of any daily summary rebuilds). This may give some loss of max/min values and times but any such loss will be minor.

Fixing any spurious data in the daily summary tables by direct editing is also possible, but it will likely be time consuming and is prone to error and hence not recommended. Consider if there is one anomalous outTemp value in a day, first you need to edit and remove the spurious data in the archive. Then you would need to locate the day concerned in the outTemp daily summary table and edit the max and maxtime fields replacing the data with the new max outTemp and the time it occurred for that day obtained by looking through that days temperature values in the archive, perhaps looking through hundreds of records. If there are other days with spurious data you would need to repeat this process for each day. Likewise for other obs (remember some obs are derived from others so an anomaly in outTemp may cause anomalies in heatindex, windchill, dewpoint etc rain anomalies may cause anomalies in rainRate). Thus just a few days of anomalous data can result in many time consuming changes being required.

Also, the --reweight option was implemented to fix an issue in earlier WeeWX versions (post v3.0.0) where archive records in databases with multiple different archive intervals were not correctly weighted in the daily summary tables. This issue was fixed in v4.3.0 and providing you are running v4.3.0 or later the reweighing of the database should only be needed as a one-off 'patch' which should have been performed automatically on startup of v 4.3.0 or later, all archive records subsequently added/modified etc by WeeWX or wee_database are automatically correctly weighted. Bottom line is users should not really need to use the --reweight option. Note the --reweight option does not perform the same function as --rebuild-daily.

Gary

[michael.k...@gmx.at]

>  but any such loss will be minor.

But often unnecessary and easily avoidable.

> Bottom line is users should not really need to use the --reweight option

I use the rebuild option when I have bad values for a single obs_type and don't want to lose any precision on min/max values and their dateTime, especially for other obs_types, when fixing them. I just have to ensure that I take care of the to-be-fixed abs_type min/max values correctly. I don't know any better way for a scenarios like I just described. I assume this doesn't apply on the user's scenario since he is missing entire archive_intervals, but anyway. 

As long as there is no option for wee_database, that manipulates min/max values and their dateTime only when it is necessary, I will not --rebuild-daily in such a scenario.

My bottom line: if you have power outages on a regular basis or even more than once the one or the other year, fiddling around correcting your database is not a good approach anyway. You should consider using a uninterruptible power supply (UPS) for the device weewx is running on (and, depending on the hardware and network setup/configuration, for the hardware, DHCP device and switch/access point) and hardware/driver that supports backfilling to the weewx database.

[kac...@gmail.com]

@gjr80 and @michael
Thank you both.