Faster way to recover from deleting latest records

[Susan Mackay]

Sometimes when we get a power failure, by WeeWx system recovers but the HP1000 console doesn't. This means that the WeeWx app carries on trying to access the console and also producing other reports, summary items etc., especially if the console is not recovered until after midnight. (At least this is how I interpret the situation.)

What this means is that there is a gap in the data in the data base: when access to the console is restored, WeeWx thinks that it records have been added even when they have not.

The solution is to delete all records after the last valid one before the power failure occurred but this also requires that all daily summary records be deleted (a la the various instructors such as at https://groups.google.com/g/weewx-user/c/ZXjuqhLz0UY). In y case I have 4 years worth of data in the database and therefore the '--rebuild-daily' takes forever.

Is there a faster way so that I can:

- delete the latest records after the gap

- do some magic step here that resets a date to say when the last valid record was 

- have WeeWx recover from there

(By the way, my HP1000 driver looks at the last entry in the database for the date from which to recover, even if the last record is 'junk' - while this may the the root cause of my problem, I don't know of a better way.)

Susan

[vince]

Run wee_database --help and check out the --from and --to and --date options.  They let you rebuild things for only a specified date or range of dates.  That should sped things up a lot.

       wee_database --rebuild-daily [--date=YYYY-mm-dd |
                                    [--from=YYYY-mm-dd] [--to=YYYY-mm-dd]]
                                    [--dry-run]

[gjr80]

Perhaps I'm not understanding the issue properly but if the console/driver is not emitting records then I would expect that WeeWX should not be adding any records to database (archive or daily summaries).

Also, not a solution to the underlying problem but if you only need rebuild part of your daily summaries (which it sounds is the case) then if you are using WeeWX v4.0 or later you should be able to use --date or --from/--to with wee_database to rebuild a single day only or a number of days.

Gary

[Susan Mackay]

I am using WeeWx V4 but my brain was stuck in V3-land - I didn't realise the --from and --to options existed.

Moral - always read the appropriate docuementation.

Thanks everyone.

Susan

[gjr80]

To be fair, it seems the addition of those new commands did not make it into the changelog, I'll take the wrap over the knuckles for that one.

Gary

[Susan Mackay]

Picking up on this again (due to another power failure at my house), I can see how the '--rebuild-daily --from' would work, but does this mean that you don't need to '--drop-daily' (which doesn't have such date-related options) first?

I other words, you can rebuild the daily records even if they already exist?

Susan

[gjr80]

On Tuesday, 29 March 2022 at 12:50:33 UTC+10 vk3...@gmail.com wrote:

Picking up on this again (due to another power failure at my house), I can see how the '--rebuild-daily --from' would work, but does this mean that you don't need to '--drop-daily' (which doesn't have such date-related options) first?

Short answer is 'usually'. You can certainly rebuild the daily summaries without first dropping them and that will solve most folks issues. There are some corner cases where dropping the daily summaries is required (for example some daily summary metadata issues are not fixed by a rebuild alone and a drop is required first) but such cases are rare. This has been the case for some time, I'm not sure if the 'dropping of the drop' occurred when the database structure changed in v3.0.0 or if it was later when we dealt with the weighting issue issue at issue #61. Am sure Tom will have the details.

 

I other words, you can rebuild the daily records even if they already exist?

Yes. The beauty of not dropping and just date/time limiting your rebuild is that you only lose the loop based max/min fidelity (if loop_hilo = True) of your daily summaries for those summaries that were rebuilt (rather than losing for all summaries if you drop).

Gary

[Susan Mackay]

Better and better - thanks

Susan