How to wipe out bad rain data

[vk3...@gmail.com]

During the time when I was developing a new driver, I managed to get some ridiculous rain values in the database - even here we don't get rain that is measured in kilometres!

I've tried to delete the very large 'rain' values form the database and also running the 'wee_database' tool to delete an recreate the daily totals but I'm still seeing very large values in any display/report that goes back beyond the current month.

Is there a way to wipe all of those bad values (happy to show that it didn't rain at all) without scrapping the whole database (only about 6 weeks so not a total loss if I have to) and starting again?

Susan

[gjr80]

Hi,

You will probably get a number of answers but essentially if you have bad data in your archive you need to fix (all of) that then drop/rebuild daily summaries or you will be forever chasing your tail. You can work out where the issue is (daily summaries or archive) by based on where the bad data is showing up (plots, month/year stats etc).  You will find some general info on how to deal with the issue in teh User's Guide under spikes in graphs (spikes or ridiculous data - same thing!) as well as in the wiki here.

Gary

[Bill Morrow]

Depending on your SQL skill, you should be able to clean them up directly in the slqlite database. (Syntax slightly different if you're using mysql.) Backup your database first!

I have a similar issue with outside temperature. For a few months my DIY station was indoors, so the outside temperature recorded was much too high.

    sqlite> select max(outTemp) from archive;
    119.984

Gives me something to aim for. In my location, it never gets above 100F.

    sqlite> update archive set outTemp=NULL where outTemp > 100.0;
    sqlite> select max(outTemp) from archive;
    99.9968

Or even 90F
    sqlite> select count(*) from archive where outTemp > 90;
    14321

You could also constrain by time. First look:
    select datetime(datetime,'unixepoch'), outTemp
    from archive
    where strftime('%m', datetime(datetime, 'unixepoch'))='02' and outTemp > 95;

and then null them out. For example
   update archive set outTemp = NULL
   where strftime('%m', datetime(datetime, 'unixepoch')) = '02'
   and outTemp > 95;
sets the outside temperature to NULL (blanks it) if it is over 95F in February.

If you know exactly when you got the driver working, you could NULL rain for all records before that time.

    update archive seet rain = NULL where datetime < 1456000000;

And then backfill

[vk3...@gmail.com]

Thanks - I managed to get that to work.

At first I tried just setting the values to 0 but I needed to set them to NULL (as per the instructions - I know!!!) before the next report would set the 'Month' displays to sensible values.

However the 'Year' summaries is still out. Is there an equivalent to '--drop-daily' and '--backfill-daily' that works for the yearly data?

Susan

[gjr80]

After 'fixing' your data (including the drop/rebuild) did you delete the NOAA format year summaries? If not then they will still show the old incorrect data, old NOAA format reports are not regenerated each report cycle, only the current month/year report is regenerated each cycle. If you did delete the reorts and they were regenerated with incorrect data then you still have incorrect data in your database. The NOAA format month and year reports pull their data from the daily summaries so no there is not '--drop-daily' and '--backfill-daily' specifically for yearly data.

If the NOAA format month or year reports are showing incorrect info there is incorrect data in your daily summaries, and since the daily summaries were recently dropped/backfilled from the archive there must still be incorrect data in your archive.

Afraid you are going to have to track down the remaining incorrect data in your archive table, then drop/rebuild the daily summaries and then delete the NOAA format reports. The offending NOAA format reports will give you an idea when the incorrect data occurred (within a month I expect for the year report). Since the values displayed in the report are aggregates it may take a bit of effort to track down the incorrect data.

Gary

[Andrew Milner]

Either - wait until tomorrow (I think it is generated once per day, at midnight) - or just delete the page (and/or image(s)) and it will be re-created.  The database data is correct - it only contains daily summaries, so there is no need for additional commands.

[vk3...@gmail.com]

Waiting until today did the trick for the graphics.

As gjr80  mentioned, the NOAA monthly report is still wrong but now I knwo how to sort that out as well.

Thanks everyone

Susan

[Andrew Milner]

What I usually do when removing/editing the archive is:

1. Stop weewx

2. Edit the archive

3. Drop daily

4. Delete ALL NOAA files

5. Delete ALL .png files

6. Delete ALL .html files

6. Backfill daily

7. Restart weewx

Probably a bit of overkill in that sequence - but it stops me getting confused and misled by old stale graphics etc ... and ensures that what I see is the result of the current state of the archive

[vk3...@gmail.com]

All sorted out now.

Thanks to all.

Susan