Deleting unwanted archive data

[Jason Rennie]

Tom and I had this exchange about how to delete bogus data from the weewx archive and stats.  One thing I noticed is that the week plots don't update immediately, but they were correct the next day.

Jason

---------- Forwarded message ----------
From: Jason Rennie <jre...@gmail.com>
Date: Tue, Nov 10, 2009 at 9:47 AM
Subject: Re: Deleting unwanted archive data
To: Thomas Keffer <kef...@threefools.org>


Worked like a charm!

Thanks,

Jason

On Mon, Nov 9, 2009 at 12:17 PM, Thomas Keffer <kef...@threefools.org> wrote:

Hi, Jason

 

OK, got it figured out. First, you'll have to download v1.0.1 to fix that nagging 'stats' bug. Find it at SourceForge: https://sourceforge.net/projects/weewx.

 

Stop weewx.

 

Install weewx as per usual. Double check the configuration file weewx.conf [it should transfer all your old settings, but...]. If you've made modifications to your templates, you might want to replace the stock 'templates' subdirectory with the version setup.py backed up.

 

Now use the tool 'sqlite3' to modify the archive database. Suppose you took the console outside on 7 Nov 2009 at 1pm. You want to delete all data before that time.

 

  prompt$ sqlite3 /home/weewx/archive/weewx.sdb [adjust for your path]

  sqlite> delete from archive where datetime(dateTime,'unixepoch','localtime')<'2009-11-07 13:00:00';

  sqlite> .exit

 

In the delete statement above, case matters. The first 'datetime' should be all lowercase (it's actually a SQL function), the second ('dateTime') in camelCase (it's actually a SQL type). The quotes around the date/time is also needed because of the spaces.

 

Now delete the stats archive. Restart weewx. It will rebuild the stats database (check /var/log/syslog to be sure). Your plots/stats should now reflect only data after the time you gave above.

 

Let me know how it goes.

 

-tk

 

-----Original Message-----
From: Jason Rennie [mailto:jre...@gmail.com]
Sent: Monday, November 09, 2009 7:02 AM
To: Thomas Keffer
Subject: Re: KMANATIC3

On Sun, Nov 8, 2009 at 11:49 AM, Thomas Keffer <tke...@gmail.com> wrote:

Re: the startup quirk. It's a bug I found just two days ago! If the stats file doesn't exist, weewx initializes it, but then forgets to read the SQL types back in. Restarting causes it to skip the initialization sequence, and just read the types. A one line fix.

Great!

 

I'm chugging away on V1.1. Biggest change is a refactoring to allow even easier customization by subclassing. I think I've figured out the Python idioms people use to allow customization and dynamic loading, but there are still a few glitches.

If I may make a suggestion, let me recommend that you be a bit wary of class inheritance in python.  I started coding python professionally a year ago after writing Java code for nearly two years and, looking back, a lot of the OOP idioms you might take from C++/Java just don't work particularly well in python.  I realize you may be past the point of designing the class/module structure and whatnot and/or you may have come up with good designs that fit python's variation of OOP, but I can tell you it may not be as easy as it looks.

Any plans to do a website?

I have a blog which I'm happy to publicize:

http://weather-station-quirks.blogspot.com/

I do publish the weewx page, but it's run of my home machine (the same one that runs weewx), so I just have it for my personal use and don't want to publicize it:

http://qwone.com/~jason/weewx/

The weewx plots are nice :)

Btw, would you mind a mailing list/group?  I'm happy to set one up on googlegroups.

Also, is there a way to permanently delete data?  I haven't been able to figure out how to get rid of data from before I put the station outside.  I tried deleting archive files and restarting, then having the Davis console clear all data, but it keeps reappearing when I restart weewx...

Jason

--
Jason Rennie
Research Scientist, ITA Software
617-714-2645
http://www.itasoftware.com/

--
Jason Rennie
Research Scientist, ITA Software
617-714-2645
http://www.itasoftware.com/

--
Jason Rennie
Research Scientist, ITA Software
617-714-2645
http://www.itasoftware.com/

[Tom Keffer]

The reason why is that plots are only regenerated if the time since
they were last generated is greater than the aggregation time. For
example, if your weekly plots specify 1 hour aggregation (the out-of-
the-box number), then weekly plots are regenerated only every hour.
Plots without any aggregation (such as dailies) are regenerated every
time.

In any case, if you want a plot to be regenerated immediately, delete
it. Missing plots are always regenerated at the first opportunity.

-tk

> > *From:* Jason Rennie [mailto:jren...@gmail.com]
> > *Sent:* Monday, November 09, 2009 7:02 AM
> > *To:* Thomas Keffer
> > *Subject:* Re: KMANATIC3

> 617-714-2645http://www.itasoftware.com/

[T Konzl]

Good morning,

I've followed these instructions to delete a date range in my archive database, but I just noticed that Tom Keffer said, "Now delete the stats archive."  I searched the forum and couldn't find an example of how to do this.  Where is this located and how do I delete it?  Would it auto-generate over time if I didn't delete it?...or do I have to manually perform this step?

BTW, thanks for posting the easy delete command for sqlite3.  I've run this to delete some overlapping WU and Cumulus dates before importing my Cumulus data.

delete from archive where datetime(dateTime,'unixepoch','localtime') between '2015-03-15 10:01:00' and '2016-01-01 00:01:00';

Thanks,

Tom K.

[T Konzl]

Okay, sorry if I'm over-posting.  I'm trying to get my system updated and I've only been on weeWX for a few weeks.  Thanks for your understanding.

I found the answer I was looking for.  It was referred to as a different name.

Finally, you need to drop the old daily summaries, which otherwise would include these bad data. Use the tool wee_database to do this:

wee_database weewx.conf --drop-daily

The summaries will be rebuilt the next time weewx starts. As this can take a long time, you may want to do this explicitly:

wee_database weewx.conf --rebuild-daily

[gjr80]

The weeWX database structure changed with the 3.0.0 release, the archive and stats databases were effectively merged into a single database.

Gary