How do I get data back from weather underground to fill in missing data in my database?

Feather Feet

unread,

Jan 31, 2016, 1:58:47 AM1/31/16

to weewx-user

Hi, I wonder if any of you might have a suggestion, this is the reverse question to normal!

I just 'lost' an SD card and now have data from 2013 to the beginning of January 2016 from a backup. I am taking data again but most of January I lost when the SD card died. The missing data IS on weather Underground. Does anyone know if an application exists to pull the data from Weather Underground and put it back in the weewx database. I know about the software to put missing data on Weather Underground but I would like to do the reverse!

Thank you for the help, and if I missed another posting or something in the docs. sorry just point me to the location.

FeatherFeet

Old HW: fine Offset Raspberry Pi B
New HW: Davis Vantage Vue, Raspberry Pi 2 hooked together directly via the 3.3V serial port on both bits of HW, and if your interested, no crazy USB or level shifter stuff in the way!

Weather station ID on Weather Underground: KCAPLEAS22
Weewx 3.4.0

Andrew Milner

unread,

Jan 31, 2016, 3:13:21 AM1/31/16

to weewx-user

You can download the daily summary data in csv form and then upload that into the archive and let weewx recreate the dsily summaries. However to get the detailed daily archive data one can only do it a day at atime - and I would also be very interested if anyone has a program that can get the daily detailed data between two dates that can then be put in the database. Someone MUST surely have done this before!!

Feather Feet

unread,

Jan 31, 2016, 3:45:45 AM1/31/16

to weewx-user

I agree someone must have done this before...but WHO...and where did they leave a copy of the code? I could download 30 ish days in CVS one at a time, but then how to upload them, and finally how to get Weewx to rebuild the daily summaries?

FeatherFeet

Feather Feet

unread,

Jan 31, 2016, 3:47:01 AM1/31/16

to weewx-user

Sorry I meant csv not cvs that would be a store....not a file format....

Andrew Milner

unread,

Jan 31, 2016, 3:58:54 AM1/31/16

to weewx-user

You can import the csv easily enough - google for import csv to SQLite or import csv to MySQL. WEEWX will rebuild daily summaries using wee_config_database --drop-daily and wee_config_database --backfill-daily

vince

unread,

Jan 31, 2016, 3:01:11 PM1/31/16

to weewx-user

On Saturday, January 30, 2016 at 10:58:47 PM UTC-8, Feather Feet wrote:

Hi, I wonder if any of you might have a suggestion, this is the reverse question to normal!

I just 'lost' an SD card and now have data from 2013 to the beginning of January 2016 from a backup. I am taking data again but most of January I lost when the SD card died. The missing data IS on weather Underground. Does anyone know if an application exists to pull the data from Weather Underground and put it back in the weewx database. I know about the software to put missing data on Weather Underground but I would like to do the reverse!

Thank you for the help, and if I missed another posting or something in the docs. sorry just point me to the location.

FeatherFeet

This has been asked and answered many times in the past on the list.

In short, you have to:

query WU for the time periods you want
reformat the data you'll get into a form compatible for restoring into weewx's archive
import into the archive
drop-daily and restart weewx to rebuild the daily stats, noaa tables, etc (this part will take a while)

Downloading:

==========

I wrote a utility years ago to do the querying part, which relies on a perl CPAN module that you'd have to install - https://raw.githubusercontent.com/vinceskahan/weewx-odds-and-ends/master/getWunderHistory.pl is there you'd find it. Disclaimer: I haven't used this in years, so I'm not certain about whether it still works. WU might have changed their interfaces since then.

A quick google search for 'how to download weather underground data' points me to this pretty good video - How to Download Weather Data from Weather Underground ...

The best one I've found was from a thread "reverse wunderfixer" in this group which points you to a pretty cool utility at http://oco-carbon.com/weather/access.html - I tried this back then, it worked. Do your downloads in small batches (a couple months at a time) or it makes you a little uneasy waiting for it to complete sometimes.

Reformatting

==========

Assuming you use sqlite3 as your db, the easiest way is to generate a sqlite3 dump/restore text file then restore the records on top of (a copy of) your archive database, which will add them to the archive table. Getting the right fields in the right order requires a little manipulation, I'm not aware of a end-to-end automated utility that handles this for you. There have been examples posted to the list of how to do this in the past.

Bottom line, you have some manual work, but once you get the data from WU it's doable. Hardest part is reorganizing your data to get the right fields into the right order left-to-right. This is one place where Excel actually isn't bad, you can drag'n'drop the columns to the correct order pretty easily in that. Just be sure to have empty columns for the (many) fields that weewx supports that WU doesn't. You wouldn't want your outTemp data written into a windspeed field in weewx :-)

Feather Feet

unread,

Jan 31, 2016, 7:48:15 PM1/31/16

to weewx-user

Thank you for the links, I will give it a try.

Feather Feet

Reply all

Reply to author

Forward