CSV Database Import

[John White]

Quick question. I've had a Davis Vantage Pro 2 (cabled) system for many years.

We have used a software program called WAIS (WeatherActive Internet Server) that a local TV station used to send the VP2 data back to the station. They have reduced the number of stations they take care of recently and so I'm now looking into publishing the data on the web vs.saved only on the local PC. I have discovered a great deal of weather data going back 5 years, in two different options for exporting to a CSV. I would like to have this data imported into WeeWx (running on MacMini) and transition from using WAIS software to WeeWx published to a webserver, most likely using the BelcherTown skin. It would be great if the NOAA reports feature in WeeWx could somehow use this prior data to have a good dataset of the weather that has taken place.

Option 1 is a "summary" of each day, with high, low, times they occured, along with rain, pressure, humidity, dewpoint ,etc. 

Option 2 is an hourly record for each day with basically the same data minus the actual times the high/low occurred.

I have been reading about wee_import and I've ran into an invalid timestamp error.

Here's a look at what the hourly data file looks like as a CSV in the image below.

Is it possible to put all the past data into WeeWx? I've got data going back to January 2017 through now.

Thanks in advance for any assistance/advice.

[vince]

Yes - certainly after massaging your historical data a bit, which shouldn't be a problem although it might take a few iterations to get it right.  Getting the time of the high/low might be the hard part.  If you can get the data into a weewx db then weewx will construct the NOAA files for you so you'll get that one for free.

Your timestamp invalid thing is wee_import looking for that column to be in a different format. The source code for wee_import suggests that you might be able to edit one line in the config file you run wee_import against

from:

    raw_datetime_format = %Y-%m-%d %H:%M:%S

to:

    raw_datetime_format = %-m/%d/%-y %H:%M:%S

Give that a try and see if you get further down the road.   Alternately tweak your excel formatting for that column to match up what wee_import expects.

If you can provide a few files someplace we can grab maybe some folks might try it to help:

• option1 format for any month - even just a partial month
  
• option2 format for the same time period
  
• whatever config file you ran wee_import against
  

[gjr80]

Really shouldn't need to dig into the source code for this, the decoding of CSV import date-time fields is covered in the raw_datetime_format option stanza under wee_import in the Utilities Guide. If you continue to have problems I would read through the CSV portions of the wee_import section of the Utilities Guide. Then, as Vince said, if you continue to have problems it would help to see your import config file as well as the actual error trace.

Based upon what you posted you should be able to import your hourly data easily and quickly. I doubt you can do much with your daily high/low summary, WeeWX does record that sort of daily high/low information in the daily summary tables, but the daily summary tables are derived from the WeeWX archive data (in your case the hourly data) and there is no effective mechanism for supplying this (daily summary table) data manually. You could conceivably reformat your daily summary data into multiple rows of timestamped data which can then be imported into the WeeWX archive using wee_import. This could preserve your high/low data but would entail manually reformatting every daily summary record into multiple uniquely timestamped records - not a small task for five years of data. Since this has not been tested I am not sure what, if any, side effects there may be.

Gary