importing data from Weather Display log files

[William Phelps]

I am making progress on this. I now have a python program that will read WD log files and parse them. Most of the effort was in adding enough checks to get around all the quirks in the data - there are lots!  In going over my several years of data, I found places where the date/time skips ahead, not too much of a surprise, but also places where it jumps backward by odd amounts. I also found several occurrences of duplicate date and time, sometimes with the same data, sometimes with different data.

The program reads these WD log files:

 nyyyylg.txt - main log file

 nyyyyvantagelog.txt - data specific to the Davis Vantage stations

 nyyyyindoorlog.txt - indoor data

The program takes start and end dates as parameters and writes a csv file, to make review and editing easier.  The next step will be to import the csv file into the weewx database. I tried using "sqllitebrowser" for this but it crashes trying to do an import.

I have a copy of Tom's convert_csv.py and it looks like it should be fairly easy to modify this to work with the csv file my program writes.

Right now the program works with a fixed archive interval.  I plan to also add the ability to change (increase) the interval, by averaging the data, since for one of my stations I changed from 60 seconds to 300 seconds.

William

[William Phelps]

I'm curious, is anyone else interested in converting WD data besides myself and Greg?
William

[William Phelps]

I did trial run of an actual conversion of WD data to weewx last night. I attempted a run on about 4 months of data from January thru April when I started running weewx.

First I ran wdConvert.py, which reads WD log files and writes a CSV file.  Running on a fairly fast PC, it took only a minute or two to produce the CSV file.

Then csvConvert.py reads the CSV file and uses the weewx engine to add records to the database.  In my case it has to run on the Raspberry Pi (at least until I figure out how to get it to run on my PC). It took a couple of hours to load about 145,000 records or so, with the little Pi running flat out.  I run with an archive interval of 60 seconds so there is a lot of data. If I were going to do this sort of thing often I'd look at writing directly to the weewx database, but it doesn't seem worth it for this application.  I did figure out how to set the syslog level so it doesn't record all the record adds, which helped.

After processing the csv file, starting up weewx took some 25 minutes as it had to generate a new stats database with almost 5 months of data.

Most of it worked well. I've got a couple of bugs to clean up - the way weewx keeps rain data is different from WD, and I'm not adjusting for that, so the graphs for yearly rain data were rather off - we don't get 300 inches of rain!.  Should be easy enough to sort out and fix that.  WD also doesn't record wind chill, so I need to add that calculation to the process.

So the basics are working, just a bit of cleanup to do and I should have it all working.

I think it's pretty clear from what I'm saw with this test that I need to switch to a 5 minute archive interval, otherwise with a couple of years or more of data weewx will take an hour or more to build the stats database!

So it looks like my next task is to modify wdConvert to take 60 second interval data and compress it to 300 second interval data so I can change the archive interval on this weather station.

William

[Oz Greg]

Great work William,  I am looking forward to running this and rebuilding my yearly history considering I keep on trashing my weewx DB's while I am in test / build mode I am going to need to some data to convert.. 

As I have my WD and weewx data already in mysql DB I was thinking of directly updating from DB to DB would this be a better approach or should I use the raw TXT files?

[Thomas Keffer]

I don't remember exactly how csvconvert worked, but if it inserts each data record as a separate transaction, it's going to be s-l-o-w on any machine, but especially on an RPi.

If you do it in one transaction context, it will be much, much faster.

-tk

William

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[William Phelps]

csvConvert is based on the convert_csv you sent, and yes it does 1 addrecord at a time. I saw in archive.py that addrecord can take a block of records, but I'm not sure it's worth figuring out how to do that given how often this program will be run...

I'd run it on my much faster PC but there is no syslog module in the Windows Python - however I see that there are several examples of how to get around this out there, so I'll probably go that route rather than keep running it on the Pi.

Thanks!
William

[William Phelps]

It turns out that it runs faster on the RPi than on my PC, which is a bit of a surprise, as the PC's clock speed is several times that of the Pi. Perhaps the PC Python is not as well optimised.

I took another look at archive.addRecord() - it can take a block of records, but it still adds them to the db one at a time.

William

On Monday, 20 May 2013 15:28:46 UTC-7, Tom Keffer wrote:

[Oz Greg]

Hi William,

Is this code ready for use??? As I would like to try importing some data :)  I grabbed the code but it looks like I am missing a heap of python modules on the raspberry..  :(

[Oz Greg]

Hi William,

I installed the missing modules and attempted to run the program, it seemed to convert ok then got stuck in a loop processing 42013vantagelog.txt over and over again..

va) line mismatch, skipping...

42013vantagelog.txt[7813]: 2013-04-06 19:18:00

(va) line mismatch, skipping...

42013vantagelog.txt[7814]: 2013-04-06 19:19:00

(va) line mismatch, skipping...

42013vantagelog.txt[7815]: 2013-04-06 19:20:00

(va) line mismatch, skipping...

42013vantagelog.txt[7816]: 2013-04-06 19:21:00

[Oz Greg]

Copy of my log files..

[William Phelps]

Greg,

I have no idea what version you are running, sorry. I would not expect a "ton" of missing modules - you don't actually say which ones were missing? It uses pretty much whatever weewx uses.

What I sent you in email was a prototype, as the email stated. I posted here that I did a test run and had some problems and will be working on them.  When I get everything fixed I will update the files on my GitHub repo.

William

[Oz Greg]

William,

No worries about the missing modules all it was missing is argparse.. :)

I also worked out the looping issue it was because my data had month 4 and 5 in it which was sending your program into a loop..

I will look for your updates github thanks..

Will you be adding support for mysql into csvConvert.py? :)

Thanks..

[William Phelps]

On Thu, May 23, 2013 at 8:42 AM, Oz Greg <ozg...@gmail.com> wrote:
> Will you be adding support for mysql into csvConvert.py? :)

at present, csvConvert doesn't use either sql directly, it uses weedb.
It should use whatever sql you have configured in weewx.conf

[William Phelps]

Success! I modified wdConvert so it calculates the rain and rainrate values correctly, and added windchill.  I just did another run using my WD data for the first 4 months of 2013, and added that to the weewx database.  Restarted weewx, which again took about 22 minutes to calculate the stats db.  Now the data for January, February, and March match what WU shows for the same period, and the NOAA text files match as well.

The weewx sqlite database went from around 14 mB to almost 38 MB.

I would like to have a full year's data in weewx, but before I do that I think I may have to switch to a longer archive interval.  Running with a 60 second interval is just too much data...

William

[Oz Greg]

That is inconsistent with your GIT version..

csvConvert.py

parser.add_argument("sql_path",

                    help="Path to the SQLITE database (Required)")

I will wait until you have had a chance to update your GIT which I hope will be soon..

Thanks..

[Oz Greg]

I just ignored the path concept and threw in my mysql DB name instead and that worked :)

Thanks..

[Oz Greg]

wee_config_database[19781]: stats: backfilled 76 days of statistics with 70902 records in 476.02 seconds cannot argue with that :)

[Oz Greg]

Feel like I am spamming your thread here :-)

Not sure if you have corrected this or not but rain is being accumulated in the archive records which I believe is incorrect

Can we add a conversion capability between Metric to Imperial, Tom has a great little function which already does this ;-) 

 weewx.archive.reconfig(old_archive_db_dict, new_archive_db_dict, target_unit_system)

Thanks

[Thomas Keffer]

Almost, but not quite.

If you look closely, you'll see that argument record_obj can be either a single record, or an iterable that yields data records. If the latter, it will process all the records handed to it by the iterable as a single transaction.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[William Phelps]

Tom,

I looked at archive.py. What I see is this:

with weedb.Transaction(self.connection) as cursor:
for record in record_list:
[...]
sql_insert_stmt = "INSERT INTO %s (%s) VALUES (%s)" %
(self.table, k_str, q_str)
try:
cursor.execute(sql_insert_stmt, value_list)
syslog.syslog(log_level, "Archive: added %s record %s"
% (self.table, weeutil.weeutil.timestamp_to_string(record['dateTime'])))

Unless I'm completely lost, it's adding one record at a time, no?

William

> You received this message because you are subscribed to a topic in the

> Google Groups "Weewx user's group" group.

> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/RLjyUnn9ZI4/unsubscribe?hl=en-US.
> To unsubscribe from this group and all its topics, send an email to

[Thomas Keffer]

Yes, it's adding one record at a time, but all within a single transaction context.

It's opening and committing the transactions that takes all the time. 

-tk

[Oz Greg]

William,

Not seen the update to your GIT hub yet, is it current??

Also as a tip for anyone converting from Metric WD..  Weewx for reasons that escape me captures rain in CM..  (Tom why is this?) and WD captures rain in MM so you need to do some kind of conversion otherwise your rain values (like mine) are all wrong..

[Oz Greg]

William,

I made the following changes to wdConvert.py

rainlastmin = float(d_flg['rainlastmin']) / 10

rain = float(d_flg['dailyrain']) / 10 - dailyrain

dailyrain = float(d_flg['dailyrain']) / 10 

if (rain < 0):

rain = 0

fwriter.writerow([utcs, 1, int(interval/60), \

d_flg['barometer'], pressureIn, altimeterIn, d_finlg['temperature'], d_flg['temperature'], d_finlg['humidity'], d_flg['humidity'], \

d_flg['windspeed'], d_flg['direction'], d_flg['gustspeed'], d_flg['direction'], rainlastmin, rain, d_flg['dewpoint'], \

0, d_flg['heatindex'], d_fvalg['ET'], d_fvalg['radiation'], d_fvalg['UV'], 

])

[Andrew Milner]

Oz Greg - I just put the annoying cm for rain down to our American cousins 'interpretation' of metric - but I bet Tom comes up with a plausible and logical reason that we have missed!!!!!!

 

They (those from across the pond) do tend to get lost when things are not in inches or cups .........

 

Hehe .... ducks and waits for the brickbats to fly .....

 

Andrew

[Stuart Turrell]

Thanks for the links, they are extremlely helpful!

One other question i forgot to mention.

I have a soil station (fully populates) and 2 temperature stations, can weeex capture this data and display it as well?

Stuart

[Niklas Berglund]

Hi.

I was wondering if this script is still working. I have finally made the switch from Weather Display to WeeWX. I have log files for a couple of years back that I would like to import to WeeWX. 

I've read through this and been trying to get wdConvert.py to work. My Python skills are not the best. 

I get the following error when trying to run wdConvert.py : 

  File "./wdConvert.py", line 23, in <module>

    import weewx.uwxutils

ImportError: No module named weewx.uwxutils

I don't know where to find the module or how to tell python where its located. 

Is there anyone that would like to help me on this ?

Niklas

[gjr80]

Hi,

Sounds like a PYTHONPATH issue. I had a quick look at the repo and did not see any guidance on what should be included on the command line. Exactly what command are you using to run the script? Try running it again but this time capture the command you used and all the output. Post this capture. Also, what type of install? setup.py, deb ?

As for your first question, does this script still work, bear in mind this script is from circa May 2013 when weewx v2.4.0 was the current version. There has been a lot of water under the bridge since then (lots of significant changes introduced in weewx v3.0.0) but since this script only produces a csv file for later import you might be lucky - the amount of interaction with weewx in the script appears minimal.

Gary

[Niklas Berglund]

Yeah, Im sure its like you say, a PYTHONPATH issue.  I don't know where to point the python path to where he'll find the module to include.. 

Im running a simple :  "sudo python ./wdConvert.py" command. No arguments added.  What sort of capture do you need me to post?

I've installed WeeWX using the DEB install. 

Thanks for taking the time with a noob :-)

Niklas

[gjr80]

Ok, two options. Option 1, copy wdConvert.py to /user/share/weewx, cd to /user/share/weewx then try the command you were using, ie

sudo cp ./wdConvert.py /usr/share/weewx
cd /usr/share/weewx
sudo python ./wdConvert.py

Option 2, try the following command:

sudo PYTHONPATH=/usr/share/weewx python ./wdConvert.py

assuming you are still in the same directory as where wdConvert.py is located.

Gary

[William Phelps]

The script was written assuming a setup install of weewx, which creates a weewx dir in the home directory.  Since you did a DEB install the dirs are different.  It's been ages since I wrote this, and it's not intended for a general audience but rather for someone familiar with the paths and file locations.  The script will also need access to your WD database and I don't remember how it finds that either,

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/RLjyUnn9ZI4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

(sent from an alien in the Andromeda Galaxy - read at your own risk!)

[Niklas Berglund]

Thanks, that did the trick (option 1) 

I now got a couple of "outMONTHYEAR.csv" files generated now from 2010 and so forth. 

Tried running the scvConvert.py script, but was missing the module "weewx.archive" 

Have also looked into the wee_import (built in command). The would probably work as well, but the CSV files I got produced from wdConvert.py isn't the correct syntax for wee_import csv files to accept. 

Niklas

[gjr80]

Hi,

So what is the format of the resulting csv? If it's one record per line then a bit of manipulation should see it be able to be imported with wee_import.

As for the scvConvert.py file, since it was written for weewx 2.x I suspect it will need a bit of work to get it to work with weewx 3.x. The weewx database structure changed significantly from weewx v2 to weewx v3.

Gary

[Niklas Berglund]

The format of the outputted CSV file is : 

dateTime,usUnits,interval,barometer,pressure,altimeter,inTemp,outTemp,inHumidity,outHumidity,windSpeed,windDir,windGust,windGustDir,rainRate,rain,dewpoint,windchill,heatindex,ET,radiation,UV

1285884000,1,1,1023.6,1023.0043,1023.2786,25.2,01.2,47,94,0,330,0,330,0.0,0.0,00.4,0,1.2,0.0,0.00,0.0

1285884060,1,1,1023.6,1023.0043,1023.2786,25.2,01.2,47,94,0,330,0,330,0.0,0.0,00.4,0,1.2,0.0,0.00,0.0

one record per line.  The "dateTime" format is weird looking though. It doesn't look anything like that in my logfiles from WD, which look like this :

day month year hour minute temperature   humidity     dewpoint   barometer  windspeed  gustspeed direction   rainlastmin    dailyrain  monthlyrain   yearlyrain  heatindex

11  4 2008 23 45 03.8  83 01.2 1009.0   0   0 112 0.0 0.0 0.0  0.0 3.8

11  4 2008 23 46 03.8  83 01.2 1009.1   0   0 112 0.0 0.0 0.0  0.0 3.8

I did find some info (going through the download section of earlier versions) about the missing modules, so I have those needed now. 

But if the database has been changed from 3.x then I guess it still won't work. 

I have access to two running RaspberryPi's.  One thats untouched about all this that I got hooked up to my Vantage/Web. And another that I was going to use for this database stuff. Ive got 3.x installed via DEB on both Pi's.  I wonder if it would help if I just installed 2.x on the standalone Pi and did the convert there..  but then again, will weewx 3.x on the other Pi be able to read and use the database that were produced in 2.x ? 

Maybe this has turned out to be a bit of an overkill.. I guess I could just pull the data I need to fill my weewx database from Weather Underground instead..  we're just curious to see if it would work. 

Niklas

[gjr80]

dateTime is an epoch timestamp, exactly what weewx uses. wee_import will import these files with little change. Have a careful read through the wee_import section of the weewx utilities guide - http://weewx.com/docs/utilities.htm.

You are game importing from WU, stick with the weather display data, it is your best source.

Gary

[Niklas Berglund]

Got it. thanks

I think I should have it all sorted now : 

pi@WeatherPi:/var/tmp $ wee_import --import-config=/var/tmp/csv.conf --config=/etc/weewx/weewx.conf --dry-run --verbose

Starting wee_import...

A CSV import from source file '/var/tmp/data.csv' has been requested.

The following options will be used:

     config=/etc/weewx/weewx.conf, import-config=/var/tmp/csv.conf

     source=/var/tmp/data.csv, from=None, to=None

     dry-run=True, calc-missing=True

     tranche=250, interval=derive, date/time_string_format=%Y-%m-%d %H:%M:%S

     rain=cumulative, wind_direction=[-360.0, 360.0]

     UV=True, radiation=True

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

Missing derived observations will be calculated.

This is a dry run, imported data will not be saved to archive.

Obtaining raw import data for period 1...

The following imported field-to-weewx field map will be used:

     source field 'barometer' in units 'inHg' --> weewx field 'barometer'

     source field 'outTemp' in units 'degree_F' --> weewx field 'outTemp'

     source field 'outHumidity' in units 'percent' --> weewx field 'outHumidity'

     source field 'rain' in units 'inch' --> weewx field 'rain'

     source field 'dateTime' in units 'unix_epoch' --> weewx field 'dateTime'

     source field 'windDir' in units 'degree_compass' --> weewx field 'windDir'

     source field 'pressure' in units 'inHg' --> weewx field 'pressure'

     source field 'windSpeed' in units 'mile_per_hour' --> weewx field 'windSpeed'

     source field 'inHumidity' in units 'percent' --> weewx field 'inHumidity'

     source field 'inTemp' in units 'degree_F' --> weewx field 'inTemp'

     source field 'windGust' in units 'mile_per_hour' --> weewx field 'windGust'

     source field 'rainRate' in units 'inch_per_hour' --> weewx field 'rainRate'

     source field 'windGustDir' in units 'degree_compass' --> weewx field 'windGustDir'

Raw import data read successfully for period 1.

Mapping raw import data for period 1...

Starting dry run import ...

132254 records identified for import.

Records processed: 132254; Unique records: 132254; Last timestamp: 2010-12-31 23:59:00 CET (1293836340)

Mapped data saved to archive successfully for period 1.

Finished dry run import. 132254 records were processed and 132254 unique records would have been imported.

pi@WeatherPi:/var/tmp $

/Niklas

[Niklas Berglund]

It looked okay for my first couple of months importing. The weewx.sdb database went from 2 MB to 11 MB. So somethings there. 

But it didn't generate the NOAA reports. Is it supposed to do that automatically from older data in the database ?

Oct 2010 look like this : 

                   MONTHLY CLIMATOLOGICAL SUMMARY for Oct 2010


NAME: Lanna, Narke, Sweden                  
ELEV: 101 meter    LAT: 59-14.76 N    LONG: 014-54.96 E


                   TEMPERATURE (°C), RAIN (mm), WIND SPEED (m/s)

                                         HEAT   COOL         AVG
      MEAN                               DEG    DEG          WIND                   DOM
DAY   TEMP   HIGH   TIME    LOW   TIME   DAYS   DAYS   RAIN  SPEED   HIGH   TIME    DIR
---------------------------------------------------------------------------------------
 01
 02
 03
 04
 05
 06
 07
 08
 09
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
---------------------------------------------------------------------------------------
     -14.7   -9.4     04  -20.4     21 1024.9    0.0 1193.8    1.1   10.3     14    251

Niklas

[Andrew Milner]

The best way of forcing the NOAA reports to be generated is to a) stop weewx    b) delete all NOAA reports    c) start weewx

Not sure if it automatically generates reports earlier than ones already present in the directory - hence deleting ALL NOAA reports and have weewx regenerate them all again from the archive.

[Peter Leban]

Where is it possible to find the script to convert WD's data to weewx database? Is there a step-by-step procedure that shows how to do it?

I'm planning to move from WD to weewx - should I start the weewx, let it run for some time and only then import data or is it better to do it earlier?

Any help is appreciated - thanks!

Dne nedelja, 08. januar 2017 07.08.19 UTC+1 je oseba Andrew Milner napisala:

[gjr80]

Hi,

A WD import module was added to wee_import a few months ago and I am aware of at least one user that has used it to successfully import numerous years of WD monthly log file data. The WD capable version of wee_import can be found in the WeeWX github development branch but will not be included in a WeeWX release until the next release. If you are using WeeWX 3.9.0 or later I can give you some fairly straightforward instructions to download, install and use the WD capable version of wee_import if you are interested. Unfortunately I am travelling at the moment and it will likely take at least a couple of days for me to put together the instructions.

In terms of what approach to take when doing the import/changeover to WeeWX will depend on a few factors. Namely what weather station you have (in particular whether the station has a data logger ala the Davis stations), how much WD data you have to import and how anal you are about losing data. If it were me and I had a significant amount of WD data to import, I would continue to run WD but install WeeWX on a machine and import all WD data except the current month. If necessary I would then install WeeWX on the final target machine (if not already done so), check it works with your station, stop WD, copy the WeeWX database containing the imported data to the WeeWX machine, import the current months WD data then finally start WeeWX. If you have a station with a logger then things are a little easier, you can afford to stop WD and take your time to install WeeWX on the final machine and import all WD data. When you finally start WeeWX it will download all the data held in the logger since WD was shut down. These are but two approaches and there are number of variations you can make.

In terms of time taken to import I think the user I referred to had about 11 years of WD data that took overnight (or maybe slightly longer) to import on a relatively new RPi.

Gary

[Peter Leban]

Hi Gary,

many thanks for your answer. My setup is as follows:

* WD running on Windows. Keeps 3.5 years of data

* Vantage Vue with USB datalogger

* temperature sensor in jar via 1-wire USB interface to log the "solar" radiation

Don't rush with instructions; I'll wait. I have all hardware in place and can start working on it anytime.

I will follow your suggestion how to start converting the data, looks reasonable and risk-free.

Side question: is there a special setting in weewx that enables data extraction from the Davis datalogger?

Peter