wee_import CSV interval = 1 vs. archive_interval = 300?

[William Burton]

Hello,

I've been testing a CSV import configuration with wee_import. The CSV data is in 1 minute intervals whereas weewx.conf has an archive_interval = 300 for 5 minutes. my csv.conf has interval = 1 to match the data interval (which has a few gaps).

What I'm finding is the imported data is still in one minute intervals while data archived from the driver has five minute intervals:

sqlite> select interval, count(*) count from archive group by interval;
interval    count
----------  ----------
1           51234
5           12259

1. Will having mixed intervals in the same database cause any issues with reporting such as incorrect time scales in graphs?
   
2. Wouldn't archive records with a one minute interval slow down report generation?
   
3. I calculated that I have around 900 files to import for a total of almost 1.3 million records. If these records could be averaged to only one record per five minutes, then it would be only 260,000 records which seems more reasonable.
   

I'm using WeeWX 4.0.0b18.

Thanks for any assistance,

-Bill

[Thomas Keffer]

Hi, Bill. 

1. WeeWX can handle mixed intervals.

2. The time, field dateTime, is indexed, so it does not slow down reports as much as you think. My database is about 1.5 M records and it performs fine on a very slow machine. The exception is queries that involving finding the min and max of the whole database.

3. If you decide you want to consolidate into 5 minute intervals, user Cameron came up with a guide that uses some clever SELECT statements to do the job. Find it on the Wiki.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0151a436-a259-4c12-be92-c20c5e796711%40googlegroups.com.

[William Burton]

Hello Thomas,

Thanks for the pointer to the SQL solution. I can see that method as fine for a one-time or infrequent interval conversions. The main part that groups and averages most columns is straightforward but the part that uses trig functions to calculate averages for wind direction and speed was completely unexpected. I'm wondering what is an authoritative source for these calculations?

For a couple of reasons, I'd prefer a solution in Python. First, the driver I'm implementing for the Columbia Weather MicroServer won't initially have the ability to automatically catch up missing data by implementing the genArchiveRecords() method in the driver so I'll have to manually import data from time to time when observations have been missed. Secondly, since I have to load 900 files, the CSV importer won't work for me as-is so it looks like I'll have to write a custom importer anyways. While I'm at it, I may try to implement accumulation logic as part of the import. The classes in bin/weewx/accum.py look like they would be very helpful. After that it would be a matter of figuring out the row grouping logic.

Thanks,

-Bill

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.