Exclude UV from WU wee_import

[Rory Gillies]

I am importing historic data from Weather Underground using wee_import and it is largely working fine, however the data I am importing has erroneous UV spikes in the data (but still valid values so not picked up by the --ignore_invalid-data_switch) . Is there any way I can exclude the UV data during import?  For the month I have imported I have manually removed the spikes from weewx.sdb, but this will take ages on a year's worth of data. TIA

[Rory Gillies]

Here's an example of the spikes, it's like this for every month. The highest UV we get up here is no more than 7.

[gjr80]

Unfortunately for a WU import wee_import imports whatever WU provides, there is no (simple) way to limit which fields are imported. There are a couple of workarounds (untested) that should work. 

1. Tell WeeWX to force the UV value to None (needs to be done before running wee_import):

• stop WeeWX
  
• before running wee_import add an entry to the [StdCalibrate] [[Corrections]] stanza of weewx.conf as follows:
  

    [StdCalibrate]

        [[Corrections]]

            ....

            UV = None

• save weewx.conf
  
• run wee_import and import the WU data
  
• remove the UV = None correction added earlier
  
• restart WeeWX

2. Manually remove the UV data using some SQL (needs to be done after running wee_import):

• stop WeeWX
  
• make a backup copy of the database
  
• if not already installed install the sqlite3 utility:
  

     $ sudo apt-install sqlite3

                 (or as appropriate for your OS)

• open your database with sqlite3:
  

     $ sqlite3 /home/weewx/archive/weewx.sdb

                 or

     $ sqlite3 /var/lib/weewx/weewx.sdb

                depending on your WeeWX install

• execute the following SQL statement (setting the date and time range accordingly) to set UV to None for the interval concerned
  

             sqlite> UPDATE archive SET UV=NULL WHERE UV IS NOT NULL AND dateTime>strftime('%s', '2022-10-01 13:00:00') AND dateTime<=STRFTIME('%s', '2022-10-21 09:15:00');

• exit sqlite3:
  

     sqlite> .quit

• update the daily summaries:

     $ wee_database --rebuild-daily --from=2022-100-01 --to=2022-10-21
            (again replacing dates with the required date range)

• restart WeeWX
  
• WeeWX should update any web pages with the new data on the next report cycle, plots will eventually update (you can force all plots to be regenerated on the next report cycle by deleting all plots of the WeeWX machine). NOAA format reports will need to be deleted from the earliest month/year with incorrect UV data through until the current month/year to force regeneration

Gary

[Rory Gillies]

Thanks Gary, I used a combination of both methods, using the QC to exclude UV values greater than 7 on import, then using the SQLite commands to change the values to be more representative. The command below clips all values above 1.8 between 01/102021 and 31/102021 as 1.8 is the typical maximum value for October here at 60ºN:

         sqlite> UPDATE archive SET UV=1.8 WHERE UV >1.8 AND dateTime>strftime('%s', '2021-10-01 00:00:00') AND dateTime<=STRFTIME('%s', '2021-10-31 23:55:00');

The resultant maximum values in my stats page looks much better now!

I'll cary on importing the rest of 2021 now I have the UV sorted. The UV sensor on the old weather station was rubbish, it only measured integer values, had loads of spikes, the values were far too high and even measured a UV index of 1 at night on occasions - usually a very bright moon! I installed the Davis Vantage PRO2 Plus at the end of January 2022 so will soon have a full year of real UV data.

Cheers,

Rory