WU Import failed

[Andre]

Hi,

First I have to say I'm not Linux professional and all my 'know-how' is based on reading a lot of tutorials.

My weather station is a Davis Vantage Vue. WeeWX is running on a Raspberry Pi 2. This setup is running and all data are fine.

My problem is I can't import any data from WU with wee_import.

---log---

wee_import --import-config=/var/tmp/wu.conf --from=2018-01-01T00:30 --to=2018-01-31T00:00 --dry-run
Starting wee_import...
Observation history for Weather Underground station 'INORDERS99' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Observations timestamped after 2018-01-01 00:30:00 CET (1514763000) and up to and
including 2018-01-31 00:00:00 CET (1517353200) will be imported.
This is a dry run, imported data will not be saved to archive.
**** radiation: cannot convert '---' to float at timestamp '2018-01-01 00:30:00 CET (1514763000)'.
**** Nothing done, exiting.

---log---

What does it mean "radiation: cannot convert '---' to float at timestamp '2018-01-01 00:30:00 CET (1514763000)'?

Is there data mismatch on WU? What can I do to solv this issue?

Regards, Andre

[Andre]

This is my wu.conv

# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM THE WEATHER UNDERGROUND
#
# Copyright (c) 2009-2016 Tom Keffer <tke...@gmail.com> and Gary Roderick.
# See the file LICENSE.txt for your rights.


##############################################################################


# Specify our source. Available options are:
#   CSV - import obs from a single CSV format file
#   WU - import obs from a Weather Underground PWS history
#   Cumulus - import obs from a one or more Cumulus monthly log files
# Format is:
#   source = (CSV | WU | Cumulus)
source = WU


##############################################################################


[WU]
    # Parameters used when importing from a WU PWS


    # WU PWS Station ID to be used for import.
    station_id = INORDERS99


    #
    # When importing WU data the following weeWX database fields will be
    # populated directly by the imported data (provided the corresponding data
    # exists on WU):
    #   barometer
    #   dateTime
    #   dewpoint
    #   outHumidity
    #   outTemp
    #   radiation
    #   rain
    #   windDir
    #   windGust
    #   windSpeed
    #
    # The following weeWX database fields will be populated from other
    # settings/config files:
    #   interval
    #   usUnits
    #
    # The following weeWX database fields will be populated with values derived
    # from the imported data provided the --calc-missing command line option is
    # used during import:
    #   altimeter
    #   ET
    #   heatindex
    #   pressure
    #   rainRate
    #   windchill
    #
    # The following weeWX fields will be populated with derived values from the
    # imported data provided the --calc-missing command line option is used
    # during import. These fields will only be saved to the weeWX database if
    # the weeWX schema has been modified to accept them. Note that the pyephem
    # module is required in order to calculate maxSolarRad - refer weeWX Users
    # Guide.
    #   appTemp
    #   cloudbase
    #   humidex
    #   maxSolarRad
    #   windrun


    # How will the interval field be determined for the imported records.
    # Available options are:
    #   derive - Derive the interval field from the timestamp of successive
    #            records. This setting is best used when the imported records
    #            are equally spaced in time and there are no missing records.
    #   conf   - Use the interval setting from weewx.conf. This setting is
    #            best used if the records to be imported have been produced by
    #            weeWX using the same archive interval as set in weewx.conf on
    #            this machine.
    #   x      - Use a fixed interval of x minutes for every record. This
    #            setting is best used if the records to be imported are
    #            equally based in time but there are some missing records.
    #            This setting is recommended for WU imports.
    # Due to WU frequently missing uploaded records, use of 'derive' may give
    # incorrect or inconsistent interval values. Better results may be
    # achieved by using the 'conf' setting (if weeWX has been doing the WU
    # uploading and the weeWX archive_interval matches the WU observation
    # spacing in time) or setting the interval to a fixed value (eg 5). The
    # most appropriate setting will depend on the completeness and (time)
    # accuracy of the WU data being imported.
    # Format is:
    #   interval = (derive | conf | x)
    interval = derive


    # Should the [StdQC] max/min limits in weewx.conf be applied to the
    # imported data. This may be useful if the source has extreme values that
    # are clearly incorrect for some observations. This is particulalrly useful
    # for WU imports where WU often records clearly erroneous values against
    # obs that are not reported. Available options are:
    #   True  - weewx.conf [StdQC] max/min limits are applied.
    #   False - weewx.conf [StdQC] max/min limits are not applied.
    # Format is:
    #   qc = (True | False)
    qc = True


    # Should any missing derived observations be calculated from the imported
    # data if possible. Available options are:
    #   True  - Any missing derived observations are calculated.
    #   False - Any missing derived observations are not calculated.
    # Format is:
    #   calc_missing = (True | False)
    calc_missing = False


    # Imported records are written to archive in transactions of tranche
    # records at a time. Increase for faster throughput, decrease to reduce
    # memory requirements. Format is:
    #   tranche = x
    # where x is an integer
    tranche = 250


    # Lower and upper bounds for imported wind direction. It is possible,
    # particularly for a calculated direction, to have a value (eg -45) outside
    # of the weeWX limits (0 to 360 inclusive). Format is:
    #
    # wind_direction = lower,upper
    #
    # where :
    #   lower is the lower limit of acceptable wind direction in degrees
    #   (may be negative)
    #   upper is the upper limit of acceptable wind direction in degrees
    #
    # WU has at times been known to store large values (eg -9999) for wind
    # direction, often no wind direction was uploaded to WU. The wind_direction
    # parameter sets a lower and upper bound for valid wind direction values.
    # Values inside these bounds are normalised to the range 0 to 360. Values
    # outside of the bounds will be stored as None. Default is 0,360
    wind_direction = 0,360

[gjr80]

Hi,

The problem appears to be that the radiation data for your station on WU is recorded as '---' instead of it being blank or a numeric. It is a simple fix to ignore the '---' but I just need to work through some other parts of wee_import to make sure I do not introduce some inconsistent behaviour. Give me a day or 2 and I should have a revised wee_import that will do the job.

Gary

[Andre]

Any news to wee_import?

Regards, Andre

[gjr80]

Hi Andre,

Unfortunately some back and neck problems over the last week have kept me from the compter, only using the iPad now and then. Hopefully the physio will have things fixed soon and I can get back to weeWX.

Gary

[Andre]

Gary, I wish you a speedy recovery.

I know that as well. Unfortunately.

Regards, Andre

[gjr80]

Andre,

I have reworked wee_import and have it here on my test system importing data from your WU station without issue. Just having a few GitHub issues, once I sort them out later today I will post some instructions for you or failing that I will email you the updated wee_import file(s).

Gary

[Andre]

Gary, thanks. This sounds good and I'll waiting for your release.

Andre

[gjr80]

Andre,

I have not yet been able to resolve my GitHub issues so I have emailed you some updated files just now.

Gary

[Andre]

Gary, here again: thank you very much for your fast support!

Regards, Andre

[gjr80]

Just to close this off I committed changes to the development branch on GitHub today (commit 3e1aa76)
to allow wee_import to ignore invalid data in an import field. This feature should be included in the next weeWX release. Should anyone require this ability before the next release it is probably easiest if they ask in the forums for instructions.

Gary