Rain Underreading: Discrepancies even when importing from csv

222 views
Skip to first unread message

Marlon

unread,
Oct 30, 2023, 6:53:37 AM10/30/23
to weewx-user
Hello
I recently started using weewx to collect my own weather data (instead of relying on external sites). The problem is strange. I tried for days, exhaustively, to fix this - wanted to avoid another support issue here that wastes your time, but alas.

My station is a froggit WH3000 and has always (still does), accurately posted to ecowitt, WU and WeatherCloud. 

I added weewx on my vps now, and it uses interceptor to get the data using my stations 'custom' posting (it has a choice of ecowitt vs wu protocol).

The data on weewx is all fine, accurate, except it underreads rain by a few millimeters everytime. All the data on the station console, WU and ecowitt matches, but weewx underreads.

For october for instance: 28.4mm is the correct value. Weewx has 23.7mm
I can see it happen on daily rainfall as well. Yesterday we had 0.2mm drizzle, but weewx still thinks we had 0.0

On a sidenote, I tried to import historic data from ecowitt (csv) to weewx, and I see the same thing happening.  

I feel bested. It got the better of me. Advice or ideas could go a long way.
Thanks in advance
Marlon

Marlon

unread,
Oct 30, 2023, 7:12:46 AM10/30/23
to weewx-user
Oct 30 13:08:42 goatbox weewx[116207] DEBUG weewx.reportengine: Report 'FTP' not enabled. Skipping.
Oct 30 13:08:42 goatbox weewx[116207] DEBUG weewx.reportengine: Report 'RSYNC' not enabled. Skipping.
Oct 30 13:08:49 goatbox weewx[116207] DEBUG user.interceptor: empty queue
Oct 30 13:08:59 goatbox weewx[116207] DEBUG user.interceptor: empty queue
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: POST: b'PASSKEY=XXXX&stationtype=EasyWeatherV1.6.6&dateutc=2023-10-30+11:08:59&tempinf=76.6&humidityin=46&baromrelin=29.661&baromabsin=29.723&tempf=71.6&humidity=44&winddir=72&windspeedmph=3.1&windgustmph=4.5&maxdailygust=15.9&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.008&monthlyrainin=1.118&yearlyrainin=32.445&totalrainin=32.445&solarradiation=831.15&uv=8&wh65batt=0&freq=868M&model=WS2900_V2.01.08'
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: raw data: b'PASSKEY=660D031A3A77C1332DFA76B2C975573C&stationtype=EasyWeatherV1.6.6&dateutc=2023-10-30+11:08:59&tempinf=76.6&humidityin=46&baromrelin=29.661&baromabsin=29.723&tempf=71.6&humidity=44&winddir=72&windspeedmph=3.1&windgustmph=4.5&maxdailygust=15.9&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.008&monthlyrainin=1.118&yearlyrainin=32.445&totalrainin=32.445&solarradiation=831.15&uv=8&wh65batt=0&freq=868M&model=WS2900_V2.01.08'
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: using rain_total 32.445 from yearlyrainin
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter b'PASSKEY=660D031A3A77C1332DFA76B2C975573C
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter stationtype=EasyWeatherV1.6.6
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter baromrelin=29.661
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter maxdailygust=15.9
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter rainratein=0.000
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter eventrainin=0.000
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter hourlyrainin=0.000
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: ignored parameter weeklyrainin=0.008
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: ignored parameter monthlyrainin=1.118
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter totalrainin=32.445
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter wh65batt=0
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter freq=868M
Oct 30 13:09:01 goatbox weewx[116207] INFO user.interceptor: unrecognized parameter model=WS2900_V2.01.08'
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: raw packet: {'dateTime': 1698664139, 'usUnits': 1, 'rain_total': 32.445, 'temperature_in': 76.6, 'humidity_in': 46.0, 'pressure': 29.723, 'temperature_out': 71.6, 'humidity_out': 44.0, 'wind_dir': 72.0, 'wind_speed': 3.1, 'wind_gust': 4.5, 'solar_radiation': 831.15, 'uv': 8.0, 'rain': 0.0}
Oct 30 13:09:01 goatbox weewx[116207] DEBUG user.interceptor: mapped packet: {'dateTime': 1698664139, 'usUnits': 1, 'pressure': 29.723, 'outHumidity': 44.0, 'inHumidity': 46.0, 'outTemp': 71.6, 'inTemp': 76.6, 'windSpeed': 3.1, 'windGust': 4.5, 'windDir': 72.0, 'radiation': 831.15, 'rain': 0.0, 'UV': 8.0}
Oct 30 13:09:01 goatbox weewx[116207] INFO weewx.manager: Added record 2023-10-30 13:08:30 SAST (1698664110) to database 'weewx'
Oct 30 13:09:01 goatbox weewx[116207] INFO weewx.manager: Added record 2023-10-30 13:08:30 SAST (1698664110) to daily summary in 'weewx'

Marlon

unread,
Oct 31, 2023, 3:34:53 AM10/31/23
to weewx-user
I nuked the DB, extracted all the data from ecowitt using @comes https://github.com/comes/ecowitt2weewx
I then imported it all back in. The data from ecowitt seems correct, still. My yearly rain so far is 824.1mm as per my station https://www.ecowitt.net/home/index?id=68712

After all of this, I still get about half the rainfall in weewx. (490.3mm) https://weewx.simple-earth.org/reports/?yr=2023

It's likely there is a weird field mapping, or table that I just don't understand. Looking at the amount of rain related issues, I'm not alone. Would like to beat this. 
Please help me save my brain. It's breaking.

gjr80

unread,
Oct 31, 2023, 5:19:18 AM10/31/23
to weewx-user
It can be tricky to maintain identical readings at all destinations for stations that post to multiple sources, ran in particular can be tricky due to it being a (typically) cumulative value whereas other observations are typically averages. Small changes or the loss of a single reading usually have little effect of weather obs average values but loss of a mm or two of rain is noticeable. Timing differences can also cause discrepancies.

You say that 'it underreads rain by a few millimeters everytime' - what does that mean, is every packet received by WeeWX under or is every day, week or month under? How often is your station posting to ecowitt.net and WeeWX? It would be good to see a startup log extract with debug = 1 so we can see your station config. If using your station custom posting/upload you should be seeing regular intercepts from your station in the WeeWX log, are you seeing all posts from your station or are some missing, do all have a rain value or are some missing? Sorry but there is likely going to be a lot of log analysis to track this down.

In terms of wee_import what does your csv file contain, does it contain cumulative rain value or per-archive period values? What does you import config file contain? There is a 'rain' config option in the CSV import config file that determines how rain values are handled - it needs to be set correctly. if the source data is correct tracking down CSV import issues should be very straightforward once we have the data and the config being used. Feel free to post your CSV source data and import config file here or email to me at gjroderick at gmail dot com and I will run through them. 

Gary

Marlon

unread,
Nov 1, 2023, 3:08:32 AM11/1/23
to weewx-user
Hey there Gary
Your response, and the time you spent, is most appreciated. Thank you.

To simplify things, let's ignore the "import" problem, that may (probably is) just user error and config. PS: for what it's worth, my rain data in the csv's are cumulative. So every hour, adds up the total until midnight where it becomes 0.0 again.

You're triggering some ideas here, regarding timing. I will share my startup log head now.
When I know the rain data matches the other sites, ecowitt, WU, Weathercloud... I will disable posting to those services and just use weewx. Even simpler. 

Under-readding seems to be at a daily level (or event, which is what I can humanly notice when I look at the console, WU, ecowitt and weewx). An event would end, and everything would say, eg: 32.5mm, but weewx would be at 26mm. Even during rain, it would be behind the console and other services.

I'm REALLY pondering the timing thing, and I think you're on to something about missing intercepts/timing.

Unfortunately, I am very much in the dark about how these timings work. I use WSView to configure my FineOffset WH clone, and the minimum rate is every 16s (which is my setting).
With what interval and "loop packets" mean, I can't figure it out.

Again, thanks for the reply. 
Startup log follows, ugly spam paste warning:

root@goatbox:/etc/weewx# service weewx start && tail -f /var/log/syslog | grep weewx
Nov  1 09:04:25 goatbox weewx[281764] INFO __main__: Locale is 'C.UTF-8'
Nov  1 09:04:25 goatbox weewx[281764] INFO __main__: Using configuration file /etc/weewx/weewx.conf
Nov  1 09:04:25 goatbox weewx[281764] INFO __main__: Debug is 1
Nov  1 09:04:25 goatbox weewx[281764] INFO __main__: PID file is /var/run/weewx.pid
Nov  1 09:04:25 goatbox weewx[281767] DEBUG __main__: Initializing engine
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Loading station type Interceptor (user.interceptor)
Nov  1 09:04:25 goatbox weewx[281753]:    ...done.
Nov  1 09:04:25 goatbox systemd[1]: Started LSB: weewx weather system.
Nov  1 09:04:25 goatbox weewx[281767] INFO user.interceptor: driver version is 0.60
Nov  1 09:04:25 goatbox weewx[281767] INFO user.interceptor: device type: wu-client
Nov  1 09:04:25 goatbox weewx[281767] INFO user.interceptor: hardware name: weatherstation via interceptor
Nov  1 09:04:25 goatbox weewx[281767] INFO user.interceptor: mode is listen
Nov  1 09:04:25 goatbox weewx[281767] INFO user.interceptor: listen on :8066
Nov  1 09:04:25 goatbox weewx[281767] INFO user.interceptor: sensor map: {'pressure': 'pressure', 'barometer': 'barometer', 'outHumidity': 'humidity_out', 'inHumidity': 'humidity_in', 'outTemp': 'temperature_out', 'inTemp': 'temperature_in', 'windSpeed': 'wind_speed', 'windGust': 'wind_gust', 'windDir': 'wind_dir', 'windGustDir': 'wind_gust_dir', 'radiation': 'solar_radiation', 'dewpoint': 'dewpoint', 'windchill': 'windchill', 'rain': 'rain', 'rainRate': 'rain_rate', 'UV': 'uv', 'txBatteryStatus': 'battery', 'extraTemp1': 'temperature_1', 'extraTemp2': 'temperature_2', 'extraTemp3': 'temperature_3', 'extraHumid1': 'humidity_1', 'extraHumid2': 'humidity_2', 'soilTemp1': 'soil_temperature_1', 'soilTemp2': 'soil_temperature_2', 'soilMoist1': 'soil_moisture_1', 'soilMoist2': 'soil_moisture_2', 'soilMoist3': 'soil_moisture_3', 'soilMoist4': 'soil_moisture_4', 'leafWet1': 'leafwetness_1', 'leafWet2': 'leafwetness_2', 'pm2_5': 'pm2_5', 'extraTemp4': 'temperature_4', 'extraTemp5': 'temperature_5', 'extraTemp6': 'temperature_6', 'extraTemp7': 'temperature_7', 'extraTemp8': 'temperature_8', 'extraHumid3': 'humidity_3', 'extraHumid4': 'humidity_4', 'extraHumid5': 'humidity_5', 'extraHumid6': 'humidity_6', 'extraHumid7': 'humidity_7', 'extraHumid8': 'humidity_8', 'soilTemp3': 'soil_temperature_3', 'soilTemp4': 'soil_temperature_4'}
Nov  1 09:04:25 goatbox weewx[281767] DEBUG user.interceptor: start tcp server
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdTimeSynch
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdTimeSynch
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdConvert
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: StdConvert target unit is 0x10
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdConvert
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdCalibrate
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdCalibrate
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdQC
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdQC
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.wxservices.StdWXCalculate
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.manager: Daily summary version is 4.0
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.wxservices.StdWXCalculate
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdWXXTypes
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdWXXTypes
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdPressureCooker
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdPressureCooker
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdRainRater
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdRainRater
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.wxxtypes.StdDelta
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdDelta
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdArchive
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Archive will use data binding wx_binding
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Record generation will be attempted in 'software'
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Using archive interval of 15 seconds (software record generation)
Nov  1 09:04:25 goatbox weewx[281767] WARNING weewx.engine: Archive delay (15) is unusually long
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Use LOOP data in hi/low calculations: 1
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdArchive
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.restx.StdStationRegistry
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.restx: StationRegistry: Station will be registered.
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.restx.StdStationRegistry
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.restx.StdWunderground
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.restx: Wunderground: Posting not enabled.
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.restx.StdWunderground
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.restx.StdPWSweather
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.restx: PWSWeather: Data for station KLEINKAROO1 will be posted
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.restx.StdPWSweather
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.restx.StdCWOP
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.restx: CWOP: Posting not enabled.
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.restx.StdCWOP
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.restx.StdWOW
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.restx: WOW: Posting not enabled.
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.restx.StdWOW
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.restx.StdAWEKAS
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.restx: AWEKAS: Data will be uploaded for user marlonza
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.restx.StdAWEKAS
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service user.owm.OpenWeatherMap
Nov  1 09:04:25 goatbox weewx[281767] INFO user.owm: service version is 0.9
Nov  1 09:04:25 goatbox weewx[281767] INFO user.owm: Data will be uploaded for 650d5a8e8885c200018ae241
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service user.owm.OpenWeatherMap
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdPrint
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdPrint
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Loading service weewx.engine.StdReport
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: 'pyephem' detected, extended almanac data is available
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Finished loading service weewx.engine.StdReport
Nov  1 09:04:25 goatbox weewx[281767] INFO __main__: Starting up weewx version 4.10.2
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.engine: Station does not support reading the time
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Using binding 'wx_binding' to database 'weewx'
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.manager: Starting backfill of daily summaries
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.manager: Daily summaries up to date
Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Starting main packet loop.
Nov  1 09:04:25 goatbox weewx[281767] DEBUG weewx.manager: Daily summary version is 4.0
Nov  1 09:04:25 goatbox weewx[281767] message repeated 2 times: [ DEBUG weewx.manager: Daily summary version is 4.0]
Nov  1 09:04:27 goatbox weewx[281767] DEBUG user.interceptor: POST: b'PASSKEY=XXXX&stationtype=EasyWeatherV1.6.6&dateutc=2023-11-01+07:04:24&tempinf=69.4&humidityin=54&baromrelin=29.554&baromabsin=29.616&tempf=68.7&humidity=60&winddir=83&windspeedmph=0.2&windgustmph=1.1&maxdailygust=6.9&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.008&monthlyrainin=0.000&yearlyrainin=32.445&totalrainin=32.445&solarradiation=654.99&uv=6&wh65batt=0&freq=868M&model=WS2900_V2.01.08'
Nov  1 09:04:27 goatbox weewx[281767] DEBUG user.interceptor: raw data: b'PASSKEY=censoredC&stationtype=EasyWeatherV1.6.6&dateutc=2023-11-01+07:04:24&tempinf=69.4&humidityin=54&baromrelin=29.554&baromabsin=29.616&tempf=68.7&humidity=60&winddir=83&windspeedmph=0.2&windgustmph=1.1&maxdailygust=6.9&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.008&monthlyrainin=0.000&yearlyrainin=32.445&totalrainin=32.445&solarradiation=654.99&uv=6&wh65batt=0&freq=868M&model=WS2900_V2.01.08'
Nov  1 09:04:27 goatbox weewx[281767] DEBUG user.interceptor: using rain_total 32.445 from yearlyrainin
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter b'PASSKEY=censored
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter stationtype=EasyWeatherV1.6.6
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter tempinf=69.4
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter humidityin=54
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter baromrelin=29.554
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter baromabsin=29.616
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter maxdailygust=6.9
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter rainratein=0.000
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter eventrainin=0.000
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter hourlyrainin=0.000
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter weeklyrainin=0.008
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter monthlyrainin=0.000
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter totalrainin=32.445
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter uv=6
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter wh65batt=0
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter freq=868M
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: unrecognized parameter model=WS2900_V2.01.08'
Nov  1 09:04:27 goatbox weewx[281767] INFO user.interceptor: skipping rain measurement of 32.445: no last rain
Nov  1 09:04:27 goatbox weewx[281767] DEBUG user.interceptor: raw packet: {'dateTime': 1698822264, 'usUnits': 1, 'rain_total': 32.445, 'temperature_out': 68.7, 'humidity_out': 60.0, 'wind_dir': 83.0, 'wind_speed': 0.2, 'wind_gust': 1.1, 'solar_radiation': 654.99, 'rain': None}
Nov  1 09:04:27 goatbox weewx[281767] DEBUG user.interceptor: mapped packet: {'dateTime': 1698822264, 'usUnits': 1, 'outHumidity': 60.0, 'outTemp': 68.7, 'windSpeed': 0.2, 'windGust': 1.1, 'windDir': 83.0, 'radiation': 654.99, 'rain': None}

Note: Rain is set to None on the first iteration. It reverts to 0.0 consequently.

Marlon

unread,
Nov 1, 2023, 3:36:02 AM11/1/23
to weewx-user
Sorry, I see a part of the log that was missing (i had it on wu-client, testing... correct setting is ecowitt-client in interceptor).
Either way, the raw POST contains this:

Nov  1 09:32:58 goatbox weewx[281767] DEBUG user.interceptor: POST: b'PASSKEY=XXXX&stationtype=EasyWeatherV1.6.6&dateutc=2023-11-01+07:32:56&tempinf=71.1&humidityin=54&baromrelin=29.540&baromabsin=29.602&tempf=69.8&humidity=57&winddir=78&windspeedmph=1.8&windgustmph=2.2&maxdailygust=11.4&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.008&monthlyrainin=0.000&yearlyrainin=32.445&totalrainin=32.445&solarradiation=839.34&uv=8&wh65batt=0&freq=868M&model=WS2900_V2.01.08'

Nov  1 09:32:58 goatbox weewx[281767] DEBUG user.interceptor: raw data: b'PASSKEY=660D031A3A77C1332DFA76B2C975573C&stationtype=EasyWeatherV1.6.6&dateutc=2023-11-01+07:32:56&tempinf=71.1&humidityin=54&baromrelin=29.540&baromabsin=29.602&tempf=69.8&humidity=57&winddir=78&windspeedmph=1.8&windgustmph=2.2&maxdailygust=11.4&rainratein=0.000&eventrainin=0.000&hourlyrainin=0.000&dailyrainin=0.000&weeklyrainin=0.008&monthlyrainin=0.000&yearlyrainin=32.445&totalrainin=32.445&solarradiation=839.34&uv=8&wh65batt=0&freq=868M&model=WS2900_V2.01.08'

A few types of rain params there.

Rainer Lang

unread,
Nov 1, 2023, 4:11:48 AM11/1/23
to weewx...@googlegroups.com

a) you should use an interceptor driver version which cleanly handles all your sensors - and also those you don't have
the original version on GitHub was never extended and is a crippled version from a today's point of view as it covers the basic sensors only and its exclusion field-map is incomplete - doesn't match the current customized server protocol sent by a modern Ecowitt console with more than the basic sensors connected - extended, complete and working versions of interceptor.py and extensions.py (for units) exist though
if you use the device_type ecowitt-client as you already found out yourself, the assignment of the rain data should work properly
But that's "beauty talk"

b) what hits my eye is your archiving interval of 15 seconds ...

Nov  1 09:04:25 goatbox weewx[281767] INFO weewx.engine: Using archive interval of 15 seconds (software record generation)

I have no idea what added value you try to get from such a short interval and it may overload the system and be - at least partially - responsible for your issue.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/433cd173-d2f6-466e-9f16-983ef4e772e1n%40googlegroups.com.

Marlon

unread,
Nov 1, 2023, 4:17:23 AM11/1/23
to weewx-user
Hi Rainer
Thank you for the reply. 
You are right, that is an absurd interval (Im not sure if late nights or desperate times caused it).
My console reports (well, POST's) every 16s (the minimum setting). I am going to increase my archive interval to 60s, or perhaps 32s. Not sure if it should be divisible by my console's abilities.

As for the driver, thanks. Didn't know Interceptor was so outdated. Unfortunately, I am not sure what else my froggit/fineOffset WH3000 can use. I couldn't get the standard ones to work.

Appreciated.
Marlon

Rainer Lang

unread,
Nov 1, 2023, 4:27:11 AM11/1/23
to weewx...@googlegroups.com

I think you have to understand better how weewx works:
even if you set a archive interval of 300 seconds (5 minutes), each data packet sent by your console every 16 seconds (shorter doesn't make sense for you anyway as the outdoor sensor array only transmits every 16 seconds) is processed by weewx and added to its Loop data according to the accumulator setup. The "loop" is a table in which all data which occur during an archive interval are collected [summarized, averaged etc.] and the last version is then archived at the end of the archiving interval. So nothing gets lost and you don't need to archive every 16 seconds ...

You can find a proper interceptor.py and extensions.py at
https://pc-wetterstation.de/weewx/usr/share/weewx/user/interceptor.zip
and
https://pc-wetterstation.de/weewx/usr/share/weewx/user/extensions.zip


Rainer Lang

unread,
Nov 1, 2023, 4:34:11 AM11/1/23
to Rainer Lang, weewx...@googlegroups.com

And, to improve your knowledge of your weather station, I recommend reading the WiKi at
https://www.wetterstationsforum.info/wiki/doku.php?id=wiki:wetterstationen:ecowitt-stationen
if you understand German (the Froggit brand may point to that).
If not, try https://www.wxforum.net/index.php?topic=40730.0 in English - not as extensive but the best on the web in English language
by the same author

Message has been deleted
Message has been deleted

Marlon

unread,
Nov 1, 2023, 8:00:35 AM11/1/23
to weewx-user
Reposting to clarify

Thank you Rainer, most helpful.

FWIW, I set the much longer archive, and got everything back to standard.
I cannot test with rain, until it rains, but I did import data from weathercloud, and the final result contains the same error. Year total report is 401 (should be 829). Another example, October reports 23.9mm, but the actual number is 28.4mm

I also previously imported from ecowitt, with the same problem. EcoWitt and WeatherCloud have the data correctly.

It feels like weewx is storing rain in a way I don't understand. Or, the wee_import suffers from the same missing issue as normal interceptor.

Edit 2: It seems almost surely that the import has the same erroneous result as normal operation.
If you want to see what I mean: https://weewx.simple-earth.org/reports/?yr=2023 <- 401mm for the year vs over 800 on ecowitt (which is also what WU and my console and my plastic rainguage agrees on).
I set import to cumulative, so midnight resets were valid. 

WeeWX is storing rain somewhere and forgetting about it when it creates graphs and reports? :D
Sorry, I am as desperate to solve this as I am confused. Two full days on this now, feeling the pain.

gjr80

unread,
Nov 1, 2023, 8:40:22 AM11/1/23
to weewx-user
As I said earlier; post or email the data you are importing along with the import config file. The import process is deterministic and relatively straightforward to troubleshoot.

Gary

Rainer Lang

unread,
Nov 1, 2023, 12:46:03 PM11/1/23
to weewx...@googlegroups.com

1. as Gary (@gjr80) already mentioned - send us your import.conf file for the CSV import and maybe 5 lines of your import CSV file including the first line with the descriptors

2. did you run the daily summary update after the import ?
sudo wee_database ...... from=    to= (see wee_database --help)

3. did you compare the values from the import CSV file with what was written to the database ?
(download/install sqlitebrowser unless already installed and start it with "sudo sqlitebroswer /var/lib/weewx/weewx.sdb"
(or whatever is the path to your weewx database) and compare the added records:
convert your date/time stamp into EPOCH to find the corresponding record in the first database column)

gjr80

unread,
Nov 2, 2023, 12:59:31 AM11/2/23
to weewx-user
On Thursday, 2 November 2023 at 02:46:03 UTC+10 lang....@googlemail.com wrote:

1. as Gary (@gjr80) already mentioned - send us your import.conf file for the CSV import and maybe 5 lines of your import CSV file including the first line with the descriptors

No, please include the whole CSV file; if the issue is the accuracy of the totals we are going to need to see all of the source data, not just a few lines.

2. did you run the daily summary update after the import ?
sudo wee_database ...... from=    to= (see wee_database --help)

There is no need to update the daily summaries after importing via wee_import, wee_import automatically updates the daily summaries once the data has been imported into the archive. 

Gary

Marlon

unread,
Nov 4, 2023, 1:33:13 PM11/4/23
to weewx-user
Hi guys
Thanks, and apologies for a latent reply. Life got hold of me for a while.
I ran the daily summaries and reweighting out of desperation, but as you mentioned, they seemed to be done auto by wee_import

I am attaching the gzipped versions of the conf and csv. This particular CSV is the latest attempt I made, and has 10 min intervals (weather cloud). Previously I did the same with ecowitt (but with their varying intervals).
Everything seems perfect, except rain - rain is wrong from either sites (yet displayed correctly ON those sites). I will also share the links to those if that helps to prove my sanity (or lack thereof).
My WeeWX site: weewx.simple-earth.org (rain wrong).
My Station on Ecowitt www.ecowitt.net/home/index?id=68712 (rain shown correctly and matching manual meters)

Thanks Gary and Rainer. Appreciated.
Marlon
weewx_import.csv.gz
import_csv.conf.gz

Rainer Lang

unread,
Nov 4, 2023, 8:23:37 PM11/4/23
to weewx...@googlegroups.com

can you remove all lines with ";;;;;;;;;;;;;;" from the import CSV file and start the import again (I assume you still have an untouched copy of your database),
re-create the NOAA reports and check again ...
The NOAA reports from your web site is the only place where I could compare the import file content with your web site database.
Obviously at certain days no data or not all data of that day were imported.
Can you also provide the syslog from this import (or from your last import) - maybe for certain timestamps already a record existed and wee_import couldn't create this record again.
wee_import only creates new records - doesn't update existing records !

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Marlon

unread,
Nov 5, 2023, 9:44:56 AM11/5/23
to weewx-user
Okie dokes.
I re-exported data from ecowitt. 4 hour intervals. I cleaned up the CSV
Then confirmed import conf is correct, confirmed row count ( | wc -l| and deleted my sqlite db so we can load from scratch.
Import seemed fine on dry run, so I launched import. Rain total is yet again, different. This time it is 470mm (still not the 800+ it should be).
Rain config set to cumulative (since ecowitt does not export yearly).

Sharing the new config, csv and logs as you requested. You should be able to compare now, Rainer.
Hope I can find a way to thank you and Gary for the help.

2023-ready-for-wee-import.csv
import_csv.conf.txt
syslog grep weewx - during import.txt
stdout during import.txt

Rainer Lang

unread,
Nov 5, 2023, 2:59:37 PM11/5/23
to weewx...@googlegroups.com

I have no idea why it should be 800+ mm
I did the work deleting all 0.0 mm value records and left for non 0.0 days only the last record (usually either 18:00 or 22:00 whatever was available)
and added them all up - result: 507.2 mm as opposite to the 470.2 mm shown in your graph and NOAA. Not 800+

PRECIPITATION (mm)

                  MAX         ---DAYS OF RAIN---
                  OBS.               OVER
 YR  MO  TOTAL    DAY  DATE   0.30   3.00  30.00
------------------------------------------------
2023 01   21.6   17.3    05      2      2      0
2023 02    6.3    2.8    20      6      0      0
2023 03  100.8   56.4    05      7      4      2
2023 04   13.0    4.6    22      5      3      0
2023 05  111.2   56.9    31     13      7      1
2023 06   79.4   22.6    15     15      5      0
2023 07   35.9   14.7    09     12      3      0
2023 08    6.6    2.0    23      5      0      0
2023 09   68.3   48.5    25      7      3      1
2023 10   27.1   11.4    21      5      4      0
2023 11    0.0    0.0    01      0      0      0
2023 12
------------------------------------------------
         470.2   56.9   May     77     31      4

also, your NOAA reports are not consistent - e.g. sum Oct 23 in the yearly view: 27.1, in the Oct 23 monthly view: 23.9

Going through the import field map and the import  CSV file,I cannot see any reason where this (meanwhile as far as I could tell in a first rough check) only 35 mm difference comes from.

All these consistency checks you could have done yourself and come to the same results as I did, couldn't you ?
I attach the Excel file I created from your import.csv - the other columns are hidden (mark adjacent columns --> Format --> Hide/Unhide)
No guarantee I missed some lines - but you can repeat the approach - and make sure all your NOAA reports are consistent amongst each other in the first place.

As a practical/pragmatic approach I suggest you complete the Excel for each month - create an additional import file with a time stamp one minute later than the last one each month (using the other data from the last record) with the missing amount and import it
Or use sqlitebrowser as suggested earlier to add the missing values of the respective days to the day already imported.
When you use sqlitebrowser, you will need to update the daily summaries (wee_database)
In the end, at least you will have a consistent database.

If Gary or someone else has the time to analyze the data more deeply to figure out why wee_import does an incomplete job, they shall feel free to do so

weewx-import-csv-test-rain-only.xlsx

gjr80

unread,
Nov 5, 2023, 11:48:06 PM11/5/23
to weewx-user
I have spent this morning working through the csv data, the NOAA format reports on your web site and the data on Ecowitt.net. There are a couple of issues going on. First, the bulk of the difference between the Ecowitt.net yearly rainfall (presently showing 824.1mm) and that imported by wee_import (approx 507mm) is due to data inconsistencies on Ecowitt.net. The yearly rainfall value displayed by Ecowitt.net is taken from the yearly rain field (presumably uploaded by the station). If you look through the 2023 tabulated data for the station on Ecowitt.net you will see the yearly rainfall value was non-existent until 6 June when it took on the value 585.1mm, but if you sum the monthly rainfall values from 1 January until 6 June you get something of the order of 254mm (it is impossible to say exactly due to discrepancies between the values in the various Ecowitt.net rain fields). This accounts for some 330mm of the difference. I have also found at least one place where the Ecowitt.net rain data goes backwards (23 February 06:00 monthly rainfall is 5.6mm, the 10:00 entry indicates 0.8mm fell taking the monthly total to 6.6mm, the 14:00 entry shows no rain but the monthly value jumps back to 5.6mm), whilst this may or may not impact the import (it depends on how the import is setup) it begs the question are there other data discrepancies that may impact the import. I haven't yet checked for any other similar discrepancies.

I have also noticed discrepancies in the NOAA format reports. For some reason the 2023 report shows the March rainfall as 100.8mm, whereas the March 2023 report shows a monthly total of 105.6mm. The archive and daily summary tables both have 105.6mm for March, but for some reason the monthly report is off. Ecowitt.net also shows 105.6mm for March. Similarly in October. The October 2023 report shows 28.4mm but the 2023 reports shows October with 27.1mm, the archive and daily summary tables both record 28.4mm. I will have to look at this further, but at present I would be wary of rainfall in the yearly NOAA reports; the monthly reports seem to show the true value.

So far I have not been able to fault wee_import. 

Gary

Rainer Lang

unread,
Nov 6, 2023, 2:04:02 AM11/6/23
to weewx...@googlegroups.com

summary data shown on the ecowitt.net dashboard are the values from the console and can be changed by the user in the console. At ecowitt.net no summarizing is done for the displayed tiles, just display of what the console sends. What is shown there is user responsibility. If the user makes adjustments, they are simply taken over and displayed. => you can only conditionally expect accuracy in the summary counts - it may or may not not match the really stored 5-minute (or longer period summarized data depending on the retention policy) data. (we can see the discrepancies between import summaries, NOAA summaries and dashboard tile summaries)

If there is a discrepancy between this summary data shown on the dashboard tiles and the summarized end-of-day data for rain, this is most likely user caused. (800+ vs. 470)
Data from the import was all properly stored - discrepancies are likely to be user caused and not we_import caused.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

Marlon

unread,
Nov 6, 2023, 2:18:45 AM11/6/23
to weewx-user
Thank you Rainer and Gary 
The effortnis greatly appreciated and I will find a way to give back to weewx community for this. 
There's a lot here on the last posts that have given me a lot of food for thought. Im going to have to launch a huge project here to find the way these other sites have created discrepancies and how I can fix that to get the data normalised. 

The 824mm is the amount that matches my plastic guage and neighbouring farms, So it's the number I need to work towards. 

Once I have that figured out, there's still the original question of whether the new events into wee will work after the interval change. Strangest thing is getting the same weirdness from Wu, weathercloud, ecowitt, etc. That does make the console sound guilty, but with that said, that number is most correct. Confusing. 

But I'll then post the findings here, and possibly some code so that this thread can have the usefulness to others in the future. 

Thank you
Marlon

Rainer Lang

unread,
Nov 6, 2023, 8:19:17 AM11/6/23
to weewx...@googlegroups.com

Hi Marlon

three more remarks:
1. comparing to neighbouring environment may not always be helpful - as rainfall is not a homogeneous phenomenon and 100 m away from your sensor there may even be no rainfall at all - or vice versa. A proper manual gauge close to your sensor makes more sense.

2. your WH65 arrray with its rain gauge may not be at the same height as your manual gauge ! To be considered.
Did you clean your rain gauge (WH65) properly ?

3. if you are looking for more accuracy:
get yourself a WH40 with a rim height extension (https://www.wxforum.net/index.php?topic=44074.0) and a GW1100 (or GW2000) as your WH2910 console (Froggit WH3000SE) cannot receive the WH40 - mount it at 1-1.5 m above ground and put your manual gauge close to it - same height.
Use a calibration factor of 1.02 for the WH40 - due to imperial/metric conversion.

And then you are much better set for your endeavours.

Your issue(s) is(are) in my opinion not weewx related but console/sensor/location related.

Reply all
Reply to author
Forward
0 new messages