QC not passed

180 views
Skip to first unread message

Francis Perea

unread,
Jan 5, 2025, 4:16:27 PMJan 5
to weewx-development
Hi all, 

I'm having a problem with my rain sensor and I get frequent messages notifying values outside limits after QC.

To be exact the function apply_qc in qc.py detects the wrong value and send a log like this:

WARNING weewx.qc: 2025-01-05 17:45:00 CET (1736095500) Archive value 'rainRate' 13.841245901639354 outside limits (0.0, 3.0)

As I understand the apply_qc function should remove the invalid reading after detecting it with the instruction 

data_dict[obs_type] = None

in qc.py line 72, but the wrong value is even so archived.

Any clue or any other way to remove the wrong observation once it is detected by apply_qc?

Thanks a lot in advance.

Graham Eddy

unread,
Jan 5, 2025, 9:10:22 PMJan 5
to Francis Perea, weewx-development
make sure the value you put in QC section is in same units as the native units in database (prob US’s quaint unit system)
[this should probably go to weewx-users group rather than weewx-development]
cheers
⊣GE⊢

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-development/c37bfae1-5beb-4178-ab1b-69423c008d49n%40googlegroups.com.

Francis Perea

unread,
Jan 9, 2025, 7:09:35 AMJan 9
to weewx-development
Hi Graham, thanks a lot for your reply.

I have the same units, I checked it already.

I post it here because I think it is a bug and the line 

data_dict[obs_type] = None

 in apply_qc() doesn't really clean the observation and it is finally written to the database.

In any case I'll post it in the user group.

Thanks once again.



Tom Keffer

unread,
Jan 9, 2025, 2:42:24 PMJan 9
to Francis Perea, weewx-development
We need the log to offer much advice. See How to get a good, useful log in the Wiki.

Also, what field are you checking in the database? Field "rain"? Or, "rainRate"?

--

Francis Perea

unread,
Jan 9, 2025, 2:58:04 PMJan 9
to weewx-development
Hi Tom, thanks a lot for your reply.

Here you have my log. 

The debug lines regarding the wind speed and vane and the "Discarding Observation" lines are mine. I had to find where the code was getting, and that's why I think the apply_qc() function is not rejecting the wrong observation.

By the moment I'm checking rain, but I've also made some testings with rainRate.

Thanks once again for your help.

ene 09 20:52:17 meteopi weewxd[28490]: INFO __main__: Starting up weewx version 5.1.0
ene 09 20:52:17 meteopi weewxd[28490]: DEBUG weewx.engine: Station does not support reading the time
ene 09 20:52:17 meteopi weewxd[28490]: INFO weewx.engine: Using binding 'wx_binding' to database 'weewx'
ene 09 20:52:17 meteopi weewxd[28490]: INFO weewx.manager: Starting backfill of daily summaries
ene 09 20:52:17 meteopi weewxd[28490]: INFO weewx.manager: Daily summaries up to date
ene 09 20:52:17 meteopi weewxd[28490]: INFO weewx.engine: Starting main packet loop.
ene 09 20:52:17 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:17 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.23
ene 09 20:52:17 meteopi weewxd[28490]: DEBUG user.byows_rpi: Resolved Wind Vane Direction: 225.0
ene 09 20:52:22 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:22 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.22
ene 09 20:52:22 meteopi weewxd[28490]: DEBUG user.byows_rpi: Unknown Wind Vane value: 0.22
ene 09 20:52:27 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:27 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.23
ene 09 20:52:27 meteopi weewxd[28490]: DEBUG user.byows_rpi: Resolved Wind Vane Direction: 225.0
ene 09 20:52:32 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:32 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.23
ene 09 20:52:32 meteopi weewxd[28490]: DEBUG user.byows_rpi: Resolved Wind Vane Direction: 225.0
ene 09 20:52:37 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:37 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.22
ene 09 20:52:37 meteopi weewxd[28490]: DEBUG user.byows_rpi: Unknown Wind Vane value: 0.22
ene 09 20:52:37 meteopi weewxd[28490]: WARNING weewx.qc: 2025-01-09 20:52:38 CET (1736452358) LOOP value 'rain' 0.616 outside limits (0.0, 0.3937007874015748)
ene 09 20:52:37 meteopi weewxd[28490]: WARNING weewx.qc: Discarding observation
ene 09 20:52:42 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:42 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.22
ene 09 20:52:42 meteopi weewxd[28490]: DEBUG user.byows_rpi: Unknown Wind Vane value: 0.22
ene 09 20:52:47 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:47 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.22
ene 09 20:52:47 meteopi weewxd[28490]: DEBUG user.byows_rpi: Unknown Wind Vane value: 0.22
ene 09 20:52:52 meteopi weewxd[28490]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 09 20:52:52 meteopi weewxd[28490]: DEBUG user.byows_rpi: Read Wind Vane value: 0.22
ene 09 20:52:52 meteopi weewxd[28490]: DEBUG user.byows_rpi: Unknown Wind Vane value: 0.22

Tom Keffer

unread,
Jan 9, 2025, 4:51:14 PMJan 9
to Francis Perea, weewx-development
Did you edit the log? There should be a lot more information than that.

To view this discussion visit https://groups.google.com/d/msgid/weewx-development/b5977ef9-7181-4ecc-9bf3-b609c4c9324en%40googlegroups.com.

Francis Perea

unread,
Jan 9, 2025, 4:53:58 PMJan 9
to weewx-development
Hi Tom,

No, I just set 

sudo journalctl -u weewx -f | tee /var/tmp/mylog

And restarted WeeWX, and stop the log once I got the first message from QC

Do you need any thing more about the log?

Thanks once again for your support

Tom Keffer

unread,
Jan 9, 2025, 7:48:21 PMJan 9
to Francis Perea, weewx-development
There should be a lot more than that. Did you try /var/log/syslog?

To view this discussion visit https://groups.google.com/d/msgid/weewx-development/daa36ff8-4154-499d-9a03-9b89eef3703cn%40googlegroups.com.

Francis Perea

unread,
Jan 10, 2025, 1:46:28 AMJan 10
to weewx-development
Hi Tom,

No, I didn't edit it. Perhaps it is shorter due that I made a restart. Here you have a full stop/start log of the service.

Are you looking for something specific? What should you expect?

For me the most important part is the line reporting the weewx.qc service value outside limits, and the fact that the instruction in apply_qc() that says 

data_dict[obs_type] = None

after that conditional doesn't really clean the value and the wrong observation in archived in the data base.

Is there any other thing I can do to reject that value once detected? Should I write my own service to discard those values?

Thank once again for your interest.

ene 10 07:39:25 meteopi systemd[1]: weewx.service: Consumed 3.297s CPU time.
ene 10 07:39:58 meteopi systemd[1]: Started weewx.service - WeeWX.
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Initializing weewxd version 5.1.0
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Command line: /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Using Python: 3.11.2 (main, Sep 14 2024, 03:00:30) [GCC 12.2.0]
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Located at:   /usr/bin/python3
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Platform:     Linux-6.6.62+rpt-rpi-v8-aarch64-with-glibc2.36
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Locale:       'es_ES.UTF-8'
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Entry path:   /usr/share/weewx/weewxd.py
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: WEEWX_ROOT:   /etc/weewx
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Config file:  /etc/weewx/weewx.conf
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: User module:  /etc/weewx/bin/user
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Debug:        1
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: User:         weewx
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Group:        weewx
ene 10 07:39:58 meteopi weewxd[54432]: INFO __main__: Groups:       weewx gpio i2c spi
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG __main__: loop_on_init: False
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG __main__: Initializing engine
ene 10 07:39:58 meteopi weewxd[54432]: INFO weewx.engine: Loading station type BYOWS (user.byows_rpi)
ene 10 07:39:58 meteopi weewxd[54432]: INFO user.byows_rpi: using driver BYOWS
ene 10 07:39:58 meteopi weewxd[54432]: INFO user.byows_rpi: driver version is 0.51
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdTimeSynch
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdTimeSynch
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: No services in service group data_services
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdConvert
ene 10 07:39:58 meteopi weewxd[54432]: INFO weewx.engine: StdConvert target unit is 0x1
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdConvert
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdCalibrate
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdCalibrate
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdQC
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdQC
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.wxservices.StdWXCalculate
ene 10 07:39:58 meteopi weewxd[54432]: INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.manager: Daily summary version is 1.0
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.wxservices.StdWXCalculate
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.wxxtypes.StdWXXTypes
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdWXXTypes
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.wxxtypes.StdPressureCooker
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdPressureCooker
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.wxxtypes.StdRainRater
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdRainRater
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.wxxtypes.StdDelta
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.wxxtypes.StdDelta
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdArchive
ene 10 07:39:58 meteopi weewxd[54432]: INFO weewx.engine: Archive will use data binding wx_binding
ene 10 07:39:58 meteopi weewxd[54432]: INFO weewx.engine: Record generation will be attempted in 'hardware'
ene 10 07:39:58 meteopi weewxd[54432]: INFO weewx.engine: Using archive interval of 300 seconds (specified in weewx configuration)
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Use LOOP data in hi/low calculations: 1
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdArchive
ene 10 07:39:58 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.restx.StdStationRegistry
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.restx: StationRegistry: Registration not requested.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.restx.StdStationRegistry
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.restx.StdWunderground
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.restx: Wunderground: Posting not enabled.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.restx.StdWunderground
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.restx.StdPWSweather
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.restx: PWSweather: Posting not enabled.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.restx.StdPWSweather
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.restx.StdCWOP
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.restx: CWOP: Posting not enabled.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.restx.StdCWOP
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.restx.StdWOW
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.restx: WOW: Posting not enabled.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.restx.StdWOW
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.restx.StdAWEKAS
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.restx: AWEKAS: Posting not enabled.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.restx.StdAWEKAS
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdPrint
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdPrint
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service weewx.engine.StdReport
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.engine: 'pyephem' detected, extended almanac data is available
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service weewx.engine.StdReport
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Loading service user.alarm.MyAlarm
ene 10 07:39:59 meteopi weewxd[54432]: INFO user.alarm: Alarm set for expression: 'windSpeed > 8'
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Finished loading service user.alarm.MyAlarm
ene 10 07:39:59 meteopi weewxd[54432]: INFO __main__: Starting up weewx version 5.1.0
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG weewx.engine: Station does not support reading the time
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.engine: Using binding 'wx_binding' to database 'weewx'
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.manager: Starting backfill of daily summaries
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.manager: Daily summaries up to date
ene 10 07:39:59 meteopi weewxd[54432]: INFO weewx.engine: Starting main packet loop.
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG user.byows_rpi: Read Wind Vane value: 0.23
ene 10 07:39:59 meteopi weewxd[54432]: DEBUG user.byows_rpi: Resolved Wind Vane Direction: 225.0
ene 10 07:40:04 meteopi weewxd[54432]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 10 07:40:04 meteopi weewxd[54432]: DEBUG user.byows_rpi: Read Wind Vane value: 0.23
ene 10 07:40:04 meteopi weewxd[54432]: DEBUG user.byows_rpi: Resolved Wind Vane Direction: 225.0
ene 10 07:40:09 meteopi weewxd[54432]: DEBUG user.byows_rpi: Wind speed: 0.0
ene 10 07:40:09 meteopi weewxd[54432]: DEBUG user.byows_rpi: Read Wind Vane value: 0.22
ene 10 07:40:09 meteopi weewxd[54432]: DEBUG user.byows_rpi: Unknown Wind Vane value: 0.22
ene 10 07:40:09 meteopi weewxd[54432]: WARNING weewx.qc: 2025-01-10 07:40:09 CET (1736491209) LOOP value 'rain' 8.096 outside limits (0.0, 0.3937007874015748)

Tom Keffer

unread,
Jan 10, 2025, 6:52:49 PMJan 10
to Francis Perea, weewx-development
That is better, but is the entry for "weewx.qc" the last entry? There should be more. Please follow the directions in the link I sent earlier. We need to see the results through the first reporting cycle. The reason why is that we need to see if the values in the archive records are being discarded.

Also, please show the results in the database for the corresponding periods. This should include both the archive table, and also the daily summaries. 



To view this discussion visit https://groups.google.com/d/msgid/weewx-development/37fd0211-8769-46e8-9c5a-940119678e2bn%40googlegroups.com.

Francis Perea

unread,
Jan 11, 2025, 3:00:39 PMJan 11
to weewx-development
Hi Tom, sorry for the inconvenience. I thought you just needed the starting of the service.

Here you have a full one with at least 3 cycles.


Unfortunately in this one I wasn't able to make the rain sensor fail, so there aren't any StdQC warnings, but as you saw in my previous posts the warnings are like this:

ene 11 20:30:16 meteopi weewxd[121652]: WARNING weewx.qc: 2025-01-11 20:30:00 CET (1736623800) Archive value 'rain' 0.583 outside limits (0.0, 0.3937007874015748)

Fot that record particularly the stored values in the data base are these:
dataTime       rain         rainRate
1736623800 NULL 2.491133333333332

My configuration in the StdQC service after several test is this by the moment:

[StdQC]
   
    [[MinMax]]
        barometer = 26, 32.5, inHg
        pressure = 24, 34.5, inHg
        outTemp = -40, 120, degree_F
        inTemp = 10, 120, degree_F
        outHumidity = 0, 100
        inHumidity = 0, 100
        windSpeed = 0, 120, mile_per_hour
        rain = 0, 10, mm
      
    [[RainRate]]
        min = 0
        max = 3
        min_delta = 0
        max_delta = 3
        reject = True

I hope it helps.

Thanks once again for your time and support.

Tom Keffer

unread,
Jan 11, 2025, 5:56:00 PMJan 11
to Francis Perea, weewx-development
It looks like the field "rain" was nulled out (otherwise, it would be 0.0, not NULL), so no bug.

The field rainRate is not null, however the fields "rain" and "rainRate" are completely different. The former represents the amount of rain that fell, the latter how fast it fell. The field rainRate may be emitted from your device, in which case, by default, WeeWX uses that value. I suspect your byows hardware does not do this, so WeeWX will calculate it using a running average over 15 minutes. So, if you get a bad rain value, it will continue to "pollute" the rainRate value for up to 15 minutes afterwards. You need to filter rainRate as well, or set its running average window, given by option rain_period, to something shorter.

[StdQC]

   
    [[RainRate]]
        min = 0
        max = 3
        min_delta = 0
        max_delta = 3
        reject = True

I don't know what any of this is. Did you write an addition to StdQC? If so, you're going to have to debug it!

-tk

Francis Perea

unread,
Jan 12, 2025, 12:51:15 PMJan 12
to weewx-development
Hi Tom, 

I get what you mean. I have been making all kind of tests to avoid wrong observations to get into the data base and I even think I began setting min-max values for rainRate, but I'll try again.

What I get from your excellent explanation is that I should try to filter rainRate, not rain, and see if that keeps "pollution" out of the DB, isn't it?

I think I'll use this:

rainRate = 0, 3

Do you think it would be enough? Or would you suggest me any other way of filtering? Do you think I should also try to modify the rain_period parameter from its default 1800 seconds?

Regarding the [[RainRate]] section you see in my config, I've to be honest and say that because of my desperation I even asked ChatGPT how to proceed and this is what he propossed. I have already removed it.

Thanks again for your support and patience.

Francis Perea

unread,
Jan 13, 2025, 11:17:44 AMJan 13
to weewx-development
Hi again Tom,

Definitely not enough with my new StdQC configuration. I still get wrong observations into my DB.

Just now I have this in my weewx.conf:

[StdQC]
   
    [[MinMax]]

        rain = 0, 10, mm
        rainRate = 0, 3, mm_per_hour 

And I still get messages like these:
ene 13 16:10:17 meteopi weewxd[193488]: WARNING weewx.qc: 2025-01-13 16:10:00 CET (1736781000) Archive value 'rainRate' 4.046533333333339 outside limits (0.0, 0.11811023609999999)

ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: 2025-01-13 17:00:00 CET (1736784000) Archive value 'rain' 0.539 outside limits (0.0, 0.3937007874015748)
ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: Discarding observation
ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: 2025-01-13 17:00:00 CET (1736784000) Archive value 'rainRate' 4.399999999999997 outside limits (0.0, 0.11811023609999999)
ene 13 17:00:17 meteopi weewxd[193488]: WARNING weewx.qc: Discarding observation

Which generate these values into my DB:

dateTime               rain                                          rainRate
1736781000 0.09899999999999998 3.079999999999999
1736784000 NULL                                 4.3119999999999985

It seems that the rainRate filter is not correct or not filtering.

Any clue?

Thanks once again for your help.


Tom Keffer

unread,
Jan 14, 2025, 7:52:52 PMJan 14
to Francis Perea, weewx-development
I am sorry, Francis, but I am unable to reproduce this.

I modified a copy of the simulator to emit an out-of-range value for "rain" randomly about 5% of the time. After letting it run for 40 minutes, I looked at the database. There was nothing but zeros in both the archive table and in the daily summaries:

sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), rain, rainRate from archive;

2025-01-14 16:10:00|0.0|0.0

2025-01-14 16:15:00|0.0|0.0

2025-01-14 16:20:00|0.0|0.0

2025-01-14 16:25:00|0.0|0.0

2025-01-14 16:30:00|0.0|0.0

2025-01-14 16:35:00|0.0|0.0

2025-01-14 16:40:00|0.0|0.0

2025-01-14 16:45:00|0.0|0.0

sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), sum, max from archive_day_rain;

2025-01-14 00:00:00|0.0|0.0

sqlite> select datetime(dateTime, 'unixepoch', 'localtime'), sum, max from archive_day_rainRate;

2025-01-14 00:00:00|0.0|0.0


There is something peculiar about your setup, but I don't know what it is. If you can send me a configuration that reproduces the behavior by using the simulator, I'm happy to take another look.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-development/91ec157e-5f06-406b-8c16-f55fb40cd694n%40googlegroups.com.

Francis Perea

unread,
Jan 15, 2025, 5:14:14 PMJan 15
to weewx-development
Hi Tom, 

I really appreciate all your efforts.

I don't really know how I could send you that simulator configuration. What would you need? My weewx.conf?

I have discovered that it is not  really a fault of my sensor but some interferences from a neighbour ham radio station. Talking with him he made me a comment and later we have confirmed that every time he transmit in a concrete frequency my station get wrong observations. At least I know now where the faults come from. And more than before I think that a correct software QC is the better way to keep wrong values to get into de DB because it seems more complicate to me to avoid the interferences.

I would really need to know how to prevent those values from getting into the database and still think that a correct StdQC service configuration would do the trick but I don't know how.

Would you please suggest any other configuration or tests I could develop to try to trim my config?

Thanks once again for all your time and interest.

Regards,

Francis

Tom Keffer

unread,
Jan 15, 2025, 6:46:43 PMJan 15
to Francis Perea, weewx-development
I don't think you've posted your weewx.conf. It could prove useful. Make sure you obfuscate any passwords first. Or, use the utility weectl debug.

Yes, if this is a bug in WeeWX, it should be possible to reproduce it using the simulator. This is what I used (diff below). It emits a bad value 5% of the time, zero the rest of the time. 

===================================================================
diff --git a/src/weewx/drivers/simulator.py b/src/weewx/drivers/simulator.py
--- a/src/weewx/drivers/simulator.py (revision 1566a768af0963a43bd1c7b174293c46226a7fb5)
+++ b/src/weewx/drivers/simulator.py (date 1736899735080)
@@ -249,6 +249,11 @@
         else:
             self.packet_number = 0
             amt = 0
+        import random
+        if random.randint(0, 100) > 95:
+            amt = 10
+        else:
+            amt = 0
         return amt
 

For StdQC, I used
[[MinMax]]
    rain = 0, 10, mm
    rainRate = 0, 3, mm_per_hour
-tk

 

To view this discussion visit https://groups.google.com/d/msgid/weewx-development/19706d14-3e4d-466a-9455-f438d2d4b7d2n%40googlegroups.com.

Francis Perea

unread,
Jan 16, 2025, 2:53:22 PMJan 16
to weewx-development
Hi Tom,

Here you have my weectl debug output:


I've been able to force the simulator to generate plenty of wrong observations for rain and rainRate by simply setting these two lines in lines number 131 of the simulator.py file:

 'rain' : Observation(magnitude=110.0,  average= 220.0, period=1.0, phase_lag= 0.0, start=start_ts),
 'rainRate' : Observation(magnitude=110.0,  average= 220.0, period=1.0, phase_lag= 0.0, start=start_ts),

But only for LOOP, and they don't go into the database:

ene 16 20:35:15 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:16 CET (1737056116) LOOP value 'rain' 323.92866511135657 outside limits (0.0, 0.3937007874015748)
ene 16 20:35:15 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:16 CET (1737056116) LOOP value 'rainRate' 323.92866511135657 outside limits (0.0, 0.11811023609999999)
ene 16 20:35:15 meteopi weewxd[166178]: INFO weewx.manager: Added record 2025-01-16 20:35:00 CET (1737056100) to database 'weewx'
ene 16 20:35:16 meteopi weewxd[166178]: INFO weewx.manager: Added record 2025-01-16 20:35:00 CET (1737056100) to daily summary in 'weewx'
ene 16 20:35:18 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:18 CET (1737056118) LOOP value 'rain' 323.7704250868808 outside limits (0.0, 0.3937007874015748)
ene 16 20:35:18 meteopi weewxd[166178]: WARNING weewx.qc: 2025-01-16 20:35:18 CET (1737056118) LOOP value 'rainRate' 323.7704250868808 outside limits (0.0, 0.11811023609999999)

MariaDB [weewx]> select dateTime, rain, rainRate from archive where dateTime >= 1737056100 order by dateTime limit 10;
+------------+------+----------+
| dateTime   | rain | rainRate |
+------------+------+----------+
| 1737056100 | NULL |        0 |
| 1737056400 |    0 |        0 |
| 1737056700 |    0 |        0 |
| 1737057000 |    0 |        0 |
+------------+------+----------+
As you can see the rain value is correctly set to Null.

In real working the StdQC errors come not only for LOOP but also for Archive, that is where I think the problem comes, but with the simulator I haven't been able to force those warnings. These are real warnings for both LOOP and Archive:

ene 16 19:27:22 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:27:22 CET (1737052042) LOOP value 'rain' 0.693 outside limits (0.0, 0.3937007874015748)
ene 16 19:27:27 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:27:27 CET (1737052047) LOOP value 'rain' 0.42899999999999994 outside limits (0.0, 0.3937007874015748)
ene 16 19:30:18 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:30:00 CET (1737052200) Archive value 'rain' 6.578 outside limits (0.0, 0.3937007874015748)
ene 16 19:30:18 meteopi weewxd[146447]: WARNING weewx.qc: 2025-01-16 19:30:00 CET (1737052200) Archive value 'rainRate' 50.06262295081965 outside limits (0.0, 0.11811023609999999)

And those coming from Archive are the ones that put garbage into the DB:

MariaDB [weewx]> select dateTime, rain, rainRate from archive where dateTime >= 1737052200 order by dateTime limit 10;
+------------+------+--------------------+
| dateTime   | rain | rainRate           |
+------------+------+--------------------+
| 1737052200 | NULL |  52.58000000000002 |
| 1737052500 |    0 |             40.392 |
| 1737052800 |    0 | 26.311999999999998 |
| 1737053100 |    0 |                  0 |
| 1737053400 |    0 |                  0 |
| 1737053700 |    0 |                  0 |
| 1737054000 |    0 |                  0 |
| 1737054300 |    0 |                  0 |
| 1737054600 |    0 |                  0 |
| 1737054900 |    0 |                  0 |
+------------+------+--------------------+

I hope you can see something I'm not able to get.

Thanks really once again. 
Reply all
Reply to author
Forward
0 new messages