StdCalibrate/Corrections isn't following the logic I set for lightning_distance

[Andrew McGinnis]

I just set up an Acurite Atlas with the lightning detector sensor add-on.  The strike_count that it outputs is cumulative (after I found I had 5000+ lightning strikes per 1min archive interval). I have it, and the strike_distance outputs mapped to weewx as below. rtl_433 is converting strike_distance from the imperial 'miles' that is natively output, to metric 'km', hence the additional units switch:

        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count]]]
            name = lightning_strike_count
            contains_total = true

        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance]]]
            name = lightning_distance
            units = km

The sensor output always has a strike count and distance, but since the strike count is cumulative, there should be no recorded distance if the delta is 0.  

In StdCalibrate/Corrections, I have these:

radiation = luminosity / 126.7 if luminosity is not None else None (<----this is working as expected)

lightning_distance = lightning_distance if lightning_strike_count > 0 else float('nan')  (<----does not work)

lightning_distance = lightning_distance if lightning_strike_count > 0 else None (<----does not work)

lightning_distance = lightning_distance if lightning_strike_count > 0 else Null (<----does not work)

The radiation observation is being recorded as 0 when luminosity is recorded as 0. But, since 0 is a valid possible output, if 0 != None, then 0/126.7 = 0. Fair enough.

The lightning_distance should be evaluating as 0 or None, but instead is being recorded as the sensor's output value, while lightning_strike_count is recorded as 0. I went into the db and ran the following, then rebuilt the affected dailies:

• update archive set lightning_strike_count = 0 where lightning_strike_count > 5;
• update archive set lightning_distance = 0 where lightning_strike_count = 0;

strike_count > 5 was because I caught my contains_total omission mid day yesterday, and there was thunder last night. Strikes before the storm were largely invalid, but distance wasn't necessarily. 

Straight away, lightning_distance is being recorded as the sensor's value, instead of 0, or ideally None/null.  

0 is a valid possible distance, assuming I get a direct strike. Without a strike, it's not 0 distance- there *isn't* a distance.

Rather than continue trying random changes to the Corrections logic, can anyone point to where the issue or mistake may lie? Is it something to do with the referenced lightning_strike_count being a cumulative number? Do I just need to yield and make the logic evaluate to 0?

[Andrew McGinnis]

ETA- 5000+ strike count/min comes from the sensor broadcasting packets in triplicate, and every 10 seconds for lightning. So up to 18 messages in each 1-min interval, with what was the true cumulative count of 303, resulted in 5454 strikes "recorded".

If anyone has a way to suppress duplicate MQTT messages, I'm all ears.  I get why sensors will send a packet multiple times to maximize the chances of reception at a station, but weewx (or, at least MQTTSubscribeDriver) sees each one and records it as a separate datapoint to aggregate in the archive period. Mostly harmless, but not entirely accurate. 

[gjr80]

A few comments.

I don't see multiple copies of a loop packet arriving causing a problem. Archive record obs that are averaged (eg outTemp) will be the same irrespective of how many time the loop packet is 'accumulated'. Per-period obs such as rain and, in this case, lightning_strike_count will similarly be unaffected as the difference for each such obs between successive duplicate loop packets will be zero. Of course cumulative obs will be nonsense, but as WeeWX does not use cumulative obs that should not be a problem.

Also, keep in mind how the [StdCalibrate] [[Corrections]] operate. Each [StdCalibrate] [[Corrections]] option value is used as the argument to a python eval() statement; if the eval() statement  causes a python exception to be raised the correction is silently ignored (and if it exists the destination obs is left unchanged, if it does not exist it is not created). I would suggest your main issue is your lightning_distance correction not working as intended (are you sure lightning_strike_count is indeed 0 or is it None?). Also, the 2nd of your three lightning_distance corrections is the one you use; WeeWX does not use 'nan' and use of Null will raise a python exception.

I would suggest you set some appropriate debug settings (I am not familiar with the MQTTSubscribe driver to know what debug settings are required) in place such that loop packets emitted by the MQTTSubscribe driver are logged. Then you will know exactly what is being fed to WeeWX and how to ensure it is processed/recorded correctly.

Gary

[vince]

Andrew - I think you need to more clearly state your setup and which problem(s) you are asking for help with because I certainly got very lost in your posts…..

What is emitting mqtt ? 

Are there actual duplicate mqtt messages with the same timestamps ?

How is your mqtt subscribe configured ?

Which mqtt subscribe variant and version are you using ?

What do a few received mqtt messages look like ?

What do a few weewx loop messages with debug=1 look like ?

What do you see ?

What do you expect/hope to see ?

[Andrew McGinnis]

What is emitting mqtt ? 

rtl_433 runs as a service on the same system as weewx is running on, set to output MQTT topics in SI units, via rtl_433 -C si -F mqtt. 

Are there actual duplicate mqtt messages with the same timestamps ?

Yes and no. The majority of all of my sensors output multiple packets, for redundancy.  Many- but not all- have a topic that is something to the effect of "sequence", but but don't otherwise differ in the payloads of the subscribed topics (notwithstanding Acurite's use of multiple message types with mixed topics across them).  Here's a message type 39 with wind avg, uv, lux, and lightning topics included. The packet was broadcast in triplicate, three copies broadcast and received, with sequence_num 1, 2, and 3:

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/time 2025-04-01 18:50:36

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/id 571

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/channel A

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/sequence_num 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/battery_ok 1

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/message_type 39

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/wind_avg_km_h 3.21869

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/uv 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/lux 5910

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 344

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 22

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/exception 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/raw_msg c23be7810084cf5696a4

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/mod ASK

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/time 2025-04-01 18:50:36

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/id 571

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/channel A

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/sequence_num 1

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/battery_ok 1

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/message_type 39

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/wind_avg_km_h 3.21869

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/uv 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/lux 5910

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 344

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 22

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/exception 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/raw_msg c63be7810084cf5696a8

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/mod ASK

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/time 2025-04-01 18:50:36

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/id 571

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/channel A

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/sequence_num 2

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/battery_ok 1

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/message_type 39

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/wind_avg_km_h 3.21869

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/uv 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/lux 5910

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 344

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 22

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/exception 0

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/raw_msg ca3be7810084cf5696ac

rtl_433/pi4b8/devices/Acurite-Atlas/A/571/mod ASK

How is your mqtt subscribe configured ?

What I pasted above was pulled straight from "mosquitto_sub -v -h localhost -t "rtl_433/pi4b8/devices/Acurite-Atlas/A/571/#"".  The weewx.conf section looks like this:

[MQTTSubscribeDriver]
    driver = user.MQTTSubscribe
    host = 10.19.76.14
    port = 1883
    keepalive = 20
    username = None
    password = None
    [[message_callback]]
        type = individual
    [[topics]]

        unit_system = METRICWX

        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/humidity]]]
            name = outHumidity
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/rain_mm]]]
            name = rain
            contains_total = true
        [[[br-weewx/Acurite-Atlas/A/571/rain_mm]]]

      **this one comes from a relay broker so I can capture the same topic twice and handle it differently for QC purposes**
            name = snow
            contains_total = false
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/temperature_C]]]
            name = outTemp
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/wind_avg_km_h]]]
            name = windSpeed
            units = km_per_hour
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/wind_dir_deg]]]
            name = windDir
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/battery_ok]]]
            name = outTempBatteryStatus
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/lux]]]
            name = luminosity
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/uv]]]
            name = UV

        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count]]]
            name = lightning_strike_count
            contains_total = true
        [[[rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance]]]
            name = lightning_distance
            units = km

Which mqtt subscribe variant and version are you using ?

pi@pi4b8:/etc/systemd/system $ weectl extension list
Using configuration file /etc/weewx/weewx.conf
Extension Name    Version   Description

MQTTSubscribe     2.3.0     Augment WeeWX records or packets with MQTT data.

What do a few received mqtt messages look like ?

Native mosquitto_sub gets what I pasted above.  MQTTSubscribe doesn't put (pretty much) anything in the log unless debug is enabled.

What do a few weewx loop messages with debug=1 look like ?

with debug enabled, the weewx log looks something like this (filtered to just lightning topics):
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> incoming rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 'lightning_strike_count' '0.0'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> incoming rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 'lightning_strike_count' '0.0'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> incoming rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 'lightning_strike_count' '0.0'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> incoming rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 'lightning_distance' '22.0'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> incoming rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 'lightning_distance' '22.0'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> incoming rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 'lightning_distance' '22.0'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> outgoing rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 'dateTime' '1743544003.5789976', 'lightning_strike_count' '0.0', 'usUnits' '17'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> outgoing rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 'dateTime' '1743544003.5918882', 'lightning_strike_count' '0.0', 'usUnits' '17'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> outgoing rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 'dateTime' '1743544003.5977087', 'lightning_strike_count' '0.0', 'usUnits' '17'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> outgoing rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 'dateTime' '1743544003.5803194', 'lightning_distance' '22.0', 'usUnits' '17'
4/1/25 17:46:43 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> outgoing rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 'dateTime' '1743544003.5927773', 'lightning_distance' '22.0', 'usUnits' '17'
4/1/25 17:46:44 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174643 EDT (1743544003) 'dateTime' '1743544003.599552', 'lightning_distance' '22.0', 'usUnits' '17'
4/1/25 17:46:44 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) TopicManager data-> outgoing rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 'dateTime' '1743544003.599552', 'lightning_distance' '22.0', 'usUnits' '17'
4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.06267', 'lightning_strike_count' '0.0', 'usUnits' '17'
4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.066883', 'lightning_strike_count' '0.0', 'usUnits' '17'
4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.0723326', 'lightning_strike_count' '0.0', 'usUnits' '17'
4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.063789', 'lightning_distance' '22.0', 'usUnits' '17'
4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.0683196', 'lightning_distance' '22.0', 'usUnits' '17'
4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.0791476', 'lightning_distance' '22.0', 'usUnits' '17'

What do you see ?

MQTTSubscribe looks to be passing the topic payload as the observation.  This is expected, as there's no logic applied to a non-cumulative value.

What do you expect/hope to see ?

StdCalibrate should be applying the correction formula to loop packets, or at least that's my understanding.  All of my WX sensors are configured through MQTT. When weewx receives an observation datapoint, regardless of the driver producing it, it should apply the correction before any record is made.  In this case,
"lightning_distance = lightning_distance if lightning_strike_count > 0 else None" should be applied akin to =IF(lightning_strike_count > 0, lightning_distance, "None"), but instead of evaluating to the False condition, it's evaluating to True.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/b3O1IzVjiCc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/2e0d1f8c-8531-401e-9bfd-f93123d8ec79n%40googlegroups.com.

[bell...@gmail.com]

I'm not quite following either. But as far as the corrections not working, I think you are probably running into the fact that MQTTSubscribe creates a packet for each topic. For 'individual' topics this works fine if observations are not dependent on another. The correction you are writing introduces a dependency between count and distance. Some additional information is in this thread, https://groups.google.com/g/weewx-user/c/NQIQ6KhG7To/m/Ke_SWuQxCwAJ
Is there an event topic? This is json message and therefore will get around the dependency limitation of 'individual' topics.
rich

[bell...@gmail.com]

Yup, each observation is in its own loop. See the following log messages

4/1/25 17:46:44 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174643 EDT (1743544003) 'dateTime' '1743544003.599552', 'lightning_distance' '22.0', 'usUnits' '17'

4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.06267', 'lightning_strike_count' '0.0', 'usUnits' '17'

4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.066883', 'lightning_strike_count' '0.0', 'usUnits' '17'

4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_count 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.0723326', 'lightning_strike_count' '0.0', 'usUnits' '17'

4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.063789', 'lightning_distance' '22.0', 'usUnits' '17'

4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.0683196', 'lightning_distance' '22.0', 'usUnits' '17'

4/1/25 17:46:54 pi4b8 weewxd[1166914]: DEBUG user.MQTTSubscribe (Driver) data-> final loop packet is rtl_433/pi4b8/devices/Acurite-Atlas/A/571/strike_distance 2025-04-01 174654 EDT (1743544014) 'dateTime' '1743544014.0791476', 'lightning_distance' '22.0', 'usUnits' '17'

If subscribing to the event topic is not possible, perhaps you could apply the correction on the archive record? See, https://www.weewx.com/docs/5.1/reference/weewx-options/stdcalibrate/#corrections

rich

[Andrew McGinnis]

 But as far as the corrections not working, I think you are probably running into the fact that MQTTSubscribe creates a packet for each topic. For 'individual' topics this works fine if observations are not dependent on another. The correction you are writing introduces a dependency between count and distance. Some additional information is in this thread, https://groups.google.com/g/weewx-user/c/NQIQ6KhG7To/m/Ke_SWuQxCwAJ
Is there an event topic? This is json message and therefore will get around the dependency limitation of 'individual' topics.
rich

This is what was happening. I followed the links and read through that thread.  Maybe I took away the wrong solution, but I added the  collect_observations = True  and single_queue = True options.  Wipe errant db records, rebuild affected daily reports, restart weewx, and boom- everything is working as intended.  Lightning strikes are recording as 0, and strike distance is registering as N/A. We'll see what happens during the next expected storm this coming Monday, but this seems to be what I needed.

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/b3O1IzVjiCc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/1e68532d-4cca-487f-aefa-bcb429c9462fn%40googlegroups.com.

[bell...@gmail.com]

Well, that is good news! I did not recommend those configuration options for a few reasons. First, I felt that I never got a confirmation that it worked. Second, those options are 'experimental', which primarily means that I am not fully comfortable with the implementation and for various reasons might not be able to fully support the functionality in the future. Knowing that it works for at least one person 'in the real world' is a good data point as I decide whether to move it out of 'experimental' to 'production'.

Thanks for letting us know that things are working.
rich

[Andrew McGinnis]

I do get occasional lightning_distance records, but 0 strikes = n/a distance for the vast majority of the time. I haven’t gone back in to debug log to see when/why they’re slipping through. Maybe the json method would be better, but I’d have to refactor all of my observations. Add to that, there’s 2 or 3 different message types, with significant overlap between them (i.e. one has temp-rain-lightning, one has temp-wind-lightning, another has temp-uv-lightning). Not quite sure how I’d handle that.  

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/fdcc73f3-44dc-4cb0-84ea-ea14d27d4735n%40googlegroups.com.