WeeWX Event Firing

[bell...@gmail.com]

Goal: Capture the min value and time of lightning strikes.

I started out binding to the PRE_LOOP and NEW_LOOP_PACKET events. When the PRE_LOOP event is handled, the min value and time were reset. In the handling of the NEW_LOOP_PACKET event, the comparison is done and the loop packet is updated as necessary. I 'quickly' realized that PRE_LOOP and POST_LOOP events have nothing to do with the archive period.

The END_ARCHIVE_PERIOD event looked promising. Resetting of the value and time could be done here. But when I analyze the order of event firing, this is what I am seeing.
2024-09-21 11:44:59  weewxd[4036220] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 11:44:59  weewxd[4036220] DEBUG user.observationtime:   RMB: packet dateTime: 1726933499 2024-09-21 11:44:59
2024-09-21 11:44:59  weewxd[4036220] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 11:44:59  weewxd[4036220] DEBUG user.observationtime:   RMB: packet dateTime: 1726933499 2024-09-21 11:44:59
2024-09-21 11:45:01  weewxd[4036220] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 11:45:01  weewxd[4036220] DEBUG user.observationtime:   RMB: packet dateTime: 1726933502 2024-09-21 11:45:02
2024-09-21 11:45:01  weewxd[4036220] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 11:45:01  weewxd[4036220] DEBUG user.observationtime:   RMB: packet dateTime: 1726933502 2024-09-21 11:45:02
2024-09-21 11:45:01  weewxd[4036220] DEBUG user.observationtime: RMBx: In end_archive_period
2024-09-21 11:45:04  weewxd[4036220] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 11:45:04  weewxd[4036220] DEBUG user.observationtime:   RMB: packet dateTime: 1726933504 2024-09-21 11:45:04
This will result in the first packet (1726933502) of the archive period using the data from the previous archive period.

Flipping the order in engine.py to the following:
    for packet in self.console.genLoopPackets():
        # Allow services to break the loop by throwing
        # an exception:
        self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet))

        # Package the packet as an event, then dispatch it.
        self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))

I would have expected to see the last packet in an archive period not being processed. But, I was not seeing that. So I added a bit more logging to try to understand what was going on. This is what I saw. Sorry, it is a bit long, I made a couple of comments with <---.

 
2024-09-21 19:39:58  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:39:58  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726961999 2024-09-21 19:39:59
2024-09-21 19:39:58  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726961999 end_archive_delay_ts: 1726962035  1726962000  35
2024-09-21 19:39:58  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:39:58  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726961999 2024-09-21 19:39:59
2024-09-21 19:39:58  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:01  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:01  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962001 2024-09-21 19:40:01
2024-09-21 19:40:01  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962001 end_archive_delay_ts: 1726962035  1726962000  35
2024-09-21 19:40:01  weewxd[235236] DEBUG user.observationtime: RMBx: In end_archive_period <--- END_ARCHIVE_PERIOD event must have fired
2024-09-21 19:40:01  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:01  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962001 2024-09-21 19:40:01 <--- And the NEW_LOOP_PACKET has fired afterward
2024-09-21 19:40:01  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:03  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop <--- processing of even packets continues
2024-09-21 19:40:03  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962004 2024-09-21 19:40:04
2024-09-21 19:40:03  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962004 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:03  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:03  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962004 2024-09-21 19:40:04
2024-09-21 19:40:03  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:06  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:06  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962006 2024-09-21 19:40:06
2024-09-21 19:40:06  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962006 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:06  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:06  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962006 2024-09-21 19:40:06
2024-09-21 19:40:06  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:08  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:08  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962009 2024-09-21 19:40:09
2024-09-21 19:40:08  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962009 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:08  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:08  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962009 2024-09-21 19:40:09
2024-09-21 19:40:08  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:11  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:11  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962011 2024-09-21 19:40:11
2024-09-21 19:40:11  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962011 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:11  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:11  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962011 2024-09-21 19:40:11
2024-09-21 19:40:11  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:13  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:13  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962014 2024-09-21 19:40:14
2024-09-21 19:40:13  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962014 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:13  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:13  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962014 2024-09-21 19:40:14
2024-09-21 19:40:13  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:16  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:16  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962016 2024-09-21 19:40:16
2024-09-21 19:40:16  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962016 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:16  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:16  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962016 2024-09-21 19:40:16
2024-09-21 19:40:16  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:18  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:18  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962019 2024-09-21 19:40:19
2024-09-21 19:40:18  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962019 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:18  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:18  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962019 2024-09-21 19:40:19
2024-09-21 19:40:18  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:21  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:21  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962021 2024-09-21 19:40:21
2024-09-21 19:40:21  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962021 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:21  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:21  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962021 2024-09-21 19:40:21
2024-09-21 19:40:21  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:23  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:23  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962024 2024-09-21 19:40:24
2024-09-21 19:40:23  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962024 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:23  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:23  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962024 2024-09-21 19:40:24
2024-09-21 19:40:23  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:26  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:26  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962026 2024-09-21 19:40:26
2024-09-21 19:40:26  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962026 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:26  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:26  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962026 2024-09-21 19:40:26
2024-09-21 19:40:26  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:28  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:28  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962029 2024-09-21 19:40:29
2024-09-21 19:40:28  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962029 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:28  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:28  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962029 2024-09-21 19:40:29
2024-09-21 19:40:28  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:31  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:31  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962031 2024-09-21 19:40:31
2024-09-21 19:40:31  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962031 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:31  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:31  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962031 2024-09-21 19:40:31
2024-09-21 19:40:31  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:33  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:33  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962034 2024-09-21 19:40:34
2024-09-21 19:40:33  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962034 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:33  weewxd[235236] DEBUG user.observationtime:  RMB: In new_loop_packet
2024-09-21 19:40:33  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962034 2024-09-21 19:40:34
2024-09-21 19:40:33  weewxd[235236] DEBUG weewx.engine: RMBx: in new_loop_packet (StdArchive)
2024-09-21 19:40:36  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:36  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962036 2024-09-21 19:40:36
2024-09-21 19:40:36  weewxd[235236] DEBUG weewx.engine: RMBx: In check_loop (StdArchive) packet dateTime: 1726962036 end_archive_delay_ts: 1726962035  1726962300  35
2024-09-21 19:40:36  weewxd[235236] DEBUG weewx.engine: RMBx: Raising BreakLoop 1726962036 1726962035 <--- due to archive_delay, the loop is 'finally' broken and the archive record can be processed.
2024-09-21 19:40:36  weewxd[235236] DEBUG weewx.engine: RMBx: handling BreakLoop exception
2024-09-21 19:40:36  weewxd[235236] DEBUG user.observationtime: RMBx: In post_loop
2024-09-21 19:40:36  weewxd[235236] DEBUG weewx.engine: RMBx: In post_loop (StdArchive)
2024-09-21 19:40:36  weewxd[235236] DEBUG user.observationtime: RMBx: In new_archive_record <--- processing the new archive record
2024-09-21 19:40:56  weewxd[235236] DEBUG user.observationtime: RMBx: In pre_loop
2024-09-21 19:40:56  weewxd[235236] DEBUG user.observationtime:  RMB: In check_loop
2024-09-21 19:40:56  weewxd[235236] DEBUG user.observationtime:   RMB: packet dateTime: 1726962039 2024-09-21 19:40:39

Because of the archive_delay, the last packet goes into the correct archive record and the END_ARCHIVE_PERIOD event fires before the NEW_LOOP_PACKET event is fired.
       
I realize that changing the order of events firing can have big impact to existing code, but I am curious about possibly making this change? It seems to me that this should only impact anyone that wants the END_ARCHIVE_PERIOD event to fire before the next packet's NEW_LOOP_PACKET is fired. Not sure if that is important to anyone... Or is there a better/more WeeWX way of accomplishing what I desire?
If no changes are made to WeeWX, my current thought is to capture the data via the NEW_LOOP_PACKET and perform the processing via the NEW_ARCHIVE_RECORD event and only update the archive record with the min value and time. This would certainly work for me.

Thanks. rich

[Tom Keffer]

Take a look at the Vantage device driver. In addition to a standard device driver, it includes a service, VantageService, that participates in the weewxd event loop. It binds to STARTUP, NEW_LOOP_PACKET, and END_ARCHIVE_PERIOD. The first two are used to reset wind gust values, the last to update the archive record with the max wind seen.

I think this is pretty similar to what you're trying to do.

-tk

[bell...@gmail.com]

Tom,
Thanks for the pointer. That is what I am attempting to do. Here is what I THINK is my problem. See 4.d below.
1. A new loop packet is generated.
2. StdEngine fires a NEW_LOOP event
3. MyService handles the NEW_LOOP event.
   a. It sees that a prior value is less then the current value in the loop packet.
   b. The prior value is added to the loop packet as min_value.
4. StdArchive handles the NEW_LOOP event.
   a. It tries to add it to the accumulator.
   b. The accumulator throws OutOfSpan exception
   c. StdArchive creates a 'new' accumulator for the next archive period.
   d. StdArchive adds the loop packet (which has been updated with prior loop packet data) to the accumulator,
5. StdEngine fires the CHECK_LOOP event.
6. StdArchive handles the CHECK_LOOP.
   a. The dateTime in the loop packet is greater than the end of the archive period.
   b. StdArchive fires the END_ARCHIVE_PERIOD event.
   c. The dateTime is not greater than the end of the archive period + the archive delay.
7. MyService handles the END_ARCHIVE_PERIOD.
   a. Min values are reset to None, etc.
   
   
8. Eventually a packet with dateTime >= the end of archive period + archive delay is generated.
   a. This causes StdArchive (in CHECK_LOOP) to raise the BREAKLOOP exception
   b. StdEngine handles the exception, raising POST_LOOP; allowing things like reports to run etc.
 
  rich

[bell...@gmail.com]

As a followup. When I flipped the order that StdEngine fires the events, such that CHECK_LOOP is before NEW_LOOP this is what I observed.

1. A new loop packet is generated.

2. StdEngine fires the CHECK_LOOP event.
3. StdArchive handles the CHECK_LOOP.

   a. The dateTime in the loop packet is greater than the end of the archive period.
   b. StdArchive fires the END_ARCHIVE_PERIOD event.
   c. The dateTime is not greater than the end of the archive period + the archive delay.

4. MyService handles the END_ARCHIVE_PERIOD.

   a. Min values are reset to None, etc.

5. StdEngine fires a NEW_LOOP event
6. MyService handles the NEW_LOOP event.
   a. It sees that no prior value exists
   b. The current value in the loop packet is added to the loop packet as min_value.
7. StdArchive handles the NEW_LOOP event.

   a. It tries to add it to the accumulator.
   b. The accumulator throws OutOfSpan exception
   c. StdArchive creates a 'new' accumulator for the next archive period.

   d. StdArchive adds the loop packet to the accumulator,

8. Eventually a packet with dateTime >= the end of archive period + archive delay is generated.
   a. This causes StdArchive (in CHECK_LOOP) to raise the BREAKLOOP exception
   b. StdEngine handles the exception, raising POST_LOOP; allowing things like reports to run etc.
 

Again, not saying that changing the order that events are fired should be taken lightly.
  rich

[bell...@gmail.com]

So to close this out. I updated the service to capture the data via the NEW_LOOP_PACKET event. Then in the NEW_ARCHIVE_RECORD I only process event data that is in the archive interval. Essentially a ‘poor mans’ accumulator. 

This has stopped the ‘bleeding’ of data from a previous archive interval into the current interval.  I still need to write some more tests, but I am pretty confident this accomplishes what I desire.

rich

[Tom Keffer]

Sorry, Rich, for not getting back to you sooner. Just got swamped with another project.

I can see how this can happen. If a LOOP packet arrives slightly after the end of an archive interval, it most properly belongs to the previous archive interval, but you're treating it as if it belongs to the next interval. 

Is there any reason why you can't use the accumulator machinery? It already handles min and max values.

You've probably already seen it but, in case you haven't, take a look at the wiki article Accumulators. 

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/d5388801-335e-482e-acc0-2fff84cd3633n%40googlegroups.com.

[bell...@gmail.com]

Tom,

I get life, no apologies necessary. I’m probably not following your thought, but here is what I think you are suggesting.

When MyService handles the NEW_LOOP_PACKET event, it checks if the data belongs to the current accumulator (or has a new archive interval started).

If it does belong to the current accumulator (current archive interval), it updates the current accumulator with the loop data.

If it does not belong (a new archive intervals has started), it creates a new accumulator and puts the data in that (keeping the old one around, similar to StdArchive (I think)).

When MyService handles the NEW_ARCHIVE_RECORD, it uses the ‘old’ accumulator to retrieve the desired values and updates the archive record.

The challenge I was facing: I could not figure out a good way for MyService to know anything about archive intervals in the NEW_LOOP_PACKET event handler.

rich

[Tom Keffer]

Perhaps if you posted your code? I'm betting you could use the existing accumulators to extract what you need from the collection of LOOP packets.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/cdd39a81-25de-4faf-b383-8d5420041d83n%40googlegroups.com.

[bell...@gmail.com]

I am guessing you mean my original code. I’ll recreate and attach shortly.

So, to recap. I had a service MyService that was augmenting loop packets with data. The standard processing of WeeWX accumulators was used to pull out the min/max/first/last values (and times of these) and update the archive record.

At first I was augmenting every loop packet with the min/max. But, I noticed that the min/max (and times) values were ‘bleeding’ into the next archive record. So, to mitigate that, I stopped putting the min/max (and times) in every loop packet. 

But there is still a slight chance that the last packet could ‘bleed’ into the next archive record (In my case, if there is a lightning strike in the last loop packet that has minimum distance smaller than any in the next archive interval). I admit it is slight, but it was enough to bother me….
rich

[bell...@gmail.com]

File attached