Calculating a moving average for wind dir / speed

[James Pattinson]

Bit of background, I have been asked to set up a simple weather screen for an airfield manned by hobbyists. They bought a Davis ISS and 6313 console combo, and I found that the console is essentially a closed/cloud system and not good for real time data.

So, I've since installed a Pi with an RTLSDR dongle and WeeWx. It's receiving data from the iSS and populating reports every 5 minutes.

At the airfield we generally need a pretty up to date wind direction and speed, averaged over the last 1 or 2 minutes, and a real time 'instant' wind display.

The instant wind I am OK with. I've installed the MQTT extension to WeeWx so I get the LOOP data every few seconds with the wind info.

Now, I'm not sure how to go about getting the average wind. Ideally, what I want is that at every update of the raw wind value (ie at the LOOP time) it also computes the average wind speed / vector over the last 1,2 and 10 minutes, so basically three moving averages all updated every few seconds, so I think these updates will be quite 'smooth' and not jumping all over the place like the instant wind reading.

Problem is I've no idea where to start with this, or even if it's possible. I would appreciate any ideas anyone may have.

[vince]

I don't know if this will help any given you want to get averages within the 5-minute archive period but does https://www.weewx.com/docs/5.0/custom/cheetah-generator/#tag-span help any ?

$span(time_delta=60).windGust.avg

$span(time_delta=120).windGust.avg

$span(time_delta=600).windGust.avg

(untested)

[vince]

Uncertain my previous answer actually helps any given how short your desired average periods are but also see  https://www.weewx.com/docs/5.0/custom/cheetah-generator/#wind

[James Pattinson]

Thanks for the responses. It’s interesting stuff. I should note that my desire is to push this average value out over MQTT as I have a web page displaying the real time data and when something changes I want it to be pushed out. So I actually don’t think anything to do with templates or cheetah will be relevant.

I am considering setting my archive interval to one minute instead of 5, am not sure it’s possible yet but it should give me almost what I want. I wonder if that would have any repercussions apart from extra storage space being required?

That said, the MQTT output has been running for a while and it’s emitting the LOOP parameters perfectly but no archive data so that’s another issue to solve!

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/AvOx55kq49g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0c672f89-b119-4566-be10-141abb358058n%40googlegroups.com.

[matthew wall]

its not a moving average, but if you have an archive interval of 5 minutes and you post both loop and archive to MQTT then you'll get 5 minute averages as well as near-real-time values.

[StdRESTful] 

    [[MQTT]]

        ...

        binding = loop, archive

 this is with the weewx-mqtt restful service:

https://github.com/matthewwall/weewx-mqtt

and this is what the result looks like in grafana with influx as the backend (this is tide data from a maxbotix ultrasonic sensor sampled every 1 second with a 5 minute archive interval)

[John Kline]

I have a plug-in that will generate a json file on every loop packet.  Furthermore, it supports rsyncing that file (on every loop packet).

You need to specify in weewx.conf what you want written to the file.  Besides tags you can write in the your reports (e.g, current.outTemp.raw, day.windGust.max), it will support arbitrary time lengths (e.g., 10m.windGust.max).  That will get you the moving averages you want (e.g, 1m.wind.avg, 2m.wind.avg).

The extension is only tested for Davis Vantage Pro 2 and RainWise CC3000, but likely works for all drivers which include all obervations on every loop packet.  Drivers generating sparse readings are very much NOT supported.

If you want to give it a try, it’s at:

On Sep 29, 2024, at 3:44 PM, matthew wall <mwall...@gmail.com> wrote:

its not a moving average, but if you have an archive interval of 5 minutes and you post both loop and archive to MQTT then you'll get 5 minute averages as well as near-real-time values.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/35efa133-9f08-45dc-bc65-fd959065aebbn%40googlegroups.com.

<tide-loop+archive.png>

[Karen K]

If you have some experience in Python, you could write a little WeeWX service, which registers with the LOOP packet loop. So, every time a new wind value is received you could store it in a variable, then calculate the moving average and then add it to the LOOP packet. There are a lot of little services around there to provide an example, how to do so.

[James Pattinson]

>  its not a moving average, but if you have an archive interval of 5 minutes and you post both loop and archive to MQTT then you'll get 5 minute averages as well as near-real-time values.

That is what I was thinking as a way to get something going while I tweak, but I saw that the MQTT extension wasn't posting archive data. I installed a different extension, which doesn't have that binding parameter, so I will replace it with your one and give that a go - thanks!

[James Pattinson]

>  I have a plug-in that will generate a json file on every loop packet.  Furthermore, it supports rsyncing that file (on every loop packet).

That's pretty interesting. Are you saying it can do averaging for periods shorter than the archive interval? It sounds like it would be possible to modify that to publish MQTT messages instead of writing the JSON file, and that'd be exactly what I need. Thanks for the pointer.

[James Pattinson]

>  If you have some experience in Python, you could write a little WeeWX service, which registers with the LOOP packet loop. So, every time a new wind value is received you could store it in a variable, then calculate the moving average and then add it to the LOOP packet. There are a lot of little services around there to provide an example, how to do so.

That's a good idea. My Python skills aren't brilliant (I still write scripts like I'm in the 1990s) but I had considered writing a script totally outside of WeeWX to subscribe to the MQTT topics, perform this calculation, and emit the average. Doing with a WeeWX service would be a more elegant solution, so I will take a look at some examples.

[Graham Eddy]

vp2 emits packets (i think including wind readings) about every 2.7 secs; weewx accumulates these and saves/reports aggregated values each interval (default 5 mins).

you could use the mqtt service/driver to publish selected packet values every 2.7 secs, and handle these youself for more realtime-oriented purposes (weewx reporting is not designed as a realtime service) in an app you write.

this simple app that subscribes to the 2.7 sec vp2 packets will publish on whatever basis you like. timing (30 sec, 1 min, 2 min, 5 min, 30 min intervals) is not the only consideration, so is reliability e.g. if you hold the aggregations in memory then if the server is restarted they are lost ➔ maybe the partial aggregations need to be persisted while being accumulated for next interval publish message

the weewx components being used are the vantage driver, which acquires realtime data from vp2 and normalises the data, and the mqtt service that extracts relevant normalised data and publishes it to your realtime wind service app. the weewx reporting would continue in parallel and provide regular reports each 5 mins on website

⊣GE⊢

[Karen K]

James Pattinson schrieb am Montag, 30. September 2024 um 09:08:00 UTC+2:

That's a good idea. My Python skills aren't brilliant (I still write scripts like I'm in the 1990s) but I had considered writing a script totally outside of WeeWX to subscribe to the MQTT topics, perform this calculation, and emit the average. Doing with a WeeWX service would be a more elegant solution, so I will take a look at some examples.

Such a service would look like this:

class YourService(StdService):

    def __init__(self, engine, conf_dict):

        super(YourService,self).__init__(engine, conf_dict)

        self.wind_readings_cache = []

        self.bind(weewx.NEW_LOOP_PACKET, self.new_loop_packet)

    def new_loop_packet(self, event):

        wind = event.packet.get('windSpeed')

        if wind is not None:

            # do some calculations involving self.wind_readings_cache

            event.packet['windSpeedAvg'] = ...

 This is a rough sketch. There are more details to consider to complete it.

[James Pattinson]

Hi Matthew

> its not a moving average, but if you have an archive interval of 5 minutes and you post both loop and archive to MQTT then you'll get 5 minute averages as well as near-real-time values.

I've tested this out. I am not exactly sure if I understand this correctly, but it looks like that the loop and archive data is getting intermingled somehow. I've set my interval to 2 minutes because I need that 2 minute average.

Now, every 2.5 seconds, I get the topic weather/loop updated with a JSON payload containing the loop data, great, I can extract things from there.

However, it also updates weather/windSpeed_mph and weather/windDir with the same values (every LOOP).

Now when the archive interval comes along I think it's putting the minute average in weather/windSpeed_mph and weather/windDir because I can pause the MQTT collection and compare against the web report. But, 2.5s later, the values are overwritten by live loop data again. There is simply no way for me to tell if what's in windSpeed_mph or windDir  is live or average. 

Is this expected? I would have liked to see a separate topic, say weather/archive completely separate from the loop data. 

Hope you can help!

James

On Sunday 29 September 2024 at 23:44:49 UTC+1 matthew wall wrote:

[James Pattinson]

On Mon, 30 Sept 2024 at 09:36, Karen K <kk44...@gmail.com> wrote:

Such a service would look like this:

class YourService(StdService):

    def __init__(self, engine, conf_dict):

        super(YourService,self).__init__(engine, conf_dict)

        self.wind_readings_cache = []

        self.bind(weewx.NEW_LOOP_PACKET, self.new_loop_packet)

    def new_loop_packet(self, event):

        wind = event.packet.get('windSpeed')

        if wind is not None:

            # do some calculations involving self.wind_readings_cache

            event.packet['windSpeedAvg'] = ...

 This is a rough sketch. There are more details to consider to complete it.

Thank you! That gives me an excellent head start on how to write a service. I will give it a go later.

[Karen K]

James Pattinson schrieb am Montag, 30. September 2024 um 14:36:29 UTC+2:

I've tested this out. I am not exactly sure if I understand this correctly, but it looks like that the loop and archive data is getting intermingled somehow. 

Yes, they do.

You can distinguish LOOP packets and ARCHIVE records within the MQTT data by looking at interval_minute. If this observation type is included, it is an ARCHIVE record, if not, a LOOP packet.

[James Pattinson]

Amazing, thank you!