websockets driver help

[Billy Jackson]

Hi,

I'm trying to write a driver for my Tempest WS.  I currently have the UDP driver from captain-coredump up and running (thank you!) on a pi but I'd like to pump data straight from the websockets API into weewx also.  I may expand it to capture archive packets from the rest API also.

The prototype is here... https://github.com/livysdad27/tempestAPI

I borrowed a LOT from captain-coredump's driver.  Where I'm struggling is with the getLoopPackets generator.  I've attempted to use the on_message handler for this but have read in another thread that it doesn't work well due to running in a separate thread.  My websockets code is a bit simple and I'm not completely sure how drivers get called for sure. 

Is it even realistic to try and run a websocket-client app implementation with "run_forever" and expect it to work correctly with weewx?  My suspicion is weewx is starting, entering the websocket app and sitting there forever because I don't get report or other messages after it kicks off.

Super grateful for any help/pointers!

Billy

[Vince Skahan]

I think somebody else was working a while back on a combination driver that could get WS and/or UDP data.  You might want to ask around over at the WF Forums to see how far they got.

[Billy Jackson]

Thanks Vince, are you the same Vince from the WF forum?

I'll drop a note there as well.  I think I know who you're talking about and I cloned their repo to review it.  That said, i wasn't able to get it to run so I'm not sure if they completed it.

I think my biggest problem is I'm using the ws.run_forever method and that's "capturing" the execution and not letting go of it.  I want to find out if there's a way, maybe using yield, to open the socket once, keep it going and let the main Weewx thread just get the iterable/data back.

The weatherflow UDP driver re-opens the connection every five minutes or so and I'd like to avoid that.

More to come.

Billy

[Vince Skahan]

yup - that's me.  Sorry I can't help you much on this one, but there are a lot of folks in the WF forums who wrote other websockets listeners so perhaps that might be the best place.  I think there are a variety of forked weewx drivers as well but I haven't looked into them in any detail since the original driver was rock solid for me (until I sold+gifted my WF gear :-)

I'm assuming you want the websockets stuff so you can grab the few elements of calculated data only present via their servers ?

I'm not certain your last statement re: the UDP driver is accurate.  My recollection is that it's just a listener that stays open, although I think it has some stuff in it to reopen a new connection if there's a timeout or something.   Code is at github (link) if anybody else is interested in taking a look.

[bell...@gmail.com]

I am guessing that run_forever is a blocking call. So, you probably need to move it into a thread.

Here, https://github.com/weewx/weewx/wiki/multi-threaded-service,  is a write up on writing a multi-threaded WeeWX service. It could be adapted for a driver. I'd probably (ok, I do in MQTTSubscribe) use a queue to communicate between the threads.

This answer talks about getting python websockets data in a separate thread

https://stackoverflow.com/questions/29145442/threaded-non-blocking-websocket-client#:~:text=1-,Answer,-Sorted%20by%3A

Hope this helps you get going. 

rich

[Billy Jackson]

Thank you!  I'll check this out.  I'm also looking closely at the queue module.  It looks like at least one MQTT driver makes use of that as well.

Billy

[Billy Jackson]

Hi Vince,

I just re-reviewed the UDP driver code.  You're correct that it has a while true loop that stays open, uses the timeout to wait long enough for a new UDP packet etc....  It's how it's called that seems to be the difference.  In my syslog I see the genLoopPackets called every five minutes on the dot.  My guess is that's how often weewx calls the driver.  At the top of Captain Coredump's genLoopPackets method is where he opens/binds the UDP socket.  Hence the need for multi-threading.

I sure have a lot to learn.  Thanks for the pointer Rich!  Vince, many many thanks for your help in both forums.

Billy

[Billy Jackson]

Hi Everyone,

Thanks for all the help.  I finally solved my problem.  As usually I was making life too hard on myself.  Here's what I finally figured out.....

• The driver basically runs all the time unless it gets interrupted.
• It seems to get interrupted every five minutes on the dot.
• Therefore there's no reason to use a long running websocket.  Five minutes gives enough data (rapid_wind every 3 sec and a obs every minute) to populate the DB.  It would be lovely to catch the interruption and close the socket cleanly but not mandatory I suspect.
• Just fire the websocket connection back up each time.

So far this is working on a test cloud instance (look ma, no local PI as a forwarder!).  I also got a version that queries/polls the rest API.  To keep that one from spamming the endpoint I put in a sleep/timeout config param.  I'm playing with the best amount of sleep there and I suspect anything less than 60 seconds, within reason, will work nicely.  I'm leaning towards the websocket driver as being the better of the two because of the more frequent wind data.  There are some other events that may be available as well but they depend on rain/lightening and today is nice and sunny.

I am curious the behavior of WEEWX when a loop_packet with a duplicate epoch/timestamp is submitted since that's the primary key?  Does it fail, just discard the packet silently or something else.  The reason I ask is that the Tempest rest API has a date-range option that may be nice to have for archive_packets or even just refilling gaps but I'd want to know how it'd play with a duplicate record.

Many thanks!

Billy

[Billy Jackson]

Also, @Vince, for the record you were right.  It just reopens the connection.  The timeout param is for the UDP connection timing out and mine being increased assisted because for some reason the UDP broadcast would sometimes overrun the timeout periodically.

Just wanted to call out that you were right on the nose and thank you.

Billy