Monitoring RESTful API calls sending weather data

[Kruse Ludington]

The below 3 lines (I put blank lines between them for readability) from my weewx.log, regarding my AmbientWeather WS-2902C weather station:

Jul 5 23:00:30 kruse-pi weewx[701] INFO weewx.restx:

Windy: Published record 2022-07-05 23:00:00 EDT (1657076400)

Jul 5 23:00:30 kruse-pi weewx[701] INFO weewx.restx: 

WindGuru: Published record 2022-07-05 23:00:00 EDT (1657076400)

Jul 5 23:00:30 kruse-pi weewx[701] INFO weewx.restx: CWOP: Published record 2022-07-05 23:00:00 EDT (1657076400)

There are many others but that is a simple example of three weather sites where I am pushing weather data.

This instance of Weewx is running on a Raspberry PI 4B 8GB running a 1TB SSD (rather than a micro SD card) for performance reasons.

Also on that same RPI, I have Home Assistant Supervised running in a docker. 

I don't care what the mechanism is other than something elegant (ideally by listening to network traffic, not by log scraping), reliable and lightweight, but I am trying to determine the best way to be notified within Home Assistant the most recent time I have sent data to each of these websites, being updated each time another API call is successfully made.

Home Assistant supports MQTT, but out of the box functionality for weewx as I understand it MQTT is only available for sending the weather data to destination/s via MQTT, not reporting when RESTful API calls are successfully published, which is the notification I need. Is there an extension for this or would it be something I need to create myself (notifications about successful API calls)?

Best Regards,

-R. Kruse Ludington

📱(M): +1.201.925.4410

📞(O): +1.862.308.7040

📞(H): +1.201.857.8307

📠(F): +1.201.857.7188

[Tom Keffer]

Honestly, scraping the logs isn't so bad.

An alternative would be to monkey-patch the member function weewx.restx.RESTThread.process_record(). The replacement version would first call the original version, then, once it returns, post success by whatever means you choose to Home Assistant. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/181d19b93b8.29af.c5c13ad976ec84a62e2892bf98a0657b%40gmail.com.

[vince]

Some creative syslog config file rewrite rules might be able to generate separate output file(s) and associated data in a format you can parse with HA, but how to do it depends on your os and which syslog daemon(s) you are running.

Personally, I'd use a cron job to publish to MQTT every few minutes and simply have HA subscribe to those topics.

If you do it by patching weewx, you'll have to keep that patch valid as weewx changes, so that's probably less good a direction to follow.

[Tom Keffer]

This monkey patch would be pretty safe. The function process_record() is intended to be overridden by subclasses, so it can't really afford to change.

Not to mention, log entries can change too!

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2265112d-89e2-4f51-91c2-0fa01d0ff98dn%40googlegroups.com.

[Kruse Ludington]

Hi again Vince and thank you also Tom. I agree to the point that the source for the overriden function "can't really afford to change" - but wouild assume any updated version however when I do update would include another copy of the same file so I'd have to go replace it each time with my own etc.... Rethinking this... even later extensions such as say to send to "Meteoservices" is also inherited from that original function so if I changed the original at that parent level it woiuld cap[ture those too?

I used to write alot of code in my previous life for trading systems on wall street but that was a long time ago - so that along with the fact this is on my spare time which I have very little of and I am new to Python and MQTT etc... not sure it it's worth the hassle to figure out how to do it in an elegant proper manner. I guess I'll continue to search high and low when I have the time to figure out how to, from within HA, to listen to the network traffic - which I don't think I will find - or as a second option to find some plugin that will scrape logs... sigh...

So.... while I look for a way for HA to listen to network traffic - do either of you guys know of any very lightweight processes I could put on the RPI to listent to traffic and publish to MQTT? The cron job is a good idea but that's still scraping the logs rather than listnining to traffic, correct?

[vince]

Sorry - no I don't know of any.  What you're asking is rather unique and for me, I don't even understand why it seems so important to you.

Many years ago at $work we had a lot of elaborate rsyslog (I think) rewrite rules to save special things to discrete files, but it's been so long that I can't remember the exact details.  I think 'example 2' of https://www.thegeekdiary.com/etc-rsyslog-conf-setup-a-filter-to-discard-or-redirect-messages/ gives one example you might fiddle with.

There are a lot of syslog daemons with varying capabilities, so I don't know what's in the os you're running.

The other very heavy option would be running Splunk (which might not even work on a pi) which can do just about anything you can cook up, but you are seriously upping the compute horsepower and complexity ante if you go down that path.

[Greg Troxel]

My reaction to all of this is that if you want to have status on whether
the upload succeeded you really want to have that flow from the code in
weewx hitting a point where it believes it got an ack. Listening to
network traffic seems like a really wrong way to do it, and if the
upload is https -- which more or less everything should be -- then it's
going to be hard to determine that there was an ack.

Watching logs is reasonable because there is a direct flow from
code believes upload succeeded
code calls logging statement
log line ends up in log

It's far more reliable and lower cost to just basically tail -f the log
(surely there is some python log-watcher module that copes with
rotation) and parses the lines and then does whatever.


This discussion makes me thing weewx (and programs in general) should
have some hook facility where there is a hook object with defined
methods that can be subclassed, and then you can add hook objects to
various parts of the code. Functionally this amounts to changing the
code, but doing so by having the ability to change part permanently in
the code and your part loaded, so it's stable across upgrades.