MQTT-Publish stanza format

[rae...@gmail.com]

I'm having a heck of a time even finding an example of how to format the stanza to publish topics under the [[MQTT]] section under the [StdRESTful] section.  

I have successfully installed this service and I see it load on startup:
https://github.com/weewx-mqtt/publish

I just wish there was a sample config to follow but that seems to be non-existent.  I don't know where to nest things.  I'm posting individual topics so can someone post an example that works?

[John Smith]

I posted a working snippet when I made a PR

https://github.com/weewx-mqtt/publish/pull/15/changes/90f1473887deef1459cd647e3aaae129a6caf5f1

However I realised how inefficient it was trying to include or exclude things like that and have since come up with a fork that loops in the python script instead of repeating the same things lots and lots of times as seen at that link

My fork can be found here: https://github.com/evilbunny2008/MQTTPublish

Here's an example stanza for that extension https://github.com/evilbunny2008/MQTTPublish/blob/main/weewx-sample.conf

However I shifted all the aggregate functionality into it's own extension so that values could be calculated once per archive interval and then all other extensions could access the values

That extension can be found here https://github.com/evilbunny2008/AggregatedValues

I'm half way through migrating the Inigo plugin to use the values calculated and so when the work on the Inigo plugin is complete both it and the MQTTPublish extension will do almost no DB lookups and I'm already seeing a significant reduction in the time to generate multiple reports.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/c3cc4816-4fbc-46c7-b793-33f89b48b53an%40googlegroups.com.

[rae...@gmail.com]

Appreciate it thanks.  So you mean I have to include every variable in the stanza and then toggle the ignore flag for each one, or can I just include the values I want?  I'm posting individual archive entries and will probably want the aggregate ones for day/week/month/year kind of thing.  Is that possible?

[rae...@gmail.com]

One more thing, installing your fork seems to point to the same mqtt-publish repo as the original.  Is that correct?

[bell...@gmail.com]

The install should have given you a ‘weewx.conf’  to start from. What does that look like?

What are you trying to do? For me, I have less fields to exclude, so it makes sense to specify those. For others, it makes more sense to specify what to include. Either is supported. 

In the soon to be released version, you can specify what to exclude on a single line.

Again, what are you trying to do. I am sure we can get it working quickly and easily.

rich

[bell...@gmail.com]

This example is overly verbose and could be simplified if one wanted to.

rich

[bell...@gmail.com]

As John noted, he has an extension to perform aggregates that can be used in tandem with this. Or you could use the built-in aggregate function of MQTTPublish.

I am looking at adding an option to publish once per day, week, month. This could be useful for expensive, rarely changing computations.

rich

[bell...@gmail.com]

This is a quickly whipped together stripped down example. It may be missing some things, but gives you an idea of how to include fields with limited config.

[MQTTPublish]
    # The MQTT server.
    # Default is localhost.
    host = localhost

    # username for broker authentication.
    # Default is None.
    username = None

    # password for broker authentication.
    # Default is None.
    password = None

    # The TLS options that are passed to tls_set method of the MQTT client.
    # For additional information see, https://eclipse.org/paho/clients/python/docs/strptime-format-codes
    [[tls]]
        # Turn tls on and off.
        # Default is true.
        enable = false

    [[topics]]
        [[[REPLACE_ME]]]
            # True if the field should not be published.
            # Valid values: True, False.
            # Default is  False
            ignore = true

            [[[[fields]]]]
                [[[[[REPLACE_ME -1]]]]]
                [[[[[REPLACE_ME -2]]]]]

On Tuesday, 7 July 2026 at 10:17:26 UTC-4 rae...@gmail.com wrote:

[bell...@gmail.com]

I could add an short ‘publish_fields’ option that is the opposite of this option, https://weewx-mqtt.github.io/publish/common-options/topics/topic-name/#ignore_fields

But noone has asked for it yet….

rich

[rae...@gmail.com]

Basically I'm looking to have access to all the calculated values that weewx produces like windchill heatindex, etc as well as historical lows and highs for these things.  That separate aggregate function looks like that would be what I'm after for that historical stuff.  Essentially my setup is I'm using mqtt subscribe to my mqtt server to get the raw data from my LoRa based weather station.  Then I want to get the calculated values from weewx back into the mqtt server so that it can be used with other mqtt connected apps.  I will also be installing the homeassistant extension as well, but I think it only creates the loop function mqtt devices I think.  I hope that helps you understand what I'm after.  Thanks again for all your help.

[Vince Skahan]

Rich doesn't ignore=false publish everything ?

[bell...@gmail.com]

It does, thanks.

So it sounds like you would be happy “opting out” of what fields to publish...

If you don’t mind installing 1.1.0 over what you installed, it will make moving forward easier, https:///github.com/weewx-mqtt/publish/releases/tag/v1.1.0

From there you should just have to update the host and topic-name. To be honest, there might be a few more tweaks, especially related to tls and userid/password.

If you have something running and are still having problems, set debug=1 and attach the log and we can go from there.

As far as aggregation, MQTTPublish has a limited function. I do believe a general purpose extension that makes it available to all of WeeWX is more appropriate. But, if you just seem quick simply aggregation, MQTTPublish does support that.

And if you want Home Assistant MQTT device discovery that is in beta.

Let me know how it goes.

rich

[bell...@gmail.com]

Yes, this was suppose to be (it is untested) example of opting in. Set ignore = true for the topic and nothing is published. Then add the fields to be published…..

[John Smith]

I am looking at adding an option to publish once per day, week, month. This could be useful for expensive, rarely changing computations.

Or you could do what I did, calc once and then cache to both ram and the storage, until and update is needed and only re-calc what needs to be re-calced... 

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/f4c519b4-6de3-4632-9a6d-96af828fac46n%40googlegroups.com.

[rae...@gmail.com]

Ok I'm a bit confused as to which extension I'm installing.  I have a [[MQTT]] section but not an [[MQTTPublish]].  Do I have the older extension installed?  Also FYI I'm installed in a docker swarm environment, but the swarm nodes have access to a NAS share that houses the data folder which contains all the customizatoins.  Do I specify 'latest' or do I specify '1.1.0'?  I thought I did install the mqtt-publish extension but these all look the same to me in terms of the install instructions unless I'm missing something.

[rae...@gmail.com]

This is the install instruction for your fork:

weectl extension install https://github.com/weewx-mqtt/publish/archive/refs/tags/latest.zip

This is the install instruction for the original

weectl extension install https://github.com/weewx-mqtt/publish/archive/refs/tags/latest.zip

That's what is confusing.  

[rae...@gmail.com]

sorry for all the posts...also it looks like your latest version is 2.0.5. 

[bell...@gmail.com]

Calculating aggregates was a poor example. If one needs to do that I would recommend using your extension and either MQTTPublish or Matthew’s MQTT extension to publish to MQTT.

But, I do see a potential need to not publish as frequently for some MQTT clients….

rich

[bell...@gmail.com]

No apologies are necessary. I hope a bit of history will help us figure out what is going on...

Matthew Wall's mqtt is the OG for publishing to MQTT. It's configuration is found under [StdRESTful][[MQTT]]. This has been around for a long time and is very stable. It is the extension most people think of when publishing to MQTT. It is here, https://github.com/matthewwall/weewx-mqtt

I need to publish to multiple topics so I developed MQTTPublish. So named to avoid confusion with Matthew's extension and align with my naming of MQTTSubscribe and MQTTReplicate. It's configuration is found under [MQTTPublish]. With these names and locations, I was hoping to make it easier for people to know what they have installed and are running. Looks like I need to do some more work on this. I have been using it for years, but only recently made it publicly available. So it has less 'real world' use and probably not as stable as Matthew's. It is here, https://github.com/weewx-mqtt/publish

John apparently has a fork of my repository. I am not sure what additional functionality he has added, nor what he is calling his extension nor what stanza he is using to configure.

I just ran the following commands:

weectl extension list
Using configuration file /home/richbell/weewx-data/weewx.conf
Extension Name    Version   Description
MQTTSubscribe     3.2.0-rc01aSource WeeWX data from MQTT.
 
weectl extension install https://github.com/matthewwall/weewx-mqtt/archive/master.zip --yes

weectl extension install https://github.com/weewx-mqtt/publish/archive/refs/tags/latest.zip --yes

weectl extension list
Using configuration file /home/richbell/weewx-data/weewx.conf
Extension Name    Version   Description
MQTTPublish       1.1.0      Publish WeeWX data to a MQTT broker.
MQTTSubscribe     3.2.0-rc01aSource WeeWX data from MQTT.
mqtt              0.24      Upload weather data to MQTT server.

Note using 'latest.zip' is just a convenience for install the latest released version of MQTTPublish. So if you want to continue. to try to use MQTTPublish, lets specify the exact version via,
weectl extension install https://github.com/weewx-mqtt/publish/archive/refs/tags/v1.1.0.zip

and see what you have installed by running
weectl extension list

Hopefully you now have MQTTPublish 1.1.0 and we can continue where we left.

Again my apologies for the confusion. I will look into how I can make this less confusing.
rich

[rae...@gmail.com]

Hey guys I think I figured everything out.  I ended up going with John's fork for the publish and got it working.  I also installed his aggregator extension but haven't ventured into it yet.  Thanks for all your help gentleman. 

[bell...@gmail.com]

Thanks for letting us know.  Let me know if there is something I can do to help reduce the confusion. 

rich

[rae...@gmail.com]

Will do, I think the only thing I got hung up on was using the correct URL to pull the extension right into the container I'm running weewx in.

[bell...@gmail.com]

Yeah, I was afraid of that when I ‘made’ MQTTPublish public. I’ll see if there is anything I can to the README.  I’ll also try to remember to ask for the exact install command used.

rich

[John Smith]

But, I do see a potential need to not publish as frequently for some MQTT clients

Wouldn't that just be a case of checking what's being published has changed? 

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/ee678375-8e16-4154-95db-aaacfefe7e4cn%40googlegroups.com.

[Greg Troxel]

John Smith <deltafo...@gmail.com> writes:

>> But, I do see a potential need to not publish as frequently for some MQTT
>> clients
>
> Wouldn't that just be a case of checking what's being published has
> changed?

Perhaps, but it gets complicated.

For me, with weewx to MQTT to Home Assistant (HA), there is a flow where
each mqtt publish results in a state update in HA which results in a
database record. Recording temperature every 15s, or whatever some
rapid LOOP does, is excessive. I have settled on every minute for a lot
of (non-weewx) temperature sensors (e.g. an Si7021 and an ESP8266 with
esphome), as a compromise between often enough and filling up the
database and making all operations slower. Probably 2 minutes would be
better.

I also have an ancient UPS (it was shiny in 1995 :-) that is polled
every second or so over serial by ups-nut, and I wrote python to send
MQTT from that. For many things, I want a value every minute or so, to
watch trends in line voltage, power usage, temperature. But, I want
immediate alerting on things like input voltage going to zero, and
battery voltage dropping sharply.

I wrote code to assess updates (vs the last transmitted values), and to
trigger a transmission if it had been long enough (60s), or if any
variable was different enough than the last report. The code is easy to
write and the art is picking "different enough". Surely my thresholds
are tuned to my UPS.

I am sending a json dictionary with everything, rather than individual
topics. That's better for MQTT, and it means all values are updated at
once, but it does mean line voltage fluctuations result in updates of
values that aren't changing.

It would be nice to have MQTTPublish/etc. be able to do this, so that
there are more rapid updates during heavy rain (every rain click feels
like news), rapid temperature drops, gusts, while not sending updates
all the time.

It would also be nice to have "send MQTT every archive interval (5
minutes) and every minute there isn't an archive transmission based on
LOOP, even if LOOP is more frequent, so I could keep my archive interval
at 5m and have 1m updates.

Of coures, one can also make a good case that sending over MQTT is cheap
and that the receiver should have the filtering. One might be sending
MQTT and using it in Belchertown, where a viewer wants updates, and in
HA, where you want a reduced update frequency when that doesn't result
in substantially wrong data*. But whether it's cheap depends on the
network path, and it could be some not-fiber-optics, ham radio, LoRa,
etc.

*By not substantially wrong data, I mean that if you do linear
interpolation between two recorded points, the data points you didn't
record will be close to that interpolated line.

[John Smith]

I am not sure what additional functionality he has added, nor what he is calling his extension nor what stanza he is using to configure.

Since I never planned to fork it, I hadn't renamed it. For now it has the uninspired name of MQTTPublishV2 until someone has a better suggestion.

As for what's changed here's the short version:

* Removed the aggregation code and turned that into it's own extension

* Added a few new configuration options to enable new functionality

* Reduced the need for repetitive configuration by shifting the heavy lifting into the python script.

* Stopped values being converted to strings by default

* Added a conversion type for compass degrees to ordinal_compass values

An exert might better illistrate 

        [[[website/stats]]]

            publish = true
            type = json
            binding = archive
            qos = 1
            retain = True
            unit_system = METRICWX
            ignore = true
            minmax = both
            append_unit_label = false
            publish_none_value = true
            day = true
            yesterday = true
            month = true
            last_month = true
            year = true
            last_year = true
            alltime = true

            [[[[fields]]]]

                [[[[[outTemp]]]]]
                    ignore = false

Adding the above to weewx.conf now turns 1 field  definition into day_outTemp_max, day_outTemp_maxtime, day_outTemp_min, day_outTemp_mintime and the same fields for yesterday_*, month_*, last_month_*, year_*, last_year_*, alltime_*

Similar things happen to other fields based on characteristics of that field, such as compass direction for wind etc a full stanza example can be viewed here.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/324cc8c3-6699-45cf-8e1c-badb414370b9n%40googlegroups.com.

[John Smith]

Perhaps, but it gets complicated.

or if any variable was different enough than the last report.  

This of course made easier by using multiple topics and setting retain on published messages.

Bonus points for parsing the retained messages on startup.

[John Smith]

I ended up going with John's fork for the publish

Since I renamed the extensiom to avoid confusion, and so the extension I forked from can live harmoniously you will need to change [MQTTPublish] to [MQTTPublisV2] if/when you install a newer version.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/b57991e9-e7c4-4487-a750-b4b1cc8c18d7n%40googlegroups.com.

[Greg Troxel]

John Smith <deltafo...@gmail.com> writes:

>>
>> Perhaps, but it gets complicated.
>
> or if any variable was different enough than the last report.
>
>
> This of course made easier by using multiple topics and setting retain on
> published messages.
>
> Bonus points for parsing the retained messages on startup.

Yes, but you need a timestamp in the message so that

weewx goes offline (down, network)

MQTT subscriber restarts

doesn't lead to logging (now, old_data). And I don't think anybody does
this. That's why I don't set retain.

[John Smith]

Yes, but you need a timestamp in the message so that

In my dealings with MQTT I've been exclusively dealing with JSON encoded strings, so a datetime is just another field in the message, did you do something different?

[rae...@gmail.com]

I preferred the individual topics because my home made weather station only updates every 3 minutes other than rain and wind which is every 5 seconds .  I didn't want to send the whole kitten kabootle every 5 seconds so that's why I went that route.  So regarding the retain question, is it better to retain or not?

[John Smith]

So regarding the retain question, is it better to retain or not?

It's not about better, but getting the last message that was sent by a device, which isn't always useful, but can be.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/0bc80c1e-71d4-4ef6-be4d-ed428826d282n%40googlegroups.com.

[bell...@gmail.com]

Thanks that should help. 

Please remove the incorrect information that MQTTPublish was forked from Matthew’s mqtt extension from your readme

[rae...@gmail.com]

I have one more question that may or may not be related to the MQTT subscribe function.  We just got some rain and my weather station sends a 0.011 inch value every bucket tip but accumulating.  So it continues to go up.  It seems the way I have my weewx set up that the rain amount is going up exponentially like it's not raining and the rain rate is huge as well as the accumulated rain is like over 1 foot which is completely ridiculous. I was under the impression that weewx would parse out that the value hasn't increased since the last read but that's not the behaviour I'm seeing.  I thought that was the default rain behaviour but maybe I'm missing something in my config.  Any thoughts?

[rae...@gmail.com]

NVM, found the contains_total flag and enabled it...

[bell...@gmail.com]

Interesting ideas. Time to move this somewhere else. I haven’t got far on any of this, but I think being able to publish on an interval bigger than the loop interval, but smaller than the archive interval might be the most interesting/useful to start with.

Leveraging WeeWX accumulators it shouldn’t be too difficult. In essence MQTTPublish would be creating its own ‘archive record’.

If anyone wants to follow along, I’ve opened https://github.com/weewx-mqtt/publish/issues/28

rich

[John Smith]

In essence MQTTPublish would be creating its own ‘archive record’.

I'm struggling to figure out the benefit of this, would this be to reduce the publishing rate? or?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/2b3fed33-5043-426e-9ac0-6740f6ed71c4n%40googlegroups.com.

[Greg Troxel]

John Smith <deltafo...@gmail.com> writes:

>> In essence MQTTPublish would be creating its own ‘archive record’.
>
> I'm struggling to figure out the benefit of this, would this be to reduce
> the publishing rate? or?

Yes, because depending on your system, that might matter.

A system with limited network capacity, or battery power to send
transmissions, or whatever, might be better sending less data, if that
less data is not meaningfully less useful.

The receiving system also requires resources. In the case of Home
Assistant, the normal practice is that each incoming update results in
a state database entry. While you can configure the MQTT receiver not
to cause that, the idea that you know the value of the observable
during a period with no data points is iffy. Using a reasonable
interval helps lot, where reasonable is a blend of short enough to
capture the signal of the underlying feature, short enough for
automations to run based on real changes, and long enough that
unnecesary and unhelpful procesing and database usage is avoided.

What if you had temperature measurements every second? Every 10 ms?

Certainly there are both people whose concerns don't lead to wanting to avoid
high-rate transmissions and people whose concerns do lead to wanting
limits.

[John Smith]

What if you had temperature measurements every second?

I solved that problem already for the soil moisture sensors + temperature/humidity sensors on my ZigBee network as the sensors send bursts of readings but there can be 30+ minutes between bursts as well.

Since the timestamps are unique, my script that copies the readings to the DB filters the readings so that there is at least 15s between readings, then later on I also prune the database so there is at least 5 minutes between readings, although the script doing the insertion could track/check when the last insertion was.

[Greg Troxel]

John Smith <deltafo...@gmail.com> writes:

>> What if you had temperature measurements every second?
>
> I solved that problem already for the soil moisture sensors +
> temperature/humidity sensors on my ZigBee network as the sensors send
> bursts of readings but there can be 30+ minutes between bursts as well.
>
> Since the timestamps are unique, my script that copies the readings to the
> DB filters the readings so that there is at least 15s between readings,
> then later on I also prune the database so there is at least 5 minutes
> between readings, although the script doing the insertion could track/check
> when the last insertion was.

That seems like an approach that works for you. There are various
transmission mechanisms, and various code running at receivers, and I
believe that a significant number of people would benefit from reduction
of communications, and ingest/db resources, especially in systems where
it's less easy to inject code (because of the person, or the system).

You seem to be suggesting that rate limits are not reasonable, because
you have a personal scheme, in one system instantiation, where you don't
need them. That's that I'm reacting to. I'm not claiming that what
you're doing is wrong, just that it's not valid to extrapolate from that
to a conclusion that all rate limits are unnecessary.

[John Smith]

I need to do better at articulating my thoughts it seems.

Yes I fully appreciate that rate limiting at the source might be beneficial and desirable for some in some circumstances, however the point I mentioned in a round about way is that while limiting loop data might be fine most of the time, there are events such as significant temperature changes before it hails and rain fall during heavy events that loop packets data would be more useful.

Which could be taken into account, like I'm planning to do, rather than blindly filtering by time.

While not applicable to this specific situation, the other point I made is not all rate limiting can be done server side.

[gary....@gmail.com]

If you just want to save Home Assistant from writing to the database, you can exclude items. They still are active in HA, just not permantly saved.
I'm using these along with some other info to make my window shades sun aware.

recorder:
  exclude:
    entities:
      - sensor.sun_altitude
      - sensor.sun_azimuth
      - sensor.solar_radiation_max_clear_sky
      - sensor.solar_radiation
    entity_globs:
      - sensor.weewx_*