Using MQTT Subscribe

[Timothy Buchanan]

I use the mqtt extension to publish to a mosquitto broker, and want to use MQTTSubscribe to receive topics posted by another device and put them into the weewx database. I installed MQTTSubscribe as an extension following the wiki, enabled it, and it didn't break anything (always good!).

Next, I may need to extend the database, since there doesn't seem to be snowLevel entries, and to map the published topics to the database. where could I find information on these two subjects? Thanks.

Timothy

[rich T]

WeeWX Version	4.1.1 using the extended schema has entries for snow.

[Timothy Buchanan]

I upgraded to 4.1.1 but I don't see these entries in the database. Do I have to install or change something to get the "extended" schema?

[Tom Keffer]

The schemas are used only when creating a new database. Upgrading to v4.x will not change your old database. 

If you wish to switch, there is a Wiki article Switching to the new wview_extended schema on how to do it.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e4f7cdca-aec1-42f7-b232-5bdec8f90501n%40googlegroups.com.

[Timothy Buchanan]

Thanks, changing the database was easy with that article. As to my other question, I'm guessing I need to put a Sensor Map in weewx.conf in the MQTTSubscribe section, something like snow/snowlevel = snow?

[Bell...@gmail.com]

What is the topic and format of the MQTT data? For example, are you publishing to a topic of 'snow/snowlevel' and the payload is the actual snow level?

Some information is here, https://github.com/bellrichm/WeeWX-MQTTSubscribe/wiki/Configuring But, I'm happy to walk you through it.

rich

[rich T]

What type of sensor are you using for measuring snow depth?

[Timothy Buchanan]

Thanks, Rich, I should be able to edit weewx.conf based on the example at the bottom of that page.

I am using an ESP8266 board with an ultrasonic sensor and a temperature sensor (to calibrate the speed of sound), and programming in the Arduino IDE. I'll 3D print a case and mount it above my deck. The materials cost about $15.

I'd be happy to post the code here, under a new topic, once I get it working.

Timothy

[Timothy Buchanan]

Attached is the extension material that I put into weewx.conf. But when I set enable = true, weewx stops archiving data. Is there an error in this configuration, or could subscribe be incompatible with another service? I'm using the Weatherflowudp driver with mqtt and GW1000 extensions.

[rich T]

I did try using a MAXBOT ultrasonic sensor to measure snow depth but found it was a little flaky.  Thinking this year going to use a VL53L0X sensor.

[Bell...@gmail.com]

So, the [[[first/topic]]] are meant to be replaced with the actual topic. So it would be something like this

```

[[Topics]]

  [[[topic name that snowDepth is published on]]]

    [[[[topic name that snowDepth is published on]]]]

      name = snowDepth

  [[[topic name that snowRate is published on]]]

    [[[[topic name that snowRate is published on]]]]

      name = snowRate

```

The duplication is an artifact of dealing with json and keyword payloads. The ```use_topic_as_fieldname``` option can be used to make the config a bit prettier.

```

[[Topics]]

  use_topic_as_fieldname = true

  [[[topic name that snowDepth is published on]]]

    name = snowDepth

  [[[topic name that snowRate is published on]]]

    name = snowRate

```

Note, if snowDepth is actually published on the topic snowDepth, the ```name``` option can be left off.

I don’t think that you want to set contains_total=true for snowDepth. This is used when the field contains a total and it needs to be converted into an increment for WeeWX.

I’ll work on clarifying the wiki.

With that said, it shouldn’t have broken WeeWX. If you are up for it, before changing the config, setting debug=1, restarting WeeWX for a couple of archive intervals and attaching the log would be appreciated .

rich

[Timothy Buchanan]

Thanks, Rich. I'll try it when I get back. I'll be in our second home for a few days: going to the gun range and the clothing-optional resort (not at the same time!).

[Timothy Buchanan]

I've attached a syslog extract showing where weewx crashed. it seems that "volt" is an invalid unit for my topic. i don't know why but for now I commented out that topic and its parameters. Now, weewx will continue running when subscribe is enabled, but subscribed topics are not being posted to the database. Here is the first topic in weewx.conf:

        # The first topic

# MQTT Topic

[[[snow/snowDepth]]]

# MQTT name

[[[[snowDepth]]]]

# weewx name

name = snowDepth

ignore = false

contains_total = false

conversion_type = float

units = inch

I used a terminal to publish "6" to snow/snowDepth on Mosquitto. Another terminal window command to subscribe to snow/snowDepth received the "6" but the database entries for snowDepth are null. Is this configuration of topics still not correct. Thanks.

[Bell...@gmail.com]

That could be simplified, but looks like it should work. The quickest and easiest way to proceed is to set debug to 1, restart WeeWX, let it run through an archive cycle during which you published to that topic and then post the log.

Re: snowBatteryStatus - The units config option is only needed if the field units do not match the units expected by the unit_system. So eliminating will at least get the data in the DB. 

Note, units is not needed for snowDepth because inches is the Units for that field in the US unit_system.

rich

[Timothy Buchanan]

syslog extract posted. weewx is loading the service, though it also seems to subscribe to the data which MQTT is publishing from weewx. the second section shows that subscribe receives the data I published from the terminal (6 inches of snow), but rejects it as outside of interval. Is there an entry to be made somewhere to sync incoming data? I note that another extension I use (GW1000) needed dateTime = datetime in a field map to work properly.

[Bell...@gmail.com]

Ah, crap.  This has to do with an attempt to quality control data by time. Next major release, I really need to rework it...

Since you have individual payloads (no timestamp from the origin) add the following under [[topics]]

ignore_start_time = true

Ignore_end_time = true

We are getting close.  Sorry for the pain.

rich

[Timothy Buchanan]

I added those this way:

# The topics to subscribe to.

    [[topics]]

        # Units for MQTT payloads without unit value.

        # Valid values: US, METRIC, METRICWX

        # Default is: US

        unit_system = US

ignore_start_time = true

Ignore_end_time = true

but I still get this:

Oct  9 20:38:29 raspberrypi weewx[30461] INFO weewx.restx: MQTT: Published record 2020-10-09 20:38:47 MDT (1602297527)

Oct  9 20:38:30 raspberrypi weewx[30461] DEBUG user.MQTTSubscribe: (Service) MessageCallbackProvider data-> incoming topic: snow/snowDepth, QOS: 0, retain: 0, payload: b'7'

Oct  9 20:38:30 raspberrypi weewx[30461] DEBUG user.MQTTSubscribe: (Service) TopicManager data-> incoming snow/snowDepth: snow/snowDepth: 7.0

Oct  9 20:38:31 raspberrypi weewx[30461] DEBUG user.MQTTSubscribe: (Service) TopicManager data-> outgoing snow/snowDepth: dateTime: 1602297510.2596123, snow/snowDepth: 7.0, usUnits: 1

Oct  9 20:38:31 raspberrypi weewx[30461] INFO user.MQTTSubscribe: (Service) TopicManager ignoring record outside of interval 1602297510.259612 1602297529.000000 1602297510.259612 dateTime: 1602297510.2596123, snow/snowDepth: 7.0, usUnits: 1

Oct  9 20:38:31 raspberrypi weewx[30461] DEBUG user.MQTTSubscribe: (Service) TopicManager data-> outgoing accumulated snow/snowDepth: 

Oct  9 20:38:31 raspberrypi weewx[30461] DEBUG user.MQTTSubscribe: (Service) data-> final packet is 2020-10-09 20:38:49 MDT (1602297529): avg_distance: 0, dateTime: 1602297529, lightning_strikes: 0, outHumidity: 23, outTemp: 13.34, outTempBatteryStatus: 2.919, pressure: 744.7, usUnits: 17

Oct  9 20:38:31 raspberrypi weewx[30461] INFO weewx.restx: MQTT: Published record 2020-10-09 20:38:49 MDT (1602297529)

Oct  9 20:38:32 raspberrypi weewx[30461] DEBUG user.MQTTSubscribe: (Service) data-> final packet is 2020-10-09 20:38:50 MDT (1602297530): dateTime: 1602297530, usUnits: 17, windDir: 336, windSpeed: 0.72

Oct  9 20:38:32 raspberrypi weewx[30461] INFO weewx.restx: MQTT: Published record 2020-10-09 20:38:50 MDT (1602297530)

[Graham Eddy]

i know nothing of this driver, but these look like driver-global attributes, which should be declared in the top-level driver section to prevent them being absorbed into a subsection (i.e. in this case, before [[topics]])

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/9baa8ed4-10ea-4722-a550-a4299dceb817n%40googlegroups.com.

[Bell...@gmail.com]

Thanks for the log. Looks like there is a bug with the ignore_start_time option.  Fix looks easy, I just need to make sure there are no side effects.

I’ll let you know when the fix is available. 

- rich

[Timothy Buchanan]

OK, thanks  for your help.

[Bell...@gmail.com]

I created a pre-release with the fix. It is here, https://github.com/bellrichm/WeeWX-MQTTSubscribe/releases/tag/v1.6.2-rc03

Hopefully this is it. 

rich

[Timothy Buchanan]

I used wee_extension to uninstall the old version then install the new. I used the configuration as shown in the attached file. When I publish to a topic, Topic Manager still ignores it as outside of interval (syslog extract in attached file).

I'm thinking that MQTTSubscribe is not compatible with the mqtt extension. I note that MQTTSubscribe is publishing records of data that it is not subscribed to, but which are published by mqtt. Have you tried running MQTTSubscribe and mqtt together?

[Bell...@gmail.com]

I've run both extensions off and on, so I don't think there is any conflict. The root-root cause is time skew between your pi and weather station and MQTTSubscribe being a bit overly aggressive on its quality control. Setting ignore_start_time and ignore_end_time should turn this off completely. Since the incoming data has no dateTime, this seems the most logical approach.

I noticed in the latest attached config, those options are in the wrong place. Move them under [[topics]] after 

  unit_system = US

  ignore_start_time = true 

  ignore_end_time = true

Also, Ignore_end_time has a capital "I".

If it is still not working, please attach the log from startup. That way we can see how the options are being processed.

I think we are real close. -rich

[Timothy Buchanan]

After moving the ignore lines, I am happy to report that the time problem is solved; that is, it is no longer rejecting the topic. However, it is still not making it into the database. I think this is because it is not converting from snow/snowDepth to snowDepth. See attached syslog extract and weewx.conf for section where topic is configured. Is there a mistake in this topic configuration?

[Bell...@gmail.com]

Sorry, I didn’t get the config quite right. I don’t subscribe to ‘individual’ topics, so this is untested, but should work.  If it doesn’t, I’ll spin up a debug environment to figure it out.

[[topics]]

  Keep the current settings, unit_system, ignore_start_time, ignore_end_time, etc.

  # The next option will take the tail end of the topic and use it as the MQTT field name

  # So, if the topic is snow/snowDepth, the MQTT field name is snowDepth

  # Only valid when the payload is type ‘individual’

  use_topic_as_fieldname = true

  [[[snow/snowDepth]]]

    # Because snowDepth is the WeeWX name and the defaults work for you, nothing else is needed.

   [[[snow/snowRate]]]

     # Not sure, but the defaults should probably work here...

Again, if this doesn’t work; post the log and I’ll spin up a debug environment to figure out what is going on.  

rich

[Timothy Buchanan]

See attached.

[Timothy Buchanan]

I  decided to try simplifying the topic to just snowDepth, as this:

 The topics to subscribe to.

    [[topics]]

        # Units for MQTT payloads without unit value.

        # Valid values: US, METRIC, METRICWX

        # Default is: US

        unit_system = US    

    ignore_start_time = true

        ignore_end_time = true

use_topic_as_fieldname = true

        

        # The first topic

        # MQTT Topic

        [[[snowDepth]]]

            # MQTT name

            [[[[snowDepth]]]]

                # weewx name

                name = snowDepth

                ignore = false

                contains_total = false

                conversion_type = float

                units = inch

when I publish 2.5 to snowDepth, I get this error in syslog:

Oct 12 08:12:52 raspberrypi weewx[29222] DEBUG user.MQTTSubscribe: (Service) MessageCallbackProvider data-> incoming topic: snowDepth, QOS: 0, retain: 0, payload: b'2.5'

Oct 12 08:12:52 raspberrypi weewx[29222] DEBUG weewx.units: Unable to convert from inch to None

Oct 12 08:12:52 raspberrypi weewx[29222] ERROR user.MQTTSubscribe: (Service) MessageCallbackProvider on_message_individual failed with: None

Oct 12 08:12:52 raspberrypi weewx[29222] ERROR user.MQTTSubscribe: (Service) **** MessageCallbackProvider Ignoring topic=snowDepth and payload=b'2.5'

Could you perhaps post a working configuration file, any topic, any unit, etc. just one that works on your weewx and I will plug it in and see what happens?

[Bell...@gmail.com]

Well, I don’t understand how that configuration even got that far. I would have expected the previous error. But just seeing log snippets is like walking around blindfolded.  Base on your last config snippet, replace it with 

        The topics to subscribe to.

        [[topics]]

        # Units for MQTT payloads without unit value.

        # Valid values: US, METRIC, METRICWX

        # Default is: US

        unit_system = US    

        ignore_start_time = true

        ignore_end_time = true

        use_topic_as_fieldname = true

        

        # The first topic

        # MQTT Topic

        [[[snowDepth]]]

This last error has to do with trying to convert the units of snowDepth. Either it WeeWX does not fully support that, or more likely I am doing something wrong. But since everything is in the US unit_system, you don’t need to specify units at the field level.

[Timothy Buchanan]

See attached.

[Bell...@gmail.com]

Agreed, two separate problems.

Not yet sure about the units problem.  But, you don’t need that option. I opened an issue to dig into it when I have time.

The second is because it is an invalid configuration. Have you tried the snippet that is in my last post?  Namely,

        

        [[topics]]

        # Units for MQTT payloads without unit value.

        # Valid values: US, METRIC, METRICWX

        # Default is: US

        unit_system = US    

        ignore_start_time = true

        ignore_end_time = true

        use_topic_as_fieldname = true

        

        # The first topic

        [[[snowDepth]]]

Or with nested topics, 

        [[topics]]

        # Units for MQTT payloads without unit value.

        # Valid values: US, METRIC, METRICWX

        # Default is: US

        unit_system = US    

        ignore_start_time = true

        ignore_end_time = true

        use_topic_as_fieldname = true

        

        # The first topic

        [[[snow/snowDepth]]]

            name = snowDepth

These would be the simplest to start with. 

And again, full logs instead of snippets would make this so much easier.

rich

[Timothy Buchanan]

The use_topic_as_fieldname=true option gives an error. Can't use it; don't need it with non-nested topics.

Not being able to define units is more important as it makes it more difficult to format for display. It'd be nice to have that working, but I can use this as is. Thanks again for your work.

[Bell...@gmail.com]

Glad it is working for you.

I'm stumped on why use_topic_as_fieldname = true doesn't work for you. It works fine for me.

As far as the units option, from what I can infer of your setup, you don't need it. You have set the unit_system to US. This tells MQTTSubscribe that observations coming in are in US units. If for some reason snowDepth was coming in as cm, you would need to set units = cm (don't quote me on cm being the right value). This would tell MQTTSubscribe to convert snowDepth from cm to inches, so it is in the same units as other incoming data. It has nothing to do with displaying, only with ensuring that the data is inserted into the DB in the correct units.

rich