MQTT for remote site and min data transfer - MQTT Driver as input to weewx

[Louis De Lange]

I  have a remote site that I want to connect to my home network using MQTT and utilize minimal data transfer.  

The plan is to have weewx on one Raspberry Pi Zero at the remote site receiving the data from the weather station, archive it and publish to a MQTT broker using mwall's MQTT uploader.   The MQTT broker will be Mosquitto running on my home network.  Finally, I want to have another instance of weewx running on a raspberry pi on my home network using Bill Morrow's MQTT driver as input.

Right now I am trying to test the arrangement using multiple PI's on my home network in a simulation.  

I am able to successfully publish the data from my local station to the MQTT broker running on a different PI, and subscribe to it  with mosquitto tool mosquitto-sub - so we know publishing works..

Where things fall apart is that I cannot get Bill Morrow's MQTT driver tor work.  Weewx starts OK, but it just waits for an archive loop and nothing else happens.

The following is the log output from starting weewx.

Sep 08 14:37:12 testpi systemd[1]: Starting LSB: weewx weather system...

Sep 08 14:37:13 testpi weewx[21197]: engine: Initializing weewx version 3.8.2

Sep 08 14:37:13 testpi weewx[21197]: engine: Using Python 2.7.13 (default, Nov 24 2017, 17:33:09)

                                     [GCC 6.3.0 20170516]

Sep 08 14:37:13 testpi weewx[21197]: engine: Platform Linux-4.14.50+-armv6l-with-debian-9.4

Sep 08 14:37:13 testpi weewx[21197]: engine: Locale is 'en_CA.UTF-8'

Sep 08 14:37:13 testpi weewx[21197]: engine: pid file is /var/run/weewx.pid

Sep 08 14:37:13 testpi weewx[21186]: Starting weewx weather system: weewx.

Sep 08 14:37:13 testpi systemd[1]: Started LSB: weewx weather system.

Sep 08 14:37:13 testpi weewx[21201]: engine: Using configuration file /etc/weewx/weewx.conf

Sep 08 14:37:13 testpi weewx[21201]: engine: Loading station type wxMesh (user.wxMesh)

Sep 08 14:37:14 testpi weewx[21201]: wxMesh: MQTT host is localhost

Sep 08 14:37:14 testpi weewx[21201]: wxMesh: MQTT topic is weather

Sep 08 14:37:14 testpi weewx[21201]: wxMesh: MQTT client is wxclient

Sep 08 14:37:14 testpi weewx[21201]: wxMesh: polling interval is 1.0

Sep 08 14:37:14 testpi weewx[21201]: wxMesh: label map is {'dateTime': 'dateTime', 'outTemp': 'outTemp', 'outHumidity': 'outHumidity', 'inTemp': 'inTemp', 'i

Sep 08 14:37:14 testpi weewx[21201]: engine: StdConvert target unit is 0x1

Sep 08 14:37:14 testpi weewx[21201]: wxcalculate: The following values will be calculated: barometer=prefer_hardware, windchill=prefer_hardware, dewpoint=pre

Sep 08 14:37:14 testpi weewx[21201]: wxcalculate: The following algorithms will be used for calculations: altimeter=aaNOAA, maxSolarRad=RS

Sep 08 14:37:14 testpi weewx[21201]: engine: Archive will use data binding wx_binding

Sep 08 14:37:14 testpi weewx[21201]: engine: Record generation will be attempted in 'hardware'

Sep 08 14:37:14 testpi weewx[21201]: engine: Using archive interval of 300 seconds (specified in weewx configuration)

Sep 08 14:37:14 testpi weewx[21201]: engine: Using binding 'wx_binding' to database 'weewx.sdb'

Sep 08 14:37:14 testpi weewx[21201]: manager: Starting backfill of daily summaries

Sep 08 14:37:14 testpi weewx[21201]: restx: StationRegistry: Registration not requested.

Sep 08 14:37:14 testpi weewx[21201]: restx: Wunderground: Posting not enabled.

Sep 08 14:37:14 testpi weewx[21201]: restx: PWSweather: Posting not enabled.

Sep 08 14:37:14 testpi weewx[21201]: restx: CWOP: Posting not enabled.

Sep 08 14:37:14 testpi weewx[21201]: restx: WOW: Posting not enabled.

Sep 08 14:37:14 testpi weewx[21201]: restx: AWEKAS: Posting not enabled.

Sep 08 14:37:14 testpi weewx[21201]: restx: MQTT: service version is 0.18

Sep 08 14:37:14 testpi weewx[21201]: engine: Starting up weewx version 3.8.2

Sep 08 14:37:14 testpi weewx[21201]: engine: Starting main packet loop.

The following is the wxMesh section of the weewx.conf file.  

[wxMesh]

    driver = user.wxMesh

    # MQTT specifics

    host = localhost    # MQTT broker hostname

        username = XXXXXXX    # MQTT broker user name. Assumption is your broker requires authentication

    password = XXXXXXXX

    topic = weather    # topic is all weather (indoor and outdoor, e.g.)

    poll_interval = 1

    [[label_map]]

        dateTime = dateTime

        outTemp = outTemp

        outHumidity = outHumidity

        inTemp = inTemp

        inHumidity = inHumidity

        barometer = barometer

I am not sure about the following things in the wxMesh section of the weewx.conf file

1. Is the topic "weather" correct?  When I subscribe to the feed using mosquitto-sub the correct syntax is "weather/#"

2. In the MQTT uploader  I can select "individual" or "aggregate" records to upload.  What does the MQTT input driver expect?

Any help from Bill or anyone else would be appreciated.

Louis  

[Pat]

Hi Louis, I don't have any specific experience with wxMesh, but maybe I can help a little with MQTT. 

In general, the # in a topic ID means "I want to see everything in this topic". This is good for debugging, but typically bad practice for anything else. For example if my broker is available to the public and anonymous users can publish (hopefully you're using an ACL if your broker is publicly available!), then they can post to weather/hello, or weather/mywebsite, etc. and your driver will ingest everything on weather because you're subscribed to weather/#. 

Depending on your weewx.conf configuration, your data is probably on weather/loop. Try to subscribe to that in your mosquitto_sub client to verify. 

Also, my guess would be you want aggregate since that will put all observations in 1 published message, as opposed to individual topics like weather/outTemp, weather/humidity, weather/barometer, etc. 

Hope this helps a little bit. 

[Louis De Lange]

Pat,

Thank you for the insight.  I already understood most of what you described, but your point that the data should be on weather/loop worked.  

So changing that now the loop is activated when data arrives, but I am getting an error about data type for dateTime.  

  File "/usr/bin/weewxd", line 64, in <module>

    weewx.engine.main(options, args)

  File "/usr/share/weewx/weewx/engine.py", line 877, in main

    engine.run()

  File "/usr/share/weewx/weewx/engine.py", line 191, in run

    self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, packet=packet))

  File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent

    callback(event)

  File "/usr/share/weewx/weewx/wxservices.py", line 45, in new_loop_packet

    self.calc.do_calculations(event.packet, 'loop')

  File "/usr/share/weewx/weewx/wxservices.py", line 191, in do_calculations

    getattr(self, 'calc_' + obs)(data_us, data_type)

  File "/usr/share/weewx/weewx/wxservices.py", line 300, in calc_maxSolarRad

    data['dateTime'], self.atc)

KeyError: 'dateTime'

I suspect the driver reads it in as a REAL number, not an integer.  I'll dig into that a bit further and report back.

[gjr80]

Hi,

That error is telling you that the field dateTime does not exist in your loop packet so I doubt the issue is whether dateTime is an integer or a float, its more fundamental than that. I would be looking at the wxMesh driver; do you have it configured properly, are there any field maps that need to be configured, typos? Remember case matters. If you can't spot a config error turn debug up as high as you can, both in WeeWX and the driver (if it is supported). I would set debug=2 in weewx.conf. Have a read of the wxMesh driver file, are there any comments up front about setting debug levels or troubleshooting, it might not be setting a option named debug, it could be debug_read etc. What you want is to get the driver to spit out the loop packet to screen/log so you can see if the driver is doing what it should or if the issue is further up the processing tree.

Gary

[Louis De Lange]

Thanks Gary,

I have been investigating and found as much - but in the end could not get it to work by config alone.

I have turned to a completely different technical solution and are able to upload the remote data at approx 5kB / archive - so it solves my immediate need.

I am still intrigued by the concept of doing this with MQTT and I will play around until I get it to work.  If I can shave off a few more kB per upload using MQTT then I can upload loops rather than archives.

[Louis De Lange]

Time to revisit this old post as I finally spent some time with morrowwm's wxMesh driver to make slight modifications required for this use case.  

It has been working well for the last week and I am making it available in case it may be useful for someone else.  If there is sufficient interest I will clean it up and upload it to weewx-wiki as a separate driver.

The following is a description from the driver comment section describing how it works.  

# Original driver created by morrowwm

# This is a specific modification for the use case of a station in a remote location

# where bandwidth and data allowance is limited and a lightweight data transfer protocol

# such as MQTT is required.

#

# The system requires a local (at the weather station) instance of weewx to collect data,

# archive locally and publish complete archive records to MQTT broker on a remote server.

# The remote server runs a remote instance of weewx that receives the complete archive

# records by subsribing to MQTT topic on the remote server, archives remotely and uploads

# generated webpages to the webserver.

#

# The local instance of weewx publishes the data to the MQTT broker using mwall's 

# weewx-mqtt uploader with the following settings

# 

#        server_url = mqtt://<username>:<password>@<mqtt_host>:1883/

#        topic = weather

#        unit_system = US

#        binding = archive

#        aggregation = aggregate

#        append_units_label = false

#

# MQTT broker set up as described in this excellent guide by Pat O'Brien

# https://obrienlabs.net/how-to-setup-your-own-mqtt-broker/

#

## The units must be weewx.US:

#   degree_F, inHg, inch, inch_per_hour, mile_per_hour

#

# To use this driver, put this file in the weewx user directory, then make

# the following changes to weewx.conf:

#

# [Station]

#     station_type = wxMesh

# [wxMesh]

#     host = localhost           # MQTT broker hostname

#     username = <username>      # MQTT broker user name. Assumption is your broker requires authentication

#     password = "password"

#     topic = weather/+          # topic

#     driver = user.wxMesh

#

# Since the subscribed topic already uses weewx standard labels no labell mapping is required.

#

# This driver has not been tested for use with Python 3