MQTT weather data as input to weewx

[Siegfried H.]

I recently discovered this great software, thx for that.

I have weewx running on a Raspberry Pi 3 (deb install) with the mqtt extension.
Current driver is simulator.
My mqtt broker is mosquitto running on a different Pi.
Setup is fine, all mqtt services are up and running, no errors in the log.

My goal is to have my existing weather data (mostly from sensors connected to an esp8266), which are published to my mosquitto server and then send to my home automation system (openHAB) as input to weewx from where I would like to upload the data to weather underground.

Is this possible and if yes, how?

Thx in advance.

[mwall]

On Saturday, August 6, 2016 at 4:03:30 AM UTC-4, Siegfried H. wrote:

My goal is to have my existing weather data (mostly from sensors connected to an esp8266), which are published to my mosquitto server and then send to my home automation system (openHAB) as input to weewx from where I would like to upload the data to weather underground.

Is this possible and if yes, how?

first you have to get the data from your sensors into weewx.  you'll have to write a driver to talk to the esp8266, but that should be pretty easy since the esp8266 interface is documented and pretty straightforward.  one approach for that is to have the weewx driver poll each of the esp8266 for data over wifi tcpip.  you could even write some simple python and run it in micropython on the esp8266 itself, so that a bunch of esp8266 devices push their data to a single weewx instance running on a heavy-duty computer such as a rpi ;)

it typically takes a few hours to write a basic weewx driver - see the weewx customization guide for details, and use one of the simple drivers such as the ultimeter (ultimeter.py) or texas weather instruments (twi.py) as a starting point.

once you have data in weewx, you'll have to get it into openhab.  that should be as easy as using the mqtt extension to weewx.  assuming you already have an mqtt broker for openhab, installing and configuring the weewx mqtt extension should take well under an hour.

you also might want to import existing data into weewx.  for that, consider gary's wee_import utility.  see this thread:

https://groups.google.com/forum/#!topic/weewx-user/gc_F0w_rtQw

unless your data are already in a standard format, it could take awhile to wrangle the data.

m

[Siegfried H.]

Thanks for the fast reply.
 

you'll have to write a driver to talk to the esp8266

Okay, then I was on the wrong track: I thought because my weather data is already published from the esp8266 to the mqtt broker (and from there into openHAB) I could simply use the weewx mqtt extension for inputting this data into weewx.
But it looks like the weewx mqtt extension is only able to output/publish data to the mqtt broker (I tried that and it works fine)

you also might want to import existing data into weewx.  for that, consider gary's wee_import utility.  see this thread:

Thx, will take a deeper look into that.

Keep up the great work with weewx!

[mwall]

On Sunday, August 7, 2016 at 1:25:08 AM UTC-4, Siegfried H. wrote:

you'll have to write a driver to talk to the esp8266

Okay, then I was on the wrong track: I thought because my weather data is already published from the esp8266 to the mqtt broker (and from there into openHAB) I could simply use the weewx mqtt extension for inputting this data into weewx.
But it looks like the weewx mqtt extension is only able to output/publish data to the mqtt broker (I tried that and it works fine)

siegried,

i had not considered that approach!  an mqtt driver that feeds weewx with data from any mqtt broker would useful for many use cases.

i have seen two general approaches: (1) publish each observation, e.g., /telemetry/outside_temperature with data 32.3, or (2) publish a group of observations as json data, e.g., /telemetry/weather/loop with data {'outside_temperature': 32.2, 'outside_humidity': 42, 'pressure': 1011.2}

what topic structure does openhab expect?

what other mqtt topic topologies have you seen?

m

[Thomas Keffer]

I have a strong preference for publishing observation packets under a single timestamp. Not only is it more efficient, but they make a natural semantic group, as they all come from the same source.

This is what WeeRT does.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Siegfried H.]

Thanks for taking your time,
 

an mqtt driver that feeds weewx with data from any mqtt broker would useful for many use cases.

Definitely.
 

(1) publish each observation, e.g., /telemetry/outside_temperature with data 32.3

This would be my usecase.

> what topic structure does openhab expect?

Actually in my approach it wouldn't matter, because the weather data is already in openHAB via mqtt through a mosquitto broker, but to answer that:
Weather data, f.e. outside temperature is sent from DS18B20 sensor via one wire protocol over wifi to an esp8266 (esp easy firmware loaded), in addition to that it is published from the esp8266 via implemented mqtt support to a broker, from the broker it is imported to openhab via openHAB mqtt binding which can be configured to any topic. An import looks like:

Number TemperatureOutside "Temperature [%.1f °C]"  { mqtt="<[mosquitto:/esp8266three/temperature:state:default]" }

so the topic is /esp8266three/temperature or any other you may have configured.

> what other mqtt topic topologies have you seen?

Despite from weather data (light intensity, pressure, humidity, temperature) you are able to control relays, switches, displays, analog in- or outputs and many many more, take a look here under "Applications".
And this is only the espeasy firmware, there are many more firmwares available for the esp8266, including arduino ide based or for lua scripting.
And: it's dirty cheap, the esp8266 with a temperature sensor comes to less than 10 Euro.

So I thought: if the weather data is already present on a mqtt broker, there may be a function to import those to weewx ... ;-)

[Siegfried H.]

This is what WeeRT does.

I'm only a weewx beginner, although I have read your wiki for WeeRT, I cannot bring in the context for my usecase, sorry. 

[Thomas Keffer]

Matthew and I have been working on a use case where an MQTT broker is used as a source for WeeRT.

-tk

On Sun, Aug 7, 2016 at 8:21 AM, Siegfried H. <siegfried...@gmail.com> wrote:

This is what WeeRT does.

I'm only a weewx beginner, although I have read your wiki for WeeRT, I cannot bring in the context for my usecase, sorry. 

--

[Siegfried H.]

Matthew and I have been working on a use case where an MQTT broker is used as a source for WeeRT.

Ahh, got it. But to be honest: it looks like I would have to install a whole lot of programs and scripts to make it work. So at the moment (not much free spare time and lacking of programming skills) I don't think that is a solution for me.
Thx anyway. 

[Robert Mantel]

I know this is an old thread, but did you guys ever get around to making a driver that subscribes to an mqtt server to get sensor data?  My RFM69 sensors connect to a gateway which parses the data into my mqtt server, so now I'm looking for a way to get it into weewx.  

On Sunday, 7 August 2016 11:55:36 UTC-4, Tom Keffer wrote:

Matthew and I have been working on a use case where an MQTT broker is used as a source for WeeRT.

-tk

On Sun, Aug 7, 2016 at 8:21 AM, Siegfried H. <siegfried...@gmail.com> wrote:

This is what WeeRT does.

I'm only a weewx beginner, although I have read your wiki for WeeRT, I cannot bring in the context for my usecase, sorry. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Bill Morrow]

On Wednesday, 20 September 2017 14:20:41 UTC-3, Robert Mantel wrote:

I know this is an old thread, but did you guys ever get around to making a driver that subscribes to an mqtt server to get sensor data?  My RFM69 sensors connect to a gateway which parses the data into my mqtt server, so now I'm looking for a way to get it into weewx.  

Robert, have a look at the driver I created and has been forked a couple of times. It is discussed in https://groups.google.com/d/topic/weewx-development/tNDnGNe9Z_M/discussion

https://github.com/morrowwm/weewxMQTT - my version

https://github.com/bonjour81/ESP8266-MQTT-WEEWX - Frederic's fork

Another fellow has worked on Frederic's version.

[Ørjan Hofstad]

Hi
Does your mqtt driver work?

[Colin Larsen]

Works great for me outputting data

On Mon, 3 Sep 2018, 09:16 Ørjan Hofstad, <oh...@comstation.net> wrote:

Hi
Does your mqtt driver work?

[Ørjan Hofstad]

Im looking for a driver that send sensor data to weewx over Mqtt. Is this the right one for me then?

Can anyone show me a example where to change settings in the config files ect.

I have installed weewx and mosquitto broker and client on my Pi.

[Ørjan Hofstad]

Here is what I have tried. But cant get it to work. I was just testing over mqtt client. Maybe Im sending the data wrong or something.

This is the setting I have set in /etc/weewx/weewx.conf

loop_on_init=true
station_type = wxMesh

[wxMesh]
host = 127.0.0.1
client = Arduino-station
username = username
password = password
topic = weather
driver = user.wxMesh
poll_interval = 1

[[label_map]]
TIME = dateTime
HUMT = outTemp
RHUM = outHumidity
INTE = inTemp
INHU = inHumidity
BARP = barometer
IRRA = radiation
PHOV = supplyVoltage
BATV = consBatteryVoltage
AMBT = extraTemp1
SYST = extraTemp2

This is the setting I have set in /usr/share/weewx/user/wxMesh.py

where to find the data file
self.host = stn_dict.get('host', '127.0.0.1')
self.topic = stn_dict.get('topic', 'weather')
self.username = stn_dict.get('username', 'username')
self.password = stn_dict.get('password', 'password')
self.client_id = stn_dict.get('client', 'Arduino-station') # MQTT client id - adjust $

[Tomasz Lewicki]

Hi Ørjan.

Did you get your config to work? If yes, where was the problem?

[Norbert F]

Here is another driver that works good for me https://github.com/bellrichm/WeeWX-MQTTSubscribe

[Ørjan Hofstad]

Hi,
No the plugin does not work. But there is another guy that have made a new plugin. Maybe that will work.

https://github.com/bellrichm/WeeWX-MQTTSubscribe

[Tomasz Lewicki]

Thank you Ørjan and NorbertF. I have done clean installation of Raspberry Pi OS, Weewx (APT method) and WeeWX-MQTTSubscribe. Before installing MQTTSubscribe everything was going without problems (no errors in weewx.log). But after install of MQTTSubscribe, trying to restart Weewx I still got the message in the log:

Jul 27 09:52:19 pogoda weewx[3676]: Stopping weewx weather system: weewx not running....
Jul 27 09:52:20 pogoda weewx[3698] INFO __main__: Initializing weewx version 4.1.1
Jul 27 09:52:20 pogoda weewx[3698] INFO __main__: Using Python 3.7.3 (default, Dec 20 2019, 18:57:59) #012[GCC 8.3.0]
Jul 27 09:52:20 pogoda weewx[3698] INFO __main__: Platform Linux-5.4.51-v7+-armv7l-with-debian-10.4
Jul 27 09:52:20 pogoda weewx[3698] INFO __main__: Locale is 'en_US.UTF-8'
Jul 27 09:52:20 pogoda weewx[3698] INFO __main__: PID file is /var/run/weewx.pid
Jul 27 09:52:20 pogoda weewx[3702] INFO __main__: Using configuration file /etc/weewx/weewx.conf
Jul 27 09:52:20 pogoda weewx[3702] INFO __main__: Debug is 0
Jul 27 09:52:20 pogoda weewx[3702] INFO weewx.engine: Loading station type MQTTSubscribeDriver (user.MQTTSubscribe)
Jul 27 09:52:20 pogoda weewx[3686]: Starting weewx weather system: weewx.
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) Version is 1.6.1
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) Log level: 0
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) Log debug setting: 0
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) Log console: None
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) Log file: None
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) overlap is 0.0
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) message_callback_provider_name is user.MQTTSubscribe.MessageCallbackProvider
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) clientid is MQTTSubscribe-5908
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) client_session is True
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) host is localhost
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) port is 1883
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) keepalive is 60
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) username is None
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) password is set
Jul 27 09:52:20 pogoda weewx[3702] INFO user.MQTTSubscribe: (Driver) Archive topic is None
Jul 27 09:52:20 pogoda weewx[3702] ERROR weewx.engine: Import of driver failed: [Errno 111] Connection refused (<class 'ConnectionRefusedError'>)
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****  Traceback (most recent call last):
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/share/weewx/weewx/engine.py", line 103, in setupStation
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      self.console = loader_function(config_dict, self)
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/share/weewx/user/MQTTSubscribe.py", line 1521, in loader
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      return MQTTSubscribeDriver(**config_dict[DRIVER_NAME])
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/share/weewx/user/MQTTSubscribe.py", line 1542, in __init__
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      self.subscriber = MQTTSubscribe(stn_dict, self.logger)
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/share/weewx/user/MQTTSubscribe.py", line 1324, in __init__
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      self.client.connect(host, port, keepalive)
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 937, in connect
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      return self.reconnect()
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 1071, in reconnect
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      sock = self._create_socket_connection()
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/local/lib/python3.7/dist-packages/paho/mqtt/client.py", line 3522, in _create_socket_connection
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/lib/python3.7/socket.py", line 727, in create_connection
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      raise err
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****    File "/usr/lib/python3.7/socket.py", line 716, in create_connection
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****      sock.connect(sa)
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****  ConnectionRefusedError: [Errno 111] Connection refused
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL __main__: Unable to load driver: [Errno 111] Connection refused
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL __main__:     ****  Exiting...

What is going on? I don't understand which connection is refused. Is there anybody to give some hint?

[Tomasz Lewicki]

W dniu poniedziałek, 27 lipca 2020 10:01:02 UTC+2 użytkownik Tomasz Lewicki napisał:

Thank you Ørjan and NorbertF. I have done clean installation of Raspberry Pi OS, Weewx (APT method) and WeeWX-MQTTSubscribe. Before installing MQTTSubscribe everything was going without problems (no errors in weewx.log). But after install of MQTTSubscribe, trying to restart Weewx I still got the message in the log: (...)

 

Jul 27 09:52:20 pogoda weewx[3702] ERROR weewx.engine: Import of driver failed: [Errno 111] Connection refused (<class 'ConnectionRefusedError'>)

Jul 27 09:52:20 pogoda weewx[3702] CRITICAL weewx.engine:     ****  ConnectionRefusedError: [Errno 111] Connection refused
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL __main__: Unable to load driver: [Errno 111] Connection refused
Jul 27 09:52:20 pogoda weewx[3702] CRITICAL __main__:     ****  Exiting...

And I reply to myself. I forgot to install Mosquitto (sudo apt install mosquitto). Shame on me...