Import MQTT value into weewx database, display via seasons skin

565 views
Skip to first unread message

George Alexander

unread,
Apr 6, 2019, 4:07:51 AM4/6/19
to weewx-user
After searching for quite a few hours, I have come up with nothing that helps what I am trying to achieve.

I have a RPi 3 that is running weewx, with a FineOffset USB station connected to it.  This works fine, happily displaying the data via a Lightpd webserver.

I also have an ESP8266 in a storage room that uses MQTT to output both the temp and humidity of the room whenever there is a change - this keeps the traffic down to only when needed - and a MQTT broker (mosquitto) runs on the RPi that runs weewx.

I would like to import the MQTT data to weewx, save it in the database and also display it on the webserver along with the FineOffset data, but how to do this seems to eluding me.

I can see how to make everything MQTT - and weewx currently exports to MQTT just out of curiosity - but I would rather not run a copy of weewx to interface with the weather station and then export to MQTT, followed by another RPi running weewx with the input being MQTT.

Surely there is a way to "read" the MQTT broker and input selected data as additional sensors along with the FineOffset data?

Thanks very much.

Thomas Keffer

unread,
Apr 6, 2019, 8:58:12 AM4/6/19
to weewx-user
You could write a WeeWX service that talks to the MQTT broker, then augments the data stream from your Fine Offset station (this would be configuration #4 in the Intake Strategies wiki entry).

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ralph Underwood

unread,
Apr 6, 2019, 9:52:06 AM4/6/19
to weewx-user
I have a small python program that subscribes to mqtt and writes a text file. I then use the filepile as a service to get the data into WeeWx. The "main" data comes from a Ultimeter station via the driver.

I came up with this approach while I was modifing/testing using the WeeWxMQTT driver a service.  It works and I put the conversion to a service on the back burner.

#!/usr/bin/python
import syslog
import time
import ftplib
import time
import logging
import paho.mqtt.client as mqtt
 
MQTT_SERVER = "10.0.1.47"
MQTT_TOPIC = "weather"

 # The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))
 
    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe(MQTT_TOPIC)
 
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    
    #print(msg.topic+" "+str(msg.payload))
    #print(str(msg.payload))
   
    m1 = (msg.payload).replace(':',' = ')
    m2 = m1.split(",")
    m3 = m2[1]
    m4 = "soilTemp4 = 32"
    file = open('/var/tmp/filepile.txt' , 'w')
    file.write(str(m3)+'\n')
    file.write(str(m4)+'\n')
    #print(m3)  
    #print (m4)       
    #print("file stent")
    file.close()
 

   
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
 
client.connect(MQTT_SERVER, 1883, 60)
 
client.loop_forever()


There are some print lines which are commented out for debugging. I also use this to set soilTemp4 to 32 degrees - that's so I can easily add a freezing level line on charts.


from weewx.conf:

# Options for extension 'filepile'
[FilePile]
    # Where to find the incoming new data:
    filename = /var/tmp/filepile.txt
    # What unit system they will be in.
    # Choices are 'US', 'METRIC', or 'METRICWX'
    unit_system = US
    # Map from incoming names, to WeeWX names.
    [[label_map]]
        # Example: incoming observation 'filelabel1' will be mapped to 'extraTemp4'
        INTE = extraTemp2
        







vince

unread,
Apr 6, 2019, 10:04:28 AM4/6/19
to weewx-user
On Saturday, April 6, 2019 at 6:52:06 AM UTC-7, Ralph Underwood wrote:
I have a small python program that subscribes to mqtt and writes a text file. I then use the filepile as a service to get the data into WeeWx. The "main" data comes from a Ultimeter station via the driver.



I have a similar setup that reads pi sensors and writes a JSON file that nginx serves up on a secondary pi.  I wrote a weewx service that grabs that file and populates a custom weewx db, although the code is somewhat ugly under the hood to say the least.

A better example is the weatherflow UDP driver.  That does a web get of a JSON formatted file, maps the elements to a database schema of your choosing, and populates the database.   Very very stable.  It would be a great example for how to grab a web endpoint and populate a custom db.


p q

unread,
Apr 6, 2019, 11:39:18 AM4/6/19
to weewx...@googlegroups.com
You might be interested in my project here:  https://hackaday.io/project/101680-solar-powered-wifi-temperature-sensor-for-weewx

I am sending the data every 2 minutes. I do it directly to the Raspb via Wifi.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Peter Quinn
(415)794-2264
Message has been deleted

George Alexander

unread,
Apr 6, 2019, 8:56:49 PM4/6/19
to weewx-user
Hi Ralph,

I think that your way of doing things is the closest to what I had envisaged.

Thus far I have managed to create a .py script that I can run manually by copying and pasting from yours, modifying the IP address to 'localhost' as they are both on the same RPi.

Watching the log of the MQTT broker, I can see that the script is indeed connecting and if I stop the broker and start it again I can see the script reconnecting as it is supposed to:

weewx@raspberrypi:~$ ./ReadMqttScript.py
Connected with result code 0

What I am not seeing is an output to /var/tmp/filepile.txt - is there anything else that I might need to do before that will happen? I am able to see from another PC that the broker is indeed publishing "weather/temperature" and "weather/humidity" so I am a bit lost as to what could be going wrong.

*sigh* I haven't even got to modifying the weewx code yet..

Ralph Underwood

unread,
Apr 7, 2019, 8:40:33 AM4/7/19
to weewx-user
Probably a permissions issue. I run my python script as sudo.  Hope that helps.


Ralph



George Alexander

unread,
Apr 8, 2019, 6:28:51 AM4/8/19
to weewx-user
Hi Ralph,

Unfortunately no, I had been running the script as root.

Thanks anyway, I'll try find another way.

Ralph Underwood

unread,
Apr 8, 2019, 10:39:31 AM4/8/19
to weewx-user
Hi George,

Try just writing the file with python - 

m1 = "test"
file = open('/var/tmp/file-test.txt' , 'w')
file.write(str(m1)+'\n')
file.close()

My first time with my python script I had a typo that I could not see for a couple of hours - I did learn a lot as I scoured the reference books for why it did not work.

Good luck.

Ralph

bgra...@umw.edu

unread,
Sep 15, 2021, 4:30:35 PMSep 15
to weewx-user
Vince and everyone,
I have a RAK7244 LoRa (chirpstack) setup on my LAN that is getting data from two LHT65 temp/hum sensors. All seems to be working correctly.
My goal is to import the data into weewx which I have running on an rpi. Not being a programmer, I seem to be WAY OVER my head with how to set up mqtt and deal with json files. I have tried the python program kindly offered by Ralph Underwood and it seems to be subscribing to the json data:

MQTT_SERVER = "192.168.1.10"
MQTT_TOPIC = "application/#"      *** [using '#' to get everything]

Connected with result code 0
application/7/device/a8404117b18312e9/rx {"applicationID":"7","applicationName":"LHT65","deviceName":"LHT65x","devEUI":"a8404117b18312e9","txInfo":{"frequency":905100000,"dr":0},"adr":true,"fCnt":1376,"fPort":2,"data":"y+wJUAHgAQlHf/8=","object":{"BatV":3.052,"Ext_sensor":"Temperature Sensor LHT65","Hum_SHT":"48.0","TempF_DS":"74.75","TempF_SHT":"74.91"}}
{"applicationID":"7","applicationName":"LHT65","deviceName":"LHT65x","devEUI":"a8404117b18312e9","txInfo":{"frequency":905100000,"dr":0},"adr":true,"fCnt":1376,"fPort":2,"data":"y+wJUAHgAQlHf/8=","object":{"BatV":3.052,"Ext_sensor":"Temperature Sensor LHT65","Hum_SHT":"48.0","TempF_DS":"74.75","TempF_SHT":"74.91"}}

application/7/device/a84041468182b4da/rx {"applicationID":"7","applicationName":"LHT65","deviceName":"LHT65-1","devEUI":"a84041468182b4da","txInfo":{"frequency":904300000,"dr":0},"adr":true,"fCnt":1367,"fPort":2,"data":"y9ELFQGRAQr2f/8=","object":{"BatV":3.025,"Ext_sensor":"Temperature Sensor LHT65","Hum_SHT":"40.1","TempF_DS":"82.51","TempF_SHT":"83.07"}}
{"applicationID":"7","applicationName":"LHT65","deviceName":"LHT65-1","devEUI":"a84041468182b4da","txInfo":{"frequency":904300000,"dr":0},"adr":true,"fCnt":1367,"fPort":2,"data":"y9ELFQGRAQr2f/8=","object":{"BatV":3.025,"Ext_sensor":"Temperature Sensor LHT65","Hum_SHT":"40.1","TempF_DS":"82.51","TempF_SHT":"83.07"}}

At this point I'm not sure what to do to get the data into weewx. If anyone can offer a suggestion I would be most grateful.
Thanks.
Bob
Message has been deleted

storm...@gmail.com

unread,
Sep 15, 2021, 6:30:15 PMSep 15
to weewx-user
Reply all
Reply to author
Forward
0 new messages