howto receive MQTT data from a running broker and put the data into database with weewx..?

[Christian Peters]

Hi,

I ran a weewx 3.9. station with Python 2 and a seperate weewx instance with the wxMesh.py driver.

This works very nice (as far as the broker isn't down or the connection breaks - then a restart of the weewx instance which holds the wxMesh driver is needed...!).

As I now switch to Python 3 and weewx 4.x I figured out that the wxMesh.py driver isn't Python 3 ready? :-(

How can I catch MQTT data into the weewx database from an external broker ilocated n my home lan?

Is this the right extension to use:

https://github.com/bellrichm/WeeWX-MQTTSubscribe

....?

What I don't understand: if I will use it as a sevice, in which database will it put my values?

Is there a mapping needed!? Can I use a seperate database?

Is there a configuration example available or does anybody set up such a station and could provide the config or give me some advice

howto get the data form the MQTT broker into weewx and the database to show the values and graphs!?

That would be great!

Thanks in advance!

Regards,

Christian

[Rich Bell]

Christian,

MQTTSubscribe should be able to meet your needs. I know the documentation is a bit sparse, but I can help get you up and running.

The first question you need to answer is whether to run as a service or driver? From what I am understanding. If you ran as a driver it would essentially be a replacement for wxmesh. Meaning it would the database configured for that WeeWX instance. If you ran MQTTSubscribe as a service, you could augment the data in your other WeeWX instance.

The next question is the format of the incoming data. Is it json, keyword (name=value), or individual (single observation per topic)? Lastly, what name mapping do you need to?

-rich

[Ken Walker]

I modified two lines in wxMesh.py and it works fine with Python 3.

changed

#!/usr/bin/python

to

#!/user/bin/python3

and

changed

except ValueError, e:

to

except ValueError as e:

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3f984aba-be57-419e-9334-fde2e668a7fdo%40googlegroups.com.

[Christian Peters]

Rich,

that's great. I'm not sure whether I should stay on the driver version or switch to the service version....which could make it maybe more easy.

But if the MQTTSubscribe as service lost connection and wouldn't reconnect I have to restart the main weewx prog to get it up again!? So maybe a second running instance

with MQTTSubscribe is the better choice...?

Difficult...I would perfer one instance as I struggle to get a second weewx instance up with systemd at the moment....

So with mosquitto_sub -h 192.168.1.143 -t weather

I got this ( in JSON format I think) :

TIME:0,radiation:0.17,cpm:31,cps:0,temp_2m:25.97,temp_5cm:27.24,temp_dht22:34.30,hum_dht22:45.30,rain:0.00

I mapped in my Python 2 env (weewx2.9, second instance with sql databse) and wxMesh:

     [[label_map]]
         TIME = dateTime
         temp_2m = extraTemp1
         temp_5cm = extraTemp2
         temp_dht22 = inTemp
         hum_dht22 = inHumidity
         rain = rain
         geiger = radiation

So do you know the syntax for the mapping for MQTTSubscribe?

As I send the data to the MQTT broker by myself it would be maybe best to use a second instance and tweak the mapping to entrys which are already in the schema.

So I could stay in the mapping posted above?

Do you set up a second instance of weewx with systemd already...?

Any help or syntax examples would be welcome, especially for the format of the label mapping of
MQTTSubscribe. I tried weew-multi but dont get the second instance up... Not sure its systemd ready...

Thanks in advance,

Christian

[Rich Bell]

Well, I haven't heard of having to restart WeeWX when the connection is lost, and if that happens I would certainly want to look into fixing that. I'll do some experiments to see if I can recreate. 

Sounds like picking the service might be the best long term choice, with the caveat about the reconnect logic needing to be verified. But if you want to get your feet wet, I would suggest that we get it up and running as a driver - perhaps even running from the command line for a bit. The actual configuration work is really the same. So you could migrate to the service as you get more comfortable or after we prove that lost connections are not a problem. 

The following assumes that you install 1.5.4-rc01. This changed the way to configure MQTTSubscribe. Once installed, there will be three key sections the [MQTTSubscribeDriver]/[MQTTSubscribeService] section. This is the MQTT connection information, and I believe this is pretty straight forward. But after looking at it, if you have questions let me know.

Next up is the [[topics]] section. Based on the information you posted, I think this is what you would want

[[topics]]

 [[[weather]]]       # The topic you are subscribing to

   unit_system = METRICWX # Not sure weather you want METRICWX or METRIC. See http://weewx.com/docs/customizing.htm#units

   [[[[temp_2m]]]] # Name in the MQTT message

       name = extraTemp1 # WeeWX name

   [[[[temp_5cm]]]]

        name = extraTemp2

   [[[[temp_dht22]]]]

        name = inTemp

   [[[[hum_dht22]]]]

      name = inHumidity

   [[[[rain]]]]

        name = rain

   [[[[geiger]]]]

      name = radiation

Not sure about your TIME field and the value of 0. I left it out and since it is missing MQTTSubscribe will use the date/time from the server it is running on. There are other options for converting the incoming data, but yours looks pretty straight forward.

Last is the [[message_callback]] section. I would call your data 'keyword'. If it was json, I would expect quotes around the fieldnames. So, something like this

[[message_callback]]

 type = keyword

 keyword_delimiter = "," # The delimiter between the name/value pairs

 keyword_separator = ":" # The separator between the name and the value

There are some additional options when running as a service to attempt to only process data that belongs to the loop packet or archive record that is being augmented. Over time I have found the defaults to be a bit too restrictive, so we might need to configure those a bit.

I haven't tried to run a second instance of WeeWX, so I can't help there. But, if you need any more help with MQTTSubscribe, just let me know.

-rich

[Rich Bell]

Christian,

What are the symptoms that cause you to have to restart WeeWX when MQTT connection is lost with wxmesh? The reason I ask, I tried just disconnection the MQTTSubscribe client from the network and then reconnecting. It seemed to behave correctly. I am hesitant to stop the broker, but I will work on setting up a second broker to do some more testing.  

Thanks for any information/help. - Rich

[Christian Peters]

Hi Rich,

thank you very much.

It works like a charm! :-)

I set up a temporary mapping to values already available in the databae. For the temp and humid values it works out of the box, for CPM, CPS and radiation I have to setup unit groups later.

This is my working setup - just to have that written down for others:

my data form the MQTT broker:

radiation:0.17,cpm:31,cps:0,temp_2m:25.97,temp_5cm:27.24,temp_dht22:34.30,hum_dht22:45.30,rain:0.00

weewx.conf:

[MQTTSubscribeService]
    host = xxx.xxx.xxx.xxx # IP of the broker
    payload_type = keyword
    console = True

    [[topics]]
     [[[weather]]]       # The topic you are subscribing to
       unit_system = METRICWX # Not sure weather you want METRICWX or METRIC. See http://weewx.com/docs/customizing.htm#units
       [[[[temp_2m]]]] # Name in the MQTT message
           name = extraTemp1 # WeeWX name
       [[[[temp_5cm]]]]
           name = extraTemp2
       [[[[temp_dht22]]]]

           name = extraTemp3
       [[[[hum_dht22]]]]
           name = extraHumid3
       [[[[rain]]]]
           name = hail
       [[[[radiation]]]]
           name = signal1
       [[[[cpm]]]]
           name = signal2
       [[[[cps]]]]
           name = signal3

[[message_callback]]
 type = keyword
 keyword_delimiter = "," # The delimiter between the name/value pairs
 keyword_separator = ":" # The separator between the name and the value

    [[Services]]
        data_services = user.MQTTSubscribe.MQTTSubscribeService

That's quite nice and more elegant as to set up a second weewx instance with a MQTT driver!

Thank you very much.

My MQTT broker runs on a BananaPi which is located in the garden near the sensors (Temp2m, Temp5cm, raingauge, temp and hum sensor inside the controlbox and a MightyOhm Geiger Counter Kit).

I can't figure out why...but somtimes I lost the WLAN connection (after 7-30 days). Then I restart the BananPi, start the mosquitto deamon...but the weewx instance running the wxMesh driver didn't reconnect.

I have to start and stop this instance, then it works again. But I saw a post in weewx-development that someone pointed out the connection was setup once at initialisation...maybe there is no check of the connection and a reconnect condition?

If the setup goes online (at the moment I run the simulator whil mograting all sensors and tweak the new webpage) and the VanatgeVP2 is connected, I will check the behavior of your drriver in my setup.

Thanks again for the excellent support. If I run into some problems I will contact you again. :-)

Regards,

Christian

P.S. Ken, thanks for you fix...it works..! :-) But the MQTTSubscribeService works great and I will prefer to run only one weewx instance to deal with all sensors.

[Rich Bell]

I should say that in the first test, I saw the disconnect and reconnect and processing pick back up.

I did a couple more tests. First, just stop publishing. No disconnect, MQTTSubscribe waited until the next message. No surprise there. Next I shut the broker down. Again, I saw MQTTSubscribe disconnect and reconnect and continue processing.

So, I think the reconnect logic is working, but there still might be some other real world case that could cause a problem.

[Rich Bell]

Great to hear. Just give a shout if you run into any other difficulties.

rich

[Bill Morrow]

Sorry for the late reply. I don't have much time to work on my original wxMesh driver. 

From what I have seen of MQTTSubscribe, it's a much more complete implementation of an MQTT subscriber, and offers the features I was planning on adding some day. When I get a chance, I plan on switching to using MQTTSubscribe.

[morr...@gmail.com]

User  norecord has made changes to wxMesh so that it should work with python 3 and weewx 4.x. Thank you!

I have merged in their changes and pushed it to the master branch.

I've also changed my local implementation so that the data is in JSON format. I'll likely push that up too, but need to make the switch to python3 first.