attaching a raspberry Pi weather station to the weewx system

[Joaquin Lopez]

Hello, at the School we have this weather station https://www.raspberrypi.org/education/weather-station/

It would be possible to configure the weewx system to be able to receive the data from the different sensosrs of our weather estation?

Thanks in davance!

[Neville Davis]

I have built my own weather station using i2C sensors and a Pi3. This generates a csv file I hold in RAM and a seperate driver pulls this data into weewx.

There have been several posts of methods of doing something similar, you maybe able to adapt one of these to your system.

My system is here http://nevspi3w.duckdns.org:8048/weather/nevsskin/index.html

and information about its construction is here https://github.com/weewx/weewx/wiki  This is still a work in process and I need to update this wiki post with my latest work but it has most of the details.

Nev

[Joaquin Lopez]

Thanks Neville, I'l try it!

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zhl4I7oRtt8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bill Morrow]

On Monday, 5 December 2016 05:32:46 UTC-4, Joaquin Lopez wrote:

It would be possible to configure the weewx system to be able to receive the data from the different sensosrs of our weather estation?

Thanks in davance!

I built a system where an Arduino reads sensors using I2C, and sends the data to a Raspberry Pi as a small packet using RF24Mesh radios. Then a process running on the Pi, modeled after one of the RF24Mesh examples, publishes to an MQTT broker. A weewx driver subscribes to the publication and writes to the weewx database. The broker and weewx are running on a different computer, but they could be on the Pi.

In your case, where the Pi already has the data, you could make something similar but simpler. Change the /home/pi/weather-station/interrupt_daemon.py script to publish to MQTT, run mosquitto or some other MQTT broker, and use my MQTT subscribing driver to read from the broker and generate loop packets (snapshopts of your readings) inside weewx.

https://github.com/morrowwm/weewxMQTT.git, look for weewxMQTT/bin/weewx/drivers/wxMesh.py

I'm new to github, so hope I've done that right.

The nice thing about putting an MQTT broker in the middle is that you can change either your collection software or weewx independently. I have created a new version of hardware, tested it with a test topic in MQTT and then put it into production by just changing the topic to be the official one which the weewx driver is subscribed to.

[mwall]

On Tuesday, December 6, 2016 at 8:37:36 PM UTC-5, Bill Morrow wrote:

I built a system where an Arduino reads sensors using I2C, and sends the data to a Raspberry Pi as a small packet using RF24Mesh radios. Then a process running on the Pi, modeled after one of the RF24Mesh examples, publishes to an MQTT broker. A weewx driver subscribes to the publication and writes to the weewx database. The broker and weewx are running on a different computer, but they could be on the Pi.

In your case, where the Pi already has the data, you could make something similar but simpler. Change the /home/pi/weather-station/interrupt_daemon.py script to publish to MQTT, run mosquitto or some other MQTT broker, and use my MQTT subscribing driver to read from the broker and generate loop packets (snapshopts of your readings) inside weewx.

https://github.com/morrowwm/weewxMQTT.git, look for weewxMQTT/bin/weewx/drivers/wxMesh.py

I'm new to github, so hope I've done that right.

bill,

very nicely done!  using mqtt as a standard protocol for a variety of sensor types is an excellent approach.

standard practice is to put a new driver in the user folder (not the weewx/drivers folder) so that you can update weewx without losing your modifications.  unless you're working on core weewx code, this approach tends to be easier than maintaining a separate fork or branch.

you could make it easier to manage (for yourself) and share (for others) if you put the mqtt driver into its own repository (i.e., just the wxMesh.py file).  that way you don't have to duplicate the entire weewx codebase.

even better, package the mqtt driver as a weewx extension so it can be easily applied to any weewx installation.  packaging should be pretty easy - see the fileparse driver as one example, and the customization guide for details:

http://weewx.com/docs/customizing.htm#extensions

if any of that is unclear or difficult to understand, please let us know so we can make it better!

m

[TasmDevil1]

Thank you

[Bill Morrow]

On Wednesday, 7 December 2016 09:47:58 UTC-4, mwall wrote:

standard practice is to put a new driver in the user folder (not the weewx/drivers folder) so that you can update weewx without losing your modifications.  unless you're working on core weewx code, this approach tends to be easier than maintaining a separate fork or branch.

you could make it easier to manage (for yourself) and share (for others) if you put the mqtt driver into its own repository (i.e., just the wxMesh.py file).  that way you don't have to duplicate the entire weewx codebase.

even better, package the mqtt driver as a weewx extension so it can be easily applied to any weewx installation.  packaging should be pretty easy - see the fileparse driver as one example, and the customization guide for details:

http://weewx.com/docs/customizing.htm#extensions

if any of that is unclear or difficult to understand, please let us know so we can make it better!

m

Thank you Matthew. I thought it was a mistake to fork all of weewx on github. I'll work on making it an extension instead.

I need to understand git and github better. I come from the close shop industrial world, where we've used everything from sccs to Mercurial. 

To start with, should I delete that entire weewxMQTT repository, and start over with only my new file(s)?

[mwall]

On Saturday, December 10, 2016 at 5:58:35 AM UTC-5, Bill Morrow wrote:

To start with, should I delete that entire weewxMQTT repository, and start over with only my new file(s)?

if you were to delete all of the files in weewxMQTT except for your files, then anyone who clones weewxMQTT would still get the entire weewx repository up to the point where you had done the clone.  that is probably not what you want.

although there are mechanisms to reach into its history and expunge things, normally git remembers *everything*

so it is probably best to start with a new repository and delete the weewxMQTT repo.

if you want to package the driver as a weewx extension, see the weewx customization guide for instructions

http://weewx.com/docs/customizing.htm#extensions

see the drivers on the weewx wiki for examples:

https://github.com/weewx/weewx/wiki

you could publish only the driver file, but it will be easier for people to use your driver if you publish it as a packaged extension.

either way, you should include a readme file in the repository that explains what your driver does, and a description of how to use it.  the readme can be readme.txt or readme.md.  if a repository contains a readme file in its root directory, github will display the contents of that file when you browse the repository.  the .md indicates a markup file - it is plain text, but you can use simple markup to style the text.  you should use one or the other.

m

[superpat]

Hi Bill,

I have just found this thread, it seems to be exactly what I need to feed my esp8266 sensors into weewx via MQTT.

Please.....  Have you put your work on github again?, if so where?

many thanks

regards

Patrick

[Bill Morrow]

Yes, I have Patrick.

https://github.com/morrowwm/weewxMQTT

I'm actually merging in indoor data collected by an Adafruit Feather Huzzah which uses esp8266 to publish on my LAN.

[superpat]

Hi Bill

Many thanks for the pointer to your github account. 

I have decided to start with a completely new build.  I am using a Raspberry Pi 3,  and a USB hard drive.   I have set the PI up to boot from the USB drive completely, using the new Pi3 MSD boot protocol,  ie. no SD card is used.

So far I have installed weewx and MQTT (mosquitto), but I have to wait now for a RTC board to arrive. In the mean time I will clean up the Esp8266 sensors, mount them on perf board, arrange battery supplies etc, lots of work!

My present station is an Oregon sensor based affair feeding a RFX  receiver  into Meteohub. It has worked well for years but the sensors are getting past their time, and I wish to replace them with more accurate ones.

I have a couple of questions regarding weewx, 

1. if I have NO "known", (in the list of weewx stations), weather station, just wxMesh, will weewx function OK  if I just add  your driver to the weewx.conf, or is there anything else to change?

2. In your MQTT driver, you state that the units have to be US,  if I have Weewx set to metric units, will the driver still work?. I cannot see any data conversions in your driver.
cheers

Patrick

[Bill Morrow]

"Pi3 MSD boot"
    Very interesting, and completely applicable to uses like weewx which are I/O intensive. I run weewx on a "real" computer, but have been thinking about switching to a Pi. My MQTT broker on the same LAN is on a Pi.

1. The only driver I have installed is wxMesh, and weewx works fine. Just configure weewx.conf. Please let me know of any issues. The documentation in the driver probably needs refinement for others to use the driver.

2. The requirement for US units is because of weewx requiring US units. The driver doesn't do any conversion, just passes the data through. That might be a good enhancement to make.

[Thomas Keffer]

2. The requirement for US units is because of weewx requiring US units. The driver doesn't do any conversion, just passes the data through. That might be a good enhancement to make.

​A weeWX driver can send data using US, METRIC, or METRICWX. It doesn't care which. It just has to be consistent.

-tk​

[Bill Morrow]

Ah, I stand corrected, thanks Tom. I believe I got that misconception from the old driver I copied to start work on mine.

Also, I'm running weewx 3.2. Long past time I updated.

[Robert Mantel]

Seems like raspberry pi systems are like snowflakes...lol.  I use a solar panel charged battery powered Particle P1 Redboard from Sparkfun that I have an I2C BME280 (temp/pressure/humidity/altitude) sensor, a rain/wind set from sparkfun as well.  The redboard sends weather data over WiFi to my Phant server every two minutes, I have a cron job on my Pi that does a jsonp query to the phant server to retrieve the latest record and parse it out into a text file with the "* = *" pairs that fileparse wants.  Then my weewx system running on my pi archives every 2 minutes.  Not realtime, but good enough for my purposes.  I also have steelseries gauges running as well.  Been a great learning experience and everyone here has been very helpful.

Many ways to skin the cat here.

[Bill Morrow]

Robert, can you share any details on the solar powered Redboard? My weather station is based on an Arduino Pro Mini, powered by an old Li-ion camera battery. The weather station sleeps most of the time, waking up briefly every 10 seconds or so to sample weather. I get about 2 weeks out of the battery, then swap it.

It's all housed inside an old solar garden lantern and a container which originally held some delicious carrot ginger soup. It communicates with a Raspberry Pi 1 in the house via nrf24L01 radios, using the RF24Mesh libraries. The radio is pretty power hungry, and the main reason for the large sleep duty cycle. On the left of the case, you can see the antenna pointing at the base station in our house.

The temperature sensor shield on the bottom is made from some sports drink bottles. 

My main project these days is developing a circuit which will use the small amount of power generated by the cell to keep the battery charged. I've done some measuring, and it should be enough. I've gone down the rabbit hole of tiny surface mount components, so making slow progress.

Oh, and this is all supposed to be done under a zero cost budget!

[Robert Mantel]

Hey Bill, no problem.  I can't provide pictures right now but will when I get home.  I'm using an 8000 mAh battery I got from ebay (I may upgrade to a 12000 mAh), that is connected to an Adafruit Solar Lipo charger.  I also bought a 6 volt solar panel from ebay or amazon, can't remember right now, but the adafruit charger works best with a 6v panel.  The charger board has a connector for the battery, the solar panel and the load as well as an older mini-usb connector to charge the battery from an ac usb adapter if you just want to top up the battery when it's too low and it's really cloudy.  You can also configure the rate of charge with the addition of a resistor to boost the charge rate to 1000 mA which is what I did.  Most lipo's can only handle a charge rate of 1C (C being the capacity of the battery) so I'm well within the 8000 mAh that the battery could take charge wise.  I'm currently experimenting with how many days I can go without intervention (hence the reason I may go with a larger 12000 mAh battery to get more days of autonomy out of the weather station).  Ideally I would like it to run 24x7 without intervention.

To save power I do with the Particle P1 redboard is that I have configured it to run in manual mode, meaning I control the wifi and cloud connections.  Because I have a local phant server collecting the raw sensor data from the redboard I have no need to connect to the cloud and because I have it in manual mode I briefly turn on the wifi every 2 minutes to dump the sensor data to my phant server in the house.  Wifi is by far the biggest power consumer so the more you can control that the better.  I'm waiting for a 6 dBi wifi antenna to put on the particle P1 because I've noticed that sometimes the wifi doesn't connect where I have it placed inside my shed.  All the sensors are outside the shed of course, but I wanted as much of the electronics sheltered from the elements for longevity sake.  The only active electronics I have outside is the BME280 sensor inside my own home brew radiation shield I made out of flower pot trays.  

While I only post data to weewx every 2 minutes, the station is processing data continuously, so rather than feed for example the current wind direction and speed, I dump the average wind direction and speed over 2 minutes to weewx.  In a previous iteration of my weather station I had it sleeping during each cycle but that proved to be a problem for getting accurate rain readings because if the station was asleep it would miss a bucket tip.  So I went with a larger battery and then controlled the wifi, which saves a ton of power.  The system has a rescue mode so that if the battery falls below 20% soc it will begin sleep cycles of 5 minutes so that the solar panel can pump all the power into the battery until it rises above that threshold and comes back online.  So far so good, hasn't had to do that yet, but it may tonight because we've had quite a few overcast days lately.  

Let me know if you want more specifics and I'll help fill in the blanks.

[Bill Morrow]

Very nice. I considered flower pots for the radiation shield also, but then that wouldn't align with your snowflake theory, would it?

I like both Sparkfun and Adafruit's ecosystem. Phant looks interesting. I'm reading temperature and humidity in the house using an Adafruit Feather Huzzah publishing over wifi to the MQTT broker. weewx subscribes to the MQTT broker to get the data. Powered by a wall wart, but the Feather has the capability of charging a battery. So in theory I could cut the cord there too.

My main goal is to harvest power from the PV cell on the top of the lantern. It maxes out at about 25 mW at 3.2 volts. I should be able to easily get 30 mAh every day, and a lot more than that on a sunny day. My station averages about 25 mAh consumption per day. I've made no attempt to cut power consumption on board, e.g there is a small power LED that is always on.

I'm working on a circuit provided by LInear Technologies: http://cds.linear.com/docs/en/design-note/dn491.pdf, which boosts almost any voltage to Li-ion charging levels. No success yet, mainly because of the difficulty of working with parts with millimeter dimensions.

You're dealing with a much higher power regime. But sounds like you have some good power management strategies going. You might be able to reduce your power needs by going to low power radios instead of wifi?

[Robert Mantel]

I'm interested in the MQTT standard, didn't really know anything about it, but may migrate to that idea and skip the phant server step altogether, but using the fileparse extention with the jsonp query of the phant server just made things simple for me.

As for the lower power radios I did consider using xbees, which I may do at some point, but I'm not sure if they will save me enough power to warrant the extra cost, we'll see how my experiment plays out.  I guess I would have to migrate from my BME280 because it's either SPI or I2C and I haven't read anything about how to deliver that over an xbee radio connection.  What radio's are you using?  

[Robert Kluszczynski]

Hi!

Robert, for low power radio please see https://www.mysensors.org/ - framework based on arduino/raspberry pi and cheap radio NRF24L01 or RFM69.

I'm using it for more than 2 years now and it's very reliable (NFR24L01 + MQTT gateway). On web site you will  find a lot of ready to use examples with different sensors.
Yesterday I give it a try and connected mysensors to weewx via MQTT and Node Red. Working so far!

Regards,

Robert

[Bill Morrow]

I have a BMP180 for barometric pressure and a si7021 for temperature and humidity connected to I2C on the Arduino Pro mini.

The Arduino uses a nrf24L01 radio and the RF24 Mesh library to squawk at the base station. I scraped off the onboard antenna and attached my homemade 5 element Yagi. I think this has improved range quite a bit, but gigahertz RF is voodoo, so who knows? I want to work on this.

If I was doing it again, I might drop down to the sub-GHz range. For example 434 MHz would have much better range. Those transceivers draw maybe 30 mA vs. 15 mA for nrf24l01. But the radio is rarely active, so should not be an issue.

I send a simple packet of data over RF24Mesh of 32 bytes. The nrf24l01 radios are supposed to handle error correction and replies, so I don't do anything for that sort of thing. That's the big advantage of something sophisticated like XBee or nrf24, you don't have to do the low level error handling yourself.

On the base station side, I created a program which is the master on the RF24Mesh, and publishes to the Mosquitto MQTT broker. It receives the 32 byte packet and converts it into a single MQTT publication that looks like this:

   TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND:  0.0,WDIR:  0.0

My weewx wxMesh driver subscribes to the MQTT weather topic, reads the key:value pairs and . The nifty thing about this approach is I can publish on the "weather" topic from multiple stations, and wxMesh will merge them all together into loop packets. The indoor station sends its data synchronously.

[Robert Mantel]

Thanks for the info guys.  When we move to our property we plan to homestead on I can see using these guys to create a whole network of sensors and other things throughout the property to keep track of a whole host of things, not just weather.  I guess with these and a very low power version of an arduino type microcontroller I could really pare down the power budget and get away with a much smaller battery and solar panel.

[Robert Mantel]

I happen to like the sparkfun stuff, turns out they have an nrf24 based radio/controller on a single board.

https://www.sparkfun.com/products/13990

On Monday, 30 January 2017 12:52:06 UTC-5, Bill Morrow wrote:

[Robert Mantel]

Hey Bill, quick question for you, are you doing realtime updates with your system?  Also can you tell me how much power draw of the arduino pro mini and the nrf24 radio?  I'm intrigued by the RF24 mesh networking capabilities and would like to make this my next project, but I want to get an idea of the power management required for these two boards combined to make them autonomously solar powered.

On Monday, 30 January 2017 12:52:06 UTC-5, Bill Morrow wrote:

[Bill Morrow]

Hi Robert,

Depends what you mean by real-time, but I would say yes, I am.

Here's some verbose debug output from the program running on the raspberry pi base station. This program listens to the remote arduino pro mini (via the RF24 mesh) and then publishes on my LAN to the MQTT broker.

Node 5 sent:

   temperature                   :  4.96 C

   pressure                      : 1009.25 mbar

   humidity                      : 79.05 %

   temperature at humidity sensor:  0.8 C

   irradiance                    : 13

   battery voltage               : 933

   solar voltage                 : 874

   mcu temperature               : 32.8

At 2017-02-02 11:19:50 waiting for up to 10 seconds for publication of TIME:0,AMBT: 4.96,BARP:1009.25,RHUM:79.05,HUMT: 0.76,IRRA:13,BATV:933,PHOV:874,SYST:32.80,WIND:  0.0,WDIR:  0.0

on topic weather for client with ClientID: survey

Message with delivery token 3 delivered return code 0

rcv

Node 5 sent:

   temperature                   :  5.21 C

   pressure                      : 1009.27 mbar

   humidity                      : 77.71 %

   temperature at humidity sensor:  1.6 C

   irradiance                    : 10

   battery voltage               : 933

   solar voltage                 : 870

   mcu temperature               : 32.9

At 2017-02-02 11:20:17 waiting for up to 10 seconds for publication of TIME:0,AMBT: 5.21,BARP:1009.27,RHUM:77.71,HUMT: 1.61,IRRA:10,BATV:933,PHOV:870,SYST:32.90,WIND:  0.0,WDIR:  0.0

on topic weather for client with ClientID: survey

Message with delivery token 4 delivered return code 0

rcv

Those packets are 27 seconds apart. I have the Arduino go to sleep for 8 seconds using a watchdog interrupt timer to wake it up, and hit the snooze button twice, so the total sleep is about 24 seconds. I consider that to be real-time enough for ambient temperature, barometric pressure and humidity.

Running the sketch that does this, the remote station (Arduino pro mini, two sensor boards and nrf24l01 radio) consumes about 25 mAh per day, as I describe below. 

Eventually, I want to be collecting wind and rain data. No sleeping allowed for that data, so I'm thinking of a daughter station using just an ATtiny which collects data continually, and then gets interrogated by the pro mini when it wakes up. The daughter board would use a wired connection, either simple serial or I2C.

[Robert Mantel]

The program running on your pi that acts as a base station, is it running on a node.js server?  I just watched a youtube video of a home automation system using node.js, mqtt and 434 MHz radios.  Just reading about those radios and their range, would almost eliminate the need for the mesh part of it, if you're just doing very low data transfer stuff like we do with weather sensors.  Outdoor range is hundreds of feet with very simple antenna arrangements.

[Bill Morrow]

On Thursday, 2 February 2017 14:10:59 UTC-4, Robert Mantel wrote:

The program running on your pi that acts as a base station, is it running on a node.js server?

No, just C language homebrew on raspbian. I'm almost completely ignorant of the whole java ecology.

 

 I just watched a youtube video of a home automation system using node.js, mqtt and 434 MHz radios.  Just reading about those radios and their range, would almost eliminate the need for the mesh part of it, if you're just doing very low data transfer stuff like we do with weather sensors.  Outdoor range is hundreds of feet with very simple antenna arrangements.

I agree, I could simplify the R24 transport. Using the mesh library started as an experiment, it worked and I haven't felt the need to change. Also, my experience is the range is barely 50m, especially on very rainy days where the radio waves get soaked up by the rain. The location I would like to ultimately use for wind measurement is likely too far for direct communication to the base. That said, I haven't actually tried mesh communications, e.g. from a very remote station through Node 5 (my current outside station) to the base.

 

I've trimmed the previous conversations from this reply.

[Robert Mantel]

Sorry for not trimming the previous replies...now one more thing, how are you getting the weather data from the mqtt broker into weewx?  From what I understand the mqtt extension only pushes weather data to an mqtt broker and not vice versa?  Maybe it does, but the wording on Mathew's GIT page suggests it only publishes it doesn't subscribe.  Would be nice if it subscribed too, then you could completely decouple weewx from the hardware a universal translator so to speak.  Maybe it does this, and I'm not understanding it completely.

[Bill Morrow]

No worries on the trimming, Robert, I'm not sure of proper etiquette here. No one's complained yet.

On Thursday, 2 February 2017 14:53:19 UTC-4, Robert Mantel wrote:

Sorry for not trimming the previous replies...now one more thing, how are you getting the weather data from the mqtt broker into weewx?

That's my MQTT driver:  https://github.com/morrowwm/weewxMQTT

 

 From what I understand the mqtt extension only pushes weather data to an mqtt broker and not vice versa?  Maybe it does, but the wording on Mathew's GIT page suggests it only publishes it doesn't subscribe.  Would be nice if it subscribed too, then you could completely decouple weewx from the hardware a universal translator so to speak.  Maybe it does this, and I'm not understanding it completely.

You have it right. That's an interesting idea, combining the publisher and subscriber. I am not sure if it makes sense in the weewx architecture. There are drivers which read incoming data, and reports, which output data. I haven't seen anything bidirectional.

[wysiwyg]

Hello Bill,

I'm just discovering your work (and I'm also discovering mqtt, and ESP8266....)

I feel like a new world of expending weewx is opening to me :-D

I plan to replace little by little my Oregon station by an homemade one.

So yesterday I setup a mosquitto server on an orange pi (an kind of raspberry) : it works great. I have tested with android app "MQTT dashboard" to build some subscriber and publicher.

I have ordered an ESP8266 and a sensor to start with (but it will take a few days or weeks to arrive.

In the mean time I plan to install wxMesh / Weewx on the orange pi and do some tests.

I can simulate a publisher either with the android app or on linux to check the wxmesh setup.

Unfortunately, I am not confortable with python and I was not able to read how the MQTT message shall looks like as input for wxMesh.

If I guessed well, the mqtt topic id should be weather.

I have many questions !! 

In a previous message of this thread, it is mentionned:

TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND:  0.0,WDIR:  0.0

But I dont see the link with weewx labels here,c an you help a bit with that ?

Also I wonder how to deal with multiple publisher: I have in mind to have at least 2 (1 for outside, 1 for inside and barometric sensor).

last question, what is the expected format for time stamp ? unix epoc timestamp? ( like 1487603802 ).

And finally: simple measurements (temp, humidity, pressure...) seems ok, but how is working measurements like rain ? ( what is the data to be put here: daily total? )

Sorry for this question avalanche !

[Bill Morrow]

Answers imbedded below.

On Monday, 20 February 2017 11:18:58 UTC-4, wysiwyg wrote:

Unfortunately, I am not confortable with python and I was not able to read how the MQTT message shall looks like as input for wxMesh.

I created the format of the message, as described below.

If I guessed well, the mqtt topic id should be weather.

Correct.

 

I have many questions !! 

In a previous message of this thread, it is mentionned:

TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND:  0.0,WDIR:  0.0

That is the message format: "keyword<colon>value<comma>.... repeated"

 

But I dont see the link with weewx labels here,c an you help a bit with that ?

In the wxMesh driver section of your weewx.conf file, list the mapping between the keywords in the MQTT message, and weewx's internal names for the data packet. I've added some documentation here: https://github.com/morrowwm/weewxMQTT/wiki/Configuration

The genLoopPackets() method in the driver converts the keyword:value pairs in to weewx's packet.

Also I wonder how to deal with multiple publisher: I have in mind to have at least 2 (1 for outside, 1 for inside and barometric sensor).

last question, what is the expected format for time stamp ? unix epoc timestamp? ( like 1487603802 ).

That is exactly how I am gathering my data. I have an inside Adafruit Feather Huzzah with a Sparkfun HTU21D temperature/humidity sensor adding the inside temperature/humidity to my outside observations. Weewx accepts both publications to the weather topic and merges them together.

I am not setting the time before the data gets to weewx, it is set inside the wxMesh driver. This was easier than trying to keep accurate time upstream in the data path, before data get to weewx. Time setting is done by these two lines in the driver:

                    if( key=="TIME" and data[key] == "0"):

                        data[key] = str(int(time.time()))

And finally: simple measurements (temp, humidity, pressure...) seems ok, but how is working measurements like rain ? ( what is the data to be put here: daily total? )

I am not collecting rain data yet. It is snowing here presently :) 

Depending on how your station collects and provides rain measurement, you might have to adjust the driver. 

My plan is to build a rain collector which will publish the rainfall for each minute or so. weewx has the capability of calculating accumulated rainfall, I believe.

Sorry for this question avalanche !

P.S. Tom. mwall et al: should this sort of conversation be moved to the weewx-development group? 

[wysiwyg]

Many thanks for so quick answer and wiki link. I will read this page carefully !

One more question: 

Would it make sense to use topic instead of all data in one topic ? 

I mean

weather/outside/temperature  = 21
weather/outside/humiduty  = 77
weather/inside/temperature  = 21
....

This is just an open question, I know mqtt since yesterday so, only have a noob point of view :D...

I understand your solution for the timestamp. The only issue I see is that it might slightly delay data...weewx will consider reading time as the measurement time which might be a few seconds earlier.

but it does not seems critical for most measurement.

Regarding raingauge, you may know that there are some raingauge with heating function to melt the snow ;-ppp

More seriously: from what I have seen: there is this Rainwise "1wire" compatible rain gauge. it uses a DS2423 counter IC.

As far as I understand, this IC just count the ticks: I mean it starts from 0 and just count, always up, never goes down.

Then, you can read it every minute, hour, day to calculate the minute, hourly or daily total rain (for the rainwise, 1 tick = 0.25mm)

This calculation should be easy to do in an ESP8266 (or anything arduino, whatever like).

We just need to know how to feed weewx correctly :-)

If I correctly understand (also after looking the "fileparser" driver). The data we have to feed in the driver match the database schemas.

there is only 2 rain data here.

('rainRate',             'REAL'),
('rain',                 'REAL'),

 




Le lundi 20 février 2017 17:13:36 UTC+1, Bill Morrow a écrit :

[Bill Morrow]

I've removed my user/password information from the github document. Thank you for alerting me to the goof, Frederic.

[Bill Morrow]

On Monday, 20 February 2017 13:04:47 UTC-4, wysiwyg wrote:

Many thanks for so quick answer and wiki link. I will read this page carefully !

One more question: 

Would it make sense to use topic instead of all data in one topic ? 

I mean

weather/outside/temperature  = 21
weather/outside/humiduty  = 77
weather/inside/temperature  = 21
....

This is just an open question, I know mqtt since yesterday so, only have a noob point of view :D...

Yes, I believe that would be more in keeping with the design of MQTT. I have not done it yet, though. I am new to MQTT also, but I like its potential.

 

I understand your solution for the timestamp. The only issue I see is that it might slightly delay data...weewx will consider reading time as the measurement time which might be a few seconds earlier.

but it does not seems critical for most measurement.

I agree. I might add a realtime clock to my station someday. Currently I am working on getting it powered by PV cell.

 

Regarding raingauge, you may know that there are some raingauge with heating function to melt the snow ;-ppp

More seriously: from what I have seen: there is this Rainwise "1wire" compatible rain gauge. it uses a DS2423 counter IC.

As far as I understand, this IC just count the ticks: I mean it starts from 0 and just count, always up, never goes down.

Then, you can read it every minute, hour, day to calculate the minute, hourly or daily total rain (for the rainwise, 1 tick = 0.25mm)

This calculation should be easy to do in an ESP8266 (or anything arduino, whatever like).

We just need to know how to feed weewx correctly :-)

If I correctly understand (also after looking the "fileparser" driver). The data we have to feed in the driver match the database schemas.

there is only 2 rain data here.

('rainRate',             'REAL'),
('rain',                 'REAL'),

I know some others have extended the schema, but I think you are on the right approach: do some work in the weather station and provide baseline data to weewx. 

[wysiwyg]

Bill,

I have some troubles to make the driver work.

While look at the code, I have the feeling that the username / password from weewx.conf are not used ?

I tried something, but seems not working better.

here is my test (it's based on other lines of the driver, I try to do something similar, but as I said, I know nothing about python)

 def __init__(self, **stn_dict):
        # where to find the data file
        self.host = stn_dict.get('host', 'localhost')
        self.topic = stn_dict.get('topic', 'weather')
        self.username = stn_dict.get('username', 'XXX')
        self.password = stn_dict.get('password', 'password')
        # how often to poll the weather data file, seconds
        self.poll_interval = float(stn_dict.get('poll_interval', 5.0))
        # mapping from variable names to weewx names
        self.label_map = stn_dict.get('label_map', {})


        loginf("host is %s" % self.host)
        loginf("topic is %s" % self.topic)
        loginf("polling interval is %s" % self.poll_interval)
        loginf('label map is %s' % self.label_map)


        self.payload = "Empty"
        #self.payloadList = [payload]
        self.client = mqtt.Client(client_id="XXX", protocol=mqtt.MQTTv31)


        #self.client.on_connect = self.on_connect
        self.client.on_message = self.on_message


        self.client.username_pw_set(self.username, self.password)
        self.client.connect(self.host, 1883, 60)
        self.client.subscribe(self.topic, qos=1)

[wysiwyg]

for information: syslog, I have only this every 1s (my poll interval)

Feb 20 21:31:59 localhost weewx[9448]: wxMesh: Sleeping for 1
Feb 20 21:32:00 localhost weewx[9448]: wxMesh: Working on payload : Empty
Feb 20 21:32:00 localhost weewx[9448]: wxMesh: Sleeping for 1
Feb 20 21:32:01 localhost weewx[9448]: wxMesh: Working on payload : Empty
Feb 20 21:32:01 localhost weewx[9448]: wxMesh: Sleeping for 1

on /var/log/mosquitto/mosquitto.log:

1487626012: New connection from 192.168.1.180 on port 1883.
1487626012: Socket error on client <unknown>, disconnecting.
1487626042: New connection from 192.168.1.180 on port 1883.
1487626042: Socket error on client <unknown>, disconnecting.

disconnection occurs when I reload weewx configuration (weewx restart)

 

[wysiwyg]

and I forget to say  I use "mqtt dashboard" android app to send the following text with weather topic:

TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND: 0.0,WDIR: 0.0

I publish the text manually by clicking on "publish" button of the app.

The text is send (and well received as I also subscribe to the topic in the same app on an other tablet)

[Bill Morrow]

Frederic, these two lines you have highlighted 

        self.username = stn_dict.get('username', 'XXX')

        self.password = stn_dict.get('password', 'password')

should pick up the username and password from the python "dictionary" that is populated when weewx reads your weewx.conf file on startup.

Do you get any sort of error message, either from weewx or Mosquitto? In weewx.conf, you can set

# Set to 1 for extra debug info, otherwise comment it out or set to zero.
debug = 1

and it will log much more information about what it is doing.

Try adding this line right after the two highlighted

        self.username = stn_dict.get('username', 'XXX')

        self.password = stn_dict.get('password', 'password')

        logdbg("Set username to %s and password to %s" % (self.username, self.password))

In Mosquitto, you have to explicitly set logging to go to syslog, otherwise it is only to stdout. Also, I recall having problems with the wrong version of the MQTT protocol when I started. I might have had to build a newer version of Mosquitto manually, instead of using the one which came with the running version of the OS.

I'm attaching a small python script which should watch everything going through your Mosquitto broker.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zhl4I7oRtt8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bill Morrow]

Oh, make sure you connect as a different client if you use that monitoring script. Otherwise it will take all the traffic from weewx!

[Bill Morrow]

OK, so no issues with MQTT.

I originally was not using the username/password facility in Mosquitto. You could try that, and simply comment out the lines in the driver.

I think this is the setting in mosquitto.conf

    allow_anonymous true

Thank you for trying the driver. When we find out why it is not working, I will document better or improve the code to make it smoother for future users.

--

[Bill Morrow]

That "Socket error" may be an MQTT protocol version mismatch.

Try changing the version which is hardcoded in my driver

import paho.mqtt.client as mqtt
....
self.client = mqtt.Client(client_id=self.client_id, protocol=mqtt.MQTTv31)

to match what your library and MQTT broke Mosquitto support. You can check your MQTT version in this file:

/usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py

or an equivalent depending on what OS you are using. I also believe I have to manually build mosquitto-1.4.10 to get the connection to work.

Another possibility is that you have Mosquitto configured to NOT need user and password.

[mwall]

i found it helpful to configure the broker so that it publishes using websockets.  then you can use a web browser to see what is happening.  this is especially useful when you're trying to decode someone else's topic hierarchy, or to figure out why your own topics are not showing up where you think they should be.

the hivemq folks wrote one of the first ones, which many people have now embedded:

http://www.hivemq.com/blog/full-featured-mqtt-client-browser

http://www.espert.io/mqtt/index.html

but many other browser-based clients are out there:

http://mitsuruog.github.io/what-mqtt/

https://www.cloudmqtt.com/docs-websocket.html

if you're using the mosquitto broker, you'll have to compile it with websocket compatibility.  but once you do that, enabling websockets is just a matter of making the broker listen on the websocket port.

m

[wysiwyg]

Hey you left me a lot of job guys! ;-) (just kidding).

So, let's try to check as much as I can:

1- Mosquitto broker version

fred@orangepipcplus:~$ mosquitto --help
mosquitto version 1.4.8 (build date Fri, 19 Feb 2016 12:03:16 +0100)


mosquitto is an MQTT v3.1 broker.


Usage: mosquitto [-c config_file] [-d] [-h] [-p port]
.....

2- Then, inside  /usr/local/lib/python2.7/dist-packages/paho/mqtt/client.py

"""
This is an MQTT v3.1 client module. MQTT is a lightweight pub/sub messaging
protocol that is easy to implement and suitable for low powered devices.
"""

So the mqtt version seems to match.

3- weewx.conf: debug = 1 (I've done it before starting my tests yesterday).

4- and now I dont what happens but I received a packet :-)

This morning:

I added the line

logdbg("Set username to %s and password to %s" % (self.username, self.password))

then I started weewx 

sudo /etc/init.d/weewx start

I publish from android mqtt dashboard a copy  paste of the wiki example:

TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND: 0.0,WDIR: 0.0

Also before going to bed yesterday, I switched back the poll interval to 10sec.

And here is what appear in my syslog:

Feb 21 06:22:59 localhost weewx[17227]: wxMesh: Got message TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND: 0.0,WDIR: 0.0
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: Working on payload : TIME:0,AMBT: 7.23,BARP:1001.97,RHUM:70.92,HUMT: 4.44,IRRA:12,BATV:974,PHOV:673,SYST:33.20,WIND: 0.0,WDIR: 0.0
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: TIME value: 1487658179
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: AMBT value:  7.23
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: BARP value: 1001.97
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: RHUM value: 70.92
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: HUMT value:  4.44
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: IRRA value: 12
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: BATV value: 974
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: PHOV value: 673
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: SYST value: 33.20
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: WIND value:  0.0
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: key: WDIR value:  0.0
Feb 21 06:22:59 localhost weewx[17227]: wxMesh: Sleeping for 10

\o/ 

I cannot do much more right now, got to go for work, but it's pretty good news !

(maybe I did something else, but don't remember what)

I will try the websockets, I think I have seen a howto somewhere yesterday.

thanks for your help both of you !

[wysiwyg]

Hello !

Regarding websockets: 

Just found the howto I was talking about:

http://www.frugalprototype.com/installation-de-votre-propre-broker-mqtt/

it's in french (very convenient for me, sorrry guys ;-).

But as a summary, it says the websocket function of mosquitto can be enabled just with a small config file.

I will try this (quick & easy to test).

Also, just to summarise my setup:

I just received last week a brand new Orange Pi PC plus (kind of low cost raspberry which as a nice advantage imho: it has integrated emmc 8GB memory).

It is setup with Armbiam Ubuntu Xenial (server), v5.25.  (I am more familiar to ubuntu so it's easier for me too)

I just setup mosquitto from depot ( apt-get install mosquitto ).

I setup weewx using the quick start guide with depot ( http://www.weewx.com/docs/apt-get.htm )

So it's all fresh install.

I only modified mosquitto config to use "user/password" and play a bit with weewx and wxMesh as you know.

finally, I have a general question regarding how the driver works:

let me know if my understanding is correct or wrong

the "on_message" stuff is kind of asynchronous

then when a message is available on the broker, this function will be trigged  and will load the message content in payload.

then the genLoopPackets is synchronous (I mean executed every n seconds, n=poll interval defined in weewx.conf)

So every n seconds, it checks if payload as some content and if yes, it will process the data and deliver the "_packet" to next weewx process

If this is correct, does it mean that if several mqtt messages are delivered inside the n seconds interval, maybe only the last one will be processed ?

The background of the question is that I'm thinking if it may be possible to use a sub topic structure for mqtt

The simplest idea I have in mind is to have:  weather/<weewx_schemas_label>:

weather/outTemp
weather/outHumidity
....

Then each of these topics may be sent at any time:  Temp, humidity, barometer may be send only every few minutes.  Wind may be sent  every 1 or few seconds,   rain may be send only when it's raining...

weewx driver may subscribe to /weather/# to get them all and process whenever they arrive ?

-Ok- I speak but don't have a clue on how to do that :-) not sure it is realistic.

The advantage could that it stick better to what I have seen from mqtt usage, then same published topics could be used by other subscribers (home-assistant, automatic watering for the garden or whatever)

Best regards,

Le mardi 21 février 2017 00:05:11 UTC+1, mwall a écrit :

[Bill Morrow]

Good news!

I have my poll_interval set to 1 second. For the wxMesh driver, that means it will check for newMQTT publications once per second. Depending on the rate of "weather" topic publications, this might be an issue. I remember now that I did have to switch to this relatively high poll interval. So perhaps this is why you were not seeing packets.

On Tuesday, 21 February 2017 02:28:45 UTC-4, wysiwyg wrote:

Hey you left me a lot of job guys! ;-) (just kidding).

So, let's try to check as much as I can:

4- and now I dont what happens but I received a packet :-)

[Bill Morrow]

This conversation might be better to continue in the weewx-development group. I am copying it over there.

Some imbedded comments below.

On Tuesday, 21 February 2017 06:06:40 UTC-4, wysiwyg wrote:

Hello !

Regarding websockets: 

Just found the howto I was talking about:

http://www.frugalprototype.com/installation-de-votre-propre-broker-mqtt/

it's in french (very convenient for me, sorrry guys ;-).

But as a summary, it says the websocket function of mosquitto can be enabled just with a small config file.

I will try this (quick & easy to test).

Also, just to summarise my setup:

I just received last week a brand new Orange Pi PC plus (kind of low cost raspberry which as a nice advantage imho: it has integrated emmc 8GB memory).

It is setup with Armbiam Ubuntu Xenial (server), v5.25.  (I am more familiar to ubuntu so it's easier for me too)

I just setup mosquitto from depot ( apt-get install mosquitto ).

I setup weewx using the quick start guide with depot ( http://www.weewx.com/docs/apt-get.htm )

So it's all fresh install.

I only modified mosquitto config to use "user/password" and play a bit with weewx and wxMesh as you know.

finally, I have a general question regarding how the driver works:

let me know if my understanding is correct or wrong

the "on_message" stuff is kind of asynchronous

then when a message is available on the broker, this function will be trigged  and will load the message content in payload.

Yes, when a message is received, the on_message callback is triggered.

 

then the genLoopPackets is synchronous (I mean executed every n seconds, n=poll interval defined in weewx.conf)

So every n seconds, it checks if payload as some content and if yes, it will process the data and deliver the "_packet" to next weewx process

Correct. I originally planned to build the loop packet in the on_message call back, but on_message actually runs in a separate thread I think, and I could not figure out how to easily get the data back to the weewx thread.  generating the loop packet only when a subscription arrives would be more efficient. I do not understand weewx or python quite well enough to design this yet.

If this is correct, does it mean that if several mqtt messages are delivered inside the n seconds interval, maybe only the last one will be processed ?

I believe any number of "loop" packets can arrive within n seconds, especially for n>1. Read the Data Architecture section here: http://www.weewx.com/docs/customizing.htm

The background of the question is that I'm thinking if it may be possible to use a sub topic structure for mqtt

The simplest idea I have in mind is to have:  weather/<weewx_schemas_label>:

weather/outTemp
weather/outHumidity
....

Then each of these topics may be sent at any time:  Temp, humidity, barometer may be send only every few minutes.  Wind may be sent  every 1 or few seconds,   rain may be send only when it's raining...

weewx driver may subscribe to /weather/# to get them all and process whenever they arrive ?

-Ok- I speak but don't have a clue on how to do that :-) not sure it is realistic.

That was exactly my original idea on how to build the driver. However, if a datum does not get set within the time interval of a RECORD (see the setting in the [StdArchive] section of weewx.conf), then it will be set to None, I think. I don't think this is what we want. Depending on the type of observation (wind, rain) we may want to keep the last value, or enter a zero.

wxMesh is definitely a work in progress. It may very well grow away from the original names

[Bill Morrow]

Please continue this thread on the weewx-development group: https://groups.google.com/d/msg/weewx-development/tNDnGNe9Z_M/ubAAPKGBCAAJ