add external sensors on remote computer to a main weewx installation?

[Christian Peters]

Hi group members,

I just assembled two additional temp sensors and a rain sensor. All this sensors are connected to an Arduino which sill emmit the data on the serial port.
As the sensors and the Arduino is located in the garden I can’t connect it to my main computer in the house.

There (in the haouse) is a weewx instance running with a VP2 connected already.

The Arduino is connected to a BananaPi (even in a box in the garden) which is connected via WLAN to my house.

I think I will collect the data from the Arduino by a program on the BananaPi (still to write) which will write the data to an InfluxDB to combine the data from the VP2 (which data is emitted by the weewx instance in the house and the weewx-influx plugin) in Grafana.

But in the main house I even generate a standard skin with weewx (VP2 data) and it would be nice to include the new additional sensors in the html page.

So the question is how to achieve this?

Should I install a second weewx on the BananaPi in the garden? But howto get the data into weewx (fileparse or a simple serial driver?).  I think a second database on the BananaPi would be better then reading the data per WLAN direct into the weewx mysql database in the house (main computer) because in case the WLAN was interrupted all data will be lost during this period of this sensors?!

Is it possible to query the remote database of the weewx instance in the garden from the main weewx instance and combine the two remote temp sensors with the outdoor temp sensor data of the VP in one plot each run the html page is build?

Thanks for suggestions! :-)

Regards,

Christian

[Thomas Keffer]

Hello, Christian

To enumerate, here are your choices:

1. Run weewx locally on your garden BananaPi. Replicate the resultant database on your "house" mysql database. Use the replicated database to run reports with both the VP2 and garden data. 

2. Run weewx locally on your garden BananaPi. Interrogate its database from the house instance of weewx.

3. Run weewx only in the house, with a second thread that interrogates your bespoke program on the Banana Pi (or even directly to the arduino). Its data is then joined with the VP2 data into a single MySQL database in the house.

4. Same as #3, except put the data in a separate database in the house.

5. Run two instances of weewx in the house. The second instance would be dedicated to collecting data from the garden. One of the two instances would then be responsible for generating reports.

Any of these can work, although I think you would find #2 to be frustrating: database protocols do not like to be run over unreliable networks. Most people in a similar situation have chosen solution #5. It's the simplest to administer.

If you choose #3, #4, or #5, I would recommend using an MQTT broker across the WLAN. They are designed to work across unreliable networks, and can even offer store messages when the WLAN is offline, replaying them later. There are a couple of weewx MQTT drivers around.

Sounds like an interesting project. Keep us informed!

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vince]

I do something similar, though less interesting than what you are doing.

I have an external building with a raspi model-B and a couple DS18B20 temperature sensors.   A cron job captures the sensor data and writes a json-formatted file to ramdisk, which is linked into a nginx web document root.   My weewx system inside the house runs a custom extension to grab that file and save the data in a separate custom weewx database.  My local skin grabs data from both weewx db and the custom db containing just the data from the custom sensors.

So I guess it's Tom's option (4) from his response.   My thoughts were:

• it's easy on the outside pi to capture the data and make it available on LAN via nginx
• it wasn't too hard to write the custom extension to add to weewx inside
• I didn't have enough compute to run multiple copies of weewx inside anyway, I run on a 128MB ram ARM processor ala a pogoplug notionally

All the MQTT magic things came out years after I had my custom solution in place, so they weren't available way back then....but they sure seem cool to say the least.  I'd probably go that way if I was going to do it over again.

But definitely store your data in a custom db in weewx, that is really helpful especially if your outside sensors throw some bad data occasionally that you need to clean up.   That way you won't need to rebuild or touch anything in your primary weewx db.

[Christian Peters]

Hi Tom,

thank you for that detailed list of possibilities!
I think #4 or #5 would be the best way as it uses a separate database. I’m not familiar with MQTT and will dive into it.
I hope my main computer is able to run two instances of weewx even with an archive interval of one minute.
Maintaining two weewx instances seems much easier to handle…even in the test phase!

Yes….I think there could be some interruption of the WLAN and I don’t want to loose data during this time. If a MQTT broker running on the BananaPi outside which stores data local and can be accessed even after an interruption it would be a really good (and my preferred ) solution. Great that MQTT drivers are already available for weewx.

It’s really a nice project and I will keep you informed. Final assembling of the distribution box placed later on the pole (VP already installed in 2m height) is nearing completion. Today I add a MightyOhm geiger counter and I think I have to modify the database of weewx to add CPS,CPM and uSv/hr to the database. Another challenge…. but already documented as I read in the docs.

@Vince: Your solution sounds good too (nice idea to use a ramdisk for that file!). I first thought of writing the data into a file and access it maybe via NFS or SMB. So you offer it by Ngnix. Another good solution. If the network connection I have would be wired Ethernet I would go your way….but I really think buffering the data outside at the BananaPi in my case would be more reliable; Good point: there really could be no data or bad data from that sensors outside. And cleaning the main database I really will avoid!

Regards,

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Christian Peters]

Hi members,

project is progressing, today we added a monoflop on the Arduino to debounce the rain gauge signal. It's working.
As described the Arduino sends it's data to a Banana-Pi via MQTT push.
On my remote computer I run a second instance of weewx with wxMesh driver which pulls the data from the MQTT server.
All well.
The main instance on my remote computer is conected to a VP2 with a 0.2mm tipping bucket rain gauge.
 
The rain parameter I got from the Arduino is mapped in wxMesh to 'hail' to implement a second rain bucket gauge.  The hardware which is connected is a rain bucket with a 0,1mm tipping bucket rain gauge!!

Now I have the problem that the 'hail' (rain of the second rain bucket) is doubled - even in the skin of the second instance and also if I do a query from the first instance like

<td class="stats_data">$day($data_binding='mqtt_binding').hail.sum<</td>

How can I set the tipping bucket rain gauge size inside the second instance to 0.1mm if I import the data via wxMesh but stay on the 0.2mm bucket size of the vantage!?

Thanks in advance!

Regards,

Christian

[gjr80]

Hi,

Perhaps I am missing something obvious but why not fix the issue at source; if the arduino is sending out the source rainfall data why not double it or calculate it there. If you follow the weeWX driver model it is the driver, ie the code that speaks direct to the station/source, that calculates how much rain occurs for each 'tip'. At least that way your data is correct as it propagates it's way through your system. If you want/need to set bucket size in a weewx.conf then you will need to modify the respective driver (which you may or may not wish to do). I don't imagine it would be easy to propagate bucket size from weewx.conf to arduino.

Gary

[Christian Peters]

Gary,

im not sure if I have to send the number of bucket tips or the rain data? Maybe that's the fault/problem? As the driver of weewx in this case is the weexMes driver, it will only map the values to weewx?
Have to look into that deeper. Maybe I have to send the rain data im mm?

Regards,

Christian

[Thomas Keffer]

The driver should emit inches or mm of rain that fell since the last packet, not bucket tips. The unit should match whatever unit system is in use in the packet. 

This is explained in the Customizing Guide, section genLoopPackets.

-tk

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Christian Peters]

Tom,

thank you for that hint. So emitting in mm was the right change.
I just wonder why my temp values are ok but my rain data isn't!?
Just found that on the wxMesh driver:

# map the data into a weewx loop packet

_packet = {'usUnits': weewx.METRIC}

So I think I have to switch to METRICWX.

Regards,

Christian

[Thomas Keffer]

That could be it. weewx.METRIC has rainfall in cm, weewx.METRICWX in mm. See the Units appendix in the Customizing Guide.

-tk

[Christian Peters]

Tom,

that seems not to solve my problem. I send 0.1 (mm) but didn't get that into weewx.
As I already have a rain bucket with the VPII I used 'hail' and 'hail rate". Could these be a problem or can't I use it for a second rain bucket (same computation as rain?).

Regard,

Chrstian

Am Sonntag, 26. November 2017 15:09:33 UTC+1 schrieb Tom Keffer:

That could be it. weewx.METRIC has rainfall in cm, weewx.METRICWX in mm. See the Units appendix in the Customizing Guide.

-tk

On Sun, Nov 26, 2017 at 8:02 AM, 'Christian Peters' via weewx-user <weewx...@googlegroups.com> wrote:

Tom,

thank you for that hint. So emitting in mm was the right change.
I just wonder why my temp values are ok but my rain data isn't!?
Just found that on the wxMesh driver:

# map the data into a weewx loop packet

_packet = {'usUnits': weewx.METRIC}

So I think I have to switch to METRICWX.

Regards,

Christian

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Thomas Keffer]

You're being a little vague on the details. What results are you getting? And, what do you mean by "send 0.1 (mm)?" Do you mean in the loop packets? If so, your genLoopPackets() function should look something like this (NOT TESTED):

def genLoopPackets():

  data = {'dateTime': time.time(), 'usUnits': weewx.METRIC}

  data['outTemp'] = (something)

  ...  

  data['hail'] = 0.1    # NB: This would be 0.1 cm, or 1mm

  yield data

There are a two other assumptions as well:

1. You have not implemented genArchiveRecords() and are, therefore, depending on software record generation to turn your loop packets into archive records.
   
2. The observation type 'hail' is in your schema. This is the normal situation with the default "wview" schema, so I'm assuming you have not changed it.
   

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Christian Peters]

Tom,

I use the wxMesh driver but I think your demo code is imlemented in that way you posted:

#!/usr/bin/python

#

# weewx driver that reads data from MQTT subscription

#

# This program is free software: you can redistribute it and/or modify it under

# the terms of the GNU General Public License as published by the Free Software

# Foundation, either version 3 of the License, or any later version.

#

# This program is distributed in the hope that it will be useful, but WITHOUT

# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS

# FOR A PARTICULAR PURPOSE.

#

# See http://www.gnu.org/licenses/

#

# The units must be weewx.US:

# degree_F, inHg, inch, inch_per_hour, mile_per_hour

#

# To use this driver, put this file in the weewx user directory, then make

# the following changes to weewx.conf:

#

# [Station]

# station_type = wxMesh

# [wxMesh]

# host = localhost # MQTT broker hostname

# topic = weather # topic

# driver = user.wxMesh

#

# If the variables in the file have names different from those in weewx, then

# create a mapping such as this:

#

# [wxMesh]

# ...

# [[label_map]]

# temp = outTemp

# humi = outHumidity

# in_temp = inTemp

# in_humid = inHumidity

from __future__ import with_statement

import syslog

import time

import paho.mqtt.client as mqtt

import weewx.drivers

DRIVER_VERSION = "0.1"

def logmsg(dst, msg):

syslog.syslog(dst, 'wxMesh: %s' % msg)

def logdbg(msg):

logmsg(syslog.LOG_DEBUG, msg)

def loginf(msg):

logmsg(syslog.LOG_INFO, msg)

def logerr(msg):

logmsg(syslog.LOG_ERR, msg)

def _get_as_float(d, s):

v = None

if s in d:

try:

v = float(d[s])

except ValueError, e:

logerr("cannot read value for '%s': %s" % (s, e))

return v

def loader(config_dict, engine):

return wxMesh(**config_dict['wxMesh'])

class wxMesh(weewx.drivers.AbstractDevice):

"""weewx driver that reads data from a file"""

def __init__(self, **stn_dict):

# where to find the data file

self.host = stn_dict.get('host', 'localhost')

self.topic = stn_dict.get('topic', 'weather')

# how often to poll the weather data file, seconds

self.poll_interval = float(stn_dict.get('poll_interval', 5.0))

# mapping from variable names to weewx names

self.label_map = stn_dict.get('label_map', {})

loginf("host is %s" % self.host)

loginf("topic is %s" % self.topic)

loginf("polling interval is %s" % self.poll_interval)

loginf('label map is %s' % self.label_map)

self.payload = "Empty"

#self.payloadList = [payload]

self.client = mqtt.Client(client_id="XXX", protocol=mqtt.MQTTv31)

#self.client.on_connect = self.on_connect

self.client.on_message = self.on_message

self.client.username_pw_set("XXX", "XXX")

self.client.connect(self.host, 1883, 60)

self.client.subscribe(self.topic, qos=1)

# The callback for when a PUBLISH message is received from the server.

def on_message(self, client, userdata, msg):

self.payload = str(msg.payload)

logdbg("Got message %s" % str(msg.payload))

def genLoopPackets(self):

while True:

self.client.loop()

# read whatever values we can get from the MQTT broker

logdbg("Working on payload : %s" % self.payload)

if self.payload != "Empty" :

data = {}

row = self.payload.split(",");

for datum in row:

(key,value) = datum.split(":")

data[key] = value

if( key=="TIME" and data[key] == "0"):

data[key] = str(int(time.time()))

logdbg("key: "+key+" value: "+data[key])

# map the data into a weewx loop packet

_packet = {'usUnits': weewx.METRICWX}

for vname in data:

_packet[self.label_map.get(vname, vname)] = _get_as_float(data, vname)

yield _packet

self.payload = "Empty"

logdbg("Sleeping for %d" % self.poll_interval)

time.sleep(self.poll_interval)

self.client.disconnect()

@property

def hardware_name(self):

return "wxMesh"

I got 0.1mm on Payload hail...but If I look at the values they doesn't match (printed on my website).
If I change to 'US' units then my Celsius values mapped wrong so I think in general changing to METRICWX in the wxMesh driver works.

I think I have to look in the database what hail values were written if I simulate a 0.1mm rain event (at the moment one switch to press...just before wirering to the bucket in the garden)....

Thanks for you help.

Regards,

Christian

[Thomas Keffer]

I'm still a little hazy on what the issue is. What do you mean by "the values don't match"? Which values? And I don't know where your website is.

​The wxMesh driver (at least as posted here) expects weewx.METRIC units, that is, rain (and hail) should be in centimeters, not millimeters.​ You can't just simply change to weewx.METRICWX in your copy, because then the other units will be wrong. 

Run weewx from the command line and watch what it prints. Tip the bucket and see what value gets printed. Is it what you expected?

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Christian Peters]

Tom,

I got it working...but I don't now why?! :-D

To clarify I used a second instance of weewx running wxMesh. As I has only one rain bucket per weewx I thougth I could change back in the weewx instance running wxMesh to the rain database entry.
My binding for wxMesh is now 'rain=rain'.
I changed in the wxMesh driver to METRICWX as I want the mm for rain and send 0.1 as value per bucket tip. 5 time tip: wxMesh grabs 0.5

wxMesh: Got message TIME:0,radiation:0.25,cpm:45,cps:1,temp_2m:50.81,temp_5cm:50.60,temp_dht22:19.70,hum_dht22:38.50,rain:0.50
Dec  4 01:17:22 sensweewx[24607]: wxMesh: Working on payload : TIME:0,radiation:0.25,cpm:45,cps:1,temp_2m:50.81,temp_5cm:50.60,temp_dht22:19.70,hum_dht22:38.50,rain:0.50
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: TIME value: 1512346642
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: radiation value: 0.25
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: cpm value: 45
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: cps value: 1
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: temp_2m value: 50.81
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: temp_5cm value: 50.60
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: temp_dht22 value: 19.70
Dec  4 01:17:22 sensweewx[24607]: wxMesh: key: hum_dht22 value: 38.50
Dec  4 01:17:22 rumo sensweewx[24607]: wxMesh: key: rain value: 0.50
Dec  4 01:17:22 rumo sensweewx[24607]: wxMesh: Sleeping for 5

The query in my main weewx instance to the wxMesh (sensweewx with mqtt):

             <tr>
                <td class="stats_label">Rain today Lambrecht</td>
                <td class="stats_data">$day($data_binding='mqtt_binding').rain.sum</td>
              </tr>

and the result is now ok:

Rain today Lambrecht: 0,5 mm

Maybe the mapping to hail was the problem...now all works as expected!?

[Thomas Keffer]

I don't think so. "hail" is treated no differently than "rain."

-tk

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscribe@googlegroups.com.

[Christian Peters]

Ok....so as usual: the user! :-D

Thanks for your patience and help!

Regards,

Christian

[bgra...@umw.edu]

Hi Christian,

I have built a MightyOhm geiger counter and would like to integrate it with my VP2 weewx station (grattans.org/wx). I'm not much of a programmer and would like more info on the hardware connections and software you have setup. Thanks in advance.

Bob