Common database for doing multi-site comparison

[Tomasz Lewicki]

I have the following idea and problem at the same time. I would like to build a small network of weather stations in the area, say 5-10 units. Of course, they will all be powered by Weewx and Raspberry Pi. Apart from the fact that each one will publish data on the internet independently of the others (one domain, separate subdomains), I am thinking about a collective presentation of data from the stations - only selected parameters, e.g., temperature, wind, precipitation.

Each Weewx instance is a separate weewx.sdb database with an identical structure. If I understand correctly, I would have to create a separate (collective) database in which the indexes of individual parameters would be different for each station, e.g., temp_pws01, temp_pws02, etc. The idea is that such a comparison would not be generated in real time, as in a standard Weewx installation, i.e., every 5-10 minutes, but, say, 4 times a day. I figured that the databases from each station would be sent via rsync to a remote server, and then specific columns would be extracted from them “somehow” and copied to a new, shared database, from which a summary would be made. Can the final step be done with Weewx, or do I need to use other tools?

The question is, has anyone done something like this before and could point me in the right direction?

Translated with DeepL.com (free version)

[Graham Eddy]

if they are all on the same box, then cheetah tags can point into another weewx database for values

e.g. in my weewx.conf. if i have another database with data_binding ‘wx_other’, i can have tag day.outTemp.max(data_binding=‘wx-other’)

i have a weewx instance ‘wxrep’ that only exists to pull data from other instances to present a consolidated whole

⊣GE⊢

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/a6e984e8-0526-4b71-93cb-fc545a942575n%40googlegroups.com.

[Tomasz Lewicki]

It sounds interesting! Could you share your configuration files? 

[Graham Eddy]

just some excerpts will suffice.

see that station ‘wxrep’ has its own database ‘weewx.sdb’ but also external database ‘vantage.sdb’ referred to using data_binding ‘vantage'

/home/weewx/wxrep/weewx.conf (excerpt)

[Databases]              

                         

    # A SQLite database is simply a single file.

    [[archive_sqlite]]   

        database_name = weewx.sdb

        database_type = SQLite

                         

    [[vantage_sqlite]]   

        database_name = vantage.sdb

        database_type = SQLite

… … … … 

[DataBindings]

         

    [[wx_binding]]

        # The database must match one of the sections in [Databases].

        # This is likely to be the only option you would want to change.

        database = archive_sqlite

        # The name of the table within the database.

        table_name = archive

        # The manager handles aggregation of data for historical summaries.

        manager = weewx.manager.DaySummaryManager

        # The schema defines the structure of the database.

        # It is *only* used when the database is created.

        #schema = schemas.wview_extended.schema

        schema = user.schemas.wxrep.schema

         

    [[vantage]]

        database = vantage_sqlite

        table_name = archive

        manager = weewx.manager.DaySummaryManager

        schema = user.schemas.vantage.schema

/home/weewx/wxrep/skins/ge-wxrep/daycharts.frag.html.tmpl (excerpt)

<figure class="chart">

    <img src="flux/images/vantage/daytempcomb.png"/>

    <div>

      <div>Out</div>

      <div>Min</div><div>$span($data_binding="vantage",$day_delta=1).outTemp.min</div><time>$span($data_binding="vantage",$day_delta=1).outTemp.mintime.format("%H:%M %d/%m")</time>

      <div>Max</div><div>$span($data_binding="vantage",$day_delta=1).outTemp.max</div><time>$span($data_binding="vantage",$day_delta=1).outTemp.maxtime.format("%H:%M %d/%m")</time>

      <div>Dew</div>

      <div>Min</div><div>$span($data_binding="vantage",$day_delta=1).dewpoint.min</div><time>$span($data_binding="vantage",$day_delta=1).dewpoint.mintime.format("%H:%M %d/%m")</time>

      <div>Max</div><div>$span($data_binding="vantage",$day_delta=1).dewpoint.max</div><time>$span($data_binding="vantage",$day_delta=1).dewpoint.maxtime.format("%H:%M %d/%m")</time>

    </div>

  </figure>

cheers

⊣GE⊢

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/65d85241-6bf2-4034-9038-00eba27ebac7n%40googlegroups.com.

[p q]

Something like Grafana might be good for this

Peter Quinn
(415)794-2264

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/6A52556B-E055-41A3-AF24-9F6AE7063477%40geddy.au.

[Vince Skahan]

Take a look at the wiki for https://github.com/weewx/weewx/wiki/dashboards for some ways to do this.  There are others.  Lots of others.

For just a few readings from just a few stations, MQTT is the simplest way:

• run a mosquitto broker on your central site
• have each weewx instance publish MQTT to a unique topic such as station1/loop, station2/loop, etc.
• have the central weewx MQTTsubscribe to the desired topics and save to one weewx db

Nice thing is MQTTsubscribe can run as a driver, so you could use that to get data from the incoming MQTT topics into one central weewx db.  You'll have to map those items to weewx db elements, so if you have a lot of readings from many stations, you'll need to add db columns 'once'.

You'd have to of course create your own weewx skin to create whatever HTML and graphs you want, but you'd be totally weewx with the exception of running a Mosquitto MQTT broker (easy to do) on your central site.

That said, if you want to have 'all' possible readings from 'many' stations then it's probably best to publish to influxdb and display with something like grafana or maybe even Home Assistant.  That will get you almost infinite extensibility but it's far more work to learn, set up, and set up securely.(note - I never looked into how to secure a grafana dashboard from Internet bots.  I was fortunately LAN-only).

[bell...@gmail.com]

If you want to try something on the bleeding edge, you could look into my weewx-mqtt/replicate extension, https://github.com/weewx-mqtt/replicate.

What is weewx-mqtt/replicate? It is my attempt to create a ‘light weight’ way of replicating weewx databases. (I say ‘light weight’ because I have decided that there is no such thing as a light weight replicator.) It uses the request/response functionality of the V5 MQTT protocol. It consists of a service running on a current (the primary) instance of WeeWX and a driver that runs on the secondary instance.

Why not just use one of the WeeWX extensions that publishes and another extension to subscribe to the topic? One limitation of this is, if the secondary instance is offline; data will not be replicated. weewx-mqtt/replicate attempts to overcome this. When the secondary instance comes on-line, it asks the service running on the primary instance for all the data since the last record in its database (aka a catchup is performed). 

Disclaimer 1: It is still very much beta (maybe alpha) code. I have not moved it into production. I only use it to replicate to a development environment. 

Disclaimer 2: It has limited documentation, so one has to be willing to work through setting it up with limited help. This includes all that goes into setting up a MQTT broker.

Disclaimer 3: Not sure if it could replicate multiple databases into a single one. But, if you are interested in experimenting I would be happy to look into supporting this.

Disclaimer 4: All of the usual stuff.

rich