Weather station on Raspberry Pi with LTE/4G connect

[Christian Radermacher]

I want to build a weatherstation based on an RPi. The Station is solar powered an has an Internet Connection with 4g/LTE.

So the best way is to push the Data to an Webserver/Database offsite the Pi.  I don´t want to store the Date local on the Pi. The Website is available over the Internet, so everyone can see it, the Pi isn't available from the Internet.

My Hardware would be a Froggit WH1080 with Anemometer and Outside Temperature/Humidity.

Is it possible to set this up with Weewx? 

Is there other Hardware wich is better for my use case?

Regards

Christian

[Graham Eddy]

do you really want to push the raw data form RPi to webserver, on which you will have to generate the reports (e.g. another instance of weewx), or generate the reports on RPi and push the reports to the webserver?

[Christian Radermacher]

I want to use the way with less Data transfer over the 4G/LTE Connection.

The second thing is less SD Card writing on the Pi. 

What Do you Think?

Regards

Christian

[michael.k...@gmx.at]

Using a SD-Card is absolutely no option, in terms of reliability. The other thing is the energy consumption. What kind of Solar Power Plant and Battery will you use?

[p q]

My experience is that the SD card is reliable. Running Weewx on a Raspi since 2016 with zero SD card issues. Maybe I'm just lucky. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/fd40fb4d-a258-445b-b6a8-94c98d64b4f2n%40googlegroups.com.

[Tom Keffer]

Me too. This installation has been running on the same SD card for over 6 years with no issues. Over 2 years of continuous uptime: http://www.threefools.org/weewx/status/index.html

Generally, sending the raw data, rather than web pages, will use less bandwidth. You could use two instances of weewx, one locally, one on the web server, linked by MQTT. I wouldn't be surprised if someone around here has done something similar.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAA1SM21nDqWyVyDywvRGY0-pQENGNjjzR9r2gD9SBbEBseMHbg%40mail.gmail.com.

[Christian Radermacher]

Generally, sending the raw data, rather than web pages, will use less bandwidth. You could use two instances of weewx, one locally, one on the web server, linked by MQTT. I wouldn't be surprised if someone around here has done something similar.

This sound good, but I can't found Information on MQTT in Weewx.

But if I configure in the Solar Powered Modul "no Report rendering, and define a MySQL Database which is remote I have the Data at my Database. Then I configure a second device with the same Database an Report-rendering on. This might be a goal, or is there something missing?

Regards

Christian

[p q]

Search this forum for performance issues with remote mysql and decide if it applies to you. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/15f5ed1b-0b62-4b60-8467-50002ab16db7n%40googlegroups.com.

[Tom Keffer]

On Mon, Mar 1, 2021 at 9:12 AM Christian Radermacher <radermache...@gmail.com> wrote:

This sound good, but I can't found Information on MQTT in Weewx.

Did you try the wiki: there are several MQTT extensions listed there. https://github.com/weewx/weewx/wiki

Also, search the forum. Lots of people have done MQTT projects.

[Karen K]

MQTT on sending side

MQTT on the receiving side

You additionally need an MQTT broker like test.mosquitto.org or you install Mosquitto on your Web-Server. The second is the better solution. The documentation of the Belchertown skin includes a very good description for setting up an MQTT broker.

[Christian Radermacher]

Thanks a lot for the reply, ich will give it a try.

I think the best solution ist MQTT an two instances of Weewx.

I´ll come back, if I have more Questions

Regards

Christian

[bell...@gmail.com]

A couple more thoughts.

1. MQTTSubscribe can be a bit complicated/convulated to configure, so don’t hesitate to reach out with any questions.

2. I do some thing similar to get data into my test environment. Currently if there is a failure (broker down, client down, etc.), data may not make it to the test system. In order to ensure data delivery, I’ve started investigating ‘persistent sessions’ (clean_session = False) and a QOS = 2. I think on the subscribe side it will be that easy - assuming the broker is configured correctly. On the publish side, I’ll probably have to keep track of failed publishes and have some type of retry - not sure I really want to deal with that...

-rich

[Graham Eddy]

i’ve implemented my own bidirectional gateway and i also parked this issue - next version’s problem, as i had no present need for it. but i am moving my system (of which weewx is a component) to mqtt backbone (e.g. my alarms weewx service moving to mqtt for more general use) so i would be very interested to discuss this issue with you as it impacts all my mqtt apps. i tracked the publish requests while they were in client-side queue but didn’t persist them, wondering about true need and performance cost

[bell...@gmail.com]

Graham,

I am still very much in the ‘idea stage’, but would be happy to chat. In order to not hijack this thread we should probably start another thread or start a discussion over here, https://github.com/bellrichm/WeeWX-MQTTSubscribe/discussions

- rich

[Graham Eddy]

i’m headed to gitub discussions now...

[Christian Radermacher]

Hi all,

i am struggling with the configuration of the MQTT Subscriber from bellrichm.

Ok is the remote station with the DP1500, a customized skin an an MQTT Publish.

    [[MQTT]]

        server_url = mqtt://user:pw@ip:1883

        topic = weather

        unit_system = METRIC

        binding = archive

        aggregation = aggregate

ok is the MQTT Broker with Mosquitto. He receives the Topic, I can see this with an MQTT Explorer.

Here is the string from the Broker:

{"dateTime": "1614942000.0", "inTemp_C": "25.580000000000005", "outTemp_C": "2.4000000000000004", "inHumidity": "33.0", "outHumidity": "91.73333333333333", "pressure_mbar": "967.3399999999997", "relbarometer": "967.34", "luminosity": "7575.2", "uvradiation": "6.0200000000000005", "UV": "0.0", "rain_cm": "0.0", "stormRain_cm": "0.61", "rainRate_cm_per_hour": "0.0", "dayRain_cm": "0.35000000000000003", "weekRain": "7.4", "monthRain_cm": "0.7400000000000001", "yearRain_cm": "0.7400000000000001", "windSpeed_kph": "0.1200002982589136", "windDir": "223.0", "windGust_kph": "1.800004473883704", "windGustDir": "223.0", "daymaxwind": "3.6", "altimeter_mbar": "1023.3070575534325", "appTemp_C": "0.5729997426277262", "barometer_mbar": "1025.6542650419428", "cloudbase_meter": "622.358803974661", "dewpoint_C": "1.1941448275071624", "heatindex_C": "2.4000000000000004", "humidex_C": "2.4000000000000004", "inDewpoint_C": "8.102394002973144", "windchill_C": "2.4000000000000004", "interval_minute": "5.0", "windrun_km": "0.010000024854909468", "hourRain_cm": "0.020000000000000018", "rain24_cm": "0.6100000000000003", "usUnits": "16.0"}

not ok is the second Station. I don't understand the config of MQTT Subscribe.

In which case I user Service, in which case I use driver?

snippet of the Weewx.conf:

[Station]

    station_type = MQTTSubscribeDriver

[MQTTSubscribeDriver]

    host = 192.168.114.xx

    port = 1883

    user = user

    password = password

[Engine]

    # The following section specifies which services should be run and in what order.

    [[Services]]

        data_services = "", user.MQTTSubscribe.MQTTSubscribeDriver

In the log I get the following:

Mar  5 11:51:56 weewx-intern weewx[1087] DEBUG __main__: Initializing engine

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__: Caught unrecoverable exception:

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****  'driver'

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****  Traceback (most recent call last):

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 151, in main

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****      engine = weewx.engine.StdEngine(config_dict)

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 81, in __init__

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****      self.setupStation(config_dict)

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 103, in setupStation

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****      driver = config_dict[station_type]['driver']

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****    File "/usr/lib/python3/dist-packages/configobj.py", line 554, in __getitem__

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****      val = dict.__getitem__(self, key)

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****  KeyError: 'driver'

Mar  5 11:51:56 weewx-intern weewx[1087] CRITICAL __main__:     ****  Exiting.

Sorry, but I can't find the solution.

[Michael]

Christian,

Two things I noticed:
It seems that you have not defined a topic for [MQTTSubscribeDriver]. Something like:
    [[topics]]
        [[[weather/loop]]]

If you have not enabled autentification for subscribing to MQTT on your server, you should remove the user and password part or comment it out with #.

 user = user
 password = password

[Christian Radermacher]

sorry, copy error from me, the topic is defined in the [MQTTSubscribeDriver] Part.

I disabled user/password, but the error is the same

Regards

Christian

[Graham Eddy]

a driver creates the original packet with a timestamp, a service changes (usually adds to) the existing packet.

some modules, like gw1000, can be configured to be in either mode (i.e. originate packets, or augment existing packets produced by some other driver). i do this, with vantage as driver and gw1000 in service mode

[Christian Radermacher]

ok, i installed the receiver Weewx new from scratch. No it seems to work without errors.

I See in the (debugLevel 2) the incoming MQTTValues. But they don't where displayed in the screen. It seems, there is one point missing.

Regards

Christian 

[Michael]

you need something like this:

[[[your Topic]]]

            # The incoming field name from MQTT.
            [[[[ inTemp_C ]]]
                # The WeeWX name.
                # Default is the name from MQTT.
                name = inTemp

...

[bell...@gmail.com]

Christian,

I’d recommend adding ‘append_units_label = False’ on the publish side. This will stop the appending of units to the label and you won’t have to rename them on the subscribing side.

I’d expect the subscribing config to look something like this.

[MQTTSubscribeDriver]

    driver = user.MQTTSubscribe

    host = 

    

    [[message_callback]]

        type = json

    

    [[topics]]

        unit_system = METRIC

        [[[weather/loop]]]

If you post the log from startup through one archive period, I could dig deeper.

Important note, with the set up above, I believe you are publishing archive records. These are received and put into loop packets by MQTTSubscribe. WeeWX would then create archive records out of them. I think this should all work, but there might be some timing issues. The log would show what is going on.

I do have alpha/experimental code that would receive the MQTT payload and process it as an archive record. It would very much be “use at your own risk”.

- rich

[bell...@gmail.com]

Thinking more, I am pretty sure the alpha code won’t work for you. 

- rich

[Christian Radermacher]

Hi Rich,

thanks for the reply. Now I can see that it is working, but my Skin would not be generated. So I search another error.

Regards

Christian

[Christian Radermacher]

Now everything is working fine, I have no idea, why the Skin would not generate. 

I will play around, an come back with other questions, if necessary.

Thanks to all, helped me

Regards

Christian 

[Christian Radermacher]

Now I have one more Question.

On the sender Site I have this Config now:

[[MQTT]]

        server_url = xxxxx

        topic = weather

        unit_system = METRIC

        binding = loop, archive

        aggregation = aggregate

        append_units_label = false

So I sent loop and archive.

Both will sent as "loop".

In this case the receiver show the same Data, with the same Time.

If I have the binding only "archive" the receiver is 5 Minutes back.

Which is the right config for me?

Regards

Christian

[bell...@gmail.com]

Christian,

You have a few options that I know of, each with its pros and cons.

1. Publish on each loop packet (Note, not on both archive and loop). In this scenario, MQTTSubscribeDriver will generate loop packets and from these WeeWX will create an archive record. In theory this should.match the archive records of your remote location. But, if for some reason some MQTT payload goes missing, there could be discrepancies. 

2. Publish on the archive record.  As you noted, there will be time lag of your archive interval. But the archive data will match and if some data does not arrive, you can export the remote data and import it.

3. If you want to live on the edge....  There is a fork of Matthew’s weewx-mqtt that publishes loop and archive data on separate topics.  It can be found here, https://github.com/moonpyk/weewx-mqtt. With this you could use MQTTSubscribeDriver’s alpha/experimental option, archive_topic.  With this set, MQTTSubscribeDriver will create loop packet’s from the loop topic. But when it is time to create an archive record, it will pull the data from the archive topic. I have this running in a very controlled environment. I have no idea how it would ‘hold up’ in the real world where messages might go missing, be delayed, etc. And again, it is alpha code - so use at your risk.  

- rich

[Christian Radermacher]

Hi Rich, thanks a lot for your reply. Great explanation.

I will test with loop sending, and wait the alpha code will be beta.

Regards

Christian