Raspberry/MQTT > VPS Server

[Gert Andersen]

Hi there

Is this scenario possible.

Currently I have an Ecowitt GW1000 which sends data to a VPS Server every 16 seconds with Weewx Interceptor, MQTT and Belchertowner installed on the VPS Server. This works fine.

I also have a Davis Vantage Pro2, which is connected to a Meetobridge via Davis USB logger. This also works fine.

Here's my new scenario:

Davis USB logger connected to rpi/weewx. On rpi/weewx I want to install mqtt with the following content:

[[MQTT]]

        server_url = mqtt: // user: password @ vps_host_name: 1883 /

        topic = weather

        binding = archive, loop

        aggregation = aggregate

        log_success = False

        log_failure = True

On <vps-host_name> I have installed Mosquitto which works fine with upload via Interceptor. I want this to be able to upload every 2 seconds.

I have not yet connected Davis to the rpi / weewx. but tried the configuration with Simulator and the above MQTT configuration. Belchertown on <vps_host_name> connects fine to MQTT, but is awaiting data. Is it because Simulator does not send data via MQTT or because my configuration is wrong.

Gert

[Pat]

Belchertown should get data as soon as there is a dateTime in the MQTT payload. Simulator should send this. What you have looks correct, but more validating needs to take place. Use another client to connect to the MQTT broker to validate you're seeing any data at all therethere. 

Or run Belchertown in debug mode and check the Chrome logs to see what it says. To run debug, add /?debug=true to the end of your site, like https://belchertownweather.com/?debug=true - then open the Chrome console by going to Settings > More Tools > Developer Tools and open the Console tab. Almost everything the skin is doing is viewable in this debug mode, from chart generation to any and all MQTT payload data it's seeing.

[Gert Andersen]

Hi

Thanks for your answer, I'll try that.

I'm a little bit unsure about  the configuration and whether it's right.

On the rpi / weewx I have installed the weewx / mqtt extension to send data on port 1883.

On vps_host I have installed weewx as well as Belchertown. I'm not expecting to install weewx / mqtt here. This is what my weewx.conf looks like on the vps_host:

[[Belchertown]]

         HTML_ROOT = / var / www / html

         skin = Belchertown

         [[[Extras]]]

             belchertown_root_url = http: // My Server

             earthquake_enabled = 1

             forecast_enabled =

             mqtt_websockets_enabled = 1

             mqtt_websockets_host = xxx-xxx-xxx-xxx.vpsCloudServer

             mqtt_websockets_port = 9001

             mqtt_websockets_ssl = 0

             mqtt_websockets_topic = weather / loop

             disconnect_live_website_visitor = 1800000

My question is also which weewx driver to install on the vps_host. In principle, weewx should only receive data via port 1883 and not use any driver (Vantage, etc) or am I wrong on it

Gert

[Pat]

Are the spaces you're adding for the html_root, belchertown root url, and mqtt topic intentional for this forum post? If not, remove those extra spaces they as they will cause a problem. 

Do you need to run weewx on the vps host? I don't think you need to. weewx on your Pi will generate the webpages needed, then it can FTP or SCP those files to the VPS host automatically. Then MQTT on vps host will help broker the data. So the config you have will go on the weewx.conf on your Pi as that information will get processed into the webpage that weewx will build. 

Port 1883 is MQTT's port; not weewx. Read through this post on how to setup MQTT and testing it for valid data and see if it helps you understand the difference between the two systems. 

Also read through this post on how weewx transfers the generated webpages to the remote web server (your vps)

[Greg Troxel]

Gert Andersen <gert.a...@gmail.com> writes:

> [[MQTT]]
> server_url = mqtt: // user: password @ vps_host_name: 1883 /
> topic = weather
> binding = archive, loop
> aggregation = aggregate
> log_success = False
> log_failure = True

For what it's worth, I have

Davis VP2
raspberry pi (running NetBSD)
weewx (current stable, ish, not v4)
MQTT add on

another computer (on the lan)
mosquitto
home assistant (HA)

and I am getting weather data (via aggregate) into HA just fine. I am
using archive only, not loop.

So there is no reason your davis/pi/weewx/mqtt should not be able to
send mqtt to a broker on a VPS (assuming TCP works with NAT/firewall
etc.)

[Gert Andersen]

Hi

Thanks a lot for your answers.

@Pat, i don't know there are spaces, The mqtt configuration is working fine with Ecowitt GW1000 uploading data to the VPS server using Interceptor.I'll ty the configuration you mention with uploading via ftp. Will I not miss the real-time data every 2 seconds?

@Greg, seems you're running the system I'm looking after. Is it possible to provide .conf for the weewx and the mqtt broker. Is far as I can see, the only difference is you're running the system on your lan and I'm running the weewx/Davis on the lan and the mqtt broker on cloud.

Thanks

Gert

[Greg Troxel]

Gert Andersen <gert.a...@gmail.com> writes:

> @Pat, i don't know there are spaces, The mqtt configuration is working fine
> with Ecowitt GW1000 uploading data to the VPS server using Interceptor.I'll
> ty the configuration you mention with uploading via ftp. Will I not miss
> the real-time data every 2 seconds?

The spaces look bizarre in the url. try dropping them.

> @Greg, seems you're running the system I'm looking after. Is it possible to
> provide .conf for the weewx and the mqtt broker. Is far as I can see, the
> only difference is you're running the system on your lan and I'm running
> the weewx/Davis on the lan and the mqtt broker on cloud.

I can't send the exact mosquitto config since it has a bunch of acls
etc, and it ends up being huge and too hard to sanitize.

In weewx, I have the following, which is pretty much like what you have
except:

- it lacks the spurious spaces [this could be a huge issue]
- I am not sending loop
- I use TLS
- I fix the pressure units from FF to SI
- I round dewpoints to 1 decimal (don't remember why clearly but surely they
are being computed and not rounded from temp/%humidity)

----------------------------------------
[[MQTT]]
server_url = mqtt://user:pass...@fully.qualified.domain.example.com:8883/
topic = weather/someplace
aggregation = aggregate
retain = True
[[[tls]]]
ca_certs = /etc/openssl/certs/ca-certificates.crt
[[[inputs]]]
[[[[barometer]]]]
name = barometer_hPa
units = hPa
[[[[altimeter]]]]
name = altimeter_hPa
units = hPa
[[[[pressure]]]]
name = pressure_hPa
units = hPa
[[[[dewpoint]]]]
format = %.1f
[[[[inDewpoint]]]]
format = %.1f
----------------------------------------

You should be aware that with mosquitto, if a write to a topic is not
permitted by acls, it just doesn't happen, and it's hard to debug.

To debug this, you should break it into pieces.

On your VPS, use mosquitto_pub and mosquitto_sub to publish and receive
someting (junk text is fine) to the topic you want to use, using hte
credentials you will use to publish and subscribe. Get that working;
it's all about mosquitto and not about weewx at all. Read the docs
about acls. Crank up debug levels if it doesn't work.

From the RPI, use telnet to the server and port 1883 (using the same
name you put int he config). Use both the -4 and -6 options if you have
an AAAA record for vps_host_name. If vps_host_name isn't a valid DNS
name and is in some host file, check extra carefully from the RPI about
this.

Then, install mosquitto on your RPI, even if you don't need it
otherwise. Repeat with mosquitto_pub on the RPI, with mosquitto_sub
running on the VPS. Then, if you want to subscribe from elsewhere, try
that.

Finally, start up weewx. Run tcpdump on port 1883 and see if you see
things. Enable debugging on your broker and watch the logs.

[Gert Andersen]

Hi

Thanks, now I have a starting point.

Gert

[Gert Andersen]

Hi

One more question. Should I just Mosquitto on vps-host(no weewx) and uploading the weewx reports?

Gert

On Wednesday, February 26, 2020 at 9:56:02 AM UTC+1, Gert Andersen wrote:

[Greg Troxel]

Gert Andersen <gert.a...@gmail.com> writes:

> One more question. Should I just Mosquitto on vps-host(no weewx) and
> uploading the weewx reports?

on vps-host, you should run the mosquitto daemon.

on the weewx computer, there is no need for a broker*.



* One can run multiple brokers and bridge them, but I do not recommend
that to anyone who is having the slightest amount of trouble!

[Gert Andersen]

Hi

Sorry, but I'm still fiddling around with the configuration. I  have made a little drawing showing what I'm trying to achieve and also ask some questions.

As mentioned earlier, I have no problems using mqtt, when I'm running weewx (uploading from Ecowitt GW1000) and the broker on the same vps host(no Raspberry). So I have a very basic understanding about mqtt. 

My problems of understanding starts when I separate weewx(Raspberry) and Mosquitto (vps host) and which components on which device.

Gert

[Greg Troxel]

Gert Andersen <gert.a...@gmail.com> writes:

> Sorry, but I'm still fiddling around with the configuration. I have made a
> little drawing showing what I'm trying to achieve and also ask some
> questions.

Please see my previous message about testing with mosquitto_pub and
mosquitto_sub. I don't think it makes sense to worry about skins and
extra brokers until you can use the test clients and confirm that they
work.

> As mentioned earlier, I have no problems using mqtt, when I'm running weewx
> (uploading from Ecowitt GW1000) and the broker on the same vps host(no
> Raspberry). So I have a very basic understanding about mqtt.

OK, but there is probably some issue with your other config and you will
need to figure out what that is.

> My problems of understanding starts when I separate weewx(Raspberry) and
> Mosquitto (vps host) and which components on which device.

The broker location should be independent of weewx and even the web
server. It is completely normal for the broker to be on a different
host than an mqtt client. I advise strongly against a second local
broker and bridging, until you have the basic setup working and then
only if you can explain why that's a good idea.

So far because you have not shown data from the test clients we have
zero idea of where in the long chain of dataflow you are having
problems. You should be able to use mosquitto_sub with options -h -p -u
-P (filled in) and subscribe to weather and see the json. And to use
mosquitto_pub on the pi and send some fake content and see that appear.

Start with pub and sub on the VPS. Then move the pub to the pi. Then
move the sub to some other random host. Then turn on weewx instead of
the pub client, but keep watching with the sub client. Add only one
thing at a time. Report back when you have one thing that works and the
next step with *only one change* that fails.

Make sure your VPS has proper DNS entries.

[Greg Troxel]

To answer your other questions in the pdf (probably best to keep text in
plain text):

As I understand it the belchertown skin, like all others, is run on
your weewx host and generates html files. Then, you arrange to
sftp/rsync these to your web server.

users load the html page and that has instructions to their browser to
connect to the broker and get rapid updates. So the page view in the
browser is rapidly updating, even though the browser is not reloading
html and the web server is not getting updated html.

I see no need to run weewx on the web server. That's just more
complexity.

websocket debugging is another layer of complexity. But you can test
with mosquitto_sub with regular sockets

[Gert Andersen]

Hi Greg

Thanks for your time to look into this.

I have a week off next week, so I can start working with your good advises.

I'll get back to give an update.

Rgds

Gert

[Gert Andersen]

Hi

Finally, up and running here with Simulator. Next week time for connecting the Davis.

http://sevejret.dk

Just some minor adjustment to the skin, then I'm done. The hardest part was to setup the FTP on AWS, but it is working now and all configurations files are kept in a safe place!

Configuration:

Raspberry running weewx, weewx mqtt extension and Belchertown skin. The skin files are uploaded with ftp every 5 minutes. 

VPS Server: AWS EC2 Instance running Ubuntu 18.04, Mosquitto Broker, Apache Webserver  and vsftpd

Many thanks to Greg and Pat who have helped me. I could not have done this without your help.

Gert

[Greg Troxel]

Can you explain what you had wrong, so that others can have an idea what
to check?