Belchertown MQTT Issues

[Matt Johnson]

I've been trouble shooting getting the Belchertown skin MQTT Websocket real time updates to work on my site shakerweather.com for a lot of this week.

I have WeeWx installed on a dedicated thin client on Ubuntu 20.04 LTS and have Mosquitto and NGINX installed on the same machine. Running bare metal, no docker or VMs here.

External access to WeeWx website is handled via NGINX reverse proxy manager with SSL certs on a different server via docker. Requests to shakerweather.com are sent to the proxy server and then to the WeeWx machine.

With this setup, the site is served up fine internally and externally with the updates at archive intervals every 5 minutes.

I know that the weewx-mqtt extension is installed correctly as I have been able to test it locally and get the websocket updates to work perfectly with the following configs:

weewx.conf

[[MQTT]]
        server_url = mqtt://user:pw@localhost:8883/
        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate

skin.conf

 # MQTT Websockets defaults
    mqtt_websockets_enabled = 1
    mqtt_websockets_host = "localhost"
    mqtt_websockets_port = 8083
    mqtt_websockets_ssl = 0
    mqtt_websockets_topic = "weather/loop"
    disconnect_live_website_visitor = 1800000

I am only able to see the the real time updates on the local machine only with WeeWx and Mosquitto. If I try to access it by IP address elsewhere on my LAN on other clients it does not connect and eventually fails. No luck externally either - despite my NGINX Reverse Proxy Manger handling serving the page and SSL certs the websocket real time updates don't pass through. That was my original thought of how it would work.

After much trial and error, and reading every thread imaginable on this along with many messages and some correspondence with Pat O'Brien I decided to go ahead and setup a Digital Ocean Ubuntu VM and install Mosquito there to serve as a cloud broker. I followed Pat's instructions exactly as he outlines in setting up the cloud broker: https://obrienlabs.net/how-to-setup-your-own-mqtt-broker/

I have the cloud MQTT broker installed correctly at Digital Ocean with Let's Encrypt, and ran tests on it. Messages can be sent when authenticated, ports are open, etc. However, I can get no further with the websockets real time updates than "Connected. Waiting for data". If I reboot the cloud MQTT broker I immediately get a disconnected message on the website so it does appear to be connecting and waiting for data. Somehow the data is simply not transferring from my WeeWx client to the cloud MQTT broker at Digital Ocean. The other weird thing is if I try to access shakerweather.com or the website by local IP address on the machine that hosts WeeWx I always get a failed message, won't even connect to the server. However, any other client on my LAN and external on WAN does not have this issue.

Here are my current configs:

weewx.conf

[[MQTT]]
        server_url = mqtt://user:p...@mqtt.beldenserver.com:8883/
        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate
         [[[tls]]]
            tls_version = tlsv1
            ca_certs = /etc/ssl/certs/ca-certificates.crt

skin.conf

# MQTT Websockets defaults
    mqtt_websockets_enabled = 1
    mqtt_websockets_host = "mqtt.beldenserver.com"
    mqtt_websockets_port = 8083
    mqtt_websockets_ssl = 1
    mqtt_websockets_topic = "weather/loop"
    disconnect_live_website_visitor = 1800000

At this point, I have spent 20+ hours on this and hoping someone here can point me in the right direction, it seems data is just not feeding the MQTT topic. I'm fine with using Digital Ocean as a cloud MQTT server just to get it up and running. My preferred state is eventually to selfhost it all.

Thanks in advance for anything I may be overlooking, advice or possible solutions.

[Les Niles]

I think the first configuration, with the local mqtt broker  isn’t going to work because mqtt_websockets_host is set to localhost, which will only resolve to the weewx/mqtt server when the web browser is running on that server.  You need to set something here that will resolve to the weewx/mqtt server from any client that you want to get realtime updates.  

Not sure why the second config, with the cloud mqtt broker, isn’t working. Are you sure the mqtt broker is configured for SSL on port 8883?  You might put “log_success = true” in the weewx [[MQTT]] config and see if the log messages tell you anything useful.  

Or forget about the cloud server and go back to getting the first config working.  You need a DNS name that will resolve to your firewall and get port-forwarded (for port 8083) to the weewx/mqtt server, for mqtt_websockets_host. That should enable external access.  And, if your firewall will do hairpinning, it should work internally as well. It may take some magic with forwarding/masquerading rules on the firewall to get hairpinning to work.  (The alternative for internal access is to have an internal DNS server that resolves that hostname directly to the internal IP of the weewx/mqtt server for clients on the internal network.)

  -Les

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/38183ff6-d5d5-4151-a1b3-93fd618aef5cn%40googlegroups.com.

[Matt Johnson]

Thanks Les. I think you have helped clear some things up. For the sake of clarity and getting to the end state I desire and your suggestion let's focus on locally hosting (my first setup) over the cloud broker. I have a robust homelab and really like to keep things in house.

think the first configuration, with the local mqtt broker  isn’t going to work because mqtt_websockets_host is set to localhost, which will only resolve to the weewx/mqtt server when the web browser is running on that server.  You need to set something here that will resolve to the weewx/mqtt server from any client that you want to get realtime updates. 

This makes sense. I initially was on this path and changed the mqtt_websockets host to the ip address of the weewx/mqtt server. This did not work for other clients on the LAN which seemed strange given how I have lots of locally running services on a few different servers in my home lab and I access them all via device IP and port, with a few that are accessible externally via reverse proxy.

You need a DNS name that will resolve to your firewall and get port-forwarded (for port 8083) to the weewx/mqtt server, for mqtt_websockets_host. That should enable external access. 

I can use my mqtt.beldenserver.com DNS name I have setup at cloudflare for this. I am running OPNSense for my firewall so I should be able to do anything. Right now I have a firewall rule setup to send port 80/443 traffic to my NGINX Reverse Proxy where I have several DNS addresses pointing to different services on different servers. One of these takes shakerweather.com through 80/443 and through the reverse proxy and points to the weewx/mqtt server to return the webpage (which is working fine)

So you are saying I just need to add a rule to forward external port 8083 requests to the weewx/mqtt server IP and port?

And, if your firewall will do hairpinning, it should work internally as well. It may take some magic with forwarding/masquerading rules on the firewall to get hairpinning to work.  (The alternative for internal access is to have an internal DNS server that resolves that hostname directly to the internal IP of the weewx/mqtt server for clients on the internal network.)

My OPNSense firewall should be able to do hairpinning. I read on that briefly as I have only heard of the term and not too familiar with it.

[Doug Jenkins]

I had a number of struggles with this when I setup the Belchertown skin on my Raspberry Pi hosting my weather site, www.largoweather.com. I think the issue is that you need your mqtt_websockets_port to be set to 443 as the websocket traffic is getting filtered out by your firewall (9001->8083)

Here is what I did to get this to work on largoweather.com:

1. Setup Cloudflare to manage the DNS proxy on 2 domains: 

   - largoweather.com (A Record)

   - wx.largoweather.com (aka your mqtt.beldenserver.com) - CNAME Record pointing the content to largoweather.com

   - Setup my SSL/TLS to Strict. I am using Cloudflare to offload my SSL processing so essentially all traffic is coming in through 443.

   

   These are pointed to my public IP address which is dynamic. I use a shell script to update cloudflare' content to keep my public IP current for their system

2. NGNIX Proxy Manager : I use this program to manage my NGINX instance that acts as a reverse proxy manager for my domains:

     2.1 : Setup:

            - largoweather.com : Setup to point to my local server ip address. The scheme is http. I use the program's lets encrypt function to get a SSL certificate and force SSL traffic to my final weewx website.

             - wx.largoweather.com : This handles my sockets setup. I forward all traffic to port 9001 and use the same ssl certificate issued for largoweather. I force SSL on this setup as well.

3. mosquitto configuration : I setup mosquitto on the same server as my weewx install since everything is running on the pi. here is my mosquitto.conf:

           pid_file /var/run/mosquitto.pid

           persistence true

           persistence_location /var/lib/mosquitto/

           log_type error

           websockets_log_level 1023

           connection_messages true

           log_dest file /mnt/*****/weewx/logs/mosquitto.log

           

           allow_anonymous true

           password_file /etc/mosquitto/passwd

           acl_file /etc/mosquitto/acl

           listener 9001

           protocol websockets

           listener 1883

           protocol mqtt

           log_type error

4. weewx/Belchertown configuration: Here I setup my MQTT to talk to my local IP address on port 1883. Remember the traffic is all coming in on port 443, so that is the port I need Belchertown skin to essentially connect to resolve the web sockets requests.

   

    [[MQTT]]

        server_url = mqtt://joeuser:xxxxxxx@<YOUR-LOCAL-SERVER-IP>:1883/

        topic = weather

        unit_system = US

        binding = archive, loop

        aggregation = aggregate

[[Belchertown]]

             skin = Belchertown

             enable = True

             [[[Extras]]]

                 site_title = Largo Weather

         mqtt_websockets_enabled = 1

                 mqtt_websockets_host = wxsocket.largoweather.com

                 mqtt_websockets_port = 443

                 mqtt_websockets_topic = weather/loop

                 mqtt_websockets_ssl = 1

                 disconnect_live_website_visitor = 1800000

I hope this helps!

Doug Jenkins

[Matt Johnson]

Ok, progress with the locally hosted MQTT server.

I changed my DNS of mqtt.beldenserver.com to point to my firewall instead of the cloud instance I had setup and added the port forwarding rules to OPNSense. I can get the mqtt websocket real time updates to work internally on any client by accessing the site by ip address of the weewx/mqtt device with the following config:

weewx.config

[[MQTT]]
        server_url = mqtt://usr:p...@mqtt.beldenserver.com:8883/

        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate

skin.conf

   # MQTT Websockets defaults
    mqtt_websockets_enabled = 1
    mqtt_websockets_host = "mqtt.beldenserver.com"
    mqtt_websockets_port = 8083

    mqtt_websockets_ssl = 0
    mqtt_websockets_topic = "weather/loop"
    disconnect_live_website_visitor = 1800000

This says to me that I successfully have the mqtt.beldenserver.com DNS setup and forwarded in my router. I still can't see it externally nor locally with the secure website https://shakerweather.com. Perhaps I was hoping the NGINX reverse proxy that is handling the SSL cert would just return it all over SSL, so I think working on that setup is next.

I initially tried this config but no luck, and it broke everything locally accessed by IP as well.

weewx.config

[[MQTT]]
        server_url = mqtt://usr:p...@mqtt.beldenserver.com:8883/

        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate
        [[[tls]]]

        tls_version = tlsv1

        ca_certs = /etc/ssl/certs/ca-certificates.crt

skin.conf

   # MQTT Websockets defaults
    mqtt_websockets_enabled = 1
    mqtt_websockets_host = "mqtt.beldenserver.com"
    mqtt_websockets_port = 8083
    mqtt_websockets_ssl = 1
    mqtt_websockets_topic = "weather/loop"
    disconnect_live_website_visitor = 1800000

What next to check?

[Matt Johnson]

Thank you Doug. I just saw this after I was posted my last message, I'll take a look.

[Matt Johnson]

Your setup seems similar to mine so I went for it. Still not working.

I went ahead and added a mqtt CNAME record to the shakerweather.com cloudflare setup. I also added mqtt.shakerweather.com to my NGINX Proxy Server to forward to port 9001 using the same SSL certificate I have for shakerweather.com that is forwarding to port 80 to serve the page.

Cloudflare:

TYPE      NAME                         CONTENT

CNAME  mqtt                            shakerweather.com

A             shakerweather.com static wan IP

weewx.conf

[[MQTT]]
        server_url = mqtt://user:p...@10.0.0.30:1883/

        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate

skin.conf

     # MQTT Websockets defaults
    mqtt_websockets_enabled = 1

    mqtt_websockets_host = "mqtt.shakerweather.com"
    mqtt_websockets_port = 443

    mqtt_websockets_ssl = 1
    mqtt_websockets_topic = "weather/loop"
    disconnect_live_website_visitor = 1800000

mosquitto.conf

persistence false

# mqtt
listener 1883
protocol mqtt

# websockets
listener 9001
protocol websockets

allow_anonymous true
password_file /etc/mosquitto/passwd

acl_file /etc/mosquitto/acl

[Matt Johnson]

Doing more trouble shooting. I have narrowed it down to a SSL issue.

To confirm this I disabled SSL at cloudflare and disabled force SSL for shakerweather.com with my NGINX reverse proxy server and ran with this setup:

weewx.config

[[MQTT]]
        server_url = mqtt://usr:p...@mqtt.shakerweather.com:1883/

        topic = weather
        unit_system = US
        binding = archive, loop
        aggregation = aggregate

skin.conf

   # MQTT Websockets defaults
    mqtt_websockets_enabled = 1
    mqtt_websockets_host = "mqtt.shakerweather.com"

    mqtt_websockets_port = 9001

    mqtt_websockets_ssl = 0
    mqtt_websockets_topic = "weather/loop"
    disconnect_live_website_visitor = 1800000

mosquitto.conf

persistence false

# mqtt
listener 1883
protocol mqtt

# websockets
listener 9001
protocol websockets

allow_anonymous true
password_file /etc/mosquitto/passwd
acl_file /etc/mosquitto/acl

I am able to see the real time websocket updates internally and externally at http://shakerweather.com (I'll leave it up for a short time so if it doesn't work if you click on it that could be why)

So at least I have narrowed down the issue. Still not sure how to resolve to ensure all is secured with SSL....

[Doug Jenkins]

Matt:

I am looking into the issue now. I think you need to offload the SSL to Cloudflare with NGNIX Proxy. I am capturing screenshots of my setup to see if that will help you.

Doug Jenkins

[Doug Jenkins]

I just checked your site in edge and pulled up the dev console. I see the main issue is that the client (me) is trying to access a non-secure websocket (your server) while the rest of the content is coming from the https stream:

paho-mqtt.min.js:37 Mixed Content: The page at 'https://shakerweather.com/' was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint 'ws://mqtt.shakerweather.com:9001/mqtt'. This request has been blocked; this endpoint must be available over WSS.

I had this problem before and what I to solve it was to have CloudFlare to translate all my traffic to SSL. 

Here is my Cloudflare and NGINX Proxy Manager setup that I use for largoweather.com. Essentially if you check my SSL Cert, it is provided by Cloudflare. Cloudflare pushes all http traffic as https to my internal server. I have firewall rules to route all incoming 80/443 traffic to my internal server in which that is properly routed to my NGINX Proxy Manager instance.

NGINX Proxy manager proxies that traffic to the correct webserver (in my case container) that hosts my weewx website. I did issue a LetsEncrypt SSL within NGINX Proxy manager to get that to work.

They trick with LetsEncrypt is that you need to expose (just for a second) your public IP Address in Cloudflare (assuming that is your name server of the domain) for the SSL Certificate to be created correctly. Once you do that, you can setup Proxy again to protect your public IP.

Once this is setup, you will immediately see the live updates working. 

[Matt Johnson]

Thanks for this detail Doug, I will try again. Your analysis is spot on as I did disable SSL temporarily to see if the websockets would in fact work that way confiming it is an SSL issue. If you go to unsecured http (http://shakerweather.com) you should see it temporarily working with unsecured websocket updates (until I go back in and try to fix it the correct way with SSL this afternoon)

Light at the end of the tunnel. I will post later today when I either reach success or get stuck again.

[Les Niles]

I don’t see the point of SSL on the websockets connection from the browser. The data is all public, there’s nothing sensitive sent by the client, and if someone spoofs the mqtt broker, who really cares?

Neither do I see much need for ssl on the weewx-to-broker connection when they’re running on the same machine, or even on an internal network. I can imagine scenarios where not using ssl allows a vulnerability, but I think the likelihood is low if the internal network itself is well secured, so I haven’t bothered with ssl. It would be different with a cloud-hosted or other external mqtt broker with the weewx-mqtt connection running over a public network. 

  -Les

On Jul 24, 2021, at 9:33 AM, Matt Johnson <mattjo...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/507ba8a1-b0e8-4ea0-9037-0d6e7beda841n%40googlegroups.com.

[Karen K]

ln77 schrieb am Sonntag, 25. Juli 2021 um 03:35:38 UTC+2:

I don’t see the point of SSL on the websockets connection from the browser. The data is all public, there’s nothing sensitive sent by the client, and if someone spoofs the mqtt broker, who really cares?

The web browser cares. If it receives the website by HTTPS (SSL encrypted), then it refuses to connect to an unencrypted websockets server.

 

 

[Les Niles]

Ah, ok thanks. I’m running the web server unencrypted as well, for the same reason.  

  -Les

On Jul 25, 2021, at 4:14 AM, Karen K <kk44...@gmail.com> wrote:

ln77 schrieb am Sonntag, 25. Juli 2021 um 03:35:38 UTC+2:

I don’t see the point of SSL on the websockets connection from the browser. The data is all public, there’s nothing sensitive sent by the client, and if someone spoofs the mqtt broker, who really cares?

The web browser cares. If it receives the website by HTTPS (SSL encrypted), then it refuses to connect to an unencrypted websockets server.

 

 

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/08aa26d4-d986-4916-9c82-4c895c8b8fddn%40googlegroups.com.

[Matt Johnson]

Doug,

I went through your doc and updated everything at Cloudflare, my Nginx Proxy Server and conf files and not working still...not sure what else it could be.

[Matt Johnson]

Holy cow, no sooner did I send that did I see the glorious sight of "connected to weather station live" !!

Not sure what happened, maybe it just took a few minutes for Cloudflare to propagate the changes? In any case it is working and tested on LAN and WAN. I am so grateful for your support Doug!!!!

[vince]

Given the long back+forth on this one, it would be really helpful if you could capture your entire setup someplace for the next person/victim who tries to do the same thing.  You have quite a lot of moving parts and it would be great to capture everything you did to set it up so it works both LAN and WAN.

[Matt Johnson]

Vince,

I'd be happy to pull it all into one doc with some supporting notes so it makes sense. This was a frustrating (at times) yet rewarding experience to figure this all out and I know many others will want to achieve a similar website and control of their data.

Matt

[Doug Jenkins]

Matt:

for some reason I did not get the updates for this thread. I am glad the setup is now working for you! I checked your site and you do not have any javascript errors now :)


Vince : I agree, I will be more than happy to do a write up on how to configure this setup (with screenshots and all). I attached a word document to this thread a while back that listed what was needed in Cloudflare to offload SSL. I can do a formal write up with some commands and configuration from Pat O'Briens MQTT documentation. 

I can also post a message on Pat O'Briens discus chat on his weather site to see if he wants to add that as a post to his labs site (obrienlabs.net). That may help the majority of the skin users in setting this up on the first go.

Doug