Best way to publish live Belchertown updates without port forwarding?

[Anton Diedericks]

Hi all,

I have WeeWX 5.3.1 running successfully on a Raspberry Pi with:

Davis Vantage Pro2 console
WeatherLinkIP data logger
Raspberry Pi
Raspbian GNU/Linux 13 (Trixie)
Belchertown skin (1.7 beta2)
Nginx
Git + GitHub Pages

Everything is working correctly on my local LAN with local IP 192.168.0.101

Belchertown updates correctly locally.

Currently I publish my weather site to GitHub Pages using a shell script that runs every minute via cron. The script stages changes, amends the previous commit, force pushes to GitHub, and GitHub Pages serves the site using a CNAME for:

www.woodlands-weather.co.za

This works, but there is still a logically delay before updates appear publicly.

My goal is near-live updates.

Unfortunately my ISP (Vodacom LTE) uses CGNAT and does not allow port forwarding, so direct hosting from the Pi via normal router forwarding is not possible.

My questions:

Is GitHub Pages fundamentally the wrong approach for live weather updates?
Would Cloudflare Tunnel be the recommended solution in this case?
Is there a better WeeWX approach for near real-time public updates when port forwarding is not available?

To set all of this up, was a lot of very late nights. My knowledge is very very limited. At least my site is live.
I have limited Linux/networking knowledge, so I would appreciate advice on the simplest and most reliable long-term setup.

Thanks
Anton

[Greg Troxel]

Anton Diedericks <voet...@gmail.com> writes:

> My goal is near-live updates.

That statement is too vague. I think you mean that you want a web page
that updates with several-second latency as loop packets happen.

> Is GitHub Pages fundamentally the wrong approach for live weather updates?

Yes. Github pages is the wrong approach for anything that changes
often. If I were github I would characterize what you are doing as
abuse, but I have no idea what github would think, were they to think
(instead of just telling everybody to use more AI).

> Would Cloudflare Tunnel be the recommended solution in this case?
> Is there a better WeeWX approach for near real-time public updates when
> port forwarding is not available?

This isn't really about weewx; it's about how to have a server on the
internet when you don't have a server on the internet :-) (Not saying
it's off topic, just that it's separable.)

There are basically two paths. One is to have a computer that is
actually on the internet, typically a virtual private server (VPS). The
other is to have someone accept connections to some name/address/port
and let you connect out.

Personally, I lean to the VPS approach, but that may be more work,
require more knowledge, and some expense. I do have the impression that
people use Cloudflare Tunnel.

[John Smith]

If you want a mqtt type service without the port forward you probably need to look into websockets.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/bb0819f6-5d6a-48f0-a274-a4036ab3f56en%40googlegroups.com.

[Vince Skahan]

Weewx+Belchertown websockets is the usual solution for realtime updates.

The github pages thing is pretty creative but more complicated than it needs to be.

The simplest way to get an internet site is to spin up a tiny VM there and have weewx rsync the web pages when the reports run.  That'll get you an internet-accessible site with updates as often as your weewx archive interval (typically 5 minutes).  I've used the smallest possible AWS Lightsail VM for years.  Cost is a few $US per month.  Basically hands-off.  All that runs there is a nginx webserver.  Mine's been up since literally the day they announced Lightsail in late 2016.

Getting Belchertown realtime updates working on LAN is easy.  Getting it working both LAN and Internet is a little harder.  I did find one way to do so last week in revisiting how to do this without permitting any incoming Internet traffic to my LAN.....

• In brief - simply run the Mosquitto MQTT broker on the Internet VM.  Point weewx + belchertown to there.  The result is that your LAN weewx instance will MQTT publish to your internet broker and the generated web pages wherever they are will look to the Internet broker for the realtime updates via websockets.  Works fine for accessing the LAN pages (when home) and also when accessing the Internet pages.

Usual best practices for running an Internet site apply:

• install the minimum software - ssh for management and rsync, webserver (I use nginx), add mosquitto-MQTT if you want websockets
• listen on the minimum ports for each service
• ideally use https for your web. Adding LetsEncrypt certs works great.  They renew every 60 days or so.
• harden your mosquitto broker. At a minimum add ACLs there to limit read/write access so only your LAN weewx instance can publish to MQTT.  If you want to 'also' put the MQTT broker behind encryption, the same LetsEncrypt certs work for mosquitto MQTT.

Be aware that once you put a webserver on Internet the bots and scanners will attack you almost instantly. As long as you have a weewx-only web there with 'no' PHP or WordPress or CGI, those attacks are generally just noise.

• If you're non-websockets just put your site behind Cloudflare proxies and them do the blocking.  Works great.
• If you need websockets and are willing to handle the blocking yourself, add fail2ban with some one-time manual setup to block the usual web probes.   I got it down to just a few attacks per day even getting to the nginx server (and being blocked immediately) by tweaking my custom block rules but it did take quite a bit of fiddling to get there.  Not too simple.

I did not find a solution where Cloudflare handled proxying my AWS web that permitted websockets to work.

I ran Belchertown+websockets for a few days and it worked fine, but I eventually reverted to non-websockets and letting Cloudflare block the bots and probes.  The realtime spinning of the wind gauge in Belchertown wasn't worth the complexity for me, but it is pretty cool....

[Anton Diedericks]

Hi all,

Following up on my original question about running WeeWX behind NAT (LTE, no port forwarding), I now have a fully working setup that may help others.

Working stack
WeeWX on Raspberry Pi
Mosquitto for MQTT + WebSockets
Belchertown skin with live updates enabled
Cloudflare Tunnel for public access
Nginx for web serving + routing

Once MQTT WebSockets, Mosquitto, and Cloudflare Tunnel ingress were correctly aligned, Belchertown live updates (wind gauge, real-time data) worked reliably through port 443.

The main challenge was not WeeWX itself, but correctly matching:

MQTT topics
WebSocket listener (Mosquitto)
Cloudflare tunnel hostname routing
Belchertown MQTT settings

Thanks to Vince Skahan and John Smith for pointing me toward the MQTT broker approach. Also to Greg Troxel for steering me away from publishing to github.  
In my case, I was able to get real-time updates working cleanly through Cloudflare Tunnel using MQTT WebSockets.

For anyone interested, I’ve documented the full installation and working configuration.
Installation Guide is linked directly from the header of my weather site.
https://www.woodlands-weather.co.za/weewx/belchertown/

This includes full step-by-step setup for WeeWX, Belchertown, MQTT, Mosquitto, Nginx, and Cloudflare Tunnel.

Happy to share config snippets if anyone wants to replicate the setup.

Regards,
Anton
PS: Yes, AI helped me a lot.

[Vince Skahan]

Try-2 - google ate my reply.

Anton - thanks much !!!

Your instructions confused me a bit re: DNS and how the tunnel is created.  I couldn't get there via your CLI instructions

I added a tunnel by going through their gui and had to add a route of type 'published application' to localhost port 80, but didn't find a way to make the mqtt hostname you're using get created nor to get 9001 to tunnel in. I must be missing something in reading your instructions....

What DNS records do you have for your web hostname and mqtt hostname ?

Also - you're missing installing mosquitto and paho-mqtt in your instructions.  I think I had to manually create /etc/cloudflared as well but I went through so many iterations I kinda lost track there.

[John Smith]

What DNS records do you have for your web hostname and mqtt hostname ?

Also - you're missing installing mosquitto and paho-mqtt in your instructions.  I think I had to manually create /etc/cloudflared as well but I went through so many iterations I kinda lost track there.

Don't worry about the DNS side of things if that's already working, just add another published route and that section will make prompt to make the DNS entry if one doesn't exist.

Basically you just point it to the websocket port set in mosquitto config, similar to what you did for the website, even if mosquitto isn't setup with a ssl cert you can still access it using a wss url.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/81bb0216-1be9-4301-9ea4-3702a66592fbn%40googlegroups.com.

[Vince Skahan]

Tried that, but doesn't work.

I have two routes both pointing to the tunnel name:

• 'weetest' points to http://localhost:80
• 'mqtt' points to http://localhost:9001
• mosquitto is listening for mqtt on 1883 and websockets on 9001
• belchertown is configured for websockets on 9001
• I have verified with MQTT Explorer the broker works ok LAN-only both MQTT and Websockets ports
• I have verified with Belchertown that if I use a browser LAN-only to the webserver page then websockets works
• but nothing I can do can get websockets to work to the Internet-facing FQDN

With Belchertown mqtt_websockets_enabled=0 - I quickly get 'WebSocket connection to 'ws://mqtt.mydomain.here:8888/mqtt' failed: A server with the specified hostname could not be found'   in the javascript console.

With Belchertown mqtt_websockets_enabled=1  - I very slowly get a 403 trying to open homepage.json and then MQTT Cannot connect to MQTT broker message in the javascript console.  WebSocket connection to 'wss://mqtt.mydomain.here:8883/mqtt' failed: The network connection was lost.

In both cases I see nothing in the mosquitto logs indicating anything came in to the mosquitto broker at all through the tunnel.

So my reading is the DNS-side of the tunnel for websockets is what's not set up correctly or Cloudflare just doesn't support what I'm trying to do.  Their documentation is so awful I'm getting nowhere trying there, and google searches etc. are not helpful.  Many many people having the same problem.

[John Smith]

Try pinging the hostname, if that fails you go into the domain and click on DNS and then click on records.

It's been updated recently and is more useful now as it shows the tunnel for hostnames routed via tunnels.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/f5ca8577-7108-4909-93c1-ff5d3b0e4684n%40googlegroups.com.

[Anton Diedericks]

Hi Vince,

Apologies for the late reply.

Logging in to https://dash.cloudflare.com, I have this.
DNS management for woodlands-weather.co.za
Search DNS Records
Your search yielded 2 results
Type | Name | Content | Proxy status | TTL
Tunnel |  mqtt |  weewx-mqtt |  Proxied |  Auto
Tunnel |  www |  weewx-mqtt |  Proxied |  Auto

Cloudflare Nameservers
Type Value
NS |  abc.ns.cloudflare.com
NS |  xyz.ns.cloudflare.com

Did Cloudflare issue you with a certificate?

cloudflared tunnel login

Hope that helps. This is above my paygrade.

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/S6z1oPAgj-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/weewx-user/CAGTinV4jxWt6_n0sdqd5EEMEsu-JPTCttJK0pZdHYwiUArwt1A%40mail.gmail.com.

--

Kind regards
Anton Diedericks

[Vince Skahan]

Thanks.  My DNS setup matches.  I'm leaning toward the certificate being the problem area here for the websockets part.  I'll tear it all down and go through it again step by step nice and slowly.  Stay tuned.

[Vince Skahan]

Got it.  Finally.  I feel like Ahab searching for the whale.

Thanks guys !!!!

I'll capture all the files that worked here this weekend and post a followup.  Some quick notes from re-re-re-re-reading Anton's webpage procedure that seems to have disappeared.

(Anton - please link it into your about.inc file in belchertown so it's available - it was very very helpful)

Some key things I finally figured out:

• you need to use the CLI to do this setup so it will create the mqtt route to ws://localhost:9001 which is impossible to create from the Cloudflare GUI.
• also after reading it a hundred times I noticed the belchertown setup in weewx.conf has to point to a websockets port of 443 which seems to be the key.  I hadn't noticed that before and was using the actual 9001 port like when you set up belchertown LAN-only without certs.
• procedurally Anton's procedure is really close to what I did.  I'll have to revisit my shell history and see if I can find any omissions there

Thanks again Anton especially.  So cool being able to get this working 'finally' after many dozen hours battling it over the years.

[John Smith]

• you need to use the CLI to do this setup so it will create the mqtt route to ws://localhost:9001 which is impossible to create from the Cloudflare GUI.

It's very possible, you just add it as a published HTTP route, similar to what you did for your website.

• also after reading it a hundred times I noticed the belchertown setup in weewx.conf has to point to a websockets port of 443 which seems to be the key.  I hadn't noticed that before and was using the actual 9001 port like when you set up belchertown LAN-only without certs.

That's because Cloudflare only proxy 80/443 and use the hostname in the HTTP headers to work out where packets need to end up.

You can do raw TCP connections by using the cloudflared program on the client end which then turns packets into essentially HTTP or pay a lot of money to Cloudflare for them to route custom ports on an enterprise plan.

That's how SSH over Cloudflare works, the ssh program calls the cloudflares program before connecting to whatever hostname.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/5d635bcd-8c8f-46dc-94f8-c101b74ebafen%40googlegroups.com.

[John Smith]

ws = http

wss = https

[Vince Skahan]

I added a page to my https://github.com/vinceskahan/belchertown-from-scratch repo on github with the procedure I wound up with after more testing....

https://github.com/vinceskahan/belchertown-from-scratch/blob/main/configure-cloudflare-tunnel.md

Thanks again Anton !!!