'[Errno 101] Network is unreachable' with MQTT.Subscribe

[Stefan Gliessmann]

Hello -

I am running a mobile WeeWX station in my RV. I use MQTT to provide real-time weather data from the basic ecowitt gateway. I augment the weather data via MQTT every 5 min or so with data from OpenWeather. Additionally, I share my GPS coordinates and a calculated altitude.

WeeWX runs on a raspberry pi in my RV. Internet is provided by an internet router via 4G mobile phone network.

All this worked great as long as I was in Germany. The moment I crossed boarder to France and the mobile phone network switches to the other country, I receive this error:

""ERROR user.MQTTSubscribe: (Service) Failed to connect to wxvm.bz3gfkrlqtrsc3sv.myfritz.net at 1883. '[Errno 101] Network is unreachable'""

I am a bit puzzled why my home MQTT broker's port cannot be reached ...

Any clue how I can trouble shoot this?

Thank you guys in advance knowing that this isn't a core WeeWX problem :)

[p q]

Can you ping your mqtt broker?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/de797053-3552-46ed-8027-5ff8aef67210n%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[vince]

I can ping that FQDN from the USA if that matters.....

[Stefan Gliessmann]

Yes, I can ping it and it gets resolved correctly ...

My internet router in my RV forwards the GPS coordinates per MQTT to my home broker correctly, too ...

[vince]

I can subscribe to your data from the USA too so you must have some kind of networking thing going on that is preventing port 1883 in from wherever you are.

[Stefan Gliessmann]

Thanks, Vince!

I am puzzled what it could be blocking here in France compared when it worked just fine in Germany. (rv.stegli.de shows the last data before crossing over to France).

[Stefan Gliessmann]

Interestingly, rv.stegli.de sees the web socket of my home MQTT broker ...

[bell...@gmail.com]

A few things you can try. None of these will fix anything, but we get some more information…

1) Can you subscribe using mosquitto_sub?

2) Does MQTTSubscribe work if you use the IP address?

3) In MQTTSubscribeService section set log = true and in ‘main’ section set debug = 1. This should output the MQTT log. I doubt this will help any, because it appears the problem is on the connect. But worth a look.

rich

[bell...@gmail.com]

It is another interesting data point. A couple of things to note.

1. The connections are using two different protocols. Belchertown is connecting over websockets and MQTTSubscribe is connecting over MQTT.

2. It is two different MQTT clients. Belchertown is using a javascript client while MQTTSubscribe uses a python client.

So, it appears that the javascript client/websockets can handle the network change and the paho mqtt client/mqtt client cannot. Now to figure out why…

Out of curiosity, what version of the paho mqtt client are you using?

rich

On Monday 7 October 2024 at 15:42:20 UTC-4 Stefan Gliessmann wrote:

[vince]

Restart weewx and see if it suddenly works. Belchertown is a little unusual and sometimes seems to need that. 

[bell...@gmail.com]

Hmm… So you can publish to the MQTT broker? What are you using to publish?

Just looking for more breadcrumbs/clues.

rich

[Stefan Gliessmann]

Hi gents -

I did not know that they are using different clients, but I knew the difference in the port numbers 1883 vs 9001 ;)

My raspbi runs python3-paho-mqtt is already the newest version (1.6.1-1).

I started with restarting weewx, then mosquitto, then the Raspberry Pi and just for good measure also the internet router ... nothing resolved the issue.

In the meantime I arrived in Spain and I have still the same issue.

My GPS router is the Teltonika RUTX11 which runs BusyBox v1.34.1 The GPS coordinates are read from a GPS receiver connected to the router and send via "Modbus TCP Devices" to my home MQTT broker under the topic RVGPS ...

I will now start enabling log = true & debug = 1

Thanks for the input so far - I will keep you posted ...

Cheers

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/K8EABxk8R7o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3e1117e3-c4d6-4ac9-9613-707141f5d58bn%40googlegroups.com.

[Stefan Gliessmann]

Continuing to find some more details/hints/breadcrumbs ...

For MQTT.Subscribe I used the IP of the broker which gets resolved via ping. Still not reachable.

Then I set MQTT.Subcribe enable = false to see if the other client can connect to the same broker. and ... no ...

ERROR user.mqtt: Failed to connect to MQTT server (mqtt://teffi:x...@wxvm.bz3gfkrlqtrsc3sv.myfritz.net:1883/): [Errno 101] Network is unreachable

ERROR weewx.restx: MQTT: Failed to publish record 2024-10-08 21:18:25 CEST (1728415105): MQTT client not available

 
So it seems to be a "port 1883 issue", but where? Pi, the RV router, the home router, the VM on which the broker runs? (But why would it work in Germany just fine ...?) 

Can the internet provider limit port 1883? 

The SIM card in the RV router is from Vodafone Germany ...

In the meantime, I rebooted the ubuntu box with my home weewx on which also my MQTT broker resides, too. Still "Errno 101" 

That the MQTT broker works at home and receives the GPS topic from my RV router and republishes them under RVGPS can be seen below (for real-time weather data, the topic would be RV:

[bell...@gmail.com]

A few of questions/clarifications

So you RV Router can publish to your MQTT broker? Is this some type of built in functionality of the router?

This is the router that the PI is behind that cannot publish or subscribe?

Are you running MQTT Explorer on the PI. If so, it appears that can subscribe, correct?

The browser is also runing on the PI, correct?

Things are pointing to something in paho mqtt client, used by both MQTTSubscribe WeeWX-MQTT… Unfortunately I still don’t have any good ideas yet. I was really hoping that the IP address would work…

Note, 1.6.1 is not the latest version. The version installed by apt is behind. The version is up to 2.1.0 

BUT, I would be very hesitant to upgrade to that. 2.0 introduced many breaking changes and many python clients broke. In theory 2.1.0 ‘fixes’ this, but since I have updated all my MQTT clients to work with 2.x and 1.6, I cannot verify that.

AND, I do not see anything in the release notes that leads me to think this would be fixed.

rich

[Stefan Gliessmann]

So your RV Router can publish to your MQTT broker? Is this some type of built in functionality of the router?

  - Yes, it is. The RV Router has a ton of features including GPS and MQTT. For GPS I was able to configure "Modbus TCP Devices" (menu point of the RV Router) to publish lat & long per MQTT under topic GPS.

This is the router that the PI is behind that cannot publish or subscribe?

  - Yes, the weewx PI is behind the RV Router (Teltonika RUTX11 is the brand and model)

Are you running MQTT Explorer on the PI. If so, it appears that can subscribe, correct?

  - Ah, no! The MQTT Explorer runs on my MBP. My MBP is also "behind" the RV Router and can connect to the home MQTT broker!

The browser is also runing on the PI, correct?

  - Hmm, no ... You mean the web browser showing "Connected. Waiting for data ..."? No, that is again on my MBP connected to a web host in the cloud. There is - at least for me - no way to have a web server on the PI behind the RV Router which does not get a static IP over 4G ... The PI ftps to the remote (cloud) web server (the provider is strato.de - I use the same for my "home" WeeWX).

Next - probably tomorrow - I will try to VPN my RV Router to my home router via WireGuard ... Maybe that will solve it temporarily (while I am on holiday)!

[vince]

Confused.   I see weewx loop data every couple seconds...

$ mosquitto_sub -t 'Wetter/loop' -h wxvm.bz3gfkrlqtrsc3sv.myfritz.net
{"dateTime": "1728419863.0", "inTemp_C": "23.299999999999997", "outTemp_C": "14.9", "inHumidity": "49.0", "outHumidity": "94.0", "pressure_mbar": "993.4", "relbarometer_mbar": "996.2", "luminosity_lux": "0.0", "uvradiation_Wpm2": "0.2", "UV": "0.0", "soilMoist1_centibar": "17.0", "soilMoist2_centibar": "20.0", "soilMoist3_centibar": "20.0", "soilMoist4_centibar": "50.0", "lightning_distance_km": "0.0", "lightning_last_det_time": "1727356355.0", "lightningcount_count": "0.0", "lightning_strike_count_count": "0.0", "heap_free_byte": "28880.0", "rain_cm": "0.0", "stormRain_cm": "1.6500000000000001", "rainRate_cm_per_hour": "0.0", "dayRain_cm": "1.73", "weekRain_cm": "1.73", "monthRain_cm": "4.5200000000000005", "yearRain_cm": "77.59", "windDir": "135.0", "windSpeed_kph": "4.32001073732089", "windGust_kph": "7.200017895534816", "daymaxwind_kph": "21.96005458138119", "wh25_batt_count": "0.0", "wh65_batt_count": "0.0", "wh51_ch1_batt_volt": "1.5", "wh51_ch2_batt_volt": "1.5", "wh51_ch3_batt_volt": "1.5", "wh51_ch4_batt_volt": "1.5", "wh57_batt_count": "2.0", "wh25_sig_count": "4.0", "wh65_sig_count": "4.0", "wh51_ch1_sig_count": "4.0", "wh51_ch2_sig_count": "4.0", "wh51_ch3_sig_count": "4.0", "wh51_ch4_sig_count": "4.0", "wh57_sig_count": "4.0", "radiation_Wpm2": "0.0", "altimeter_mbar": "1002.8582768770635", "appTemp_C": "15.304960048993621", "barometer_mbar": "1002.747303342425", "cloudbase_meter": "198.40961163679043", "dewpoint_C": "13.941895630939834", "heatindex_C": "14.899999999999995", "humidex_C": "18.237831152343837", "inDewpoint_C": "11.976036849734465", "maxSolarRad_Wpm2": "0.0", "windchill_C": "14.9", "hourRain_cm": "0.0", "rain24_cm": "1.7299999999999955", "usUnits": "16.0"}

and I can see you moving.....

$ mosquitto_sub -t 'RVGPS/loop' -h wxvm.bz3gfkrlqtrsc3sv.myfritz.net
{"dateTime": "1728418872", "latitude": "43.367867", "longitude": "-3.032074"}
{"dateTime": "1728418876", "latitude": "43.367867", "longitude": "-3.032073"}
{"dateTime": "1728418882", "latitude": "43.367867", "longitude": "-3.032072"}

Is there any chance you're not getting to the websockets port 9001 ?

[Stefan Gliessmann]

😉

Under the topic Wetter one can read the weather at my home.

Under the topic RV would be the weather from the RV.

The RV router shares lat & long under topic GPS.

I needed to reformat the json payload and that one is republished under topic RVGPS.

Eventually, I should do a network diagram as from time to time, I am getting confused, too …

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/78f8ba90-64ee-49c8-87a2-0d243dba80ben%40googlegroups.com.

[bell...@gmail.com]

Well, the fact that MBP is behind the router and connect via websockets (Belchertown) and MQTT (MQTT Explorer) could connect to your broker, makes it seem like it is something with the PI or pahoo mqtt.

While not exact, this seems similar https://github.com/eclipse/paho.mqtt.python/issues/154

I''d try running mosquitto_sub from the PI and MBP and see if it works on both.

You could also try running the attached on both. From my understanding, it is used to get the IP to create the socket. I'm wondering if on the PI it is not returning a 'good' IP.

I'm definitely not a network guy and I'm wading deep into pahoo mqtt and the the python socket module. So, yes, I am throwing stuff against the wall to see what will stick...

rich

[Stefan Gliessmann]

Hello Rich et al.,

I was a bit busy exploring Spain and, so, did not do a lot of trouble shooting on this issue ...

Well - I still have the same issue:

When I run the little socket info py, I get this:

which looks good to me ... I am continued to be puzzled ...

Please throw more stuff at me to see if anything might stick and we have an inlay to the problem ...

Cheers,

Stefan

[bell...@gmail.com]

Stefan, 

Here are a couple of more things you could try.

First, try subscribing via mosquitto_sub to via the host name and the IPV4 and IPV6 addresses.

mosquitto_sub  -v -t '#' -h wxvm.bz3gfkrlqtrsc3sv.myfritz.net

mosquitto_sub  -v -t '#' -h 77.21.85.171
mosquitto_sub  -v -t '#' -h 2a02:8108:9340:757c:efa5:3c2b:a158:b1d9

You could also try to run the attached  (python3 connect.py). If I have read the paho mqtt and socket code correctly, this should simulate what it does to establish a connection.

rich

[Stefan Gliessmann]

Rick -

I finally tried the mosquitto_sub ... here are the results:

teffi@rvwx:~ $ sudo systemctl status weewx.service 

● weewx.service - WeeWX

     Loaded: loaded (/lib/systemd/system/weewx.service; enabled; preset: enabled)

     Active: active (running) since Tue 2024-10-15 11:28:28 CEST; 1 day 9h ago

       Docs: https://weewx.com/docs

   Main PID: 799 (python3)

      Tasks: 1 (limit: 3909)

        CPU: 1min 23.979s

     CGroup: /system.slice/weewx.service

             └─799 python3 /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf

Oct 16 20:34:21 rvwx weewxd[799]: INFO user.MQTTSubscribe: (Service) min_delay is 1

Oct 16 20:34:21 rvwx weewxd[799]: INFO user.MQTTSubscribe: (Service) max_delay is 120

Oct 16 20:34:21 rvwx weewxd[799]: INFO user.MQTTSubscribe: (Service) password is set

Oct 16 20:34:21 rvwx weewxd[799]: INFO user.MQTTSubscribe: (Service) Archive topic is None

Oct 16 20:34:21 rvwx weewxd[799]: DEBUG user.gw1000: Next update in 5 seconds

Oct 16 20:34:26 rvwx weewxd[799]: ERROR user.MQTTSubscribe: (Service) Failed to connect to wxvm.bz3gfkrlqtrsc3sv.myfritz.net at 1883. '[Errno 101] Network is unreachable'

Oct 16 20:34:26 rvwx weewxd[799]: DEBUG user.gw1000: Next update in 5 seconds

Oct 16 20:34:27 rvwx weewxd[799]: INFO user.gw1000: GatewayCollector thread has been terminated

Oct 16 20:34:27 rvwx weewxd[799]: CRITICAL __main__: Caught WeeWxIOError: [Errno 101] Network is unreachable

Oct 16 20:34:27 rvwx weewxd[799]: CRITICAL __main__:     ****  Waiting 60.0 seconds then retrying...

teffi@rvwx:~ $ mosquitto_sub  -v -t '#' -h wxvm.bz3gfkrlqtrsc3sv.myfritz.net

^C

teffi@rvwx:~ $ mosquitto_sub  -v -t '#' -h 77.21.85.171

^C

teffi@rvwx:~ $ mosquitto_sub  -v -t '#' -h 2a02:8108:9340:757c:efa5:3c2b:a158:b1d9

Error: Network is unreachable

teffi@rvwx:~ $ sudo python3 getinfo.py 

(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('77.21.85.171', 1883))

(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('77.21.85.171', 1883))

(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('77.21.85.171', 1883))

(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2a02:8108:9340:757c:efa5:3c2b:a158:b1d9', 1883, 0, 0))

(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('2a02:8108:9340:757c:efa5:3c2b:a158:b1d9', 1883, 0, 0))

(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_RAW: 3>, 0, '', ('2a02:8108:9340:757c:efa5:3c2b:a158:b1d9', 1883, 0, 0))

So, only for the IPv6 Address I got the Error:Network is unreachable.

Host name and IPv4 did not do anything and I needed to cancel out.

Tomorrow, I am back in Germany and I am wondering if it will work again ...

Cheers,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6d9f4c63-2cf8-46b5-be4e-c1bcda193b3an%40googlegroups.com.

[Stefan Gliessmann]

and the output from the connect.py:

teffi@rvwx:~ $ sudo python3 connect.py

(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('77.21.85.171', 1883))

  ERROR: timed out

(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2a02:8108:9340:757c:efa5:3c2b:a158:b1d9', 1883, 0, 0))

  ERROR: [Errno 101] Network is unreachable

On Tue, Oct 15, 2024 at 7:22 PM bell...@gmail.com <bell...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6d9f4c63-2cf8-46b5-be4e-c1bcda193b3an%40googlegroups.com.

[Stefan Gliessmann]

It must be a Raspberry Pi network thingy ... I will trouble shoot is further along this https://forum.iobroker.net/topic/69165/raspberry-networkmanager-keine-ipv6-verbindung-von-extern/10

On Tue, Oct 15, 2024 at 7:22 PM bell...@gmail.com <bell...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6d9f4c63-2cf8-46b5-be4e-c1bcda193b3an%40googlegroups.com.

[Stefan Gliessmann]

Well - it must be a raspberry bookworm networking thingy. However, not the one I linked in my previous post.

Today, I used ethernet rather wifi connection. And - now - my home broker can be reached again ... Really odd!

Even crazier than that, the home broker was also reachable for several hours once I returned back to Germany / German 4G Coverage ...

I hope, the ethernet cable is really the solution and not just a temporary ...

Thank you guys, we helping me along on this odd issue ...