MQTT Connection Refused

[Jeff Beckman]

I'm having trouble getting data into Home Assistant via MQTT.  I've connected to Home Assistant with MQTT Explorer and been able to publish to a test topic, so that end of things appears to be working.  

I did my best to follow the instructions at https://github.com/weewx/weewx/wiki/mqtt, skipping the python-cjson since weewx is using python 3 (via Deb package).  The MQTT portion of my weewx.config looks like this in my latest attempt:

    [[MQTT]]

        server_url = mqtt://192.168.1.35:1883

        topic = weather

        unit_system=METRIC

        [[[inputs]]]

            [[[[inTemp]]]]

                units = degree_F

Initially I was trying with server_url = mqtt://user:pas...@192.168.1.35:1883, but recently enabled anonymous connections in case a password typo was the problem.  

With debug = 1 my syslog seems to indicate the problem is possibly with authentication (connection refused).  Any ideas what I'm doing wrong.

Jan 25 20:39:49 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: Found station at bus= device=

Jan 25 20:39:49 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: next read in 18 seconds

Jan 25 20:40:01 raspberrypi CRON[1952]: (pi) CMD (python3 /home/pi/cron_jobs/humid_checker.py > /dev/null)

Jan 25 20:40:07 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: Found station at bus= device=

Jan 25 20:40:07 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: next read in 12 seconds

Jan 25 20:40:19 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: Found station at bus= device=

Jan 25 20:40:19 raspberrypi /weewxd: inside humid: found value of 53.2

Jan 25 20:40:19 raspberrypi weewx[29660] INFO weewx.manager: Added record 2021-01-25 20:40:00 MST (1611632400) to database 'weewx.sdb'

Jan 25 20:40:19 raspberrypi weewx[29660] INFO weewx.manager: Added record 2021-01-25 20:40:00 MST (1611632400) to daily summary in 'weewx.sdb'

Jan 25 20:40:22 raspberrypi weewx[29660] DEBUG weewx.reportengine: Running reports for latest time in the database.

Jan 25 20:40:22 raspberrypi weewx[29660] DEBUG weewx.reportengine: Running report 'StandardReport'

Jan 25 20:40:22 raspberrypi weewx[29660] DEBUG user.mqtt: Failed upload attempt 1: [Errno 111] Connection refused

Jan 25 20:40:23 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: Found station at bus= device=

Jan 25 20:40:23 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: next read in 6 seconds

Jan 25 20:40:23 raspberrypi weewx[29660] DEBUG weewx.reportengine: Found configuration file /etc/weewx/skins/Seasons/skin.conf for report 'StandardReport'

Jan 25 20:40:23 raspberrypi weewx[29660] DEBUG weewx.cheetahgenerator: Using search list ['weewx.cheetahgenerator.Almanac', 'weewx.cheetahgenerator.Station', 'weewx.cheetahgenerator.Current', 'weewx.cheetahgenerator.Stats', 'weewx.cheetahgenerator.UnitInfo', 'weewx.cheetahgenerator.Extras']

Jan 25 20:40:23 raspberrypi weewx[29660] DEBUG weewx.manager: Daily summary version is 3.0

Jan 25 20:40:27 raspberrypi weewx[29660] DEBUG user.mqtt: Failed upload attempt 2: [Errno 111] Connection refused

Jan 25 20:40:29 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: Found station at bus= device=

Jan 25 20:40:29 raspberrypi weewx[29660] DEBUG weewx.drivers.acurite: next read in 18 seconds

Jan 25 20:40:32 raspberrypi weewx[29660] DEBUG user.mqtt: Failed upload attempt 3: [Errno 111] Connection refused

Jan 25 20:40:35 raspberrypi weewx[29660] INFO weewx.cheetahgenerator: Generated 8 files for report StandardReport in 12.36 seconds

Jan 25 20:40:35 raspberrypi weewx[29660] DEBUG weewx.manager: Daily summary version is 3.0

Jan 25 20:40:37 raspberrypi weewx[29660] ERROR weewx.restx: MQTT: Failed to publish record 2021-01-25 20:40:00 MST (1611632400): Failed upload after 3 tries

Jan 25 20:40:42 raspberrypi weewx[29660] INFO weewx.imagegenerator: Generated 14 images for report StandardReport in 6.25 seconds

Jan 25 20:40:42 raspberrypi weewx[29660] INFO weewx.reportengine: Copied 0 files to /var/www/html/weewx

[vince]

Connection refused usually means the other side literally not listening on that port.  You might want to check that 192.168.1.35 is listening on port 1883.  That should be the ip address of your MQTT broker, not the ip address of the weewx system that's trying to connect to it.

Here's me checking my MQTT server pi

pi@pi3:~ $ sudo ss -tupl|grep 1883

tcp   LISTEN 0      100                         0.0.0.0:1883            0.0.0.0:*                      users:(("mosquitto",pid=607,fd=5))

tcp   LISTEN 0      100                            [::]:1883               [::]:*                      users:(("mosquitto",pid=607,fd=6))

...and checking that process 607 is the MQTT server...

pi@pi3:~ $ ps -aux|grep 607

mosquit+   607  0.0  0.5   8924  5128 ?        Ss   Jan12   7:49 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Quick test with mosquitto is to get on a client computer and run:

mosquitto_sub -h 192.168.1.35 -t testingmqtt

And in another window send a message to that topic:

mosquitto_pub -h 192.168.35 -t testingmqtt -m hellothere

(and the first window that's subscribing should display the 'hellothere' message the other window sent to the broker)

[Jeff Beckman]

OK, that produced some interesting results.  For your first suggestion, my broker computer is running Hass.io (Home Assistant), which doesn't have the ss command.  I'll keep digging there to see if there's another way to double check that the port is open, but the log seems to indicate it's listening on 1883.  

I installed  mosquitto-clients on a raspberry pi that I'm not using for Weewx or my broker.  So far everything I try there results in the same connection refused error.  I've tried both as anonymous and as a user.  Here's an example of an attempt:

pi@piopolis:~ $ mosquitto_sub -h 192.168.1.35 -t test -u homeassistant -P reallylongpassword

Error: Connection refused

Checking the log on the broker, I see no evidence of the attempt, which would point to the port not actually being open.  But then I try to connect from my windows machine using MQTT Explorer and everything works.  I can publish messages and see them pop up on my broker machine.  The log looks normal.  

MQTT Explorer setup:

Logs from mosquitto broker:

[22:30:06] INFO: Setup mosquitto configuration 

[22:30:06] WARNING: SSL not enabled - No valid certs found! 

[22:30:06] INFO: No local user available 

[22:30:07] INFO: Initialize Hass.io Add-on services 

[22:30:07] INFO: Initialize Home Assistant discovery [22:30:07] INFO: Start Mosquitto daemon 

1611639007: mosquitto version 1.6.3 starting 

1611639007: Config loaded from /etc/mosquitto.conf. 

1611639007: Loading plugin: /usr/share/mosquitto/auth-plug.so 

1611639007: |-- *** auth-plug: startup 

1611639007: ├── Username/password checking enabled. 

1611639007: ├── TLS-PSK checking enabled. 

1611639007: └── Extended authentication not enabled. 

1611639007: Opening ipv4 listen socket on port 1883. 

1611639007: Opening ipv6 listen socket on port 1883. 

1611639007: Opening websockets listen socket on port 1884. 

1611639007: Warning: Mosquitto should not be run as root/administrator. 

1611639012: New connection from 172.30.32.1 on port 1883. [INFO] found homeassistant on local database 

1611639012: New client connected from 172.30.32.1 as 5yoX9kmQROmBbxG2B0yAct (p2, c1, k60, u'homeassistant'). 

1611639366: New connection from 192.168.1.185 on port 1883. [INFO] found homeassistant on local database 

1611639366: New client connected from 192.168.1.185 as mqtt-explorer-a18267b6 (p2, c1, k60, u'homeassistant').  

[Greg Troxel]

Jeff Beckman <jeffrey...@gmail.com> writes:

> OK, that produced some interesting results. For your first suggestion, my
> broker computer is running Hass.io (Home Assistant), which doesn't have the
> ss command. I'll keep digging there to see if there's another way to
> double check that the port is open, but the log seems to indicate it's
> listening on 1883.
>
> I installed mosquitto-clients on a raspberry pi that I'm not using for
> Weewx or my broker. So far everything I try there results in the same
> connection refused error. I've tried both as anonymous and as a user.
> Here's an example of an attempt:
> pi@piopolis:~ $ mosquitto_sub -h 192.168.1.35 -t test -u homeassistant -P
> reallylongpassword
> Error: Connection refused

That's great progress; you have shown that your MQTT setup is troubled
without even involving weewx.

It's time to:

check the config file for the broker to see if it is listening on
localhost only (which doesn't make a lot of sense) or to *:* or to
that address specifically. Setup questions for a hass.io broker are
probably best posed to some HA forum.

check all your firewall configurations. Firewalls are notorious for
breaking intended communication, and the fun you have fixing that is
the tradeoff for whatever protection you get. I would do this by:

reading the firewall configs

querying the firewall stats and logs of filtered packets

running tcpdump on both the weewx computer and the hass.io computer.
If either is some kind of container, also add the computer hosting
that container.

> Checking the log on the broker, I see no evidence of the attempt, which
> would point to the port not actually being open. But then I try to connect

No, it points to the operating system not doing all of receiving a SYN and
successfully completing the TCP 3-way handshake. Brokers more or less
all just use accept(2) to listen for connections and don't process TCP.

> from my windows machine using MQTT Explorer and everything works. I can
> publish messages and see them pop up on my broker machine. The log looks
> normal.

That's an intersting clue that perhaps the problem is a firewall that
affects the weewx machine and not the windows machine.

You might also try basic things like ping of the broker from the weewx machine.

> 1611639012: New connection from 172.30.32.1 on port 1883. [INFO] found
> homeassistant on local database
> 1611639012: New client connected from 172.30.32.1 as 5yoX9kmQROmBbxG2B0yAct
> (p2, c1, k60, u'homeassistant').
> 1611639366: New connection from 192.168.1.185 on port 1883. [INFO] found
> homeassistant on local database
> 1611639366: New client connected from 192.168.1.185 as
> mqtt-explorer-a18267b6 (p2, c1, k60, u'homeassistant').

That certainly looks like it is accepting connections from off machine.

An interesting mix of RFC1918 addresses, but maybe that's intended.

Make sure your entire home LAN setup has a consistent addressing plan,
with no reuse.

[Jeff Beckman]

I feel like I might not get much sleep this week, this is driving me nuts.  I installed linux on a VM on my desktop (the one that has working MQTT Explorer) and it worked.  The default (oracle virtualbox) was to just attach to my NAT, so it got an IP of 10.0.2.15 (which my desktop passes through to 192.168.1.185?).  Changing it to a bridge adapter, it picked up a DHCP address of 192.168.1.218, and then it did not work.

Here's my full setup:  Router is a stand-alone box running pfsense.  I have a server that runs freebsd doing virtualization (static, 192.168.1.30).  Home Assistant is a VM (static, 192.168.1.35).  Weewx is a raspberry pi 2 (static, 192.168.1.17).  Desktop windows machine (DHCP, 192.168.1.185).

I have another unrelated VM running on that free bsd machine for a minecraft server (static, 192.168.1.36).  Remoting in there gives the same problems trying to publish a message.  This machine's network setup is almost identical to the Home Assistant machine and it is able to accept connections on its minecraft port, 1935 or something.  

I tried moving the Weewx machine to DHCP since it's an easy test, but no change.  

So the working tests are: DHCP on 192.168.1.185, windows and linux mint, MQTT Explorer and mosquitto-clients

The non-working tests are: static and dynamic on 192.168.1.*, linux mint and raspbian, mosquitto-clients

So the answer may be that I should be asking in a home networking forum, or possibly a Hass.io forum.  Maybe I'll do some more tests from other machines to try and gather some more information.  

[Greg Troxel]

Jeff Beckman <jeffrey...@gmail.com> writes:

> I feel like I might not get much sleep this week, this is driving me nuts.
> I installed linux on a VM on my desktop (the one that has working MQTT
> Explorer) and it worked. The default (oracle virtualbox) was to just
> attach to my NAT, so it got an IP of 10.0.2.15 (which my desktop passes
> through to 192.168.1.185?). Changing it to a bridge adapter, it picked up
> a DHCP address of 192.168.1.218, and then it did not work.

Unless you really know what you are doing, it seems unlikely that having
10.0.2.15 and 192.168.1.85 on the same LAN is sensible. Generally each
Ethernet, real or virtual, should have one prefix and at most one DHCP
server. Once there is address confusion, i suspect routing confusion.

My advice is to first completely straighten out your home LANs
addressing plan.

> Here's my full setup: Router is a stand-alone box running pfsense. I have
> a server that runs freebsd doing virtualization (static, 192.168.1.30).
> Home Assistant is a VM (static, 192.168.1.35). Weewx is a raspberry pi 2
> (static, 192.168.1.17). Desktop windows machine (DHCP, 192.168.1.185).
>
> I have another unrelated VM running on that free bsd machine for a
> minecraft server (static, 192.168.1.36). Remoting in there gives the same
> problems trying to publish a message. This machine's network setup is
> almost identical to the Home Assistant machine and it is able to accept
> connections on its minecraft port, 1935 or something.

So where is 10/8? That's a bug, IMHO, and perhaps related, perhaps not.

[vince]

10.0.2.15 is the default host-only address you get in VirtualBox with the default NAT out to your LAN.   You likely need to expose port 1883 on the outside (192.168.1.x side) so that something can answer the phone, so to speak. 

It would be helpful if you'd draw us a network picture with which systems+VMs are where and what ip addresses are on which interface.

One howto for VirtualBox is at https://www.howtogeek.com/122641/how-to-forward-ports-to-a-virtual-machine-and-use-it-as-a-server/ - ignore the bridged example and check out how to do the port forwarding in the NAT'd example.   I'm guessing there's a way to do the same thing without a gui on the VM but I never do native VirtualBox any more, I always use Vagrant as a front end and then it's trivial to expose ports through the underlying NAT'd VM.   One example for ubuntu is at https://github.com/vinceskahan/weewx-vagrant/tree/master/pkg/ubuntu2004 if you wanted to see the Vagrantfile there.   The provision.sh script will install a Simulator weewx and nginx and expose the web on port 8105.

Short answer is that if you use a default VirtualBox setup then you'll be NAT'ing and you'll need to do port forwarding in VirtualBox so that port 1883 on your host (incoming) gets to port 1883 in the running VM on that host.

I would recommend not doing a bridged VM for security reasons.  Just expose a port and it'll work.  Really.

[Jeff Beckman]

After much searching I've found the problem.  There seems to have been a problem with a GS110EMX switch on my network that was preventing MQTT traffic from getting through, but only from IP addresses that were originally requested from Linux machines.  I'm not sure how that's possible, but that seems to have been the problem.  After a firmware update on the switch everything works normally.