Belchertown Skin and MQTT With Username Not Working

[G Hammer]

Does anyone have the Belchertown skin working with MQTT using a server that requires a username and password such as CloudMQTT?

I have tried several different ways of configuring the skin and it fails to connect or it shows 'Connecting to weather station real time data' forever without connecting.

The data is being sent to the server fine and I have subscribed to it using client software (see below).

Thanks for any input, I'm at a loss here.

[Colin Larsen]

Not working yet as far as I know.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Philip Kutzenco]

I have it working on my own externally hosted Mosquitto server (on Digital Ocean). My Mosquitto MQTT broker is set up requiring a username and password for publishing. Additionally it has TLS/SSL implemented (with Let's Encrypt certificates). It allows subscribing anonymously and also runs Websockets so that it can feedthe Belchertown skin. I used Pat's MQTT "tutorial" to do this. My website is https://wx.kutzenco.com.

phil

[Gary Hammer]

Do you connect the client (skin) via websockets or any other way using a username and password?

That is the question.

 

From: weewx...@googlegroups.com <weewx...@googlegroups.com> On Behalf Of Philip Kutzenco
Sent: Monday, October 8, 2018 3:32 PM
To: weewx-user <weewx...@googlegroups.com>
Subject: [weewx-user] Re: Belchertown Skin and MQTT With Username Not Working

 

I have it working on my own externally hosted Mosquitto server (on Digital Ocean). My Mosquiutto MQTT broker is set up requiring a username and password for publishing. Additionally it has TLS/SSL implemented (with Let's Encrypt certificates). It allows subscribing anonymously and also runs Websockets so that it can feedthe Belchertown skin. I used Pat's MQTT "tutorial" to do this. My website is https://wx.kutzenco.com.

 

phil

On Monday, October 8, 2018 at 10:21:50 AM UTC-4, G Hammer wrote:

 

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/5Qn_6oZjLP4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[Philip Kutzenco]

The answer to that question is no. I've attached a file with sanitized excerpts of my weewx.conf file showing the stanzas related to:

1. MQTT (publishing) - which specifies a username and password but not websocket (port 8883 - SSL not websocket)

2. Belchertown Highcharts

3. Belchertown (subscribing) - which specifies websockets but no username/password (port 9001 - SSL and websocket)

You'll need to check with Pat, but I expect he saw no reason to lock down the subscriptions with username/password when programming his skin, only locking down the publishing (which is done by MWall's MQTT extension). I think the rationale is you don't care who sees the output (after all, it's being published on an open web site), but you don't want any unauthorized uploading of data which you'll be outputting and displaying to others.

So, if the MQTT broker requires a username/password for subscribing over websockets, I don't know if the skin provides for that. I assume you've tried to prepend <username>:<password>@ to the host name in the Belchertown Extras stanza without success.

Hopefully Pat can weigh in here.

phil

[Gary Hammer]

The publishing is not a problem, but that is not in the skin.

As I said, I have tried nearly any combination of config, all to no avail.

 

I may have to look at a different MQTT host, one that does not require username and password for all connections as it seems that is not working in the Belchertown skin.

 

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/5Qn_6oZjLP4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[G Hammer]

Well, I have followed Pat's guide and reconfigured my server's mosquitto install.

Websockets is a mysterious beast.

This is what I get when I open my weather webpage:

1539079347: SNI: Unknown ServerName: ftp.ghammer.net
1539079348: Socket error on client <unknown>, disconnecting.
1539079348: SNI: Unknown ServerName: ftp.ghammer.net
1539079368: SNI: Unknown ServerName: ftp.ghammer.net

However, that is the server's name and the name in the SSL cert.

1539077963: Initial logging level 5
1539077963: Libwebsockets version: 2.0.3 unknown-build-hash
1539077963: IPV6 not compiled in
1539077963: libev support compiled in but disabled
1539077963: libuv support compiled in but disabled
1539077963:  Threads: 1 each 1024 fds
1539077963:  mem: platform fd map:  8192 bytes
1539077963:  Compiled with OpenSSL support
1539077963: Creating Vhost 'default' port 9001, 3 protocols
1539077963:  Using SSL mode
1539077963:  SSL ECDH curve 'prime256v1'
1539077963:  Listening on port 9001
1539077963:  mem: per-conn:          920 bytes + protocol rx buf
1539077963:  canonical_hostname = ftp.ghammer.net
1539077964: lws_protocol_init

After about 15 minutes, voila!

All is working with zero changes to any config.

Quite puzzled, but not touching a thing.

[Philip Kutzenco]

Mosquitto is fussy, especially about its config files. In Pat's guide, he mentions that extra spaces at the end of lines messes things up. He says Mosquitto sometimes needs to be restarted completely. I found that Mosquitto also requires a newline at the end of config files. If you start (or restart) Mosquitto and it sees something it doesn't like in a config file, it immediately exits. Starting Mosquitto again, even if you first fix any errors in the config files, causes it to fail immediately again.

For me, issuing a stop command first (sudo /etc/init.d/mosquitto stop) and then starting it (sudo /etc/init.d/mosquitto start) works. My theory  (with no real evidence :-)) is that when Mosquitto exits badly, it leaves its pid file, or some other flag, in place. When you try to start Mosquitto again, it sees that the pid file is there and won't start. If you first issue the stop command, Mosquitto clears the pid (or some other flag) which allows it to start when issued a start command.

I wonder if Mosquitto was unhappy about starting or restarting cleanly for you because of a leftover artifact, which cleared itself after time? Just speculation with no evidence.

Glad you have it running, at least.

phil

[G Hammer]

I rarely cut & paste when configuring. 

I too have found that restarting any service doesn’t always produce good results.  I just stop then start things. 

I’ll post my config files later as an example for others. 

I’m just happy to have it running in house at long last. 

The public test.mosquitto.org was not real reliable.  

[dhin...@djhindley.com]

Hi 

 

I am trying to set up MQTT on the Belchertown skin, and just can't get it to work, but think I am very nearly there.  Like Philip Kutzenco, I am using Digital Ocean to host my MQTT broker, with my own domain name, and have followed Pat's guide to set up the MQTT broker with SSL using Let's Encrypt.   MQTT all seems to be working, as per Pat's guide.  My weather website (ashteadweather.com hosted at 1&1 with SSL certificate) says "Connected.  Waiting for data", which sounds promising, but it just stays at that message.  I copied the weewx.conf settings format for MQTT and the skin.conf format for Belchertown skin, as suggested by Philip Kutzenco, as per below:

 

 

 

weewx.conf:

 

 

[[MQTT]]

        server_url = mqtt://xxxx:zzzz...@mqttdh.uk:8883/

        topic = weather

        unit_system = METRIC

        aggregation = aggregate

        binding = archive,loop

        log_success = False

        log_failure = True

        [[tls]]

           tls_version = tlsv1

           ca_certs = /etc/ssl/certs/ca-certificates.crt

 

 

 

Belchertown:

 # MQTT Defaults

    mqtt_enabled = 1

    mqtt_host = mqttdh.uk

    mqtt_port = 9001

    mqtt_ssl = 1

    mqtt_topic = "weather/#"

    disconnect_live_website_visitor = 0

 

Syslog does give any clues as to what is going wrong - no errors reported, for example.  Any suggestions from those that have got a similar set up.  I must be doing something basic wrong.  For example, do I need to put any certificates to follow the line   "ca_certs = /etc/ssl/certs/ca-certificates.crt" in weewx.conf?  Or is it something to with needing user names and passwords on publishing/subscribing perhaps?

Many Thanks 

David.

[Colin Larsen]

You'll also need to supply the username and password to "receive" the MQTT data (in skin.conf or Belchertown) but that is not yet supported as far as I know.

Colin

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[Philip Kutzenco]

David,

You don't need to specify a username/password to receive data if you have sett up your broker as Pat detailed in his post. The weewx.conf stanzas in the file I attached in my earlier post are working fine at https://wx.kutzenco.com. One minor change is that I changed the subscription to "weather/loop" instead of "weather/#". It works with #, but I think best practice is to only subscribe to loop.

Here are some things to look at:

1. Did you set up a password for Mosquitto? It is done as follows:

sudo mosquitto_passwd -c /etc/mosquitto/passwd <your username>

Note that the username is just for Mosquitto. It doesn't need to be a linux account name.

1.  Did you set up an acl file (/etc/mosquitto/acl)? It should contain:

# Allow anonymous access to the sys
topic read $SYS/#
 # Allow anonymous to read weather
topic read weather/#

# weewx readwrite to the loop
user <your username from above>
topic weather/#

2. Does your Mosquitto myconfig.conf (/etc/mosquitto/conf.d/myconfig.conf) file contain the following? It should have:

persistence false
allow_anonymous true
password_file /etc/mosquitto/passwd
acl_file /etc/mosquitto/acl
listener 1883
protocol mqtt

# websockets
listener 9001
protocol websocket

I am far from an expert on this, but if you post copies of those files. I will look at them in addition to the weewx.conf stanzas you already published, ans see if I can spot a reason for it not working for you. (I probably won't get a chance to look until tomorrow).

phil

On Thursday, October 11, 2018 at 5:13:23 PM UTC-4, Colin Larsen wrote:

You'll also need to supply the username and password to "receive" the MQTT data (in skin.conf or Belchertown) but that is not yet supported as far as I know.

Colin

On Fri, 12 Oct 2018, 07:59 , <dhin...@djhindley.com> wrote:

Hi 

 

I am trying to set up MQTT on the Belchertown skin, and just can't get it to work, but think I am very nearly there.  Like Philip Kutzenco, I am using Digital Ocean to host my MQTT broker, with my own domain name, and have followed Pat's guide to set up the MQTT broker with SSL using Let's Encrypt.   MQTT all seems to be working, as per Pat's guide.  My weather website (ashteadweather.com hosted at 1&1 with SSL certificate) says "Connected.  Waiting for data", which sounds promising, but it just stays at that message.  I copied the weewx.conf settings format for MQTT and the skin.conf format for Belchertown skin, as suggested by Philip Kutzenco, as per below:

 

 

 

weewx.conf:

 

 

[[MQTT]]

        server_url = mqtt://xxxx:zzzzzzzzz@mqttdh.uk:8883/

[David Hindley]

Phil/Pat

Many Thanks for you reply. 

I did set up a password for Mosquitto and also the acl file, as per your email below.

However, my myconfig.conf file is different to the one you listed below, as I am using Let's Encrypt SSL, so followed the format towards the end of Pat's post ( MQTT "tutorial" ), as follows:

persistence false

allow_anonymous true

password_file /etc/mosquitto/passwd

acl_file /etc/mosquitto/acl

#Insecure mqtt to localhost only and secure mqtt with ssl

listener 1883 localhost

listener 8883

certfile /etc/letsencrypt/live/mqttdh.uk/cert.pem

cafile /etc/letsencrypt/live/mqttdh.uk/chain.pem

keyfile /etc/letsencrypt/live/mqttdh.uk/privkey.pem

protocol mqtt

# websockets

listener 9001

certfile /etc/letsencrypt/live/mqttdh.uk/cert.pem

cafile /etc/letsencrypt/live/mqttdh.uk/chain.pem

keyfile /etc/letsencrypt/live/mqttdh.uk/privkey.pem

protocol websockets

Did you not use SSL on your set up for  https://wx.kutzenco.com? Maybe I have done something wrong with trying to set this part up.  It is really frustrating, as the syslog reports that MQTT is sending records, as it contains several lines like:

Oct 12 09:58:27 raspberrypi weewx[1147]: restx: MQTT: Published record 2018-10-12 09:58:28 BST (1539334708)

Pat - if you see this, do you have any ideas what I might be doing wrong - my hunch is that it is something to do with the settings for SSL for MQTT in weewx.conf, which are shown below.  Do I need to create the ca-certificates.crt file?  Or I guess it could be some issue with my web host for www.ashteadweather.com which is 1&1 (with SSL).

Thanks

David.

weewx.conf file

  [[MQTT]]

        server_url = mqtt://xxxxx:zzz...@mqttdh.uk:8883/

        topic = weather

        unit_system = METRIC

        aggregation = aggregate

        binding = archive,loop

        # log_success = False

        # log_failure = True

        [[tls]]

           tls_version = tlsv1

           ca_certs = /etc/ssl/certs/ca-certificates.crt

The Belchertown skin.conf MQTT content is as follows:

 # MQTT Defaults

    mqtt_enabled = 1

    mqtt_host = mqttdh.uk

    mqtt_port = 9001

    mqtt_ssl = 1

    mqtt_topic = "weather/loop"

    disconnect_live_website_visitor = 0

        server_url = mqtt://xxxx:zzzz...@mqttdh.uk:8883/

[David Hindley]

Not sure it will help solve this or not, but the Mosquitto log shows the following:

 New connection from 86.27.145.159 on port 8883.

1539340809: OpenSSL Error: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

1539340809: Socket error on client <unknown>, disconnecting.

1539340811: Client connection from 86.27.145.159 failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number.

1539340814: New connection from 86.27.145.159 on port 8883.

So, it does seem to be SSL related, but I am not sure how to solve this.  Any ideas please anyone?

David.

[Philip Kutzenco]

I think I found it.

In weewx.conf, you have:

[[tls]]

            tls_version = tlsv1
            ca_certs = /etc/ssl/certs/ca-certificates.crt

It should be:

[[[tls]]]

            tls_version = tlsv1
            ca_certs = /etc/ssl/certs/ca-certificates.crt

Note 3 brackets around tls. Try that.

BTW. I do have the extra lines you note in /etc/mosquitto/conf.d/myconfig.conf. I was in a huury when posting last night.

phil

        server_url = mqtt://xxxxx:zzzzzzz@mqttdh.uk:8883/

[David Hindley]

Phil - you are a genius. Thank you so much.  Three brackets did it, and www.ashteadweather.com is now live!! Not sure how I managed to get only two brackets!  For some bizarre reason, however, the temperature readings are now way off after making this change - showing temperature as oscillating between 66.6 degC (a tad high) and 19.2 deg c (correct) here in autumnal England - must be a units issues somewhere, which I will explore, but if anyone has seen this before and knows how to fix it, please advise.

David. 

On Fri, 12 Oct 2018 at 12:26, Philip Kutzenco <kutz...@gmail.com> wrote:

I may have found it.

in weewx.conf, you have a stanza:

[[tls]]
            tls_version = tlsv1
            ca_certs = /etc/ssl/certs/ca-certificates.crt

It should be:

[[[tls]]]

            tls_version = tlsv1
            ca_certs = /etc/ssl/certs/ca-certificates.crt

with 3 brackets.

Try that.

BTW: I do have the added lines in myconfig.conf that you do. I was in hurray when I posted last night.

phil

On Friday, October 12, 2018 at 6:42:41 AM UTC-4, David Hindley wrote: