NodeRed refuses browser connection

[Ian Eagland]

Hi

This is a follow up to https://groups.google.com/forum/#!topic/node-red/H10Vskn-brk

I discovered the 521 error occurs because the origin web server refused the connection from Cloudflare. ... Something on the web server or hosting provider's network is blocking Cloudflare's requests 

I tried to connect over the local LAN http://192.168.1.27:2086/admin/#flow/1f59a540.e0a65b

and got:- 

This site can’t be reached

192.168.1.27 refused to connect.

So I don't think it is a cloudflare issue. What is really strange is that if I have my  Nodered dashboard active before the error occurs the dashboard is still active after the error without any problems as far as I can tell.

If I try to open my dashboard after the error occurs it just hangs with a blank web page.

So I had a look at syslog. I found a reference to UFW. So I immediately disabled UFW. Unfortunately this has not solved the problem. I cannot see anything else untoward in the log.

Its reached the stage that whenever try to deploy a change I get the error. I have to stop and start Nodered so I can deploy.

I disabled Dataplicity (my remote access to the Pi) in case that was causing a problem. No change.

My mosquitto installation is on the same Pi as NodeRed. I have a number of flows that are event driven via mqtt. They are all working perfectly even though I cannot reach nodered from a browser.

This has me baffled (Not difficult as I am a relative newcomer to the world of Linux though I have learnt a few things)

As I cannot find any reference to anything similar to this problem by googling I guess it is unique to my installation.

Anyone any suggestions as to what I could try next. I am thinking of starting from scratch on another Pi and just copying all my nodered flows across to see if the problem goes away.

There is one line in sylog that I see freqeuntly:-

Sep  7 15:56:07 NodeRed rsyslogd-2007: action 'action 18' suspended, next retry is Thu Sep  7 15:57:07 2017 [try http://www.rsyslog.com/e/2007 ]

What does that mean and could it be relevant?

Other than that start up looks fine (to me)

Sep  7 15:56:24 NodeRed systemd[1]: Starting Node-RED graphical event wiring tool....

Sep  7 15:56:24 NodeRed systemd[1]: Started Node-RED graphical event wiring tool..

Sep  7 15:56:26 NodeRed Node-RED[3653]: 7 Sep 15:56:26 - [info]

Sep  7 15:56:26 NodeRed Node-RED[3653]: Welcome to Node-RED

Sep  7 15:56:26 NodeRed Node-RED[3653]: ===================

Sep  7 15:56:26 NodeRed Node-RED[3653]: 7 Sep 15:56:26 - [info] Node-RED version: v0.17.5

Sep  7 15:56:26 NodeRed Node-RED[3653]: 7 Sep 15:56:26 - [info] Node.js  version: v6.11.3

Sep  7 15:56:26 NodeRed Node-RED[3653]: 7 Sep 15:56:26 - [info] Linux 4.9.35-v7+ arm LE

Sep  7 15:56:27 NodeRed Node-RED[3653]: 7 Sep 15:56:27 - [info] Loading palette nodes

Sep  7 15:56:33 NodeRed Node-RED[3653]: 7 Sep 15:56:33 - [info] Dashboard version 2.3.5 started at /ui

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [info] Settings file  : /home/pi/.node-red/settings.js

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [info] User directory : /home/pi/.node-red

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [info] Flows file     : /home/pi/.node-red/flows_NodeRed.json

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [info] Server now running at http://127.0.0.1:2086/admin/

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] loaded flow revision: 187d13c6434f067c076375bd8d11d233

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/runtime/nodes/credentials.load : no user key present

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/runtime/nodes/credentials.load : using default key

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [info] Starting flows

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : global

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : f4989015.ba371

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : a0043d38.5ffbc

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 141ba2d1.ebe45d

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 31a07a41.ce5f86

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 1f59a540.e0a65b

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 25501a65.daafe6

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : d2f33ad4.2d0cc8

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 69a55057.4a692

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 124f9a2a.edb066

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 9356c150.eaf35

Sep  7 15:56:35 NodeRed Node-RED[3653]: 7 Sep 15:56:35 - [debug] red/nodes/flows.start : starting flow : 4db73f69.000cc

Sep  7 15:56:36 NodeRed Node-RED[3653]: 7 Sep 15:56:36 - [info] Started flows

Sep  7 15:56:36 NodeRed Node-RED[3653]: 7 Sep 15:56:36

[Julian Knight]

Does the Pi have a firewall configured? Also, can you try accessing http://127.0.0.1:2086/admin/ using wget or Lynx from a Pi command line?

[Ian Eagland]

Hi Julian

I had UFW installed but I have disabled it for testing.

pi@NodeRed:~ $ sudo ufw status verbose

Status: inactive

pi@NodeRed:~ $

The result from the wget suggested:-

pi@NodeRed:~ $ wget http://127.0.0.1:2086/admin/

--2017-09-08 09:05:26--  http://127.0.0.1:2086/admin/

Connecting to 127.0.0.1:2086... failed: Connection refused.

pi@NodeRed:~ $

[Julian Knight]

This is what I get on my pi:

$ wget http://127.0.0.1:1880/red/
--2017-09-08 14:11:12--  http://127.0.0.1:1880/red/
Connecting to 127.0.0.1:1880... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

Note that I use /red/ for my admin ui.

Of course, it would help if I remembered that it is set up with a self-signed cert to use TLS :-}

$ wget https://127.0.0.1:1880/red
--2017-09-08 14:12:32--  https://127.0.0.1:1880/red
Connecting to 127.0.0.1:1880... connected.
ERROR: The certificate of ‘127.0.0.1’ is not trusted.
ERROR: The certificate of ‘127.0.0.1’ hasn't got a known issuer.
The certificate's owner does not match hostname ‘127.0.0.1’

Try the following command (from the Pi command line):

sudo netstat -tnlp | grep :2086

That should tell you whether anything is actually listening on the port. You should get something like:

$ sudo netstat -tnlp | grep :1880
tcp        0      0 0.0.0.0:1880            0.0.0.0:*               LISTEN      7685/node-red

If you get no response then you don't actually have a service running on that port at all! In that case, try running

sudo netstat -tnlp | grep node-red

If you still get nothing then node-red isn't actually running on that device or at least it hasn't yet opened any ports. Something like:

sudo ps -ef | grep node-red

Should show if node-red is running at all (you should get 2 entries, one for grep and the other for node-red itself)

[Ian Eagland]

Thanks Julian

Results as follows when in error condition ie cannot deploy a change:-

Last login: Fri Sep  8 09:03:11 2017 from 192.168.1.17

pi@NodeRed:~ $ sudo netstat -tnlp | grep :2086

pi@NodeRed:~ $ sudo netstat -tnlp | grep node-red

pi@NodeRed:~ $ sudo ps -ef | grep node-red

pi         446     1  0 07:56 ?        00:03:02 node-red                        

pi        1151  1119  0 16:18 pts/0    00:00:00 grep --color=auto node-red

pi@NodeRed:~ $

First 2 no response. Last one did get a response.

If I stop and restart NodeRed I get

 pi@NodeRed:~ $ sudo netstat -tnlp | grep :2086                                  tcp        0      0 0.0.0.0:2086            0.0.0.0:*               LISTEN      1221/node-red

pi@NodeRed:~ $ sudo netstat -tnlp | grep node-red                               tcp        0      0 0.0.0.0:2086            0.0.0.0:*               LISTEN      1221/node-red

pi@NodeRed:~ $

pi@NodeRed:~ $ sudo ps -ef | grep node-red                                      pi        1221     1  5 16:25 ?        00:00:17 node-red                        

pi        1298  1119  0 16:31 pts/0    00:00:00 grep --color=auto node-red

Just out of interest I just realised that if the error occurs after I have managed to connect to the server, debug post as responses as normal 

Also how did you present the output that you got so neatly on the forum?

On Thursday, 7 September 2017 17:02:49 UTC+1, Ian Eagland wrote:

[Julian Knight]

OK, so that shows that the Node-RED http server doesn't have it's port open. I think we can generally agree that is a "bad thing"tm

I can only assume that something is going rather badly wrong somewhere. Have you tried upping the logging level and turning on the audit log? In settings.js - that might shed some light.

The formatting comes from using the Google Groups web interface rather than email, I prefer that. The line that is grey with no box around it came from a copy/paste from the web. The ones with boxes round them come from using the last icon in the web editor "Highlight code syntax".

On Friday, 8 September 2017 17:33:24 UTC+1, Ian Eagland wrote:

Thanks Julian

[Ian Eagland]

Thanks Julian

I had set logging to trace.

Nothing untoward showed up. I am guessing something other than node red is getting in on the act. I am going to set up a new Pi and copy my nodered flows and absolutely nothing else. As a relative newcomer to linux I think that will quickest way to get back up and running properly.

Regards

Ian

[Julian Knight]

You may well be right Ian.

If it helps, I personally find it easier in the long run to install NR locally rather than globally. I uninstall the node.js that comes with raspbian. Add the official repo from the node site, create a project folder, do "npm init" to get a package.json and add both node-red and any required node modules to the dependencies list then do "npm install". I then create a .data sub-folder and adjust the Node-RED startup to use that as the user data folder (settings.js, etc). I have a public subfolder for static web resources.

total 68K
drwxr-xr-x   7 pi pi 4.0K Sep  8 08:35 ./
drwxr-xr-x  14 pi pi 4.0K Sep  7 19:13 ../
-rwxr-xr-x   1 pi pi  249 Jan 13  2017 check-status.sh*
drwxr-xr-x   7 pi pi 4.0K Sep  8 15:06 .data/
drwxr-xr-x   4 pi pi 4.0K Nov  5  2016 .data-clean/
drwxr-xr-x 434 pi pi  20K Sep  8 09:19 node_modules/
-rw-r--r--   1 pi pi 2.8K Sep  8 08:38 package.json
drwxr-xr-x   8 pi pi 4.0K Nov 20  2016 public/
-rw-r--r--   1 pi pi  737 Aug 25  2016 README.md
-rwxr-xr-x   1 pi pi   30 Jan 13  2017 restart.sh*
lrwxrwxrwx   1 pi pi   22 Jan 13  2017 showlog.sh -> /home/pi/nrlive-log.sh*

Whilst this takes a bit more work to set up initially, once done, NR upgrades happen with npm update in normal node.js fashion, node.js updates happen via apt-get update as you would expect from any other software.

It also means that you can back up a fully working version with a simple copy; you can run multiple instances with their own versions of Node-RED, transfer a working version to a new machine or, of course, directly onto a new card after a rebuild of the OS.

I also keep a clean version of the .data folder in case I need to reset everything or anything.

$ ls -la .data-clean/
total 48K
drwxr-xr-x 4 pi pi 4.0K Nov  5  2016 ./
drwxr-xr-x 7 pi pi 4.0K Sep  8 08:35 ../
drwxr-xr-x 5 pi pi 4.0K Nov  5  2016 certs/
-rw-r--r-- 1 pi pi  15K Aug 24  2016 .config.json
-rw-r--r-- 1 pi pi  518 Aug 25  2016 envFile.txt
drwxr-xr-x 3 pi pi 4.0K Aug 24  2016 lib/
-rw-r--r-- 1 pi pi    0 Oct 27  2016 nrlive.db
-rw-r--r-- 1 pi pi 8.1K Nov  5  2016 settings.js

[Julian Knight]

Uninstall the version of Node-RED that comes with Rasbian as well of course.

[Julian Knight]

Here is an Ansible task file that shows all the steps to completely remove and purge the default versions of node.js and Node-RED. You should be able to follow the commands even if you aren't using Ansible.

---
# Remove the old versions of Node-RED, Node.JS and npm
# from a vanilla Raspberry Pi (Rasbian) install


# sudo apt-get remove -y nodered
- name: Remove old Node-RED
  apt: name=nodered state=absent
  become: true
# sudo rm -rf /usr/local/lib/node_modules/node-red* /usr/local/bin/node-red*
- name: Remove local lib/bin modules
  command: rm -rf /usr/local/lib/node_modules/node-red* /usr/local/bin/node-red*
  become: true
# sudo rm -rf /usr/lib/node_modules/node-red* /usr/bin/node-red*
- name: Remove lib/bin modules
  command: rm -rf /usr/lib/node_modules/node-red* /usr/bin/node-red*
  become: true
# sudo apt-get remove -y nodejs nodejs-legacy npm
- name: Remove old NodeJS
  action: >
    {{ ansible_pkg_mgr }} name={{ item }} state=absent
  with_items:
    - nodejs
    - nodejs-legacy
    - npm
  become: true
# sudo dpkg -r nodejs
- name: Final purge of nodejs
  command: dpkg -r nodejs
  become: true
# sudo dpkg -r node
- name: Final purge of node
  command: dpkg -r node
  become: true
# sudo rm -rf /usr/local/bin/npm*
- name: Remove local npm bin modules
  command: rm -rf /usr/local/bin/npm*
  become: true
# sudo apt-get autoremove -y
- name: Remove useless packages from the cache
  command: apt-get autoremove -y
  # apt: autoremove=true
  become: true

[Ian Eagland]

Julian

Many thanks for the time you spent on this.
I did go for a complete new install as I was concerned that it was not node-RED that was the issue but something else I may have messed up on the pi.
Wasn't easy because mosquitto will not install on stretch so I had to get the last version of jessie.
Any way, now up an running with a duplicate node-RED. Tomorrow I substitute the old installation with the new and hopefully find the problem solved.

Ian