Node red dying on first deploy on raspberry pi... cannot ever get it to run again.

[Dan Hoover]

Ok. I'm baffled.  I'm having a problem with many sd cards on several pi's that has me baffled.

In April I built a "special" image with all my nodes preinstalled.  I've used it on a dozen pis or without issue.  And now I'm getting the same problem on every new pi I try.

I have an mqtt server on a pi zero with no messages coming or going.

I take a fresh copy of my sd card.  

I boot the pi.

I go to node-red

I take an mqtt input node. Subscribe to #  and send it to a debug.

I hit deploy

Node red just hangs and will not ever run again.  It tries to load and just hangs when trying to load all the nodes on the left side.

It does the exact same thing when I do a fresh install of raspbian. Follow the same steps. NR never runs again. 

What could be happening?  How do I see any logs?

[Dan Hoover]

I think I'm narrowing it down to the 2 raspberry pi's I bought at the arrow.com sale.  Hmm.  I'd still like to check the logs and see what's happening, but I can't find them to save my life.

[fluri...@gmail.com]

In addition to pi zero which pi types are using (pi 1, pi 2, pi 3 )?

[Dan Hoover]

They're all Pi 3 v 1.2s. 

It's so repeatable. 

One other weird thing is if I put a "botched" card in one of the Pi's I ordered from Amazon, it sort of "fixes" the card. The flow loads perfectly and then I put it back in the arrow.com pi and it loads. 

I'm thinking the arrow.com ones have either bad sd slots or bad ram?

[Colin Law]

How are you starting node red?

Colin

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/8544f976-926c-4e1a-b48f-b5a8030cfaa4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave C-J]

When you built the image, did that include installing/upgrading node.js ? If so what Pi did you build it on ? Node.js for pi2 or 3 won't run on pi0 (or old pi). Other way round does work. So you need so build on the old pi, or have two builds.

However that wouldn't explain why it runs up until the point you deploy. So as Colin says, how are you starting it ? If you just run node-red command then logs are in the console.

[Dan Hoover]

Right the image was built on a pi 3 and has all the updates.  Node red auto starts on boot.  

[Dave C-J]

so I would not expect that to work on a Pi Zero...

[Dan Hoover]

Right. I'm not using a pi zero. I'm using a Pi 3 1.2

Just doing mqtt in from a local mqtt server.

Topic # straight to debug.  NR crashes.

What else should I check?

[Colin Law]

The node red log should be in /var/log/syslog. You say it is crashing but until you see a crash message you cannot say it is crashing. It may just be locking up in a loop of some sort. Run top in a terminal and see if the CPU goes to a high value. Since it seems that it may be related to mqtt, run mosqitto_sub in a terminal watching all topics and see if you have an mqtt loop.

Colin

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/ac88d094-7e58-453c-a8a9-b097cb85cc61%40googlegroups.com.

[Dave C-J]

Generally subscribing to # shouldn't be an issue - but of course on a busy broker there could be thousands of topics which if loads were retained - could then cause a connect storm. What happens if you a) do as Colin suggests and use an external command line client to test and b) subscribe to a named topic rather than # ?

[Dan Hoover]

Yeah. A flood of messages seems to be the problem. Is there a way to subscribe to only all new messages?

[Colin Law]

It will only give you each message once. If you are getting repeat
messages containing the same data then you are somewhere sending
repeat messages. A common cause of this is something like a dashboard
switch node with an mqtt topic as its input and also sending its value
to an mqtt out. If the Pass Through checkbox (or something like that)
is selected for the switch then messages just go round and round as
fast as the system can manage.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to node-red+u...@googlegroups.com.

> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/9f75d790-c980-4ce7-bf0f-3add5a1936c0%40googlegroups.com.

[Nick O'Leary]

You could also use a delay node set to rate limit node to slow down the processing of the messages and confirm that is the actual issue.

There is no way to subscribe for only new messages - you'll always get the retained messages as well. So the question to ask is whether you need them to be retained or not.

Nick

To post to this group, send an email to node...@googlegroups.com.

Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLt6PojvFn0qSUzh_H-71NGQ9-tMm6F7B7Fz0H%2BV9_%3D%2Bew%40mail.gmail.com.

[Dan Hoover]

Thanks to both of you. I'll start searching out these options!

[Dan Hoover]

Looking at the syslog w1_master_driver is crashing.

It's pretty frustrating because I can't cd ~/.node-red on the pi because it tells me it's not there. I've never been able to get into the NR folder so I can't uninstall a node that crashes. I can only reburn the whole sd card.

Once it crashes, disabling the 1wire bus doesn't fix it.  Nothing is plugged into any gpio pins at all.

Any ideas?

[Colin Law]

Copy/Paste the node red startup sequence from the log here. To copy
from the terminal use Ctrl-Shift-C.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/73bba559-85f9-476c-9137-384e0a23c6be%40googlegroups.com.

[Nick O'Leary]

Dan. assuming you've used our script to setup your pi, you should have a command called node-red-log that will display the most recent log output.

Nick

On 14 August 2017 at 21:27, Dan Hoover <mud...@gmail.com> wrote:

Node red starts on boot and I can't figure out how to reboot it where I can copy those messages.  Any advice?

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/de3817ac-5f9b-4bc8-b0fb-d40a00115f76%40googlegroups.com.

[Dan Hoover]

Thanks for your continued help.  Here's what I've figured out.

It does appear to be a flood of messages as shown by the startup messages...

Welcome to Node-RED

===================

14 Aug 20:18:04 - [info] Node-RED version: v0.15.3

14 Aug 20:18:04 - [info] Node.js  version: v0.10.29

14 Aug 20:18:04 - [info] Linux 4.9.35-v7+ arm LE

14 Aug 20:18:04 - [info] Palette editor disabled : npm command not found

14 Aug 20:18:04 - [info] Loading palette nodes

14 Aug 20:18:11 - [info] Settings file  : /root/.node-red/settings.js

14 Aug 20:18:11 - [info] User directory : /root/.node-red

14 Aug 20:18:11 - [info] Flows file     : /root/.node-red/flows_raspberrypi.json

14 Aug 20:18:11 - [info] Server now running at http://127.0.0.1:1880/

14 Aug 20:18:11 - [info] Starting flows

14 Aug 20:18:11 - [info] Started flows

14 Aug 20:18:11 - [info] [mqtt-broker:13f6dda8.53fb72] Connected to broker: mqtt://192.168.0.222:1883

14 Aug 20:18:12 - [warn] [delay:f86c57b0.d16d88]  buffer exceeded 1000 messages

^C14 Aug 20:18:19 - [info] Stopping flows

Setting a delay of limit 1msg per second does nothing to fix it.

doing a mosquitto_sub to the same server only shows me 20 or so messages.  It's a brand new mosquitto server so it is not like there are lots of messages on it.

I still can't get to my node-red folder from the pi command line, so once it crashes, it seems to be crashed forever.

[Nick O'Leary]

Dan,

that log output shows you what User directory it's using. For some reason you have it set to run as root. So instead of /home/pi/.node-red your nodes are installed under /root/.node-red

Nick

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/de2d900d-962f-4b7a-b4b0-0ab23897780e%40googlegroups.com.

[Dan Hoover]

I'm pretty sure that one is a default raspbian install.  No telling.  I keep putting new cards in and getting the same result. How do I access /root/ from the command line of a pi.

su doesn't work

sudo doesn't work

[Dan Hoover]

Actually, I just thought that one through. 

sudo -i

cd ~/.node-red 

does the trick

It doesn't fix my core problem, but at least I can uninstall nodes

[Mark Setrem]

You are also using a very old no longer supported version of nodejs

14 Aug 20:18:04 - [info] Node-RED version: v0.15.3
14 Aug 20:18:04 - [info] Node.js version: v0.10.29

Given the issues you've been having might it be quicker to start with a new raspbian image and then follow the upgrade steps as listed on the nodered website to end up with a correctly installed, correct permissions,
running as pi, up-to-date version of node-red?

[Dan Hoover]

I understand...but the problem spans all versions.  I have about 10 sd cards I'm cycling through because every time I get this problem I can never run node-red again. I can't figure out for the life of me where these flows are being stored so I can delete them.  My "production" pi has the latest nr/njs/npm

[Nick O'Leary]

Dan, now you have learnt how to access the log output you can start to understand what's going on.

As discussed, the log output includes the user directory. It also tells you the flows file its using and other important pieces of information. So now you know where the flows are stored.

Nick

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/723a3486-0617-42bc-a19e-df51fc34b65c%40googlegroups.com.

[Dan Hoover]

Ok.   Progress!  I'm still confused, but progress.

So, it crashed in the state of:

One timestamp inject going to an mqtt out to a topic.

One mqtt in (#) out to debug.

All of this was in a SINGLE flow.

I went to the folder the log showed and nano'd flows_raspberry.json

Double clicked everything. Ctrl-k to cut. Saved the file.

Restart node-red

Oddly, even though the json file was blank, it came back up with the inject and output node still there but the mqtt in node and debug gone.

Weird.  

However, it still crashes if I subscribe to #. Eventually I'm going to have lots of messages so I need to figure out why it can't do the rate limiting. Any ideas?

Thanks SO MUCH for all your time.

[Dan Hoover]

To add one more thing to the mix...if I add the mqtt in and deploy, it's fine.

If I add mqtt and a delay node. It's fine.

It's only when I add the debug node that the whole thing crashes.

[Dave C-J]

Any chance you can share the flow ?

sent from phone

[Dan Hoover]

Sure! 

[{"id":"f3922ace.a04868","type":"inject","z":"ed614483.76f2b8","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":107,"y":112,"wires":[["5d0ddf79.dd5b2"]]},{"id":"5d0ddf79.dd5b2","type":"mqtt out","z":"ed614483.76f2b8","name":"","topic":"testTopic","qos":"","retain":"","broker":"13f6dda8.53fb72","x":629.0000152587891,"y":108.00000190734863,"wires":[]},{"id":"1a804b90.c60714","type":"mqtt in","z":"ed614483.76f2b8","name":"","topic":"#","qos":"2","broker":"13f6dda8.53fb72","x":102.10000610351562,"y":237.80001831054688,"wires":[["71ab14c5.6b0cec"]]},{"id":"71ab14c5.6b0cec","type":"delay","z":"ed614483.76f2b8","name":"","pauseType":"rate","timeout":"500","timeoutUnits":"milliseconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"x":286.1000061035156,"y":235.00000762939453,"wires":[["b0f65cc.76452a"]]},{"id":"b0f65cc.76452a","type":"debug","z":"ed614483.76f2b8","name":"","active":true,"console":"false","complete":"false","x":477.10003662109375,"y":235.60000610351562,"wires":[]},{"id":"13f6dda8.53fb72","type":"mqtt-broker","z":"","broker":"192.168.0.222","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"willTopic":"","willQos":"0","willPayload":"","birthTopic":"","birthQos":"0","birthPayload":""}]

[Dan Hoover]

It's definitely a retained message issue, which I don't understand. Somehow I have 70k+ retained messages on my internal network on a server that has been up for 2 days. Gotta be some sort of loop. Lots of googling basically tells me there's no easy way to get rid of this crud.  Frustrating. 

[Dan Hoover]

Through lots of sh and python tomfoolery, I think I was able to get the old messages out of my mqtt server, but I'm still really concerned about the way that worked.  I never even touch the "retain" drop down, but I guess I have to explicitly say false, so I don't get stuck with lots of old messages. Thanks so much for walking me through this. This group has been a massive help to me over the last few months.

[Dan Hoover]

Closing the loop...Apparently during my tinkering, there was at one point an if/else loop on an arduino that had a test mqtt publish on the wrong side of the else statement.  It may have attempted to publish about 10k messages per second with the retain flag on. For some reason the pi found them, even though mosquitto_sub didn't. 

Lesson learned. Over 3 days of pulling my hair out. Thanks!

[Colin Law]

Exactly what mosquitto_sub command were you using?

Colin

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/a9daf42c-0576-4c52-8e79-67a7bdf8ba97%40googlegroups.com.

[Julian Knight]

For future reference, if you need to clear the retained messages and don't care about history, simply delete the db file that Mosquitto maintains. You can find that from the log. Stop the Mosquitto service, delete the db and restart the service.