Node Red freezes after some time

[Eckehard Fiedler]

Hy all,

I´m using NodeRed on several Pi´s for some time now. I had different setups and versions running, currently Im using TheThingBox 2.4.0.7 with NodeRed 0.15.2.

The setup was very similar on all devices. Node red is used as a gateway for serial data to any cloud platform. Data Input via serial port as a JSON-string, output to different platforms via REST or MQTT. The data rate is fairly low, one JSON-string every 30s.

 

On all installations I face the same issue: After some time Node Red stops working. This might also be a serial port issue, but sometimes i can even not reach the node red ui anymore. 

As far as I can see this is not a memory issue, also a pi Mode one usually freezes earlier than a 2B. I had different approaches to find the reason, but without success. There is also nothing in the log files that gave any hint.

The only thing that works till now is to reset node red frequently. Currently I use a littel script that is placed in /etc/cron.hourly to restart node red every hour:

#!/bin/bash

set +e

systemctl stop thethingbox.service

killall node-red

wait

echo ThingBox Stopped

node /root/thethingbox/thethingbox.js > /dev/null 2>&1 &

echo ThingBox Started !!!!!!!!

There are several topics in this forum that report similar issues, but I´m not sure if this is a common issue.

Any suggestion how to find the reason?? 

Best regards,

Eckehard

[Colin Law]

Are you still able to get terminal access to the pi when it has failed?

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/e7d8adcb-70d5-453b-8d57-0084c42bce02%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Colin Law]

On 5 November 2017 at 14:52, Colin Law <cla...@gmail.com> wrote:
> Are you still able to get terminal access to the pi when it has failed?

If so does top show you that something is hogging memory or cpu?

Colin

[Mark Setrem]

Have you tried using a far newer version of node-red?
If you start with a new raspbian installation there’s instructions on the website for one command that you can run which will end up with the latest version of node-red and the latest version of
Node.js

[Eckehard Fiedler]

thank you for the suggestions.

as mentioned I tried several combinations of node-red, raspian and hardware but could not find a proper reason. I can log in over ssh and restart node red manually. I had a script logging free memory every 5 minutes that did not show any leaks. 

I tried to use more verbose logging of node red, but did not get more info. By the way: are there any more advanced tools for debugging in node red that could help to find similar issues?

Best regards,

Eckehard

[Mark Setrem]

Given the speed of node-red and Node.js development you
would be better doing any debugging on the latest version of node-red rather than a version of node-red and Node.js that was released almost a year ago.

[Colin Law]

On 5 November 2017 at 19:54, Eckehard Fiedler <fiedl...@gmail.com> wrote:
> thank you for the suggestions.
>
> as mentioned I tried several combinations of node-red, raspian and hardware
> but could not find a proper reason. I can log in over ssh and restart node
> red manually. I had a script logging free memory every 5 minutes that did
> not show any leaks.

And what about cpu usage. Run top in a terminal to find out.

Also worth, when it hangs, stopping the mqtt server in case you have
an mqtt flood.

Where is the mqtt server? in the pi or remote?

Colin

[Julian Knight]

Looks like the ThingBox project is dead. So probably time to restructure around a full Node-RED install anyway.

[Eckehard Fiedler]

The latest Thingbox was released 2017 Sept 6 on node red 0.17.5, does not sound too dead. I´ll give this a try and will report later. Anyway I´m pretty sure this is not the answer. 

From my previous set ups it seemd that some kind of overflow might be a reason. My current test case uses only a REST interface to send data. But even there Node Red has to wait for a response. 

Does anybody know what happens to node red, if some part or the flow is too slow ? Maybe the http-request is not answered when a new request ist sent? Maybe not an mqtt flood but a node-red flood?? Is there any kind of indication for such a situation?

Best regards

Eckehard 

[Julian Knight]

Well the website is dead. Or it was when I tried it earlier.

[The ThingBox]

I'm afraid you just came to the website on an update moment....

The Thingbox project is not dead. We just released the last version yesterday adding a WiFi hotspot to connect easily to Pi Zero.

The ThingBox Project - A ready to flash Raspberry Pi image that contains Node-RED and Internet Of Things software for non developers.

Find it at http://TheThingbox.io !!!

[Eckehard Fiedler]

Update: Issue Solved!

Thank you all for your help. That helped me to find the reason:

My flow was triggered from the serial input. For test reasons I added a separate inject node to push some data every some seconds. This runs smoothly, while the serial stops after some hours. So this is clearly a serial blockage.

The interesting part: Serial data come from an Arduino, which ist visibly running all the time. I can reset the Arduino with no reaction. If I unplug the serial line this is recognized in node red. After reconnection Serial ist recognized again.

So, it seems that serial gets blocked for some reason, maybe som missing bits. 

I will try to install some kind of watchdog to recognize this situation. 

Does anybody have a suggestions, how to restart serial input by node red?? Restarting node red works, but this is kind of brute force method.

Best regards,

Eckehard 

[Julian Knight]

It is likely that the serial port isn't closing correctly. That means, when you replug, you probably get a new serial port. You should be able to check this using the "dmesg" command.

I expect there are ways to force the serial port closed and released but I'm afraid that is beyond my knowledge.

[Colin Law]

You said that sometimes you could not access the node red UI, a stuck serial port should not cause that.

Colin

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/8ef314c6-5024-4e9f-b531-9dd809195e92%40googlegroups.com.

[Eckehard Fiedler]

On Thursday, November 9, 2017 at 9:14:27 PM UTC+1, Colin Law wrote:

You said that sometimes you could not access the node red UI, a stuck serial port should not cause that.

Colin

Hy Colin,

When I came here, it was not sure, what precisely caused the issue. I could only see that data where missing. 

Then I changed my setup using two triggers, the serial in and an inject node. So, now it´s sure node-red ist running while serial data flow stops. It´s possible to get it up and running again

a) unplugging serial port

b) resetting node red

c) deploy the current flow

So, any way to reconnect seems to work.

I found other posts that suggest, that there is some issue with node-red-serial-node: https://github.com/node-red/node-red/issues/881

It seems, serial connection is somehow unreliable. 

[Eckehard Fiedler]

Same issue here: https://github.com/raspberrypi/linux/issues/1187

[Julian Knight]

Hi, the issue you reference isn't really a Node-RED one, it stems from how the Pi works and the limitations of the hardware and the USB to serial library.

The result is that serial output is never going to be 100% reliable - to be honest, that is probably a sensible assumption on any platform. As I mentioned in the last post in that issue, the sensible approach is to simply test and throw away any responses that fail the check. If you can't use JSON, you should consider using a checksum on your serial records.

However, that shouldn't be causing NR to hang and certainly doesn't in my setup.

You could try the workaround mentioned in the other link you posted, forcing the USB port to use v1.1 instead of v2.0 as that might well have a positive benefit.