Slow deployment and (possible) crashes

[Csongor Varga]

Hi Guys,

Is there a troubleshooting document available somewhere which explains what to check if there are some suspected issues with NR?

These are my symptoms:

- Deployment is much slower than usual. I always select deploy only modified nodes and even after making changes to a single node (e.g. function node), it takes about 20-30 seconds to deploy. Small changes used to be much quicker. My NR is running on a raspberry PI 3.

- I have a flow which collects information periodically and stores it in context/global, and a physical display that shows this. If the context/global is empty I see that on the screen. Today I noticed for the second time that my context/globals are gone. Like when NR restarts, but there was no actual restart. I was not even in the editor in the last few hours. I should get an email when flows are started, but there was no email. I can also see all my UI elements empty which only get periodic update.

- When I noticed this, I logged into the editor, and it was not loading the tab and was saying that connection to the server is lost. I refreshed the page after about 1 minute and everything loaded OK. I have a system info page, and it was showing 15 minutes CPU load of 0.25, and CPU temp of 55C. Both dropping since.

I am running 0.17.5. I have installed and removed a few nodes via the palette (I did via the command line in the past).

Any pointers?

Thanks,

Csongor

[Andy]

I've noticed the same thing.  After about 50 (I didn't count them, but it's about the same each time) deployments it starts to slow down.  A few deployments beyond that and it's all but locked up.  It will eventually squeeze one through, but a restart of node-red returns it to normal.  I haven't found any logs that explain the situation yet.  The NR server will run without any issue for months if I am not actively developing.  I have one instance running on a Pi zero W running 4 PID loops and a small dashboard and that's been going nonstop for a long time.

[Colin Law]

A couple of ideas. Run, in a terminal

tail -f /var/log/syslog

and make sure that there is not something going on all the time. Then deploy and see if there is anything untowards there then.

Are you running the UI display on the pi or on a remote browser?

Have you got showing charts in the UI?  If so and the browser is running on the pi then close down all the tabs except the editor and see if that makes a difference to the deploy time.

CPU load of 0.25 is high.  Run top in a terminal and see what is using the processor.

If you restart node-red and try again does it still take a long time to deploy?

If you reboot the pi and try again does it still take a long time?

Colin

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/4b149930-4f6e-4e4b-b104-0f8850dfb429%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nick O'Leary]

Hi,

in your settings.js file, you should find a 'logging' section, with 'level' set to "info". If you change "info" to "trace" and restart node-red, you'll get a lot more output in the Node-RED log. That includes useful information around the deploy action to see where time is being spent. That would help you see if there are any nodes in particular causing the slow down.

Nick

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLupePkaRnUTU2%2Bqekr2NteZmooMG8u4OK1Asr07Scy1xw%40mail.gmail.com.

[Csongor Varga]

Hi Colin,

The display is a separate ESP which is displaying data sent by NR. So it is not the dashboard. The PI running headless, if I use the dashboard it is on a separate machine, but only occasionally.

I have attached the syslog and the top. I don't see anything out of the ordinary. The snmpwalk is my process, runs hourly, the w1_bus_master process reading a 1wire temp sensor connected to the PI. That has been there for more than a year now.

I will try restarting the PI completely in the evening when there is peace and quiet.

Thanks,

Csongor

You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/5Nq2UElXyHY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/CAL%3D0gLupePkaRnUTU2%2Bqekr2NteZmooMG8u4OK1Asr07Scy1xw%40mail.gmail.com.

[cflurin]

Which node version are you running?

Try updating to the last version (on my rpi3  it's v6.12.3)

[Csongor Varga]

I am running 6.11.2. I am a bit cautious about running upgrades, last time I did it, my NR was unusable for days. And of course I forgot what caused the issue.

I have restarted the pi, and node red. So far so good. The first deploy was slow, but the second was quicker. I keep and eye on it for a day or so. I will also try changing the debug mode to trace.

On Wed, Jan 3, 2018 at 7:25 PM, cflurin <fluri...@gmail.com> wrote:

Which node version are you running?

Try updating to the last version (on my rpi3  it's v6.12.3)

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/5Nq2UElXyHY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+unsubscribe@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/b3389946-fa28-4fed-8162-6d33cb992218%40googlegroups.com.

[cflurin]

As far I remember this was the issue in node version 6.11.2, so I would update node.

Just use the script "update-node-js-and-node red" described here:

https://nodered.org/docs/hardware/raspberrypi

[Julian Knight]

Your system load values are low - anything over 2 is high I seem to remember.

But your CPU utilisation for Node-RED is high. Mine runs at about 10% avg varying from about 2% to about 25%.

But you have some swap activity by the looks of things and if you are running off an SD card, this will be massively slow.

Looks like something is causing Node-RED to have very high memory utilisation so maybe check what you are keeping in memory. Do you have some global/flow/context variables that are constantly growing? My memory for NR is around 113M RES and 220M VIRT (11%, yours is around 30%).

You might also want to install Glances which gives a better resource display than top.

[Csongor Varga]

OK, I just noticed another crash, and it is memory. This is what I see in the log:

Jan  4 07:02:32 OpenHAB Node-RED[474]: Timeout: No Response from 192.168.1.55

Jan  4 07:02:32 OpenHAB Node-RED[474]: 4 Jan 07:02:32 - [info] [exec:Page count] error:Error: Command failed: snmpwalk -v 2c -c public 192.168.1.55 1.3.6.1.2.1.43.10.2.1.4.1.1 1515045737388

Jan  4 07:02:32 OpenHAB Node-RED[474]: Timeout: No Response from 192.168.1.55

Jan  4 07:02:32 OpenHAB Node-RED[474]: 4 Jan 07:02:32 - [info] [exec:Toner Level] error:Error: Command failed: snmpwalk -v 2c -c public 192.168.1.55 1.3.6.1.2.1.43.11.1.1.9.1.1 1515045737388

Jan  4 07:02:32 OpenHAB Node-RED[474]: Timeout: No Response from 192.168.1.55

Jan  4 07:02:32 OpenHAB Node-RED[474]: 4 Jan 07:02:32 - [info] [exec:Toner Max] error:Error: Command failed: snmpwalk -v 2c -c public 192.168.1.55 1.3.6.1.2.1.43.11.1.1.8.1.1 1515045737388

Jan  4 07:02:32 OpenHAB Node-RED[474]: Timeout: No Response from 192.168.1.55

Jan  4 07:05:03 OpenHAB dhcpcd[512]: wlan0: fe80::1 router available

Jan  4 07:09:01 OpenHAB CRON[22707]: (root) CMD (  [ -x /usr/lib/php5/sessionclean ] && /usr/lib/php5/sessionclean)

Jan  4 07:09:40 OpenHAB dhcpcd[512]: wlan0: fe80::1 router available

Jan  4 07:15:01 OpenHAB CRON[23787]: (root) CMD (python /home/pi/miflora.py C4:7C:8D:60:9E:08 miflora.json)

Jan  4 07:15:04 OpenHAB CRON[23783]: (CRON) info (No MTA installed, discarding output)

Jan  4 07:17:01 OpenHAB CRON[24147]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Jan  4 07:17:56 OpenHAB dhcpcd[512]: wlan0: fe80::1 router available

Jan  4 07:26:43 OpenHAB Node-RED[474]: <--- Last few GCs --->

Jan  4 07:26:43 OpenHAB Node-RED[474]: 44723064 ms: Mark-sweep 776.5 (257.1) -> 775.3 (257.1) MB, 1794.7 / 0.0 ms (+ 2.1 ms in 1 steps since start of marking, biggest step 2.1 ms) [allocation failu

re] [GC in old space requested].

Jan  4 07:26:43 OpenHAB Node-RED[474]: 44724819 ms: Mark-sweep 775.3 (257.1) -> 782.9 (257.1) MB, 1752.3 / 0.0 ms (+ 2.0 ms in 1 steps since start of marking, biggest step 2.0 ms) [last resort gc].

Jan  4 07:26:43 OpenHAB Node-RED[474]: 44726537 ms: Mark-sweep 782.9 (257.1) -> 790.0 (257.1) MB, 1718.5 / 0.0 ms [last resort gc].

Jan  4 07:26:43 OpenHAB Node-RED[474]: <--- JS stacktrace --->

Jan  4 07:26:43 OpenHAB Node-RED[474]: ==== JS stack trace =========================================

Jan  4 07:26:43 OpenHAB Node-RED[474]: Security context: 0x3cb2b6f1 <JS Object>

Jan  4 07:26:43 OpenHAB Node-RED[474]: 2: init(aka RP$initInterceptor) [/home/pi/.node-red/node_modules/request-promise-core/configure/request2.js:45] [pc=0x64f8425c] (this=0x65b84079 <a Request wi

th map 0x535d6165>,requestOptions=0x65b84231 <an Object with map 0x4af3ac41>)

Jan  4 07:26:43 OpenHAB Node-RED[474]: 3: new constructor(aka Request) [/home/pi/.node-red/node_modules/request/request.js:128] [pc=0x64f82570] (this=0x65b84079 <a Request with map 0x535d6165>,opt.

..

Jan  4 07:26:43 OpenHAB Node-RED[474]: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Jan  4 07:26:43 OpenHAB systemd[1]: nodered.service: main process exited, code=killed, status=6/ABRT

Jan  4 07:26:43 OpenHAB systemd[1]: Unit nodered.service entered failed state.

Jan  4 07:26:43 OpenHAB systemd[1]: nodered.service holdoff time over, scheduling restart.

Jan  4 07:26:43 OpenHAB systemd[1]: Stopping Node-RED graphical event wiring tool....

Jan  4 07:26:43 OpenHAB systemd[1]: Starting Node-RED graphical event wiring tool....

Jan  4 07:26:43 OpenHAB systemd[1]: Started Node-RED graphical event wiring tool..

Jan  4 07:26:46 OpenHAB Node-RED[25880]: 4 Jan 07:26:46 - [info]

Jan  4 07:26:46 OpenHAB Node-RED[25880]: Welcome to Node-RED

Jan  4 07:26:46 OpenHAB Node-RED[25880]: ===================

Jan  4 07:26:46 OpenHAB Node-RED[25880]: 4 Jan 07:26:46 - [info] Node-RED version: v0.17.5

Jan  4 07:26:46 OpenHAB Node-RED[25880]: 4 Jan 07:26:46 - [info] Node.js  version: v6.11.2

Jan  4 07:26:46 OpenHAB Node-RED[25880]: 4 Jan 07:26:46 - [info] Linux 4.9.35-v7+ arm LE

Jan  4 07:26:47 OpenHAB Node-RED[25880]: 4 Jan 07:26:47 - [info] Loading palette nodes

Jan  4 07:26:53 OpenHAB Node-RED[25880]: 4 Jan 07:26:53 - [info] Dashboard version 2.4.3 started at /ui

Jan  4 07:26:54 OpenHAB Node-RED[25880]: 4 Jan 07:26:54 - [info] Settings file  : /home/pi/.node-red/settings.js

Jan  4 07:26:54 OpenHAB Node-RED[25880]: 4 Jan 07:26:54 - [info] User directory : /home/pi/.node-red

Jan  4 07:26:54 OpenHAB Node-RED[25880]: 4 Jan 07:26:54 - [info] Flows file     : /home/pi/.node-red/flows_OpenHAB.json

Jan  4 07:26:54 OpenHAB Node-RED[25880]: 4 Jan 07:26:54 - [info] Server now running at https://127.0.0.1:1880/

Jan  4 07:26:58 OpenHAB Node-RED[25880]: 4 Jan 07:26:58 - [info] Starting flows

Jan  4 07:27:26 OpenHAB dhcpcd[512]: wlan0: fe80::1 router available

Jan  4 07:28:27 OpenHAB Node-RED[25880]: 4 Jan 07:28:27 - [info] Started flows

What now? I do keep stuff in the global/context but nothing massive and it is not growing. Just a of single variables, and some small objects. That should not be a problem.

[Csongor Varga]

This PI was originally my OpenHAB test system. I removed the OpenHAB now (at least I think I did) that my help with the memory issue.

[cflurin]

Have look here:

https://groups.google.com/forum/#!searchin/node-red/node-red$20deploy$20slow/node-red/qfZFs-oPEK0/xBHOBwizAQAJ

[Julian Knight]

Indeed, upgrade node.js to the latest v6 version.

[cflurin]

Actually the node-red team recommends LTS 8.x

[Dave C-J]

​Node.js 6 or 8 are fine... 4 should work but is getting a bit out of date now. ​

 

[Csongor Varga]

Let me ask this here, because last time I did an upgrade I was really struggling to get it working again.

So, I have a PI3, and I remember that I started with a preinstalled version of Node Red. Therefore based on this:

https://nodered.org/docs/hardware/raspberrypi

I need to run the bash script on the top, and it will take care of it, right?

Regards,

Csongor

[Dave C-J]

Yes

Run it as your normal user (not root). 

--

Sent from phone.

[Csongor Varga]

OK, I am done. The script executed, I am on v6.12.3, the editor loads much faster and deployment is also faster.

NR is running for 1 hour now, and top says <10% CPI and 15% MEM. I assume this is reasonable.

I have also updated my nodes and so far everything is working fine.

Thank you for all the assistance... again.

Csongor