TCP: too many orphaned sockets

[Dion Mes]

I have node-red running on a Raspberry Pi 3. Installed with the install/update script. The web interface (dashboard and editor) crash about every 24 hours. Nodered keeps running since I have some schedule timers which still work. The first message in the log I see is;

Dec 14 07:27:57 Home kernel: [3013902.907523] net_ratelimit: 142 callbacks suppressed

After that a couple of lines with;

Dec 14 07:27:57 Home kernel: [3013902.907527] TCP: too many orphaned sockets

I have no idea if there is a node creating these orphans, what would be a good way debugging this ?

The nodes I am using with a relation to networking are:

- The HTTP request node.
- Zwave controller node
- Hue magic node

[Dion Mes]

So anyone an idea how to debug this? This way it crashes every 24 hours or so, like last evening with again this in the logs, no further clues :

Dec 15 20:53:53 Home kernel: [3148660.061496] net_ratelimit: 1580 callbacks suppressed

Dec 15 20:53:53 Home kernel: [3148660.061499] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.132646] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.141536] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.191507] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.311498] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.311549] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.311587] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.321549] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.331561] TCP: too many orphaned sockets

Dec 15 20:53:54 Home kernel: [3148660.331672] TCP: too many orphaned sockets

Regards.

[Nick O'Leary]

Have you googled for what that error message means (it isn't a node-red specific error)?

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/101fd1c1-8d0e-4dc6-aaa4-797fd4d2b9aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave C-J]

I would guess it's from one of those extra nodes - but couldn't say which... maybe try grep-ing  the node_modules directory where they are installed searching for "Home kernel:"  will give you a clue

[Dion Mes]

Hi, yes I did. So the TCP connection is still waiting for packets but the process initiating the TCP connection already ended, that is what orphaned socket means as I understand it. Normally these get cleaned up, but somehow to many sockets stay open which in the end results in a crash of node-red. It starts with the dashboard en the node editor which stop responding, while node is till running. So I need to figure out which process creates these TCP connections?!?! And I don't know how.

Thanks for the reply.

Dion

[Dion Mes]

Well the raspberry pi has a hostname Home, and the messages come from the kernel. So that explains the 'home kernel' I think .

[Nick O'Leary]

> maybe try grep-ing  the node_modules directory where they are installed searching for "Home kernel:"  will give you a clue

No. I think it's a message in the system log being generated by the kernel - not a log message you'll find in any node.js code.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/48c5a401-b74f-4581-a4ec-5f76c2bf4249%40googlegroups.com.

[Dion Mes]

Exactly.

So how can I find out the node is my issue.

[Dion Mes]

Identified with ss -nap an lot of outgoing connections from node-red to  54.210.217.45:9553 .  I will try to find out which node connects to that port.

On Thursday, 14 December 2017 21:39:55 UTC+1, Dion Mes wrote:

[Dion Mes]

Ok, so it is the one not listed, I also had a node-red-contrib-nest which is connecting to AWS. Removed this node and we'll see. 

Thanks for giving me a push in analysing the sockets.

Dion

[Dave C-J]

good old google... always watching you .

[Dion Mes]

Ha, yup. Knew that when I bought these nest devices. Stil strange that they connect to an AWS host instead of a google host. It was the port number what that made me identify the nest node. Hope this will solve it.

[Dion Mes]

Solved.

After almost two days I can confirm that node-red-contrib-nest was the culprit.  Informed the developer.

Regards and thanks.