UNIX/Windows Sockets/Pipes - idea for new node from Max Hadley

[Julian Knight]

Hi guys, took the liberty of starting a new thread. 

Here are the original emails from the other thread:

Max Hadley

08:57 (5 hours ago)

Re: [node-red] Re: A Node-RED templated JavaScript object creator node ...

Thinking along rather different lines, I was pondering the uses of a node-red-contrib-UNIX-socket-in node, which would listen on a configurable socket and generate a message for each text line received. The idea being that s shell script would use (BSD) netcat to send the text. This would let system cron jobs notify a node-RED flow when they were run.

You could use such a node to receive test data generated by suitable script. By chance, I just also learned about jo http://jpmens.net/2016/03/05/a-shell-command-to-create-json-jo/ which might make another piece of a solution.

How does this look to you guys?

Max

me (Julian Knight change)

09:25 (5 hours ago)

Re: [node-red] Re: A Node-RED templated JavaScript object creator node ...

Max, I really like the idea of being able to define a UNIX socket/pipe so that system services and commands could send information directly to NR (and efficiently too). Even better if it also worked with Windows sockets/pipes too! Not so commonly known on Windows but it certainly also supports them.

There is room for both if people have the time to build them! I love the dummy-json module because it works so elegantly and should be fairly easy for non-UNIXy/non-programmery type people (e.g 99.999% of the world :-) ) to get their heads around. Sockets on the other hand would suit a different audience. While it would be possible to run the dummy-json module in a separate service and use a socket to transfer the data to NR with your idea, it would require fairly significant system-level skills to set up and maintain.

- show quoted text -

Dave C-J

11:34 (3 hours ago)

Re: [node-red] Re: A Node-RED templated JavaScript object creator node ...

I agree - these are two separate things/nodes.

How well do pipes work on Windows these days ? would this be cross platform in any way ? What about cloud ?

(This should be in another thread...)

To answer your question Dave. While I'm no expert, my understanding is that pipes work well as they have done for many years. Windows is meant to be POSIX compliant which is where I think the standards lie. 

Because we are using Node.js, it should handle many of the cross-platform issues but of course this isn't always the case.

For the cloud, Azure Web Apps are really running Windows Servers with IIS and IISnode so anything that works on Windows should work. Can't really comment on other cloud platforms. I do now have access to Azure so I could always test there. The only problem is that I haven't yet managed to get Node-Red to run as an Azure Web App. I've only tried via a Git deployment so far though and that appears to fail on the serial node compilation. It is also possible to deploy via FTP so the server doesn't need to do a compile (in that case, you upload the node_modules folder too) but the cheap Azure Web App host is 32bit only and I'm currently using 64bit on my Windows PC so I need to do some faffing.

Max, if you wanted to create some test code, I'd be happy to test it on Windows.

[Max Hadley]

I have now created a simple node-red-contrib-ipc (Inter-Process Communication) node and pushed it to Github. I've only really tested it on MAC OS X so far.

I've used the node to create a test socket, e.g. /Users/max/tst.socket and then sent messages to it using BSD Netcat:

ls | nc -U /Users/max/tst.socket

This generates one message for each file in the current directory.

I haven't tried it on Windows, but in theory it ought to work! There is one known issue: if the node creates the socket path and successfully starts listening on it, and then somebody else maliciously deletes the socket, the node does not detect it and falsely continues to report it is listening for input. Normal exit, and redeployment of the flow, cause the socket file to be deleted so that when the flow starts up again it can create the socket file and listen on it. If the file already exists when the flow starts, the node status will show 'Path in use' and keep retrying to listen at 2s intervals in the hope someone will notice and delete the offending file. If the file cannot be created - for example because of inadequate permissions or because an intermediate directory does not exist - the status will show 'disconnected' and nothing will happen until the problem is fixed and the flow restarted.

I haven't been able to provoke a node-RED crash, but I'm probably just not trying hard enough.

Treat this as very much alpha code. I won't publish it to npm until someone has given it a good thrashing on Windows (Julian, I'm looking at you!) It's easy to install, as there are no dependencies. You need only the .js, .html & .png (icon) files.

[Julian Knight]

Hi Max, I'll have a go when I get a chance, I usually manage to break things :-)

[Julian Knight]

Oh, something else that might be interesting. Not many people seem to realise that you can start a node.js server on a socket instead of an IP port. Food for thought. Micro-service integration between Node-Red and other node.js servers.

On Sunday, 13 March 2016 21:58:42 UTC, Max Hadley wrote:

I have now created a simple node-red-contrib-ipc (Inter-Process Communication) node and pushed it to Github. I've only really tested it on MAC OS X so far.

...

[Julian Knight]

OK, I've had a quick try on Windows.

The good news is that the listener starts listening!

The bad news is that it appears to get exclusive use of the pipe and so I can't send anything to it.

The reverse is equally true. If I start a new node-pipe server, it works fine but I can't then create the node on the same pipe.

I used the npm module "named-pipes" which clearly works on Windows as I can use it to send and receive data just fine.

I wonder whether the send/receive functions aren't the wrong way round? In the nomenclature of pipes (at least in node.js), it seems that the sending part seems to be referred to as the server and the receiver as the client. You've implemented a listening node so it should be the client part.

Here is the code that works. I ran each 1/2 in turn to test it with your node, changing the pipe name each time to make sure it wasn't stuck:

// SERVER side

var NamedPipes = require('named-pipes');

var mypipe = 'pipeme3';

server = NamedPipes.listen(mypipe);

server.on ('connect', function (client) {

  console.log('New Client')

  client.send('welcome', 'something') //# THis gets passed to emit on the client side

});

// Client Side

pipe = NamedPipes.connect(mypipe)

pipe.on('welcome', function(str) {

  console.log(str) // # Outputs 'something'

});

I also tried some tests using Node.JS's native pipe/socket functions and got the same issues.

λ node namedpipe2.js

events.js:141

      throw er; // Unhandled 'error' event

      ^

Error: listen EADDRINUSE \\.\pipe\pipeme2

    at Object.exports._errnoException (util.js:870:11)

    at exports._exceptionWithHostPort (util.js:893:20)

    at Server._listen2 (net.js:1223:19)

    at listen (net.js:1272:10)

    at Server.listen (net.js:1362:5)

    at PipeEmitter.listenToPipe (C:\Users\julian\.node-red\node_modules\named-pipes\lib\named-pipes.js:54:32)

    at new PipeEmitter (C:\Users\julian\.node-red\node_modules\named-pipes\lib\named-pipes.js:40:14)

    at Object.module.exports.listen (C:\Users\julian\.node-red\node_modules\named-pipes\lib\named-pipes.js:15:14)

    at Object.<anonymous> (C:\Users\julian\.node-red\namedpipe2.js:5:21)

    at Module._compile (module.js:409:26)

On Sunday, 13 March 2016 21:58:42 UTC, Max Hadley wrote:

I have now created a simple node-red-contrib-ipc (Inter-Process Communication) node and pushed it to Github. I've only really tested it on MAC OS X so far.

...

[Julian Knight]

I also tried with a command line tool compiled from example code on MSDN. I get a very similar issue. Running the server (which sends the data) with the same pipe name that your listener is using doesn't allow access to the pipe.

On Tuesday, 15 March 2016 22:25:26 UTC, Julian Knight wrote:

OK, I've had a quick try on Windows.

The good news is that the listener starts listening!

The bad news is that it appears to get exclusive use of the pipe and so I can't send anything to it.

...

[Max Hadley]

This is very weird. I'll have a look at the named-pipes package and try to figure out what it does, but don't let me stop you from investigating - I'm away for a few days next week so I'm unlikely to get much done before Easter.

[Max Hadley]

Julian,

I've had a look at the named-pipes package, and one thing did strike me (apart from the fact it's in coffeescript!) is that all the backslashes in the pipe name string are escaped:

@pipeAddress = "\\\\.\\pipe\\#{@pipeName}"

However I'm not escaping anything - could it be as simple as that the error message is wrong? Presumably you could just type the \\\\s in the dialog box - can you try with a pipe name like that and let me know what happens, please?

I've also found a copy of issue 50 of The Delphi Magazine from 1999 (I keep things) which explains the new named pipes facility of Windows NT and gives examples - in Delphi. It seems the 'server' is the process which creates the pipe, and the 'client' the one that connects to an existing pipe. The creation flags specify if the pipe is one- or two-directional, and which way the communication goes.

I've tried installing my node on a Windows 7 PC but I don't have an easy way to send data to a pipe. If you have access to a known good pipe client in some language or other could you point me to it!

Thanks

Max

[Niels vd spek]

wauw, i was just starting to think about how to communicate between a freepascal program and node-red. my current idea is to use unix sockets or pipes (not sure yet what the best approche is) as IPC.
and now this message poped up.

i hope your node (max) is wat i was searching for.

[Max Hadley]

Niels

The node is working OK on MAC OS X & GNU Linux, and I've now just about got it working on Windows as well. It should hit npm shortly - hang on, only a bit more testing to do!

Max

[Niels vd spek]

Max,

do i get it correct that your node is only setup as server side that can receive messages?
i need a bedirectional approche.
searching for a unix socket solution i found node-IPC here: https://github.com/RIAEvangelist/node-ipc
maybe it helps you.

[Max Hadley]

Alex,

Yes, I have implemented a server (which is intrinsically bidirectional), but it only handles incoming text data. It should be possible to create a complementary node which created a server that handles only outgoing data, but a true bidirectional server is harder to fit in to node-RED's flow-based operation.

I've just now published v 0.0.2 to npm

Max