TCP server node msg when client disconnects

[Dean Sellers]

Hi,

First, loving node-red. I am updating an express based web-app for some automation tasks. There are a few different devices connected to it all with different apis. I have embedded node-red in my express app as part of the admin/maintenance section and plan on writing all of the device gateway code as flows to standarise them on a pub/sub model using MQTT. So far, so easy!

As part of this I have a tcp server that listens for device connections and routes through a function node to MQTT nodes based on the messages they send. When the device connects it sends an identifier that I store against the msg._session.id. This maps the device data to an MQTT topic. Again, so far so easy.

But when a client connection drops, I don't know. I can update my connection table when (if) the device reconnects, but I can't let subscribers know that the device has gone offline. My first instinct is to extend the TCP node to emit a NULL msg, or a msg with the current (maybe incomplete) buffer and a property that indicates the client has gone (for whatever reason), but I thought I would ask first to see if there were any better ideas out there.

Anyone?

Regards,

D'o

[Dave C-J]

Hi Dean

very timely question... we were just debating something similar...

right now it should send any remaining data in the incoming buffer so you shouldn't miss data... but yes you don't know if it has ended...

Options we are debating include...

1) setting another property (eg msg.end=true;) - downside is that for cases when the buffer is empty we would also send a null payload to the next node that may not be expecting it. (Or indeed to a node where a null payload is valid).

2) Using the new catch node (in master branch) - and log it as an "error" - downside is it doesn't "feel" like an error.

3) Create a new "status" node that reports the same thing as the status labels under each node - which currently (in the case of tcp server) should reflect the number of active connections. - downside is that is seems a bit over the top.

More thoughts please....

[Julian Knight]

What about a second output connection that only fires when the connection status changes? Then people could do whatever they wanted with it?

[Dean Sellers]

Hi Dave,

Glad I have happened on a hot topic :)

In my case, since the output of the node is used in a function node, I can easily deal with a message property like msg.end. In this instance the data doesn't mean much to me anyway, so I would discard it. This is the way I was probably leaning if I 'subclassed' the tcp node for my situation.

I did wonder what the implications of sending a NULL msg were? Which nodes would see this as valid? In my use case the function that gets the msg.end would update the connection table and then send a NULL to the MQTT output node to publish a 0 length message. Subscribers would interpret this as a disconnection and act however they see fit. So in the case of a disconnect with no data in a buffer yet the NULL message works well, the intermediate function node can act on the local state, and the downstream MQTT node just does as it is told, publishes a NULL to the subscribers.

Of course this doesn't work in the case where you have persistent, or retained, topics. You don't want to push a NULL, so you would need to add an intermediate function to swallow NULLs.

I did look at the feature definitions for the various nodes and the query node is the only one that specifies a NULL msg in the case where 'they cannot provide the appropriate payload'. In the case of a broken pipe, or client disconnect, the query node cannot provide the appropriate payload so this seems consistent to me. There doesn't seem to be a specification yet for how input nodes should deal with a NULL msg, and it is probably node dependent.

Maybe on NULL they check msg.end if set use the node.warn/node.error api, if not set it is a valid NULL (for some other reason) and the flow continues?

I think that one of the beauties of node is the simplicity, nodes have input(s) and output(s). Once you have conditional outputs, or flow outside the main flow, things get complicated. Both to implement and to deal with at the user end. If a node has well defined behavior under normal and exception conditions and this doesn't suit the use case you can add function nodes intermediate to deal with the exceptional cases.

One last option that has just occurred to me is to push msg.payload = <partial>, if there is an incomplete buffer (maybe with a property msg.partial = true) then push a null payload with msg.end = true. This seems consistent with the input node description which says something like, an input node must create a new msg{} object for each distinct output. In this case downstream nodes connected to this will do as they would have done anyway and any use case that requires special handling can use a function node to do it.

Edge cases, don't you love them!

 

[Dave C-J]

Dean

yes MQTT and nulls was the one I was thinking of also... publishing a null to a retained topic will clear it. I was also thinking something like the file node may get upset and write a blank file - but I think we actually handle that already and ignore it.

We don't want to send "other" things in the payload (like <partial>) as then things like the file node would have to know what to do with them - especially a pain when the incoming packets are actually a binary buffer. So option 1 still feels best... but...   

Keep thinking...

[Dave C-J]

Hmm, actually the file node currently would try to write a null file, so that would need to be handled. Maybe some other nodes to think about / check also.

And thanks Julian two outputs is an option. (Not one I like but ... On the list)

[Dean Sellers]

Sorry about the confusion, what I meant by the last option was really just an extension of option 1. Basically in the case that you get a disconnect with data in the buffer. Currently the behavior is that data is sent to the downstream, what I thought about was that this still occurs, the partial buffer is sent in the payload;

msg.payload=<the contents of the buffer>

I addition there is another property added to the msg, msg.partial=true

Then the node sends the null payload with the end flag.

I was probably just over-egging the pudding. The partial message with an end flag is probably just as useful.

[Dave C-J]

Ah right... err - but we don't know if it's really partial. If something sends (say) 2000 bytes with an MTU of 1500 and then closes the connection the second chunk would be 500 bytes and be the valid end as well as the "end" of the connection. I think if you see and end property then you have to assume that is all you're going to get... It's up to the app layer above to work out if that is complete or not, and what to do if not.

[Dean Sellers]

Yeah I agree, I am not sure exactly why that one popped out, brain explosion...

The more I think about it the more I am coming to the conclusion that the send msg.end with the buffer contents if any, or no payload, is the way to go. To deal with the NULL payload situation there are two easy options I see;

1. Document the behavior, for those downstream who care a function node to swallow a msg with (msg.end && !msg.payload) could be added in the flow.

or

2. To maintain strict backwards compatibility a boolean setting (defaulting false) on the TCP node "Send msg.end on disconnection" on the updated TCP node.

PS: Since I need this I am happy to contribute time to implement it when the time comes.

[Dave C-J]

In my test version I've decided to add an end:true property to the _session object instead. As it is already there. Seems to work..... But is throwing up other knock on implications that need to be considered before we do anything.

As it is, the nulls cause issues with several nodes including file, http, serial, so making it optional is a possible way out.

But also. When we operate the other way round. Ie we connect out. Then we ought to do the same for when the far end goes away... But we are the client so we try to reconnect automatically. When we do so then no doubt someone will want the connect message also... And it all spirals onwards.

So still thinking.

[Dean Sellers]

Ok, so I have written an small extension to the TCP node.

Basically it adds the option for the node to send _session.end=true on the socket.end event. In the case of the buffered string a partial string in the buffer will be sent as well. Otherwise and empty string, "", or 'null' will be sent as the payload depending on the buffer or string setting.

This only happens for listening (server) socket nodes. My rationale here is that client nodes have different semantics anyway. On a client socket there is already logic to try a reconnect quickly and then go to the reconnect logic. And notification here gets complicated, and I don't need it for my use case :).

There is a diff attached on the current master, or see my fork on github at;

https://github.com/Lighthouse-Automation/node-red there is a branch tcp-disconnect.

Thoughts?