TCP input to JSON

[Mattias Månsson]

Hi!

I have a little question about the JSON node. It seems as it can only accept full messages (i.e. you can't stream to it). But if I connect it to a TCP input and have a server pushing notifications in json format, can I really guarantee that it won't receive partial json messages? 

I have seen errors in the log like this:

7 Apr 18:20:50 - [error] [json:8d2eada5.72d15] Unexpected token {

so I expect that it could be due to that. I really don't wanted to add a delimiter on the TCP input as I then would have to change my protocol (from just json only, to json with some delimiters). How can I solve this?

Thanks in advance!

Mattias

[Dave C-J]

Hi Mattias,

Yes the JSON node does require compete objects or strings to operate on... 

The TCP node should deliver complete TCP packets.... but that cannot be guaranteed - of course they will be split based on the TCP packet size (MTU) - but should be reassembled correctly.

A possibly more likely scenario is that two or more packets are being sent very quickly - such that the send buffer doesn't flush - so that received TCP packet contains two messages - {blah:blah}{blah:blah} - which would then not be parsed correctly by the JSON node.... could that be a possibility ?

[Mattias Månsson]

That could certainly be a possibility. I'm parsing notifications from xbmc (media player) and it sometimes sends two in a row. How could I solve this without writing my own json parser/splitter? It is hard to split without full parsing as there are lists inside the json objects like this:

{"jsonrpc":"2.0","method":"GUI.OnScreensaverDeactivated","params":{"data":false,"sender":"xbmc"}}{"jsonrpc":"2.0","method":"GUI.OnScreensaverActivated","params":{"data":null,"sender":"xbmc"}}

[Julian Lyndon-Smith]

http://stackoverflow.com/questions/12872563/issues-when-reading-a-string-from-tcp-socket-in-node-js

It seems to be a common problem - there are solutions out there. Perhaps you could write your own node (xmbc-node)  based on the tcp node that is tailored for your particular need with the routines described in stack overflow

Julian

[Mattias Månsson]

I was expecting that unfortunately... :/ Oh well, guess I write my own node then. I have to try that too sometime I guess... :) Thanks for the help!

Btw...I know little about websockets, I just saw the description in the info bar. Could I use that somehow?

WebSocket input node.

By default, the data received from the WebSocket will be in msg.payload. The socket can be configured to expect a properly formed JSON string, in which case it will parse the JSON and send on the resulting object as the entire message.

--
http://nodered.org
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/Q9cHRBbnwvU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Lyndon-Smith]

if xmbc shoves stuff out through websockets, then go for it  ;)

[Mattias Månsson]

Ah, that's a special protocoll I guess... THen no, it's just json over tcp.

[Nicholas O'Leary]

A quick read of this: http://kodi.wiki/view/JSON-RPC_API#Notifications reveals a couple things.

On the tcp stream they say this:

>  Please note that no delimiters are provided in between notifications and/or responses. As such, your client needs to be able to deal with this, eg. by counting and matching curly braces ({}).

Which isn't very helpful - force everyone to do incremental parsing of the stream to find the packets. (Counting brackets as they suggest might work... but as soon as a track title has a { or } in it, oh dear, you now have to count " as well to track when you're inside a string...)

But that same page also says there is a websocket interface available - http://kodi.wiki/view/JSON-RPC_API#WebSocket which may make things easier.

Nick

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.

[Mattias Månsson]

Ah, that's nice! :) Thanks a lot :)

[Dave C-J]

If they really are always sending JSON  { }    and they may come  back to back...  but    }{   is invalid JSON - so you may not need to count...  (though there may be whitespace in between)...

[Mattias Månsson]

That's smart! I guess I could also make it simpler by assuming there will never be brackets inside strings from xbmc... (at least I don't think so)

On Tue, Apr 7, 2015 at 9:54 PM, Dave C-J <dce...@gmail.com> wrote:

If they really are always sending JSON  { }    and they may come  back to back...  but    }{   is invalid JSON - so you may not need to count...  (though there may be whitespace in between)...

--

[Dave C-J]

Yes - is possibly risky so not a "general" solution... but I'm sure you can verify it enough for your situation.

[Dave C-J]

example flow with }{ splitter.... uses new node.send capability in Node-RED  v0.10.6....

[{"id":"fc1ff116.03e01","type":"inject","name":"Set temp to 20","topic":"","payload":"{\"foo\":20}{\"bar\":30}{\"doh\":40}","payloadType":"string","repeat":"","crontab":"","once":false,"x":195,"y":1662,"z":"dd17b386.22e85","wires":[["5363832b.ac9c7c"]]},{"id":"5363832b.ac9c7c","type":"function","name":"}{ splitter","func":"\nwhile (msg.payload.indexOf(\"}{\") !== -1) {\n    var s = msg.payload.indexOf(\"}{\");\n    var a = msg.payload.substr(0,s+1);\n    node.send({payload:a});\n    msg.payload = msg.payload.substr(s+1);\n}\n\nreturn msg;","outputs":1,"valid":true,"x":401,"y":1628,"z":"dd17b386.22e85","wires":[["13c67db9.ec3982"]]},{"id":"13c67db9.ec3982","type":"json","name":"","x":583,"y":1629,"z":"dd17b386.22e85","wires":[["9d55a9d1.62aa58"]]},{"id":"9d55a9d1.62aa58","type":"debug","name":"","active":true,"console":"false","complete":"false","x":749,"y":1671,"z":"dd17b386.22e85","wires":[]}]

[Mattias Månsson]

Works perfectly! :) Thanks a lot for all the help guys!

On Apr 7, 2015 11:04 PM, "Dave C-J" <dce...@gmail.com> wrote:

example flow with }{ splitter.... uses new node.send capability in Node-RED  v0.10.6....

[{"id":"fc1ff116.03e01","type":"inject","name":"Set temp to 20","topic":"","payload":"{\"foo\":20}{\"bar\":30}{\"doh\":40}","payloadType":"string","repeat":"","crontab":"","once":false,"x":195,"y":1662,"z":"dd17b386.22e85","wires":[["5363832b.ac9c7c"]]},{"id":"5363832b.ac9c7c","type":"function","name":"}{ splitter","func":"\nwhile (msg.payload.indexOf(\"}{\") !== -1) {\n    var s = msg.payload.indexOf(\"}{\");\n    var a = msg.payload.substr(0,s+1);\n    node.send({payload:a});\n    msg.payload = msg.payload.substr(s+1);\n}\n\nreturn msg;","outputs":1,"valid":true,"x":401,"y":1628,"z":"dd17b386.22e85","wires":[["13c67db9.ec3982"]]},{"id":"13c67db9.ec3982","type":"json","name":"","x":583,"y":1629,"z":"dd17b386.22e85","wires":[["9d55a9d1.62aa58"]]},{"id":"9d55a9d1.62aa58","type":"debug","name":"","active":true,"console":"false","complete":"false","x":749,"y":1671,"z":"dd17b386.22e85","wires":[]}]

--