Parsing multiple json messages

[Dan Hoover]

I have some flexibility on how I format this data, but here's what I have for now.  

I was orignally visiting php page and getting a single json response, but now I need to be able to parse multiple json strings and turn each one into its own mqtt message.  How would I do this?  

Basically, I'm getting a block of 6 json messages that look like this...

{"setpoint":"44.44","company":"name","campus":"location","building":"maint","room":"shop","zone":"zone0"}{"setpoint":"44.44","company":"name","campus":"location","building":"maint","room":"shop","zone":"zone1"}{"setpoint":"44.44","company":"name","campus":"location","building":"maint","room":"shop","zone":"zone2"}{"setpoint":"44.44","company":"name","campus":"location","building":"maint","room":"shop","zone":"zone3"}{"setpoint":"44.44","company":"name","campus":"location","building":"maint","room":"shop","zone":"zone4"}{"setpoint":"44.44","company":"name","campus":"location","building":"maint","room":"shop","zone":"zone5"}

How can I break each of these messages into their own message so each one fires off an individual mqtt query?

If I absolutely have to format them all into one json string, I can, but it will require reprogramming a bunch of arduinos so they can properly parse the messages.

[Dan Hoover]

I should also note that this response comes all at once. It doesn't come in 6 bursts from the server.

[steve rickus]

Hi Dan,

It's not clear what the block of 6 json message represents... is the a big string returned from a php page or something? Did you copy that from node-red, or from a browser? Would be less ambiguous if you could show the expanded output in the debug sidebar.

If that block is coming as a string from some php server, I would try to change the page source to return a valid json array -- just wrap the whole thing in square brackets "[" ... "]" and add a comma "," separator in place of the space between the curly braces like so "},{"

Then you could wire the output to a json node to get an array of objects, wire that to a split node to get six different msgs, and wire that into mqtt out...

--

Steve

[Dan Hoover]

Yep. It's a big string returned from php. I copied it from the browser because nr didn't know how to parse all of it. When it only returns 1 it can do it, but not all 6 at once.  

I will give that a shot. I can concatenate in php and see what happens. I'll keep you posted. Thanks for taking the time to respond!

[Dave C-J]

Dan

As long as you have an identifiable and unique character (or characters)  then the split node should do what you want... often a line break is sufficient "\n" - but can be anything that won't turn up in the rest of the line.

(looking at your string you could use }  but if you do that then you will need to add that character back on to each string before then feeding into a json node (or whatever else you wanted to do with it).

[Dan Hoover]

Ok. I was able to adjust my php so it formats the json string as you suggested...

[ {msg1}, {msg2}, {msg3}, {msg4}, {msg5}, {msg6} ] 

First of all, I never really noticed the json node and it doesn't really have any sort of configuration. When I dropped it into my flow it just seemed like the info passed right through.

What I was doing before when there was only 1 was using a function to parse that single json string like this....

msg.topic = msg.payload.company + '/' + msg.payload.location  + '/' + msg.payload.building  + '/' + msg.payload.room + msg.payload.zone + '/' +'/setpoint';

msg.payload = msg.payload.setpoint;

return msg;

If some of those names above seem weird, I obscured them for a little security through obscurity.

Anyway, what was happening was it was formatting an mqtt topic like this...

organization/campus/building/room/zone/setpoint

Then it was using the setpoint as a payload.

The gist of what is happening is if an arduino has been rebooted unexpectedly, it sends its arduino number to the webserver. The webserver queries the db and sends its last setpoint and state back.

The problem is that as I'm tying more sensors on to the arduino I would like to essentially foreach through them and send each individual setpoint back.

But I need to know how to break up that long json string into x number of msgs and send out an mqtt per json string.

I know I could probably parse this differenly on the arduino side, but I absolutely hate writing this kind of stuff in c and every arduino is already setup to parse the zone/setpoint if I can just send it the message.

I hope that makes sense.

[Dan Hoover]

I just noticed the split node part.  Another node I never understood until it just clicked. I don't have a line break in there, but I can give that a whirl.  THANKS!

[Colin Law]

On 28 September 2017 at 22:06, Dan Hoover <mud...@gmail.com> wrote:
> Ok. I was able to adjust my php so it formats the json string as you
> suggested...
> [ {msg1}, {msg2}, {msg3}, {msg4}, {msg5}, {msg6} ]
>
> First of all, I never really noticed the json node and it doesn't really
> have any sort of configuration. When I dropped it into my flow it just
> seemed like the info passed right through.

No, it would have converted it from an string to an array of objects,
it just looks similar in the debug node. Having got it into an array
if you feed it into an split node I think it should split it into a
sequence of messages.

Colin

>
> What I was doing before when there was only 1 was using a function to parse
> that single json string like this....
>
> msg.topic = msg.payload.company + '/' + msg.payload.location + '/' +
> msg.payload.building + '/' + msg.payload.room + msg.payload.zone + '/'
> +'/setpoint';
> msg.payload = msg.payload.setpoint;
> return msg;
>
> If some of those names above seem weird, I obscured them for a little
> security through obscurity.
>
> Anyway, what was happening was it was formatting an mqtt topic like this...
>
> organization/campus/building/room/zone/setpoint
> Then it was using the setpoint as a payload.
>
> The gist of what is happening is if an arduino has been rebooted
> unexpectedly, it sends its arduino number to the webserver. The webserver
> queries the db and sends its last setpoint and state back.
>
> The problem is that as I'm tying more sensors on to the arduino I would like
> to essentially foreach through them and send each individual setpoint back.
>
> But I need to know how to break up that long json string into x number of
> msgs and send out an mqtt per json string.
>
> I know I could probably parse this differenly on the arduino side, but I
> absolutely hate writing this kind of stuff in c and every arduino is already
> setup to parse the zone/setpoint if I can just send it the message.
>
> I hope that makes sense.
>

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/node-red/5178dc78-3e96-4f83-8739-6338f8491568%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Dan Hoover]

Yep. So in this case, I didn't want to concatenate them all together with [] and ,

Especially since there are commas in the json strings themselves.

I just stuck a * in between each of them and it worked perfectly. One long message became 6 and I'm off to the races. Thank you to both of you!

On Thursday, September 28, 2017 at 1:05:56 PM UTC-8, Dave C-J wrote:

[Dave C-J]

either method should work to get you to 6 separate messages which you can then change / send as you wish. good luck

[Dan Hoover]

Yeah. It's not working as is. I'll have to do a little more debugging before I waste everyone's time, but something is breaking in the split process.

[Dan Hoover]

What can I do to the string in html that won't break it when I try to decode the individual json strings?

If I do the [] and ,

the split doesn't automatically split. It doesn't seem to know what to do with it.

If I do something off the wall like a * it splits them fine, but it loses something and all my variables come back as undefined on the other side of the split.

[Colin Law]

Are you saying that if you add the [] and feed it through json then
split then it does not split it? If so then show us what you see in a
debug node for the string and for the o/p of the json node and the o/p
of the split.

Colin

> --
> http://nodered.org
>
> Join us on Slack to continue the conversation: http://nodered.org/slack
> ---
> You received this message because you are subscribed to the Google Groups
> "Node-RED" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to node-red+u...@googlegroups.com.
> To post to this group, send email to node...@googlegroups.com.
> Visit this group at https://groups.google.com/group/node-red.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/node-red/39574264-05fe-47cb-bc37-b1063f0f5b95%40googlegroups.com.

[Dan Hoover]

Here's a before and after.  The message comes through the split like this....

_msgid: "478170d9.619b4"

topic: ""

payload: "{"setpoint":"44.44","company":"companyname","campus":"alaska","building":"lodge","room":"office","zone":"zone1"}"

statusCode: 200

Then I use a function like this...

msg.topic = msg.payload.company + '/' + msg.payload.campus  + '/' + msg.payload.building  + '/' + msg.payload.room + msg.payload.zone +'/setpoint';

msg.payload = msg.payload.setpoint;

return msg;

And it comes back as 

object

_msgid: "3d9bba2b.0a2c46"

topic: "undefined/undefined/undefined/undefinedundefined/setpoint"

statusCode: 200

[Dan Hoover]

Just did. See below

[Dave C-J]

using * method... 

[{"id":"b872eba3.7ef008","type":"inject","z":"17c6da83.e39e95","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":189.5,"y":855,"wires":[["bc7dd448.a3f3e8"]]},{"id":"bc7dd448.a3f3e8","type":"function","z":"17c6da83.e39e95","name":"","func":"msg.payload = '{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone0\"}*{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone1\"}*{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone2\"}*{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone3\"}*{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone4\"}*{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone5\"}';\nreturn msg;","outputs":1,"noerr":0,"x":335,"y":856,"wires":[["eb26b683.9f7f58"]]},{"id":"eb26b683.9f7f58","type":"split","z":"17c6da83.e39e95","name":"","splt":"*","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":493.5,"y":857,"wires":[["ade5aeec.25aac"]]},{"id":"ade5aeec.25aac","type":"json","z":"17c6da83.e39e95","name":"","pretty":false,"x":626.5,"y":856,"wires":[["be2a707c.452a3"]]},{"id":"be2a707c.452a3","type":"debug","z":"17c6da83.e39e95","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","severity":40,"x":786,"y":862,"wires":[]}]

[Dave C-J]

or array method

[{"id":"96ed2697.7a31d8","type":"inject","z":"17c6da83.e39e95","name":"","topic":"","payload":"","payloadType":"str","repeat":"","crontab":"","once":false,"x":191,"y":916,"wires":[["1fe697c4.1e6438"]]},{"id":"1fe697c4.1e6438","type":"function","z":"17c6da83.e39e95","name":"","func":"msg.payload = '[{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone0\"},{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone1\"},{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone2\"},{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone3\"},{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone4\"},{\"setpoint\":\"44.44\",\"company\":\"name\",\"campus\":\"location\",\"building\":\"maint\",\"room\":\"shop\",\"zone\":\"zone5\"}]';\nreturn msg;","outputs":1,"noerr":0,"x":336.5,"y":917,"wires":[["f4e9dfb0.acf5e"]]},{"id":"f4e9dfb0.acf5e","type":"json","z":"17c6da83.e39e95","name":"","pretty":false,"x":486,"y":919,"wires":[["f9972726.382a78"]]},{"id":"f9972726.382a78","type":"split","z":"17c6da83.e39e95","name":"","splt":"*","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":628,"y":923,"wires":[["9f1b2eb0.d39d7"]]},{"id":"9f1b2eb0.d39d7","type":"debug","z":"17c6da83.e39e95","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","severity":40,"x":787.5,"y":923,"wires":[]}]

[Nick O'Leary]

Hi Dan,

the output:

 payload: "{"setpoint":"44.44","company":"companyname","campus":"alaska","building":"lodge","room":"office","zone":"zone1"}"

shows that msg.payload is a String, not an object of key/value pairs you can do anything with immediately.

You need to pass that msg through a JSON node which will parse its contents to give you a javascript object which you can then do things like msg.payload.setpoint to access the individual values within.

Nick

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/10675967-1297-4f97-aa39-f34f2c2f4615%40googlegroups.com.

[Dan Hoover]

Ok. The extra json node seemed to rejsonize the output after the split. Even though, as you said it "looked" right in the debug window, my function didn't seem to recognize it as json.  Which is weird, because when there was only one of them that same function worked perfectly.  

Two questions just to help me understand.  What does the split function do that breaks the json and what does the json function do.  It seems to me that it's just kind of passing it through but there has to be some voodoo going on.

For the hundredth time. THANK YOU. I never would have gotten that just by guessing. 

[Dave C-J]

you were sending in a string to start with (from your php) - so using the * method - we just split the string into smaller strings... that are now json - which then needed to be made into an actual javascript object which the json node does.

or the array way - it now starts as a correct json string - so we can turn it into a javascript object (an array) first .... then split the array into individual objects.

[steve rickus]

Ok, this is getting confusing... there are really two approaches here:

1. php returns a valid json string that represents an array of msg objects, or

2. php returns a string with a character separator that can be use to segregate the json strings

I like approach #1 because you don't have to deal with strings on the node-red side, and it can be used for either 1 or more than 1 msg. The json node has 2 modes, as outlined in the node info: if msg.payload is a json string, it replaces the msg.payload string with an actual js object, so you can pull out the properties. If msg.payload is an object, it turns it into the equivalent json string. This is why the node does not seem to change the payload too much -- but if you look at the type in the debug sidebar, it will show "string [size]" or "object".

SO, for #1 the flow would be: php json string -> json node -> split (array) -> mqtt (or switch or whatever is downstream)

and, for #2 the flow would be: php blob -> split (by char) -> json node -> mqtt (or switch or whatever is downstream)

The biggest edge case is when the php code returns a single msg as a json string, that must be wrapped in square brackets, to represent an array of one object. In this case the json node creates an array of 1 object, and the split node only outputs that one msg. I hope that's clear?

--

Steve

[Dan Hoover]

On Thursday, September 28, 2017 at 2:02:36 PM UTC-8, Dave C-J wrote:

you were sending in a string to start with (from your php) - so using the * method - we just split the string into smaller strings... that are now json - which then needed to be made into an actual javascript object which the json node does.

or the array way - it now starts as a correct json string - so we can turn it into a javascript object (an array) first .... then split the array into individual objects.

You're absolutely right.   

[Dan Hoover]

You're right. I made a bad assumption that the function I wrote would do some sort of type coersion and figure it out. That's obviously wrong. 

What was happening was that that "string" was coming from a get request, and the http request node was expecting a json string back, so it just parsed it, but when I concatenated multiple json strings together, it didn't know what to do with that, so it just passed it as a string.

[Dan Hoover]

Absolutely clear.  I wound up going with option 1 primarily because of the way my php server was dealing with the data.  

So my flow is 

mqtt in -> http request -> split -> json -> my custom json parser function -> mqtt out

It's doing what I need it to do and it's nice that it doesn't care how many sensors I have, etc.  

I have dozens of arduinos with anywhere between 1 and 12 sensors on them and now when the power goes it, each arduino 

MQTT's a welcome message
Node red turns that into a get request (there's a lot of security/firewalling going on here)

The nginx server queries the db, returns my blob with a * in between each json string.

The splitter splits

The parser parses

My function reformats each one to the proper mqtt message

Node red blasts it out.

Works like a champ and I didn't have to recompile a bit of C code thanks to you all.  Seriously. You're one of the best coding communities on the web.

[Colin Law]

I don't understand what you mean by mqtt in, then http request. Why do you need the http request?

Colin

--

http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.

To unsubscribe from this group and stop receiving emails from it, send an email to node-red+unsubscribe@googlegroups.com.

To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.

To view this discussion on the web, visit https://groups.google.com/d/msgid/node-red/01f4e0fd-060f-4ec6-93a7-3cc9885eca04%40googlegroups.com.

[Colin Law]

Oh, I get it, the http in is just the trigger for the http.

Colin

>> email to node-red+u...@googlegroups.com.

[Dan Hoover]

Right. The arduino sends an mqtt request when it wants it last values back.