Asynchronous REST call

[Daniel Ellison]

Hey all,

I'm trying to do something that should be pretty straightforward but
isn't obvious to me in the context of a function node. I'm a bit of a
newbie when it comes to server-side JavaScript, though not to
programming in general.

I simply want to call a URL from within a function and retrieve the
results. In this case the result will simply be a number from 0-255.
I've enabled the "http" node.js module and it's working fine: I can see
that I get the correct results from the external server. The problem
comes in with the asynchronous aspect of the call. My function returns
before it gets the result. This will come as no surprise to JS people. :)

I'm pretty sure I need to enable something like Async.js. To that end
I've already installed node-red-contrib-async, which makes available
Async.js and Underscore. Sadly, the documentation is lacking so there's
no clear example of how to use it. It seems to assume (probably rightly)
prior experience with Async.js.

Backtracking a bit, this has to be done in a function node as opposed to
an HTTP node because I need to dynamically call several URLs in a row,
not knowing the URLs in advance (they're read in from a JSON file).

Does anyone have a simple example I can follow to get past this sticking
point? If you need any more details, just say the word.

Thanks in advance!
Daniel

[Roberto Calderon]

Hi There,

The proper way is to be a bit more transparent and avoid writing too much code on the function node. This way other people looking at your flow can figure out what is happening. You don't want to manage all your async calls. Let Node RED do it for you! ;)

This is the way to do all the http calls you want and wait for data to complete. Copy and paste this flow:

[{"id":"ce5f98f9.31a068","type":"function","name":"Wait for all workers to finish","func":"context.data = context.data || {};\n\nswitch (msg.topic) {\n    case \"task1\":\n        context.data.task1 = msg.payload;\n        msg = null;\n        break;\n    case \"task2\":\n        context.data.task2 = msg.payload;\n        msg = null;\n        break;\n    \n    default:\n        msg = null;\n    \tbreak;\n\n}\n\nif(context.data.task1 != null && context.data.task2 != null ) {\n\tmsg2 = {};\n\t//do something with data.\n\t// Task 1 is in context.data.task1,\n\t// Task 2 is in context.data.task2\n    context.data=null; //delete context\n\treturn msg2;\n} else return msg;","outputs":1,"noerr":0,"x":815,"y":306,"z":"ec8bd99d.137428","wires":[["1375fff2.ec8a"]]},{"id":"e7e548e2.181ab8","type":"function","name":"add topic \"task1\"","func":"msg.topic = \"task1\";\nreturn msg\n\n","outputs":1,"noerr":0,"x":564,"y":186,"z":"ec8bd99d.137428","wires":[["ce5f98f9.31a068"]]},{"id":"7b2c5afd.84d3a4","type":"http request","name":"","method":"GET","ret":"txt","url":"","x":307,"y":177,"z":"ec8bd99d.137428","wires":[["e7e548e2.181ab8"]]},{"id":"34b4cae2.cb4b36","type":"function","name":"add topic \"task2\"","func":"msg.topic = \"task2\";\nreturn msg\n\n","outputs":1,"noerr":0,"x":553,"y":347,"z":"ec8bd99d.137428","wires":[["ce5f98f9.31a068"]]},{"id":"ff3b8a05.00c478","type":"http request","name":"","method":"GET","ret":"txt","url":"","x":332,"y":334,"z":"ec8bd99d.137428","wires":[["34b4cae2.cb4b36"]]},{"id":"dbd9de7f.24262","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":161,"y":253,"z":"ec8bd99d.137428","wires":[["7b2c5afd.84d3a4","ff3b8a05.00c478"]]},{"id":"1375fff2.ec8a","type":"debug","name":"","active":true,"console":"false","complete":"false","x":1112,"y":299,"z":"ec8bd99d.137428","wires":[]}]

This is what is happening here: When you hit the inject node each of the http nodes are called. A small function node adds a "topic" element that will be used by a second function node that will wait for all topics (task1, task2, taskn) to complete. Once they are all complete you can agregate the data and do your magic. If you read the http node you can set up the url dynamically with moustache strings and the "msg.url" element in the input. ;)

Hopefully this helps.

[Daniel Ellison]

Thanks, Roberto! One thing I don't understand - probably because I'm not (yet!) a Node-RED expert - is how the "Wait for all workers to finish" node actually waits. It seems to return either "msg" or "msg2" immediately. I would love to be able to do this in a more Node-RED idiomatic way, believe me.

[Dave C-J]

Hi Daniel,
I would do it slightly differently... I would use a file node to read the file... Then a function to split that into separate URLs... (See docs on functions returning multiple messages).. Setting msg.url... And feeding that into an http request node... Etc.
That way the http request will be called once for each url in the file.

Is that what you want or is there some more subtle inner feedback I'm missing ?

[Daniel Ellison]

Hey Dave,

Thanks for the response! I do currently read the JSON file with a file node. I use a function node to extract the URLs and loop over them, using http.request to call each URL. The callback uses node.send() to send the values to the next function node.

If I understand, you would suggest setting msg.url dynamically in a message for each URL and then passing an array of these messages on to an HTTP request node. Would the HTTP request node subsequently pass its results on as an array to the next node? If not, how do I make a function node wait until it has all of its information before continuing on? If the follow-on node receives, say, three messages sequentially, how does it stay in the function until I want it to continue on? As far as I can tell, function nodes simply exit as soon as they've done their work.

These are probably very basic questions and if so, I apologize. I'm not currently at my home computer so I can't test any of this. I'm really enjoying learning Node-RED and plan to use it extensively in my home automation project.

Thanks,

Daniel

[Roberto Calderon]

The node works by looping through the cases "task1" and "task2" and using the context object:

if (context.data.task1 != null && context.data.task2 != null){

  //code will run only when both tasks are not null.

}

You can add more tasks, as many as you do.

[Julian Knight]

Hi Roberto,

What Dave was referring to is here: http://nodered.org/docs/writing-functions.html

When you set the output message to an array, you can make it send multiple messages to the next node, you do it like so:

return [ [msg1, msg2, msg3] ]

Note the array within an array. That's because the outer array controls multiple outputs instead of multiple messages.

So if you return each url in an inner array, the downstream node will get each one as a separate message.

The alternative method for function nodes is to use the send() function to manually send messages instead of using return. This is useful when you need to put the message output inside an async call.

[Daniel Ellison]

Hey all,

I finally came up with a solution to my problem. First, I'll state the problem again for those who skim: I need to periodically query and publish the status of a varying number of attributes on a varying number of devices. For example, I have a Z-Wave deadbolt which can report its state (open/closed) and battery level (1-100). I need to publish the lock's full status to an MQTT broker. The querying is done via REST calls to URLs, one for each attribute.

The solution is as follows: on startup I load the device data from a local JSON file into context.global.devices. This file includes the URLs needed to query the device for each status attribute. An example consisting of one device would be:

{
    "03f508ea-34f3-11e5-ba67-c4e98407dfb6": {
        "product": "Kwikset SmartCode Deadbolt",
        "name": "Side Door Lock",
        "type": "lock",
        "status": {
            "lock": "/ZWaveAPI/Run/devices[3].DoorLock.data.mode.value",
            "battery": "/ZWaveAPI/Run/devices[3].Battery.data.last.value"
        },  
        "actions": {
            "close": "/ZWaveAPI/Run/devices[3].DoorLock.Set(255)",
            "open": "/ZWaveAPI/Run/devices[3].DoorLock.Set(0)"
        }  
    }  
}

In order to publish the device status at regular intervals I started with an "inject" node with "Repeat" set to "interval". In a function node I loop over the devices in context.global.devices (keyed on a UUID). I want to send all of the status URLs at once to the waiting HTTP request node, so for each device I loop over the "status" structure, creating a message which contain, the device ID, the status name, e.g. "battery", and the REST URL. These messages are each pushed to an array. Once I run out of status URLs I push a final "EOT" message ("End Of Transmission" in ASCII) with a null name and node.send() the array in an enclosing array (see http://nodered.org/docs/writing-functions.html#multiple-messages). The function does this for each device in the JSON structure (the outer loop). The resulting 

The waiting HTTP request node receives this array of messages for each device and does a GET request using the URL in each message. As it receives a response it emits a message. Conveniently, the request node passes along any attributes it didn't need (e.g. "id" and "name").

The next function node receives each message from the request node, one by one. It keeps a status structure in "context" and updates it for every message that it receives, returning null so no message is passed out of the node. When it receives an "EOT" message with a null name it packages up the context.status structure into msg.payload and returns that message. The waiting MQTT out node publishes the payload to the MQTT broker.

That took way longer to explain than the code that implements it. The final result is that status messages are published at an interval for every device in the system. The messages end up looking like this:

{
    gateway: 'zwave',
    device: '03f508ea-34f3-11e5-ba67-c4e98407dfb6',
    status: {
        lock: '255',
        battery: '100'
    }  
}

If this interests anyone I can send along the flow or provide further explanation. If you have any ideas for improvement I'm more than happy to hear them.

~Daniel

[Roberto Calderon]

I'm glad you found a solution Daniel.

You can share your flow in the flow library (http://flows.nodered.org/add), people will definitely benefit from it there ;)

[Dave C-J]

+1

[Daniel Ellison]

I'll definitely do that. I'd like to document it properly, so it'll be a day or so.

On Tuesday, August 4, 2015 at 2:57:31 PM UTC-4, Dave C-J wrote:

+1