[Announce] node-red-contrib-ecolect - A Natural Language Processing node

[Dean Cording]

node-red-contrib-ecolect is a wrapper around the Ecolect natural language matching library.  Whilst it was developed as a complementary module for node-red-contrib-google-action to process speech input, it can process text messages received from anywhere.

This node takes a plain text string and tries to match it to one of the configured topics using fuzzy pattern matching of training phrases. The node can also extract values from the string such as numbers, dates, times, and sub-strings. The value extraction is quite powerful and will interpret values like 'next Friday', 'in 15 minutes', '2.4 million'.

Messages for each topic are sent on separate outputs so that they can be processed by a flow specific to that topic. The last output is always for messages that the processor could not recognise.

The output message contains the matched topic and the extracted values. All other message properties are passed through unchanged.

The Ecolect library works by comparing each word in the input with the corresponding word in each training phrase to calculate the edit distance.  It then returns the topic for the phrase with the lowest total edit distance.  This is not as powerful as the approach used by cloud based systems such as Google Assistant which use Bayesian techniques to calculate the probability that an input phrase matches a topic based of the frequency of word combinations in the training phrases.  The Bayesian approach is good for handling a large variety of topics from a large population of speakers but requires a lot of training.  Ecolect doesn't require as much training but isn't good at matching on input that is only vaguely similar to the training phrases.

Here is a simple example application:

I have a SonoffTH sensor running Tasmota on each of the three levels of my house which feed temperature and humidity information to Node Red via MQTT.  From these readings the comfort level is calculated for each area and all of the information is recorded in the global context.

This flow allows me to query the current temperature, humidity or comfort level for each area using Google Assistant

Action Request is the Google Action request listener.  Intent is a switch node that directs the flow according to whether the conversation is initiating, processing, or closing. An incoming message for the query "what is the temperature upstairs?" from Google Action looks like:

{"conversationId":"1514954494935",
 "intent":"actions.intent.TEXT",
 "dialogState":{},
 "closeConversation":true,
 "userId":"151298",
 "payload":"what is the temperature upstairs",
"_msgid":"56de9922.33bec8"}

Ecolect is the Natural Language Matching node that interprets the incoming text.  In this example it only has one topic configured that looks like:

The topic has two values defined - room and reading.  Using enumerations improves the accuracy of the matching.  The phrases specify what the topic will be matched by and include the locations of the values.

If the topic is matched, the message is sent out of the first output, otherwise it will be sent out of the last output. After processing by Ecolect it looks like:

{"topic":"climate",
 "conversationId":"1514954494935",
 "intent":"actions.intent.TEXT",
 "dialogState":{},
 "closeConversation":true,
 "userId":"151298",
 "payload":"what is the temperature upstairs",
 "_msgid":"56de9922.33bec8",
 "values":{
   "reading":"temperature",
   "room":"upstairs"},
 "score":1}

The message is passed to the Answer Climate Query node.  This is a simple change node which uses the following Jsonata to respond to the query by pulling the reading from the global context.

($room := (values.room = "upstairs" ) ? $globalContext("climate.upper") : (
            (values.room = "downstairs") ? $globalContext("climate.lower") :
            $globalContext("climate.middle"));

$reading := (values.reading = "temperature") ? $room.temperature : (
            (values.reading = "humidity") ? $room.humidity : $room.comfort);

$unit := (values.reading = "temperature") ? "degrees" : (
            (values.reading = "humidity") ? "percent" : "");

"The " & values.room & " " & values.reading & " is " & $reading & " " & $unit)

After processing by the response node, it looks like:

{"topic":"climate",
 "conversationId":"1514954494935",
 "intent":"actions.intent.TEXT",
 "dialogState":{},
 "closeConversation":false,
 "userId":"1512980920890",
 "payload":"The upstairs temperature is 30.6 degrees",
 "_msgid":"56de9922.33bec8",
 "values":{
   "reading":"temperature",
   "room":"upstairs"},
 "score":1}

This is sent back to Google and my Google home announces "The upstairs temperature is 30.6 degrees".

dialogState is an object that is passed back and forwards with Google for the life of the conversation.  If I stored this queries values in there then I can add contextual awareness to my app.  For instance, if I asked 'what is the temperature upstairs?'  and then ask "what is it downstairs?" the app could understand that I was still talking about temperature. 

Enjoy,

Dean

[Dean Cording]

Opps, forgot the link:

https://www.npmjs.com/package/node-red-contrib-ecolect

Dean

[Paul Reed]

Thanks Dean.

Using the example "turn on the kitchen light at 9pm" from the node info, I get;

I'm assuming it's because of an issue in the ecolet library - https://github.com/aholstenson/ecolect-js/issues/2 that you've reported?

Paul

[Dean Cording]

I've actually implemented a work-around for those issues.

What phrases are you using?

[Paul Reed]

I'm using;

turn {item} {state} in {room} at {when}

turn {state} {room} {item}

turn {room} {item} {state}

and asking the question "turn on the kitchen light at 9pm"

The 3 node flow is;

[{"id":"d9a60edf.cba5","type":"ecolect","z":"644f7043.705ed","name":"","topics":[{"name":"gardenlight","phrases":"turn {item} {state} in {room} at {when}\nturn {state} {room} {item}\nturn {room} {item} {state}","values":[{"name":"room","type":"text","enumerations":[]},{"name":"item","type":"text","enumerations":[]},{"name":"state","type":"boolean","enumerations":[]},{"name":"when","type":"date-time","enumerations":[]}]}],"outputs":2,"x":268,"y":415,"wires":[["77c77df1.599574"],[]]},{"id":"77c77df1.599574","type":"debug","z":"644f7043.705ed","name":"","active":true,"console":"false","complete":"true","x":413,"y":416,"wires":[]},{"id":"6811a970.bf4e38","type":"inject","z":"644f7043.705ed","name":"","topic":"","payload":"turn on the kitchen light at 9 pm","payloadType":"str","repeat":"","crontab":"","once":false,"x":111,"y":415,"wires":[["d9a60edf.cba5"]]}]

[Csongor Varga]

I have just installed this and started working with it. I love it. So easy to use. Dean, thanks for sharing this.

Paul, I think you need the following phrase: "turn {state} the {room} at {when}

I found that you really need to add all combination, even an extra "the" will result in an unrecognized command. Also in your room enumeration add "kitchen light" as only "kitchen" is not enough.

[Csongor Varga]

Hi Dean,

I don't want to steal this topic, but do you have an "almost" step-by-step instructions how to set up Google Assistant integration? I read on the contrib node that you need an action to be set up and also make some changes to json files, etc.

Thanks,

Csongor

[Dean Cording]

I'd suggest using enumerations for item and room as otherwise it is a bit ambiguous as that what is a room and what is an item.

[Dean Cording]

Paul,

Try this

[{"id":"c9e9e9a0.faa628","type":"ecolect","z":"30254762.66c5f8","name":"","topics":[{"name":"gardenlight","phrases":"turn {state} the {room} {item} at {when}\nturn {item} {state} in {room} at {when}\nturn {state} {room} {item}\nturn {room} {item} {state}","values":[{"name":"room","type":"enumeration","enumerations":["kitchen","bedroom","bathroom"]},{"name":"item","type":"enumeration","enumerations":["light","fan","radio"]},{"name":"state","type":"boolean","enumerations":[]},{"name":"when","type":"date-time","enumerations":[]}]}],"outputs":2,"x":240,"y":360,"wires":[["9933520b.09721"],["9933520b.09721"]]},{"id":"9933520b.09721","type":"debug","z":"30254762.66c5f8","name":"","active":true,"console":"false","complete":"true","x":385,"y":361,"wires":[]},{"id":"ff746222.bd305","type":"inject","z":"30254762.66c5f8","name":"","topic":"","payload":"turn on the kitchen light at 9pm","payloadType":"str","repeat":"","crontab":"","once":false,"x":83,"y":360,"wires":[["c9e9e9a0.faa628"]]}]

As Csongor said, you need to add as many phrases as you can think of.  The matching algorithm works more on variations of words within phrases rather than variations of phrases.

Dean

[Paul Reed]

Thanks, that's better. (I was following your example in the node info panel).

Above, you commented about dialogState - "If I stored this queries values in there then I can add contextual awareness to my app."

Have you any examples of how to implement this please.

Paul

[Dean Cording]

I haven't had a chance to write one just yet but it would go something like:

1. merge payload.values into payload.dialogState

2. on the next request, if a particular value is missing then use the one from payload.dialogState 

You would need to add phrases that omit the value, so the complete set would look something like:

what is the {reading} in the {room}
what is the {reading}
what about in the {room}

So, if you first asked "what is the temperature in the bedroom", you could then ask "what is the humidity" to get the humidity in the bedroom or "what about in the study" to get the temperature in the study.

I've been using jsonata to build responses.  It's quite powerful but isn't all that flexible.  One trick to consider is that you can do multiple steps on the same property in a change node rather than try to do everything in on go.  The change node runs through each change sequentially.  However, once things become reasonably complex I think you are much better off writing code in a function node.

Dean

[Dean Cording]

On Wednesday, 3 January 2018 20:22:28 UTC+10, Csongor Varga wrote:

Hi Dean,

I don't want to steal this topic, but do you have an "almost" step-by-step instructions how to set up Google Assistant integration? I read on the contrib node that you need an action to be set up and also make some changes to json files, etc.

Thanks,

Csongor

Hi Csongor,

Create an account on Google Actions - https://developers.google.com/actions/

In the console, create a new project and make a note of the project id.

Use the actions.json  from https://github.com/DeanCording/node-red-contrib-google-action/blob/master/action.json as it has the basic config for using with Node Red. You just need to set your server's url.

Use the gaction CLI utility (https://developers.google.com/actions/tools/gactions-cli) to publish your app onto Google Assistant.

  gactions test -preview_mins 9999999 -action_package action.json -project your-project-id

You can test you app using the simulator in the Google Actions console or from any device linked to your Google account. To access your app say:

"Hey Google, talk to my test app"

You can also set up a shortcut in the Google Home app, such as 'Node Red' -> 'talk to my test app', to make it easier.

Dean

 

[final G]

Hi Dean, I like this your project and I’ve been trying to build it but wasn’t successful. I have followed the steps you have mentioned above and I’m able to successfully test my app with the simulator in the google assistance console but when i  add the google action listener to node-red I don’t seem to get it connected. I’ve secured my node red i believe thats all I need. And couldn’t seem to get where I will find the SSL certificate file and SSL private key file that I will add to google action listener node on node-red Please can you help?

[Dean Cording]

Hi,

You can create a new SSL certificate and key pair for this node and put them anywhere where they are accessible from Node Red.  The Google Action node runs a separate http server from Node Red so that you don't need to expose your Node Red server to the internet.  You do however need to set up a port forward on your router from the Internet to your Google Action node.

Dean 

[final G]

Hi Dean 

thank you for your reply. I'm able to get the SSL certificates and I tried to port forward to my Google action node but the action node is not responding. it keeps on saying "my test app isn't responding".  can you take a look on how I port forward please  ? on the action.json file the url used is: ( https://127.0.0) because this is the secured url. google action node listen on 9091. and the node-red unsecured server is (http://192.168.) which is the IP I used on port forward

[Dean Cording]

In your action.json, the URL needs to be the external IP address of your server (or port forwarding router).  This is the address that Google tries to contact.