[announce] node-red-contrib-google-action

[Dean Cording]

I recently acquired a Google Home smart speaker and have spent a couple of days understanding the Google Assistant architecture and how to integrate it with Node Red.

Google Assistant comprises of a number of layers:

1. Android or Home smart devices to perform speech capture and playback

2. Actions to perform speech recognition and initial application routing

3. DialogFlow to add contextual understanding to the user's request

4. (Optional) IFTTT to add logic to processing requests

5. Third party servers like Belkin Wemo or Sonoff eWeLink to control the actual hardware

This node is a wrapper around Google's actions-on-google-nodejs client library and connects to layer 2 above.  It receives the raw text from the speech recognition of the user's request and sends back a response to be spoken to the user.  Each request also includes some state information to allow a conversation to be maintained over multiple requests.

The node runs it's own Express web server, rather than use the Node Red one, so that it can be opened to the Internet without exposing all of Node Red.  Keep in mind that there is no security implemented at the moment.

Also keep in mind that Google Assistant is not intended to host private apps.  However, if you keep your app in perpetual test, then it works like a private app as it is only accessible from devices linked to your account.

There is a sample flow included which demonstrates a simple app that responds to queries containing the work 'number' or 'fancy' such as 'tell me a number', 'what is your number', 'what number is your favourite', 'say something fancy', 'talk fancy to me' etc

The node is available from node-red-contrib-google-action

Enjoy

Dean

[Paul Reed]

Thanks for submitting Dean, I've had great fun playing with this over the past week, however it does seem to rely upon query-response conversation (as you have said in your readme) instead of Natural Language Processing (NLP) which is used in other Google Assistant systems, such as Dialogflow.

This falls down when a query is made which is not captured by the NR code.

A simple example of this could include retrieving the name of a neighbour, by reference to their house number, so user expressions of "who lives at number 8", "number 8 - who lived there" etc.

If the NR query is something like;

     if (question.match( /\b(?:lives|lived)\b/g )) {

         if (question.match( /\b(?:number)\b/g )) {

         .....then return the respective answer....

         }

     }

Then the query would be understood, and the correct answer returned. But if the user expression was "Who resides at number 8", clearly would result in "Sorry, I don't understand", as 'resides' has not been defined in the query.

However, had the 2 example user expressions been added to Dialogflow instead, google's NLP would understand that 'resides' == 'lives' and the query would be fulfilled.

[Dean Cording]

Thanks Paul,

From my brief play with DialogFlow, I was under the impression that you needed to properly register your application with Google to use it and that presented some restrictions.   Also, part of DialogFlow's machine learning capability relies on multiple users making many varied queries so that it can expand its understanding.  I don't know if you would get the same advantage with one user making a few queries. I could be wrong so anyone feel free to correct me if I have misunderstood. 

I am considering making a follow-on node to do NLP in Node Red.  There are a number of NLP packages available that could be easily wrapped in a node.  The advantages to this as you can pick the NLP library that best suits your application (or use multiple) and you can process text from other sources.  

I would also question whether Node Red is the best choice for implementing ad hoc query apps like your example.  In the IoT context, there isn't much scope for free form queries.  We are either turning something on or off (turn lights on), setting a level (set temperature to 25), changing state (I'm leaving now), querying a value (what is the temperature).  Under Google Assistant's architecture, processing complex interactions is passed off on to specialist apps that only understand the language of their particular domain.

These are just my opinions and I'm interested in any feedback.

Dean

[Paul Reed]

I was just using the above as a simple example, but my real usage is to access my temperature sensors, control attached lighting etc, all of which is controlled via node-red & MQTT.

I could be wrong about NLP, but it appears that even in test-app mode, dialogflow takes advantage of google's vast database of NLP. I've found this by throwing obscure queries into a dialogflow test-app, all of which have been correctly fulfilled.

I guess I was looking more towards the model shown below, where the NLP and context is handled by google, and fulfillment is made via a webhook into node-red.

Not ideal though, as it still leaves us grappling with Dialogflow, which despite many hours of reading I still find difficulty comprehending!!

[Paul Reed]

Dean, have you looked at http://compromise.cool/

Paul

[Dean Cording]

I am actually looking at Talkify which uses NLP to map phrases onto skills (actions that the app can perform).  It has a few classifier plugin that allow you to use the NLP approach best suited for your environment.  Plus it also able to keep track of context.

 

[Glenn]

How many betas can you have. I thought you could only have 1 which makes it pretty useless? Correct me if I am wrong.

[Paul Reed]

Do you mean 1 test-app?

If so, yes that's correct, but that one app can be multi functional, and fulfill many actions.
So, pretty useful really.

[Dean Cording]

You can actually have more than one beta by registering and naming your applications but that gets difficult because you need to follow Google's naming requirements of having a two word name that does not contain any known brand words and is unique across all Google Assistant apps ever.  You also have to supply a link to a privacy policy and jump through a few other hoops.

[Glenn]

My test skill is called Sonos and it works fine.

I say "Hey Google, talk to Sonos"

But then I want another one to talk to MusicCast another for lights that aren't on Google home etc.

[Paul Reed]

Hi Dean, I see you're looking at ecolect for NLP.
Have you considered IBM Watson natural-language-classifier or does a cloud based service present issues?

The upside is being a IBM service, there may be (some) support from the node-red gurus...

...just a suggestion.from a newbie!

Paul

[Dean Cording]

Hi Paul,

Yes, I've been looking at a few NLP libraries including Natural and Ecolect with the idea of feeding commands and requests into Node Red.  So far Ecolect is the better choice because it allows you to use templates in your phrases to identify values that you want to extract.  This will make processing requests downstream almost trivial.  I have a prototype NLP node based on Ecolect working now and should have it released in the next day or so. 

As for the use of cloud services for NLP,  I started this project to learn a bit more about current NLP technologies.  It is now quite practical to string together a bunch of cloud based services to implement voice controlled actions, but then I wouldn't be using Node Red if that was the way I wanted to go.

Dean

[Paul Reed]

https://groups.google.com/d/msg/node-red/qDeVXGeSjss/5husg2MoDgAJ

Link added to the Ecolect node which has been developed to work alongside the google-action node.

Paul