Collaboration to Integrate into Unreal Engine 4

[Eric Green]

My team and I have been working to stream line the learning process for Unreal Engine 4's workflow and we heard about Lucida on CBC Spark. We would like to extend our efforts to integrate UE4 VR Editor with Lucida to make it easier to use, build with and explore. 

UE4 has a plugin for Javascript support, and python plugins.

Where would the community recommend we begin in exploring ways to integrate the system.

[Johann Hauswald]

Hi Eric,

If I understand correctly you'd like to use Lucida as a way for users to learn how to use UE4? Can you describe a little more about what you'd like to accomplish so I can give you some better tips how to get started? Seems like what you could use to start is some Speech Recognition + a Question-Answering system that has a DB of FAQs relevant to UE4 that users can ask as they navigate.

[Eric Green]

Hi Johann ,

Yes that is a starting point, but it can be so much more than that. Our implementation involves some deep learning algorithm modifications coupled with an adaptive and predictive GUI. Speech recognition in Unreal Engine 4 is already available. UE4 is C++, so we have had to recruit a Java developer. Such features are ideal for what Epic Games has currently put out there, so take a look at their new VR Editor

https://www.youtube.com/watch?v=JKO9fEjNiio

It needs some work, and we would like to add some needed functionality. It was through CBC Spark that we heard of 'Lucida'. 

We will be modifying the following section of code:

https://github.com/EpicGames/UnrealEngine/tree/4.12/Engine/Source/Editor/VREditor

To gain access to UE4's source code, check out:

https://www.unrealengine.com/ue4-on-github

We have also been experimenting with IBM's Watson, but we are somewhat drawn to open source! 

[Eric Green]

Any advice where to start? 

[Johann Hauswald]

Sounds like figuring out the domain of questions users would ask and then design a domain specific QA system.

We've been working on designing domain specific QA systems and used Open Ephyra as a starting point. It was tuned for a specific type of question so there are many rules and patterns specific to a corpus of questions that were used for a challenge a few years ago. We're now taking the approach of aggregating information from multiple offline sources (https://github.com/brmson/yodaqa is another open source QA system we are working with) and online sources then designing heurisitics to select the best answer.

Being able to infer the type or domain of the question (maybe you're looking to answer questions from different domains in UE4) helps tremendously. We are also working on adding this layer of text classification that can tell us whether the text is a factoid question, a command, a question about an image, etc. This allows us to route the query to the necessary components in Lucida since we have micro services that accomplish many different tasks.

This is all fairly high level so if you use slack we can definitely chat some more there: https://lucida-ai.slack.com (I can send you an invite to the channel).

[Shayne G. Brown]

Hi Eric,

As an UE4 dev who has been wanting to do something with IBM Watson and UE4 for a long time and knows just how powerful that would make UE4. Do you have any updates or things you could share with us on it. I am looking at ways right now myself into linking it into a new game I am working on that's very ambitious and AI heavy which requires something more than I am capable of creating. Anyways great to see the VR editor stuff and hope to hear something back.

Thanks
Shayne Brown
Singularity Studio