Unity World Action Descriptors

[Lake Watkins]

This isn't of much concern to most OpenCog developers, but I thought I'd field it anyway to get some wider feedback.

As we're nearing completion of the alpha for the Unity Game World framework, there's a need to create more explicit actions for testing.  The old version of the framework provided actions which were both high-level (e.g. move to a location) and low-level (e.g. fry an egg) in a sense that was particular to that specific game world.  This was problematic because moving to a location often required several context-sensitive actions along the way (e.g. switching animations between walking, turning, etc.) that had to be provided from the Unity side.  In effect, we would have to construct simple plans on the Unity side when given high-level actions from OpenCog.

In a perfect world, OpenCog would do all context-sensitive planning and simply deliver us low-level actions in real-time.  Failing that, it would give us a detailed plan of low-level actions to perform with some contingencies.  However, given that this is not currently the case, and given that we have other competing concerns (such as NPC behaviours to worry about), it seemed prudent to come up with a solution that could work around some of the current weaknesses in the Embodiment framework and eventually transition into something we would still need in the future.  Since the abstraction-level of actions will also be important once we apply Embodiment to Hanson's robots (via canned animations, etc.), I figured it was a good time to start thinking about this in more detail.

Attached to this email is a draft list of actions broken into High-, Mid-, and Low-Level action descriptors.  The idea is that OpenCog can request an action of either a high, middle, or low level of abstraction by combining zero or one descriptors from each level.  For example, "Move" is a high-level descriptor, "Forward" is a mid-level descriptor, and "Walk" is a low-level descriptor.  We might receive from Embodiment any combination of actions such as "WalkMove", "ForwardMove", "WalkForward", etc., together with a target position that it would like a given game agent to move to.  On the Unity side, we would fulfil this request by applying goal-directed behaviour trees constituted of low-level actions.  The high-level action "Move" would invoke a behaviour tree composed of various sub-trees, including the "Forward" sub-tree, which might eventually invoke the WalkForwardMove action.  Given the high-level nature of "Move", however, it might be possible that we'd move left, right, forward, or backward (or any combination) and that we'd use a variety of movement modes, such as walking, climbing, or even flying.  More descriptors effectively filters actions available to us.

This is useful because we can re-use these behaviour trees when dealing with agents which are not controlled by OpenCog, such as Players or Unity-Controlled NPCs.  Mid-Level behaviour-trees correspond to keyboard and mouse inputs, for example, providing context sensitive actions whether a player presses a key or OpenCog requests an action.  Defining descriptors in this way also allows us to use simple concept blending (i.e. random) until a more complex system for that is in place.

Some actions can't be invoked directly by OpenCog, but they can be requested as a goal.  For example, destroying a block is more like a consequence of activating a character's ability to produce fire than it is an action in its own right.  In this sense, some actions are passive, like falling, some are reactions, like block destruction, and others are a bit of both, like destroying a block you fall on.  These types of gameplay interactions will be handled by Unity but still perceived by OpenCog through Embodiment's perception interface.

The attached list is about as comprehensive as I can make it.  What don't we need?  Are behaviour trees a good way to handle this? How should we handle action/animation transitions?  Should we allow actions to run in parallel (for example, WalkForwardLeftMove = WalkForwardMove + WalkLeftMove).  

Any questions / comments appreciated.

Regards,

-Lake

[Joel Pitt]

Hi Lake -

Thanks for the update. Cool to see how the project is progressing.

When I initially implemented the low level action interface, I aimed
at making it as generic as possible, and open to introspection to
agents. E.g. Agents didn't have to have prior knowledge of any of the
actions or their effects. Instead Agents were expected to infer their
effects from experience..

To this end, I made the Unity world pretty dumb. Support for NPCs that
were not controlled by OpenCog (or another remote AI systems through
the embodiment protocols), was intentionally left out... since that
felt like duplication of effort. In essence it'd be building two sets
of control systems. On the other hand, I can also see the need to
allow for faster responses to the game world, which can only really be
done by having the decision logic on the Unity side. There is also a
fair bit of variety to abstractness to the actions, taking a single
step is obviously a lower level than a "walk" action. So it probably
makes sense to have a hierarchy.

Ideally the Unity world would be a dumb "experience chamber", which
would stream data to/from OpenCog.

Anyhow, these are just my thoughts from my involvement more than a
year and a half ago. I'm sure the project has come a long way since
then... and I'm aware that my vision is potentially no longer in
alignment with goals and priorities that you guys are facing. Good
luck though! ;-)

Cheers
Joel

> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+u...@googlegroups.com.
> To post to this group, send email to ope...@googlegroups.com.
> Visit this group at http://groups.google.com/group/opencog?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Joel Pitt]

Thanks Lake for more info on what's been happening. Sounds really
promising! I look forward to seeing/playing with the alpha :-)

J

On 14 May 2013 17:20, Lake Watkins <nem...@gmail.com> wrote:
> Hey Joel,
>
> Yeah thanks, in some ways there's been a lot of changes to the Unity
> framework, but we've also been able to reuse a lot of your work from before.
> Since I haven't really taken the time to update you and the other
> contributors about recent progress, here's a short list of what's changed:
>
> Voxel Engine: the code for rendering blocks and generating collision meshes
> has been gutted and replaced. We can handle hundreds of thousands of blocks
> in the game world now with much more stability when blocks are created or
> destroyed. New chunks of blocks stream in cleanly to allow for an
> infinitely expanding world. This is more than the 3D Space Map can handle
> at the moment, but for smaller, static worlds it works great.
> Minecraft Support: we've adopted the same file format as minecraft, allowing
> us to load worlds created in that game or related tools (such as MCEdit) and
> save our own world in that format with just a little bit of overhead for our
> own needs. We primarily use MCEdit to mock up pre-generated worlds now.
> PAI and Avatar decoupling: there were a number of tight-couplings between
> the Perception Action Interface and the Avatar class. I think the
> abstraction here was well intended, but the execution created too many
> problems for testing when changes (like new actions) were frequent. The new
> framework eliminates the need for a binding Avatar class by separating each
> concern into a separate component of the game characters. This way, a
> failure in executing a particular animation doesn't propagate up the
> Perception/Action chain and cause planning to fail on the embodiment server
> (like it once did). There were other reasons too, but mainly the fact that
> Avatar touched everything from animations to actions to physics just stood
> out as a red flag to me.
> Protocol Buffers: We're in the middle of providing real support for
> protobuf, but we haven't weaned ourselves off of XML serialization just yet.
> My understanding is that ZeroMQ and ProtoBuf will be the serialization
> methods of choice for future systems in OpenCog, so we've been doing our
> part to support that.
> Behaviour Trees: we've added support for Behave, an implementation of
> behaviour trees for Unity. This saves us a fair amount of work, and lets us
> put trees together graphically pretty quickly.
> Ease-of-Use and Extensibility: Unity doesn't generally support threading,
> especially in standalone executables, so we switched over to using
> co-routines to maintain speed. I've also been working on a variety of
> extensions to the Unity Inspector's editor, both for myself and for future
> contributors. Hopefully this will help new guys navigate the Unity Editor's
> interface more easily while also promoting better coding practices. To give
> everyone an idea, without these extensions contributors would be
> incentivized to publicly expose all of their classes' data members, rather
> than using properties, etc. Unity's just a headache at times.
>
> Aside from the above, I've also outlined a tentative game design that we'll
> be aiming for in the event this ever reaches critical mass (I'll share it if
> anyone's interested). Since my focus has been on making a good testing
> framework (or "experience chamber"), however, I haven't really put much
> effort into gameplay yet. It's really only now that I've had a chance to
> dig into the action interface, so that's what promoted the first email.
>
> One of the conceptual changes I've made in this regard has been to change
> the Unity Game's interface from Noun/Verb to Verb/Noun. I can see that you
> guys were aiming for something like what they did in the Sims, where you
> effectively select the object that you want to manipulate and then choose
> the context-dependent action that you want to perform on it. This is fine
> for games like the Sims which aren't action-oriented, or CAD tools, but most
> action-oriented games today use a Verb/Noun selection scheme where each
> keyboard or mouse input corresponds to a type of action you can take. The
> exact context-dependent action taken then depends on the target. For
> example, before you would select a ball and then select the kick action from
> a menu, which would cause the ball to go flying. In the new scheme, you can
> kick at any time, but a ball's only going to go flying if you're standing in
> front of it. Especially once we started adding more animations for each
> type of action, we found this a lot easier to work with.
>
> A lot of things, like a hierarchy of behaviour trees, followed from that.
> I'd still like to keep Unity as dumb as possible and allow OpenCog to do the
> heavy lifting, so for example I'm not promoting giving actions an explicit
> hierarchy in the atomspace or anything. OpenCog should be able to figure
> out the conceptual relationships of actions on its own - it's just that
> we'll handle the edge cases in Unity when we're not given enough context by
> OpenCog. This isn't really code duplication since we can reuse the same
> trees for NPCs; effectively we can just replace where the Emodiment server
> would plug in to Unity with a NPC Behaviour tree instead, etc.
>
> I hope that clears a few things up. I'll post an updated draft of the
> action descriptors after I add some annotations for Nicky, our super-amazing
> artist.
>
> Regards,
> -Lake

[Lake Watkins]

Hey Joel,

Yeah thanks, in some ways there's been a lot of changes to the Unity framework, but we've also been able to reuse a lot of your work from before.  Since I haven't really taken the time to update you and the other contributors about recent progress, here's a short list of what's changed:

• Voxel Engine: the code for rendering blocks and generating collision meshes has been gutted and replaced.  We can handle hundreds of thousands of blocks in the game world now with much more stability when blocks are created or destroyed.  New chunks of blocks stream in cleanly to allow for an infinitely expanding world.  This is more than the 3D Space Map can handle at the moment, but for smaller, static worlds it works great.
• Minecraft Support: we've adopted the same file format as minecraft, allowing us to load worlds created in that game or related tools (such as MCEdit) and save our own world in that format with just a little bit of overhead for our own needs.  We primarily use MCEdit to mock up pre-generated worlds now.
• PAI and Avatar decoupling: there were a number of tight-couplings between the Perception Action Interface and the Avatar class.  I think the abstraction here was well intended, but the execution created too many problems for testing when changes (like new actions) were frequent.  The new framework eliminates the need for a binding Avatar class by separating each concern into a separate component of the game characters.  This way, a failure in executing a particular animation doesn't propagate up the Perception/Action chain and cause planning to fail on the embodiment server (like it once did).  There were other reasons too, but mainly the fact that Avatar touched everything from animations to actions to physics just stood out as a red flag to me.
• Protocol Buffers: We're in the middle of providing real support for protobuf, but we haven't weaned ourselves off of XML serialization just yet.  My understanding is that ZeroMQ and ProtoBuf will be the serialization methods of choice for future systems in OpenCog, so we've been doing our part to support that.
• Behaviour Trees: we've added support for Behave, an implementation of behaviour trees for Unity.  This saves us a fair amount of work, and lets us put trees together graphically pretty quickly.
• Ease-of-Use and Extensibility: Unity doesn't generally support threading, especially in standalone executables, so we switched over to using co-routines to maintain speed.  I've also been working on a variety of extensions to the Unity Inspector's editor, both for myself and for future contributors.  Hopefully this will help new guys navigate the Unity Editor's interface more easily while also promoting better coding practices.  To give everyone an idea, without these extensions contributors would be incentivized to publicly expose all of their classes' data members, rather than using properties, etc.  Unity's just a headache at times.

Aside from the above, I've also outlined a tentative game design that we'll be aiming for in the event this ever reaches critical mass (I'll share it if anyone's interested).  Since my focus has been on making a good testing framework (or "experience chamber"), however, I haven't really put much effort into gameplay yet.  It's really only now that I've had a chance to dig into the action interface, so that's what promoted the first email.  

One of the conceptual changes I've made in this regard has been to change the Unity Game's interface from Noun/Verb to Verb/Noun.  I can see that you guys were aiming for something like what they did in the Sims, where you effectively select the object that you want to manipulate and then choose the context-dependent action that you want to perform on it.  This is fine for games like the Sims which aren't action-oriented, or CAD tools, but most action-oriented games today use a Verb/Noun selection scheme where each keyboard or mouse input corresponds to a type of action you can take.  The exact context-dependent action taken then depends on the target.  For example, before you would select a ball and then select the kick action from a menu, which would cause the ball to go flying.  In the new scheme, you can kick at any time, but a ball's only going to go flying if you're standing in front of it.  Especially once we started adding more animations for each type of action, we found this a lot easier to work with.

A lot of things, like a hierarchy of behaviour trees, followed from that.  I'd still like to keep Unity as dumb as possible and allow OpenCog to do the heavy lifting, so for example I'm not promoting giving actions an explicit hierarchy in the atomspace or anything.  OpenCog should be able to figure out the conceptual relationships of actions on its own - it's just that we'll handle the edge cases in Unity when we're not given enough context by OpenCog.  This isn't really code duplication since we can reuse the same trees for NPCs; effectively we can just replace where the Emodiment server would plug in to Unity with a NPC Behaviour tree instead, etc.    

I hope that clears a few things up.  I'll post an updated draft of the action descriptors after I add some annotations for Nicky, our super-amazing artist.

Regards,

-Lake

On Tue, May 14, 2013 at 4:23 AM, Joel Pitt <joel...@gmail.com> wrote:

[Linas Vepstas]

Hi Lake,

On 14 May 2013 00:20, Lake Watkins <nem...@gmail.com> wrote:

• Minecraft Support: we've adopted the same file format as minecraft,

Bravo!

 

• PAI and Avatar decoupling:

I'm confused. Is this in Unity?  Because OpenCog has classes/directories with these names, and I don't think I saw any changes being checked into the git repo for this work... as I like to remind people: on git, merge early, merge often, make unit  tests run.

 

• Protocol Buffers: We're in the middle of providing real support for protobuf, but we haven't weaned ourselves off of XML serialization just yet.  My understanding is that ZeroMQ and ProtoBuf will be the serialization methods of choice for future systems in OpenCog, so we've been doing our part to support that.

This statement makes me very nervous.  Last time I looked at code in opencog that used these, it was a horrible mess.  These two protocols seemed to be invasive (they'd get used in random code for no apparent reason) they'd cause build breaks (forcing circular dependencies between unrelated directories, each of which required the other lib to link, thus forcing all libs to be static, not shared) and based on old emails, they seemed to be very slowww, with a terrible performance profile.

The same way that you are concerned about the overall architecture of PAI and Avatar and Voxel engine, I'm concerned about the overall architecture of commo.  Randomly attaching random parts of the code without really thinking about the point of it all just gives me the willies.   I've no specific critques .. but ... jeez, be careful, don't create a bowl of spaghetti.  Just like OO design, commo also has design patterns, of which the most common and evil is 'spaghetti'.

One of the conceptual changes I've made in this regard has been to change the Unity Game's interface from Noun/Verb to Verb/Noun.  I can see that you guys were aiming for something like what they did in the Sims, where you effectively select the object that you want to manipulate and then choose the context-dependent action that you want to perform on it.  This is fine for games like the Sims which aren't action-oriented, or CAD tools, but most action-oriented games today use a Verb/Noun selection scheme where each keyboard or mouse input corresponds to a type of action you can take.  The exact context-dependent action taken then depends on the target.  For example, before you would select a ball and then select the kick action from a menu, which would cause the ball to go flying.  In the new scheme, you can kick at any time, but a ball's only going to go flying if you're standing in front of it.  Especially once we started adding more animations for each type of action, we found this a lot easier to work with.

This tickled me.  I don't know if you know this, but in NLP, in dependency grammar parsing (and in other grammar styles), the verb is the head word.  That is, for sentences, its the verb that dominates the est of the words in the sentence; the nouns are there to be kicked around by the verb.  This would hold for English and most any other language as well.  So, in some deep psychological sense, human beings are verb machines.  Well .. so are dogs: my dog knows more verbs than nouns (come, go, sit, heel, down, jump, fetch ... vs ball, bone.)   Curious.

-- Linas

[Linas Vepstas]

On 15 May 2013 03:54, Lake Watkins <nem...@gmail.com> wrote:

Yesterday Nicky showed me that we can setup actions in a piecewise manner (telling each limb how to animate separately, etc.), so I'll be updating the low-level descriptors to match that.  With any luck, the RoboKind API will share many analogous low-level action descriptors or there'll be a strong correspondence 

Let me know when you start using adjectives and adverbs  ("rotate limb SLOWLY", "swing arm TO THE LEFT".)   These all have particular dependency parses (trees); I wonder how deeply linguistic structure can mesh with, or interface to, action planning.  Or rather, how unified/how similar the structures can be, and if there are any parts of action planning and motion control for which a linguistic approach would NOT be the best/easiest way to do things ... 

-- Linas

[Lake Watkins]

I'm confused. Is this in Unity?  Because OpenCog has classes/directories with these names, and I don't think I saw any changes being checked into the git repo for this work... as I like to remind people: on git, merge early, merge often, make unit  tests run.

I'm only working at it from the Unity side, but I estimate this is only half the problem.  Also, you'll be glad to know that I merge early and often (sometimes several times a day), I've been teaching everyone else here how to use mergetools (I prefer p4merge myself), and even though I haven't made a lot of unit tests I've been talking to David Hart about ways to get the Unity player working with unit tests.  My process isn't perfect, but once I start getting my feet wet in the real OpenCog code I promise to fork, make pull requests, and write unit tests properly.

This statement makes me very nervous.

 Right now our ProtoBuf implementation is little more than a class attribute at the top of our models:  

[ProtoContract(ImplicitFields = ImplicitFields.AllPublic)]

So it's minimally invasive.  I'm staving off the fuller implementation for now since I couldn't find a reliable solution for cross-cutting concerns in Unity.  I tried a few Aspect-Oriented approaches that work for standard C#, but no luck yet.  I'm well aware that an invasive, randomly applied serialization protocol will do more harm than good.  At the end of the day we need serialization for many different reasons (runtime to edittime persistence, communicating with the embodiment server, etc.), so we'll have to use serialization in some way, shape, or form.  Hope that eases some concerns.

This tickled me.

Just wait 'till you see the tickle monster!  Actually I'm glad that this happens to jive with the NLP code and maybe in some deeper psychological sense.  I was just basing this off prior gaming experience and instinct, really, but hopefully this is indicative of good training and not blind luck.  Since I first starting making these changes, Ruiting and others have been running me through NLP a bit, so now I've got a better understanding.  The Tell high-level action descriptor in particular was designed for the eventual inclusion of Ruiting's NLP code.

Yesterday Nicky showed me that we can setup actions in a piecewise manner (telling each limb how to animate separately, etc.), so I'll be updating the low-level descriptors to match that.  With any luck, the RoboKind API will share many analogous low-level action descriptors or there'll be a strong correspondence between the BML framework they're using and the Behaviour Trees that I'm designing now.

Regards,

-Lake