GUI for drag and drop (blocks) scripting off-world

[Josmas Flores]

Hi guys, how is it going? I'm working on a project and would like to
know if you guys (especially Morris) think if it is feasible or I'm
way too far off!

I'm planning to use npc and scripting modules to create a programable
bot for educational purposes; think about soft skills, or actors in a
simulation, or stuff like that. The problem is that for most
educators, the idea of installing wonderland and the needed modules,
and start scripting (in javascript or whatever), it's just way too
much. Some of them wouldn't have the skill to do it, some others won't
have the time or interest.

So here comes my project. I'm planning to create a little interface 'a
la scratch' (http://www.youtube.com/watch?v=ix6Hlt8xL3E&feature=related)
to allow to write scripts by dragging and dropping blocks on-screen.
It will be quite simple, at least at the beginning, and I'm planning
to use jquery ui for that. The output of the 'script window' (or div
in this case) will be translated into a series of scripts. I'm going
for a web interface cause it can be hosted (one thing less to install
for educators!).

I am planning on packing up the resultant scripts and write a content
importer that will place the scripts in appropriate places. I think
the structure of the scripts is quite fixed and they live in the user
area, so it should be possible to do... is this right?
I am only in planning stage so I will have to go back to the scripting
tutorials and start messing with the npc modules! And of course, it
will not be all that straight forward in the end.

There are examples of this kind of GUI all over the place, such as the
already mentioned scratch or scratch4SL which translates into LSL (SL
and opensim scripting). So I guess it could also be used to teach
introductory programming, but my target audience right now is
educators, and limited to bots (not general scripting). Making it more
general/flexible would be way to much for me, and I would want to have
it working before the next semester (wishful thinking!)

I'll push the GUI project and the importer to my github when I start
messing with code (hopefully this very week!) so feel free to have a
look if you fancy.

Any feedback will be much appreciated!

cheers,
José

[Nicole Yankelovich]

José,

Hi. It's great that you're planning to tackle this project. While the block scripting approach might work, I would suggest that you at least consider some alternative UI mechanisms. The place to start is with some user stories. Can you articulate a number of different things educators might want to do with bots? Here are some things on my list:

• Greeter - have an automated avatar greet visitors to a space. This involves a stationary avatar with recorded audio and gestures.
• Presenter - an automated avatar, potentially representing the instructor, that gives a presentation synchronized with multi-media effects like slideshows and video. This would involve a mostly stationary avatar, recorded audio, and triggers for multi-media events like changing slides or playing a segment of video.
• Tour Guide - have an automated avatar escort visitors along a fixed path. This involves a sequence of movements from location to another, audio and gestures along the way, and audio and gestures at each stop along the tour. A moving platform that visitors could ride on to follow the tour guide might be a nice touch.
  
• Demonstrator - bot that shows visitors how to do or use something. This involves an animated avatar that performs a sequence of actions, synchronized with recorded audio.
• Conversational Partner - particularly for language learning applications, have an automated character that can respond to speech input. This involves the use of a speech recognizer, a recognition grammar, a dialog flow, text-to-speech or canned responses, and corresponding animations.

The first three seem readily doable to me. The last two are much more difficult. Do you have some additional use cases in mind?

For the simple cases, you might want to consider an in-world "bot wizard." In other words, a structured sequence of dialogs that walk you through the creation process. First, select an avatar. Configure clothing, etc. Next select a behavior template (greeter, presenter, tour guide, etc.). Depending on the template, the next steps would be different. You would need to be taken through recording or uploading audio, specifying animations and triggers, demonstrating a path for the bot by moving your avatar along the desired path, and so forth. What you end up with is a script behind the scenes, but from the UI perspective, you end up with "bot properties" that can easily be updated or modified.

Nicole.

[Morris Ford]

Hi José,

Not only is what you are talking about feasible but most of the mechanisms necessary to implement automated bots has been created. The documents and tutorials for scripting are here:

https://sites.google.com/site/openwonderland/tutorials/scripting

The file scriptingClass.pdf is a first tutorial and goes through most of the basics of the scripting component.

The files robotScripting2.pdf and robotScripting.pdf document the robot control script mechanism.

The file WlAvatarPath.pdf documents the mechanism for having avatars follow paths.

Now, some explanation.

I created a 'script editor' for managing the creation, testing and maintenance of scripts in order to facilitate this process. Before this it was necessary to manually create the scripts in an external editor and then place them in the proper locations on the WL server. The original version of this editor is explained in the first tutorial. When I started working with automated bots and then later with the avatar path following, I added a 'tab' to the script editor to allow for the creation of these two types of scripts. The scripts generated out of this tool are not scripts in the sense that javascript scripts are scripts but are files of purpose specific steps. The WlAvatarPath.pdf file documents the use of this tool for creating the path following scripts. I have not finished testing or documenting the bot script part.

The bot scripting that I created allows for all of the types of bot functionality that Nicole has mentioned save the voice recognition in the Conversation Partner. I have created and tested bot scripts that do these things so I know that they work. 

As for the creation path. The philosophy that I have about doing anything with scripting is to use scripting to do it. As such, the script editor is a script and is implemented in java as a script in the jsr223 sense. The method you described of using external (to WL) web interfaces, while workable, removes to extra-wonderland the whole process and makes the create, implement, try, modify, try, etc. iterative loop more clumsy. I would highly recommend implementing the creation interface by modifying what I did or creating some other script inside the scripting component structure. If you wanted to be really adventuresome, you could implement a new special purpose scripting language for this creation process that complies with the jsr223 interfacing specs and run that inside WL.

I don't want to burden you with too much info if you're not interested. I would be glad to continue the conversation at your leisure.

Morris

[Josmas Flores]

Hi guys, thanks so much for the replies.

Nicole, the user stories are a great start and I've been working on
that for a bit. I have sorted a couple of interviews with educators to
help me finalise basic requirements and I was planning to use those
stories as examples to pack with the GUI.

Morris, I went through the scripting classes when they were released
earlier this year, but I still have to catch up with all the bot
functionality. You've done a lot of great work in this area and it
will be great to reuse as much as I could.

You both seem to agree that working in-world would be a better fit.
The wizard sounds great but unfortunately I do not have the time (or
knowledge) to create a module like that.
The scripting editor is a good option but it might be out of reach for
some of my audience, and my intent here is to make the creation of
bots as easy as possible. The idea of the externally hosted,
block-dragging interface is to cater for my target audience: educators
that might know very little (to nothing) about programming.

Regarding communication with the exterior (conversational robot), my
plan is to finish the restful communicator module that I started
earlier in the year (and then abandoned!). That would allow the bot to
communicate with any external service with a REST interface, but then
again, this is a lot of wishful thinking on my part!

I have a tendency to like tools that can be used out of the world, and
that's why my first OWL project attempt ever was the (unfortunately
unfinished!) restful communicator. I generally see advantages in two
fronts:
1. From a development point of view I can work on the tool without
depending on OWL; development is faster, testing is faster; deployment
is easier, and hosting is very flexible.
2. From a users point of view, they will not depend on the world being
available to work with the tool.

The tricky bit then is connecting both sides. But OWL comes with a lot
of goodies like importers that you can extend and make that connection
happen in some cases.

I understand what Morris mentions about the coding cycle for the bot
(trial and error) being clumsy. I do agree completely and should put a
bit more thought on that. I could try to script the drag and drop
mechanism within the scripting editor but that would be very
adventurous, and I really need at least a working proof of concept for
next semester.

What I really need to do is to take a year off work and do all the
stuff that I really want to do!!! If only bills would pay for
themselves! :D

thanks again for the input guys, really appreciated!
José

[Morris Ford]

José,

A few points.

The scripting editor section for creating bot 'scripts' requires no programming knowledge. It basically is; move your avatar to a location, select animations, audio, chat output and/or object synchronization messages for that spot then record the step and continue until the whole 'presentation' has been laid out. It would be simple enough to add avatar selection to this process and to create a set of generic bot operation scripts. 

For me, a huge overriding consideration is that this process is going on in-world and the development cycle becomes create a step or two, try it, change it, try it again, and move on through the process. The whole idea of the virtual world is subverted by moving processes such as this 'outside'. Another issue is that creation of a 'presentation' such as this out-world will eventually require an emulation of the world to try out the bot, in other words, creating a WL instance inside the creation tool. Trying to do this out-world really becomes almost impossible when multiple object synchronizations are involved in the 'presentation'.
Morris

[Josmas Flores]

Hi Morris, I do remember trying out that GUI now; sorry about that, I
have to take a look at those modules asap.
I also agree with the problems with presentation dev cycle, and I was
not planning to add an instance of the world for trying it out
although I do agree that would be the way to go. I was thinking more
of a very limited preview, and animation with a 'dot' showing the
location of the bot and the interactions, but I'm aware of the
limitations of this approach.

thanks for the input again; will have a go at the modules and I will
probably have a ton of questions after that; if you don't mind me
disturbing you with all this of course! :)

cheers,
José

[Morris Ford]

Anytime.

Morris

[Josmas Flores]

Hi Morris, just gone through the scripting tutorials(except for the
walk and the bug ones) and although I have a couple more questions,
will start by just 2:

1. Is it still needed to modify the xml files to get the avatars named
correctly?
2. what does ICE stand for?

cheers,
José

[Morris Ford]

Yes on the avatar file change. All the avatars have the same name but to sort out the animation events a unique name is required.

ICE is inter-cell event, the event used for all the communications between cells.
Morris

On Nov 9, 2010 8:36 PM, "Josmas Flores" <josmas...@gmail.com> wrote:

Hi Morris, just gone through the scripting tutorials(except for the
walk and the bug ones) and although I have a couple more questions,
will start by just 2:

1. Is it still needed to modify the xml files to get the avatars named
correctly?
2. what does ICE stand for?

cheers,
José

On 9 November 2010 13:08, Morris Ford <morri...@gmail.com> wrote:
> Anytime.
> Morris
>

> On Tue...

[Josmas Flores]

Thanks Morris. it makes sense (both answers) but have to take that
into account if I want non-programmers to use this stuff! :)

Will go through the 2 remaining documents today (hopefully). I have a
bunch of notes from my experiments (also had a couple of mates going
through the classes with me), would you be interested in the feedback
about the documentation? I can send it privately if you want to have a
read.

I started documenting some of the process in the community wiki
(thanks Maggie for adding the Bots_101 page!):
http://faithhold.dyndns.org:8080/JSPWiki/Wiki.jsp?page=Bots%20101

cheers,
José

[Morris Ford]

José,

I would be interested in any feedback that you have. No need to send it privately unless you want to do it that way. I have read what you have done so far in the wiki and I think it is quite good. 

The way I think of the scripting environment is as a layer on top of the basic client that facilitates the creation of dynamic content. My impression of what you are working toward is a higher level layer on top of the scripting layer for the creation of bots. That matches my philosophy quite well. I would love to see what you create be a tool inside wonderland but that is your choice. I will help you in whatever way I can.

Morris

[Josmas Flores]

Morris,

Yes, your description is spot on. Scripting is great to create content
but it is targeted to developers. Your Robots tab is a great way to
abstract at a higher level, but it is still quite technical (and this
is not a critique, I think you are doing an awesome job).
I am aiming to abstract even further and allow non-technical people to
mess about with the bots.

I would also agree that a tool within the world would be the way to go
but I'm a bit worried about finding problems that will stop me from
getting anywhere (and this is ultimately my own limitations!). So I
might start outside and then translate it into an in-world module at
some stage when I have something already in place.

A couple of questions about the Bug tutorial. You are using Maya and
then exporting to collada... would it be possible to use other tools,
say blender?
Or is the modulator module specific for Maya output? I am assuming
that the modulator module is only used to create this kind of animated
art, right?
And the last one... You are scripting the animation in Maya, but also
scripting the object through the scripting component... I thought the
animations were embedded in the dae file itself... I'm obviously
missing something here but I don't quite get it.

cheers,
José

[Morris Ford]

José,

I agree about the level of expertise required to utilize the scripting and the Robots stuff and my intention was to create a layer that made it easier compared to doing the same things in java code and to facilitate the creation of more layers on top like what you are doing. 

About the bugs and other animations. I used Maya because I had access to a copy, I had a little experience with it and the art path from Maya->ColladaMaya->Wonderland had been used before for more than static objects. The code changes I made inside the wonderland loader for collada files were to parse out the animations and insert them into the scene graph inside wonderland. I am certain that the same thing, in the sense of enabling another art path, could be done, eg, with Blender and the end result in terms of what ended up in the scene graph would be the similar if not the same. I built the Modulator module (underlying module for bugs, clocks, etc) to take the model information including animations from the scene graph with the assumption that if a different art path were utilized the information in the scene graph would be laid out the same.

The relationship between the scene graph animations and the scripting component is that the model animations animate the object and the scripting instructs the module about when and how to animate. The inclusion of scripting in this sense furthers the idea of a dynamic world where many objects operate in concert. Also, the scripting module is not doing any of the animation directly, the modulator code parses the animations out of the scene graph and then waits for instructions to do the animations. In this sense a modulator module could be modified to have the instructions built in to do the animations and not have a scripting interface.

The issue I was dealing with when doing this stuff was to allow complex animations. Complex in the sense of translations, rotations and scalings of multiple nodes. I was unable to achieve such complexity using the basic animations of a cell and especially when I wanted to animate more than one node. For example, to animate an aircraft rolling into and out of a turn while changing altitude was completely out of reach for me (add a propellor and I start having migraine issues) but I can animate that in a tool such as Maya and activate it in wonderland. 

Hope that helps a bit.

Morris

[Josmas Flores]

Morris, thanks for the explanations; it helps a lot!

I'm following the avatar path tutorial and I cannot write to a file;
after creating the path I click on 'write path file' and I get this on
console:

ScriptingComponent : Cell 3 : Script file
http://134.226.112.171:8080//webdav/content/scripts/cubo/startup.js
not found
10-Nov-2010 16:23:13
org.jdesktop.wonderland.client.cell.ChannelComponent messageReceived
WARNING: No listener for message class
org.jdesktop.wonderland.common.cell.messages.CellServerComponentMessage
from cell org.jdesktop.wonderland.client.cell.ModelCell status VISIBLE
Inside setScriptContext
10-Nov-2010 16:27:20
org.jdesktop.wonderland.modules.webdav.common.WebdavContentCollection
getChild
WARNING: In path paths element paths not found
Exception for contentWriteFile
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at properties.contentWriteFile(Unknown Source)
at properties.jButtonWriteFileActionPerformed(Unknown Source)
at properties.access$1500(Unknown Source)
at properties$17.actionPerformed(Unknown Source)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
at java.awt.Component.processMouseEvent(Component.java:6267)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
at java.awt.Component.processEvent(Component.java:6032)
at java.awt.Container.processEvent(Container.java:2041)
at java.awt.Component.dispatchEventImpl(Component.java:4630)
at java.awt.Container.dispatchEventImpl(Container.java:2099)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4577)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
at java.awt.Container.dispatchEventImpl(Container.java:2085)
at java.awt.Window.dispatchEventImpl(Window.java:2478)
at java.awt.Component.dispatchEvent(Component.java:4460)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:599)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: java.lang.NullPointerException
at org.jdesktop.wonderland.modules.scriptingComponent.client.ScriptingComponent.contentWriteFile(ScriptingComponent.java:1177)
... 33 more

Any idea what is going on? I'm using Ubuntu32.
Is the module supposed to be writing to the global webdav space? any
permission issues maybe? I tried unchecking the 'Global Webdav' box
but same output.

cheers,
José

[Josmas Flores]

Tried in a mac client against the same Linux server and I get the same problem.
The modules are installed from source and both wonderland and
wonderland-modules are up to date.

cheers,
José

[Josmas Flores]

Bit more on this; the directory 'paths' did not exist at: ...run/content/paths/
I have created it and given 777 permissions but still not able to
write; I guess the permissions come from webdav?

the exception in the console is gone but the server output looks like:
[WebdavServlet] Permission denied writing
/home/jos/.wonderland-server/0.5-dev/run/content/paths/path_1 by jos

cheers,
José

[Morris Ford]

The user you are running has to have admin privileges to write to global webdav. The paths really need to be in global due to the different users involved.

Morris

[Morris Ford]

Also, the bot generation path of that screen has a few problems. I am working on it.

[Josmas Flores]

Great stuff! you knew about it anyway.
when you say 'admin' does that mean i need to install authentication?
i've never messed with groups/auth and son on, so i'm a bit lost that
end... but will give that a go after the dev meeting.

thanks for all your help Morris!

[Josmas Flores]

Got it working! it is really cool!
Have to make a case for it now... have a bit of pressure to work with
http://www.openmetaverse.org/projects/libopenmetaverse instead; so
have to come up with something clever... really want to do this with
OWL.

Morris, will probably send the feedback directly cause I don't want to
spam the list.

thanks again!
José

[Morris Ford]

I assume you mean the avatar path building part. The bot program creation part will be more or less the same but a bit more complex. Anything I can to do to keep you from having to go to the dark side (.net) let me know.

Morris

[Josmas Flores]

yeah, only followed the document... you mention a different doc for
the complex one but not sure that's available?

and YES! the .net issue is a big one... I tend to look at things on a
positive way and think about it as an opportunity to mess about with
ironruby or ironpython, but won't go there if I can avoid it!!! :)

cheers,
José

[Morris Ford]

The document for the bot program creation is in process. Maybe tomorrow.

On Nov 10, 2010 7:30 PM, "Josmas Flores" <josmas...@gmail.com> wrote:

yeah, only followed the document... you mention a different doc for
the complex one but not sure that's available?

and YES! the .net issue is a big one... I tend to look at things on a
positive way and think about it as an opportunity to mess about with
ironruby or ironpython, but won't go there if I can avoid it!!! :)

cheers,
José

On 11 November 2010 00:23, Morris Ford <morri...@gmail.com> wrote:

> I assume you mean the avata...

[Josmas Flores]

Oh, thanks for that, no rush... I've a ton of notes to write out
neatly and probably won't have much time till the weekend anyway. so
please take your time... I will give it a go if I have time anyway...
see if I can work it out! :)
time for bed long gone now so see you all tmw; have fun!

José