Client UI for VWs

[Ryan McDougall]

Summer vacation is over for us here, and we're starting up the second
half of 2009 very soon.

One of the topics that will be coming up for us is making "The Right"
UI for our client viewer, and I wanted to ask the collected wisdom of
this list what they are doing or thinking is the right way forward.

The problem with VWs is they are multi-use. They host an application
(the way a web browser hosts a web app), and must provide a means of
*both* supporting the application with a structured framework to build
on, and allowing it the freedom to draw application specific elements.

Moreover, what elements are required for a successful VW UI? Should
there be an bookmark menu and address input bar, or tabs, like a web
browser? Or does inspiration come more from video games?

For us, we have decided to use Qt as a framework for creating
graphical elements, due to Qt's excellent JavaScript integration,
which will allow us to "farm out" many UI tasks to HTML5+JavaScript.

However I feel that's just the technological underpinning, and we owe
it more to our users to make a coherent interaction design of some
kind.

One example of a web-driven UI that looks really good is Palm's Pre.
It's basically WebKit rendering HTML5+JS based UI elements called
"activity cards":
http://www.palm.com/us/products/phones/pre/index.html

However, to make it feel like an "native app" -- not just a web
browser, they've had to make a special purpose "window manager", which
is at the heart of webOS:

http://developer.palm.com/index.php?option=com_learn&view=learn
http://developer.palm.com/index.php?option=com_content&view=article&id=1772

Palm appears to be writing an O'Reilly book based on it's webOS,
starting here: http://developer.palm.com/index.php?option=com_content&view=article&id=1761

I feel that design an VW client UI essentially boils down to writing a
small application-specific window manager. Something that accepts
requests to show new UI elements, instantiates the correct objects,
manages layout with competing UI elements, handles input events, and
composites the UI over the 3D world.

Do you agree? If so what form should it take?

Cheers,

[Ryan McDougall]

Forgot to link the community dev page for webos: http://www.webos-internals.org/

[Ryan McDougall]

On Thu, Aug 13, 2009 at 9:49 AM, Ryan

Infoworld article on webOS/mojo:
http://infoworld.com/d/mobilize/palms-mojo-sdk-pre-mixes-simplicity-rough-edges-032

[Hurliman, John]

Fortunately, this is one problem that you don't have to get right the first time. As long as all of the UI components are fully scriptable and replaceable, the community will figure out what the right interface(s) are.

John

[Ewen Cheslack-Postava]

That depends on what you allow users to do with the UI. Its true that
if all of your UI components are static or loaded from well known
names from the network, then upgrading can make sense. We've also
discussed the use case of users being able to create scripted UIs for
their objects. While you can still upgrade and fix bugs, there's a
much bigger burden for support moving forward. Its like the
difference between a game being able to just test and upgrade their
scripts vs. the compatibility nightmare involved with user created
scripts.

Sirikata is going the web based route for a number of reasons - a lot
of implementation is handled for us, backwards compatibility is
limited to what *we* add to integrate with our system, since the
browser should deal with rendering backwards compatibility, and its a
well understood system for generating UIs.

-Ewen

[Ryan McDougall]

Ewen, that's all quite true, and mimics fairly well our own positions,
however the sad thing is I know that that's not enough to launch a
consumer product with. There is a whole field of interaction
design/user experience that I admire greatly although I admit I've not
training/experience in it. We can't just say "hey designer, do 'it' in
HTML5" and wash our hands of it -- clearly Palm Pre didn't do this.

The fact that we are server or graphics geeks means we are missing out
on a whole bunch of what real people need to have usable software, and
it's a shame. I think it's a major reason why VW aren't nearly as
popularly successful as we think they should. :(

Cheers,

[Ewen Cheslack-Postava]

You're right, and I don't think we should just say "go build it in
HTML5." At a minimum, there's plenty of interaction between the
browser and the rest of the system, at worst we admit that raw HTML5
is not great for constructing UIs and we should provide a higher level
abstraction.

I'm actually struggling with this question right now, as we try to
decide what UI stuff for an application should live in sirikata's
repository and what should live in the app's repository. The reason
I'm struggling is that I can imagine more than 2 layers here. I tend
to think of this like the X - Gtk/Qt - App layering. The browser
provides the basic facilities for drawing and interacting with the
rest of the system. Something above that may provide something more
akin to Gtk or Qt, a set of widgets that is easily extended and makes
constructing the application significantly easier. I think a lot of
this is non-specific to the applications, so I would prefer to reuse
existing tools - see YUI for example
(http://developer.yahoo.com/yui/). Then the top layer actually
constructs the UI the connects it to whatever application level logic
is necessary. Of course we always come back to the interoperability
question and whether having more than one option for that middle layer
hurts interoperability, even between instances of, say, Sirikata-based
worlds.

Of course there's also everything that falls "outside" the top level
UI elements, i.e. the "window manager." This concerns me less because
ideally its mostly irrelevant to the UI designer aside from a small
amount of interaction (ability to pull up new window, get resize
events, etc).

-Ewen