Jmol and SparshUI

[Robert Hanson]

the following link is a little YouTube demo of using Sparsh-UI with Jmol on the HP IQ600 under Vista. Very preliminary -- still a few kinks -- but I think it shows promise.

http://www.youtube.com/watch?v=7x8uDRQKIg0

Bob Hanson

--
Robert M. Hanson
Professor of Chemistry
St. Olaf College
1520 St. Olaf Ave.
Northfield, MN 55057
http://www.stolaf.edu/people/hansonr
phone: 507-786-3107


If nature does not answer first what we want,
it is better to take what answer we get.

-- Josiah Willard Gibbs, Lecture XXX, Monday, February 5, 1900

[Jay Roltgen]

Bob,

Very nice!  To my knowledge, you're the first person to have implemented Sparsh-UI in a browser environment.  This is a fantastic contribution.  Just a few ideas for questions you raised in the video:

I would suggest that sometimes when the app does not seem to be responding, it is simply the device driver that is not picking up the touch points reliably.  You seemed to be moving your fingers usually in a "diagonal" motion, that which is the least accurate on the HP touchsmart (because the touch sensors are in the upper corners). 

Also, one of the (very few!) current limitations of Sparsh-UI is that it does not know how to handle multiple clients that may be obscuring each other.  In order to determine which client application was "on top," Sparsh would have to communicate with the OS windowing environment, and we never implemented it.  I noticed that when one of the browser apps didn't seem to be responding, jmol was actually behind it.  It's possible that jmol was stealing the sparsh-ui events from the applet, and that this wasn't a firefox problem.  Maybe give that a shot.

This is great to see this working for your application!  We have been considering making a page either on the google group or the googlecode website of "people using Sparsh-UI" and were wondering if you would be willing to be included on that page (just your name, affiliation, and link to website).  Let me know.

Thanks,

Jay Roltgen

--

You received this message because you are subscribed to the Google Groups "sparsh-ui" group.
To post to this group, send email to spar...@googlegroups.com.
To unsubscribe from this group, send email to sparsh-ui+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sparsh-ui?hl=en.

[Robert Hanson]

Jay,

On Thu, Dec 10, 2009 at 3:55 PM, Jay Roltgen <jrol...@gmail.com> wrote:

Bob,

Very nice!  To my knowledge, you're the first person to have implemented Sparsh-UI in a browser environment.  This is a fantastic contribution.  Just a few ideas for questions you raised in the video:

Oh, boy! Excellent!
 

I would suggest that sometimes when the app does not seem to be responding, it is simply the device driver that is not picking up the touch points reliably.  You seemed to be moving your fingers usually in a "diagonal" motion, that which is the least accurate on the HP touchsmart (because the touch sensors are in the upper corners). 

My testing suggests it's just my algorithm. Very crude -- but I'll look into that. I can just start

JmolMultiTouchDriver.exe -test > data.txt

and collect reams of data. That whole session was captured. both from the driver end and the Jmol/Gesture server end.
 

Also, one of the (very few!) current limitations of Sparsh-UI is that it does not know how to handle multiple clients that may be obscuring each other.  In order to determine which client application was "on top," Sparsh would have to communicate with the OS windowing environment, and we never implemented it.  I noticed that when one of the browser apps didn't seem to be responding, jmol was actually behind it.  It's possible that jmol was stealing the sparsh-ui events from the applet, and that this wasn't a firefox problem.  Maybe give that a shot.

This appears to work fine. At least for the applet/application I did a test with them in front of each other -- actually in this demo if you carefully, that's the case -- and it works fine. As part of the getGroupId() call, I just check to see if the owned window has focus. Nothing more than that. That said, there is a problem with applets that are hidden by the tab system of Firefox. I still have to look into why that is and see what kind of cross-communication is happening there.
 

This is great to see this working for your application!  We have been considering making a page either on the google group or the googlecode website of "people using Sparsh-UI" and were wondering if you would be willing to be included on that page (just your name, affiliation, and link to website).  Let me know.

Sure. The details about the best way to implement this are a bit sketchy. Right now you have to have JmolMultiTouchDriver.exe and NWMultiTouch.dll in a directory on the HP TouchSmart and then manually start JmolMultiTouchDriver.exe with the -testsocket option so that it goes into a cycle mode waiting for a server. (Then, if it loses the server, it just starts cycling again until you press ESC.) We can change the flags and default behavior to whatever you think is reasonable.

By the way, is there a simple way to tell a C++ program to close out when its DOS window is closed? If so, I'll add that.

We're still seeing a few bugs where single-touch gestures are lost, but I should be able to figure out what's going wrong there.

Bob

 

[Stephen Gilbert]

Nice work, Bob! Keep those demos coming!

Stephen

--
Stephen B. Gilbert, Ph.D.
Associate Director
Virtual Reality Applications Center
Human Computer Interaction
Research Assistant Professor, Psychology
Iowa State University
515.294.6782

[Robert Hanson]

Latest executables and sample files are at http://chemapps.stolaf.edu/jmol/docs/examples-11/Jmol-Sparshui-Demo-v0_3.zip

Question: Did you ever ask the engineers at NextWindow why they didn't produce a timely TOUCH_UP event? That one little issue has caused me (and Jay and Tony, I think) untold hours of grief.

Bob

--

You received this message because you are subscribed to the Google Groups "sparsh-ui" group.
To post to this group, send email to spar...@googlegroups.com.
To unsubscribe from this group, send email to sparsh-ui+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sparsh-ui?hl=en.

[Jay Roltgen]

Hey Bob!

Sorry for the delay in responding to you - it's been a little busy here with finals and such.  We didn't ever ask why the folks at NextWindow why they didn't do this - but this is a deeper issue than NextWindow:

I was actually out with some folks from the VRAC at a bar a few months ago and found out that a peer had done some stuff with HP touchsmarts too - he found that there is an HP service that runs on top of the nextWindow driver and acts as a sort of "moderator" for the touch events.  He discovered that if you disable this service, you receive timely death events.  I tried getting this to work, and I was able to get it to work with the sample code nextWindow provided, but I still had the same problems with my own code.

I don't remember the name of the service, but it should have either "hp" or "touch" in the name somewhere.  Good luck.

~Jay

[Robert Hanson]

oooh, good hint!

[Robert Hanson]

Stephen, an update

An adapted SparshUI interface will be going into a new Nanooze (http://www.nanooze.org/) exhibit at the Epcot Center in a couple of weeks. We're just putting the final touches on the graphical interface. The name of the activity is "Touch a Molecule", and when it is up and public, I will certainly send you more about it.

What I can say now is that we are driving multiple windows within the Firefox browser, and everything is as slick as can be. Each applet has the capability of starting (and restarting) a server, and the driver is set up to cycle when it loses a connection, trying to find a new connection once a second.

So when a page is closed, if any other page present on the machine needs service, it starts its own gesture server and locks in the connection with the driver. Doesn't matter which applet locks in first; the rest can adapt and be clients only.

The process is initiated by having a parameter  multiTouchSparshUI="true" within the applet tag or


?PARAM=multTouchSparshUI:true

on the web page. The nice thing about this is that once this version of Jmol gets distributed, you can use multitouch on your machine at any remote Jmol site just be having that in the command line. In fact, right now you can go to any site and use the following on the command line and have instant multitouch service (with the driver running on your machine, of course!)


[any web page running a Jmol applet]?PARAM=multiTouchSparshUI:true&JMOLJAR=http://chemapps.stolaf.edu/jmol/docs/examples-11/JmolAppletSigned.jar


Because I didn't want this to interfere with your "real" SparshUI system, I set it up to operate on a client port of 5946 and a driver port of 5947. Hope you approve.

I'm only finding one glitch -- If you navigate from one page having a Jmol applet directly to another, so the multitouch server has to shut down and start again, then it can happen that the driver misses that and doesn't try to reconnect. Do you know how I can send a message to the driver saying, "gesture server shutting down now!" ?

Bob

On Thu, Dec 10, 2009 at 8:27 PM, Stephen Gilbert <stephen....@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "sparsh-ui" group.
To post to this group, send email to spar...@googlegroups.com.
To unsubscribe from this group, send email to sparsh-ui+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sparsh-ui?hl=en.

[Jay Roltgen]

Bob,

Cool!  We're excited to have Sparsh-UI in the public's eye like this :)

Sure, it's pretty easy to send a message to your driver.  You can simply use the socket connection present in Sparsh-UI and the input device to send it the message.  Since you're not currently sending data to the input device at all, even a single byte sent would be enough to flag it to shut down.

To do this, you'll need to modify the InputDeviceConnection class to have a method that is called serverShutDown() or something.  In that method, just write a byte out on the socket (see the ClientProtocol class for an example).

In your driver, you'll need to start another thread that listens for the byte - basically, you'll create another thread (similar to the touchpointkiller) and then, from within that thread, call read() with a length of 1 - see the MSDN socket documentation (here's a good starting point: http://msdn.microsoft.com/en-us/library/bb530746(VS.85).aspx)

Then, you just check to make sure the byte you received was the one you were expecting from Sparsh to shut down, and then do whatever "restart" operation you like.

Good luck!

~Jay