New multitouch patch, fully backwards-compatible and low impact

[luke]

I have put together a new patch for supporting multitouch scaling on
the G1.

http://web.mit.edu/~luke_h/www/Android_MultitouchV2.zip (driver
source, boot.img containing a rebuilt kernel, and a demo app .apk all
included)

This version is much less invasive than the previous patch. I pushed
all multitouch support into the synaptics driver, and only have the
driver report a single event for multitouch situations, consisting of
the x and y coords set to the midpoint between the two touch points,
and the size field of the event set to the distance between the touch
point and each finger (i.e. the radius of stretch). Size == 0 when
there is only one finger on the screen. This patch is 100% compatible
with all current apps and the Android event pipeline, as far as I can
tell. Apps can be written to detect multitouch events by checking the
size field, and if it is > 1.0 (the max value returned by the current
driver, although this value appears to just consist of noise
currently), then the program can initiate a multitouch operation.
Under the new system, the size values stored in MotionEvents are in
screen coordinates just like x and y.

From a user standpoint, this patch could easily give multitouch
scaling in MapViews and WebViews, but does not allow for more
complicated operations such as rotation (see my previous post for the
reason why).

Comments please from core framework developers? I understand a driver
was checked into git for a "real" iPhone-style multitouch screen a
while back, but there doesn't appear to be any UI code checked in for
it yet, and obviously there is no announced device. Could something
like this be pushed upstream in the meantime though to at least
support multitouch scaling on the G1? Or is this unlikely to ever
pass?

Thanks,
Luke Hutchison

[Dianne Hackborn]

Hi, this is really cool work, but I don't think we would accept something like this as a patch.  At the very least, re-using the size field to report pseudo-multi-touch information is a pretty big hack.

The API I would expect to see for multi-touch would be something along the lines of a new set of action codes for when additional fingers are down, moved, or released, and a new field on MotionEvent for which finger that event is for.  (So it would always be 0 for the current action codes, and some value 1-n for the new action codes, where n is the maximum number of fingers that can be detected.)  Then each finger would be a new distinct MotionEvent stream, which would be delivered separately and the ones with the new action codes would be ignored by existing applications.

--
Dianne Hackborn
Android framework engineer
hac...@android.com

Note: please don't send private questions to me, as I don't have time to provide private support.  All such questions should be posted on public forums, where I and others can see and answer them.

[Jean-Baptiste Queru]

On the other hand, if the panel can track all fingers in a single
processing pass, it'd be unfortunate to report that as multiple
events: at some point in time the app would be in a state that doesn't
match anything that the screen has ever read, i.e. where it has the
new location for one finger but the old location for the other one.

JBQ

--
Jean-Baptiste M. "JBQ" Queru
Android Engineer, Google.

[luke]

What is the size field intended for currently?

How do action codes work? Can you just specify which codes your app
is listening for? It sounds like you are describing some sort of "opt-
in" filter system.

On Jan 13, 5:01 pm, "Dianne Hackborn" <hack...@android.com> wrote:
> Hi, this is really cool work, but I don't think we would accept something
> like this as a patch.  At the very least, re-using the size field to report
> pseudo-multi-touch information is a pretty big hack.
>
> The API I would expect to see for multi-touch would be something along the
> lines of a new set of action codes for when additional fingers are down,
> moved, or released, and a new field on MotionEvent for which finger that
> event is for.  (So it would always be 0 for the current action codes, and
> some value 1-n for the new action codes, where n is the maximum number of
> fingers that can be detected.)  Then each finger would be a new distinct
> MotionEvent stream, which would be delivered separately and the ones with
> the new action codes would be ignored by existing applications.
>
>
>
> On Tue, Jan 13, 2009 at 1:53 PM, luke <luke.hu...@gmail.com> wrote:
>
> > I have put together a new patch for supporting multitouch scaling on
> > the G1.
>

> >http://web.mit.edu/~luke_h/www/Android_MultitouchV2.zip<http://web.mit.edu/%7Eluke_h/www/Android_MultitouchV2.zip> (driver

> hack...@android.com

[Jean-Baptiste Queru]

Another problem is that if you put a finger down (finger 0), then
another one (finger 1), then lift the first one, the only finger being
tracked is now finger 1, and that might not work that well with
existing apps.

JBQ

[Dianne Hackborn]

On Tue, Jan 13, 2009 at 2:08 PM, luke <luke....@gmail.com> wrote:

What is the size field intended for currently?

It is the size of the finger on the screen, for ex tip or thumb.
 

How do action codes work?  Can you just specify which codes your app
is listening for?  It sounds like you are describing some sort of "opt-
in" filter system.

No, they are just the code identifying the action associated with the event -- DOWN, MOVE, UP, etc.  The app looks at the code to decide what to do with the event.

--
Dianne Hackborn
Android framework engineer

hac...@android.com

[Dianne Hackborn]

On Tue, Jan 13, 2009 at 2:14 PM, Jean-Baptiste Queru <j...@google.com> wrote:

Another problem is that if you put a finger down (finger 0), then
another one (finger 1), then lift the first one, the only finger being
tracked is now finger 1, and that might not work that well with
existing apps.

Whether this is reported as the first or second finger going up is an implementation detail (in the screen firmware as well as the framework).

[luke]

On Jan 13, 5:06 pm, Jean-Baptiste Queru <j...@google.com> wrote:
> On the other hand, if the panel can track all fingers in a single
> processing pass, it'd be unfortunate to report that as multiple
> events: at some point in time the app would be in a state that doesn't
> match anything that the screen has ever read, i.e. where it has the
> new location for one finger but the old location for the other one.

You are right, but the rate that motion events are passed along makes
this effect almost negligible... There is a lot of messiness like
this in event processing, I have come to discover...

[luke]

On Jan 13, 5:14 pm, Jean-Baptiste Queru <j...@google.com> wrote:
> Another problem is that if you put a finger down (finger 0), then
> another one (finger 1), then lift the first one, the only finger being
> tracked is now finger 1, and that might not work that well with
> existing apps.

Correct. This is actually a hardware limitation, and needs to be
handled with smarts at some level. Apparently the iPhone SDK leaves
it up to the user to handle this.

[luke]

On Jan 13, 5:15 pm, "Dianne Hackborn" <hack...@android.com> wrote:

> On Tue, Jan 13, 2009 at 2:08 PM, luke <luke.hu...@gmail.com> wrote:
> > What is the size field intended for currently?
>
> It is the size of the finger on the screen, for ex tip or thumb.

Not for the synaptics driver -- size information is passed in the
pressure field. Sometimes you can convince yourself the value gets
smaller just as a touchpoint lifts off the screen, but it's a
stretch. Also the range of values [0.0,1.0] are quantized to 15
levels at the driver level. I tried really hard to find meaning for
the value in the size field, and just couldn't believe it was used for
anything meaningful -- and certainly the information is not reliable
enough to do anything meaningful with. It is effectively just noise.

> > How do action codes work?  Can you just specify which codes your app
> > is listening for?  It sounds like you are describing some sort of "opt-
> > in" filter system.
>
> No, they are just the code identifying the action associated with the event
> -- DOWN, MOVE, UP, etc.  The app looks at the code to decide what to do with
> the event.

Oh, those codes. That makes sense as a way of building in multitouch
without creating a major new API.

[luke]

On Jan 13, 5:01 pm, "Dianne Hackborn" <hack...@android.com> wrote:
> Hi, this is really cool work, but I don't think we would accept something
> like this as a patch.  At the very least, re-using the size field to report
> pseudo-multi-touch information is a pretty big hack.

No problem, totally understandable. I will probably put this out
there as a "community hack" and see if it catches on (i.e. whether
Market apps start appearing that use it). I am going to hack together
a map and browser apps that use this for scaling, and people can
choose whether or not to use this on their phone as a workaround,
until multitouch is officially available.

Thanks for answering my questions.

[luke]

Actually I have one more question if anyone can answer it -- does
anyone know for sure if the threat of being sued by Apple really is a
barrier to getting multitouch on Android? I don't need an official
word from Google or anybody else -- I just need to know if this is
just rumor or if it is a real possibility.

Thanks!

[Dianne Hackborn]

On Tue, Jan 13, 2009 at 2:27 PM, luke <luke....@gmail.com> wrote:

Not for the synaptics driver -- size information is passed in the
pressure field.  Sometimes you can convince yourself the value gets
smaller just as a touchpoint lifts off the screen, but it's a
stretch.  Also the range of values [0.0,1.0] are quantized to 15
levels at the driver level.  I tried really hard to find meaning for
the value in the size field, and just couldn't believe it was used for
anything meaningful -- and certainly the information is not reliable
enough to do anything meaningful with.  It is effectively just noise.

If we are talking about accepting a patch (and I need to emphasize, I can't make an promises about anything here, except that the current approach will not be accepted), it can't be just designed around what the synaptics driver happens to currently do.

What you are doing is great work for shimming in a multi-touch demo that touches as little code as possible.  For a patch that would add multitouch APIs, though, the approach is much different and should result in an API that is clear and easy to understand and sufficient for a wide variety of devices.  There would also be other things to deal with, for example a way for an app to find out up-front if the screen is capable of multi-touch.

[Disconnect]

Great work btw :)

And everything I have seen says apple patented -gestures-. EG pinch to zoom, etc. So .. ymmv. But it cannot prevent general multitouch from working (ahem, ms surface) or - likely - even stretch to zoom (prior art)..

[Tanu]

plzzz... can u tell me....
how to install the bootimage file inthe windows environmet....
how to run the multitouch event application........

thanks and regards
Tanuja

> > > Thanks for answering my questions.- Hide quoted text -
>
> - Show quoted text -

[Miroslav Mandov]

> For a patch that would add multitouch
> APIs, though, the approach is much different and should result in an API
> that is clear and easy to understand and sufficient for a wide variety of
> devices.  There would also be other things to deal with, for example a way
> for an app to find out up-front if the screen is capable of multi-touch.
>
> --
> Dianne Hackborn

Could it be possible to add an "option" in so "settings", where the
user could select whether his device supports MT or just a single
touch? In this way Android would have this nice feature for devices
that support it, and be able to work as it is now for devices that are
not MT capable.

[Dianne Hackborn]

For hardward that doesn't support multitouch this would need to always be false, and if it does I don't see why you would want to turn it off, so I'm not sure I the point of a user setting for it...?

[Miroslav Mandov]

On Jan 26, 8:14 pm, Dianne Hackborn <hack...@android.com> wrote:
> For hardward that doesn't support multitouch this would need to always be
> false, and if it does I don't see why you would want to turn it off, so I'm
> not sure I the point of a user setting for it...?

May be I did not explain my point correctly.
My idea was to have the MT functionality in Android OS and at the same
time to keep the existing GUI for devices that are not MT capable.
Android concept is that all applications should be compatible with all
Android devices that could lead to the bad idea to keep it only as
"single touch" platform.
Yes, it could be an "option" that the manufacturer is selecting, or
Android can select it based on the screen drivers.

Unfortunately I just saw that Apple was awarded "Touch screen device,
method, and graphical user interface for determining commands by
applying heuristics" patent:
http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=/netahtml/PTO/search-bool.html&r=1&f=G&l=50&co1=AND&d=PTXT&s1=multi-touch&s2=multitouch&OS=multi-touch+AND+multitouch&RS=multi-touch+AND+multitouch

BTW I am using JF's 1.41 version of ADP1, which implements Luke's MT
and it works quite good.

[Jean-Baptiste Queru]

Please don't discuss patents on this list. Thanks.

JBQ

--

[Dianne Hackborn]

Well yes of course if or when multi-touch is introduced into the platform, you will NOT be able to rely on it being there.  Android will be running on devices without a multitouch screen, you can count on it.

[Bob]

Engadget just had an interesting article that discussed the Apple
"multi-touch" patent, suggesting that pinch/zoom isn't claimed.
http://www.engadget.com/2009/01/28/apple-vs-palm-the-in-depth-analysis/

Please think about soft keyboards when considering multi-touch APIs,
where the problem isn't detecting the distance between two finger, but
rather is allowing a 2nd finger (or thumb) to come down on a 2nd key
before the first finger (or thumb) has been released. The proposed
API to provide only a centroid and the distance between fingers won't
work at all for this sort of application.

I think you'd ultimately want to package this as additional
GestureDetector subclasses, depending on whether the type of gesture
detection wanted was pinch/zoom or multiple/overlapping taps.
Keyboards (and maybe games) would use one class, browsers another.

Providing the right information from a touch panel to drive either
detector might be challenging, especially if the panel provides
'unpaired' x and y coordinates rather than individual touch locations,
but I believe it could be done.

On Jan 13, 4:08 pm, Disconnect <dc.disconn...@gmail.com> wrote:
> And everything I have seen says apple patented -gestures-. EG pinch to zoom,
> etc. So .. ymmv. But it cannot prevent general multitouch from working
> (ahem, ms surface) or - likely - even stretch to zoom (prior art)..
>

[Dianne Hackborn]

Please don't talk about patents on this list, thanks.

[Al Sutton]

Thread already doing the patent thing to death on -developers;

http://groups.google.com/group/android-developers/browse_thread/thread/d1970d9896878f74/3d2f85cc0f4e1205#3d2f85cc0f4e1205

It probably should go to -discuss if anyone wants to continue it.

Al.

Dianne Hackborn wrote:
> Please don't talk about patents on this list, thanks.
>
> On Thu, Jan 29, 2009 at 6:04 PM, Bob <bob....@gmail.com
> <mailto:bob....@gmail.com>> wrote:
>
>
> Engadget just had an interesting article that discussed the Apple
> "multi-touch" patent, suggesting that pinch/zoom isn't claimed.
> http://www.engadget.com/2009/01/28/apple-vs-palm-the-in-depth-analysis/
>
> Please think about soft keyboards when considering multi-touch APIs,
> where the problem isn't detecting the distance between two finger, but
> rather is allowing a 2nd finger (or thumb) to come down on a 2nd key
> before the first finger (or thumb) has been released. The proposed
> API to provide only a centroid and the distance between fingers won't
> work at all for this sort of application.
>
> I think you'd ultimately want to package this as additional
> GestureDetector subclasses, depending on whether the type of gesture
> detection wanted was pinch/zoom or multiple/overlapping taps.
> Keyboards (and maybe games) would use one class, browsers another.
>
> Providing the right information from a touch panel to drive either
> detector might be challenging, especially if the panel provides
> 'unpaired' x and y coordinates rather than individual touch locations,
> but I believe it could be done.
>
> On Jan 13, 4:08 pm, Disconnect <dc.disconn...@gmail.com

> <mailto:dc.disconn...@gmail.com>> wrote:
> > And everything I have seen says apple patented -gestures-. EG
> pinch to zoom,
> > etc. So .. ymmv. But it cannot prevent general multitouch from
> working
> > (ahem, ms surface) or - likely - even stretch to zoom (prior art)..
> >
> > On Tue, Jan 13, 2009 at 5:31 PM, luke <luke.hu...@gmail.com
> <mailto:luke.hu...@gmail.com>> wrote:
> >
> > > Actually I have one more question if anyone can answer it -- does
> > > anyone know for sure if the threat of being sued by Apple
> really is a
> > > barrier to getting multitouch on Android? I don't need an
> official
> > > word from Google or anybody else -- I just need to know if this is
> > > just rumor or if it is a real possibility.
> >
> > > Thanks!
> >
> > > On Jan 13, 5:29 pm, luke <luke.hu...@gmail.com

> <mailto:luke.hu...@gmail.com>> wrote:
> > > > On Jan 13, 5:01 pm, "Dianne Hackborn" <hack...@android.com

> <mailto:hack...@android.com>> wrote:
> >
> > > > > Hi, this is really cool work, but I don't think we would
> accept
> > > something
> > > > > like this as a patch. At the very least, re-using the
> size field to
> > > report
> > > > > pseudo-multi-touch information is a pretty big hack.
> >
> > > > No problem, totally understandable. I will probably put
> this out
> > > > there as a "community hack" and see if it catches on (i.e.
> whether
> > > > Market apps start appearing that use it). I am going to
> hack together
> > > > a map and browser apps that use this for scaling, and people can
> > > > choose whether or not to use this on their phone as a
> workaround,
> > > > until multitouch is officially available.
> >
> > > > Thanks for answering my questions.
>
>
>
>
> --
> Dianne Hackborn
> Android framework engineer

> hac...@android.com <mailto:hac...@android.com>

>
> Note: please don't send private questions to me, as I don't have time
> to provide private support. All such questions should be posted on
> public forums, where I and others can see and answer them.
>

--
======
Funky Android Limited is registered in England & Wales with the
company number 6741909. The registered head office is Kemp House,
152-160 City Road, London, EC1V 2NX, UK.

The views expressed in this email are those of the author and not
necessarily those of Funky Android Limited, it's associates, or it's
subsidiaries.

[Jeff Hamilton]

Please do not discuss patents on any of the lists. Thanks.

-Jeff

[Al Sutton]

So where would you prefer discussions about specific patents and their potential effects on functionality to be discussed?

Al.

[Dianne Hackborn]

Most software engineers working at a company simply can not discuss specific patents in public forums, because of the legal repercussions.  Whenever possible, it is best simply to not know about them.

So we'd prefer if there was no talking about specific patents on these lists.  You can talk about them somewhere else, but many of the people working on Android will not be able to join you.

[mike quinn]

And therein lies the problem.

These patents do nothing but stifle innovation for the fear of legal backlash, the best people to discuss this with are the very people that cannot comment on it.

[Al Sutton]

First off I've put a blog entry up so that this can be discussed
off-list as requested;
http://alsutton.wordpress.com/2009/01/31/will-us-patent-laws-strange-android/

The only reason that software patent discussions don't take place
regularly for most "software engineers at a company" is because for most
of us they are not an issue. There are no legal repercussions because
they are not enforceable in a vast majority of the world.

If the Android team limits Androids functionality based on patents that
are only valid in the US then Android will *never* become a major player
in the mobile OS market because it will get absolutely trashed in any
comparison with products from Nokia, RIM, and all the other companies
who have non-US development shops that ship to markets outside the US.

And if nobody wants to discuss them then you're just ignoring the
elephant in the room, and I was always taught that in law ignorance is
not defence.

Al.

> > > > <mailto:bob....@gmail.com <mailto:bob....@gmail.com>>>

> > > > <mailto:dc.disconn...@gmail.com

> <mailto:dc.disconn...@gmail.com>>> wrote:
> > > > > And everything I have seen says apple patented
> -gestures-. EG
> > > > pinch to zoom,
> > > > > etc. So .. ymmv. But it cannot prevent general
> multitouch from
> > > > working
> > > > > (ahem, ms surface) or - likely - even stretch to zoom
> (prior art)..
> > > > >
> > > > > On Tue, Jan 13, 2009 at 5:31 PM, luke
> <luke.hu...@gmail.com <mailto:luke.hu...@gmail.com>

> > > > <mailto:luke.hu...@gmail.com

> <mailto:luke.hu...@gmail.com>>> wrote:
> > > > >
> > > > > > Actually I have one more question if anyone can
> answer it -- does
> > > > > > anyone know for sure if the threat of being sued by Apple
> > > > really is a
> > > > > > barrier to getting multitouch on Android? I don't
> need an
> > > > official
> > > > > > word from Google or anybody else -- I just need to
> know if this is
> > > > > > just rumor or if it is a real possibility.
> > > > >
> > > > > > Thanks!
> > > > >
> > > > > > On Jan 13, 5:29 pm, luke <luke.hu...@gmail.com
> <mailto:luke.hu...@gmail.com>

> > > > <mailto:luke.hu...@gmail.com

> <mailto:luke.hu...@gmail.com>>> wrote:
> > > > > > > On Jan 13, 5:01 pm, "Dianne Hackborn"
> <hack...@android.com <mailto:hack...@android.com>

> > > > <mailto:hack...@android.com

> <mailto:hac...@android.com <mailto:hac...@android.com>>

[Jean-Baptiste Queru]

Two words: treble damages.

JBQ

--

[Al Sutton]

Two words : No jurisdiction.

US courts can't award and enforce any penalty on a non-US legal entity
conducting business outside the US, so if Android is going to be
modified to comply with US patents when US based legal issues arise you
can be pretty sure it'll lose when it's compared around the world with a
phone not developed in the US.

Al.

[Jean-Baptiste Queru]

We're just following the guidelines established by our legal department.

JBQ

--

[Bradley Young]

And they set that guideline due to two words:

Discoverable email.

Discussions about patents are *not* the kind of thing that you want done via email.  It is quite understandable that Google has this restriction, and they host the list (not only that, but it seems that they are paying employees to participate!).  You should respect that, as you would show respect in someone else's house.

Bradley

On Jan 31, 2009 6:43 AM, "Jean-Baptiste Queru" <j...@google.com> wrote:


We're just following the guidelines established by our legal department.

JBQ

On Sat, Jan 31, 2009 at 6:12 AM, Al Sutton <a...@funkyandroid.com> wrote: > > Two words : No jurisdic...

[Al Sutton]

Don't get me wrong, I understand the restrictions you're under because
you're US based, but my concern is that because you're US based Android
will suffer.

I understand that this conversation has now gone beyond a platform issue
and onto a wider discussion about how US patents could affect android,
which is why I'd suggest any further comments can be posted on my blog
entry (http://bit.ly/UtGy) and are kept away from the Android lists in
case a specific example of a patent is needed to make a point.

Al.

[Jean-Baptiste Queru]

The majority of smartphone OSes have some part of their development
done in the US. In fact several of the most visible members of that
family are developed primarily in Santa Clara County, California.

JBQ

--

[Al Sutton]

Nokia sold 42.4% of Smarphones in Q3 2008[1]. The bulk of their
development is done in Finland, Germany, and the UK. With their US
presence mostly doing things for the local US market.

I can understand that out of the 5 or so OSes 3 may be mainly developed
in the US, so numerically the number of smartphone OSes developed in the
US is more than the number developed outside the US, but when an OS in
the non-US minority holds over 40% market share (which is more than it's
4 nearest competitors combined), I think that also says something very
important.

Al.

[1]http://compoundsemiconductor.net/blog/2008/12/global_smartphone_sales_growth.html

[Muthu Ramadoss]

Now I know why Nokia is winning. Thanks Al.

take care,
Muthu Ramadoss.

http://linkedin.com/in/tellibitz +91-9840348914
http://androidrocks.in - Android Consulting.

[Jean-Baptiste Queru]

That's interesting, it wasn't that long ago that they were trying to
recruit me in their engineering office in Boston to work on some core
technologies that ship on all their European phones, or that I was
working directly with their engineering office in San Diego on
products for the Americas (not just the US), or that a good friend of
mine was working in engineering in one of their Bay Area offices on
phone stuff that ships mostly in Asia.

Seriously, Nokia does have some significant engineering presence in
the US, and not just for the US market (though they indeed do a lot of
their engineering work outside of the US).

JBQ

--