Re: Active stylus support on ChromeOS

[Rick Byers]

+input-dev team

Hi Rob.

Thanks, that's great!  I've hoped to expose some additional stylus-type properties on TouchEvent for awhile, but the standards situation here is quite difficult (with some vendors backing the pointer events API).  We don't add new non-standard APIs to blink, so the standards process is the real gate here.  You can follow along and participate in the Touch Events Community Group if you're interested.

That said, we already have something that kind of works on Android with stylus input.  There we send touch events for the stylus, and plumb things like stylus pressure through to TouchEvent.force.  The main functionality that's missing with that is an inability to tell where the stylus is when it's not contacting the screen.  As a workaround (until we have a better API), we send mousemove events in that scenario.  This has been good enough for users in practice for awhile.  You can see this behavior, for example, on a Samsung Galaxy Note with their SPen.

So first step is definitely getting ChromeOS to parity with our Android behavior here.  The additional steps on top of that are tracked in that bug and this document.

Rick

On Mon, Jan 5, 2015 at 7:49 AM, Rob Bradford <robert....@intel.com> wrote:

Hi,

I recently added support for passive / Wacom tablets to the ChromeOS
Ozone stack; the events that the backend generated in that case were
MouseMove/Click events.

I've now got some new hardware that features a touchscreen and an active
stylus and I believe it would be logical for those events to be touch
events (i.e. no mouse cursor) rather than mouse events. What do you
think?

I see from crbug.com/393462 you planned to add some stylus specific
functionality for use on Android. Did you make any progress on that? I
think that the active stylus on touch will be able to generate the same
types of event details.

Cheers,

Rob

[Rob Bradford]

Hi Rick & input-dev team,

On Wed, 2015-01-07 at 10:32 -0500, Rick Byers wrote:
> +input-dev team
>
>
> Hi Rob.
> Thanks, that's great! I've hoped to expose some additional
> stylus-type properties on TouchEvent for awhile, but the standards
> situation here is quite difficult (with some vendors backing the
> pointer events API). We don't add new non-standard APIs to blink, so
> the standards process is the real gate here. You can follow along and
> participate in the Touch Events Community Group if you're interested.

Thanks, i've subscribed to the mailing list now to keep an eye out for
updates on this.

>
> That said, we already have something that kind of works on Android
> with stylus input. There we send touch events for the stylus, and
> plumb things like stylus pressure through to TouchEvent.force. The
> main functionality that's missing with that is an inability to tell
> where the stylus is when it's not contacting the screen. As a
> workaround (until we have a better API), we send mousemove events in
> that scenario. This has been good enough for users in practice for
> awhile. You can see this behavior, for example, on a Samsung Galaxy
> Note with their SPen.
>
>
> So first step is definitely getting ChromeOS to parity with our
> Android behavior here. The additional steps on top of that are
> tracked in that bug and this document.
>

After wrangling with the I2C driver i've now got both the touchscreen
and the stylus working on my hardware. The implementation I made for the
external tablets works with zero modifications with the on screen
stylus.

This code currently generates MouseEvent (as in ui/events) for movement
and button press/release and moves the mouse cursor. Currently this
cursor is the usual arrow.

If I understand the Android SPen behaviour correctly I need to change
this to:

- Generate MouseEvent (ET_MOUSE_MOVED) when the stylus is in proximity
but not touching the screen (as currently so no change)
- Generate TouchEvent, ET_TOUCH_PRESSED/RELEASED when pressing &
releasing and ET_TOUCH_MOVED when dragging.
- Plumb the stylus pressure in as the touch event force.

Couple of implementation questions:

- Should the cursor move with the stylus? Perhaps it should be some
other graphical representation, like a cross hair (that's what I
observed on Windows 8.1)
- What shall we put for the radius of the touch event, zero?
- When some mechanism for identifying that this is a stylus vs a finger
comes into blink we'll also need to extend TouchEvent.

Rob

[Rick Byers]

Sounds right.

Couple of implementation questions:

- Should the cursor move with the stylus? Perhaps it should be some
other graphical representation, like a cross hair (that's what I
observed on Windows 8.1)

I guess that's really a UI question which I have no strong opinion for.  On Samsung SPen devices I see they use a little circle as the cursor.  Doing something similar on CrOS would be reasonable.  kuscher@ may have more input on the CrOS UI side.

- What shall we put for the radius of the touch event, zero?

On my Galaxy Note I see 0,0 (which the extensions note defines to mean "not known"), but personally I think 1,1 would be more appropriate.  You do really know the size of the contact area - it's very precise.  This should have the effect of disabling touch adjustment - no need to do sloppy targetting when you're using a precise input device.

You should also make sure you report interaction media features properly (same as a mouse device I guess).   See Mustaq's recent CLs on this bug for details.

 

- When some mechanism for identifying that this is a stylus vs a finger
comes into blink we'll also need to extend TouchEvent.

The public API?  Right this would be nice, but we need a standard and support from the other browser vendors for this.  Having use cases would help.  Do you know anyone trying to build websites to take advantage of stylus input (differently from how they handle mouse/touch)?

Rob

[Alexander Kuscher]

I guess that's really a UI question which I have no strong opinion for.  On Samsung SPen devices I see they use a little circle as the cursor.  Doing something similar on CrOS would be reasonable.  kuscher@ may have more input on the CrOS UI side.

I think seeing a mouse cursor when using a stylus would look very odd. Crosshair or circle are fine. Crosshair if the resolution is very precise and circle if it is somewhat coarse. Adding UX for input and potential assets etc

[Roma Shah]

+1 to what Kuscher said, although it makes sense to reserve judgment till we see how it actually feels. I talked to +Alan about helping out on the assets.  

[Jenn Chen]

SGTM. Thanks all!

--

Jenn Chen

User Experience Designer | Google

[Alan Bettes]

Hi all, can I get a bug to track this? I don't have a galaxy note available..are there any specs I can get on the required size of the asset? Size of the stylus tip? Whatever's relevant. Thanks.   

On Thu, Jan 15, 2015 at 1:23 PM, Roma Shah <ro...@google.com> wrote:

--

Alan Bettes
Visual Designer // Chrome

[Rob Bradford]

On Tue, 2015-01-13 at 17:20 -0500, Rick Byers wrote:

> - Should the cursor move with the stylus? Perhaps it should be
> some
> other graphical representation, like a cross hair (that's what
> I
> observed on Windows 8.1)
>
>
> I guess that's really a UI question which I have no strong opinion
> for. On Samsung SPen devices I see they use a little circle as the
> cursor. Doing something similar on CrOS would be reasonable.
> kuscher@ may have more input on the CrOS UI side.

Cool.

>
> - What shall we put for the radius of the touch event, zero?
>
>
> On my Galaxy Note I see 0,0 (which the extensions note defines to mean
> "not known"), but personally I think 1,1 would be more appropriate.
> You do really know the size of the contact area - it's very precise.
> This should have the effect of disabling touch adjustment - no need to
> do sloppy targetting when you're using a precise input device.
>

Right, this is now implemented in my CL here:
https://codereview.chromium.org/847663004/

Still need to work on the unit tests but interactive testing shows the
desired behavior with monitorEvents.

>
> You should also make sure you report interaction media features
> properly (same as a mouse device I guess). See Mustaq's recent CLs
> on this bug for details.
>
> - When some mechanism for identifying that this is a stylus vs
> a finger
> comes into blink we'll also need to extend TouchEvent.
>
>
> The public API? Right this would be nice, but we need a standard and
> support from the other browser vendors for this. Having use cases
> would help. Do you know anyone trying to build websites to take
> advantage of stylus input (differently from how they handle
> mouse/touch)?

I was actually referring to the TouchEvent class in ui/events, so not
the public API. Of course that will only be plumbing until the standard
is available.

Unfortunately i'm not really aware of any stylus using websites, but
anything that's focussed on sketching would be able to take advantage of
it. A quick search shows a few doodling and sketching apps in the chrome
store when you search for "stylus".

Rob