debugging HID support

[Jason12 C12]

Background

I'm fairly new to ChromeOS, but I am developing a USB device that works like a absolute position mouse.   It is basically what one finds in the  absmouse Arduino library.  (source link).

It works on Windows and Mac, but not on the recent production Chromebook I have for testing.  The mouse simply does not move.   When I plug in the device, it does show up at chrome://system#lsusb as using usbhid.  I'll post evtest output in a moment.

I've built a similar mouse that uses relative positioning and that works fine on the Chromebook.

QQ

How do I debug this?   What logs do I have access to that might help with forensics?   Where would I find the code that probably is deciding to ignore this abs mouse?

Thanks in advance.

evtest output

crosh> evtest
No device specified, trying to scan all of /dev/input/event*
Not running as root, no devices may be available.
Available devices:
/dev/input/event0: Lid Switch
/dev/input/event1: Sleep Button
/dev/input/event2: Power Button
/dev/input/event3: AT Translated Set 2 keyboard
/dev/input/event4: GTCH7503:00 2A94:A803
/dev/input/event5: Tablet Mode Switch
/dev/input/event6: Elan Touchpad
/dev/input/event7: sof-glkda7219max Headset Jack
/dev/input/event8: sof-glkda7219max HDMI/DP, pcm=5 Jack
/dev/input/event9: sof-glkda7219max HDMI/DP, pcm=6 Jack
/dev/input/event10: sof-glkda7219max HDMI/DP, pcm=7 Jack
/dev/input/event11: Adafruit ItsyBitsy 32u4 5V 16MHz
Select the device event number [0-11]: 11
Input driver version is 1.0.1
Input device ID: bus 0x3 vendor 0x239a product 0x800e version 0x101
Input device name: "Adafruit ItsyBitsy 32u4 5V 16MHz"
Supported events:
Event type 0 (EV_SYN)
Event type 1 (EV_KEY)
Event code 272 (BTN_LEFT)
Event code 273 (BTN_RIGHT)
Event code 274 (BTN_MIDDLE)
Event type 3 (EV_ABS)
Event code 0 (ABS_X)
Value 6826
Min 0
Max 32767
Event code 1 (ABS_Y)
Value 3337
Min 0
Max 32767
Event type 4 (EV_MSC)
Event code 4 (MSC_SCAN)
Properties:
Testing ... (interrupt to exit)
Event: time 1640537269, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537269, -------------- SYN_REPORT ------------
Event: time 1640537269, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537269, -------------- SYN_REPORT ------------
Event: time 1640537269, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537269, -------------- SYN_REPORT ------------
Event: time 1640537269, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537269, -------------- SYN_REPORT ------------
Event: time 1640537269, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537269, -------------- SYN_REPORT ------------
Event: time 1640537269, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537269, -------------- SYN_REPORT ------------
Event: time 1640537270, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537270, -------------- SYN_REPORT ------------
Event: time 1640537270, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537270, -------------- SYN_REPORT ------------
Event: time 1640537270, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)
Event: time 1640537270, -------------- SYN_REPORT ------------
Event: time 1640537270, type 3 (EV_ABS), code 0 (ABS_X), value in range (min, max)
Event: time 1640537270, -------------- SYN_REPORT ------------
Event: time 1640537270, type 3 (EV_ABS), code 0 (ABS_X), value in range (min, max)
Event: time 1640537270, -------------- SYN_REPORT ------------

[Jason12 C12]

There is more detailed evtest (while logged in as root this time) that also resulted in no mouse pointer movement..

localhost ~ # evtest /dev/input/event11

Input driver version is 1.0.1
Input device ID: bus 0x3 vendor 0x239a product 0x800e version 0x101
Input device name: "Adafruit ItsyBitsy 32u4 5V 16MHz"
Supported events:
Event type 0 (EV_SYN)
Event type 1 (EV_KEY)
Event code 272 (BTN_LEFT)
Event code 273 (BTN_RIGHT)
Event code 274 (BTN_MIDDLE)
Event type 3 (EV_ABS)
Event code 0 (ABS_X)

Value 0

Min 0
Max 32767
Event code 1 (ABS_Y)

Value 0

Min 0
Max 32767
Event type 4 (EV_MSC)
Event code 4 (MSC_SCAN)
Properties:
Testing ... (interrupt to exit)

Event: time 1640556733.363285, type 3 (EV_ABS), code 0 (ABS_X), value 614
Event: time 1640556733.363285, type 3 (EV_ABS), code 1 (ABS_Y), value 606
Event: time 1640556733.363285, -------------- SYN_REPORT ------------
Event: time 1640556733.438242, type 3 (EV_ABS), code 0 (ABS_X), value 648
Event: time 1640556733.438242, -------------- SYN_REPORT ------------
Event: time 1640556733.513246, type 3 (EV_ABS), code 0 (ABS_X), value 682
Event: time 1640556733.513246, -------------- SYN_REPORT ------------
Event: time 1640556733.588247, type 3 (EV_ABS), code 0 (ABS_X), value 716
Event: time 1640556733.588247, -------------- SYN_REPORT ------------
...

[ggg]

Jason,

I suspect the developers who could answer this question are out this week. Ping this thread again next week if you haven't found the answer.

My guess is Chrome OS is just ignoring the device.  Are you signed in or expecting the mouse to move at the login screen?

Chrome OS changed behavior in 2021 to ignore external USB devices (with an "allow list" for exceptions like USB ethernet) if someone isn't signed in.

I expect you'd need to find that allow list and either post an acceptable change to it or emulate an existing entry.

cheers,

grant

[ggg]

On Tuesday, December 28, 2021 at 10:48:45 AM UTC-8 ggg wrote:

Jason,

I suspect the developers who could answer this question are out this week. Ping this thread again next week if you haven't found the answer.

My guess is Chrome OS is just ignoring the device.  Are you signed in or expecting the mouse to move at the login screen?

Chrome OS changed behavior in 2021 to ignore external USB devices (with an "allow list" for exceptions like USB ethernet) if someone isn't signed in.

I expect you'd need to find that allow list and either post an acceptable change to it or emulate an existing entry.

Or sign in to see if the mouse is still being ignored. If your mouse is still being ignored when signed in, then it's something else.

cheers,

grant

[Harry Cutts]

We haven't implemented support for absolute mice in our input stack, so that'll be why it's not working. Specifically, the exact bit of code that will be deciding to ignore the mouse is EventDeviceInfo::HasMouse, which only recognizes mice with REL_X and REL_Y (as opposed to ABS_X and ABS_Y). However, even if you modified that to accept the absolute axes, then the gestures library's MouseInterpreter wouldn't know what to do with them, so nothing would happen.

I haven't encountered the idea of an absolute mouse before, except as a VM interface. What features are you hoping to gain from making it absolute rather than relative? I'm guessing you're interested in being able to click on things by their absolute position on the screen, in which case you could have the device identify as a drawing tablet instead (see EventDeviceInfo::HasTablet).

[Grant Grundler]

On Thu, Dec 30, 2021 at 5:33 AM Harry Cutts <hcu...@chromium.org> wrote:

We haven't implemented support for absolute mice in our input stack, so that'll be why it's not working. Specifically, the exact bit of code that will be deciding to ignore the mouse is EventDeviceInfo::HasMouse, which only recognizes mice with REL_X and REL_Y (as opposed to ABS_X and ABS_Y). However, even if you modified that to accept the absolute axes, then the gestures library's MouseInterpreter wouldn't know what to do with them, so nothing would happen.

I haven't encountered the idea of an absolute mouse before,

Amusingly, I wrote a driver for one decades ago (look for CONFIG_OBMOUSE)

     http://lkml.iu.edu/hypermail//linux/kernel/0401.0/0861.html

to support the HP Omnibook 600C / 600CT "Pop-out" mouse:

     https://www.cs.hmc.edu/~fleck/iowa-lab/omnibook.html

But since USB mice, I've never heard of another (but I also mostly stopped dealing with input devices then too).

It's not particularly hard to convert the driver to emit relative moves instead or even make ABS vs REL a setting accessible via /sys. Obmouse driver does that if you are looking for an example (even if some parts of the driver are very outdated).

except as a VM interface. What features are you hoping to gain from making it absolute rather than relative? I'm guessing you're interested in being able to click on things by their absolute position on the screen, in which case you could have the device identify as a drawing tablet instead (see EventDeviceInfo::HasTablet).

But it's not a tablet - i.e. gestures would be "mouse like". Better to modify the driver to emit REL coordinates.

cheers,

grant

[Evan Green]

In case it helps during development, I happened to write (with Harry's
help) a silly usermode app that translates an absolute coordinate
device into a relative one. In my case I wanted to use the lower
portion of the touchscreen as a mouse. Probably not what you're
looking for long term, but maybe helpful temporarily.

https://github.com/evangreen/trackscreen

-Evan

> --
> --
> Chromium OS Developers mailing list: chromiu...@chromium.org
> View archives, change email options, or unsubscribe:
> https://groups.google.com/a/chromium.org/group/chromium-os-dev

[Jason12 C12]

Thanks everyone.  This is great help all around.

I'm now looking in to configuring the device to pose as a digitizer / graphics tablet as recommended in this thread.   I do have a physical graphics pad that appears to behave on ChromeOS exactly as I'd want, so it should be possible to have our device behave the same way.   I am still, a week later, testing hypothesizes for why my device is not treated like a graphics tablet.   But at this point, my device is no longer being ignored, so that's progress.  But it seems to be acting like a touch screen finger device.  (The mouse cursor/pointer disappears and does not move, but the "finger" does have the ability to act on the desktop.)    I'll keep on working on it.  Thanks for the link to the relevant ChromeOS source code.

Re: why one might want an absolute position mouse - In my scenario we have a presenter in front of an audience using a mouse like device as he wanders the room.  He needs to be able click and to point, etc on whatever computer (Mac, Windows, ChromeOS, Linux, microscope, other) that is being projected to the audience.    Using relative coordinates works great unless the mouse pointer ends up on another (unseen) screen of the desktop.  Then the UX is poor.  We'd like to keep the mouse on the public screen and let the presenter trivially return it there if a second mouse has left it off screen.  

So, why not just emulate a graphics tablet?   A: we might have to..  It looks like that approach would work great if the mouse is connected to ChromeOS laptop, but if the laptop is Windows, Windows treats graphics tablets differently than an ordinary mouse.  It uses s different mouse pointer/cursor icon.  Clicks are treated differently.  Etc.  That is not suitable for our presenters.   They are accustomed to mouse-like behavior so that's what we'd like to give them.

So at this point, our device might need to be able to act as a abs mouse on some desktops and a graphics tablet on ChromeOS.  But of course the mouse doesn't know which it's connected to.  

[Harry Cutts]

On Wednesday, January 5, 2022 at 7:35:07 AM UTC-8 jason.c...@merlyn.org wrote:

Thanks everyone.  This is great help all around.

I'm now looking in to configuring the device to pose as a digitizer / graphics tablet as recommended in this thread.   I do have a physical graphics pad that appears to behave on ChromeOS exactly as I'd want, so it should be possible to have our device behave the same way.   I am still, a week later, testing hypothesizes for why my device is not treated like a graphics tablet.   But at this point, my device is no longer being ignored, so that's progress.  But it seems to be acting like a touch screen finger device.  (The mouse cursor/pointer disappears and does not move, but the "finger" does have the ability to act on the desktop.)    I'll keep on working on it.  Thanks for the link to the relevant ChromeOS source code.

Re: why one might want an absolute position mouse - In my scenario we have a presenter in front of an audience using a mouse like device as he wanders the room.  He needs to be able click and to point, etc on whatever computer (Mac, Windows, ChromeOS, Linux, microscope, other) that is being projected to the audience.    Using relative coordinates works great unless the mouse pointer ends up on another (unseen) screen of the desktop.  Then the UX is poor.  We'd like to keep the mouse on the public screen and let the presenter trivially return it there if a second mouse has left it off screen.  

Ah, I see. You might want to play around with your graphics tablet on a multi-monitor Chrome OS setup to get a feel for how it works currently — at the moment, we apply the cursor movements from the tablet on whatever screen the cursor is currently on, so you wouldn't be able to return the cursor to the visible screen this way. I think we also allow the graphics tablet to move the cursor from one screen to another by touching it to the corresponding edge (e.g. if the cursor's on your left monitor and you move the pen all the way to the right edge of the tablet, the cursor will move to the left edge of your right monitor). So, this might not be the exact behaviour you want.

 

So, why not just emulate a graphics tablet?   A: we might have to..  It looks like that approach would work great if the mouse is connected to ChromeOS laptop, but if the laptop is Windows, Windows treats graphics tablets differently than an ordinary mouse.  It uses s different mouse pointer/cursor icon.  Clicks are treated differently.  Etc.  That is not suitable for our presenters.   They are accustomed to mouse-like behavior so that's what we'd like to give them.

So at this point, our device might need to be able to act as a abs mouse on some desktops and a graphics tablet on ChromeOS.  But of course the mouse doesn't know which it's connected to.  

If you or someone else at Merlyn are interested in contributing code to Chromium to support absolute mice, I would be open to accepting it, and I'd be happy to give you code pointers etc.. Feel free to email me directly to discuss more if you're interested.

[dragon788 (dragon788)]

Hi Harry et al,

Do you know offhand if there are any bugs filed against ChromeOS for supporting absolute positioning mice as inputs or at least tracking/displaying the cursor position when one of these devices is attached? Clicks seem to work (maybe a separate part of the HID spec from positioning?) but changing the mouse position doesn't.

I've got a fairly concrete use case or two, but I don't want to generate extra noise if it isn't necessary.

First case, using a USB KM synchronizer (like a KVM without the V) to allow directing inputs from an attached USB keyboard and mouse to a single device (or many if the KM supports synchronized output mode), and switching between devices using a touch against the screen edge.

This switching actually works on ChromeOS currently but is very hard to keep track of with invisible cursors on the device screens. I think it still "works" because this 'edge detection' basically uses a huge virtual display with a bunch of stitched together regions at 1080p or 4k (and very occasionally the resolutions can be configured via a driver), and the KM accepts the relative mouse moves from a USB HID mouse and translates them to absolute positions on the large virtual display to determine whether to switch the target device, and then sends translated absolute mouse locations from within the current device's region to the target device. It may actually just detect the edge and reset the absolute X,Y for each new device so they can skip one level of translation, but for the resolution configurable devices I'm not sure how this would work.

Second case, there are USB DP/HDMI KVMs that also support mouse glide/seamless mouse switching/edge switching, and the main difference between the two implementations is the KVM could actually "know" what resolution the device's output display is thanks to the DP/HDMI EDID, but some of the devices may still implement a more naive 1080p default (which I think I've seen having tested on a 1280x800 device where the right "edge" was beyond the edge of the screen).

This type of device is common in control rooms for law enforcement, video production, site monitoring etc, and I think ChromeOS devices especially as they become even more capable with ChromeOS Plus would make ideal workstations for these environments, but only partial compatibility with the switching devices would make that unlikely.

[Harry Cutts]

On Thu, 4 Jan 2024 at 18:20, dragon788 (dragon788) <drag...@gmail.com> wrote:

Hi Harry et al,

Do you know offhand if there are any bugs filed against ChromeOS for supporting absolute positioning mice as inputs or at least tracking/displaying the cursor position when one of these devices is attached? Clicks seem to work (maybe a separate part of the HID spec from positioning?) but changing the mouse position doesn't.

I've got a fairly concrete use case or two, but I don't want to generate extra noise if it isn't necessary.

First case, using a USB KM synchronizer (like a KVM without the V) to allow directing inputs from an attached USB keyboard and mouse to a single device (or many if the KM supports synchronized output mode), and switching between devices using a touch against the screen edge.

I'm not actually on the ChromeOS input team anymore (I've moved over to Android), but I'm not aware of any bugs filed for it. Feel free to file one in the ChromeOS Public Tracker > Peripherals > Input component! Including details of the exact model(s) of KM switch you're testing with would be helpful, as would filing a feedback report as instructed in the bug template.

[Dmitry Torokhov]

+Theodore Tso +Kenneth Albanowski 

On Mon, Jan 8, 2024 at 3:42 AM Harry Cutts <hcu...@chromium.org> wrote:

On Thu, 4 Jan 2024 at 18:20, dragon788 (dragon788) <drag...@gmail.com> wrote:

Hi Harry et al,

Do you know offhand if there are any bugs filed against ChromeOS for supporting absolute positioning mice as inputs or at least tracking/displaying the cursor position when one of these devices is attached? Clicks seem to work (maybe a separate part of the HID spec from positioning?) but changing the mouse position doesn't.

I've got a fairly concrete use case or two, but I don't want to generate extra noise if it isn't necessary.

First case, using a USB KM synchronizer (like a KVM without the V) to allow directing inputs from an attached USB keyboard and mouse to a single device (or many if the KM supports synchronized output mode), and switching between devices using a touch against the screen edge.

I'm not actually on the ChromeOS input team anymore (I've moved over to Android), but I'm not aware of any bugs filed for it. Feel free to file one in the ChromeOS Public Tracker > Peripherals > Input component! Including details of the exact model(s) of KM switch you're testing with would be helpful, as would filing a feedback report as instructed in the bug template.

Ted has a "TESmart KVM" switch which apparently in "fast switching mode" presents a device with ABS_X/Y, so there are few such devices:

Input driver version is 1.0.1

Input device ID: bus 0x3 vendor 0x46d product 0xc506 version 0x110
Input device name: "HID 046d:c506"

Supported events:
  Event type 0 (EV_SYN)
  Event type 1 (EV_KEY)
    Event code 272 (BTN_LEFT)
    Event code 273 (BTN_RIGHT)
    Event code 274 (BTN_MIDDLE)

    Event code 275 (BTN_SIDE)
    Event code 276 (BTN_EXTRA)
  Event type 2 (EV_REL)
    Event code 8 (REL_WHEEL)
    Event code 11 (REL_WHEEL_HI_RES)

  Event type 3 (EV_ABS)
    Event code 0 (ABS_X)

      Value      0
      Min        0
      Max     1024
    Event code 1 (ABS_Y)
      Value      0
      Min        0
      Max      768

[dragon788]

Yeah the TESmart KVMs with the ability to toggle the pass through (I think turning it off which disables extra buttons and presents an emulated mouse to allow a double tap of the cursor at the edge of the screen to trigger a switch) are reasonably priced and have a nice set of features, plus the ability to control via serial or LAN.

[dragon788]

If you could file a bug for getting mouse pointer support for absolute positioning devices that would be amazing.

I have the HKS1601A1U which is a similar vintage to yours with 16 ports, vs the older model like the HKS1601A10 which does NOT support the passthrough and so has no way to toggle fast switching.

Somehow I only found out about the mouse switching option in the last few weeks as it wasn't called out in the Amazon listing or most of the documentation I'd read.

I was extremely excited and enabled it once I found a keyboard that still had a Scroll Lock key, but when I tested it with my ChromeOS devices I was shattered to see that it behaved just like the KM switchers from Belkin and MT-VIKI et al and reported as an absolute positioning mouse and therefore didn't show a cursor in ChromeOS.

I _can_ use the mouse to quick switch between devices with a double tap of the cursor against the right or left edges, and once a device is selected the keyboard will interact with it, but not everything in ChromeOS is fully keyboard accessible, especially with the translated input interactions between Android apps and Linux apps and the ChromeOS input layer munging things differently for different destinations.

In Crosh running evtest I'm seeing the same thing you are where ChromeOS or at least the evtest layer does see the EV_ABS events in fast switching and EV_REL in passthrough, but it doesn't update the cursor location (or even show the cursor unless a relative mouse is plugged in or the trackpad is brushed, it hides the cursor once I start typing from the KVM attached keyboard, likely assuming a touchscreen or stylus driven interface). 

On Wed, Jan 10, 2024 at 10:15 AM Theodore Tso <ty...@google.com> wrote:

Let me know if you'd like me to a file a bug, or if you want me to run any experiments.   For now I've just disabled fast switching (which I had accidentally enabled when I switched to a new keyboard --- a Nuphy Air 96 V2).    The specific TESmart KVM switch is a 2018 model, HKS0401A1U (purchased from Amazon). 

With Fast Switching enabled:

Select the device event number [0-10]: 7    

Properties:

Testing ... (interrupt to exit)

Event: time 1704836973, type 3 (EV_ABS), code 0 (ABS_X), value in range (min, max)

Event: time 1704836973, type 3 (EV_ABS), code 1 (ABS_Y), value in range (min, max)

With Fast Switching disabled:

Working evtest report:
Select the device event number [0-10]: 7

Input driver version is 1.0.1

Input device ID: bus 0x3 vendor 0xc45 product 0x7403 version 0x110
Input device name: "HID 0c45:7403"

Supported events:
  Event type 0 (EV_SYN)
  Event type 1 (EV_KEY)
    Event code 272 (BTN_LEFT)
    Event code 273 (BTN_RIGHT)
    Event code 274 (BTN_MIDDLE)

  Event type 2 (EV_REL)
    Event code 0 (REL_X)
    Event code 1 (REL_Y)

    Event code 8 (REL_WHEEL)
    Event code 11 (REL_WHEEL_HI_RES)

  Event type 4 (EV_MSC)
    Event code 4 (MSC_SCAN)
Properties:
Testing ... (interrupt to exit)

Event: time 1704837426, type 2 (EV_REL), code 0 (REL_X)
Event: time 1704837426, -------------- SYN_REPORT ------------
Event: time 1704837426, type 2 (EV_REL), code 0 (REL_X)

The mouse connected to the TESmart KVM switch is a Logitech G MX518 Gaming Mouse.

- Ted

[Dmitry Torokhov]

OK, I filed https://issuetracker.google.com/issues/319472351