aRDP Pro - Hold key on keyboard is a series of key presses

[Sean Reid]

Hi, I am setting up my new S22 Ultra for remote workstation use.  I am a game developer, and aRDP facilitates almost everything I need.  However, after much testing, I have realized that holding a key on a bluetooth or wired keyboard via the type-c usb port on the phone, keystrokes are sent as a series if key up/down events.  This can be proven by holding the shift key, and Windows will popup the sticky keys dialog as though I pressed the shift key 5 times in a row in quick succession.

This obviously is not adequate behavior for a game developer.  I need to be able to move my character around on the screen using WASD.  For example, in an overhead game, I hold down the D key to move right.  What actually happens is that the character barely moves, and when he moves it is very stuttery.  In other words, it's like the player is rapidly pressing D, and barely any of those repeated strokes get through anyway.  More specifically, the character is mostly responsive as though only 5% of these repeated keystrokes are getting through.

Any help would be appreciated.  I also tried the RD Client app, and that one doesn't popup the Sticky Keys while holding shift.  However, my game doesn't receive any key events whatsoever.

Finally, I used the Windows 11 Remote Desktop application on a 2nd PC, and it works perfectly.  The game itself will draw at a decent framerate (as does aRDP Pro), but the keyboard behavior works flawlessly on the native Windows 11 RD application.  I just need aRDP not send repeated keystrokes.

[Sean Reid]

Here is some additional information.

While connected via Windows 11 Remote Desktop via 2nd PC, here is the debug logging in my game related to keystrokes:

on key: 68 32 1
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 2
on key: 68 32 0

The 68 means "D" key, "1" means key-down or press, "2" means key-repeat, and "0" means key-up or release.

Please ignore the key repeat event as I do not use it.  The important events are key-down and key-up.

Here is the observed behavior when using aRDP Pro:

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

on key: 68 32 1

on key: 68 32 0

As you can see, as I hold down the "D" key, the remote desktop is just getting a series of key-down and key-up events in quick succession.  This explains the Sticky Keys popup if I am holding the shift key.

[i iordanov]

Hi Sean,

This is expected behavior for the app, since it was originally
designed only with soft-keyboard input in mind where there is no
concept of key held down, really. The app has historically handled
Android KeyEvent events in a stateless manner, since it assumes all
information for any software key event is contained in the event
itself. To complicate matters further, the app is actually 3 apps in
one with support for three different protocols. As a result, the app
sends modifier keys on keydown and releases modifier keys on keyup to
keep things architecturally simple. With soft input there is no
guarantee that there will be a keyup event sent at all, so in order to
not remain with modifier keys stuck in a pressed state, the app
prefers to send modifier keys as soon as possible.

It's only recently that more and more users have started using the app
with hardware keyboards, where there is a concept of a modifier key
held down while others are pressed, but of course the keyboard
implementation has remained the same. I guess maybe it's time to try
to figure out what the best way to add multi key-press modifiers.

Let's start by first upgrading you to the beta version of aRDP where I
made a number of changes to the keyboard handling logic. There won't
be a fix, for your issue, but if I find time to work on this you'll
get the feature as soon as I have a version up in Beta.

Also, if you don't mind, please open a feature request here:
https://github.com/iiordanov/remote-desktop-clients/issues

Thanks!
iordan

> --
> You received this message because you are subscribed to the Google Groups "bVNC, aRDP, aSPICE, Opaque Remote Desktop Clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bvnc-ardp-aspice-opaque-remot...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bvnc-ardp-aspice-opaque-remote-desktop-clients/bf4a1089-1ded-40d3-bd61-50ee0936139fn%40googlegroups.com.



--
The conscious mind has only one thread of execution.

[Sean Reid]

Thanks for the reply.  Everything you've said makes sense.  It's been a while, but I released a game with gamepad input support for the Android version.  I used my original nvidia shield (the portable one with the xbox style gamepad and screen attached) as the testing device.  All of the so-called "key up" and "key down" events were actually gamepad events, so that's why it makes sense that the "letter" keys are sent as key down/up pairs to the remote desktop.

When I was evaluating more RDP clients for Android, I noticed that some of them do send key up and key down events, but only for the non "letter" keys such as the cursor keys, CTRL, SHIFT, ALT, Escape, and others such as Backspace and Return (I believe, but I might be mistaken).  The default mapping for Player 1 is the keyboard keys WASD, but I also provide secondary, tertiary, and 4th optional input mappings.  For the purposes of game development, I just added the cursor keys to the character's movement, and then Right Shift for the dash action, and Backspace for the Interact action (to open doors, chests, etc).

The 2 clients that support this are Microsoft's Remote Desktop, and another called Remote Desktop Manager.  Those two clients have their flaws (from my perspective as a game developer).  Here are the flaws:

- Microsoft Remote Desktop: video performance is bad, when I make my game's window larger past a certain size, the entire client lags including video, audio, and inputs.

- Remote Desktop Manager: I cannot quite pinpoint the issue with Shift and/or Caps Lock, but let's say I want to quickly type "PLAYER_LIFE" but without engaging caps lock.  The capitalized letters will actually be random case, for example "PLaYer_LifE".  I suspect it's because they send the Shift as a key up/down pair, and sometimes it is caught up with a letter, so there is a 50/50 chance your letter is sent as a true capitalized letter.  As for Caps Lock, it is the same issue.  Even if I held the "S" key while caps lock is engaged, the following output to the remote desktop would be "SSsssSsSSssSsSs"

So both of the aforementioned remote desktop clients are a no-go.  The best performing remote desktop app is definitely aDRP, and if it facilitates the "non letter keys" as key up/down events (cursor keys, ctrl, alt, shift, etc), then it would turn my new S22 Ultra into the power-house remote development machine along with Samsung Dex and a bluetooth keyboard and mouse.  I will be looking into posting a feature request.

Thank you for the reply.

[i iordanov]

Took a stab at:

- deduplicating repeated modifier events (android appears to be
sending repeats of modifiers when Linux doesn't, for instance).
- sending a modifier key to the remote only when local vs remote
metastate has changed.
- avoiding the sending of "simulated" modifiers (derived from the
KeyEvent metaState state) if the remote has already had a hardware
metastate sent.

There are probably a ton of issues with the code still, so treat as a
first draft, so to speak:

https://github.com/iiordanov/remote-desktop-clients/releases/download/v5.0.7/freeaRDP-v5.0.7_1-final.apk

Cheers!
iordan

> To view this discussion on the web visit https://groups.google.com/d/msgid/bvnc-ardp-aspice-opaque-remote-desktop-clients/0749d7ca-8f01-4311-80f2-8f86c2ac2191n%40googlegroups.com.

[Sean Reid]

So far, so good!  I am able to witness in my game via remote desktop the following keys with key up and down events:

• Left Shift
• Left Alt
• Left Ctrl
• Right Shift
• Return/Enter: this key is also sending the Key Repeat event, in addition to key up/down events.  Although I do not make use of Key Repeat.

The following keys are received as key up/down events properly, but my game is seeing "-1" as the key code.  This might not have anything to do with aRDP, however.  I am using GLFW, and it's sending the keycode of -1 to my game.  I would have to look into why the keycode cannot be determined and sends "-1" instead.  For the most part, GLFW is just handing-off events from the Windows system event loop directly (i.e. WM_KEYUP, WM_SYSKEYUP, WM_KEYDOWN, WM_SYSKEYDOWN), with a few tweaks here and there.

• Right Alt
• Right Ctrl

The following keys do not send key up/down events, which is expected behavior because you only worked on the modifier keys:

• Arrow cursor keys: very important key because WASD can use arrow keys as an alternate non-letter key
• Escape Key: somewhat important key since that's how players back-out of menus or open the game main menu
• Backspace Key: this one may not be supported because it might be handled by the Android keyboard system like the letter keys.

My only Bluetooth keyboard is a 60% mechanical keyboard, so I am unable to test other keys at this time (like Page Up/Down, Insert, Numpad keys including math symbols, the Numpad Return key, etc.)

I am excited to see if more keys can be supported, because the streaming/video performance of aRDP is by far the best.

[Sean Reid]

I created the official feature request on Github.  It can be found here:

https://github.com/iiordanov/remote-desktop-clients/issues/348

[i iordanov]

Hey, thanks for that. Let's move the discussion into the issue tracker.

iordan

> To view this discussion on the web visit https://groups.google.com/d/msgid/bvnc-ardp-aspice-opaque-remote-desktop-clients/a56369be-40e2-425a-bd04-132b0f2d1942n%40googlegroups.com.

[Cat Stevens]

Hey guys you might not like what I have to say but I was trying to disable the Opaque app from bVNC. It didn't suit my needs you can find out more on one of my repositories on GitHub. It was scrcpy, bVNC and x11vnc. Android doesn't support a vncserver or any other thing like it. It works what I built however I had to add suinput.c to x11vnc. It's a super user input but it works. I've also built 64bit DTS for Android no root needed. It plays mp3 and other ALSA supported music formats as Digital Theater Sound. The sound is killer I use Poweramp and it rocks. The bit per second rate is 6144 nothing like it out there you can find my GitHub page. Just Google toshiba6012 GitHub. There you will find some of my work take care.

Best regards

Ryan Johnson

To view this discussion on the web visit https://groups.google.com/d/msgid/bvnc-ardp-aspice-opaque-remote-desktop-clients/CAMS0tn3S0gqsvPAB_N8dMqE5-itd5KiLPPwvgSLQk806OsUzEg%40mail.gmail.com.

[Cat Stevens]

I've been booted off GitHub I'm pissed. To boot they pulled down scrcpy from apt. Give me some time I'll post my work on Mega with scrcpy compiled.

Best regards

Ryan Johnson

[Cat Stevens]

My GitHub account is still up I have a new Kali Linux install and I having a lot of issues. My GitHub address is http://github/toshiba6012

There you will find a lot of stuff the Android remote repositories are almost done and will be uploaded soon.

Best regards

Ryan Johnson

[i iordanov]

Hey,

Opaque, aRDP, aSPICE, and bVNC can be built totally independently of
each other through Android Studio and even with gradle on the
command-line. There should be absolutely no need to remove Opaque from
bVNC.

Thanks!
iordan