Injected keyboard input events as "key events", rather than "text events".

[vivimillin]

Is your feature request related to a problem? Please describe.
East Asian language input methods (IME) cannot be typed correctly on DroidVC-NG, and keyboard input can only be recognized as English characters, and cannot trigger IME recognition and conversion to characters.

Describe the solution you'd like
Injected keyboard input events (at the Android API level) as "key events", rather than "text evetns".
More specificly, numbers and "special characters" can be inserted using text events, but letters should be injected using key events, so that the keyboard behaves as expected in games, as well as in IME.
A good reference for implementation is scrcpy. And its implementation on keyboard input：https://github.com/Genymobile/scrcpy/blob/master/doc/keyboard.md

Your environment (please complete the following information):

• droidVNC-NG version: v2.8.0
• Android version: 14
• Client-side OS and version: Windows 11
• VNC client and version: TightVNC 2.8.81

[bk138]

Looks interesting, but I suspect it requires root or some manual setup on the device? Can you give us some insights @vivimillin ?

[bk138]

AFAICT, AOAv2 only works for USB devices which can "speak" it, uhid OTOH needs to be loaded as a kernel module, which, if present, would require root. So this seems out of scope...

[vivimillin]

no, no ScrCpy (and its input implementation) works perfect without root. :-)

Among its several options of keyboard input modes, what I suggest equals the default one (sdk), not aoa/uhid. That the keyboard input events are injected at the Android API level.

[bk138]

Among its several options of keyboard input modes, what I suggest equals the default one (sdk), not aoa/uhid. That the keyboard input events are injected at the Android API level.

I'm a bit confused ;-) On https://github.com/Genymobile/scrcpy/blob/master/doc/keyboard.md it says the SDK method is only able to inject "ASCII and some other characters", so there would be no improvement w.r.t. the current situation? Can you point us at the exact API docs maybe?

Edit: is this maybe related/same-as #208?

Edit edit: specifically, the API linked in #208 (comment) ?

[vivimillin]

I think in ScrCpy, the input is implemented with method: InputManager.injectInputEvent

It's a hidden method, as explained:
"Some of them need to inject input events to the system. To do so, they use the
hidden method [InputManager.injectInputEvent()] (exposed by the
[InputManager wrapper][inject-wrapper])."

And codes here: /genymobile/scrcpy/wrappers/InputManager.java#L12
/genymobile/scrcpy/device/Device.java#L69

Also some other discussions:
https://stackoverflow.com/questions/18699614/android-inputmanager-injectinputevent

Sorry I fact I'm not expert on Android programming :-)

[vivimillin]

then, the emulated key input was injected as KEYCODE with ACTION_DOWN & ACTION_UP, so that the keyboard behaves better as expected in games, as well as in IME.

https://github.com/Genymobile/scrcpy/blob/5ae01749bf4fad9fb4d8bf7c879dc60f479c1013/server/src/main/java/com/genymobile/scrcpy/device/Device.java#L72

https://github.com/Genymobile/scrcpy/blob/5ae01749bf4fad9fb4d8bf7c879dc60f479c1013/app/src/keyboard_sdk.c#L48

[bk138]

@vivimillin Thanks for the pointers, this makes sense. As droidVNC-NG is an app distributed through the app store, we can't use hidden API though. Our best best will be #208 (comment) 🤷. I think this could possibly handle non-English symbols as well, let's continue the research there :-).