Support using android device's gyros as mouse, for aiming assist #1200
Conversation
|
@Williangalvani, i've noticed that when using Steam controller gyro set as mouse input, if i also send a low value analog input (like left trigger, but lower than any game's input threshold) at the same time, the interface won't switch from Xinput to KB/M. Could you pass on some ghost analog input value when you do the conn.SendMouseMove(x, y) ? not to mention that an in-stream toggle would also be helpful (maybe mirror from turning mouse emulation on/off by holing the start button, one can do the same by holding the select button) |
|
@comanel great ideas, I'll try it. I also experimented with sending only joystick values instead, but that didn't quite do it either. I'll try your suggestions! |
|
Any plans to continue on this PR? Can we merge it as it currently is, and leave the "gyro as stick" to another PR? |
|
Hi dude,
GyroMouse thing works as it should, if you ignore the weird UI behavior
switching from controller to keyboard prompts, it's good enough.
Gyro as stick has some quirks though. Worst one i can think of is upon
re-connecting to a running session, any continuous Joystick input isn't
sent properly... it's like it's being interrupted each frame / packet.
Try merging them both, dive into it and see if there's anything you can
improve.
…On Wed, Jul 12, 2023 at 5:36 PM Carlos Solís ***@***.***> wrote:
Any plans to continue on this PR? Can we merge it as it currently is, and
leave the "gyro as stick" to another PR?
—
Reply to this email directly, view it on GitHub
<#1200 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCVYQLBQITBUP75P4LW73TXP2Y6ZANCNFSM6AAAAAAXAJFYR4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
It's a good PoC, but this PR can't be merged as is. There are a few issues:
Now that we can send motion events to the host natively using the new Sunshine protocol extensions, I think the best approach is to implement 4 gyro control options:
I suspect the first 3 will cover the vast majority of use cases. Since we're passing the gyroscope and accelerometer data to the host via the emulated DualShock 4 controller now in Sunshine nightly, users can do whatever stuff they want with it using Steam Input or similar tools. They can use it natively in games that support motion controls or map it to whatever control they want with Steam Input. I can take a look at implementing these since I've been working on sensor stuff the last few days already. |
|
I'm sorry, I've been low on time and last time I tried, the docker I used to build the apk was no longer working properly, my apks were not getting sensor data, idk why. |
|
I've tried the latest nightly for sunshine and built latest master on
moonlight android. There is some sensor input read in Steam UI, as -0.1
acceleration on tilt axis inside the DS4 gyro calibration menu.
allowUseOfGamepadMotionSensors is true, but something tells me that this
has nothing to do with any sensors on the android device itself (duuh).
Didn't get the chance to actually connect a controller with gyro sensors to
the android device and see if the data passes through normally.
Any chance you could mirror device sensor data as you do with controller
gyro sensor data ?
…On Fri, Jul 14, 2023 at 8:04 AM Cameron Gutman ***@***.***> wrote:
It's a good PoC, but this PR can't be merged as is. There are a few issues:
- It changes the manifest so Moonlight won't install on devices
without a gyroscope
- There's no way to actually turn it off (SensorEventListener is
always registered using additional power)
- Sensor readings come in a fixed orientation and need to be adjusted
for the orientation of the screen (landscape vs reverse landscape)
- It doesn't disable sensors when the activity enters PiP mode
Now that we can send motion events to the host natively using the new
Sunshine protocol extensions, I think the best approach is to implement 4
gyro control options:
- Use gamepad motion sensors (already implemented)
- Use device motion sensors for gamepads that lack motion sensors
- Use device motion sensors for on-screen controls
- Use device motion sensors for mouse control
I suspect the first 3 will cover the vast majority of use cases. Since
we're passing the gyroscope and accelerometer data to the host via the
emulated DualShock 4 controller now in Sunshine nightly, users can do
whatever stuff they want with it using Steam Input or similar tools. They
can use it natively in games that support motion controls or map it to
whatever control they want with Steam Input.
—
Reply to this email directly, view it on GitHub
<#1200 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCVYQJVMNX6OFIKI3HUFPLXQDHNJANCNFSM6AAAAAAXAJFYR4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Yep that's the plan. |
|
The latest code in master adds support for gamepad motion sensor emulation using device sensors (off by default). Let me know how that works for you all. |
I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even with DS4 and all options checked (Sunshine nightly) |
|
Could you please share a link with the android build, build on appveyor and
I can't build it myself too soon..
…On Sun, Sep 17, 2023, 12:33 Marocco2 ***@***.***> wrote:
The latest code in master adds support for gamepad motion sensor emulation
using device sensors (off by default). Let me know how that works for you
all.
I've compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro
even with DS4 and all options checked (Sunshine nightly)
—
Reply to this email directly, view it on GitHub
<#1200 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Nevermind. Apparently it doesn't work with touch controls. Attaching a real controller and disabling gamepad motion but leaving emulation enabled will work as intended |
I can share my compiled version: https://send.vis.ee/download/24e7ddf2be463119/#QGyF4fUiOUAJ_9vh13L6Ug |
|
Thanks, unfortunately this doesn't work for me either, using a oneplus 7t
pro paired with a razer kishi. Ds4 gyro is passed through as it was in
previous builds though.
…On Sun, Sep 17, 2023, 13:27 Marocco2 ***@***.***> wrote:
Could you please share a link with the android build, build on appveyor
and I can't build it myself too soon..
… <#m_-2870162213784852305_>
On Sun, Sep 17, 2023, 12:33 Marocco2 *@*.*> wrote: The latest code in
master adds support for gamepad motion sensor emulation using device
sensors (off by default). Let me know how that works for you all. I've
compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even
with DS4 and all options checked (Sunshine nightly) — Reply to this email
directly, view it on GitHub <#1200 (comment)
<#1200 (comment)>>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4
<https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4>
. You are receiving this because you were mentioned.Message ID: @.*>
I can share my compiled version:
https://send.vis.ee/download/24e7ddf2be463119/#QGyF4fUiOUAJ_9vh13L6Ug
—
Reply to this email directly, view it on GitHub
<#1200 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCVYQJNPH337T2B6ZLTRG3X23GB3ANCNFSM6AAAAAAXAJFYR4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Boy do I feel stupid, forgot to check the frigging box :))
Works wonderfully!
…On Sun, Sep 17, 2023, 13:43 Adrian Coman ***@***.***> wrote:
Thanks, unfortunately this doesn't work for me either, using a oneplus 7t
pro paired with a razer kishi. Ds4 gyro is passed through as it was in
previous builds though.
On Sun, Sep 17, 2023, 13:27 Marocco2 ***@***.***> wrote:
> Could you please share a link with the android build, build on appveyor
> and I can't build it myself too soon..
> … <#m_-4829164415291168043_m_-2870162213784852305_>
> On Sun, Sep 17, 2023, 12:33 Marocco2 *@*.*> wrote: The latest code in
> master adds support for gamepad motion sensor emulation using device
> sensors (off by default). Let me know how that works for you all. I've
> compiled and tried it on a Pixel 7 Pro but Steam won't recognize gyro even
> with DS4 and all options checked (Sunshine nightly) — Reply to this email
> directly, view it on GitHub <#1200 (comment)
> <#1200 (comment)>>,
> or unsubscribe
> https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4
> <https://github.com/notifications/unsubscribe-auth/ACCVYQNIMVFNWWA44G7F7E3X227XHANCNFSM6AAAAAAXAJFYR4>
> . You are receiving this because you were mentioned.Message ID: @.*>
>
> I can share my compiled version:
> https://send.vis.ee/download/24e7ddf2be463119/#QGyF4fUiOUAJ_9vh13L6Ug
>
> —
> Reply to this email directly, view it on GitHub
> <#1200 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACCVYQJNPH337T2B6ZLTRG3X23GB3ANCNFSM6AAAAAAXAJFYR4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
This allows for a Breath-of-the-wild-ish experience. This does confuse a couple of games such as Red dead redemption (where the tips keep switching between keyboard/mouse and joystick very fast).
This also makes using the touchscreen as mouse a bit weird.
Maybe this should add the mouse movement to the joystick readings instead?
anyway, opening this for visibility and feedback =]
Related to #1155 ?