[FEAT] `Trackpad mode` for touchscreens

[aptalca]

Is this a new feature request?

• I have searched the existing issues

Wanted change

Implement a new touchscreeen mode that can be enabled or switched to where the touchscreen is treated as a trackpad.

• Cursor position would be static
• Placing finger anywhere on the screen and swiping up by 100 pixels would move the cursor up by 100 pixels.
• Tapping on the screen would register a click at the current cursor position (rather than the finger tap spot).
• Tap and hold would register a right click action.
• Two finger moves would register a scroll action

Reason for change

Current touchscreen method registers a click at the tap or touch location on the screen, which is very difficult on smaller screens like a small tablet or a phone. Double click is also very difficult as the taps are likely registered in different spots.

The trackpad mode would provide very precise actions on touchscreen devices, especially on smaller screens.

Bonus: The trackpad mode also allows for displaying part of a screen on a small screen device and scrolling left and right as well as up and down as needed, following the cursor, so one can have a larger resolution or desktop size than what the small device allows.

Proposed code change

Here's some relevant info: https://bbs.archlinux.org/viewtopic.php?id=283077

Some more relevant info: https://www.oesf.org/forum/index.php?topic=35185.0

Video demonstration: https://www.youtube.com/watch?v=bsJn3eMvG7w

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[thelamer]

selkies-project/selkies@d51f99f
@aptalca noble base is ready for testing.
The icon to activate it is in the sidebar at the top looks like a trackpad, it replaces the gaming mode on touch.
With your approval will add to all of the bases.

[aptalca]

I've been testing this on a selkies noble rebased calibre and it works really well. I can actually use it on a cell phone screen, which is pretty impressive.

I can confirm that the trackpad button shows up on the menu in chrome Android (but the press status isn't saved in the browser so every time I reopen, I have to switch to it).
Single click works. Click and hold does the right click (although I have to fire up the rdp and chrome remote desktop apps to double check the common behavior because I can't fully recall. They might have the right click action assigned to a 2 finger tap instead and the click and hold may trigger drag. We may want to keep it consistent with them. I'll let you know).

[thelamer]

The drag is what I am interested in, the only thing I could think of is you would have a click and hold for a length of time to actually hold the cursor.
Just from a logical standpoint and how it works touching the screen and moving your finger has to move the cursor and it cant click it every time you do that so how would you activate this is the question.

[aptalca]

My understanding of this tech is next to nil, but I'll attempt an explanation based on my assumptions.
My main assumption is that whatever underlying library is used, detects touch actions and converts them to a compatible mouse input. Below is a list of actions and expected mouse inputs

Touch action	Mouse Input
One finger quick tap	Left click
One finger swipe	Directional mouse movement
Two finger quick tap	Right click
One finger quick tap and then hold second touch	Hold left click (until touch release)
Two finger swipe	Scroll wheel (up/down and left/right)

With a mouse, a drag is achieved by holding left click and moving the mouse. On the touchscreen in trackpad mode, that would be achieved by tapping twice but holding the second tap, which would be interpreted as a left click hold, and then swiping the finger, which would last until the touch is released.

Does that make sense?

[thelamer]

Yeah perfect sense will get this in

[thelamer]

Calibre demo image has been updated.

[aptalca]

Hot damn. It all works. 2 finger tap for right click, 2 finger swipe for scroll, even drag and drop via tap and hold.

Is it possible to store the state of mode selection in device browser cache? Right now, I have to enable the trackpad mode every time I open the page on the same device.

[thelamer]

It should be stored now I tested that too, try clearing cache and app storage or test from a clean private mode.
It should hydrate the setting and tell the server to show the cursor.
We are rolling Arch with the features but a side effect is client cache and localstorage is a burden.
It works on my phone in a clean context opening and closing but as a larger issue:
#10
We need settings to be a first class citizen right now it is not, also on top of what this user is explaining modularity to what is presented the the user, every setting should be able to be set limited or capped via env by admins and the user is only presented those options and attempts from the client to manually set them will be denied.

It is basic application setup stuff and is boring but it needs to be done.

[aptalca]

One minor request. Can we flip the scroll direction? Currently when you move your fingers down, the screen content moves up, just like it does when you use a trackpad or mouse to scroll. However when you use a touchscreen, you want the content to move in the same direction as your fingers.