[FEAT] The Road to WebRTC

[ehfd]

Is this a new feature request?

• I have searched the existing issues

Wanted change

• 1. Stabilize input - We cannot compromise on latency in any case; It needs to include coroutines aggressively across the code base, since WebRTC will do what the WebSockets data stream did using DataChannels.
• Implement something similar to https://github.com/Xpra-org/xpra/blob/a5a13a993139d9d48a60734e7bab3ec9d53c919a/xpra/server/mixins/input.py#L257 , to handle repeated keys gracefully in disconnections or congestion.
• Implement the ability to choose between including custom headers natively in Pixelflux/PCMflux (For WebSockets, to not have an extra copy inside Python) vs not including any custom headers set by Pixelflux/PCMflux at all (for WebRTC).
• Implement the ability to request the keyframe/IDR Frame (same thing) upstream from the web client to Pixelflux.
• With time, migrate more of the code to asyncio instead of thread-blocking code.
• Then, WebRTC can be wired up.

Reason for change

WebRTC is managed completely by the browser stacks and is free from CPU consumption or Latency issues. This is the de-facto standard protocol for cloud gaming-level or 3D workloads as well.

Proposed code change

Above.

[ehfd]

From a collaborator:

I gave aiortc a try and wired the audio and video of gstreamer pipeline to aiortc tracks just to get some hands-on. Aiortc does support pre-encoded data, but with caveat, as you highlighted, an IDR frame request has to be handled by us explicitly. For that I modified the aiortc rtcrtpsender to emit the pli event and assigned a handler to trigger the an IDR frame on video pieline.
It's all working, but the video seem to lag at times, need to look into it. Will try with pixelflux shortly. I guess to request an IDR frame from pixelflux requires some communication back from application logic to pixelflux package. In gstreamer it's simple, we just need to push event upstream request for key_frame.

I know we are leaving gstreamer behind, I just tried this to see things.

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[ehfd]

Should be kept open.

[hgw77]

My understanding was that Webtop already utilizes WebRTC through Selkies and Gstreamer?

Since the current streaming method works very well with older, less powerful client hardware (e.g. old Linux based Laptops/Macbooks), I hope the stack changes will maintain this compatibility.

[ehfd]

My understanding was that Webtop already utilizes WebRTC through Selkies and Gstreamer?

To note, Webtop uses WebSocket as the main method now. We implemented WebRTC separately, but with almost all of the same optimizations as WebSocket.