[Enhancement] Test client CPU and GPU consumption through `MediaStreamTrackGenerator`/`VideoTrackGenerator`

[ehfd]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

• Client browser CPU and GPU consumption is still pretty high compared to the original Selkies (but much lower than KasmVNC), likely due to the addition of <canvas> instead of solely using <video> when it can use the latter only - use <canvas> only with stripe encode mode. High 3D rasterization utilization from the GPU along with CPU spikes, not video decode. The spikes are NOT in the striped encode mode, they are in NVENC and x264enc modes.
• Direct usage of MediaStreamTrackGenerator/VideoTrackGenerator would be zero-copy in Chromium and Safari, so there are no disadvantages and only advantages. However, Firefox doesn't support this yet.
• The usage of OffscreenCanvas in the polyfill https://jan-ivar.github.io/polyfills/mediastreamtrackgenerator.js is very interesting as well (nonetheless, it will only be used in Firefox). There is a tradeoff between latency and resource consumption compared to directly painting as a Canvas through drawImage.
• Audio should stay in AudioWorkletNode.

Expected Behavior

No response

Steps To Reproduce

Reproduce in a power-efficient laptop that has power constraints.

Environment

Ubuntu 24.04

Docker creation

git clone https://github.com/selkies-project/selkies.git
cd selkies
git checkout -f main
docker run --rm -it \
  --shm-size=1gb \
  -e DEV_MODE=selkies-dashboard \
  -e PUID=1000 \
  -e PGID=1000 \
  -v $(pwd):/config/src \
  -p 3000:3000 \
  -p 3001:3001 linuxserver/webtop:debian-xfce bash

Container logs

N/A.

[thelamer]

This is known, that is why striped is not default

[ehfd]

@thelamer The latter of the checkmarks can still be implemented eventually in non-striped mode (to skip canvas and directly use video). The issue is in non-striped mode; it just gets worse in striped mode.

[ehfd]

Possible through MediaStreamTrackGenerator/VideoTrackGenerator. The only catch is that it is not in Firefox, and other browsers have it in a Window but not a DedicatedWorker. This is fixable through a polyfill below.

https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamTrackGenerator
https://developer.mozilla.org/en-US/docs/Web/API/VideoTrackGenerator
https://webrtchacks.com/real-time-video-processing-with-webcodecs-and-streams-processing-pipelines-part-1/
w3c/webcodecs#212
https://github.com/webrtc/samples/blob/gh-pages/src/content/capture/worker-process/js/main.js

Issue that needs to be resolved: mdn/browser-compat-data#24569

Polyfill for MediaStreamTrackGenerator:
https://blog.mozilla.org/webrtc/unbundling-mediastreamtrackprocessor-and-videotrackgenerator/

Example: https://github.com/w3c/webtransport/blob/main/samples/webcodecs-echo/js/main.js

For transporting the processed image somewhere while keeping latency low, you could consider WebTransport or WebRTC’s RTCDataChannel.
For rendering, you could render directly to a canvas through drawImage, using WebGPU, or via an <video> element through VideoTrackGenerator (also defined in MediaStreamTrack Insertable Media Processing using Streams).

[thelamer]

Given the polyfill just writes then reads off a canvas do you still think this is worth pursuing. There is no logical way this would improve performance.

[ehfd]

The polyfill is a fallback; if it doesn't fallback it is the same pathway to WebRTC MediaStreams.

This is zero-copy on Chromium and Safari, so there are no disadvantages; only advantages.