linux: add support for driver 595.80 and 610.43.02

[Gelmo]

Fixes: #1081

Fixes: #1079

Note: I am not running 595.80 so I am unable to test. I did confirm the patched string, so this is expected to work. Someone running this driver should test.

I did test 610.43.02 with NvFBCCUDAAsync, which was successful.

HOWEVER, for some reason gpu-screen-recorder is crashing with a general protection fault when using libnvidia-encode.so.610.43.02. I haven't determined why just yet, and MOST of the nvidia samples run fine. This might be the same issue seen here - #889

I'm submitting this as a draft in hopes that others will test as well, for both drivers included in this PR. If other people are having issues, I will further review to see if there was a more significant change that requires more digging. I'm hoping that nobody else has issues and this is just an issue with gpu-screen-recorder.

Tests from Capture_Linux_v9.0.0:

❮ ./NvFBCCUDAAsync
Application version: 5
NvFBC API version: 1.9

Selected backend: X11
Creating an asynchronous capture session of 10 frames with 1 second internal between captures.
Reallocated 6075 KB of system memory
New frame id 1 grabbed in 0 ms, downloaded in 2 ms, saved in 18 ms, now sleeping for 980 ms
Frame id 2 grabbed in 0 ms, downloaded in 0 ms, saved in 18 ms, now sleeping for 982 ms
Frame id 3 grabbed in 0 ms, downloaded in 1 ms, saved in 17 ms, now sleeping for 982 ms
New frame id 4 grabbed in 0 ms, downloaded in 1 ms, saved in 17 ms, now sleeping for 982 ms
New frame id 5 grabbed in 0 ms, downloaded in 3 ms, saved in 17 ms, now sleeping for 980 ms
Frame id 6 grabbed in 0 ms, downloaded in 2 ms, saved in 18 ms, now sleeping for 980 ms
New frame id 7 grabbed in 0 ms, downloaded in 7 ms, saved in 18 ms, now sleeping for 975 ms
New frame id 8 grabbed in 0 ms, downloaded in 7 ms, saved in 18 ms, now sleeping for 975 ms
New frame id 9 grabbed in 0 ms, downloaded in 5 ms, saved in 18 ms, now sleeping for 977 ms
New frame id 10 grabbed in 0 ms, downloaded in 5 ms, saved in 18 ms, now sleeping for 977 ms

❮ ./NvFBCMultiThread
Application version: 4
NvFBC API version: 1.9

Selected backend: X11
Thread 0: creating a capture session of 10 RGB frames cropped to 2680x1440+0+0 and of size 2680x1440.
Selected backend: X11
Thread 1: creating a capture session of 10 RGB frames cropped to 2680x1440+2680+0 and of size 2680x1440.
Thread 0: New frame id 1 grabbed in 18 ms, saved in 35 ms.
Thread 1: New frame id 1 grabbed in 19 ms, saved in 35 ms.
Thread 0: New frame id 2 grabbed in 7 ms, saved in 34 ms.
Thread 1: New frame id 2 grabbed in 12 ms, saved in 34 ms.
Thread 0: New frame id 3 grabbed in 15 ms, saved in 34 ms.
Thread 1: New frame id 3 grabbed in 15 ms, saved in 34 ms.
Thread 0: New frame id 4 grabbed in 13 ms, saved in 36 ms.
Thread 1: New frame id 4 grabbed in 15 ms, saved in 37 ms.
Thread 0: New frame id 5 grabbed in 14 ms, saved in 37 ms.
Thread 1: New frame id 5 grabbed in 11 ms, saved in 34 ms.
Thread 0: New frame id 6 grabbed in 10 ms, saved in 35 ms.
Thread 1: New frame id 6 grabbed in 15 ms, saved in 34 ms.
Thread 0: New frame id 7 grabbed in 15 ms, saved in 35 ms.
Thread 1: New frame id 7 grabbed in 15 ms, saved in 34 ms.
Thread 0: New frame id 8 grabbed in 14 ms, saved in 35 ms.
Thread 1: New frame id 8 grabbed in 15 ms, saved in 34 ms.
Thread 0: New frame id 9 grabbed in 14 ms, saved in 35 ms.
Thread 1: New frame id 9 grabbed in 15 ms, saved in 34 ms.
Thread 0: New frame id 10 grabbed in 14 ms, saved in 35 ms.
Thread 1: New frame id 10 grabbed in 15 ms, saved in 34 ms.

Possible issue with Pipewire?

❮ ./NvFBCPipeWire
Application version: 1
NvFBC API version: 1.9

Creating a capture session without a restore token...

pci id for fd 10: 10de:2482, driver (null)
pci id for fd 11: 10de:2482, driver (null)
pci id for fd 10: 10de:2482, driver (null)
pci id for fd 11: 10de:2482, driver (null)

Destroying the capture session...
A capture session has not been created for this NvFBC client

Long times with this test:

❮ ./NvFBCSharedContext
Application version: 2
NvFBC API version: 1.9

Worker thread: Capturing 10 RGB frames of size 5360x1440.
Worker thread: New frame id 1 grabbed in 754 ms, saved in 67 ms.
Worker thread: New frame id 2 grabbed in 4440 ms, saved in 66 ms.
Worker thread: New frame id 3 grabbed in 433 ms, saved in 67 ms.
Worker thread: New frame id 4 grabbed in 4929 ms, saved in 66 ms.
Worker thread: New frame id 5 grabbed in 4945 ms, saved in 66 ms.
Worker thread: New frame id 6 grabbed in 4930 ms, saved in 65 ms.
Worker thread: New frame id 7 grabbed in 418 ms, saved in 67 ms.
Worker thread: New frame id 8 grabbed in 48 ms, saved in 67 ms.
Worker thread: New frame id 9 grabbed in 34 ms, saved in 67 ms.
Worker thread: New frame id 10 grabbed in 254 ms, saved in 66 ms.

I'm able to reproduce the GPF with NvFBCToGLEnc:

❮ ./NvFBCToGLEnc
Application version: 2
NvFBC API version: 1.9

Selected backend: X11
X framebuffer size is 5360x1440.
fish: Job 1, './NvFBCToGLEnc' terminated by signal SIGSEGV (Address boundary error)

[ 4835.589046] traps: NvFBCToGLEnc[52452] general protection fault ip:7f053eacdcd0 sp:7ffeddcfe3e8 error:0 in libnvidia-encode.so.610.43.02[6cd0,7f053eacb000+37000]

Since I can reproduce the GPF with that test, I think I need to keep digging a bit.

[Gelmo]

Thread 1 "NvFBCToGLEnc" received signal SIGSEGV, Segmentation fault.
0x00007ffff79abcd0 in ?? () from /usr/lib/libnvidia-encode.so.1
(gdb) bt
#0  0x00007ffff79abcd0 in ?? () from /usr/lib/libnvidia-encode.so.1
#1  0x000055555555bda8 in __frame_dummy_init_array_entry ()
#2  0x00007ffff7ffd000 in ?? ()
#3  0x00007fffffffdd38 in ?? ()
#4  0x00005555555562d3 in main (argc=<error reading variable: Cannot access memory at address 0xffffffffffffbb7c>, argv=<error reading variable: Cannot access memory at address 0xffffffffffffbb70>) at NvFBCToGLEnc.c:539
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[Gelmo]

Application version: 2
NvFBC API version: 1.9

Selected backend: X11
X framebuffer size is 5360x1440.
Process 102041 stopped
* thread #1, name = 'NvFBCToGLEnc', stop reason = signal SIGSEGV: sent by kernel (SI_KERNEL)
    frame #0: 0x00007ffff79abcd0 libnvidia-encode.so.1`___lldb_unnamed_symbol_6c10 + 192
libnvidia-encode.so.1`___lldb_unnamed_symbol_6c10:
->  0x7ffff79abcd0 <+192>: outl   %eax, %dx
    0x7ffff79abcd1 <+193>: movq   %r12, %rsi
    0x7ffff79abcd4 <+196>: callq  *(%rax)
    0x7ffff79abcd6 <+198>: movl   %eax, %r13d
(lldb) bt
* thread #1, name = 'NvFBCToGLEnc', stop reason = signal SIGSEGV: sent by kernel (SI_KERNEL)
  * frame #0: 0x00007ffff79abcd0 libnvidia-encode.so.1`___lldb_unnamed_symbol_6c10 + 192
    frame #1: 0x000055555555bda8 NvFBCToGLEnc`__frame_dummy_init_array_entry + 8

[MystifiedSky]

Testing PR #1083 on Unraid 7.3.1 with NVIDIA driver 610.43.02 and an RTX 2080 Ti.

Environment:

OS: Unraid 7.3.1
GPU: NVIDIA GeForce RTX 2080 Ti
Driver: 610.43.02
NVIDIA-SMI: 610.43.02
CUDA UMD Version: 13.3
Docker runtime: NVIDIA runtime via Unraid NVIDIA Driver plugin

I tested the PR branch:

git clone --branch 610.43.02 --single-branch https://github.com/Gelmo/nvidia-patch.git /tmp/nvidia-patch-pr1083
cd /tmp/nvidia-patch-pr1083
bash ./patch.sh

The patch script reported success:

Detected nvidia driver version: 610.43.02
libnvidia-encode.so
Attention! Backup not found. Copying current libnvidia-encode.so to backup.
Patched!

However, after applying the patch, NVENC encoding failed immediately using this FFmpeg test:

docker run --rm --runtime=nvidia \
  -e NVIDIA_VISIBLE_DEVICES=all \
  -e NVIDIA_DRIVER_CAPABILITIES=all \
  jrottenberg/ffmpeg:6.1-nvidia \
  -hide_banner \
  -loglevel verbose \
  -f lavfi -i testsrc2=size=1920x1080:rate=30 \
  -t 30 \
  -c:v h264_nvenc \
  -preset p4 \
  -f null -

With the patched library, the test exited almost immediately and showed:

ERROR: driverInitFileInfo 578 result=11
ERROR: init 664 result=11
ERROR: init 250 result=11

I then restored the original backup library:

cp -a /opt/nvidia/libnvidia-encode-backup/libnvidia-encode.so.610.43.02 /usr/lib64/libnvidia-encode.so.610.43.02
ldconfig

After restoring the original library, the same FFmpeg NVENC test worked and nvidia-smi showed ffmpeg actively using the GPU:

GPU Memory Usage: ~358 MiB
Process: ffmpeg

So on my setup, the PR patch applies and reports Patched!, but the patched libnvidia-encode.so.610.43.02 breaks NVENC initialization. Restoring the original backup library immediately makes NVENC work again.

Let me know if you want the SHA1/SHA256 hashes of the patched and original libraries or any additional logs.

[Gelmo]

@MystifiedSky Thank you for testing! This confirms that the issue is not just on my end. I will look further into this. Looks like the changes are a bit more significant this time around but I should be able to find it pretty easily once I can dedicate more time to it. MAYBE later tonight, but likely not until tomorrow or Saturday. At least NvFBC is working for now; it's just NvENC that needs to be fixed. On the bright side, whatever we find should work for most of 610.x