cuda: extract Q1_0 elements via __byte_perm

[dfriehs]

Overview

Unpack Q1_0 elements via __byte_perm, leading to a nice increase in t/s (+5-10%) and a modest one for pp (+1-2.5%). I expect this PR to stay in draft until after #24127 is merged, at which point I will rebase. I consider it complete, though.

Tagging @khosravipasha, in case you are interested in this one as well.

Additional information

test-backend-ops test passes before and with 8373d2b, and KL divergence is 0 between the two. n=4 shows a slowdown in test-backend-ops perf, but llama-batched-bench shows increased t/s even for B=4. Maybe there is some tuning to update?

I'm not able to test HIP/ROCm or MUSA. If either don't support __byte_perm or slow down I will add a fallback path.

test-backend-ops perf

before 8373d2b:

Backend 1/2: CUDA0
  Device description: NVIDIA GeForce RTX 3090
  Device memory: 24120 MB (23374 MB free)

  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   443040 runs -  22.60 us/run - 117.44 MFLOP/run -  5.20 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   386382 runs -  25.89 us/run - 234.88 MFLOP/run -  9.07 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   330292 runs -  30.30 us/run - 352.32 MFLOP/run - 11.63 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   273705 runs -  36.54 us/run - 469.76 MFLOP/run - 12.85 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   225378 runs -  44.39 us/run - 587.20 MFLOP/run - 13.23 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   139849 runs -  71.55 us/run - 939.52 MFLOP/run - 13.13 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):  16034 runs - 623.73 us/run -  60.13 GFLOP/run - 96.40 TFLOPS
  Backend CUDA0: OK

with 8373d2b:

Backend 1/2: CUDA0
  Device description: NVIDIA GeForce RTX 3090
  Device memory: 24120 MB (23250 MB free)

  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   511200 runs -  19.58 us/run - 117.44 MFLOP/run -  6.00 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   482232 runs -  20.74 us/run - 234.88 MFLOP/run - 11.33 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   349036 runs -  28.65 us/run - 352.32 MFLOP/run - 12.30 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   253896 runs -  39.40 us/run - 469.76 MFLOP/run - 11.92 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   239400 runs -  41.78 us/run - 587.20 MFLOP/run - 14.06 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   148302 runs -  67.43 us/run - 939.52 MFLOP/run - 13.93 TFLOPS
  MUL_MAT(type_a=q1_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):  16404 runs - 609.66 us/run -  60.13 GFLOP/run - 98.63 TFLOPS
  Backend CUDA0: OK

llama-batched-bench

merged output of
llama-batched-bench -no-kvu -ngl all -fit off -ub 1024 -npp 0,256,4096 -ntg 256 -npl 1,2,4,8 -m Bonsai-8B-Q1_0.gguf
and
llama-batched-bench -no-kvu -ngl all -fit off -ub 1024 -npp 16384,32768 -ntg 256 -npl 1 -m Bonsai-8B-Q1_0.gguf

before 8373d2b:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	1.141	224.31	1.141	224.31
0	256	2	512	0.000	0.00	1.401	365.40	1.401	365.40
0	256	4	1024	0.000	0.00	2.015	508.09	2.015	508.09
0	256	8	2048	0.000	0.00	2.786	735.10	2.786	735.10
256	256	1	512	0.051	5032.44	1.136	225.40	1.187	431.47
256	256	2	1024	0.094	5455.92	1.435	356.72	1.529	669.65
256	256	4	2048	0.181	5663.62	2.089	490.15	2.270	902.22
256	256	8	4096	0.345	5933.45	2.886	709.70	3.231	1267.77
4096	256	1	4352	0.740	5532.66	1.317	194.31	2.058	2114.87
4096	256	2	8704	1.464	5595.90	1.779	287.73	3.243	2683.61
4096	256	4	17408	2.917	5616.59	2.777	368.71	5.694	3057.07
4096	256	8	34816	5.826	5624.60	4.214	486.05	10.039	3467.93
16384	256	1	16640	3.708	4419.06	1.832	139.74	5.540	3003.85
32768	256	1	33024	9.600	3413.21	2.535	100.98	12.135	2721.28

with 8373d2b:

PP	TG	B	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s	T s	S t/s
0	256	1	256	0.000	0.00	1.017	251.66	1.017	251.66
0	256	2	512	0.000	0.00	1.118	457.92	1.118	457.92
0	256	4	1024	0.000	0.00	1.809	566.01	1.809	566.01
0	256	8	2048	0.000	0.00	2.564	798.83	2.564	798.83
256	256	1	512	0.050	5167.54	1.008	254.04	1.057	484.28
256	256	2	1024	0.092	5577.10	1.153	444.08	1.245	822.66
256	256	4	2048	0.176	5829.71	1.884	543.52	2.060	994.34
256	256	8	4096	0.338	6066.06	2.664	768.90	3.001	1364.81
4096	256	1	4352	0.722	5674.43	1.188	215.49	1.910	2278.72
4096	256	2	8704	1.427	5740.23	1.488	344.10	2.915	2985.89
4096	256	4	17408	2.842	5765.03	2.563	399.55	5.405	3220.80
4096	256	8	34816	5.676	5772.59	3.974	515.39	9.650	3607.81
16384	256	1	16640	3.647	4492.57	1.704	150.23	5.351	3109.74
32768	256	1	33024	9.476	3458.00	2.407	106.37	11.883	2779.16

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: No.