Intel Consumer Level AVX-VNNI Support

[callmegaga]

Reminder

• I have read the above rules and searched the existing issues.

System Info

I am not sure.Does kt support the intel consumer level avx-vnni instructions whitch is 256 bit, it is different with server side avx-vnni which is 512 bit.Do you assess whether doing this is worthwhile? If so, I can do this work.I'm also trying to make the NPU, iGPU, and CPU work together on Intel processors.Can you give me some advice?

Reproduction

Put your message here.

Others

No response

[ErvinXie]

Hey, thanks so much for reaching out — and for being willing to contribute!

To clarify the current state: kt-kernel supports AVX2 for consumer CPUs, but AVX2-VNNI (the 256-bit VNNI variant on Alder/Raptor Lake P-cores) is not yet explicitly supported. So there's a real opportunity here!

Adding AVX2-VNNI support would be a meaningful contribution — it would unlock better INT8 performance on a wide range of consumer hardware without needing AMX or AVX512. If you're up for it, we'd love to see what you can do. A good starting point would be the kernel dispatch logic in ktransformers/operators/cpu_backend/.

For the NPU+iGPU+CPU direction — that's a really cool idea and something we find genuinely interesting. It's not our current focus, but we'd be excited to see experiments in that direction from the community.

Looking forward to seeing your work!

---

This reply was co-authored by an AI agent and a human.

[callmegaga]

Okay, if these two points are technically feasible, then I will complete the relevant development. Thank you.

[wqshmzh]

LOOKING FORWARD TO THE IMPROVEMENT ON SUPPORTING VNNI !

[NukePeace]

VNNI int8 does not work that effectively due to the quantization range for Q4_K_M will not all be within int8, but vnni int16 is the best thing intel ever implemented, despite only a side product of vnni int8-- expect a 2x acceleration from it though, that's still a lot.

[blazingphoenix7]

I have been working in exactly this lane: RAWINT4 serving on consumer CPUs with the 256-bit AVX-VNNI (here an Arrow Lake Core Ultra 5 225: AVX2 + AVX-VNNI, no AVX512, no AMX).

First result is a correctness fix. The AVX-VNNI-256 RAWINT4 backend could not load weights through the serving path at all, because its TP wrapper only accepted flat weight buffers while NativeMoEWrapper always passes per-expert pointers. Fixed in #2092 together with per-commit equivalence tests over the RAWINT4 load paths.

On performance there is real headroom in the current kernel. Cycle-counted microbenchmarks of the AVX-VNNI-256 RAWINT4 GEMM on this machine (rdtscp, arms interleaved per round, min over rounds; kernel level, not end to end): reworking the accumulator scheme gives 1.4x to 1.8x on decode shapes depending on shape and group size, and streaming the weights as packed nibbles instead of the load-time-unpacked int8 reaches about 2.2x on large prefill shapes. The packed-nibble direction also halves resident expert weight memory, since the backend currently unpacks int4 to int8 at load time. I plan to send these incrementally, each with the same equivalence-test treatment.