Adding two sequence-level and consistency-based uncertainty measures: Predictive Kernel Entropy and Spectral uncertainty

[nassimwalha]

• Predictive Kernel Entropy (PKE): https://arxiv.org/abs/2310.05833
• Spectral Uncertainty (SU): https://arxiv.org/pdf/2509.22272

Both methods rely on the following stat calculators:

• BlackboxSamplingGenerationCalculator or SamplingGenerationCalculator to generate multiple samples from the target llm for a single input query.
• SampleSentenceEmbeddingsCalculator (newly created in this pull request), to compute sentence embeddings of the sampled answers using a SentenceTransformer model. The SentenceTransformer model is managed by the newly created SentenceEmbedder class in lm_polygraph.utils.sentence_embedder (implemented analogously to Dberta).

PKE computes pairwise kernel values of the embeeddings and then averages them out, SU creates an empirical kernel matrix out of the pairwise values, normalizes it and computes its Von Neumann Entropy.

PS: In the paper, SU comes with a decomposition into aleatoric and epistemic uncertainty using input clarifications. As this is not necessairly specific to SU and can be applied to any other sequence-level uncertainty method, implementing the decomposition will come in a future pull request.

Additional change: compatibility fixes of the class AutoModelForVision2Seq for transformers >= 5.0 (these were included in some parts of the repo already, but not everywhere)