diff --git a/.gitignore b/.gitignore index 14e628d..ef33457 100644 --- a/.gitignore +++ b/.gitignore @@ -76,3 +76,6 @@ r/beam/vignettes/*.html # Generated example reports (built in CI by scripts/generate_example_reports.py) docs/reports/*.html + +/.quarto/ +**/*.quarto_ipynb diff --git a/README.md b/README.md index 4f443f8..df11431 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ beam_run("beam.yaml") - [Documentation site](https://imallona.github.io/beam/): vignettes, how-tos, explanations, and the Python API reference. - Ontology release: `docs/beam.owl.ttl` (OWL) and `docs/beam.skos.ttl` (SKOS), regenerated from the cards on each release. -## Licence +## License - Code: GPL-3.0-or-later (`LICENSE`). - Metric cards under `src/beam/metrics/`: CC-BY-4.0 (`src/beam/metrics/LICENSE.md`). diff --git a/_quarto.yml b/_quarto.yml index 0b73e71..baaa139 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -50,54 +50,32 @@ website: text: Normalization and scales - href: docs/explanations/aggregation-methods.md text: Aggregation methods - - href: docs/explanations/aggregation-agreement.md - text: Aggregation agreement - - href: docs/explanations/normalization-agreement.md - text: Normalization agreement + - href: docs/explanations/choice-agreement.md + text: Choice agreement - href: docs/explanations/rank-sensitivity.md - text: What moves the ranking - - href: docs/explanations/specification-curve.md - text: Specification curve + text: Rank sensitivity and the specification curve - href: docs/explanations/analysis-blinding.md text: Analysis blinding - href: docs/explanations/weighting-schemes.md text: Weighting schemes - - href: docs/explanations/comparing-methods-across-datasets.md - text: Comparing methods across datasets - - href: docs/explanations/dataset-concordance.md - text: Dataset concordance - - href: docs/explanations/dataset-discrimination.md - text: Dataset discrimination + - href: docs/explanations/critical-difference.md + text: Critical difference + - href: docs/explanations/dataset-concordance-and-discrimination.md + text: Dataset concordance and discrimination - href: docs/explanations/attribution-synthesis.md text: Attribution synthesis - - href: docs/explanations/skillings-mack.md - text: Skillings-Mack - - href: docs/explanations/pairwise-superiority.md - text: Pairwise superiority - - href: docs/explanations/transitivity.md - text: Pairwise transitivity - - href: docs/explanations/bayesian-comparison.md - text: Bayesian comparison + - href: docs/explanations/pairwise-method-comparison.md + text: Pairwise method comparison - href: docs/explanations/missing-data.md text: Missing data - href: docs/explanations/reference-levels.md text: Reference levels - href: docs/explanations/card-data-consistency.md text: Card and data consistency - - href: docs/explanations/convergent-discriminant-validity.md - text: Metric validity - - href: docs/explanations/reliability.md - text: Metric reliability - - href: docs/explanations/dimensionality.md - text: Metric dimensionality - - href: docs/explanations/heterogeneity-mixed-effects.md - text: Mixed-effects heterogeneity - - href: docs/explanations/heterogeneity-bradley-terry.md - text: Bradley-Terry trees - - href: docs/explanations/full-rankings-and-bounded-metrics.md - text: Plackett-Luce and bounded metrics - - href: docs/explanations/comet.md - text: COMET + - href: docs/explanations/metric-set-diagnostics.md + text: Metric-set diagnostics + - href: docs/explanations/method-by-dataset-heterogeneity.md + text: Method-by-dataset heterogeneity - href: docs/explanations/funky-heatmaps-and-robustness.md text: Funky heatmaps - href: docs/explanations/ontology-mappings.md diff --git a/docs/explanations/aggregation-agreement.md b/docs/explanations/aggregation-agreement.md deleted file mode 100644 index 99e9408..0000000 --- a/docs/explanations/aggregation-agreement.md +++ /dev/null @@ -1,37 +0,0 @@ -# Aggregation agreement: does the benchmark result depend on the aggregation method? - -beam offers [five aggregations](aggregation-methods.md): SAW, TOPSIS, VIKOR, PROMETHEE II and [COMET](comet.md). Each combines the per-metric scores into a composite under each aggregation method own assumptions, and the headline ranking commits to one of them The question tat follows is whether that commitment changes the evaluation results. [`beam.mcda.aggregation_agreement`](../reference/aggregation_agreement.qmd) addresses this by re-ranking the same (normalized) input metrics matrix under each aggregation, holding the weighting fixed. Then, the agreement of the ranks is reported. - -This can be read as a diagnostic of [choice sensitivity](rank-sensitivity.md), complementing beam's leave-one-metric-out and leave-one-dataset-out functions and weight variation via stochastic multicriteria acceptability analysis (SMAA). This aggregation agreement procedure evaluates the remaining degree of freedom to choose a final rank. - -## Method - -[Normalization](normalization-and-scales.md) and [weighting](weighting-schemes.md) happen before aggregation and do not depend on the aggregation rule, so the weight vector is the same across the five methods. So only the aggregation step is evaluated here. The diagnostic runs the full pipeline once per method, collects the per-tool ranks, and compares every pair of rankings. - -Agreement between two rankings is measured with the Kendall tau-b coefficient. tau-b counts the concordant and discordant pairs of tools and corrects for ties, which matters here because beam uses competition ranking, so ties are common. tau-b is 1 for identical orderings, -1 for exact reversals, and near 0 when the two methods order the tools randomly. The report plots the full method-by-method tau matrix and a single mean of its off-diagonal entries as a summary. - -The report also gives a consensus ranking, the ranking of the per-method mean ranks, and a diagnostic flag reporting whether the top ranked method is stable across aggregations. The per-tool smallest and largest rank across the methods are the rank span drawn in the consensus panel of the [funky heatmap](funky-heatmaps-and-robustness.md). - -## Interpretation - -A high mean tau means the recommendation is stable under the aggregation choice: whichever method a reader prefers, they reach about the same order. A low mean tau means the aggregation rule is itself deciding the order. - -The [Duo 2018 clustering benchmark](../../examples/duo2018/duo2018.qmd) is a worked example. On the three-metric pooled matrix (the Adjusted Rand Index, runtime, Shannon entropy difference) under equal weights, the five aggregations agree at a mean pairwise tau-b of 0.65, and Seurat is the unanimous top under every one of them. So the top recommendation does not depend on the aggregation choice. The agreement is not complete, though. PROMETHEE II correlates only 0.34 to 0.54 with the others, so the ordering in the middle of the ranking depends on which method is used. - -## How to use it - -Call `aggregation_agreement(scores, polarity)` with a tool by metric matrix and the per-metric polarity from [`beam.cards.polarities_for`](../reference/polarities_for.qmd). Pass the normalization context from `beam.mcda.registry_context` so the comparison rests on the same normalized matrix as the headline ranking. The weighting is held fixed at whatever you pass; the default is equal weights. - -In case an aggregation method fails to run on the input then it is dropped from the report rather than failing the whole analysis. At least two methods must produce a ranking, otherwise there is nothing to compare. When every tool scores identically the orderings are all-ties, tau-b is undefined and the run fails. - -## What it does not do - -The diagnostic measures whether the methods agree, not which one is correct. The five aggregations encode different value judgements: TOPSIS rewards closeness to an ideal point, VIKOR seeks a compromise solution, PROMETHEE II nets pairwise outranking flows. A disagreement among them is a disagreement about what a composite metric should do. The consensus ranking is a convenience summary. Typically, incompatible results from different aggregation methods point to some disagreement that has to be further evaluated. - -## See also - -- [Aggregation methods](aggregation-methods.md) - -## References - -- Kendall, M. G. A new measure of rank correlation. Biometrika 30 (1938). DOI [10.1093/biomet/30.1-2.81](https://doi.org/10.1093/biomet/30.1-2.81). diff --git a/docs/explanations/aggregation-methods.md b/docs/explanations/aggregation-methods.md index 8936e05..9111a21 100644 --- a/docs/explanations/aggregation-methods.md +++ b/docs/explanations/aggregation-methods.md @@ -1,6 +1,6 @@ # Aggregation methods -After [normalizing](normalization-and-scales.md) the metrics results, beam holds a tool by metric matrix in the unit interval, with every column oriented so higher is better, plus a weight per metric. Aggregation turns that matrix into one preference score per tool, which then becomes a ranking. beam offers four aggregation methods on this page: SAW, TOPSIS, VIKOR and PROMETHEE II ([COMET](comet.md) has its own page). They differ in what they compute and in what they assume about the scale of the normalized scores. +After [normalizing](normalization-and-scales.md) the metrics results, beam holds a tool by metric matrix in the unit interval, with every column oriented so higher is better, plus a weight per metric. Aggregation turns that matrix into one preference score per tool, which then becomes a ranking. beam offers five aggregation methods: SAW, TOPSIS, VIKOR, PROMETHEE II and COMET. They differ in what they compute and in what they assume about the scale of the normalized scores. beam wraps pymcdm for these methods rather than reimplementing their algorithms. Each beam function calls the matching pymcdm method with an identity normalization, so pymcdm runs on beam's already normalized matrix, and with every metric typed as positive because the matrix is oriented higher is better. beam keeps the shared input and output contract, applies the higher-is-better convention described below, and handles the corner cases (for example a single tool) that pymcdm does not. @@ -14,7 +14,7 @@ Simple additive weighting (SAW) is the dot product of the normalized scores and ## TOPSIS -TOPSIS measures each tool's distance to two reference points. It weights the matrix, then finds the ideal solution (the best value in each weighted column) and the anti-ideal (the worst). Each tool's score is its relative closeness: the distance to the anti-ideal divided by the sum of the distances to the ideal and the anti-ideal. A tool near the ideal and far from the anti-ideal scores near 1. Like SAW, TOPSIS uses Euclidean distance on the weighted values, so it assumes an interval scale where differences are comparable across metrics. As a key difference with SAW, TOPSIS rewards a tool for being close to the best on every metric at once, not just for a high total. So a balanced tool can rank above a tool with one very high and one very low score, even when their sums match. +TOPSIS measures each tool's distance to two reference points. It weights the matrix, then finds the ideal solution (the best value in each weighted column) and the anti-ideal (the worst). Each tool's score is its relative closeness: the distance to the anti-ideal divided by the sum of the distances to the ideal and the anti-ideal. A tool near the ideal and far from the anti-ideal scores near 1. Like SAW, TOPSIS uses Euclidean distance on the weighted values, so it assumes an interval scale where differences are comparable across metrics. Unlike SAW, TOPSIS rewards a tool for being close to the best on every metric at once, not just for a high total. So a balanced tool can rank above a tool with one very high and one very low score, even when their sums match. ## VIKOR @@ -32,9 +32,17 @@ PROMETHEE II compares every ordered pair of tools rather than scoring each tool Because the usual preference function looks only at the sign of each difference, PROMETHEE II in this default form is ordinal per metric. That is, it uses the order of the tools on a metric, not the size of the gaps. This makes it the lightest method on scale assumptions of the four. Brans and Vincke define five other preference functions that soften this with indifference and preference thresholds in the metric's own units, which add back the size of the differences. +## COMET + +COMET, the Characteristic Objects Method (Salabun 2015), is the one aggregation whose ranking is rank reversal free: adding or removing a tool cannot change the order of the others. The four methods above score each tool against the other tools present, so the reference they use shifts when the set of tools changes. That matters for a growing leaderboard, such as community benchmarking in hackathons, or a sensitivity analysis that drops one method at a time. COMET instead fits on a fixed grid of reference points, the characteristic objects, and scores the tools against that grid afterward. beam wraps pymcdm for it. + +The grid is the Cartesian product of a few characteristic values per metric. beam uses the endpoints of the normalized scale, $0$ and $1$, by default, so with two metrics the objects are the four corners of the unit square; a caller can pass an interior value such as $0.5$ to add anchors. Ordering the objects needs a judgement of which is better, supplied in COMET's original form by a human expert comparing them pairwise. A benchmarking pipeline has none, so beam uses a deterministic rule: one characteristic object outranks another when the weighted sum of its coordinates is larger. Each tool is then scored by triangular fuzzy interpolation between the surrounding objects, a preference in $[0, 1]$ where higher is better. Because the grid and its ordering are fixed before any tool is scored, two tools receive the same scores whatever else is in the table. + +COMET costs more than the others. The number of characteristic objects grows as the product of the per-metric value counts, so it is exponential in the number of metrics: ten metrics with two values each is already $2^{10} = 1024$ objects, and the scores are harder to read by hand than a weighted sum. It suits cases where rank stability matters more than simplicity. + ## Choosing an aggregation method -SAW is the easiest choice when transparency matters and the metrics are on a comparable interval scale. TOPSIS and VIKOR both reward balance across metrics, with VIKOR adding explicit control for how much a single weak metric should count. PROMETHEE II is the most conservative about scale, since in its default form it reads only the order of the tools on each metric. Running more than one method and comparing the rankings is itself a sensitivity check: a recommendation that holds across methods is more reliable than one that depends on the choice of aggregation. This is what [aggregation agreement](aggregation-agreement.md) measures. +SAW is the easiest choice when transparency matters and the metrics are on a comparable interval scale. TOPSIS and VIKOR both reward balance across metrics, with VIKOR adding explicit control for how much a single weak metric should count. PROMETHEE II is the most conservative about scale, since in its default form it reads only the order of the tools on each metric. COMET is the choice when the tool set changes over time and the order should not shift as tools are added or dropped, at the cost of the exponential grid. Running more than one method and comparing the rankings is itself a sensitivity check: a recommendation that holds across methods is more reliable than one that depends on the choice of aggregation. This is what [aggregation agreement](choice-agreement.md) measures. The nature of the input metrics and their redundancy, e.g. speed or memory versus biologically-driven evaluation, and their (lack of) correlation, impact the aggregation method choice. @@ -44,4 +52,5 @@ The nature of the input metrics and their redundancy, e.g. speed or memory versu - Opricovic, S. Multicriteria optimization of civil engineering systems. Faculty of Civil Engineering, Belgrade (1998). - Opricovic, S. and Tzeng, G.-H. Compromise solution by MCDM methods: a comparative analysis of VIKOR and TOPSIS. European Journal of Operational Research (2004). DOI [10.1016/S0377-2217(03)00020-1](https://doi.org/10.1016/S0377-2217%2803%2900020-1). - Brans, J.-P. and Vincke, P. A preference ranking organisation method: the PROMETHEE method for multiple criteria decision-making. Management Science (1985). DOI [10.1287/mnsc.31.6.647](https://doi.org/10.1287/mnsc.31.6.647). +- Salabun, W. The Characteristic Objects Method: a new distance-based approach to multicriteria decision-making problems. Journal of Multi-Criteria Decision Analysis (2015). DOI [10.1002/mcda.1525](https://doi.org/10.1002/mcda.1525). - OECD. Handbook on Constructing Composite Indicators (2008), on weighting and aggregation. DOI [10.1787/9789264043466-en](https://doi.org/10.1787/9789264043466-en). diff --git a/docs/explanations/analysis-blinding.md b/docs/explanations/analysis-blinding.md index 4ce1ea9..457151f 100644 --- a/docs/explanations/analysis-blinding.md +++ b/docs/explanations/analysis-blinding.md @@ -2,7 +2,7 @@ If a benchmarker can see which method is which while choosing the [weighting](weighting-schemes.md), the [aggregation](aggregation-methods.md) and the metric set, those choices can shift toward a preferred method. This can happen via unconscious biases and without intent. Different choices give different rankings, and it is easy to settle on the one that fits an expected result (Gelman and Loken 2013; Simmons, Nelson and Simonsohn 2011). One way to avoid it is blind analysis: run the analysis on methods whose labels are hidden, then reveal the labels (MacCoun and Perlmutter 2015; Klein and Roodman 2005). -`beam.blind` applies this to a score table. It replaces the tool names with opaque labels and shuffles the rows under a seed, so neither the names nor the order carry the methods' identity. The analyst runs the full beam pipeline on the blinded scores, fixes the configuration, then calls `beam.unblind` with the seal to restore the true names. +`beam.blind` applies this to a score table. It replaces the tool names with opaque labels and shuffles the rows under a seed, so neither the names nor the order carry the methods' identity. The analyst runs the full beam pipeline on the blinded scores, fixes the configuration, then calls `beam.unblind` with the seal to restore the true names. Unblinding only renames the rows and does not re-rank the methods. ```python import beam @@ -27,11 +27,7 @@ beam is controlled by users and not the way around. To add an extra layer of aud "blinding": {"blinded": true, "seal_sha256": "6ee805357098..."} ``` -The seal file, kept separately, records that the configuration was fixed before the labels were revealed. A reviewer who has the manifest and the seal can confirm the analysis ran on scores blinded under that seal. This mechanisms suits preregistration well, but does not enforce it. - -## Why the blinding does not affect results - -Unblinding only renames the rows and does not re-rank methods. +The seal file, kept separately, records that the configuration was fixed before the labels were revealed. A reviewer who has the manifest and the seal can confirm the analysis ran on scores blinded under that seal. This mechanism suits preregistration well, but does not enforce it. ## Usage @@ -43,7 +39,7 @@ beam unblind result.json --seal seal.json --out result.unblinded.json ## See also -- [Specification curve](specification-curve.md) +- [Specification curve](rank-sensitivity.md#the-specification-curve) - [Rank sensitivity](rank-sensitivity.md) ## References diff --git a/docs/explanations/attribution-synthesis.md b/docs/explanations/attribution-synthesis.md index b9f5c38..ea3f9ad 100644 --- a/docs/explanations/attribution-synthesis.md +++ b/docs/explanations/attribution-synthesis.md @@ -1,8 +1,8 @@ # Attribution synthesis -A benchmark ranking is affected by three major factors. The analyst's choices, including the [weighting](weighting-schemes.md) and the [aggregation rule](aggregation-methods.md). The datasets: a method that ranks first on one dataset trails on another. The benchmarker also does two benchmarks of the same task, each with its own pipeline and methods and datasets, rank shared methods differently. beam already measures these one at a time. [`rank_sensitivity`](rank-sensitivity.md) splits a ranking between the choices and the dataset. [`source_variance_decomposition`](heterogeneity-mixed-effects.md) splits a method's standing between the benchmark and the method. The two use different scales, so they cannot be compared directly. +A benchmark ranking is affected by three factors. The analyst's choices, the [weighting](weighting-schemes.md) and the [aggregation rule](aggregation-methods.md). The datasets: a method that ranks first on one dataset trails on another. And the benchmarker: two benchmarks of the same task, each with its own pipeline, methods and datasets, rank the shared methods differently. beam already measures these one at a time. [`rank_sensitivity`](rank-sensitivity.md) splits a ranking between the choices and the dataset. [`source_variance_decomposition`](method-by-dataset-heterogeneity.md) splits a method's standing between the benchmark and the method. The two use different scales, so they cannot be compared directly. -[`attribution_synthesis`](../reference/attribution_synthesis.qmd) puts them on one scale. For each setting it gives three independent axis, for analyst choice, dataset and benchmarker, that sum to one. Compared across settings, from one benchmark to a contrast where the datasets are held fixed, the axes show how the source of the movement changes as the dataset contribution is removed. +[`attribution_synthesis`](../reference/attribution_synthesis.qmd) puts them on one scale. For each setting it gives three independent axes, for analyst choice, dataset and benchmarker, that sum to one. Compared across settings, from one benchmark to a contrast where the datasets are held fixed, the axes show how the source of the movement changes as the dataset contribution is removed. ## Approach @@ -18,5 +18,5 @@ The axes are descriptive, without confidence intervals, and limited by data avai ## See also -- [Mixed-effects model](heterogeneity-mixed-effects.md) -- [Specification curve](specification-curve.md) +- [Mixed-effects model](method-by-dataset-heterogeneity.md) +- [Specification curve](rank-sensitivity.md#the-specification-curve) diff --git a/docs/explanations/bayesian-comparison.md b/docs/explanations/bayesian-comparison.md deleted file mode 100644 index 487d766..0000000 --- a/docs/explanations/bayesian-comparison.md +++ /dev/null @@ -1,34 +0,0 @@ -# Bayesian comparison of two methods - -The [critical-difference diagram](comparing-methods-across-datasets.md) and the [pairwise superiority](pairwise-superiority.md) report p-value, as they test the chance of the observed split if the two methods scored the same. For analysts choosing a method that score is not that informative; ideally, the probability that one method scores higher than the other given the data is more informative. [`beam.mcda.bayesian_sign_comparison`](../reference/bayesian_sign_comparison.qmd) reports the latter, using the Bayesian sign test of Benavoli et al. (2017) on the same per-pair counts [`pairwise_superiority`](../reference/pairwise_superiority.qmd) produces. - -## Procedure - -For a method pair, each shared dataset falls into one of three regions: A scores higher by more than the region of practical equivalence (the ROPE), B scores higher by more than the ROPE, or the two are within the ROPE. The ROPE is taken from the superiority report, often the metric's [noise floor](reference-levels.md), so a difference too small to interpret counts as a tie. - -The share of datasets in each region is unknown. The test models the three shares with a Dirichlet posterior whose parameters are the observed counts plus a small prior. The function reports three probabilities that sum to one: that A is practically better than B, that the two are practically equivalent, and that B is practically better than A. - -A pair gets a decisive label when one of the three reaches the threshold (0.95 by default), and is inconclusive otherwise. The function also reports the posterior mean share of each region, the Dirichlet mean. It reports a standing score per method: the mean over the other methods of the probability of scoring at least as high as them. - -The default places one prior pseudo-observation on the equivalence region, matching the baycomp default. Two other placements are available: `uniform` spreads the prior across the three regions, and `neutral` spreads it across the two directional regions. One pseudo-observation is outweighed after a few datasets. - -## Relation to the other comparison tools - -beam's four comparison tools read the same tool by dataset evidence and answer different questions: - -1. The critical-difference diagram tests separability on one metric. -2. The pairwise superiority report gives a frequentist effect size and a sign-test p-value. -3. The [transitivity check](transitivity.md) asks whether the pairwise majorities admit one consistent order. -4. The Bayesian sign test reports the probability that A is practically better than B. - -## Limits - -The posterior describes the datasets in hand, not a population of datasets. With few datasets most pairs are inconclusive at the threshold. The result depends on the ROPE, so set the band from the metric's noise floor. The sign test uses only the direction of each dataset's difference, not its size. - -## References - -Benavoli A, Corani G, Demsar J, Zaffalon M. Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis. Journal of Machine Learning Research 2017, 18(77):1-36. - -Corani G, Benavoli A. A Bayesian approach for comparing cross-validated algorithms on multiple data sets. Machine Learning 2015, 100(2-3):285-304. - -The reference implementation is the baycomp package (), against which beam is cross-checked. diff --git a/docs/explanations/card-data-consistency.md b/docs/explanations/card-data-consistency.md index ce42105..588fbd6 100644 --- a/docs/explanations/card-data-consistency.md +++ b/docs/explanations/card-data-consistency.md @@ -1,8 +1,8 @@ # Card and data consistency -A metric card is a formalization and hence, to some extent, a contract. It declares the metric value range, a baseline of what obtainable by chance, an ideal target, and a noise floor. The [multi-criteria decision analysis (MCDA) pipeline](cards-and-pipeline.qmd) trusts each one: the range bounds the [normalization](normalization-and-scales.md), the baseline anchors `baseline_relative` scaling and the beats-chance check, the target drives `target_relative` scaling, and the [noise floor](reference-levels.md) sets which method differences are interpretable. If the score matrix contradicts a declared value, every later step loses usefulness. +A metric card declares the metric value range, a baseline of what is obtainable by chance, an ideal target, and a noise floor. The [multi-criteria decision analysis (MCDA) pipeline](cards-and-pipeline.qmd) reads each of these: the range bounds the [normalization](normalization-and-scales.md), the baseline anchors `baseline_relative` scaling and the beats-chance check, the target drives `target_relative` scaling, and the [noise floor](reference-levels.md) sets which method differences are interpretable. If the score matrix contradicts a declared value, every later step loses usefulness. -[`beam.mcda.card_data_consistency`](../reference/card_data_consistency.qmd) reads the raw scores against the card values, before any normalization, and reports where they disagree. It is the data-facing companion to the schema validation: `validate.py` checks that the requested aggregation is licit for the declared [scale type](measurement-theory.md), and this audit checks that the data honours the declared values. +[`beam.mcda.card_data_consistency`](../reference/card_data_consistency.qmd) reads the raw scores against the card values, before any normalization, and reports where they disagree. It is the data-facing companion to the schema validation: `validate.py` checks that the requested aggregation is licit for the declared [scale type](measurement-theory.md), and this audit checks the data against the declared values. A recurring failure is a unit mismatch. A metric defined on the `[0, 1]` fraction scale is reported as a percentage, so the column runs to 100 against a card that declares the `[0, 1]` range. The column is still numeric and still interval, so it passes the schema validation and the scale-versus-method check. It then distorts the min-max normalization for that metric, which stretches its contribution to the composite, and the [weighting](weighting-schemes.md) that rests on it. The audit catches this on the raw scores: it reports the metric by name, counts how many tools fall outside the range, and gives the worst value, in one pass over every metric. @@ -26,7 +26,7 @@ The `ok` flag is true when there are no violations. Notes do not clear it but ar ## Redundant checks -[`beam.mcda.normalize`](../reference/normalize.qmd) also has a narrow guard: it raises when a column's minimum or maximum falls outside the declared bounds. That guard fires inside the ranking call, stops at the first offending column, names a column index rather than the metric, and checks only the range. `card_data_consistency` is the comprehensive standalone audit. It reads all metrics in one pass, names each one, grades the findings, and adds the baseline, target, noise-floor, and degeneracy checks the normalization guard does not make. +[`beam.mcda.normalize`](../reference/normalize.qmd) also has a narrow guard: it raises when a column's minimum or maximum falls outside the declared bounds. That guard fires inside the ranking call, stops at the first offending column, names a column index rather than the metric, and checks only the range. `card_data_consistency` is the full standalone audit. It reads all metrics in one pass, names each one, grades the findings, and adds the baseline, target, noise-floor, and degeneracy checks the normalization guard does not make. ## Running diff --git a/docs/explanations/cards-and-pipeline.qmd b/docs/explanations/cards-and-pipeline.qmd index 0768b49..2ad8d91 100644 --- a/docs/explanations/cards-and-pipeline.qmd +++ b/docs/explanations/cards-and-pipeline.qmd @@ -1,4 +1,4 @@ -# What the multi-criteria decision analysis (MCDA) pipeline reads from the metric cards +# Metric cards and the MCDA pipeline A metric card declares fields covering identity, kind, inputs, output, semantics, comparability, implementations, examples, and provenance. diff --git a/docs/explanations/choice-agreement.md b/docs/explanations/choice-agreement.md new file mode 100644 index 0000000..874f75e --- /dev/null +++ b/docs/explanations/choice-agreement.md @@ -0,0 +1,31 @@ +# Choice agreement: aggregation and normalization + +Two analyst choices can move a ranking with no change to the data: which [aggregation rule](aggregation-methods.md) combines the per-metric scores, and which [normalization](normalization-and-scales.md) rescales them first. beam has a paired diagnostic for each. Both re-rank the same matrix under the alternatives and report how far the rankings agree. They join the other checks on the analyst's degrees of freedom: leave-one-metric-out, leave-one-dataset-out, the weight sampling in stochastic multicriteria acceptability analysis (SMAA), and the [variance decomposition](rank-sensitivity.md) that splits a ranking across all of those choices at once. + +## Measuring agreement + +Each diagnostic runs the full pipeline once per alternative, collects the per-tool ranks, and compares every pair of rankings with the Kendall tau-b coefficient. tau-b counts the concordant and discordant pairs of tools and corrects for ties, which matters here because beam uses competition ranking, so ties are common. It is 1 for identical orderings, -1 for exact reversals, and near 0 when two orderings are unrelated. The report carries the full tau matrix, the mean of its off-diagonal entries as a summary, a consensus ranking (the ranking of the per-alternative mean ranks), and a flag for whether the top method holds across the alternatives. The per-tool smallest and largest rank become the rank span the [funky heatmap](funky-heatmaps-and-robustness.md) draws. + +## Aggregation agreement + +[`beam.mcda.aggregation_agreement`](../reference/aggregation_agreement.qmd) varies the aggregation rule across the five aggregations: SAW, TOPSIS, VIKOR, PROMETHEE II and [COMET](aggregation-methods.md#comet). Normalization and [weighting](weighting-schemes.md) happen before aggregation and do not depend on the rule, so the weight vector is the same across the five and only the aggregation step changes. + +A high mean tau means the recommendation is stable under the aggregation choice. A low mean tau means the aggregation rule is itself deciding the order. On the [Duo 2018 clustering benchmark](../../examples/duo2018/duo2018.qmd), on the three-metric pooled matrix (the Adjusted Rand Index, runtime, Shannon entropy difference) under equal weights, the five aggregations agree at a mean pairwise tau-b of 0.65, and Seurat is the unanimous top under every one of them. PROMETHEE II still correlates only 0.34 to 0.54 with the others, so the middle of the ranking depends on which method is used. + +Call `aggregation_agreement(scores, polarity)` with a tool by metric matrix and the per-metric polarity from [`beam.cards.polarities_for`](../reference/polarities_for.qmd), and pass the normalization context from `beam.mcda.registry_context` so the comparison rests on the same normalized matrix as the headline ranking. An aggregation that fails on the input is dropped rather than failing the analysis; at least two must produce a ranking. When every tool scores identically the orderings are all-ties, tau-b is undefined and the run fails. + +The five aggregations encode different value judgements: TOPSIS rewards closeness to an ideal point, VIKOR seeks a compromise solution, PROMETHEE II nets pairwise outranking flows. A disagreement among them is a disagreement about what a composite metric should do. The consensus ranking is a convenience summary. + +## Normalization agreement + +[`beam.mcda.normalization_agreement`](../reference/normalization_agreement.qmd) varies the normalization instead. By default it compares the four scale-agnostic strategies, `min_max`, `log_min_max`, `rank` and `zscore`, each applied to every column. `baseline_relative` and `target_relative` are left out of the uniform sweep: they need a per-metric reference or target from the card and are tied to the metric's meaning, so they are not a free choice the analyst makes column by column. The card-recommended per-metric normalization can be passed as one labelled candidate, and `beam.rank` passes it as the `recommended` column so the report compares the headline default against the uniform alternatives. + +One difference from aggregation agreement: the objective [weights](weighting-schemes.md) (entropy, standard deviation, CRITIC, MEREC) are computed from the normalized matrix, so changing the normalization also changes those weights. That is intended, and the report shows the normalization's total effect on the order, not just the part that acts through the aggregation step. + +On the same Duo 2018 matrix under equal weights, the card-recommended normalization and the four uniform strategies agree at a mean pairwise tau-b of 0.64, and Seurat is again the unanimous top. The recommended normalization agrees with `min_max` at tau-b 0.79 but with `rank` at only 0.36, because the rank normalization drops the gap sizes that the other strategies keep, so the ordering in the middle of the table moves. + +Call `normalization_agreement(scores, polarity)` with `recommended=RegistryContext.normalization` and the same `bounds`, `baselines` and `targets` the headline ranking used. A candidate that cannot run is dropped: `log_min_max` needs strictly positive values, and a `target_value` metric admits only `target_relative`, so the uniform strategies all drop on a matrix that contains one. `beam.plot.normalization_agreement(run)` draws the tau-b heatmap and `beam.plot.normalization_effect(run)` a bump chart of each tool's rank across the strategies; the funky heatmap gains a normalization rank-span panel with `show_normalization_consensus=True`. When the strategies disagree, report the disagreement and the reason the card recommends what it does, covered in [Normalization and scales](normalization-and-scales.md). + +## References + +- Kendall, M. G. A new measure of rank correlation. Biometrika 30 (1938). DOI [10.1093/biomet/30.1-2.81](https://doi.org/10.1093/biomet/30.1-2.81). diff --git a/docs/explanations/comet.md b/docs/explanations/comet.md deleted file mode 100644 index 46440d4..0000000 --- a/docs/explanations/comet.md +++ /dev/null @@ -1,36 +0,0 @@ -# COMET - -COMET, the Characteristic Objects METhod (Salabun 2015), is one of the [aggregation methods](aggregation-methods.md) in the multi-criteria decision analysis (MCDA) pipeline. It takes the same input as the others, a matrix of tools by metrics already [normalized](normalization-and-scales.md) to the unit interval and oriented so higher is better, plus a [weight per metric](weighting-schemes.md), and returns one preference value per tool. What sets it apart is that its ranking is rank reversal free: adding or removing a tool cannot change the order of the others. This page explains how it reaches that property and when to use COMET over weighted sum or TOPSIS. - -beam wraps pymcdm for COMETf. beam supplies the characteristic values and the expert rule described below, and pymcdm builds the Matrix of Expert Judgement, the preference of each characteristic object, and the fuzzy interpolation that scores the tools. - -## The rank reversal problem and COMET's approach - -Most aggregation methods score a tool by comparing it to the other tools in the table. TOPSIS measures distance to the best and worst observed values, so the scale it uses depends on which tools are present. Weighted sum is closer to safe, but once a normalization step rescales each column to the observed minimum and maximum, the same dependence returns. The practical consequence is rank reversal: drop a poor method from the comparison, recompute, and two of the survivors can swap places even though nothing about them changed. For a leaderboard that grows over time, _such as those for community benchmarking in hackathons_, or a sensitivity analysis that drops one method at a time, this is a practical problem. The order is supposed to describe the methods, not which other methods are present. - -COMET avoids the problem by never fitting its model on the tools. Instead it fits on a fixed grid of reference points called characteristic objects, and the tools are scored afterward against that fixed grid. - -The grid is built per criterion. For each metric the method picks a few characteristic values that span its scale. beam uses the endpoints of the normalized scale by default, $0$ and $1$, which is two characteristic values per metric. A caller can pass three values per metric, for example $0$, $0.5$ and $1$, to give the model an interior anchor. The characteristic objects are the Cartesian product of these per-metric values: with two metrics and the endpoints, the four objects are the four corners of the unit square; with three values each, the nine points of a $3 \times 3$ grid. The grid does not depend on the tools, so it does not move when a tool is added or removed. That is the source of the rank reversal free property. - -COMET still needs to know which characteristic objects are better. In its original form a human expert compares every pair of objects and records the judgement in a matrix, the Matrix of Expert Judgement (MEJ). Entry $(i, j)$ is $1$ when object $i$ is preferred to object $j$, $0.5$ when they are judged equal, and $0$ when $j$ is preferred. The row sums of this matrix give the Summed Judgement (SJ) of each object. Grouping objects by distinct SJ value and spreading those groups evenly across the unit interval gives each object a preference $P$, from $0$ for the lowest group to $1$ for the top. - -A benchmarking pipeline has no human expert in the loop, so beam supplies a deterministic one. The expert rule here is the weighted sum of a characteristic object's coordinates: object a is preferred to object b when its weighted sum is larger, equal when the two weighted sums are equal. beam passes this rule to pymcdm as the expert function, and pymcdm builds the MEJ from it. This is simple additive weighting used as a pairwise judge of the reference grid. It is auditable, it reproduces, and it lets the method run without a person rating object pairs. The choice of expert is the one design decision a COMET user makes that the other methods do not expose. - -Once every characteristic object has a preference $P$, a tool is scored by fuzzy interpolation between the surrounding objects. Each characteristic value of a metric carries a triangular fuzzy number peaked at that value: membership is $1$ at the value, falls linearly to $0$ at the neighbouring characteristic values, and is $0$ beyond them. A tool's score on one metric turns into a set of membership degrees, one per characteristic value, that sum to $1$ and say how close the tool sits to each anchor on that metric. The membership degrees for a whole characteristic object are the product of the per-metric memberships across criteria, the tensor product over the grid. The tool's preference is the sum over all characteristic objects of that object's preference $P$ weighted by the tool's membership in it. A tool sitting exactly on a corner of the grid inherits that corner's preference; a tool in between is a blend of the surrounding corners. The result is in $[0, 1]$ and higher is better. - -Because the preferences $P$ are fixed before any tool is scored, two tools always receive the same scores no matter which other tools are in the table. That is what makes the method rank reversal free. - -## When to prefer COMET - -COMET costs more than weighted sum or TOPSIS, in two senses. The number of characteristic objects grows as the product of the per-metric value counts, so it is exponential in the number of metrics: ten metrics with two values each is already $2^{10} = 1024$ objects. And the scores are harder to read off by hand than a weighted sum. So it is not the default. - -COMET suits cases where rank stability matters more than simplicity. It can be cost-effective for living benchmarks, where the comparison set changes over time. - -## See also - -- [Aggregation agreement](aggregation-agreement.md) -- [Rank sensitivity](rank-sensitivity.md) - -## References - -- Salabun, W.. The Characteristic Objects Method: a new distance-based approach to multicriteria decision-making problems. Journal of Multi-Criteria Decision Analysis (2015). DOI [10.1002/mcda.1525](https://doi.org/10.1002/mcda.1525). diff --git a/docs/explanations/comparing-methods-across-datasets.md b/docs/explanations/comparing-methods-across-datasets.md deleted file mode 100644 index 33f8c26..0000000 --- a/docs/explanations/comparing-methods-across-datasets.md +++ /dev/null @@ -1,42 +0,0 @@ -# Comparing methods across datasets: Friedman and Nemenyi - -The [multi-criteria decision analysis (MCDA) aggregation](aggregation-methods.md) builds a metric composite to give a rank and consolidated score to each method. It does not say whether that ranking is real. If the methods are close and the influence of the datasets is large, the order at the top can be an artifact of which datasets happened to be included. Demsar (2006) gives the standard answer for this setting: rank the methods on each dataset, then test the ranks. beam implements it in [`beam.mcda.critical_difference`](../reference/critical_difference.qmd). - -## Implementation - -The input is a tool by dataset matrix for one metric or for an MCDA composite. On each dataset the methods are ranked, with 1 for the best. Each method then has an average rank across the datasets. The Friedman test asks whether these average ranks differ more than chance would produce if all methods were equivalent. A small p-value means at least one method is consistently ahead or behind. - -The Friedman test does not say which methods differ from which. For that, beam runs the Nemenyi post-hoc. Its critical difference is the smallest gap between two average ranks that counts as significant at the chosen level. Two methods whose average ranks differ by less than the critical difference cannot be told apart from the data at hand. - -## The critical difference - -The Nemenyi critical difference is $q \sqrt{k (k + 1) / (6 N)}$, where $k$ is the number of methods, $N$ the number of datasets, and $q$ the Studentized range value for $k$ at the chosen alpha, divided by the square root of two. beam computes $q$ exactly with scipy, so it is correct for any number of methods, not only the small tables printed in the paper. As a check, for five methods at alpha 0.05 the $q$ term is 2.728, the value in Demsar's Table 5. - -The formula shows the two ways to earn power: more datasets shrink the critical difference, and fewer methods shrink it. With many methods and few datasets the critical difference is large, and most pairs come out unseparable. - -## Cliques - -A critical-difference diagram draws the methods along a rank axis and connects the ones that are not significantly different. beam returns these groups as cliques: maximal runs of methods, consecutive in rank order, whose first and last average ranks lie within the critical difference. A method that shares no clique with another is significantly separated from it. The cliques are the data behind the diagram; the vignette draws the bars. - -## How to use it - -Call `critical_difference(scores, higher_is_better=True)` with a tool by dataset matrix. Set `higher_is_better=False` for a cost metric such as runtime, so the faster method still ranks near 1. The report carries the average ranks, the Friedman statistic and p-value, the critical difference, and the cliques. - -The test needs at least three methods and at least two datasets. It adds to the MCDA composite rather than replacing it: the composite says which method to prefer under a stated [weighting](weighting-schemes.md), and the Friedman-Nemenyi result says whether the data support drawing a line between the methods at all. - -When the matrix has missing cells the Friedman ranks are no longer defined. The Skillings-Mack (1981) test generalizes the global statistic to incomplete blocks at the cost of the Nemenyi cliques; see [Skillings-Mack: coverage-aware Friedman](skillings-mack.md). - -## Limitations - -The Nemenyi post-hoc compares every pair of methods and is conservative. When the question is whether one new method beats a fixed set of baselines, comparing to a single control with the Bonferroni-Dunn correction has more power, as Demsar notes. beam currently implements the all-pairs Nemenyi case. - -## See also - -- [Pairwise superiority](pairwise-superiority.md) -- [Transitivity](transitivity.md) -- [Bayesian comparison](bayesian-comparison.md) -- [Skillings-Mack: coverage-aware Friedman](skillings-mack.md) - -## References - -- Demsar, J. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7 (2006). diff --git a/docs/explanations/convergent-discriminant-validity.md b/docs/explanations/convergent-discriminant-validity.md deleted file mode 100644 index 5e259c4..0000000 --- a/docs/explanations/convergent-discriminant-validity.md +++ /dev/null @@ -1,32 +0,0 @@ -# Convergent and discriminant validity of a metric set - -Most benchmarks use different performance metrics to score the methods and, in some cases, metrics are grouped depending on the meaning. The single-cell [scIB integration benchmark](../../examples/openproblems/openproblems.qmd) scores groups its metrics into biological conservation and batch correction. The grouping carries an assumption: the metrics in one group measure the same thing, and the metrics in different groups measure different things. [`beam.mcda.metric_validity`](../reference/metric_validity.qmd) tests that assumption against the scores, following Campbell and Fiske (1959). - -## Implementation - -Each method-by-dataset cell is one observation, and the metrics are the variables. The function updates every metric so that higher means better, reading the polarity from the cards and negating a lower-is-better metric. It then computes the Spearman rank correlation between every pair of metrics over the observations they share. - -Spearman, a rank correlation, allows to search for association no matter the metric scale. The correlation is pairwise-complete, and left as NaN otherwise. - -## Usage - -The function requires a label per metric, supplied by the user This is a domain judgement, so beam takes it as an argument rather than reading it from the cards. Typically, labels reflect computational characteristics (max set size, wallclock time; and biology-driven metrics). Grouping the correlations by that label splits them in two. - -Within-group correlations are the convergent evidence. Metrics that claim to measure one construct should agree with each other. Between-group correlations are the discriminant evidence. Metrics that claim to measure different constructs should agree less. When the mean within-group correlation is higher than the mean between-group correlation, the grouping holds up, and treating the groups as separate criteria in the [weighting](weighting-schemes.md) is supported by the data. `discriminant_ok` records that comparison. - -beam reports two kinds of metric discrimination problems or properties: -- A redundant pair is two metrics in the same group whose correlation is at or above a threshold (0.9 by default). They order the methods almost identically, so carrying both adds little and double-counts one construct. One is a candidate to drop. -- A crossloading metric correlates more, on average, with another group than with its own. It behaves more like a different construct than the one its label claims. This is the per-metric form of a discriminant-validity failure, and it points at a metric that is mislabelled or genuinely ambiguous. - -## Example - -On the [OpenProblems batch integration scores](../../examples/openproblems/openproblems.qmd), the bio/batch grouping is supported but weak: mean within-group correlation 0.38 against mean between-group correlation 0.30. The biological metrics agree more among themselves (0.45) than the batch metrics do (0.24), and `graph_connectivity`, a batch metric, correlates more with the biological group than with its own. The reading is that the two axes are separable but not orthogonal. - -## See also - -- [Reliability](reliability.md) -- [Dimensionality](dimensionality.md) - -## References - -- Campbell, D. T., Fiske, D. W.. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin (1959). DOI [10.1037/h0046016](https://doi.org/10.1037/h0046016). diff --git a/docs/explanations/critical-difference.md b/docs/explanations/critical-difference.md new file mode 100644 index 0000000..e50d329 --- /dev/null +++ b/docs/explanations/critical-difference.md @@ -0,0 +1,77 @@ +# Critical difference: Friedman, Nemenyi, Skillings-Mack + +The [multi-criteria decision analysis (MCDA) aggregation](aggregation-methods.md) builds a metric composite to give a rank and consolidated score to each method. It does not say whether that ranking is real. If the methods are close and the influence of the datasets is large, the order at the top can be an artifact of which datasets happened to be included. Demsar (2006) gives the standard answer for this setting: rank the methods on each dataset, then test the ranks. beam implements it in [`beam.mcda.critical_difference`](../reference/critical_difference.qmd). + +## Implementation + +The input is a tool by dataset matrix for one metric or for an MCDA composite. On each dataset the methods are ranked, with 1 for the best. Each method then has an average rank across the datasets. The Friedman test asks whether these average ranks differ more than chance would produce if all methods were equivalent. A small p-value means at least one method is consistently ahead or behind. + +The Friedman test does not say which methods differ from which. For that, beam runs the Nemenyi post-hoc. Its critical difference is the smallest gap between two average ranks that counts as significant at the chosen level. Two methods whose average ranks differ by less than the critical difference cannot be told apart from the data at hand. + +## The critical difference + +The Nemenyi critical difference is $q \sqrt{k (k + 1) / (6 N)}$, where $k$ is the number of methods, $N$ the number of datasets, and $q$ the Studentized range value for $k$ at the chosen alpha, divided by the square root of two. beam computes $q$ exactly with scipy, so it is correct for any number of methods, not only the small tables printed in the paper. As a check, for five methods at alpha 0.05 the $q$ term is 2.728, the value in Demsar's Table 5. + +The formula shows the two ways to earn power: more datasets shrink the critical difference, and fewer methods shrink it. With many methods and few datasets the critical difference is large, and most pairs come out unseparable. + +## Cliques + +A critical-difference diagram draws the methods along a rank axis and connects the ones that are not significantly different. beam returns these groups as cliques: maximal runs of methods, consecutive in rank order, whose first and last average ranks lie within the critical difference. A method that shares no clique with another is significantly separated from it. The cliques are the data behind the diagram; the vignette draws the bars. + +## How to use it + +Call `critical_difference(scores, higher_is_better=True)` with a tool by dataset matrix. Set `higher_is_better=False` for a cost metric such as runtime, so the faster method still ranks near 1. The report carries the average ranks, the Friedman statistic and p-value, the critical difference, and the cliques. + +The test needs at least three methods and at least two datasets. It adds to the MCDA composite rather than replacing it: the composite says which method to prefer under a stated [weighting](weighting-schemes.md), and the Friedman-Nemenyi result says whether the data support drawing a line between the methods at all. + +When the matrix has missing cells the Friedman ranks are no longer defined. The Skillings-Mack (1981) test generalizes the global statistic to incomplete blocks at the cost of the Nemenyi cliques, covered below. + +## Skillings-Mack for incomplete blocks + +Real benchmarks rarely give a complete matrix: methods time out, error on an input, or were not run on every dataset. beam's [missing-data policy](missing-data.md) does not fill those gaps by imputation, so `critical_difference` reduces the matrix to its complete observations, and on a wide table with many partial methods there may be none. The Skillings-Mack (1981) test fills the gap with a Friedman-type statistic that does not need a complete matrix, in [`beam.mcda.skillings_mack`](../reference/skillings_mack.qmd) and under the alias [`beam.mcda.coverage_aware_critical_difference`](../reference/coverage_aware_critical_difference.qmd). + +It answers the same global question, whether the methods are separable across the datasets, on the same tool by dataset matrix with NaN allowed. It gives no pairwise comparisons. The Nemenyi critical difference assumes every pair is ranked on every dataset with equal block sizes; with incomplete blocks the per-method average rank no longer shares a denominator, so beam returns the global test only and leaves pairwise statements to a `critical_difference` run on the complete block. + +Within each block (column) $j$ the methods present are ranked from 1 (lowest score) to $k_j$ (highest score), with average ranks for ties. The within-block rank for method $i$ is centred and standardized by the block size: + +$$ +A_{ij} = \left(R_{ij} - \frac{k_j + 1}{2}\right) \sqrt{\frac{12}{k_j + 1}} +$$ + +The factor $\sqrt{12 / (k_j + 1)}$ makes the variance of $A_{ij}$ under the null (the method's rank is uniform over $1, \dots, k_j$) equal to $k_j - 1$, whatever the block size. Summing over the blocks where method $i$ appears gives the per-method statistic $A_i$. The vector $A$ has length equal to the number of methods and sums to zero. Its null covariance is + +$$ +\Sigma_{ii} = \sum_{\text{blocks } j \text{ containing method } i} (k_j - 1) +$$ + +$$ +\Sigma_{ij} = -(\text{number of blocks containing both } i \text{ and } j), \quad i \neq j +$$ + +Every row of $\Sigma$ sums to zero, so $\Sigma$ is rank-deficient by one. Dropping any single row and column gives a positive definite $(n - 1) \times (n - 1)$ submatrix that can be inverted, and the test statistic is the quadratic form + +$$ +T = A_{\text{reduced}}^{\top} \, \Sigma_{\text{reduced}}^{-1} \, A_{\text{reduced}} +$$ + +which is $\chi^2$ distributed with $n_{\text{methods}} - 1$ degrees of freedom under the null. Which row and column is dropped does not affect the statistic, because $A$ lies in the column space of $\Sigma$. + +On a complete matrix Skillings-Mack collapses to the Friedman $\chi^2$: every block has the same $k$, the standardizing factor is constant, and the covariance structure matches the rank-sum formulation. beam tests this equivalence at every random seed in the test suite to within $10^{-10}$, with one caveat. scipy's `friedmanchisquare` applies a tie correction that divides the statistic by $1 - T / (k (k^2 - 1) N)$, where $T$ is the standard ties term; the 1981 Skillings-Mack formulation does not. The two statistics differ when there are within-block ties, and agree to machine precision without them. + +Two ways to use the pair: + +- Restrict the matrix to the complete block of methods and datasets where every method ran, and call `critical_difference`. This gives a global test and the Nemenyi cliques but drops every dataset where any method failed to run. It fits when the complete block is large enough. +- Keep every observed score and call `skillings_mack` on the partial matrix. This gives a global test only, with no pairwise cliques. It fits when restricting to the complete block drops most of the data. + +## Limitations + +The Nemenyi post-hoc compares every pair of methods and is conservative. When the question is whether one new method beats a fixed set of baselines, comparing to a single control with the Bonferroni-Dunn correction has more power, as Demsar notes. beam currently implements the all-pairs Nemenyi case. + +## See also + +- [Pairwise method comparison](pairwise-method-comparison.md) + +## References + +- Demsar, J. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7 (2006). +- Skillings, J. H., Mack, G. A. On the use of a Friedman-type statistic in balanced and unbalanced block designs. Technometrics 23(2), 171-177 (1981). DOI [10.1080/00401706.1981.10486261](https://doi.org/10.1080/00401706.1981.10486261). diff --git a/docs/explanations/dataset-concordance.md b/docs/explanations/dataset-concordance-and-discrimination.md similarity index 57% rename from docs/explanations/dataset-concordance.md rename to docs/explanations/dataset-concordance-and-discrimination.md index 9ed5bcb..3dd66b2 100644 --- a/docs/explanations/dataset-concordance.md +++ b/docs/explanations/dataset-concordance-and-discrimination.md @@ -1,4 +1,4 @@ -# Dataset concordance +# Dataset concordance and discrimination A ranking pooling different metrics aims to answer, across all the datasets at once, which method performs best on average. It cannot say whether the datasets agree on that order. @@ -34,7 +34,24 @@ This is a within-method comparison. On the [Duo 2018 clustering benchmark](../../examples/duo2018/duo2018.qmd) (three metrics, equal weights, pooled across the twelve datasets) the mean tau-b is about 0.34, so the datasets share only a moderate ordering. The datasets group into a Koh pair, a Kumar, Sim and Trapnell cluster, and a Zhengmix cluster, which mirrors the way the source studies built the data. The rank-deviation table shows the disagreement concentrating in a few cells: SAFE places last on the harder simulated datasets while sitting mid-table on average, and ascend collapses on the Koh datasets. So the pooled ranking is a reasonable summary for the bulk of the methods, while a handful of method-dataset combinations carry most of the spread. -The diagnostic sits next to the other ways beam evaluates multi-metric composite rankings. Leave-one-dataset-out asks whether the ranking depends on any single dataset. The [critical-difference](comparing-methods-across-datasets.md) and [Skillings-Mack](skillings-mack.md) tests ask whether the methods are separable on one metric. The [Bradley-Terry tree](heterogeneity-bradley-terry.md) splits the datasets by their declared features. Dataset concordance reports the agreement structure among the datasets from the rankings alone, with no features required and no assumption that the datasets are exchangeable. +The diagnostic sits next to the other ways beam evaluates multi-metric composite rankings. Leave-one-dataset-out asks whether the ranking depends on any single dataset. The [critical-difference](critical-difference.md) and [Skillings-Mack](critical-difference.md#skillings-mack-for-incomplete-blocks) tests ask whether the methods are separable on one metric. The [Bradley-Terry tree](method-by-dataset-heterogeneity.md#bradley-terry-trees) splits the datasets by their declared features. Dataset concordance reports the agreement structure among the datasets from the rankings alone, with no features required and no assumption that the datasets are exchangeable. + +## Dataset discrimination + +Concordance needs shared methods: it compares method orders across datasets. A property defined per dataset needs no shared methods, and every benchmark can report it: how much a dataset separates the methods it scores. [`beam.mcda.dataset_discrimination`](../reference/dataset_discrimination.qmd) computes it. A dataset on which the methods score about the same cannot rank them; one on which they differ can. This is the per-dataset form of the metric-level idea in the [weighting code](weighting-schemes.md), where a metric on which methods do not differ has no discrimination. + +beam computes two values per dataset. + +- Spread, the effect size. Each metric is oriented to higher-is-better and min-max scaled across the benchmark's cells, so metrics are comparable and a dataset on which every method scores near the maximum keeps a small spread. The metrics are pooled to one score per method, and the spread is the standard deviation across methods. +- Concordance, the consistency. Kendall's W over the dataset's method-by-metric matrix, with its Friedman p value. A high W means the metrics order the methods the same way; a low W means they do not, so a single ranking on that dataset is unstable. + +A dataset with high spread and high W separates the methods, and its metrics agree on the order. The scaling is per benchmark, so spreads are comparable within a benchmark and only roughly across benchmarks. A method or metric not observed on a dataset stays NaN and is handled available-case, never imputed: spread uses the observed methods, and Kendall's W uses the complete method-by-metric block, reported only when at least `min_methods` methods and two metrics remain. + +### Hard datasets + +A dataset can be hard because the biology is complex, in which case every method struggles, or because it puts some methods at a disadvantage, for example label quality for semi-supervised methods that would not affect fully unsupervised ones. [`beam.mcda.difficulty_concordance`](../reference/difficulty_concordance.qmd) separates the two. It splits the methods into groups, measures each dataset's difficulty for each group as the group's mean pooled score, and correlates the per-group difficulty profiles across datasets with Spearman. High concordance means the hardness comes from the data; low concordance means it comes from the kind of method. + +On the [OpenProblems batch-integration task](../../examples/openproblems/openproblems.qmd) the deep-learning and classical methods agree (Spearman about 0.89). On the Shen 2026 benchmark, whose scenarios degrade annotation quality, they agree weakly (about 0.32), and the hard cases are harder for the label-using deep-learning methods. ## References diff --git a/docs/explanations/dataset-discrimination.md b/docs/explanations/dataset-discrimination.md deleted file mode 100644 index f511cc9..0000000 --- a/docs/explanations/dataset-discrimination.md +++ /dev/null @@ -1,27 +0,0 @@ -# Dataset discrimination - -Most cross-benchmark comparisons (e.g., analyzing multiple benchmarks aiming to address the same task) ask whether benchmarks agree on the method order, which requires shared methods. When the method sets are disjoint, that comparison is not possible. Yet a property defined per dataset is still available: how much a dataset separates the methods it scores. Every benchmark can report it for every dataset. - -[`beam.mcda.dataset_discrimination`](../reference/dataset_discrimination.qmd) computes it from the scores. A dataset on which the methods score about the same cannot rank them; a dataset on which they differ can. This is the per-dataset form of the metric-level idea in the [weighting code](weighting-schemes.md), where a metric on which methods do not differ has no discrimination. It complements [dataset concordance](dataset-concordance.md), which asks whether datasets agree on the order. - -## Implementation - -beam computes two values per dataset. - -- Spread, the effect size. Each metric is oriented to higher-is-better and min-max scaled across the benchmark's cells, so metrics are comparable and a dataset on which every method scores near the maximum keeps a small spread. The metrics are pooled to one score per method, and the spread is the standard deviation across methods. -- Concordance, the consistency. Kendall's W over the dataset's method-by-metric matrix, with its Friedman p value. A high W means the metrics order the methods the same way; a low W means they do not, so a single ranking on that dataset is unstable. - -A dataset with high spread and high W separates the methods, and its metrics agree on the order. The scaling is per benchmark, so spreads are comparable within a benchmark and only roughly across benchmarks. - -A method or metric not observed on a dataset stays NaN and is handled available-case, never imputed. Spread uses the observed methods. Kendall's W uses the complete method-by-metric block and is reported only when at least `min_methods` methods and two metrics remain. - -## Hard datasets - -A dataset can be hard because the biology is complex, in which case every method struggles, or because it provides a disadvantage to some methods, e.g. label quality for semi-supervised methods that wouldn't affect fully unsupervised methods. [`beam.mcda.difficulty_concordance`](../reference/difficulty_concordance.qmd) aims to separate the two. It splits the methods into groups, measures each dataset's difficulty for each group as the group's mean pooled score, and correlates the per-group difficulty profiles across datasets with Spearman. High concordance means the hardness comes from the data; low concordance means it comes from the kind of method. - -On the [OpenProblems batch-integration task](../../examples/openproblems/openproblems.qmd) the deep-learning and classical methods agree (Spearman about 0.89). On the Shen 2026 benchmark, whose scenarios degrade annotation quality, they agree weakly (about 0.32), and the hard cases are harder for the label-using deep-learning methods. - -## See also - -- [Dataset concordance](dataset-concordance.md) -- [OpenProblems as a data source](openproblems-as-a-data-source.md) diff --git a/docs/explanations/dimensionality.md b/docs/explanations/dimensionality.md deleted file mode 100644 index e4143ed..0000000 --- a/docs/explanations/dimensionality.md +++ /dev/null @@ -1,42 +0,0 @@ -# Latent dimensionality of metrics - -Benchmarks often read a group of metrics as one criterion, but splits are possible. For instance, the [scIB single-cell batch integration benchmark](../../examples/openproblems/openproblems.qmd) score groups its metrics into biological conservation and batch correction and weights the two groups 0.6 and 0.4, so each group is read as a single composite scale. The [reliability](reliability.md) check asks how consistently a group's metrics agree, and reports Cronbach's alpha. Alpha reads a group as one reliable scale when it is high, but that reading rests on an assumption alpha cannot test: that the group is a single factor, one underlying quantity each metric measures with noise. [`beam.mcda.metric_dimensionality`](../reference/metric_dimensionality.qmd) tests the assumption directly by counting the factors in the group. - -## Implementation - -Each method-by-dataset cell is one observation, and the metrics are the variables. The function orients every metric so that higher means better, reading the polarity from the cards, then computes the Spearman rank correlation between every pair of metrics over the observations they share. This is the same oriented correlation that the [validity](convergent-discriminant-validity.md) and [reliability](reliability.md) checks use, so all three diagnostics rest on the same numbers. - -For each group the function takes the eigenvalues of the within-group correlation matrix. This is principal component analysis on the correlation matrix. A correlation matrix of `k` metrics has `k` eigenvalues that sum to `k`. One large eigenvalue with the rest small means a single factor underlies the group. Several eigenvalues of similar size mean several factors. The report carries, per group, the eigenvalues in descending order, the share of variance the first component explains (the first eigenvalue divided by `k`), and two counts of how many factors the group holds. - -The Kaiser (1960) rule keeps every component whose eigenvalue is above one, the variance of a single standardized metric. It is the quick rule and tends to keep more components than the data supports, because in a finite sample the later eigenvalues sit above one by chance alone. - -Parallel analysis (Horn 1965) corrects for that. It draws many random matrices of the same size, with no real association between the columns, and reads off the eigenvalues they produce by chance at each rank. A component is kept when its observed eigenvalue is larger than the random level at that rank. `metric_dimensionality` uses the 95th percentile of the random eigenvalues, following Glorfeld (1995), which holds the false-retention rate down where Horn's original mean rule lets noise through. Parallel analysis is the count the report uses for its verdict: a group is reported as unidimensional when parallel analysis keeps only one component. - -The random draws have a fixed seed for reproducibility. - -## Dimensionality vs reliability - -Dimensionality and reliability answer different questions, and they can disagree. Alpha rises with the mean correlation between the metrics and with the number of metrics. A long group can then reach a high alpha while holding more than one factor, and a short group can sit at a low alpha while holding a single factor its metrics track only weakly. The four combinations all occur: one factor read consistently, one factor read weakly, several factors that happen to agree, several factors that do not. - -The case this check exists to surface is a high alpha on a group that turns out to carry two factors. The group is internally consistent enough to pass as one scale, but it is not one thing, and the 0.6/0.4 weighting that treats it as a single criterion is then a coarser modelling choice than the alpha alone suggests. - -## Limitations - -Counting factors is not the same as naming them. Parallel analysis says how many dimensions the group has, and not what they are. Reading the eigenvectors, or splitting and relabelling the group is responsability of the user. The count is also descriptive of the methods and datasets in the input. A small benchmark gives a coarse estimate, and a few observations relative to the number of metrics make the eigenvalues unstable, so the function declines to score a group when the input has too few observations for its size. - -The correlations are computed pairwise, over the observations each metric pair shares, so the within-group matrix need not be positive semidefinite and a late eigenvalue can come out slightly negative. The function reports the eigenvalues as they are. When a within-group pair has too few shared observations to correlate, the group cannot be decomposed, and the report lists it as undefined rather than guessing. - -## On OpenProblem's batch integration. - -On the [OpenProblems batch integration scores](../../examples/openproblems/openproblems.qmd), with the same bio/batch grouping the validity and reliability checks use, the two groups split apart. The biological group, seven metrics with a high alpha of 0.85, carries two factors by parallel analysis: its first component explains 0.54 of the variance, but a second component stands clear of the chance level, so the high alpha is partly the size of the group rather than a single underlying quantity. The batch group, five metrics with a low alpha of 0.62, carries one factor: its metrics track a single dimension but weakly, which is why the alpha is low while the dimensionality is one. Reliability and dimensionality point in opposite directions on each group. - -## See also - -- [Reliability](reliability.md) -- [Convergent and discriminant validity](convergent-discriminant-validity.md) - -## References - -- Kaiser, H. F.. The application of electronic computers to factor analysis. Educational and Psychological Measurement (1960). DOI [10.1177/001316446002000116](https://doi.org/10.1177/001316446002000116). -- Horn, J. L.. A rationale and test for the number of factors in factor analysis. Psychometrika (1965). DOI [10.1007/BF02289447](https://doi.org/10.1007/BF02289447). -- Glorfeld, L. W.. An improvement on Horn's parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement (1995). DOI [10.1177/0013164495055003002](https://doi.org/10.1177/0013164495055003002). diff --git a/docs/explanations/full-rankings-and-bounded-metrics.md b/docs/explanations/full-rankings-and-bounded-metrics.md deleted file mode 100644 index 3bf0d7f..0000000 --- a/docs/explanations/full-rankings-and-bounded-metrics.md +++ /dev/null @@ -1,24 +0,0 @@ -# Full rankings and bounded metrics - -The [mixed-effects model](heterogeneity-mixed-effects.md) and the [Bradley-Terry tree](heterogeneity-bradley-terry.md) both start from a method by dataset score matrix and process it as pairwise comparisons. Two further models in `beam.heterogeneity` take different views of the same matrix: Plackett-Luce reads each dataset as a full ranking of the methods, and the glmmTMB beta models a metric bounded in (0, 1) without the Gaussian approximation. - -## The Plackett-Luce model - -When each dataset gives a full ranking of the methods, not just pairwise wins, Plackett-Luce turns those orderings into one worth per method, with the worths summing to one. It generalizes the Bradley-Terry strengths from pairwise comparisons to full orderings, and it reduces to Bradley-Terry when the data are pairwise. - -[`beam.heterogeneity.plackett_luce`](../reference/plackett_luce.qmd) builds the per-dataset dense ranking from the score matrix, oriented by the metric polarity so a lower-is-better metric is reversed. Ties are shared and any method missing from a column is dropped from that dataset's ranking. The report carries the worth, the log-worth, and the quasi-standard-errors. The quasi-standard-errors are reference-free, so any two methods compare without picking a baseline. The report also says whether the ranking network is connected, which is the condition for the worths to be jointly identified. - -As a limitation, in rankings that mix ties with partial coverage, the quasi-standard-errors can be unavailable, because the variance refit fails on that shape. When that happens the worths are still reported and the standard-error fields are NA with a warning. - -Plackett-Luce and the Bradley-Terry tree answer different questions. Plackett-Luce gives one global ranking with uncertainty across all datasets. The tree locates where that ranking reverses, splitting the datasets by their features. Use Plackett-Luce when the input is a per-dataset ordering and the question is the overall order; use the tree when the question is which dataset features change the order. - -## The glmmTMB beta engine - -The lme4 mixed-effects model assumes a Gaussian residual. That is an approximation for a metric bounded in (0, 1), and it is worst near the bounds, where the Gaussian tail runs past 0 or 1 but the metric cannot. The glmmTMB beta engine models the bound directly with a beta likelihood. - -Call `mixed_effects(engine="glmmtmb", family="beta")`. It fits the same `score ~ method + (1 | dataset)` structure with a beta likelihood. The marginal means and the variance components come back on the link (logit) scale, and the report's `scale` field says "link". The variance components are not directly comparable to the Gaussian-scale lme4 numbers, but the method ordering is comparable across the two fits. Auto family resolution picks beta only for scores strictly in (0, 1), so an unbounded metric stays Gaussian. - -## Running and R dependency - -Both need the R toolchain. Check `plackett_luce_available()` and `glmmtmb_available()` before calling. The conda environment [envs/heterogeneity.yml](https://github.com/imallona/beam/blob/main/envs/heterogeneity.yml) provides PlackettLuce, qvcalc and glmmTMB. - diff --git a/docs/explanations/funky-heatmaps-and-robustness.md b/docs/explanations/funky-heatmaps-and-robustness.md index 6498263..b6453d0 100644 --- a/docs/explanations/funky-heatmaps-and-robustness.md +++ b/docs/explanations/funky-heatmaps-and-robustness.md @@ -1,33 +1,29 @@ # Funky heatmaps and robustness -The funky heatmap is the glyph table that dynbenchmark/dynverse, OpenProblems and other benchmarking platformas and benchmark use to summarize a multi-metric benchmark at a glance. +The funky heatmap is the glyph table that dynbenchmark/dynverse, OpenProblems and other benchmarking platforms use to summarize a multi-metric benchmark at a glance. Methods are the rows, sorted best first. Metrics are the columns. Each cell is a circle whose radius grows with the score, with its colour marking the metric group, and a final overall column carries the aggregate. It is a dense summary. -Methods are the rows, sorted best first. Metrics are the columns. Each cell is a circle whose radius grows with the score, with its colour marking the metric group, and a final overall column carries the aggregate. It is a dense summary. - -Read alone it looks like a final ranking, though the order can change under a different analysis. Both the circle sizes and the row order depend on the [normalization](normalization-and-scales.md), which is usually min-max, and a different normalization moves the circles and can reorder the rows. beam reads the normalization from the metric cards rather than fixing min-max, and then adds panels that let the table carry its own robustness. +Read on its own the table can pass for a settled ranking, but the order is not fixed. The circle sizes and the row order both depend on the [normalization](normalization-and-scales.md), which is usually min-max, and changing it moves the circles and can reorder the rows. beam reads the normalization from the metric cards instead of fixing min-max, and it adds panels next to the grid that show how far the order would hold under a different analysis. ## The glyph grid and its overlays -beam extends the funkyheatmap with some robustness scores. `beam.reporting.funky_heatmap` draws the glyph grid and the optional robustness panels beside it. `beam.reporting.funky_heatmap_from_run` builds the whole figure from a `beam.rank` `RunResult`. Each overlay answers one question about whether the row order is real, and each is fed by a beam primitive. +`beam.reporting.funky_heatmap` draws the glyph grid and the optional robustness panels beside it; `beam.reporting.funky_heatmap_from_run` builds the whole figure from a `beam.rank` `RunResult`. Each panel is fed by one beam primitive and probes the order in a different way. -The leave-one-dataset-out rank span answers whether the order depends on any single dataset. For each method it draws the span of ranks the method takes when each dataset is dropped in turn, with the pooled rank marked on the span. The values are `rank_low`, `rank_high` and `rank_stability` from [`beam.mcda.leave_one_dataset_out`](../reference/leave_one_dataset_out.qmd). A single point means the rank does not change whatever dataset is dropped. A wide span means the order depends on which datasets are in the pool. +Leave-one-dataset-out rank span. For each method it draws the span of ranks the method takes as each dataset is dropped in turn, with the pooled rank marked. The values are `rank_low`, `rank_high` and `rank_stability` from [`beam.mcda.leave_one_dataset_out`](../reference/leave_one_dataset_out.qmd). A single point means the rank holds whatever dataset is dropped; a wide span means the order depends on which datasets are in the pool. -The aggregation-consensus rank span answers whether the order is an artefact of the chosen aggregation. Holding the [weighting](weighting-schemes.md) fixed, it draws the span of ranks a method takes across the [five aggregations](aggregation-methods.md): SAW, TOPSIS, VIKOR, PROMETHEE II and [COMET](comet.md). The values are `consensus_low` and `consensus_high`, the `rank_low` and `rank_high` of [`beam.mcda.aggregation_agreement`](../reference/aggregation_agreement.qmd), which also reports the pairwise Kendall tau-b agreement behind the span (see [Aggregation agreement](aggregation-agreement.md)). A wide span means the order reflects the aggregation rule rather than the methods. +Aggregation-consensus rank span. Holding the [weighting](weighting-schemes.md) fixed, it draws the span of ranks a method takes across the [five aggregations](aggregation-methods.md): SAW, TOPSIS, VIKOR, PROMETHEE II and [COMET](aggregation-methods.md#comet). The values are `consensus_low` and `consensus_high`, the `rank_low` and `rank_high` of [`beam.mcda.aggregation_agreement`](../reference/aggregation_agreement.qmd), which also reports the Kendall tau-b agreement behind the span (see [Aggregation agreement](choice-agreement.md)). A wide span means the aggregation rule, not the methods, is setting the order. -The stochastic multicriteria acceptability analysis (SMAA) rank-acceptability stacked bar answers what the ranks look like under weight uncertainty. For each method it shows the share of random weightings that place it at rank 1, rank 2, rank 3 and so on. The values are `smaa_acceptability` from [`beam.mcda.smaa`](../reference/smaa.qmd), the rank_acceptability_index. This is the full distribution of ranks across sampled weights, more informative than a single confidence number. +SMAA rank-acceptability bar. Stochastic multicriteria acceptability analysis samples random weightings; for each method the stacked bar shows the share that place it at rank 1, rank 2, and so on. The values are `smaa_acceptability` from [`beam.mcda.smaa`](../reference/smaa.qmd). It is the full rank distribution under weight uncertainty rather than a single confidence number. -The worth panel answers whether two adjacent methods are separable. It draws the latent strength per method from a model as points with horizontal confidence intervals, the `worth` and `worth_ci`. The model is [Plackett-Luce](full-rankings-and-bounded-metrics.md) with reference-free quasi-standard-errors, Bradley-Terry, or the [mixed-effects](heterogeneity-mixed-effects.md) marginal means. When two adjacent intervals overlap the methods are not separable, which the overall bar does not show. This is where the heterogeneity models contribute to the heatmap. +Worth panel. It draws the latent strength per method as points with horizontal confidence intervals, `worth` and `worth_ci`, from [Plackett-Luce](method-by-dataset-heterogeneity.md#plackett-luce-full-rankings) with reference-free quasi-standard-errors, Bradley-Terry, or the [mixed-effects](method-by-dataset-heterogeneity.md) marginal means. Two adjacent intervals that overlap mark methods the aggregate bar cannot tell apart. -The critical-difference cliques answer whether the methods are statistically separable across datasets. They draw brackets grouping the rows that the Friedman-Nemenyi test cannot separate, the `cliques` from [`beam.mcda.critical_difference`](../reference/critical_difference.qmd). A bracket over the top several rows says those methods cannot be told apart from the data at hand. +Critical-difference cliques. Brackets group the rows the Friedman-Nemenyi test cannot separate, the `cliques` from [`beam.mcda.critical_difference`](../reference/critical_difference.qmd). A bracket over the top rows says those methods are not distinguishable from the data at hand. -Each overlay asks one version of the same question: would this row order stay the same under a reasonable change. The change is dropping a dataset, choosing a different aggregation, sampling the weights, or accounting for the model uncertainty. The plain funky heatmap answers none of these. +Between them the panels ask whether the row order would survive dropping a dataset, choosing a different aggregation, resampling the weights, or accounting for model uncertainty. The plain funky heatmap answers none of these. ## How to use it -Call `funky_heatmap_from_run(run)` on a `RunResult`. The leave-one-dataset-out span, the aggregation consensus and the SMAA panel come from the run automatically. The worth with intervals and the cliques are passed in by the caller, because the worth comes from the R-backed models and needs the heterogeneity toolchain. The result is the glyph grid with whichever panels the caller supplied. - -The HTML report from `beam.report` embeds this figure by default in its "Robustness at a glance" section, with the panels the run can supply without the R toolchain. Pass `funky_heatmap=False` to leave it out. +Call `funky_heatmap_from_run(run)` on a `RunResult`. The leave-one-dataset-out span, the aggregation consensus and the SMAA panel come from the run. The worth intervals and the cliques have to be passed in, since the worth comes from the R-backed heterogeneity models rather than from the run itself. -Two worked cases are in the vignettes. The OpenProblems batch integration case shows a fragile top of the order, with wide leave-one-dataset-out spans and overlapping worth intervals among the leading methods. The Duo 2018 case is stable: the top method holds rank one across every leave-one-dataset-out run, the spans are narrow, and the worth intervals separate. +`beam.report` embeds it by default in the "Robustness at a glance" section, with the panels a run can supply without the R toolchain; pass `funky_heatmap=False` to drop it. -The worth panel draws on the [Bradley-Terry tree](heterogeneity-bradley-terry.md) and the mixed-effects fit, which is where the latent strengths and their intervals come from. The cliques come from the [Friedman-Nemenyi test](comparing-methods-across-datasets.md), and the leave-one-dataset-out span shares its logic with that test's question of whether the order depends on one dataset. The [OpenProblems](../../examples/openproblems/openproblems.qmd) and [Duo 2018](../../examples/duo2018/duo2018.qmd) vignettes show the fragile and the stable case side by side. +The [OpenProblems](../../examples/openproblems/openproblems.qmd) and [Duo 2018](../../examples/duo2018/duo2018.qmd) vignettes show the two extremes side by side. OpenProblems batch integration has a fragile top: wide leave-one-dataset-out spans and overlapping worth intervals among the leaders. Duo 2018 is stable: the top method holds rank one across every leave-one-dataset-out run, the spans are narrow, and the worth intervals separate. diff --git a/docs/explanations/heterogeneity-bradley-terry.md b/docs/explanations/heterogeneity-bradley-terry.md deleted file mode 100644 index 3a16df9..0000000 --- a/docs/explanations/heterogeneity-bradley-terry.md +++ /dev/null @@ -1,35 +0,0 @@ -# Heterogeneity: Bradley-Terry trees - -The [mixed-effects model](heterogeneity-mixed-effects.md) asks how much of the score variation is a method-by-dataset interaction. The Bradley-Terry tree in [`beam.heterogeneity.bradley_terry_tree`](../reference/bradley_terry_tree.qmd) asks another question: which dataset properties produce that interaction, and what is the ranking inside each subgroup. It follows Strobl, Wickelmaier and Zeileis, who combine a Bradley-Terry model with model-based recursive partitioning. - -For one metric, each dataset gives a ranking of the methods. The tree works with the pairwise form of that ranking: for every method pair, the dataset records which method scored higher (a win for one, a loss for the other), a tie on an exact equality, or a missing comparison where a method has no score there. The metric polarity orients the comparison, so a lower-is-better metric needs no manual flipping. [`beam.heterogeneity.paired_comparisons`](../reference/paired_comparisons.qmd) builds this design from a method by dataset matrix; it is pure Python and is the part you can inspect without R. - -The datasets are the subjects of the model and the methods are the objects being compared. The order is reversed from the usual reading because the tree splits on properties of the datasets. - -## The Bradley-Terry model - -A Bradley-Terry model turns pairwise wins into a latent strength per method, the worth, with the worths summing to one. The strongest method is the one most likely to win a pairwise comparison. Fit on all datasets at once, it gives a single ranking, the `global_worth` in the report. - -Model-based recursive partitioning fits the Bradley-Terry model at the root, then tests whether the worth parameters are stable across the candidate dataset features. When the stability test flags a feature, the datasets split on it and the model is refit in each child. The recursion continues until no further split is warranted or a node would fall below the minimum size. The result is a tree whose leaves each carry their own Bradley-Terry ranking, with inner nodes that read as "datasets with feature X above threshold Z go this way". - -In the context of Strobl and colleagues, instead of one ranking averaged over datasets that may disagree, the tree gives "prefer method A on datasets like this, method B on datasets like that", with a statistical test deciding where the split is real. `reversed_leaves` reports the leaves whose strongest method differs from the global one, the subgroups where the pooled recommendation does not hold. - -## The small-sample limit - -Recursive partitioning needs enough datasets to support a split. With a dozen datasets the parameter-stability test rarely separates a feature-dependent regime from sampling noise, and the tree reduces to the single flat ranking. The report says so through `did_split` and the summary, rather than inventing a split. This is the same limit the critical-difference diagram and the mixed-effects fit hit on a small benchmark. The tree is useful where there are many datasets carrying real feature variation. - -## How to run and R dependency - -Call `bradley_terry_tree(matrix, method_names, dataset_names, numeric_features=, categorical_features=, polarity=)`. The features are per-dataset descriptors, numeric ones as continuous splitters and categorical ones as factors; pass at least one. `minsize` sets the smallest leaf and `alpha` the split test level. The report carries the tree nodes (split variables, breakpoints, parameter-stability p-values), the per-leaf worths with standard errors, the leaf assignment per dataset, the global flat ranking, and a plain-language summary. `node_ranking`, `datasets_in_node` and `reversed_leaves` read the leaves. - -The fit runs in R's psychotree through a subprocess, so it needs the R toolchain. Check `bttree_available()` first; the conda environment [envs/heterogeneity.yml](https://github.com/imallona/beam/blob/main/envs/heterogeneity.yml) provides psychotree and partykit. - -## Related checks - -The [mixed-effects model](heterogeneity-mixed-effects.md) measures how much interaction there is; the Bradley-Terry tree locates it in dataset features and reports the subgroup rankings. Leave-one-dataset-out asks whether the composite ranking depends on any one dataset; the [Friedman-Nemenyi diagram](comparing-methods-across-datasets.md) asks whether the methods are statistically separable on a metric. Read together they move from "is there heterogeneity" to "where is it and what do I do about it". - -The [Duo 2018 vignette](../../examples/duo2018/duo2018.qmd) works this through on the Adjusted Rand Index (ARI) scores, where the tree degrades to a flat ranking on 12 datasets. - -## References - -- Strobl, C., Wickelmaier, F., Zeileis, A.. Accounting for individual differences in Bradley-Terry models by means of recursive partitioning. Journal of Educational and Behavioral Statistics (2011). DOI [10.3102/1076998609359791](https://doi.org/10.3102/1076998609359791). diff --git a/docs/explanations/heterogeneity-mixed-effects.md b/docs/explanations/heterogeneity-mixed-effects.md deleted file mode 100644 index 5e66bb0..0000000 --- a/docs/explanations/heterogeneity-mixed-effects.md +++ /dev/null @@ -1,39 +0,0 @@ -# Heterogeneity: the mixed-effects variance decomposition - -A pooled [multi-criteria decision analysis (MCDA)](aggregation-methods.md) ranking treats the datasets as interchangeable and reports one order of the methods. The heterogeneity question is whether that order is stable across datasets or an average over datasets that disagree. The mixed-effects model in [`beam.heterogeneity.mixed_effects`](../reference/mixed_effects.qmd) answers it for one metric, following Eugster, Hothorn and Leisch (2008). - -## Definition - -For one metric, every benchmark score is one observation labelled by its method and its dataset. The model is - - score ~ method + (1 | dataset) - -The method is a fixed effect: each method gets a marginal mean over datasets, with a standard error. The dataset is a random intercept: it absorbs the fact that some datasets are easy and some are hard, a shift that moves every method together. What is left after the method effect and the dataset shift is the residual. - -The variance splits into two parts: the dataset variance and the residual variance. The intraclass correlation, the dataset variance over the total, is the share of the spread that is a pure dataset shift. A high value means the datasets differ mostly in difficulty, not in which method they favour. A low value means most of the variation is within datasets, where the method-by-dataset interaction occurs. - -A single global ranking is safe when the interaction is small: the method that leads on average leads nearly everywhere. When the interaction is large, the average hides reversals, and an appropriate split is "use method A on data like this, method B on data like that". This is the [Strobl critique](heterogeneity-bradley-terry.md), that one method does not fit all datasets, written as a variance component. - -With one run per (method, dataset) cell, the usual single-run benchmark, the interaction cannot be separated from measurement noise. There is one number per cell and two things to explain it with, so they are confounded; the residual is their sum. beam reports the residual share as the upper bound on the interaction and says so. To split them you need replicates, several runs of the same method on the same dataset. When the input has them, the auto path fits - - score ~ method + (1 | dataset) + (1 | dataset:method) - -and reports the interaction as its own component, so `interaction_share` is defined instead of `None`. - -The per-cell residuals are the local signal. A large residual is a cell where a method does much better or worse than its global effect predicts on that dataset. `top_outliers` returns the largest ones. They point to where the interaction sits, which methods on which datasets, even when its size cannot be separated from noise. - -## How to use it - -Call `mixed_effects(methods, datasets, scores)` with three parallel sequences for one metric, or `mixed_effects_from_matrix(matrix, method_names, dataset_names)` with a method by dataset matrix. NaN scores are dropped. The report carries the method marginal means and their standard errors, the variance components, the dataset ICC, the interaction or residual share, the residuals, and the outlier cells. - -The fit runs in R's lme4 through a subprocess, so it needs the R toolchain. Check `r_available()` first; the conda environment [envs/heterogeneity.yml](https://github.com/imallona/beam/blob/main/envs/heterogeneity.yml) provides it. lme4 uses a Gaussian likelihood, which is an approximation for a metric bounded in [0, 1] such as the Adjusted Rand Index (ARI). - -## Relation to leave-one-dataset-out - -Both evaluate a pooled ranking across the datasets, and they answer different questions. Leave-one-dataset-out ([`beam.mcda.leave_one_dataset_out`](../reference/leave_one_dataset_out.qmd)) re-ranks with each dataset removed and asks whether the recommendation depends on any single dataset. The mixed-effects model asks how much of the score variance is interaction at all. The first is a stability check on the composite; the second is a split of the raw metric. Read together, a stable leave-one-dataset-out result with a low interaction share is a ranking you can trust across the datasets at hand. The separate question of whether the methods are statistically separable on a metric is the [Friedman-Nemenyi check](comparing-methods-across-datasets.md). - -The [Duo 2018 vignette](../../examples/duo2018/duo2018.qmd) works this through on the ARI scores. - -## References - -- Eugster, M. J. A., Hothorn, T., Leisch, F.. (Psycho-)analysis of benchmark experiments. Technical Report 30, Department of Statistics, LMU Munich (2008). diff --git a/docs/explanations/method-by-dataset-heterogeneity.md b/docs/explanations/method-by-dataset-heterogeneity.md new file mode 100644 index 0000000..5eb0398 --- /dev/null +++ b/docs/explanations/method-by-dataset-heterogeneity.md @@ -0,0 +1,71 @@ +# Method-by-dataset heterogeneity + +A pooled [MCDA](aggregation-methods.md) ranking treats the datasets as interchangeable and reports one order of the methods. The heterogeneity question is whether that order is stable across datasets or an average over datasets that disagree. beam has four models in `beam.heterogeneity`, each reading one metric's method by dataset score matrix, all fitting in R: + +- the mixed-effects variance decomposition: how much of the score variation is a method-by-dataset interaction; +- the Bradley-Terry tree: which dataset features produce that interaction, and the ranking inside each subgroup; +- Plackett-Luce: one global ranking with uncertainty when each dataset gives a full ordering; +- the glmmTMB beta engine: the mixed-effects decomposition for a metric bounded in (0, 1). + +## Mixed-effects variance decomposition + +[`beam.heterogeneity.mixed_effects`](../reference/mixed_effects.qmd) follows Eugster, Hothorn and Leisch (2008). For one metric, every benchmark score is one observation labelled by its method and its dataset. The model is + + score ~ method + (1 | dataset) + +The method is a fixed effect: each method gets a marginal mean over datasets, with a standard error. The dataset is a random intercept: it absorbs the fact that some datasets are easy and some are hard, a shift that moves every method together. What is left after the method effect and the dataset shift is the residual. + +The variance splits into two parts: the dataset variance and the residual variance. The intraclass correlation, the dataset variance over the total, is the share of the spread that is a pure dataset shift. A high value means the datasets differ mostly in difficulty, not in which method they favour. A low value means most of the variation is within datasets, where the method-by-dataset interaction occurs. + +A single global ranking is safe when the interaction is small: the method that leads on average leads nearly everywhere. When the interaction is large, the average hides reversals, and an appropriate split is "use method A on data like this, method B on data like that". This is the Strobl critique, that one method does not fit all datasets, written as a variance component. + +With one run per (method, dataset) cell, the usual single-run benchmark, the interaction cannot be separated from measurement noise. There is one number per cell and two things to explain it with, so they are confounded; the residual is their sum. beam reports the residual share as the upper bound on the interaction and says so. To split them you need replicates, several runs of the same method on the same dataset. When the input has them, the auto path fits + + score ~ method + (1 | dataset) + (1 | dataset:method) + +and reports the interaction as its own component, so `interaction_share` is defined instead of `None`. + +The per-cell residuals are the local signal. A large residual is a cell where a method does much better or worse than its global effect predicts on that dataset. `top_outliers` returns the largest ones. They point to where the interaction sits, which methods on which datasets, even when its size cannot be separated from noise. + +Call `mixed_effects(methods, datasets, scores)` with three parallel sequences for one metric, or `mixed_effects_from_matrix(matrix, method_names, dataset_names)` with a method by dataset matrix. NaN scores are dropped. The report carries the method marginal means and their standard errors, the variance components, the dataset ICC, the interaction or residual share, the residuals, and the outlier cells. + +## Bradley-Terry trees + +The mixed-effects model measures how much interaction there is. The Bradley-Terry tree in [`beam.heterogeneity.bradley_terry_tree`](../reference/bradley_terry_tree.qmd) asks which dataset properties produce it, and what the ranking is inside each subgroup. It follows Strobl, Wickelmaier and Zeileis, who combine a Bradley-Terry model with model-based recursive partitioning. + +For one metric, each dataset gives a ranking of the methods. The tree works with the pairwise form: for every method pair, the dataset records which method scored higher (a win for one, a loss for the other), a tie on an exact equality, or a missing comparison where a method has no score there. The metric polarity orients the comparison, so a lower-is-better metric needs no manual flipping. [`beam.heterogeneity.paired_comparisons`](../reference/paired_comparisons.qmd) builds this design from a method by dataset matrix; it is pure Python and is the part you can inspect without R. The datasets are the subjects of the model and the methods are the objects being compared, reversed from the usual reading because the tree splits on properties of the datasets. + +A Bradley-Terry model turns pairwise wins into a latent strength per method, the worth, with the worths summing to one. Fit on all datasets at once, it gives a single ranking, the `global_worth` in the report. Model-based recursive partitioning fits the model at the root, then tests whether the worth parameters are stable across the candidate dataset features. When the stability test flags a feature, the datasets split on it and the model is refit in each child. The recursion continues until no further split is warranted or a node would fall below the minimum size. The result is a tree whose leaves each carry their own Bradley-Terry ranking, with inner nodes that read as "datasets with feature X above threshold Z go this way". Instead of one ranking averaged over datasets that may disagree, the tree gives "prefer method A on datasets like this, method B on datasets like that", with a statistical test deciding where the split is real. `reversed_leaves` reports the leaves whose strongest method differs from the global one. + +Recursive partitioning needs enough datasets to support a split. With a dozen datasets the parameter-stability test rarely separates a feature-dependent regime from sampling noise, and the tree reduces to the single flat ranking. The report says so through `did_split` and the summary, rather than inventing a split. This is the same limit the critical-difference diagram and the mixed-effects fit hit on a small benchmark. + +Call `bradley_terry_tree(matrix, method_names, dataset_names, numeric_features=, categorical_features=, polarity=)`. The features are per-dataset descriptors, numeric ones as continuous splitters and categorical ones as factors; pass at least one. `minsize` sets the smallest leaf and `alpha` the split test level. The report carries the tree nodes (split variables, breakpoints, parameter-stability p-values), the per-leaf worths with standard errors, the leaf assignment per dataset, the global flat ranking, and a plain-language summary. `node_ranking`, `datasets_in_node` and `reversed_leaves` read the leaves. + +## Plackett-Luce: full rankings + +When each dataset gives a full ranking of the methods, not just pairwise wins, Plackett-Luce turns those orderings into one worth per method, with the worths summing to one. It generalizes the Bradley-Terry strengths from pairwise comparisons to full orderings, and reduces to Bradley-Terry when the data are pairwise. + +[`beam.heterogeneity.plackett_luce`](../reference/plackett_luce.qmd) builds the per-dataset dense ranking from the score matrix, oriented by the metric polarity so a lower-is-better metric is reversed. Ties are shared and any method missing from a column is dropped from that dataset's ranking. The report carries the worth, the log-worth, and the quasi-standard-errors. The quasi-standard-errors are reference-free, so any two methods compare without picking a baseline. The report also says whether the ranking network is connected, the condition for the worths to be jointly identified. In rankings that mix ties with partial coverage the quasi-standard-errors can be unavailable, because the variance refit fails on that shape; the worths are still reported and the standard-error fields are NA with a warning. + +Plackett-Luce and the Bradley-Terry tree answer different questions. Plackett-Luce gives one global ranking with uncertainty across all datasets; the tree locates where that ranking reverses. Use Plackett-Luce when the input is a per-dataset ordering and the question is the overall order, and the tree when the question is which dataset features change the order. + +## glmmTMB beta: bounded metrics + +The lme4 mixed-effects model assumes a Gaussian residual. That is an approximation for a metric bounded in (0, 1), and it is worst near the bounds, where the Gaussian tail runs past 0 or 1 but the metric cannot. The glmmTMB beta engine models the bound directly with a beta likelihood. + +Call `mixed_effects(engine="glmmtmb", family="beta")`. It fits the same `score ~ method + (1 | dataset)` structure with a beta likelihood. The marginal means and the variance components come back on the link (logit) scale, and the report's `scale` field says "link". The variance components are not directly comparable to the Gaussian-scale lme4 numbers, but the method ordering is comparable across the two fits. Auto family resolution picks beta only for scores strictly in (0, 1), so an unbounded metric stays Gaussian. + +## R dependency + +All four models fit in R through a subprocess, so they need the R toolchain. lme4 backs the Gaussian mixed-effects fit, psychotree and partykit the Bradley-Terry tree, PlackettLuce and qvcalc the Plackett-Luce worths, and glmmTMB the beta engine. Check the matching availability function before calling: `r_available()`, `bttree_available()`, `plackett_luce_available()`, `glmmtmb_available()`. The conda environment [envs/heterogeneity.yml](https://github.com/imallona/beam/blob/main/envs/heterogeneity.yml) provides all of them. + +## Relation to the other checks + +These models split the raw metric across datasets. Two composite-level checks sit next to them. Leave-one-dataset-out ([`beam.mcda.leave_one_dataset_out`](../reference/leave_one_dataset_out.qmd)) re-ranks with each dataset removed and asks whether the recommendation depends on any single dataset; a stable result with a low interaction share is a ranking you can trust across the datasets at hand. Whether the methods are statistically separable on a metric is the [Friedman-Nemenyi check](critical-difference.md). + +The [Duo 2018 vignette](../../examples/duo2018/duo2018.qmd) works these through on the Adjusted Rand Index (ARI) scores, where the tree degrades to a flat ranking on 12 datasets. + +## References + +- Eugster, M. J. A., Hothorn, T., Leisch, F. (Psycho-)analysis of benchmark experiments. Technical Report 30, Department of Statistics, LMU Munich (2008). +- Strobl, C., Wickelmaier, F., Zeileis, A. Accounting for individual differences in Bradley-Terry models by means of recursive partitioning. Journal of Educational and Behavioral Statistics (2011). DOI [10.3102/1076998609359791](https://doi.org/10.3102/1076998609359791). diff --git a/docs/explanations/metric-set-diagnostics.md b/docs/explanations/metric-set-diagnostics.md new file mode 100644 index 0000000..0ee413c --- /dev/null +++ b/docs/explanations/metric-set-diagnostics.md @@ -0,0 +1,64 @@ +# Metric-set diagnostics: validity, reliability, and dimensionality + +A benchmark often treats a group of metrics as one criterion. The single-cell [scIB integration benchmark](../../examples/openproblems/openproblems.qmd) splits its metrics into biological conservation and batch correction, and weights the two groups 0.6 and 0.4, so each is read as a single composite scale. That grouping is an assumption, and beam has three checks on it, all built from the same oriented correlation between the metrics: + +- validity: are same-group metrics more alike than different-group metrics, so the split is the right one? +- reliability: does a group hold together consistently enough to read as one scale? +- dimensionality: is a group really one underlying factor, or several? + +## The shared correlation + +Each method-by-dataset cell is one observation, and the metrics are the variables. The function orients every metric so that higher means better, reading the polarity from the cards and negating a lower-is-better metric. It then takes the Spearman rank correlation between every pair of metrics over the observations they share. A rank correlation finds association regardless of the metric scale. Pairs with no shared observations are left as NaN. All three checks below start from this correlation. + +The grouping itself is a label per metric, passed in as an argument. It is a domain judgement, so beam does not read it from the cards. In the scIB case the labels are biological conservation and batch correction. + +## Validity + +[`beam.mcda.metric_validity`](../reference/metric_validity.qmd) follows Campbell and Fiske (1959). Correlations within a group are the convergent evidence: metrics that claim to measure one construct should agree. Correlations between groups are the discriminant evidence: metrics that claim to measure different constructs should agree less. `discriminant_ok` is true when the mean within-group correlation is higher than the mean between-group correlation, which is the case where treating the groups as separate criteria in the [weighting](weighting-schemes.md) has support in the data. + +beam reports two kinds of metric discrimination problems or properties: + +- A redundant pair is two metrics in the same group whose correlation is at or above a threshold (0.9 by default). They order the methods almost identically, so carrying both adds little and double-counts one construct. One is a candidate to drop. +- A crossloading metric correlates more, on average, with another group than with its own. It behaves more like a different construct than the one its label claims. This is the per-metric form of a discriminant-validity failure, and it points at a metric that is mislabelled or genuinely ambiguous. + +On the [OpenProblems batch integration scores](../../examples/openproblems/openproblems.qmd), the bio/batch grouping is supported but weak: mean within-group correlation 0.38 against mean between-group correlation 0.30. The biological metrics agree more among themselves (0.45) than the batch metrics do (0.24), and `graph_connectivity`, a batch metric, correlates more with the biological group than with its own. + +## Reliability + +[`beam.mcda.metric_reliability`](../reference/metric_reliability.qmd) reports standardized Cronbach's alpha per group, following Cronbach (1951): + + alpha = k * r_bar / (1 + (k - 1) * r_bar) + +where `k` is the number of metrics in the group and `r_bar` is their mean inter-item correlation. The standardized form, built from correlations rather than from raw covariances, is appropriate here because the metrics are on different scales. It is the rank-based analogue of classical alpha, in line with the way the rest of beam compares metrics by rank. + +Alpha runs up to 1. A common rule of thumb treats 0.7 as the point above which a group reads as one reliable scale, and `metric_reliability` flags any group below that cutoff. The cutoff is somewhat arbitrary, so the report carries `r_bar` and `k` next to each alpha, so analysts make a decision. + +Alpha rises with the mean correlation and with the number of metrics, so a long group can reach a high alpha on modest agreement, and a short group needs stronger agreement to reach the same cutoff. When two groups differ in size, the mean inter-item correlation is the comparison to use, because it does not carry the size effect. + +For a group of three or more metrics, the report recomputes the group's alpha with each metric removed in turn. A metric whose removal raises the group's alpha agrees with its labelled construct less than the others do. It is the metric to reconsider first, whether to relabel it, drop it, or treat it as a separate criterion. A metric in a group of two has no alpha-if-dropped entry, because dropping one leaves a single metric and alpha is undefined for one item. + +Alpha is not a validity check. A group can be reliable and still measure the wrong thing. + +On the OpenProblems scores, where there is an expertly-curated classification of metrics in bio/batch, the biological conservation group is reliable: alpha 0.85 over seven metrics, mean inter-item correlation 0.45. The batch correction group does not reach the cutoff: alpha 0.62 over five metrics, mean inter-item correlation 0.24. Dropping `pcr` is the only batch removal that raises the batch alpha by more than a rounding step, from 0.62 to 0.67, so `pcr` is the batch metric least consistent with the rest of its group. + +## Dimensionality + +Alpha reads a group as one scale when it is high, but that rests on an assumption alpha cannot test: that the group is a single factor, one underlying quantity each metric measures with noise. [`beam.mcda.metric_dimensionality`](../reference/metric_dimensionality.qmd) tests it directly by counting the factors. + +For each group the function takes the eigenvalues of the within-group correlation matrix. This is principal component analysis on the correlation matrix. A correlation matrix of `k` metrics has `k` eigenvalues that sum to `k`. One large eigenvalue with the rest small means a single factor underlies the group. Several eigenvalues of similar size mean several factors. The report carries, per group, the eigenvalues in descending order, the share of variance the first component explains (the first eigenvalue divided by `k`), and two counts of how many factors the group holds. + +The Kaiser (1960) rule keeps every component whose eigenvalue is above one, the variance of a single standardized metric. It is the quick rule and tends to keep more components than the data supports, because in a finite sample the later eigenvalues sit above one by chance alone. Parallel analysis (Horn 1965) corrects for that. It draws many random matrices of the same size, with no real association between the columns, and reads off the eigenvalues they produce by chance at each rank. A component is kept when its observed eigenvalue is larger than the random level at that rank. `metric_dimensionality` uses the 95th percentile of the random eigenvalues, following Glorfeld (1995), which holds the false-retention rate down where Horn's original mean rule lets noise through. Parallel analysis is the count the report uses for its verdict: a group is reported as unidimensional when parallel analysis keeps only one component. The random draws have a fixed seed for reproducibility. + +Dimensionality and reliability answer different questions and can disagree. Alpha rises both with the mean correlation between the metrics and with the number of metrics. A long group can then reach a high alpha while holding more than one factor, and a short group can sit at a low alpha while holding a single factor that its metrics track only weakly. The case this check exists to surface is a high alpha on a group that turns out to carry two factors. The group is internally consistent enough to pass as one scale, but it is not one thing, and the 0.6/0.4 weighting that treats it as a single criterion is then a coarser modelling choice than the alpha alone suggests. + +Counting factors is not the same as naming them. Parallel analysis says how many dimensions the group has, not what they are. Reading the eigenvectors, or splitting and relabelling the group, is left to the user. The count is descriptive of the methods and datasets in the input: a small benchmark gives a coarse estimate, and a few observations relative to the number of metrics make the eigenvalues unstable, so the function declines to score a group when the input has too few observations for its size. The correlations are computed pairwise, so the within-group matrix need not be positive semidefinite and a late eigenvalue can come out slightly negative; the function reports the eigenvalues as they are. When a within-group pair has too few shared observations to correlate, the group cannot be decomposed, and the report lists it as undefined rather than guessing. + +On the OpenProblems scores, with the same bio/batch grouping, the two groups come apart. The biological group has seven metrics and a high alpha of 0.85, but parallel analysis finds two factors: the first component explains 0.54 of the variance and a second component still stands clear of the chance level, so part of that alpha is the size of the group rather than one underlying quantity. The batch group has five metrics and a low alpha of 0.62, yet it reads as a single factor whose metrics track one dimension weakly. + +## References + +- Campbell, D. T., Fiske, D. W. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin (1959). DOI [10.1037/h0046016](https://doi.org/10.1037/h0046016). +- Cronbach, L. J. Coefficient alpha and the internal structure of tests. Psychometrika (1951). DOI [10.1007/BF02310555](https://doi.org/10.1007/BF02310555). +- Kaiser, H. F. The application of electronic computers to factor analysis. Educational and Psychological Measurement (1960). DOI [10.1177/001316446002000116](https://doi.org/10.1177/001316446002000116). +- Horn, J. L. A rationale and test for the number of factors in factor analysis. Psychometrika (1965). DOI [10.1007/BF02289447](https://doi.org/10.1007/BF02289447). +- Glorfeld, L. W. An improvement on Horn's parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement (1995). DOI [10.1177/0013164495055003002](https://doi.org/10.1177/0013164495055003002). diff --git a/docs/explanations/missing-data.md b/docs/explanations/missing-data.md index 4dd690b..7b0dce8 100644 --- a/docs/explanations/missing-data.md +++ b/docs/explanations/missing-data.md @@ -2,7 +2,7 @@ Benchmark tables have gaps. A method crashes on one dataset, times out on another, does not accept a given input shape, or was never run for one metric. These gaps are normal. beam has to decide what a gap means before it ranks. By default, beam does not impute (but could impute on user request). -This page explains how beam treats missing cells and why it refuses to choose for you. +This page explains how beam treats missing cells and why it leaves the imputation choice to you. The problem with imputation is what to use as a filler for NAs: the column mean, a zero, the midpoint of the scale... Each one of these imputes a value the method never generated, and the imputed value changes the ranking. Filling a missing accuracy with the mean shifts a method's score toward the average; filling it with zero treats a crash as the worst possible result; filling it with the midpoint improves a method's apparent score despite not running. None of these is wrong per se. Which one is right depends on the benchmark, so beam will not pick one without asking the user. A missing value often indicates inability to run, so treating it as the worst outcome is reasonable. @@ -20,7 +20,7 @@ Once the dataset axis has been summarized, what remains is a tool by metric matr `error` refuses any missing cell and names the alternatives. This is the default because a recommendation must not rest on a choice you did not make. -`available` is available-case ranking with simple additive weighting. Each tool is scored on the metrics it was measured on, with the weights renormalized over that tool's observed metrics. A method measured on accuracy and runtime but not memory is scored on accuracy and runtime. The composites then rest on different metric sets across tools, so the result carries a warning. Only SAW supports this, because its composite is a sum you can take over a subset of metrics. [TOPSIS, VIKOR, PROMETHEE II](aggregation-methods.md) and [COMET](comet.md) cannot: they each need every tool placed on every criterion to define an ideal point, a pairwise comparison, or a fuzzy membership. The objective [weight schemes](weighting-schemes.md) (entropy, standard deviation, CRITIC, MEREC) measure the spread of a metric across the tools, which needs a complete column. So under `available` these methods and weightings refuse and point you to a policy that completes the matrix or to a per-subset analysis. +`available` is available-case ranking with simple additive weighting. Each tool is scored on the metrics it was measured on, with the weights renormalized over that tool's observed metrics. A method measured on accuracy and runtime but not memory is scored on accuracy and runtime. The composites then rest on different metric sets across tools, so the result carries a warning. Only SAW supports this, because its composite is a sum you can take over a subset of metrics. [TOPSIS, VIKOR, PROMETHEE II](aggregation-methods.md) and [COMET](aggregation-methods.md#comet) cannot: they each need every tool placed on every criterion to define an ideal point, a pairwise comparison, or a fuzzy membership. The objective [weight schemes](weighting-schemes.md) (entropy, standard deviation, CRITIC, MEREC) measure the spread of a metric across the tools, which needs a complete column. So under `available` these methods and weightings refuse and point you to a policy that completes the matrix or to a per-subset analysis. `worst` is the explicit decision that a missing cell means the method did not run and should count as the worst outcome. After normalization each gap is set to 0, the worst score on the higher-is-better scale, the matrix is complete, and every method runs. Many benchmarkers prefer this: a tool that fails to run ranks lower than one that runs and scores poorly. It is a statement about the benchmark, not a guess at an unknown value, and beam records it in a warning. @@ -28,11 +28,11 @@ Once the dataset axis has been summarized, what remains is a tool by metric matr ## The critical-difference test -The [Friedman test and its Nemenyi post-hoc](comparing-methods-across-datasets.md) rank the methods within each dataset, which is only defined over a complete column. beam refuses a tool by dataset table with missing cells rather than dropping or filling them, and suggests restricting the diagram to the block of methods and datasets where all of them ran. The missing-data generalization of the Friedman test is the [Skillings-Mack (1981) test](skillings-mack.md), available as [`beam.mcda.skillings_mack`](../reference/skillings_mack.qmd). +The [Friedman test and its Nemenyi post-hoc](critical-difference.md) rank the methods within each dataset, which is only defined over a complete column. beam refuses a tool by dataset table with missing cells rather than dropping or filling them, and suggests restricting the diagram to the block of methods and datasets where all of them ran. The missing-data generalization of the Friedman test is the [Skillings-Mack (1981) test](critical-difference.md#skillings-mack-for-incomplete-blocks), available as [`beam.mcda.skillings_mack`](../reference/skillings_mack.qmd). -## What never happens +## No silent filling -beam does not fill by default and much less without a warning. A missing value is *only imputed under the explicit `worst` and `impute` policies*, both of which the user chooses by name and both of which warn. Everywhere else, a missing cell is summarized around (the dataset axis), carried through untouched (normalization), or refused with a warning. +beam never fills a gap by default, and never without a warning. A missing value is imputed only under the explicit `worst` and `impute` policies, both of which the user chooses by name and both of which warn. Everywhere else, a missing cell is summarized around (the dataset axis), carried through untouched (normalization), or refused with a warning. ## See also diff --git a/docs/explanations/network-meta-analysis.md b/docs/explanations/network-meta-analysis.md index f8d54df..13e39e0 100644 --- a/docs/explanations/network-meta-analysis.md +++ b/docs/explanations/network-meta-analysis.md @@ -10,13 +10,13 @@ The treatments are the methods. The studies are the (benchmark, dataset) blocks. The P-score is the ranking. It is the share of competing methods a method outperforms, averaged over the ranking uncertainty, in 0 to 1. Because a benchmark rank of 1 is best, beam treats small values as desirable, so a higher P-score indicates a higher-ranked method. -Benchmarks publish one score per method per dataset per metric, with no replicate runs, so there is no sampling standard error in the usual sense. This wrapper takes the variability across the metrics within a (benchmark, dataset) block as the within-arm spread. That treats the metrics as repeated readings of the same quantity, which they are not exactly: the Adjusted Rand Index (ARI), average silhouette width (ASW), k-nearest-neighbour batch-effect test (kBET) and local inverse Simpson index (LISI) measure related but distinct aspects of integration quality. The pooled ranking is therefore a descriptive summary of the evidence as published, not an inference back to a population of runs. An arm resting on fewer than two metrics has no estimable standard deviation and is dropped, and netmeta then keeps only the studies that still have two or more arms. +Benchmarks publish one score per method per dataset per metric, with no replicate runs, so there is no sampling standard error in the usual sense. This wrapper takes the variability across the metrics within a (benchmark, dataset) block as the within-arm spread. That treats the metrics as repeated readings of the same quantity, which they are not exactly: the Adjusted Rand Index (ARI), average silhouette width (ASW), k-nearest-neighbour batch-effect test (kBET) and local inverse Simpson index (LISI) measure related but distinct aspects of integration quality. The pooled ranking is a descriptive summary of the evidence as published, not an inference back to a population of runs. An arm resting on fewer than two metrics has no estimable standard deviation and is dropped, and netmeta then keeps only the studies that still have two or more arms. Two scores evaluate the pooled ranking. The heterogeneity Q (the within-design part) measures how much the studies of the same design disagree beyond chance. The inconsistency Q (the between-design part) measures whether the direct and the indirect evidence for the same comparison agree. A large inconsistency Q is the formal version of "the benchmarks contradict each other". The ranking that pools them is then an average over evidence that does not line up. beam reports both parts where the design structure supports the split, along with tau-squared and I-squared. ## Related checks -The network meta-analysis and the source-variance decomposition (see [the mixed-effects explanation](heterogeneity-mixed-effects.md)) answer complementary questions. The mixed-effects [`source_variance_decomposition`](../reference/source_variance_decomposition.qmd) puts a number on how much of the spread is the benchmark rather than the method (the method-by-benchmark variance share). The network meta-analysis pools the same evidence into a single ranking with a hierarchy of methods, and its inconsistency statistic is the disagreement read as a model lack-of-fit rather than as a variance component. A high method-by-benchmark variance share and a significant inconsistency Q are the same finding from two directions. The benchmarks disagree, and the pooled ranking is an average over that disagreement. [Attribution synthesis](attribution-synthesis.md) combines this with the analyst-choice and dataset axes. +The network meta-analysis and the source-variance decomposition (see [the mixed-effects explanation](method-by-dataset-heterogeneity.md)) answer complementary questions. The mixed-effects [`source_variance_decomposition`](../reference/source_variance_decomposition.qmd) puts a number on how much of the spread is the benchmark rather than the method (the method-by-benchmark variance share). The network meta-analysis pools the same evidence into a single ranking with a hierarchy of methods, and its inconsistency statistic is the disagreement read as a model lack-of-fit rather than as a variance component. A high method-by-benchmark variance share and a significant inconsistency Q both say the benchmarks disagree, so the pooled ranking averages over that disagreement. [Attribution synthesis](attribution-synthesis.md) combines this with the analyst-choice and dataset axes. ## How to run and R dependency diff --git a/docs/explanations/normalization-agreement.md b/docs/explanations/normalization-agreement.md deleted file mode 100644 index 04874da..0000000 --- a/docs/explanations/normalization-agreement.md +++ /dev/null @@ -1,46 +0,0 @@ -# Normalization agreement: does the recommendation depend on the rescaling? - -Before any weighting or aggregation, beam rescales each metric column to the unit interval so the metrics can be combined. The strategy is read per metric from the card field `comparability.recommended_normalization`, but it is a real analyst choice. A rank normalization keeps the order of the methods and drops the size of the gaps between them. `log_min_max` keeps the multiplicative structure of a ratio metric, so one slow method no longer compresses the differences among the fast ones. `zscore` standardizes the column and compresses an outlier smoothly. These can produce different orders from the same raw scores. [`beam.mcda.normalization_agreement`](../reference/normalization_agreement.qmd) asks how much that choice moves the ranking, the same question [`aggregation_agreement`](aggregation-agreement.md) asks about the aggregation rule. - -This is a choice-sensitivity diagnostic, the same family as the others beam ships. Leave-one-metric-out and leave-one-dataset-out vary the inputs, stochastic multicriteria acceptability analysis (SMAA) varies the weights, `aggregation_agreement` varies the aggregation rule, and this varies the normalization. The normalization is the one analyst choice the others leave untested. - -## Rationale - -The diagnostic re-ranks the same matrix under several normalizations, holding the weighting and the aggregation fixed. It runs the full pipeline once per candidate, collects the per-tool ranks, and compares every pair of rankings. - -One point separates this from `aggregation_agreement`. Objective [weights](weighting-schemes.md) (entropy, standard deviation, CRITIC, MEREC) are computed from the normalized matrix, so changing the normalization also changes those weights. That is intended. The normalization choice propagates through the whole pipeline, and the report shows its total effect on the order, not just the part that acts through the aggregation step. - -Agreement between two rankings is measured with the Kendall tau-b coefficient. tau-b counts the concordant and discordant pairs of tools and corrects for ties, which matters because beam uses competition ranking, so ties are common. tau-b is 1 for identical orderings, -1 for exact reversals, and near 0 when the two orderings are unrelated. The report carries the full label-by-label tau matrix and the mean of its off-diagonal entries as a summary. It also gives a consensus ranking, the ranking of the per-label mean ranks, and a flag that is true only when every normalization puts the same tool first. The per-tool smallest and largest rank across the labels are the rank span the [funky heatmap](funky-heatmaps-and-robustness.md) draws in its normalization panel. - -## Which normalizations are compared - -By default the diagnostic compares the four scale-agnostic strategies: `min_max`, `log_min_max`, `rank` and `zscore`, each applied to every metric column. `baseline_relative` and `target_relative` are left out of this uniform sweep. They need a per-metric reference or target value from the card and are tied to the metric's meaning, so they are not a free choice the analyst makes column by column. - -The card-recommended per-metric normalization can be passed in as one labelled candidate. When `beam.rank` runs the diagnostic for the HTML report it passes the recommended strategies as the `recommended` column, so the report compares the headline default against the uniform alternatives rather than comparing only the uniform ones with each other. - -A candidate that cannot run on the input is dropped rather than failing the whole analysis. `log_min_max` requires strictly positive values, so a column with a zero or a negative value drops that candidate. A `target_value` metric admits only `target_relative`, so the uniform strategies all drop on a matrix that contains one. At least two candidates must produce a ranking, otherwise there is nothing to compare and the call raises. - -## Interpreting results - -A high mean tau means the recommendation is stable under the normalization choice: whichever rescaling a reader prefers, they reach about the same order. A low mean tau means the rescaling is itself deciding the order. - -The [Duo 2018 clustering benchmark](../../examples/duo2018/duo2018.qmd) is a worked example. On the three-metric pooled matrix (the Adjusted Rand Index, runtime, Shannon entropy difference) under equal weights, the card-recommended normalization and the four uniform strategies agree at a mean pairwise tau-b of 0.64, and Seurat is the unanimous top under every one of them. So the top recommendation does not depend on the rescaling. The agreement is not complete. The recommended normalization agrees with `min_max` at tau-b 0.79 but with `rank` at only 0.36, because the rank normalization drops the gap sizes that the other strategies keep, so the ordering in the middle of the table moves. This is the typical pattern. The leader is often stable while the mid-tier is not. - -## How to run - -Call `normalization_agreement(scores, polarity)` with a tool by metric matrix and the per-metric polarity from [`beam.cards.polarities_for`](../reference/polarities_for.qmd). Pass `recommended=RegistryContext.normalization` so the card default is compared against the uniform strategies, and pass the same `bounds`, `baselines` and `targets` the headline ranking used. The weighting and the aggregation are held fixed at whatever you pass; the defaults are equal weights and weighted sum. - -For plotting, `beam.plot.normalization_agreement(run)` generates the tau-b heatmap and `beam.plot.normalization_effect(run)` gives a bump chart of each tool's rank across the strategies, so you can follow which methods move. The funky heatmap gains a normalization rank-span panel when you pass `show_normalization_consensus=True`. - -## Limitations - -The diagnostic measures whether the normalizations agree, not which one is correct. The strategies encode different decisions about what a comparable score is: `min_max` treats the observed range as the scale, `rank` treats only the order as meaningful, `log_min_max` treats ratios as the meaningful structure. A disagreement among them is a real disagreement about how the metric should be read. The consensus ranking is a convenience summary, not a preferred normalization. When the strategies disagree, report the disagreement and the reason the card recommends what it does, covered in [Normalization and scales](normalization-and-scales.md). - -## See also - -- [Normalization and scales](normalization-and-scales.md) -- [Aggregation agreement](aggregation-agreement.md) - -## References - -- Kendall, M. G. A new measure of rank correlation. Biometrika 30 (1938). DOI [10.1093/biomet/30.1-2.81](https://doi.org/10.1093/biomet/30.1-2.81). diff --git a/docs/explanations/normalization-and-scales.md b/docs/explanations/normalization-and-scales.md index 8ffc472..48be042 100644 --- a/docs/explanations/normalization-and-scales.md +++ b/docs/explanations/normalization-and-scales.md @@ -51,7 +51,7 @@ The `beam.scenarios` module ships two cases that make the failure concrete. In t ## See also -- [Normalization agreement](normalization-agreement.md) +- [Choice agreement](choice-agreement.md#normalization-agreement) - [Weighting schemes](weighting-schemes.md) - [Aggregation methods](aggregation-methods.md) diff --git a/docs/explanations/ontology-mappings.md b/docs/explanations/ontology-mappings.md index 2131283..efc12d6 100644 --- a/docs/explanations/ontology-mappings.md +++ b/docs/explanations/ontology-mappings.md @@ -68,4 +68,57 @@ The artefact is deposited on Zenodo per release for a permanent identifier. The The cards draw four fields from closed enumerations: polarity, scale_type, allowed_transformations and comparability.recommended_normalization. `src/beam/owl/skos.py` publishes those enumerations as SKOS concept schemes in `docs/beam.skos.ttl`. Each allowed value becomes one `skos:Concept` with a prefLabel and a definition. A `skos:broader` edge records where one value specializes another: the Stevens scale ladder from nominal to ratio, and the normalizations that are special cases of an affine transform. The enum values are read from the JSON Schema, so a value no card uses yet still gets a concept. The definitions are kept in the generator, taken from the [measurement-theory](measurement-theory.md) and [normalization-and-scales](normalization-and-scales.md) essays. Regenerate with `python -m beam.owl.skos`. -A draft of upstream STATO proposals is available at [stato-term-proposals.md](stato-term-proposals.md). +## Draft STATO proposals + +The coverage table above lists the metrics with no STATO term. The proposals below are drafts to file against the STATO tracker (https://github.com/ISA-tools/stato/issues) rather than fill with a beam-private IRI. Other measurands (runtime, peak_memory) and the [transportation metrics](../../examples/transportation/transportation.qmd) (cost, speed, co2) are out of STATO scope. + +To file one, open an issue with the label, the definition, and the parent class named below, and attach the primary reference. When STATO assigns an IRI, write it into the matching card under `mappings.stato`, regenerate the OWL with `python -m beam.owl.generate`, and update the coverage table above. + +### normalized mutual information (partition similarity) + +- Definition: a clustering-agreement measure equal to the mutual information between two partitions divided by a normalizing function of their entropies, valued in the unit interval. +- Parent: a measure of clustering agreement, sibling of the adjusted Rand index (STATO_0000593, already mapped on the ari card). +- beam card: nmi. +- Reference: Strehl and Ghosh 2002, Cluster ensembles, JMLR 3:583-617. + +### silhouette coefficient + +- Definition: a cluster-validity measure equal to the mean over points of the difference between the mean nearest-other-cluster distance and the mean within-cluster distance, normalized by the larger of the two. +- Parent: cluster validity index, a kind of summary statistic. +- beam cards: silhouette, and as children the scIB silhouette variants asw_batch, asw_label and isolated_label_asw, which apply the same coefficient over a batch or a label assignment. One upstream silhouette term covers the family; the variant stays recorded on the beam card. +- Reference: Rousseeuw 1987, Silhouettes, Journal of Computational and Applied Mathematics 20:53-65, DOI [10.1016/0377-0427(87)90125-7](https://doi.org/10.1016/0377-0427%2887%2990125-7). + +### k-nearest-neighbour batch-effect test statistic + +- Definition: a batch-mixing test that compares, over local k-neighbourhoods, the observed batch-label composition against the global composition by a chi-squared statistic, with the rejection rate reported. +- Parent: test statistic, a kind of chi-squared based statistic. +- beam card: kbet. +- Reference: Buttner, Miao, Wolf, Teichmann and Theis 2019, A test metric for assessing single-cell RNA-seq batch correction, Nature Methods 16:43-49, DOI [10.1038/s41592-018-0254-1](https://doi.org/10.1038/s41592-018-0254-1). + +### local inverse Simpson index + +- Definition: a local-neighbourhood diversity measure equal to the inverse Simpson index of the label composition (cLISI) or the batch composition (iLISI) within a perplexity-weighted neighbourhood. +- Parent: diversity index, a kind of summary statistic. +- beam cards: clisi, ilisi. +- Reference: Korsunsky et al. 2019, Fast, sensitive and accurate integration of single-cell data with Harmony, Nature Methods 16:1289-1296, DOI [10.1038/s41592-019-0619-0](https://doi.org/10.1038/s41592-019-0619-0). + +### Shannon entropy of a partition + +- Definition: the Shannon entropy, in nats or bits, of the label distribution of a partition. A STATO term at this level would cover the difference reported by beam's card and any future entropy-based metric. +- Parent: information-theoretic measure, a kind of summary statistic. +- beam card: shannon_entropy_diff. +- Reference: Shannon 1948, A mathematical theory of communication, Bell System Technical Journal 27:379-423, DOI [10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x). + +### symmetric mean absolute percentage error + +- Definition: a forecasting-accuracy error equal to the mean over horizons of the absolute forecast error divided by the average of the absolute actual and forecast values, in percent. +- Parent: a measure of forecast error, a kind of percentage. +- beam card: smape. +- Reference: Makridakis, Spiliotis and Assimakopoulos 2020, The M4 Competition, International Journal of Forecasting 36:54-74, DOI [10.1016/j.ijforecast.2019.04.014](https://doi.org/10.1016/j.ijforecast.2019.04.014). + +### mean absolute scaled error + +- Definition: a forecasting-accuracy error equal to the mean absolute forecast error scaled by the in-sample mean absolute error of a naive one-step forecast. +- Parent: a measure of forecast error, a kind of summary statistic. +- beam card: mase. +- Reference: Hyndman and Koehler 2006, Another look at measures of forecast accuracy, International Journal of Forecasting 22:679-688, DOI [10.1016/j.ijforecast.2006.03.001](https://doi.org/10.1016/j.ijforecast.2006.03.001). diff --git a/docs/explanations/openproblems-as-a-data-source.md b/docs/explanations/openproblems-as-a-data-source.md index 53ff963..783bba0 100644 --- a/docs/explanations/openproblems-as-a-data-source.md +++ b/docs/explanations/openproblems-as-a-data-source.md @@ -1,6 +1,6 @@ # OpenProblems as a data source -beam needs benchmark results that are already a method by dataset by metric tensor with declared metric directions. Most published benchmarks are not in that shape, and putting them together is the data-engineering bottleneck the cross-benchmark meta-analysis runs into. OpenProblems in Single-Cell Analysis (openproblems.bio, Nature Biotechnology 2025, DOI [10.1038/s41587-025-02694-w](https://doi.org/10.1038/s41587-025-02694-w)) already produces that shape, so beam ingests it directly. +beam needs benchmark results that are already a method by dataset by metric tensor with declared metric directions. Most published benchmarks are not in that shape, and putting them together is the data-engineering step the cross-benchmark meta-analysis would otherwise face. OpenProblems in Single-Cell Analysis (openproblems.bio, Nature Biotechnology 2025, DOI [10.1038/s41587-025-02694-w](https://doi.org/10.1038/s41587-025-02694-w)) already produces that shape, so beam ingests it directly. OpenProblems runs a continuous automated single-cell benchmark. Within a task, every method is scored on every dataset by every metric, and the full results are published as CC-BY JSON per task in the openproblems-bio/website GitHub repository. So one task is a method by dataset by metric tensor with the metric directions stated. That is the harmonization beam would otherwise have to do by hand. @@ -9,10 +9,10 @@ beam vendors small derived tables in long format, with columns `method_id`, `dat No single OpenProblems task gives both many metrics and many datasets, so beam bundles two and each runs a different part of the stack. - `batch_integration` has 19 methods, 6 datasets and 13 scIB metrics (Luecken et al. 2022). It is metric-rich, so it runs the multi-criteria decision analysis (MCDA) layer over many criteria. beam re-derives the per-metric [normalization](normalization-and-scales.md) from the metric cards and runs explicit MCDA with [sensitivity](rank-sensitivity.md). This can rank a different method first than the platform's mean-of-scaled-scores rule. In the bundled data, beam with equal weights and TOPSIS leads with combat while the mean-of-scores rule leads with scanvi. The aggregation rule, not the data alone, determines the top method. Coverage is uneven: the `hvg_overlap` column and some method-by-dataset cells are sparse, which beam surfaces as [NaN](missing-data.md). -- `spatially_variable_genes` has 14 methods, 50 datasets and one correlation metric. It is dataset-rich, which is what the [Bradley-Terry tree](heterogeneity-bradley-terry.md) needs to find feature-based splits. The dataset features (spatial assay technology, organism, cancer condition) parse out of the dataset id. The tree splits on technology: spark_x leads the pooled ranking, but spanve leads on the visium and xenium datasets and nnsvg leads on the seqfish, slideseqv2 and slidetags datasets. The top spatially-variable-gene method depends on the assay. +- `spatially_variable_genes` has 14 methods, 50 datasets and one correlation metric. It is dataset-rich, which is what the [Bradley-Terry tree](method-by-dataset-heterogeneity.md#bradley-terry-trees) needs to find feature-based splits. The dataset features (spatial assay technology, organism, cancer condition) parse out of the dataset id. The tree splits on technology: spark_x leads the pooled ranking, but spanve leads on the visium and xenium datasets and nnsvg leads on the seqfish, slideseqv2 and slidetags datasets. The top spatially-variable-gene method depends on the assay. Because no one task carries both breadth in metrics and breadth in datasets, `batch_integration` covers the MCDA breadth and `spatially_variable_genes` covers the heterogeneity depth. The published results JSON does not include dataset-level numeric features such as cell counts or batch design, so beam uses only the categorical descriptors parseable from the dataset id, and for the spatial data the technology and organism are partly confounded. This bounds what the tree can attribute a split to. -OpenProblems re-derives its leaderboard by a fixed mean-of-normalized-scores rule. beam applies MCDA aggregation plus sensitivity, as the `batch_integration` example shows. Because OpenProblems is a sister continuous-benchmarking platform to omnibenchmark, an ingestion also serves as a reference for the future omnibenchmark adapter. +OpenProblems re-derives its leaderboard by a fixed mean-of-normalized-scores rule. beam applies MCDA aggregation plus sensitivity, as the `batch_integration` example shows. Because OpenProblems is a sister continuous-benchmarking platform to omnibenchmark, an ingestion is also a reference for the future omnibenchmark adapter. The [OpenProblems vignette](../../examples/openproblems/openproblems.qmd) works both tasks through end to end. diff --git a/docs/explanations/pairwise-method-comparison.md b/docs/explanations/pairwise-method-comparison.md new file mode 100644 index 0000000..d5ee91d --- /dev/null +++ b/docs/explanations/pairwise-method-comparison.md @@ -0,0 +1,81 @@ +# Pairwise method comparison + +The [critical-difference diagram](critical-difference.md) says which methods differ beyond chance across the datasets. It does not say by how much, how often one method outperforms another, or whether the pairwise results agree with a single order. Its mean-rank post-hoc also depends on the whole pool: add or drop a method and two others can change from "different" to "the same" (Benavoli, Corani and Mangili 2016). beam has three pool-independent checks that read the same pairwise evidence: a frequentist effect size, a transitivity test, and a Bayesian posterior. + +## The pairwise counts + +For a pair of methods, look at the datasets they share. On each one, the method with the higher score outperforms the other on that dataset, for that metric. Over the datasets this gives three numbers: how often A outperforms B, how often B outperforms A, and how often the two are equivalent. + +Two methods are equivalent on a dataset when their scores differ by no more than the region of practical equivalence, the ROPE. Set the ROPE to the metric's [noise floor](reference-levels.md) (`comparability.noise_floor`), the smallest difference the card calls interpretable, and one method counts as outperforming the other only when the difference clears that floor. With a ROPE of zero, any non-zero difference counts. All three checks read these same counts. + +## Probability of superiority + +[`beam.mcda.pairwise_superiority`](../reference/pairwise_superiority.qmd) reports the effect size. The probability of superiority of A over B is the fraction of shared datasets on which A outperforms B, a common-language effect size (Grissom 1994): how often A scores higher than B on a dataset like these. A sign test on the decisive datasets, equivalences dropped, says whether the difference is more than chance. + +```python +from beam.mcda import pairwise_superiority +from beam.cards import properties_for + +floor = properties_for(["ari"])[0].noise_floor +report = pairwise_superiority(ari_by_dataset, "higher_is_better", rope=floor, + method_names=method_names) +report.order[0] # the method with the highest standing +report.probability_superior # P(row outperforms column), a matrix +report.equivalent_pairs # pairs the sign test cannot tell apart +``` + +`standing` is a Copeland-style score per method: the mean over the other methods of the chance of outperforming or being equivalent to them, in `[0, 1]`. One outperforms every other method on every dataset, 0.5 is even. `equivalent_pairs` lists the pairs whose sign test does not reach the chosen level, so no decisive difference. + +On the [Duo 2018](../../examples/duo2018/duo2018.qmd) Adjusted Rand Index (ARI) scores, with the ARI noise floor of 0.01 as the ROPE, SC3 has the highest standing. Most method pairs are practically equivalent: the sign test cannot separate more than half of them, because the differences across the twelve datasets are often within the noise floor. This agrees with the critical-difference reading, where most methods fall in overlapping cliques, and states it as an effect size: even the two leading methods are within the floor on most datasets rather than one clearly outperforming the other. + +## Transitivity + +An aggregation returns one ordering of the methods. That ordering is a fair summary only when the pairwise comparisons behind it agree with a single order, and they do not always agree. Method A can outperform B on most of the datasets they share, B outperform C, and C outperform A. These three results form a cycle, and no ordering agrees with all the pairwise majorities. + +[`beam.mcda.pairwise_transitivity`](../reference/pairwise_transitivity.qmd) reads the superiority report and turns the counts into a single relation: method `i` outperforms method `j` when it does so more often than `j` outperforms `i` across their shared datasets. A pair that splits evenly, or that shares no decisive dataset, is left with no edge and recorded as tied. Because the counts already apply the ROPE, a difference inside the noise floor does not create an edge, so near-ties do not produce spurious cycles. + +From the relation the function reports four things. + +- The method preferred to every other method by pairwise majority, when one exists. Condorcet described this case in his 1785 essay on majority voting; it is the option preferred to every other one in a head-to-head majority vote. Such a method might not exist, and when it does the rest of the relation can still be inconsistent. +- The circular triads: sets of three methods whose edges form a cycle. Their count, out of all method triples, measures how far the relation is from transitive. +- Kendall and Babington Smith's (1940) coefficient of consistence, `1 - d / d_max`, where `d` is the circular-triad count and `d_max` is the maximum possible for this relation size. It runs from 1 (transitive) down to 0 (least consistent). It is defined only when every pair is decided, so it is reported as undefined when any pair is tied. +- Whether the relation is transitive, and the single order it implies when it is transitive and every pair is decided. + +```python +from beam.mcda import pairwise_transitivity + +trans = pairwise_transitivity(report) +trans.is_transitive # whether one order is consistent with the pairwise majorities +trans.circular_triads # the cyclic triples, if any +trans.condorcet_choice # method preferred to all others, or None +trans.coefficient_of_consistence # Kendall's coefficient, or None when pairs are tied +``` + +On the Duo 2018 ARI scores, with the noise floor of 0.01 as the equivalence band, SC3 is preferred to every other method by pairwise majority, the same method the standing score, the marginal means and the [Bradley-Terry ranking](method-by-dataset-heterogeneity.md#bradley-terry-trees) pick out. Even so the relation is not transitive: there is a cycle and no single order agrees with every pairwise majority. The cycle is among methods that the critical-difference diagram and the pairwise effect sizes already place in overlapping groups, where a few datasets can swing the majority either way. Raising the equivalence band from zero to the noise floor cuts the cycles from five to one. + +## Bayesian sign comparison + +The sign test and the critical-difference diagram report a p-value, the chance of the observed split if the two methods scored the same. [`beam.mcda.bayesian_sign_comparison`](../reference/bayesian_sign_comparison.qmd) reports the probability that one method scores higher than the other given the data, using the Bayesian sign test of Benavoli et al. (2017) on the same per-pair counts. + +For a method pair, each shared dataset falls into one of three regions: A scores higher by more than the ROPE, B scores higher by more than the ROPE, or the two are within it. The share of datasets in each region is unknown. The test models the three shares with a Dirichlet posterior whose parameters are the observed counts plus a small prior, and reports three probabilities that sum to one: that A is practically better than B, that the two are practically equivalent, and that B is practically better than A. A pair gets a decisive label when one of the three reaches the threshold (0.95 by default), and is inconclusive otherwise. The report also carries the posterior mean share of each region and a standing score per method, the mean over the other methods of the probability of scoring at least as high as them. + +The default places one prior pseudo-observation on the equivalence region, matching the baycomp default. Two other placements are available: `uniform` spreads the prior across the three regions, and `neutral` spreads it across the two directional regions. One pseudo-observation is outweighed after a few datasets. + +## Related checks + +[`critical_difference`](critical-difference.md) tests separability across all methods at once. [`noise_floor_separation`](reference-levels.md) reads the aggregate scores, one value per method, and flags pairs no metric separates above the floor. [`aggregation_agreement`](choice-agreement.md) shows when the aggregation rules disagree; a cycle in the pairwise relation is one reason they can. + +## Limitations + +These reports describe the datasets in hand, not a population of datasets. The comparison is paired by dataset, so it needs the same datasets for both methods; a pair with little overlap is compared on whatever they share, and the count says how many that was. The sign test drops equivalences, so a pair equivalent on most datasets has few decisive ones and a weak test. The Bayesian posterior stays close to the prior when the datasets are few, so most pairs are inconclusive at the threshold. All three use only the direction of each dataset's difference, not its size, and all depend on the ROPE, so set the band from the metric's noise floor. + +## References + +- Benavoli, A., Corani, G., Mangili, F. Should we really use post-hoc tests based on mean-ranks? Journal of Machine Learning Research 17 (2016). +- Grissom, R. J. Probability of the superior outcome of one treatment over another. Journal of Applied Psychology (1994). DOI [10.1037/0021-9010.79.2.314](https://doi.org/10.1037/0021-9010.79.2.314). +- Condorcet, M. de. Essai sur l'application de l'analyse a la probabilite des decisions rendues a la pluralite des voix. Imprimerie Royale, Paris (1785). +- Kendall, M. G., Babington Smith, B. On the method of paired comparisons. Biometrika 31, 324 (1940). DOI [10.1093/biomet/31.3-4.324](https://doi.org/10.1093/biomet/31.3-4.324). +- Benavoli, A., Corani, G., Demsar, J., Zaffalon, M. Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis. Journal of Machine Learning Research 18(77):1-36 (2017). https://jmlr.org/papers/v18/16-305.html +- Corani, G., Benavoli, A. A Bayesian approach for comparing cross-validated algorithms on multiple data sets. Machine Learning 100(2-3):285-304 (2015). https://doi.org/10.1007/s10994-015-5486-z + +The reference implementation is the baycomp package (https://github.com/janezd/baycomp), against which beam is cross-checked. diff --git a/docs/explanations/pairwise-superiority.md b/docs/explanations/pairwise-superiority.md deleted file mode 100644 index fee0bf2..0000000 --- a/docs/explanations/pairwise-superiority.md +++ /dev/null @@ -1,44 +0,0 @@ -# Comparing two methods at a time - -The [critical-difference diagram](comparing-methods-across-datasets.md) says which methods differ beyond chance. It does not say by how much, or how often one method outperforms another. Its mean-rank post-hoc also depends on the whole pool: add or drop a method and two others can change from "different" to "the same" (Benavoli, Corani and Mangili 2016). [`beam.mcda.pairwise_superiority`](../reference/pairwise_superiority.qmd) reports an effect size that does not depend on the pool. - -## How it works - -For a pair of methods, look at the datasets they share. On each one, the method with the higher score outperforms the other on that dataset, for that metric. Over the datasets this gives three numbers: how often A outperforms B, how often B outperforms A, and how often the two are equivalent. - -Two methods are equivalent on a dataset when their scores differ by no more than the region of practical equivalence, the ROPE. Set the ROPE to the metric's [noise floor](reference-levels.md) (`comparability.noise_floor`), the smallest difference the card calls interpretable, and one method counts as outperforming the other only when the difference clears that floor. With a ROPE of zero, any non-zero difference counts. - -The probability of superiority of A over B is the fraction of shared datasets on which A outperforms B. It is a common-language effect size, the standard name for it (Grissom 1994): how often A scores higher than B on a dataset like these. A sign test on the decisive datasets, equivalences dropped, says whether the difference is more than chance. - -## What it reports - -```python -from beam.mcda import pairwise_superiority -from beam.cards import properties_for - -floor = properties_for(["ari"])[0].noise_floor -report = pairwise_superiority(ari_by_dataset, "higher_is_better", rope=floor, - method_names=method_names) -report.order[0] # the method with the highest standing -report.probability_superior # P(row outperforms column), a matrix -report.equivalent_pairs # pairs the sign test cannot tell apart -``` - -`standing` is a Copeland-style score per method: the mean over the other methods of the chance of outperforming or being equivalent to them, in `[0, 1]`. One outperforms every other method on every dataset, 0.5 is even. `equivalent_pairs` lists the pairs whose sign test does not reach the chosen level, so no decisive difference. - -## On real data - -On the [Duo 2018](../../examples/duo2018/duo2018.qmd) Adjusted Rand Index (ARI) scores, with the ARI noise floor of 0.01 as the ROPE, SC3 has the highest standing. But most method pairs are practically equivalent: the sign test cannot separate more than half of them, because the differences across the twelve datasets are often within the noise floor. This agrees with the critical-difference reading, where most methods fall in overlapping cliques, and states it as an effect size: even the two leading methods are within the floor on most datasets rather than one clearly outperforming the other. - -## Related checks - -[`critical_difference`](comparing-methods-across-datasets.md) tests significance across all methods at once. [`noise_floor_separation`](reference-levels.md) reads the aggregate scores, one value per method, and flags pairs no metric separates above the floor. `pairwise_superiority` works across the datasets, pair by pair, and reports an effect size with the noise floor as the equivalence band. Significance, aggregate separation, and per-dataset effect size are three readings of the same question. - -## Limitations - -The probability of superiority describes these datasets, not a population of datasets. The sign test drops equivalences, so a pair that is equivalent on most datasets has few decisive ones and a weak test, which is the expected reading when the methods are close. The comparison is paired by dataset, so it needs the same datasets for both methods; a pair with little overlap is compared on whatever they share, and the count says how many that was. - -## References - -- Benavoli, A., Corani, G., Mangili, F.. Should we really use post-hoc tests based on mean-ranks?. Journal of Machine Learning Research 17 (2016). -- Grissom, R. J.. Probability of the superior outcome of one treatment over another. Journal of Applied Psychology (1994). DOI [10.1037/0021-9010.79.2.314](https://doi.org/10.1037/0021-9010.79.2.314). diff --git a/docs/explanations/rank-sensitivity.md b/docs/explanations/rank-sensitivity.md index d33f737..49d6346 100644 --- a/docs/explanations/rank-sensitivity.md +++ b/docs/explanations/rank-sensitivity.md @@ -1,4 +1,4 @@ -# What moves the ranking +# Rank sensitivity and the specification curve A ranking can move for two reasons. One is the data: a method that ranks first on one dataset can trail on another. The other is the analyst's choices: which [weighting scheme](weighting-schemes.md) sets the weights, and which [aggregation rule](aggregation-methods.md) combines the scores. [`beam.mcda.rank_sensitivity`](../reference/rank_sensitivity.qmd) measures both at once. The weighting, the aggregation, and the dataset are each a small set of options, so beam runs every combination instead of sampling. For each tool this gives a table of its rank over the full grid. An analysis of variance then splits the rank variance into a share for each factor and a share for their interactions. The shares sum to one. @@ -9,7 +9,7 @@ For a deterministic function of a few categorical factors over a balanced full g The headline is the share each factor carries, pooled over the tools: -- A large dataset share means the ranking depends on which dataset you use. That is method heterogeneity, the same thing the [Bradley-Terry tree](heterogeneity-bradley-terry.md) and the [mixed-effects decomposition](heterogeneity-mixed-effects.md) read in other ways. +- A large dataset share means the ranking depends on which dataset you use. That is method heterogeneity, the same thing the [Bradley-Terry tree](method-by-dataset-heterogeneity.md#bradley-terry-trees) and the [mixed-effects decomposition](method-by-dataset-heterogeneity.md) read in other ways. - A large weighting or aggregation share means the ranking depends on a choice the analyst could make differently. That is a degree of freedom the report should disclose. - A large interaction share means the choice matters more on some datasets than others. @@ -45,13 +45,39 @@ On the [Duo 2018 clustering benchmark](../../examples/duo2018/duo2018.qmd) the d The default weightings are equal, entropy, standard deviation, and CRITIC. MEREC is left out by default because it takes the logarithm of the scores and refuses a zero, and the default min_max normalization maps each column minimum to zero. Pass MEREC explicitly with a normalization that keeps the scores positive. -Every combination must produce a ranking. The distance and outranking aggregations (TOPSIS, VIKOR, PROMETHEE II, [COMET](comet.md)) refuse a slice with missing cells, so a tensor with gaps needs `missing="worst"` to complete the matrix, or a restriction to the feasible subset. A factor level that still fails on the input is dropped, and the report names what it dropped. +Every combination must produce a ranking. The distance and outranking aggregations (TOPSIS, VIKOR, PROMETHEE II, [COMET](aggregation-methods.md#comet)) refuse a slice with missing cells, so a tensor with gaps needs `missing="worst"` to complete the matrix, or a restriction to the feasible subset. A factor level that still fails on the input is dropped, and the report names what it dropped. COMET is slow on a metric-rich benchmark. It builds characteristic objects whose count grows fast with the number of metrics, so on a task with a dozen metrics the other four aggregations are the practical choice. The shares describe this set of options on this data. They are not an inference back to a population of datasets or a population of analysts. A tool that holds the same rank in every combination has no variance to split, so its per-tool shares are undefined and reported as such. +## The specification curve + +The shares say which choice moves the ranking. The specification curve reads the same grid a different way: it lists the ranking each combination produces and counts how often the top method stays the same (Simonsohn, Simmons and Nelson 2020; Steegen, Tuerlinckx, Gelman and Vanpaemel 2016). [`beam.mcda.specification_curve`](../reference/specification_curve.qmd) takes the `RankSensitivityReport` and post-processes it, without re-ranking: + +- `specifications`: one record per combination, with its factor levels, the full tool ordering, and the top-ranked tool. +- `most_frequent_top_fraction`: the fraction of combinations that rank the same tool first. Near 1 means the top is stable across the choices. +- `modal_order_fraction`: the fraction that produce the single most common ordering of all tools. This is stricter, since it asks the whole order to repeat. +- `n_distinct_top_tools`: how many tools rank first in at least one combination. +- `curve_order`: the combinations sorted by the rank the most-frequent top method takes, so a plot reads left to right from its best rank to its worst. + +```python +from beam.mcda import specification_curve + +curve = specification_curve(report) # report from rank_sensitivity above +print(curve.most_frequent_top_fraction, curve.n_distinct_top_tools) +``` + +On [Duo 2018](../../examples/duo2018/duo2018.qmd) over the Adjusted Rand Index, runtime and the Shannon entropy difference, the analyst-choice grid has 20 combinations: four weightings by five aggregations on the pooled matrix. Seurat ranks first in every one, so the top does not depend on the weighting or the aggregation, but the full ordering repeats in only 15 percent, so the middle of the table reshuffles. Adding the twelve datasets as a third factor grows the grid to 240 combinations: Seurat now ranks first in 49 percent, and five tools rank first in at least one, which is the data contribution on top of the analyst's freedom. On the [M4 forecasting competition](../../examples/m4/m4.qmd) the 20-combination choice grid leaves Pawlikowski first in 90 percent, and the 120-combination grid that adds the six frequency bands drops the top tool, Smyl, to 33 percent with five tools reaching the top. + +The HTML report plots the specification curve for a tensor input, in the same section as the variance decomposition. The top panel plots the rank of the most-frequent top method across the combinations sorted from its best to its worst; the panel beneath marks which weighting, aggregation and dataset each used. + ## See also -- [Specification curve](specification-curve.md) - [Attribution synthesis](attribution-synthesis.md) +- [Analysis blinding](analysis-blinding.md) + +## References + +- Simonsohn, U., Simmons, J. P., Nelson, L. D. Specification curve analysis. Nature Human Behaviour 4, 1208-1214 (2020). DOI [10.1038/s41562-020-0912-z](https://doi.org/10.1038/s41562-020-0912-z). +- Steegen, S., Tuerlinckx, F., Gelman, A., Vanpaemel, W. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11, 702-712 (2016). DOI [10.1177/1745691616658637](https://doi.org/10.1177/1745691616658637). diff --git a/docs/explanations/reference-levels.md b/docs/explanations/reference-levels.md index b647698..116bbd6 100644 --- a/docs/explanations/reference-levels.md +++ b/docs/explanations/reference-levels.md @@ -26,5 +26,5 @@ The chance baseline is exact only where a metric has a well-defined chance level ## See also -- [Pairwise superiority](pairwise-superiority.md) -- [Bayesian comparison](bayesian-comparison.md) +- [Pairwise superiority](pairwise-method-comparison.md) +- [Bayesian comparison](pairwise-method-comparison.md#bayesian-sign-comparison) diff --git a/docs/explanations/reliability.md b/docs/explanations/reliability.md deleted file mode 100644 index aeb7b91..0000000 --- a/docs/explanations/reliability.md +++ /dev/null @@ -1,37 +0,0 @@ -# Internal-consistency reliability of a metric group - -A benchmark often treats a group of metrics as one criterion. The [scIB integration score](../../examples/openproblems/openproblems.qmd) groups its metrics into biological conservation and batch correction and weights the two groups 0.6 and 0.4, so each group is read as a single (composite) scale. The [convergent and discriminant validity](convergent-discriminant-validity.md) check asks whether that grouping is the right split. Reliability asks the next question: if a group is going to be read as one scale, how consistently do its metrics measure one thing? [`beam.mcda.metric_reliability`](../reference/metric_reliability.qmd) answers it with standardized Cronbach's alpha, following Cronbach (1951). - -## Rationale - -Each method-by-dataset cell is one observation, and the metrics are the variables. The function orients every metric so that higher means better, reading the polarity from the cards, then computes the Spearman rank correlation between every pair of metrics over the observations they share. This is the same orientation and correlation step the validity check uses, so the two diagnostics rest on the same numbers. - -For each group the report gives the standardized alpha - - alpha = k * r_bar / (1 + (k - 1) * r_bar) - -where `k` is the number of metrics in the group and `r_bar` is their mean inter-item correlation. The standardized form, built from correlations rather than from raw covariances, is appropriate here because the metrics are on different scales. It is the rank-based analogue of classical alpha, in line with the way the rest of beam compares metrics by rank. - -## Interpretation - -Alpha runs up to 1. A common rule of thumb treats 0.7 as the point above which a group reads as one reliable scale, and `metric_reliability` flags any group below that cutoff. The cutoff is somewhat arbitrary, so the report carries `r_bar` and `k` next to each alpha, so analysts make a decision. - -Alpha rises with the mean correlation and with the number of metrics, so a long group can reach a high alpha on modest agreement, and a short group needs stronger agreement to reach the same cutoff. When two groups differ in size, the mean inter-item correlation is the comparison to use, because it does not carry the size effect. - -For a group of three or more metrics, the report recomputes the group's alpha with each metric removed in turn. A metric whose removal raises the group's alpha agrees with its labelled construct less than the others do. It is the metric to reconsider first, whether to relabel it, drop it, or treat it as a separate criterion. - -A metric in a group of two has no alpha-if-dropped entry, because dropping one leaves a single metric and alpha is undefined for one item. - -## Limitations - -Alpha assumes the group is one reflective factor, a single underlying quantity that each metric reads with noise, which the [dimensionality](dimensionality.md) check tests. A low alpha is evidence against that assumption, not a measurement of which metric fails; the alpha-if-dropped diagnostic narrows that down but does not settle it. Alpha is also not a validity check. A group can be reliable and still measure the wrong thing, which is why reliability reads alongside the convergent and discriminant validity rather than instead of it. - -The result is descriptive of the methods and datasets in the input. A small benchmark gives a coarse estimate, and the grouping it scores is the analyst's, so the diagnostic informs that judgement. - -## On beam's examples: OpenProblems batch integration - -On the [OpenProblems batch integration scores](../../examples/openproblems/openproblems.qmd), where there is an expertly-curated classification of metrics in bio/batch, the biological conservation group is reported to be reliable: alpha 0.85 over seven metrics, mean inter-item correlation 0.45. The batch correction metrics group does not reach the cutoff: alpha 0.62 over five metrics, mean inter-item correlation 0.24, the one group flagged as low reliability. Dropping `pcr` is the only batch removal that raises the batch alpha by more than a rounding step, from 0.62 to 0.67, so `pcr` is the batch metric least consistent with the rest of its group. - -## References - -- Cronbach, L. J.. Coefficient alpha and the internal structure of tests. Psychometrika (1951). DOI [10.1007/BF02310555](https://doi.org/10.1007/BF02310555). diff --git a/docs/explanations/skillings-mack.md b/docs/explanations/skillings-mack.md deleted file mode 100644 index 573a58d..0000000 --- a/docs/explanations/skillings-mack.md +++ /dev/null @@ -1,56 +0,0 @@ -# Skillings-Mack: coverage-aware Friedman - -The Demsar [Friedman-Nemenyi test](comparing-methods-across-datasets.md) in [`beam.mcda.critical_difference`](../reference/critical_difference.qmd) only runs on a complete tool by dataset matrix. Real benchmarks rarely give one: methods time out, error on an input, or were not run on every dataset. beam's [missing-data policy](missing-data.md) refuses to fill those gaps by imputation, so the critical-difference test does not run on the matrix and reduces it to the complete observations. On a wide table with many partial methods, there might be no complete observations. - -The Skillings-Mack (1981) test fills the gap with a Friedman-type statistic that does not need a complete matrix. It is provided as [`beam.mcda.skillings_mack`](../reference/skillings_mack.qmd) and as the convenience alias [`beam.mcda.coverage_aware_critical_difference`](../reference/coverage_aware_critical_difference.qmd). - -## Rationale - -The input is the same tool by dataset matrix the Friedman test takes, with NaN allowed. The test answers the same global question, "are the methods separable across the datasets, or does the apparent ranking fall within noise". It does not give pairwise comparisons; the Nemenyi post-hoc needs the complete-matrix construction that Skillings-Mack generalizes. - -Within each block (column) $j$ the methods that are present are ranked from 1 (lowest score) to $k_j$ (highest score), with average ranks for ties. The within-block rank for method $i$ is then centred and standardized by the block size: - -$$ -A_{ij} = \left(R_{ij} - \frac{k_j + 1}{2}\right) \sqrt{\frac{12}{k_j + 1}} -$$ - -The factor $\sqrt{12 / (k_j + 1)}$ makes the variance of $A_{ij}$ under the null (the method's rank is uniformly distributed over $1, \dots, k_j$) equal to $k_j - 1$, whatever the block size. Summing over the blocks where method $i$ appears gives the per-method statistic $A_i$. The vector $A$ has length equal to the number of methods and sums to zero. - -The covariance matrix of $A$ under the null is - -$$ -\Sigma_{ii} = \sum_{\text{blocks } j \text{ containing method } i} (k_j - 1) -$$ - -$$ -\Sigma_{ij} = -(\text{number of blocks containing both } i \text{ and } j), \quad i \neq j -$$ - -Every row of $\Sigma$ sums to zero, so $\Sigma$ is rank-deficient by one. Dropping any single row and column gives a positive definite $(n - 1) \times (n - 1)$ submatrix that can be inverted. The test statistic is the quadratic form - -$$ -T = A_{\text{reduced}}^{\top} \, \Sigma_{\text{reduced}}^{-1} \, A_{\text{reduced}} -$$ - -which is $\chi^2$ distributed with $n_{\text{methods}} - 1$ degrees of freedom under the null. The choice of which row and column to drop does not affect the statistic, because $A$ lies in the column space of $\Sigma$. - -## Implications - -The Nemenyi critical difference depends on every pair of methods being ranked on every dataset. With incomplete blocks the per-method "average rank" no longer has the same denominator across methods, and the studentized range distribution that justifies Nemenyi assumes equal block sizes. beam returns the global Skillings-Mack test only. For pairwise statements the user must restrict the matrix to the block of methods and datasets where all of them ran and run `critical_difference` there. - -On a complete matrix Skillings-Mack collapses to the Friedman $\chi^2$ statistic. Every block has the same $k$, the standardizing factor is constant, and the covariance structure becomes the same as the rank-sum formulation in Friedman. beam tests this equivalence at every random seed in the test suite to within $10^{-10}$. With a caveat. scipy's `friedmanchisquare` applies a tie correction that divides the statistic by $1 - T / (k (k^2 - 1) N)$ where $T$ is the standard ties term. The standard Skillings-Mack formulation in the 1981 paper does not include this correction. So the two statistics differ when there are within-block ties. Without ties they agree to machine precision. - -## Choosing `skillings_mack` vs `critical_difference` - -- Restrict the matrix to the complete block of methods and datasets where every method ran, and call `critical_difference`. This gives a global test and the Nemenyi cliques but drops every dataset where any method failed to run. It fits when the complete block is large enough. -- Keep every observed score and call `skillings_mack` on the partial matrix. This gives a global test only, with no pairwise cliques. It fits when restricting to the complete block drops most of the data. - -## See also - -- [Comparing methods across datasets](comparing-methods-across-datasets.md) -- [Mixed-effects model](heterogeneity-mixed-effects.md) - -## References - -- Skillings JH, Mack GA. On the use of a Friedman-type statistic in balanced and unbalanced block designs. Technometrics 1981, 23(2):171-177. DOI [10.1080/00401706.1981.10486261](https://doi.org/10.1080/00401706.1981.10486261). -- Demsar J. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7 (2006). diff --git a/docs/explanations/specification-curve.md b/docs/explanations/specification-curve.md deleted file mode 100644 index 52de873..0000000 --- a/docs/explanations/specification-curve.md +++ /dev/null @@ -1,59 +0,0 @@ -# The specification curve - -A ranking depends on a few choices: which [weighting scheme](weighting-schemes.md), which [aggregation rule](aggregation-methods.md), and, when there is more than one dataset, which dataset to read. If only a ranking is reported from a single set of choices there is no possibility to know how other choices would have impacted the result. The specification curve runs every combination of the choices and reports the ranking each one gives, allowing to evaluate their variation (Simonsohn, Simmons and Nelson 2020; Steegen, Tuerlinckx, Gelman and Vanpaemel 2016). - -beam already runs the full grid in [`beam.mcda.rank_sensitivity`](rank-sensitivity.md): every combination of the weighting, the aggregation and the dataset. `rank_sensitivity` splits the rank variance into a share per factor, which says which choice moves the ranking. [`beam.mcda.specification_curve`](../reference/specification_curve.qmd) reads the same grid, lists the rankings, and counts how often the top method stays the same. - -## Outputs - -`specification_curve` takes a `RankSensitivityReport` and returns a `SpecificationCurveReport`. It does not re-rank methods but, rather, post-processes the grid that `rank_sensitivity` already ran. - -- `specifications`: one record per combination, with the factor levels that define it, the full tool ordering it produces, and the top-ranked tool. -- `most_frequent_top_fraction`: the fraction of combinations that rank the same tool first. Near 1 means the top is stable across the choices. -- `modal_order_fraction`: the fraction of combinations that produce the single most common ordering of all tools. This is stricter than the top fraction, since it asks the whole order to repeat. -- `n_distinct_top_tools`: how many tools rank first in at least one combination. -- `curve_order`: the combinations sorted by the rank that the method ranking first most often takes, so a plot reads left to right from its best rank to its worst. - -## Usage - -The `rank_sensitivity` grid has two factors for a tool-by-metric matrix (the weighting and the aggregation) and three for a tool-by-dataset-by-metric tensor (the aforementioned two plus the dataset). They can be used to answer different questions - -Running `rank_sensitivity` on the tool-by-metric matrix gets the analyst-choice possibilities: every weighting by every aggregation, on the ranking pooled across datasets. Running it on the tool-by-dataset-by-metric adds the data component, which might be interesting if they are heterogeneous. - -```python -from beam.mcda import rank_sensitivity, specification_curve, registry_context - -ctx = registry_context(metric_ids, "saw") -report = rank_sensitivity( - tensor, # (n_tools, n_datasets, n_metrics) - ctx.polarity, - normalization=list(ctx.normalization), - bounds=list(ctx.bounds), - baselines=list(ctx.baselines), - targets=list(ctx.targets), - missing="worst", - tool_names=tool_names, - dataset_names=dataset_names, -) -curve = specification_curve(report) -print(curve.most_frequent_top_fraction, curve.n_distinct_top_tools) -``` - -## On beam's examples - -On [Duo 2018](../../examples/duo2018/duo2018.qmd) over the Adjusted Rand Index, runtime and the Shannon entropy difference, the analyst-choice multiverse has 20 combinations: four weightings by five aggregations on the pooled matrix. Seurat ranks first in every one, so the top does not depend on the weighting or the aggregation. The full ordering repeats in only 15 percent of them, so the middle of the table reshuffles with the choice even though the top does not. - -If adding the twelve datasets as a third factor, the grid grows to 240 combinations. Seurat now ranks first in 49 percent, and five tools rank first in at least one. Hence reflecting the data contribution beyond analysist's freedom. - -On the [M4 forecasting competition](../../examples/m4/m4.qmd) the pattern is similar. The 20-combination choice multiverse leaves Pawlikowski first in 90 percent. The 120-combination grid that adds the six frequency bands drops the top tool, Smyl, to 33 percent, with five tools reaching the top. The ranking is stable to how weighting and aggregation are done, and unstable to the frequency bands (data). - -The HTML report plots the specification curve for a tensor input, in the same section as the variance decomposition. The top panel plots the rank of the method that ranks first most often, across the combinations sorted from its best to its worst; the panel beneath marks which weighting, aggregation and dataset each combination used. - -## See also - -- [Analysis blinding](analysis-blinding.md) - -## References - -- Simonsohn, U., Simmons, J. P., Nelson, L. D. Specification curve analysis. Nature Human Behaviour 4, 1208-1214 (2020). DOI [10.1038/s41562-020-0912-z](https://doi.org/10.1038/s41562-020-0912-z). -- Steegen, S., Tuerlinckx, F., Gelman, A., Vanpaemel, W. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11, 702-712 (2016). DOI [10.1177/1745691616658637](https://doi.org/10.1177/1745691616658637). diff --git a/docs/explanations/stato-term-proposals.md b/docs/explanations/stato-term-proposals.md deleted file mode 100644 index b402cf7..0000000 --- a/docs/explanations/stato-term-proposals.md +++ /dev/null @@ -1,60 +0,0 @@ -# STATO term proposals for gapped metrics - -The per-card mappings table in [ontology mappings](ontology-mappings.md) lists the metrics that carry no STATO term today. This document drafts some additions. They can be filed against the STATO issue tracker at https://github.com/ISA-tools/stato/issues rather than filled by a beam-private IRI. - -Other measurands (runtime, peak_memory) and the [transportation metrics](../../examples/transportation/transportation.qmd) (cost, speed, co2) are out of STATO scope. - -## How to file a proposal - -Open an issue on the STATO tracker with the proposed label, the definition, and the parent class named below. Attach the primary reference. When STATO assigns an IRI, write it into the matching card under `mappings.stato`, regenerate the OWL artefact with `python -m beam.owl.generate`, and update the coverage table in [ontology mappings](ontology-mappings.md). - -## Proposals (as of July 2026) - -### normalized mutual information (partition similarity) - -- Definition: a clustering-agreement measure equal to the mutual information between two partitions divided by a normalizing function of their entropies, valued in the unit interval. -- Parent: a measure of clustering agreement, sibling of the adjusted Rand index (STATO_0000593, already mapped on the ari card). -- beam card: nmi. -- Reference: Strehl and Ghosh 2002, Cluster ensembles, JMLR 3:583-617. - -### silhouette coefficient - -- Definition: a cluster-validity measure equal to the mean over points of the difference between the mean nearest-other-cluster distance and the mean within-cluster distance, normalized by the larger of the two. -- Parent: cluster validity index, a kind of summary statistic. -- beam cards: silhouette, and as children the scIB silhouette variants asw_batch, asw_label and isolated_label_asw, which apply the same coefficient over a batch or a label assignment. One upstream silhouette term covers the family; the variant stays recorded on the beam card. -- Reference: Rousseeuw 1987, Silhouettes, Journal of Computational and Applied Mathematics 20:53-65, DOI [10.1016/0377-0427(87)90125-7](https://doi.org/10.1016/0377-0427%2887%2990125-7). - -### k-nearest-neighbour batch-effect test statistic - -- Definition: a batch-mixing test that compares, over local k-neighbourhoods, the observed batch-label composition against the global composition by a chi-squared statistic, with the rejection rate reported. -- Parent: test statistic, a kind of chi-squared based statistic. -- beam card: kbet. -- Reference: Buttner, Miao, Wolf, Teichmann and Theis 2019, A test metric for assessing single-cell RNA-seq batch correction, Nature Methods 16:43-49, DOI [10.1038/s41592-018-0254-1](https://doi.org/10.1038/s41592-018-0254-1). - -### local inverse Simpson index - -- Definition: a local-neighbourhood diversity measure equal to the inverse Simpson index of the label composition (cLISI) or the batch composition (iLISI) within a perplexity-weighted neighbourhood. -- Parent: diversity index, a kind of summary statistic. -- beam cards: clisi, ilisi. -- Reference: Korsunsky et al. 2019, Fast, sensitive and accurate integration of single-cell data with Harmony, Nature Methods 16:1289-1296, DOI [10.1038/s41592-019-0619-0](https://doi.org/10.1038/s41592-019-0619-0). - -### Shannon entropy of a partition - -- Definition: the Shannon entropy, in nats or bits, of the label distribution of a partition. A STATO term at this level would cover the difference reported by beam's card and any future entropy-based metric. -- Parent: information-theoretic measure, a kind of summary statistic. -- beam card: shannon_entropy_diff. -- Reference: Shannon 1948, A mathematical theory of communication, Bell System Technical Journal 27:379-423, DOI [10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x). - -### symmetric mean absolute percentage error - -- Definition: a forecasting-accuracy error equal to the mean over horizons of the absolute forecast error divided by the average of the absolute actual and forecast values, in percent. -- Parent: a measure of forecast error, a kind of percentage. -- beam card: smape. -- Reference: Makridakis, Spiliotis and Assimakopoulos 2020, The M4 Competition, International Journal of Forecasting 36:54-74, DOI [10.1016/j.ijforecast.2019.04.014](https://doi.org/10.1016/j.ijforecast.2019.04.014). - -### mean absolute scaled error - -- Definition: a forecasting-accuracy error equal to the mean absolute forecast error scaled by the in-sample mean absolute error of a naive one-step forecast. -- Parent: a measure of forecast error, a kind of summary statistic. -- beam card: mase. -- Reference: Hyndman and Koehler 2006, Another look at measures of forecast accuracy, International Journal of Forecasting 22:679-688, DOI [10.1016/j.ijforecast.2006.03.001](https://doi.org/10.1016/j.ijforecast.2006.03.001). diff --git a/docs/explanations/transitivity.md b/docs/explanations/transitivity.md deleted file mode 100644 index ed87dd7..0000000 --- a/docs/explanations/transitivity.md +++ /dev/null @@ -1,47 +0,0 @@ -# Transitivity or whether one ranking is consistent with the pairwise evidence - -Every aggregation in beam returns one ordering of the methods. That ordering is a fair summary only when the pairwise comparisons behind it agree with a single order. They do not always agree. Method A can outperform B on most of the datasets they share, B outperform C, and C outperform A. These three results form a cycle. When a cycle is present, no ordering of the methods agrees with all the pairwise majorities. - -[`beam.mcda.pairwise_transitivity`](../reference/pairwise_transitivity.qmd) checks for this. It reads the pairwise superiority report and asks whether the pairwise majority relation admits one consistent order. - -## Rationale - -For each pair of methods, [`pairwise_superiority`](../reference/pairwise_superiority.qmd) already counted how often each one outperforms the other across the datasets they share. `pairwise_transitivity` turns those counts into a single relation: method `i` outperforms method `j` when it does so more often than `j` outperforms `i` across their shared datasets. A pair that splits evenly, or that shares no decisive dataset, is left with no edge and recorded as tied. Because the counts already apply the region of practical equivalence, a difference inside the noise floor does not create an edge, so near-ties do not produce spurious cycles. - -From the relation the function reports four things. - -- The method preferred to every other method by pairwise majority, when one exists. Condorcet described this case in his 1785 essay on majority voting; it is the option preferred to every other one in a head-to-head majority vote. Such a method might not exist, and when it does the rest of the relation can still be inconsistent. -- The circular triads: sets of three methods whose edges form a cycle. Their count, out of all method triples, measures how far the relation is from transitive. -- Kendall and Babington Smith's (1940) coefficient of consistence, `1 - d / d_max`, where `d` is the circular-triad count and `d_max` is the maximum possible for this relation size. It runs from 1 (transitive) down to 0 (least consistent). It is defined only when every pair is decided, so it is reported as undefined when any pair is tied. -- Whether the relation is transitive, and the single order it implies when it is transitive and every pair is decided. - -## Usage - -```python -from beam.mcda import pairwise_superiority, pairwise_transitivity -from beam.cards import properties_for - -floor = properties_for(["ari"])[0].noise_floor -sup = pairwise_superiority(ari_by_dataset, "higher_is_better", rope=floor, - method_names=method_names) -trans = pairwise_transitivity(sup) -trans.is_transitive # whether one order is consistent with the pairwise majorities -trans.circular_triads # the cyclic triples, if any -trans.condorcet_choice # method preferred to all others, or None -trans.coefficient_of_consistence # Kendall's coefficient, or None when pairs are tied -``` - -## On beam's examples - -On the [Duo 2018](../../examples/duo2018/duo2018.qmd) Adjusted Rand Index (ARI) scores, with the ARI noise floor of 0.01 as the equivalence band, SC3 is preferred to every other method by pairwise majority, the same method the standing score, the marginal means and the [Bradley-Terry ranking](heterogeneity-bradley-terry.md) pick out. Even so the relation is not transitive, as there is a cycle and no single order agrees with every pairwise majority. The cycle is among methods that the critical-difference diagram and the pairwise effect sizes already place in overlapping groups, which is where a few datasets can swing the majority either way. Raising the equivalence band from zero to the noise floor cuts the cycles from five to one. - -The reading is that SC3 ranks first on Duo 2018, but the order among the methods below it is not settled by the pairwise evidence, and the noise floor is what separates an actual cycle from one made of differences too small to interpret. - -## Related methods - -[`critical_difference`](comparing-methods-across-datasets.md) tests whether the methods are separable across datasets on one metric. [`pairwise_superiority`](pairwise-superiority.md) reports, pair by pair, how often one outperforms the other. `pairwise_transitivity` reads that same pairwise relation and asks the prior question: does a single ranking even make sense here, or do the pairwise results contradict each other. [`aggregation_agreement`](aggregation-agreement.md) shows that the aggregation rules disagree; a cycle is one reason they can. - -## References - -- Condorcet, M. de. Essai sur l'application de l'analyse a la probabilite des decisions rendues a la pluralite des voix. Imprimerie Royale, Paris (1785). -- Kendall, M. G., Babington Smith, B.. On the method of paired comparisons. Biometrika 31, 324 (1940). DOI [10.1093/biomet/31.3-4.324](https://doi.org/10.1093/biomet/31.3-4.324). diff --git a/docs/explanations/weighting-schemes.md b/docs/explanations/weighting-schemes.md index af2a1d1..b1d3b68 100644 --- a/docs/explanations/weighting-schemes.md +++ b/docs/explanations/weighting-schemes.md @@ -15,7 +15,7 @@ Entropy weights, in the Shannon sense, give a metric more weight when its scores Standard deviation weights follow the same idea as entropy, that a metric that spreads the tools out should count more, but measure the spread directly with the sample standard deviation. They assume the columns are already on a common scale, which they are after normalization. It suits a plain, readable measure of spread. -CRITIC weights add a second idea on top of spread: conflict. CRITIC stands for CRiteria Importance Through Intercriteria Correlation. A metric increases its weight when its scores spread out and when it disagrees with the other metrics. If two metrics rank the tools the same way, they carry the same information, and counting both at full weight would double count that information. CRITIC measures the disagreement as one minus the correlation between columns and multiplies it by the standard deviation. It aims to remove the contribution of redundant metrics that track the same underlying property and, hence, do not contribute extra information. +CRITIC weights add a second idea on top of spread: conflict. CRITIC stands for CRiteria Importance Through Intercriteria Correlation. A metric increases its weight when its scores spread out and when it disagrees with the other metrics. If two metrics rank the tools the same way, they carry the same information, and counting both at full weight would double count that information. CRITIC measures the disagreement as one minus the correlation between columns and multiplies it by the standard deviation. This downweights redundant metrics that track the same underlying property. MEREC weights use a different approach. MEREC stands for Method based on the Removal Effects of Criteria. It scores how much the overall ranking changes when each metric is removed. A metric whose removal barely moves the scores provides little information and gets a small weight. A metric whose removal shifts the scores a lot gets a large weight. The aggregate it uses is logarithmic, so it needs strictly positive scores. A column normalized with plain min-max can contain a hard zero, which MEREC cannot take the logarithm of. Pair it with a normalization that stays away from zero, such as the logistic z-score strategy. MEREC fits when a weight should reflect each metric's marginal effect on the result rather than its raw spread. @@ -34,7 +34,7 @@ Equal weights are the default baseline. An objective scheme fits when the data s ## See also - [Rank sensitivity](rank-sensitivity.md) -- [Specification curve](specification-curve.md) +- [Specification curve](rank-sensitivity.md#the-specification-curve) ## References diff --git a/docs/how-to/plot-a-beam-run.md b/docs/how-to/plot-a-beam-run.md index e22e7f2..018a182 100644 --- a/docs/how-to/plot-a-beam-run.md +++ b/docs/how-to/plot-a-beam-run.md @@ -2,7 +2,7 @@ `beam.plot` returns matplotlib figures you can show in a notebook, drop into a Quarto vignette, or write to a file. Every function returns a `matplotlib.figure.Figure`, so you can adjust it before saving. The figure code is shared with the [HTML report](../reference/report.qmd), so a plot you draw here is the same one the report embeds. -This recipe shows the three groups of plots and how to save them. +This recipe covers the plot families and how to save them. ## Get a run @@ -58,7 +58,7 @@ plot.critical_difference(report) # canonical Friedman-Nemenyi clique-bar diag plot.critical_difference_band(report) # the shaded-band alternative ``` -The [critical-difference plot](../explanations/comparing-methods-across-datasets.md) is the canonical Demsar diagram: each tool at its average rank, with a blue bar joining each clique the Nemenyi test cannot separate. +The [critical-difference plot](../explanations/critical-difference.md) is the canonical Demsar diagram: each tool at its average rank, with a blue bar joining each clique the Nemenyi test cannot separate. ## Grids, heterogeneity, and building blocks diff --git a/docs/how-to/rerun-from-manifest.md b/docs/how-to/rerun-from-manifest.md index 264f087..a1f9e15 100644 --- a/docs/how-to/rerun-from-manifest.md +++ b/docs/how-to/rerun-from-manifest.md @@ -1,6 +1,6 @@ # Rerun a beam analysis from a manifest -Every [`beam.rank`](../reference/rank.qmd) run writes a manifest: a JSON record of the input file path and content hash, the metric card ids and versions, the [weighting scheme](../explanations/weighting-schemes.md), the [aggregation method](../explanations/aggregation-methods.md), the [SMAA](../reference/smaa.qmd) seed and sample count, the [per-metric normalization](../explanations/normalization-and-scales.md), the software fingerprint (python, numpy, scipy, pymcdm, pyyaml, jsonschema versions), and the host fingerprint. The manifest is what travels with a publication: a reader can run beam against it and reproduce the analysis byte for byte where possible. +Every [`beam.rank`](../reference/rank.qmd) run writes a manifest: a JSON record of the input file path and content hash, the metric card ids and versions, the [weighting scheme](../explanations/weighting-schemes.md), the [aggregation method](../explanations/aggregation-methods.md), the [SMAA](../reference/smaa.qmd) seed and sample count, the [per-metric normalization](../explanations/normalization-and-scales.md), the software fingerprint (python, numpy, scipy, pymcdm, pyyaml, jsonschema versions), and the host fingerprint. The manifest is the record to keep with a publication: a reader can run beam against it and reproduce the analysis byte for byte where possible. This recipe walks through rerunning a recorded analysis. diff --git a/docs/how-to/run-from-beam-yaml.md b/docs/how-to/run-from-beam-yaml.md index 1588ffc..f4497ae 100644 --- a/docs/how-to/run-from-beam-yaml.md +++ b/docs/how-to/run-from-beam-yaml.md @@ -53,13 +53,13 @@ sensitivity turns the sensitivity analysis on or off by its presence. Include th outputs is optional and each entry is optional. report writes the self-contained HTML report. manifest writes manifest.json, the run record. scores_normalized writes the normalized tool-by-metric matrix as a CSV. -## The reproducible artifact +## Keep the run reproducible -Keep beam.yaml and the written manifest.json together with the scores file. manifest.json records the input and its hash, the metrics, the parameters and the normalization that beam read from the cards. A reviewer with the same scores file reruns the ranking with one command. The manifest lets them confirm they got the same inputs and parameters you did. +Keep beam.yaml and the written manifest.json together with the scores file. The manifest records the input and its hash, the metrics, the parameters and the normalization beam read from the cards, so a rerun can be checked against it. See [Rerun from a manifest](rerun-from-manifest.md). ## Pinning a card version -Each metric entry may pin a [card](../explanations/cards-and-pipeline.qmd) version, which is the artifact a reviewer needs to reproduce a ranking exactly: +Each metric entry may pin a [card](../explanations/cards-and-pipeline.qmd) version, so a rerun uses the exact card the ranking was built on: ```yaml metrics: diff --git a/docs/how-to/run-heterogeneity-from-the-cli.md b/docs/how-to/run-heterogeneity-from-the-cli.md index 56cc8ee..67c146d 100644 --- a/docs/how-to/run-heterogeneity-from-the-cli.md +++ b/docs/how-to/run-heterogeneity-from-the-cli.md @@ -14,9 +14,9 @@ beam heterogeneity scores.csv --model mixed-effects --metric ari --out report.js ## The three models -[`mixed-effects`](../reference/mixed_effects.qmd) fits `score ~ method + (1 | dataset)` in lme4 and splits the [score variation into a method effect](../explanations/heterogeneity-mixed-effects.md), a between-dataset shift, and the residual. The report carries the variance components, the dataset ICC (the share of variation that is the dataset rather than the method), and the per-method marginal means. A high ICC means most of the spread is between datasets, so a single pooled ranking hides much of the structure. +[`mixed-effects`](../reference/mixed_effects.qmd) fits `score ~ method + (1 | dataset)` in lme4 and splits the [score variation into a method effect](../explanations/method-by-dataset-heterogeneity.md), a between-dataset shift, and the residual. The report carries the variance components, the dataset ICC (the share of variation that is the dataset rather than the method), and the per-method marginal means. A high ICC means most of the spread is between datasets, so a single pooled ranking hides much of the structure. -[`bradley-terry-tree`](../reference/bradley_terry_tree.qmd) fits a [Bradley-Terry tree](../explanations/heterogeneity-bradley-terry.md) in psychotree. It splits the datasets by their features so each leaf has its own method ranking, and flags the leaves where the pooled top method does not hold. It needs a dataset features file, passed with `--features`: +[`bradley-terry-tree`](../reference/bradley_terry_tree.qmd) fits a [Bradley-Terry tree](../explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) in psychotree. It splits the datasets by their features so each leaf has its own method ranking, and flags the leaves where the pooled top method does not hold. It needs a dataset features file, passed with `--features`: ``` beam heterogeneity scores.csv --model bradley-terry-tree --metric ari \ @@ -31,7 +31,7 @@ d1,531,9,real d2,3994,3,sim ``` -[`plackett-luce`](../reference/plackett_luce.qmd) fits a Plackett-Luce model in PlackettLuce on the per-dataset rankings of the methods, and reports the worth per method with quasi-standard errors. It is the [full-ranking generalization](../explanations/full-rankings-and-bounded-metrics.md) of the Bradley-Terry model and needs no features. +[`plackett-luce`](../reference/plackett_luce.qmd) fits a Plackett-Luce model in PlackettLuce on the per-dataset rankings of the methods, and reports the worth per method with quasi-standard errors. It is the [full-ranking generalization](../explanations/method-by-dataset-heterogeneity.md#plackett-luce-full-rankings) of the Bradley-Terry model and needs no features. ## The output diff --git a/docs/tutorials/quickstart.md b/docs/tutorials/quickstart.md index e0abfab..ae0e211 100644 --- a/docs/tutorials/quickstart.md +++ b/docs/tutorials/quickstart.md @@ -2,7 +2,7 @@ beam ranks the tools in a benchmark from a tool-by-metric table of scores. It reads the metric meaning (polarity, scale, normalization) from [metric cards](../explanations/cards-and-pipeline.qmd), [normalizes](../explanations/normalization-and-scales.md) the scores, [combines them into one composite score](../explanations/aggregation-methods.md) per tool, and reports a ranking with a sensitivity analysis. -This tutorial goes from a small CSV to an HTML report, first in Python and then from the command line. Copy each block as you go and run it. +This tutorial goes from a small CSV to an HTML report, first in Python and then from the command line. ## What you need @@ -26,7 +26,7 @@ sc3,0.74,0.71,310.5 monocle,0.69,0.66,88.0 ``` -## Step 2: rank and report in five lines +## Step 2: rank and report ```python import beam @@ -93,4 +93,4 @@ To run a whole pipeline from one declarative file (so a reviewer reruns it with For the concepts behind the steps above, see the explanations: [normalization and scales](../explanations/normalization-and-scales.md), [weighting schemes](../explanations/weighting-schemes.md) and [aggregation methods](../explanations/aggregation-methods.md). -When each metric is calculated on more than one dataset, `beam.rank` also reports whether the datasets agree on the order, with [dataset concordance](../explanations/dataset-concordance.md). The bundled vignettes ([Duo](../../examples/duo2018/duo2018.qmd), [M4](../../examples/m4/m4.qmd), [OpenProblems](../../examples/openproblems/openproblems.qmd), [transportation](../../examples/transportation/transportation.qmd)) show it on real and illustrative data. +When each metric is calculated on more than one dataset, `beam.rank` also reports whether the datasets agree on the order, with [dataset concordance](../explanations/dataset-concordance-and-discrimination.md). The bundled vignettes ([Duo](../../examples/duo2018/duo2018.qmd), [M4](../../examples/m4/m4.qmd), [OpenProblems](../../examples/openproblems/openproblems.qmd), [transportation](../../examples/transportation/transportation.qmd)) show it on real and illustrative data. diff --git a/examples/cell_type_annotation/cell_type_annotation.qmd b/examples/cell_type_annotation/cell_type_annotation.qmd index dc1f704..9d4b68b 100644 --- a/examples/cell_type_annotation/cell_type_annotation.qmd +++ b/examples/cell_type_annotation/cell_type_annotation.qmd @@ -181,7 +181,7 @@ top_per_combo = {head[int(np.argmin(row))] for row in ranks_grid} print("methods top-ranked in at least one of the 20 configurations:", ", ".join(sorted(top_per_combo))) ``` -[`aggregation_agreement`](../../docs/explanations/aggregation-agreement.md) measures that: the Kendall tau-b agreement of the rankings the five aggregations give at equal weights, and whether they agree on the top method. +[`aggregation_agreement`](../../docs/explanations/choice-agreement.md) measures that: the Kendall tau-b agreement of the rankings the five aggregations give at equal weights, and whether they agree on the top method. ```{python} from beam.mcda import aggregation_agreement @@ -236,9 +236,9 @@ nf = run.noise_floor print("noise-floor check active:", nf.active, "(no floor declared on the annotation cards)") ``` -## Are the methods separable on agreement +## Critical difference on agreement -A Demsar [critical-difference diagram](../../docs/explanations/comparing-methods-across-datasets.md) ranks the methods on each of the 36 datasets and asks, with a Friedman test and a Nemenyi post-hoc, whether the differences are larger than sampling noise. +A Demsar [critical-difference diagram](../../docs/explanations/critical-difference.md) ranks the methods on each of the 36 datasets and asks, with a Friedman test and a Nemenyi post-hoc, whether the differences are larger than sampling noise. ```{python} from beam.mcda import critical_difference @@ -258,7 +258,7 @@ for clique in cd.cliques: plot.critical_difference(cd) ``` -[`pairwise_superiority`](../../docs/explanations/pairwise-superiority.md) is the effect-size companion: for each method pair, the share of the 36 datasets where one method scores higher, the readable version of the significance the diagram gives. +[`pairwise_superiority`](../../docs/explanations/pairwise-method-comparison.md) is the effect-size companion: for each method pair, the share of the 36 datasets where one method scores higher, the readable version of the significance the diagram gives. ```{python} from beam.mcda import pairwise_superiority @@ -269,7 +269,7 @@ for i in np.argsort(-ps.standing): print(f" {head[i]:14s} {ps.standing[i]:.2f}") ``` -[`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) reads the same relation and reports whether one order is [consistent with the pairwise majorities](../../docs/explanations/transitivity.md). +[`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) reads the same relation and reports whether one order is [consistent with the pairwise majorities](../../docs/explanations/pairwise-method-comparison.md#transitivity). ```{python} from beam.mcda import pairwise_transitivity @@ -284,7 +284,7 @@ The matrix below orders the methods by how many others they outperform. A transi plot.pairwise_majority(trans) ``` -[`bayesian_sign_comparison`](../../docs/explanations/bayesian-comparison.md) reports, for each pair, the posterior probability that one method is practically better than another. With 36 datasets the posterior is sharper, so more pairs reach a decisive label than on the smaller benchmarks. +[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) reports, for each pair, the posterior probability that one method is practically better than another. With 36 datasets the posterior is sharper, so more pairs reach a decisive label than on the smaller benchmarks. ```{python} from beam.mcda import bayesian_sign_comparison @@ -295,9 +295,9 @@ print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pa plot.bayesian_comparison(bayes) ``` -## What moves the ranking: the data or the choice +## Rank sensitivity: data or analyst choice -A common question is whether the ranking would hold under a different analysis. The sections above each measured part of this. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) combines them: it ranks the methods on each dataset, under every weighting and aggregation, and splits each method's rank variance into the dataset, the weighting, the aggregation and their interactions. This is the dataset-weight question stated directly. +The sections above each measured part of whether the ranking holds under a different analysis. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) combines them: it ranks the methods on each dataset, under every weighting and aggregation, and splits each method's rank variance into the dataset, the weighting, the aggregation and their interactions. ```{python} from beam.mcda import rank_sensitivity @@ -327,7 +327,7 @@ plot.rank_sensitivity(rs) The dataset accounts for almost all of the rank variance. The weighting and the aggregation account for less than one percent combined. Leave-one-dataset-out reports whether the pooled order changes when one dataset is dropped. -The shares above are pooled over the methods. The per-method version splits the same variance one method at a time. It separates a method whose rank depends on the dataset from one that depends on the weighting or the aggregation. The span next to each bar is the difference between the method's best and worst rank. +The figures so far pool the shares over all methods. The per-method breakdown attributes each method's rank spread on its own, telling apart a method that moves with the dataset from one that moves with the analyst's choice. The bar span is the method's best minus worst rank. ```{python} plot.rank_sensitivity_by_tool(rs) @@ -339,7 +339,7 @@ print(f"top method rank stability under leave-one-dataset-out: {lodo.rank_stabil print(f"most influential dataset: {block_datasets[lodo.most_influential_dataset]}") ``` -[`specification_curve`](../../docs/explanations/specification-curve.md) lists the rankings the grid produces. The choice-only multiverse (weighting by aggregation on the pooled matrix) shows whether GPT-4 holds the top across the modeling choices; the full grid adds the dataset. +[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) lists the rankings the grid produces. The choice-only multiverse (weighting by aggregation on the pooled matrix) shows whether GPT-4 holds the top across the modeling choices; the full grid adds the dataset. ```{python} from beam.mcda import specification_curve @@ -363,9 +363,9 @@ print(f"choices only ({pooled.n_specifications} combinations): " f"{pdom} first in {pooled.most_frequent_top_fraction * 100:.0f}%") ``` -## Dataset concordance: do the datasets agree on the order? +## Dataset concordance -The pooled ranking averages over the datasets in the complete block. [`dataset_concordance`](../../docs/explanations/dataset-concordance.md) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not. It needs no replicates and assumes nothing about the datasets being interchangeable. +The pooled order sits on top of 36 per-dataset orders. [`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) recovers each of those, ranking the methods within a dataset and comparing every pair of orderings with Kendall tau-b. A high mean says the datasets broadly agree with the pooled order; a low mean says they do not. It assumes nothing about the datasets being interchangeable and needs no replicates. ```{python} conc = run.dataset_concordance @@ -384,7 +384,7 @@ for cell in conc.notable_cells[:5]: plot.dataset_concordance(run) ``` -The companion view marks where each method places higher or lower than its own typical rank, so a reader sees which methods carry the dataset disagreement rather than which method ranks best overall. +The companion view marks where each method places higher or lower than its own typical rank, showing which methods carry the dataset disagreement rather than which ranks best overall. ```{python} plot.dataset_struggle(run) @@ -408,7 +408,7 @@ print("top method after unblinding:", restored.top_tool) print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12]) ``` -## Dataset weight, made concrete: equal versus cell-weighted pooling +## Equal versus cell-weighted pooling beam weights each dataset equally by default. The paper's per-cell-type accuracy weights a dataset by how many cell types it has. The dataset accounts for almost all of the variance, so the dataset weight changes the magnitudes. Here it does not change the top rank. @@ -422,9 +422,9 @@ for i in np.argsort(-equal_pool): print(f"{head[i]:14s} {equal_pool[i]:.3f} {weighted_pool[i]:.3f}") ``` -## Heterogeneity: how much of the agreement variance is the dataset +## Mixed-effects on agreement -`rank_sensitivity` decomposes the ranks. A [mixed-effects model](../../docs/explanations/heterogeneity-mixed-effects.md) decomposes the scores: it fits `agreement ~ method + (1 | dataset)` in R's lme4, separating the stable method effect from the dataset being uniformly easy or hard. The fit needs the R toolchain, so this section runs only when it is available. +`rank_sensitivity` decomposes the ranks. A [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) decomposes the scores: it fits `agreement ~ method + (1 | dataset)` in R's lme4, separating the stable method effect from the dataset being uniformly easy or hard. The fit needs the R toolchain, so this section runs only when it is available. ```{python} from beam.heterogeneity import mixed_effects_from_matrix, r_available @@ -443,9 +443,9 @@ else: print("Provision it with envs/heterogeneity.yml (conda or mamba).") ``` -## Heterogeneity: does cancer tissue reverse the ranking +## Bradley-Terry tree on tissue features -A practical question is whether the ranking reverses on data like the reader's. A [Bradley-Terry tree](../../docs/explanations/heterogeneity-bradley-terry.md) turns each dataset into pairwise method comparisons and splits the datasets by their features, so each leaf gets its own ranking and a parameter-stability test decides where a split is real. The candidate features are the source, tissue, species, sample type (normal or cancer) and the number of cell types. One thing to check is whether the gap between GPT-4 and the other methods shrinks on cancer tissue, where canonical markers are less reliable. The fit needs the R toolchain. +A further question is whether the ranking reverses on particular kinds of data. A [Bradley-Terry tree](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) turns each dataset into pairwise method comparisons and splits the datasets by their features, so each leaf gets its own ranking and a parameter-stability test decides where a split is real. The candidate features are the source, tissue, species, sample type (normal or cancer) and the number of cell types. One thing to check is whether the gap between GPT-4 and the other methods shrinks on cancer tissue, where canonical markers are less reliable. The fit needs the R toolchain. ```{python} from beam.heterogeneity import bradley_terry_tree, bttree_available @@ -470,7 +470,7 @@ else: print("Provision it with envs/heterogeneity.yml (conda or mamba).") ``` -[`plackett_luce`](../../docs/explanations/full-rankings-and-bounded-metrics.md) fits the same preference idea as a single full-ranking model over the methods, a check on the Bradley-Terry global ranking. +[`plackett_luce`](../../docs/explanations/method-by-dataset-heterogeneity.md#plackett-luce-full-rankings) fits the same preference idea as a single full-ranking model over the methods, a check on the Bradley-Terry global ranking. ```{python} from beam.heterogeneity import plackett_luce, plackett_luce_available @@ -484,7 +484,7 @@ else: print("Provision it with envs/heterogeneity.yml (conda or mamba).") ``` -## Is the manual annotation reliable +## Reliability of the manual annotation The agreement metric is defined against the manual annotation, so beam uses it as the reference and cannot check it directly. One indirect check treats the five annotators as independent raters and looks for cell types where they agree on a broad lineage that differs from the manual label. Those cell types are candidates for a fine-grained mislabel or an ambiguous reference. Item response theory is the model-based version of this check, separating a hard cell type from a mislabeled one through item difficulty and discrimination; that module is not yet implemented, so this section uses the broad-lineage consensus the bundled table records. @@ -522,7 +522,7 @@ Most of these are a granularity difference rather than an error: the manual labe ## Partial coverage: the full 54-dataset set -The complete block dropped the 18 datasets where SingleR and ScType were not run. beam can also rank on the full set without completing the matrix. [Available-case SAW](../../docs/explanations/missing-data.md) scores each method on the datasets where it was measured, with the weights renormalized over its support; the [Skillings-Mack test](../../docs/explanations/skillings-mack.md) is the coverage-aware Friedman test for a matrix with gaps. +The complete block dropped the 18 datasets where SingleR and ScType were not run. beam can also rank on the full set without completing the matrix. [Available-case SAW](../../docs/explanations/missing-data.md) scores each method on the datasets where it was measured, with the weights renormalized over its support; the [Skillings-Mack test](../../docs/explanations/critical-difference.md#skillings-mack-for-incomplete-blocks) is the coverage-aware Friedman test for a matrix with gaps. ```{python} from beam.mcda import skillings_mack @@ -562,7 +562,7 @@ beam.report(run, "cell_type_annotation_report.html", ground_truth_tool="GPT-4") print("wrote cell_type_annotation_report.html") ``` -## Cross-benchmark: do the LLM annotation benchmarks agree? +## Cross-benchmark: LLM annotation benchmarks Three more recent benchmarks score LLMs at cell type annotation: DeepCellSeek (Briefings in Bioinformatics 2025, [10.1093/bib/bbaf677](https://doi.org/10.1093/bib/bbaf677)), AnnDictionary (Nature Communications 2025, [10.1038/s41467-025-64511-x](https://doi.org/10.1038/s41467-025-64511-x)) and Single-Cell Omics Arena (SOAR, [arXiv:2412.02915](https://arxiv.org/abs/2412.02915)). They use different models, datasets and metrics, so they cannot be pooled on a shared score. The common currency is the within-benchmark rank of the methods each benchmark has in common with the others. The three classical tools (CellMarker2.0, SingleR, ScType) connect GPTCelltype, DeepCellSeek and SOAR; GPT-4o connects DeepCellSeek, AnnDictionary and SOAR. The provenance and license of each bundled table are in `src/beam/data/`; DeepCellSeek and SOAR scores come from a pinned source, and the SOAR and AnnDictionary folders carry a note on their license status. @@ -700,4 +700,4 @@ On the 36 datasets where all five methods were run, GPT-4 ranks first on the mea On the critical-difference diagram GPT-4 separates from the other four methods, which sit in overlapping cliques and are not separable on agreement at 36 datasets. The rank-sensitivity decomposition puts about 96 percent of the rank variance on the dataset and less than one percent on the weighting and the aggregation combined. -A cross-benchmark version of this analysis, checking whether several LLM annotation benchmarks agree on the ranking, is not possible yet. The recent ones (DeepCellSeek, Single-Cell Omics Arena, AnnDictionary) publish their code and summary figures but not a reusable per-method per-dataset score table, so their numbers cannot be re-ranked under one rule. This is the same reusability gap a shared metric layer addresses, and the reason beam reduces each benchmark to an explicit score tensor. +A cross-benchmark version of this analysis, checking whether several LLM annotation benchmarks agree on the ranking, is not possible yet. The recent ones (DeepCellSeek, Single-Cell Omics Arena, AnnDictionary) publish their code and summary figures but not a reusable per-method per-dataset score table, so their numbers cannot be re-ranked under one rule. diff --git a/examples/cross_benchmark/cross_benchmark.qmd b/examples/cross_benchmark/cross_benchmark.qmd index 64d4562..1af649f 100644 --- a/examples/cross_benchmark/cross_benchmark.qmd +++ b/examples/cross_benchmark/cross_benchmark.qmd @@ -1,5 +1,5 @@ --- -title: "Cross-benchmark meta-analysis: do single-cell integration benchmarks disagree about the methods, the data, or the benchmarker?" +title: "Cross-benchmark meta-analysis" author: "Izaskun Mallona" date: today format: @@ -135,7 +135,7 @@ for b in BENCH: display(fig) ``` -## A fifth source with a different lens: BatchBench +## A fifth source: BatchBench BatchBench scores the methods with two entropy metrics rather than the scIB family: batch entropy, where a higher value means the batches are better mixed, and cell-type entropy, where a higher value means the cell types are more blurred, so a lower value is better for biology. Ruben Chazarra-Gil provided the scores by personal communication. The two metrics show the batch-mixing against biology tradeoff directly. @@ -163,9 +163,9 @@ plt.show() A method low and to the right mixes batches well while keeping the cell types separate. harmony sits high and to the right: it mixes batches well but blurs the cell types more than the others. A benchmark that weights biology equally then ranks it below fastMNN, which is what BatchBench does, and which is why the fifth source increases the disagreement measured below. -## How much of the variance is the benchmark +## Benchmark share of the variance -A cross-benchmark [variance decomposition](../../docs/explanations/heterogeneity-mixed-effects.md) puts a number on the split. The model is `score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark)`, where the method-by-benchmark component is the disagreement attributable to the benchmark rather than the method. The chunk fits it twice, with the four scIB-family sources and again with BatchBench added, so the effect of the fifth source is visible. The fit needs R's lme4, so the chunk runs only when it is available. +A cross-benchmark [variance decomposition](../../docs/explanations/attribution-synthesis.md) puts a number on the split. The model is `score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark)`, where the method-by-benchmark component is the disagreement attributable to the benchmark rather than the method. The chunk fits it twice, with the four scIB-family sources and again with BatchBench added, so the effect of the fifth source is visible. The fit needs R's lme4, so the chunk runs only when it is available. ```{python} from beam.heterogeneity import r_available, source_variance_decomposition @@ -211,7 +211,7 @@ else: print("Provision it with envs/heterogeneity.yml.") ``` -With BatchBench in the pool, harmony no longer ranks first on its own: it ties with liger at the top, fastMNN follows, and combat ranks last. The I-squared is high and the inconsistency Q is significant: the direct and the indirect evidence for the same method pairs do not line up, which is the network-model reading of the same finding the variance decomposition reports as a large method-by-benchmark share. The pooled ranking is an average over benchmarks that disagree, and the inconsistency statistic quantifies it. +With BatchBench in the pool, harmony no longer ranks first on its own: it ties with liger at the top, fastMNN follows, and combat ranks last. The I-squared is high and the inconsistency Q is significant: the direct and the indirect evidence for the same method pairs do not line up, which is the network-model reading of the same finding the variance decomposition reports as a large method-by-benchmark share. ## Same data, two pipelines: the human pancreas @@ -253,7 +253,7 @@ plt.show() ## Per-benchmark fragility: which top method is closest to flipping -Within each benchmark, the smallest single-weight change that swaps the top-ranked method tells the reader how much the recommendation depends on the analyst's weighting. beam uses the Triantaphyllou-Sanchez closed-form weight delta on the mean-rank matrix; [SAW](../../docs/explanations/aggregation-methods.md) with equal weights is the baseline so the perturbation is interpretable as "how much weight off one metric is needed to flip the top". +Within each benchmark, the smallest single-weight change that swaps the top-ranked method measures how much the recommendation depends on the analyst's weighting. beam uses the Triantaphyllou-Sanchez closed-form weight delta on the mean-rank matrix; [SAW](../../docs/explanations/aggregation-methods.md) with equal weights is the baseline so the perturbation is interpretable as "how much weight off one metric is needed to flip the top". ```{python} from beam.mcda import smallest_weight_perturbation @@ -280,7 +280,7 @@ OpenProblems is the most fragile of the three: a 0.11 absolute change to the ASW ## Specification curve per benchmark -The fragility delta is one number per benchmark. The specification curve lists the whole multiverse of analyst choices: every weighting by every aggregation on the mean-rank matrix. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) runs the grid and [`specification_curve`](../../docs/explanations/specification-curve.md) reports how often the top method holds. The benchmarks differ here too: on Tran and scIB the top method holds across all 20 combinations, while on OpenProblems it does not. +The fragility delta is one number per benchmark. The specification curve lists the whole multiverse of analyst choices: every weighting by every aggregation on the mean-rank matrix. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) runs the grid and [`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) reports how often the top method holds. The benchmarks differ here too: on Tran and scIB the top method holds across all 20 combinations, while on OpenProblems it does not. ```{python} from beam.mcda import rank_sensitivity, specification_curve @@ -327,11 +327,11 @@ print("order, unblinded: ", order_unblinded) print("same order after unblinding:", order_named == order_unblinded) ``` -## A sixth source with disjoint methods: dataset discrimination +## A sixth source: dataset discrimination -Every analysis above compares method orders, so it needs shared methods. Shen, He and Guan (2026, [10.1371/journal.pcbi.1014008](https://doi.org/10.1371/journal.pcbi.1014008)) score ten methods, five semi-supervised (scANVI, scGEN, STACAS, scDREAMER, ItClust) and five unsupervised (Seurat, scVI, Harmony, scanorama, scCRAFT), on six datasets, each under annotation scenarios that degrade label supervision from full to none. Only harmony and scanorama overlap the classical five, too few to rank-harmonize. A property defined per dataset is still available: how much each dataset separates the methods it scores. Any benchmark can report it for every dataset, whatever methods it ran. See the data README for provenance and the [dataset-discrimination explanation](../../docs/explanations/dataset-discrimination.md). +Every analysis above compares method orders, so it needs shared methods. Shen, He and Guan (2026, [10.1371/journal.pcbi.1014008](https://doi.org/10.1371/journal.pcbi.1014008)) score ten methods, five semi-supervised (scANVI, scGEN, STACAS, scDREAMER, ItClust) and five unsupervised (Seurat, scVI, Harmony, scanorama, scCRAFT), on six datasets, each under annotation scenarios that degrade label supervision from full to none. Only harmony and scanorama overlap the classical five, too few to rank-harmonize. A property defined per dataset is still available: how much each dataset separates the methods it scores. Any benchmark can report it for every dataset, whatever methods it ran. See the data README for provenance and the [dataset discrimination explanation](../../docs/explanations/dataset-concordance-and-discrimination.md#dataset-discrimination). -[`dataset_discrimination`](../../docs/reference/dataset_discrimination.qmd) computes it from the scores. beam treats each (dataset, scenario) pair as one unit and reports two values per unit: the spread of the pooled method scores, the effect size, and Kendall's W over the unit's method-by-metric matrix, the consistency of the order across metrics. It complements [`dataset_concordance`](../../docs/explanations/dataset-concordance.md): concordance asks whether datasets agree on the order, discrimination asks whether a dataset separates the methods at all. +[`dataset_discrimination`](../../docs/reference/dataset_discrimination.qmd) computes it from the scores. beam treats each (dataset, scenario) pair as one unit and reports two values per unit: the spread of the pooled method scores, the effect size, and Kendall's W over the unit's method-by-metric matrix, the consistency of the order across metrics. It complements [`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md): concordance asks whether datasets agree on the order, discrimination asks whether a dataset separates the methods at all. ```{python} from beam.datasets import load_openproblems, load_semisupervised_integration @@ -348,7 +348,7 @@ display(plot.dataset_discrimination(disc, top=15)) The strongest discriminators are the corrupted-label scenarios (`randomly_wrong_70` and the heavily missing ones): wrong labels separate the methods that use them from the unsupervised ones. The weakest are the mild perturbations, where the methods score alike and the unit cannot rank them. On the high-spread units the colour is dark, so the metrics agree on the order there. -### Are hard datasets as hard for deep learning as for classical methods? +### Difficulty concordance: classical versus deep learning Splitting the methods into two categories, classical and deep learning, separates two kinds of hardness. [`difficulty_concordance`](../../docs/reference/difficulty_concordance.qmd) correlates the per-category difficulty profiles across the datasets. A high concordance places the hardness in the data; a low one places it in the kind of method. @@ -389,7 +389,7 @@ print("methods now shared with Shen:", shared) display(plot.difficulty_concordance(scib_conc, title="scIB: do DL and classical find the same datasets hard?")) ``` -On scIB the two categories agree (about 0.90), as on OpenProblems and unlike Shen, even though the datasets and metrics are held fixed and only the method changes. The low Shen concordance is therefore a property of annotation degradation, not of being a deep-learning method. Extracting these methods also lifts the deep-learning overlap: scVI, scANVI and scGen now join harmony and scanorama as methods shared with Shen, where before no deep-learning method was carried from scIB at all. +On scIB the two categories agree (about 0.90), as on OpenProblems and unlike Shen, even though the datasets and metrics are held fixed and only the method changes. The low Shen concordance is a property of annotation degradation, not of being a deep-learning method. Extracting these methods also lifts the deep-learning overlap: scVI, scANVI and scGen now join harmony and scanorama as methods shared with Shen, where before no deep-learning method was carried from scIB at all. ## Two levels of agreement diff --git a/examples/duo2018/duo2018.qmd b/examples/duo2018/duo2018.qmd index 071c6c5..4658645 100644 --- a/examples/duo2018/duo2018.qmd +++ b/examples/duo2018/duo2018.qmd @@ -17,11 +17,11 @@ format: This dataset also has a [generated standard report](../../docs/reports/duo2018.html), the self-contained HTML `beam.report` emits for any run. This vignette is the narrative walkthrough behind it. -This vignette re-analyses the fourteen single-cell clustering methods benchmarked in Duo, Robinson and Soneson (2018) with the beam metric registry and the MCDA decision module. It loads the published method by dataset by metric tensor, pulls the [metric cards](../../docs/explanations/cards-and-pipeline.qmd), and pools each metric across the twelve datasets using the rule its card recommends. It then runs the MCDA pipeline and checks how stable the ranking is under different weightings, aggregation methods, and weight perturbations. It closes with a Demsar critical-difference diagram that asks whether the methods are separable on ARI at all. +This vignette re-analyses the fourteen single-cell clustering methods benchmarked in Duo, Robinson and Soneson (2018) with the beam metric registry and the [MCDA](../../docs/explanations/aggregation-methods.md) decision module. It loads the published method by dataset by metric tensor, pulls the [metric cards](../../docs/explanations/cards-and-pipeline.qmd), and pools each metric across the twelve datasets using the rule its card recommends. It then runs the MCDA pipeline and checks how stable the ranking is under different weightings, aggregation methods, and weight perturbations. It closes with a Demsar critical-difference diagram that asks whether the methods are separable on ARI at all. ## Set-up -The five [aggregation methods](../../docs/explanations/aggregation-methods.md) used below (SAW, TOPSIS, VIKOR, PROMETHEE II, [COMET](../../docs/explanations/comet.md)) are wrapped from pymcdm. beam normalizes by metric card, then calls pymcdm on the normalized matrix and keeps the higher-is-better convention. The [weighting schemes](../../docs/explanations/weighting-schemes.md) are beam's own, since pymcdm's reject the zeros beam's normalization produces. +The five [aggregation methods](../../docs/explanations/aggregation-methods.md) used below (SAW, TOPSIS, VIKOR, PROMETHEE II, [COMET](../../docs/explanations/aggregation-methods.md#comet)) are wrapped from pymcdm. beam normalizes by metric card, then calls pymcdm on the normalized matrix and keeps the higher-is-better convention. The [weighting schemes](../../docs/explanations/weighting-schemes.md) are beam's own, since pymcdm's reject the zeros beam's normalization produces. ```{python} %matplotlib inline @@ -173,7 +173,7 @@ print(" " + ", ".join(sorted(top_per_combo))) ## Critical-difference diagram on ARI -The MCDA composite gives one ranking, but it does not say whether the methods are separable. The Demsar (2006) [Friedman test](../../docs/explanations/comparing-methods-across-datasets.md) ranks the methods on each dataset and asks whether the average ranks differ more than chance. The Nemenyi post-hoc gives the critical difference: two methods whose average ranks differ by less than it are not distinguishable from this data. +The MCDA composite gives one ranking, but it does not say whether the methods are separable. The Demsar (2006) [Friedman test](../../docs/explanations/critical-difference.md) ranks the methods on each dataset and asks whether the average ranks differ more than chance. The Nemenyi post-hoc gives the critical difference: two methods whose average ranks differ by less than it are not distinguishable from this data. This runs on ARI alone, on the block of methods observed on all twelve datasets. SAFE and ascend have missing ARI cells, so they are dropped here to keep the block complete; the remaining twelve methods are observed on all twelve datasets. @@ -206,7 +206,7 @@ plot.critical_difference(cd) ## Pairwise superiority on ARI -The critical-difference diagram says which methods are separable, but not by how much or how often one method outperforms another. [`pairwise_superiority`](../../docs/reference/pairwise_superiority.qmd) [compares the methods two at a time](../../docs/explanations/pairwise-superiority.md) across the datasets, with the ARI noise floor (0.01) as the equivalence band: a method counts as outperforming another on a dataset only when the difference clears that floor. The probability of superiority is the fraction of datasets on which one method outperforms the other, an effect size that does not depend on the pool. +The critical-difference diagram says which methods are separable, but not by how much or how often one method outperforms another. [`pairwise_superiority`](../../docs/reference/pairwise_superiority.qmd) [compares the methods two at a time](../../docs/explanations/pairwise-method-comparison.md) across the datasets, with the ARI noise floor (0.01) as the equivalence band: a method counts as outperforming another on a dataset only when the difference clears that floor. The probability of superiority is the fraction of datasets on which one method outperforms the other, an effect size that does not depend on the pool. ```{python} from beam.mcda import pairwise_superiority @@ -218,11 +218,11 @@ print(f"highest standing: {top_method} ({sup.standing[sup.order[0]]:.2f})") print(f"method pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}") ``` -Most pairs are not distinguishable: across the twelve datasets the ARI differences are often inside the 0.01 floor, so neither method outperforms the other decisively. This is the critical-difference reading stated as an effect size, and it agrees with it. Beyond the method with the highest standing, the rest are largely interchangeable on ARI at this dataset count. +Most pairs are not distinguishable: across the twelve datasets the ARI differences are often inside the 0.01 floor, so neither method outperforms the other decisively. This is the critical-difference reading stated as an effect size. Beyond the method with the highest standing, the rest are largely interchangeable on ARI at this dataset count. ## Pairwise transitivity on ARI -The ranking is one order of the methods, but the pairwise majorities behind it need not agree with any single order. [`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) reads the same pairwise relation and asks whether one order is [consistent with it](../../docs/explanations/transitivity.md): it reports the method preferred to every other one, the circular triads (a method outperforms a second, the second a third, the third the first), and whether the relation is transitive. +The ranking is one order of the methods, but the pairwise majorities behind it need not agree with any single order. [`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) reads the same pairwise relation and asks whether one order is [consistent with it](../../docs/explanations/pairwise-method-comparison.md#transitivity): it reports the method preferred to every other one, the circular triads (a method outperforms a second, the second a third, the third the first), and whether the relation is transitive. ```{python} from beam.mcda import pairwise_transitivity @@ -241,7 +241,7 @@ plot.pairwise_majority(trans) ## Bayesian comparison on ARI -The tests above report a p-value: the chance of the observed split if two methods scored the same. A reader choosing a method wants the probability that one method scores higher, given the data. [`bayesian_sign_comparison`](../../docs/explanations/bayesian-comparison.md) reads the same pairwise relation and reports, for each pair, the posterior probability that one method is practically better, that the two are practically equivalent within the floor, and that the other is practically better. +The tests above report a p-value: the chance of the observed split if two methods scored the same. [`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) reads the same pairwise relation and reports, for each pair, the posterior probability that one method is practically better, that the two are practically equivalent within the floor, and that the other is practically better. ```{python} from beam.mcda import bayesian_sign_comparison @@ -365,9 +365,9 @@ beam.report(run, "duo2018_report.html", ground_truth_tool="Seurat") print("wrote duo2018_report.html") ``` -## Dataset concordance: do the datasets agree on the order? +## Dataset concordance -The pooled ranking averages over the twelve datasets. [`dataset_concordance`](../../docs/explanations/dataset-concordance.md) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not, and the per-dataset orderings diverge. The diagnostic needs no replicates and assumes nothing about the datasets being interchangeable, so it measures the heterogeneity rather than asking how many datasets would be enough. +The pooled ranking averages over the twelve datasets. [`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not, and the per-dataset orderings diverge. The diagnostic needs no replicates and assumes nothing about the datasets being interchangeable, so it measures the heterogeneity rather than asking how many datasets would be enough. ```{python} conc = run.dataset_concordance @@ -395,9 +395,9 @@ The companion view locates where the disagreement comes from. Each cell is a met plot.dataset_struggle(run) ``` -## Specification curve: every choice at once +## Specification curve -[`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) runs every combination of the weighting, the aggregation and the dataset, and splits the rank variance between them. [`specification_curve`](../../docs/explanations/specification-curve.md) reads the same grid and lists the rankings, then reports how often the top method holds. The choice-only grid (weighting by aggregation, on the pooled matrix) and the choice-plus-data grid (the dataset joins it) answer different questions: the first asks whether the recommendation survives the modeling choices, the second whether it survives a change of dataset. +[`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) runs every combination of the weighting, the aggregation and the dataset, and splits the rank variance between them. [`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) reads the same grid and lists the rankings, then reports how often the top method holds. The choice-only grid (weighting by aggregation, on the pooled matrix) and the choice-plus-data grid (the dataset joins it) answer different questions: the first asks whether the recommendation survives the modeling choices, the second whether it survives a change of dataset. ```{python} from beam.mcda import rank_sensitivity, specification_curve @@ -442,7 +442,7 @@ The pooled grid keeps Seurat first in every weighting-by-aggregation combination plot.specification_curve(curve) ``` -## Blind analysis: fix the pipeline before seeing the names +## Blind analysis A benchmarker who can see which method is which while choosing the weighting and the metric set has room to tune the choices toward a method they expect to rank first. [`beam.blind`](../../docs/explanations/analysis-blinding.md) masks the method names and shuffles the rows under a seed, so the pipeline is fixed on opaque labels; `beam.unblind` restores the names afterward. The ranking does not change, because beam ranks on the score values and unblinding only renames the rows. What blinding adds is a record: the seal fingerprint goes into the run manifest. @@ -463,7 +463,7 @@ print("top method after unblinding:", restored.top_tool) print("blinding fingerprint in manifest:", blind_run.manifest["blinding"]["seal_sha256"][:12]) ``` -## Heterogeneity: where does the ARI variance live +## Mixed-effects variance decomposition Leave-one-dataset-out asks whether the pooled ranking depends on any single dataset. A mixed-effects model asks a different question: how much of the score variation is a stable method effect and how much is the method-by-dataset interaction that a single ranking does not show. [`beam.heterogeneity.mixed_effects`](../../docs/reference/mixed_effects.qmd) fits `score ~ method + (1 | dataset)` on one metric in R's lme4, with the method as a fixed effect and the dataset as a random intercept that absorbs how easy or hard each dataset is for every method alike. The fit needs the R toolchain, so the section below runs only when it is available. @@ -495,9 +495,9 @@ if me is not None: display(plot.model_effects(me, xlabel="ARI marginal mean over datasets (higher ranks first)")) ``` -The dataset intercept takes about 0.71 of the ARI variance, so most of the spread is datasets being uniformly easy or hard rather than methods reordering. SC3 and Seurat have the highest marginal means and are not separable from each other given the standard errors, which matches the critical-difference reading above. The residual share, the upper bound on the method-by-dataset interaction at one run per cell, concentrates on the two lowest-scoring methods (RaceID2 and FlowSOM) on a few simulated and four-group datasets, not on a reshuffling of the higher-scoring methods. See [the mixed-effects explanation](../../docs/explanations/heterogeneity-mixed-effects.md). +The dataset intercept takes about 0.71 of the ARI variance, so most of the spread is datasets being uniformly easy or hard rather than methods reordering. SC3 and Seurat have the highest marginal means and are not separable from each other given the standard errors, which matches the critical-difference reading above. The residual share, the upper bound on the method-by-dataset interaction at one run per cell, concentrates on the two lowest-scoring methods (RaceID2 and FlowSOM) on a few simulated and four-group datasets, not on a reshuffling of the higher-scoring methods. See [the mixed-effects explanation](../../docs/explanations/method-by-dataset-heterogeneity.md). -## Heterogeneity: does a dataset feature reverse the ranking +## Bradley-Terry tree on dataset features The variance decomposition says how much interaction there is. The Bradley-Terry tree asks which dataset feature drives it. For one metric, each dataset becomes a set of pairwise method comparisons (the higher ARI is the preferred method), and `psychotree::bttree` splits the datasets by their features, so each leaf gets its own Bradley-Terry ranking and a parameter-stability test decides where a split is real. The dataset features (number of cells, number of true clusters, real vs simulated, family) ship next to the benchmark. The fit needs the R toolchain, so the section runs only when psychotree is available. @@ -526,9 +526,9 @@ else: print("Provision it with envs/heterogeneity.yml (conda or mamba).") ``` -On these 12 datasets the parameter-stability test finds no split, so the tree reduces to a single Bradley-Terry ranking led by SC3, the same order the net pairwise preferences and the marginal means give. A dozen datasets is too few for the test to separate a feature-dependent regime from sampling noise, which is also why the critical-difference diagram and the variance decomposition both report a stable global ranking here. The tree is more informative on a benchmark with many datasets carrying real feature variation; the [OpenProblems spatially_variable_genes vignette](../openproblems/openproblems.qmd) is where a split appears. See [the Bradley-Terry explanation](../../docs/explanations/heterogeneity-bradley-terry.md). +On these 12 datasets the parameter-stability test finds no split, so the tree reduces to a single Bradley-Terry ranking led by SC3, the same order the net pairwise preferences and the marginal means give. A dozen datasets is too few for the test to separate a feature-dependent regime from sampling noise, which is also why the critical-difference diagram and the variance decomposition both report a stable global ranking here. The tree is more informative on a benchmark with many datasets carrying real feature variation; the [OpenProblems spatially_variable_genes vignette](../openproblems/openproblems.qmd) is where a split appears. See [the Bradley-Terry explanation](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees). -## What moves the ranking: the data or the choice +## Rank sensitivity: data or analyst choice The sections above measure three things separately: leave-one-dataset-out (does the pooled order depend on one dataset), the weighting-by-aggregation grid (does the order depend on the modeling choice), and the mixed-effects ICC (how much score variance is the dataset). `rank_sensitivity` puts all three on one scale. It ranks the methods on each dataset, under every weighting and aggregation, and splits each method's rank variance into the dataset, the weighting, the aggregation and their interactions. @@ -558,7 +558,7 @@ print(f" most influential factor: {rs.most_influential_factor}") plot.rank_sensitivity(rs) ``` -The dataset carries the most, close to the 0.71 the mixed-effects ICC put on the ARI variance, reached here by an exact factorial decomposition of the ranks rather than a random-effects model of the scores. Two methods, the same reading. This is not in tension with the stable leave-one-dataset-out result above: that asks whether the pooled order survives dropping one dataset, and it does, because pooling averages over the per-dataset disagreement. The dataset share measures that disagreement directly, by ranking on each dataset on its own. The pooled order is stable; the per-dataset orders are not, and a single pooled ranking does not show that disagreement. The weighting filled the partial cells with the lowest score (`missing="worst"`), a stated choice the share is read against. +The dataset carries the most, close to the 0.71 the mixed-effects ICC put on the ARI variance, reached here by an exact factorial decomposition of the ranks rather than a random-effects model of the scores. This is not in tension with the stable leave-one-dataset-out result above: that asks whether the pooled order survives dropping one dataset, and it does, because pooling averages over the per-dataset disagreement. The dataset share measures that disagreement directly, by ranking on each dataset on its own. The weighting filled the partial cells with the lowest score (`missing="worst"`), a stated choice the share is read against. The shares above are pooled over the methods. The per-method version splits the same variance one method at a time. It separates a method whose rank depends on the dataset from one that depends on the weighting or the aggregation. The span next to each bar is the difference between the method's best and worst rank, read alongside its shares. diff --git a/examples/m4/m4.qmd b/examples/m4/m4.qmd index 455bb98..fed9099 100644 --- a/examples/m4/m4.qmd +++ b/examples/m4/m4.qmd @@ -19,7 +19,7 @@ This dataset also has a [generated standard report](../../docs/reports/m4.html), This vignette runs beam on the M4 forecasting competition (Makridakis, Spiliotis and Assimakopoulos 2020, [10.1016/j.ijforecast.2019.04.014](https://doi.org/10.1016/j.ijforecast.2019.04.014)), a large real benchmark from outside bioinformatics. M4 forecast 100,000 anonymized time series across six frequency bands (yearly, quarterly, monthly, weekly, daily, hourly). Each submitted method was scored by two error metrics, sMAPE and MASE, both lower is better. beam reads those two metrics from its registry as the [cards](../../docs/explanations/cards-and-pipeline.qmd) `smape` and `mase`, treats the six frequency bands as the datasets, and runs the same pipeline as the bioinformatics examples. -Three things are shown on real data. The registry does not care about the domain: forecasting metrics carry polarity, scale and [normalization](../../docs/explanations/normalization-and-scales.md) the same way clustering metrics do. The method ranking depends heavily on the frequency band, which is why leave-one-dataset-out and the [critical-difference diagram](../../docs/explanations/comparing-methods-across-datasets.md) across frequencies matter here. And the choice of how to pool the bands changes which method ranks first, so the recommendation is partly a decision about pooling rather than a fact about the methods. +Three things are shown on real data. The registry does not care about the domain: forecasting metrics carry polarity, scale and [normalization](../../docs/explanations/normalization-and-scales.md) the same way clustering metrics do. The method ranking depends heavily on the frequency band, which is why leave-one-dataset-out and the [critical-difference diagram](../../docs/explanations/critical-difference.md) across frequencies matter here. And the choice of how to pool the bands changes which method ranks first, so the recommendation is partly a decision about pooling rather than a fact about the methods. ## Provenance of the bundled table @@ -151,9 +151,9 @@ plot.dataset_stability(run) The hourly band is the most influential. It has only a few hundred series and very different dynamics from the long yearly and quarterly series, so a method tuned for it moves a lot in the ranking when it is dropped. This is the forecasting version of the method-by-dataset interaction the other examples show. -## Dataset concordance: do the bands agree on the order? +## Dataset concordance -The pooled ranking averages over the six frequency bands. [`dataset_concordance`](../../docs/explanations/dataset-concordance.md) ranks the methods within each band and compares every pair of per-band orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual bands; a low mean says it does not. It needs no replicates and assumes nothing about the bands being interchangeable. +The pooled ranking averages over the six frequency bands. [`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the methods within each band and compares every pair of per-band orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual bands; a low mean says it does not. It needs no replicates and assumes nothing about the bands being interchangeable. ```{python} conc = run.dataset_concordance @@ -175,7 +175,7 @@ M4 is the contrast to [Duo](../duo2018/duo2018.qmd): the bands disagree more, wh plot.dataset_concordance(run) ``` -The companion view marks where each method places higher or lower than its own typical rank, so a reader sees which methods carry the band disagreement rather than which method ranks best overall. +The companion view marks where each method places higher or lower than its own typical rank, showing which methods carry the band disagreement rather than which ranks best overall. ```{python} plot.dataset_struggle(run) @@ -193,9 +193,9 @@ funky_heatmap_from_run(run, title="M4: scores and rank robustness") There are only two metrics, so the glyph grid is small. The robustness panels carry more: many of the 25 methods change rank when a band is dropped or the aggregation is changed, and the SMAA bar spreads the top ranks across several methods rather than one. The methods score close together, so the pooled order is not firm. -## Heterogeneity: how much of the sMAPE variance is the band +## Mixed-effects on sMAPE -The leave-one-band-out check asks whether the ranking leans on one band. A [mixed-effects model](../../docs/explanations/heterogeneity-mixed-effects.md) asks the complementary question: how much of the sMAPE variation is a stable method effect and how much is the method-by-band interaction. It needs R's lme4, so the chunk runs only when it is available. +The leave-one-band-out check asks whether the ranking leans on one band. A [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) asks the complementary question: how much of the sMAPE variation is a stable method effect and how much is the method-by-band interaction. It needs R's lme4, so the chunk runs only when it is available. ```{python} from beam.heterogeneity import mixed_effects_from_matrix, r_available @@ -210,7 +210,7 @@ else: print("R with lme4 not available; skipping the mixed-effects fit.") ``` -On M4 the band intercept takes most of the sMAPE variance: the bands differ mostly in how hard they are to forecast for every method alike, which is the opposite of the transportation example, where the terrain decides which mode ranks first. A [Bradley-Terry tree](../../docs/explanations/heterogeneity-bradley-terry.md) on the six bands, with the seasonal period as the splitting feature, has too few datasets to find a stable split; this is the same small-sample limit the Duo benchmark hits, and why the OpenProblems spatial task (50 datasets) is where a split appears. +On M4 the band intercept takes most of the sMAPE variance: the bands differ mostly in how hard they are to forecast for every method alike, which is the opposite of the transportation example, where the terrain decides which mode ranks first. A [Bradley-Terry tree](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) on the six bands, with the seasonal period as the splitting feature, has too few datasets to find a stable split; this is the same small-sample limit the Duo benchmark hits, and why the OpenProblems spatial task (50 datasets) is where a split appears. ```{python} from beam.heterogeneity import bradley_terry_tree, bttree_available @@ -234,7 +234,7 @@ else: print("R with psychotree not available; skipping the Bradley-Terry tree.") ``` -## What moves the ranking: the band or the choice +## Rank sensitivity: band or analyst choice The leave-one-band-out check and the mixed-effects ICC both say the band matters. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) puts that on the same scale as the two modeling choices and says how much each one moves the ranking. It runs every combination of [weighting scheme](../../docs/explanations/weighting-schemes.md), [aggregation rule](../../docs/explanations/aggregation-methods.md) and band, then splits each method's rank variance into a share for each factor by analysis of variance. The design is a balanced full factorial, so the shares are exact, not sampled. @@ -264,7 +264,7 @@ print(f" most influential factor: {rs.most_influential_factor}") plot.rank_sensitivity(rs) ``` -The band accounts for almost all the rank variance and the two choices almost none. The M4 order is a question of which frequency you score on, not how you weight or aggregate. This is the same reading as the mixed-effects ICC above, reached a different way: an exact factorial decomposition of the ranks rather than a random-effects model of the scores. The headline method ranks differently across bands, which is the variance the band share counts. +The band accounts for almost all the rank variance and the two choices almost none. The M4 order is a question of which frequency you score on, not how you weight or aggregate. The mixed-effects ICC above reaches the same number a different way: an exact factorial decomposition of the ranks rather than a random-effects model of the scores. The headline method ranks differently across bands, which is the variance the band share counts. The shares above are pooled over the methods. The per-method version splits the same variance one method at a time. It separates a method whose rank depends on the band from one that depends on the weighting or the aggregation. The span next to each bar is the difference between the method's best and worst rank. @@ -280,7 +280,7 @@ for band, mean_rank in zip(rs.dataset_names, rs.headline_rank_by_dataset): print(f"{band:12s} {mean_rank:.1f}") ``` -[`specification_curve`](../../docs/explanations/specification-curve.md) lists the rankings the same grid produces and reports how often the top method holds. The full grid mixes the choices with the band; running it on the pooled matrix instead isolates the choices, so the gap between the two fractions is the band's doing. +[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) lists the rankings the same grid produces and reports how often the top method holds. The full grid mixes the choices with the band; running it on the pooled matrix instead isolates the choices, so the gap between the two fractions is the band's doing. ```{python} from beam.mcda import specification_curve @@ -326,7 +326,7 @@ print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12 ## Pairwise superiority across the bands -`rank_sensitivity` showed the band carries the ranking. [`pairwise_superiority`](../../docs/explanations/pairwise-superiority.md) reads the same fact pair by pair: how often one method outperforms another across the six bands on sMAPE. sMAPE declares no noise floor, so the equivalence band is zero here; any difference counts. +`rank_sensitivity` showed the band carries the ranking. [`pairwise_superiority`](../../docs/explanations/pairwise-method-comparison.md) reads the same fact pair by pair: how often one method outperforms another across the six bands on sMAPE. sMAPE declares no noise floor, so the equivalence band is zero here; any difference counts. ```{python} from beam.mcda import pairwise_superiority @@ -343,7 +343,7 @@ print(f"{top_method} outperforms {m4.method_names[runner]} on {n_top} of {pair.n With only six bands the sign test has little power, so few pairs reach significance, and the method with the highest standing outperforms the others on some bands and not on others. This is the band dependence the variance decomposition measured, now as a pairwise record: no method outperforms the field across every frequency. -[`pairwise_transitivity`](../../docs/explanations/transitivity.md) asks whether those pairwise majorities agree with a single order. With the ranking moving across the bands, they need not. +[`pairwise_transitivity`](../../docs/explanations/pairwise-method-comparison.md#transitivity) asks whether those pairwise majorities agree with a single order. With the ranking moving across the bands, they need not. ```{python} from beam.mcda import pairwise_transitivity @@ -358,7 +358,7 @@ The matrix below orders the methods by how many others they outperform. A transi plot.pairwise_majority(trans) ``` -[`bayesian_sign_comparison`](../../docs/explanations/bayesian-comparison.md) puts the same comparison on the probability scale: for each pair, the posterior probability that one method is practically better. With six bands the posterior is coarse, so most pairs stay inconclusive at the 0.95 threshold. +[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) puts the same comparison on the probability scale: for each pair, the posterior probability that one method is practically better. With six bands the posterior is coarse, so most pairs stay inconclusive at the 0.95 threshold. ```{python} from beam.mcda import bayesian_sign_comparison diff --git a/examples/openproblems/openproblems.qmd b/examples/openproblems/openproblems.qmd index 16a1e4e..b3e3716 100644 --- a/examples/openproblems/openproblems.qmd +++ b/examples/openproblems/openproblems.qmd @@ -156,7 +156,7 @@ from beam import plot plot.ranking(result, ground_truth_tool=methods[mean_order[0]]) ``` -### A funky heatmap that shows its own rank robustness +### Funky heatmap with rank robustness The [funky heatmap](../../docs/explanations/funky-heatmaps-and-robustness.md) shows a multi-metric benchmark with methods as rows sorted best first, metrics as circles sized by the score and coloured by group (here the scIB biological-conservation and batch-correction groups), and an overall bar. beam adds three panels: a worth panel with the ARI marginal mean and a 95 percent interval from the mixed-effects model (overlapping intervals mean two methods are not separable), the span of ranks each method takes across the leave-one-dataset-out runs, and the SMAA rank-acceptability bar (the share of random weightings that put a method at each rank). The row order and circle sizes depend on the normalization, which beam takes from the metric cards rather than defaulting to min-max. @@ -196,9 +196,9 @@ funky With only six datasets the leave-one-dataset-out spans are wide: the top methods shift by several ranks when a single dataset is dropped, while the bottom methods stay put. The worth panel agrees: the ARI marginal-mean intervals of the top methods overlap, so they are not separable. The SMAA bar puts the top ranks on several methods rather than one. The top of this order is not settled. -### Where the ARI variance lives +### Mixed-effects variance decomposition on ARI -A [mixed-effects model](../../docs/explanations/heterogeneity-mixed-effects.md) on one metric splits the score variation into a stable method effect and the method-by-dataset interaction. We run it on ARI. The fit needs R's lme4, so the chunk runs only when it is available. +A [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) on one metric splits the score variation into a stable method effect and the method-by-dataset interaction. We run it on ARI. The fit needs R's lme4, so the chunk runs only when it is available. ```{python} from beam.heterogeneity import mixed_effects_from_matrix, r_available @@ -221,9 +221,9 @@ else: print("Provision it with envs/heterogeneity.yml.") ``` -### Do the metric groups measure different things? +### Metric validity -The scIB score weights two metric groups, biological conservation at 0.6 and batch correction at 0.4, on the premise that they measure different things. [`beam.mcda.metric_validity`](../../docs/reference/metric_validity.qmd) checks that premise against the scores (Campbell and Fiske 1959). It treats each method-by-dataset cell as one observation, orients every metric to higher-is-better from the cards, and correlates the metrics with Spearman rank correlation. Metrics in the same group should agree (convergent); metrics in different groups should agree less (discriminant). See [the validity explanation](../../docs/explanations/convergent-discriminant-validity.md). +The scIB score weights two metric groups, biological conservation at 0.6 and batch correction at 0.4, on the premise that they measure different things. [`beam.mcda.metric_validity`](../../docs/reference/metric_validity.qmd) checks that premise against the scores (Campbell and Fiske 1959). It treats each method-by-dataset cell as one observation, orients every metric to higher-is-better from the cards, and correlates the metrics with Spearman rank correlation. Metrics in the same group should agree (convergent); metrics in different groups should agree less (discriminant). See [the validity explanation](../../docs/explanations/metric-set-diagnostics.md). ```{python} from beam.cards import polarities_for @@ -248,9 +248,9 @@ for name, group, within, between, nearest in validity.crossloading_metrics: The grouping holds, but barely. Metrics in the same group agree a little more (0.38 on average) than metrics in different groups (0.30). The biological metrics agree more with each other (0.45) than the batch metrics do (0.24). Three batch metrics, `graph_connectivity` most of all, agree more with the biological group than with their own. So the two groups overlap on this data, and the 0.6 / 0.4 weighting treats them as more separate than the scores show. -### Does each group hold together as one scale? +### Metric reliability -Validity asks whether the bio/batch split is the right axis. Reliability asks whether each side of that split reads as one scale. [`beam.mcda.metric_reliability`](../../docs/reference/metric_reliability.qmd) reports standardized Cronbach's alpha per group from the same oriented Spearman correlations (Cronbach 1951). A group above the conventional 0.7 cutoff reads as one reliable scale; below it the group is a looser collection. See [the reliability explanation](../../docs/explanations/reliability.md). +Validity asks whether the bio/batch split is the right axis. Reliability asks whether each side of that split reads as one scale. [`beam.mcda.metric_reliability`](../../docs/reference/metric_reliability.qmd) reports standardized Cronbach's alpha per group from the same oriented Spearman correlations (Cronbach 1951). A group above the conventional 0.7 cutoff reads as one reliable scale; below it the group is a looser collection. See [the reliability explanation](../../docs/explanations/metric-set-diagnostics.md#reliability). ```{python} from beam.mcda import metric_reliability @@ -277,9 +277,9 @@ for name, group, alpha_without in reliability.alpha_if_dropped: The biological group holds together (alpha 0.85 over seven metrics), the batch group does not at the cutoff (alpha 0.62 over five). Dropping `pcr` is the only batch removal that raises the batch alpha, so it is the batch metric least consistent with the rest of its group. The reading pairs with the validity result: the bio/batch axis is the right split, and its biological side is a coherent scale while its batch side is not. -### Is each group really one factor? +### Metric dimensionality -A high alpha can mean one factor measured consistently, or several factors in a group long enough to average out to a high alpha. [`beam.mcda.metric_dimensionality`](../../docs/reference/metric_dimensionality.qmd) separates the two by counting the factors in each group, with parallel analysis on the same oriented Spearman correlations (Horn 1965, Glorfeld 1995). See [the dimensionality explanation](../../docs/explanations/dimensionality.md). +A high alpha can mean one factor measured consistently, or several factors in a group long enough to average out to a high alpha. [`beam.mcda.metric_dimensionality`](../../docs/reference/metric_dimensionality.qmd) separates the two by counting the factors in each group, with parallel analysis on the same oriented Spearman correlations (Horn 1965, Glorfeld 1995). See [the dimensionality explanation](../../docs/explanations/metric-set-diagnostics.md#dimensionality). ```{python} from beam.mcda import metric_dimensionality @@ -299,11 +299,11 @@ print() print("read as one factor:", list(dimensionality.unidimensional_groups)) ``` -Dimensionality and reliability point in opposite directions here. The biological group has the higher alpha but carries two factors, so its 0.85 is partly the size of the group rather than one quantity. The batch group has the lower alpha but is one factor, weakly tracked. A group can be reliable without being one thing, and one thing without being reliable, which is why the two checks read together. +Dimensionality and reliability point in opposite directions here. The biological group has the higher alpha but carries two factors, so its 0.85 is partly the size of the group rather than one quantity. The batch group has the lower alpha but is one factor, weakly tracked. ## What moves the batch-integration ranking -The OpenProblems leaderboard pools by the mean of the scaled scores; beam can pool any of several ways. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) asks how much that choice matters against the data. It runs every combination of weighting, aggregation and dataset, and splits the rank variance. [COMET](../../docs/explanations/comet.md) is left out of the aggregations here: it builds characteristic objects whose count grows fast with the number of criteria, so it is slow on a 12-metric task. The other four aggregations are quick. +The OpenProblems leaderboard pools by the mean of the scaled scores; beam can pool any of several ways. [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) asks how much that choice matters against the data. It runs every combination of weighting, aggregation and dataset, and splits the rank variance. [COMET](../../docs/explanations/aggregation-methods.md#comet) is left out of the aggregations here: it builds characteristic objects whose count grows fast with the number of criteria, so it is slow on a 12-metric task. The other four aggregations are quick. ```{python} from beam.mcda import rank_sensitivity @@ -330,13 +330,13 @@ print(f" most influential factor: {rs.most_influential_factor}") The dataset is still the largest single factor, but here it is not the only one the way it is on M4. The aggregation choice and the dataset-by-choice interaction carry a share too. With 12 metrics that can disagree, how the metrics are combined changes the order more than it does on the two-metric M4 task. So on batch integration the ranking depends on which dataset and on how the metrics are pooled, which is the contrast with the OpenProblems mean-of-scores leaderboard the section above draws. -The shares above are pooled over the methods. The per-method version splits the same variance one method at a time. It separates a method whose rank depends on the dataset from one that depends on the weighting or the aggregation. The span next to each bar is the difference between the method's best and worst rank. +Those shares are averaged over the methods. Splitting them one method at a time shows which methods owe their rank movement to the dataset and which to the weighting or aggregation choice; the span beside each bar is that method's best-to-worst rank range. ```{python} plot.rank_sensitivity_by_tool(rs) ``` -[`specification_curve`](../../docs/explanations/specification-curve.md) reads the same grid as a list of rankings. Because the aggregation carries a real share here, the choice-only multiverse is less unanimous than on M4: the top method does not hold in every weighting-by-aggregation combination. +[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) reads the same grid as a list of rankings. Because the aggregation carries a real share here, the choice-only multiverse is less unanimous than on M4: the top method does not hold in every weighting-by-aggregation combination. ```{python} from beam.mcda import specification_curve @@ -363,9 +363,9 @@ print(f"choices only ({pooled.n_specifications} combinations): " plot.specification_curve(curve) ``` -## Dataset concordance: do the datasets agree on the order? +## Dataset concordance -The pooled ranking averages over the batch-integration datasets. [`dataset_concordance`](../../docs/explanations/dataset-concordance.md) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not. It needs no replicates and assumes nothing about the datasets being interchangeable. +Pooling averages over the six batch-integration datasets. [`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) re-ranks the methods within each dataset on its own and correlates every pair of those orderings with Kendall tau-b. A high mean means the pooled order represents the datasets; a low mean means it papers over their disagreement. No replicates are needed and the datasets need not be exchangeable. ```{python} conc = result.dataset_concordance @@ -411,7 +411,7 @@ print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12 ## Pairwise superiority on ARI -[`pairwise_superiority`](../../docs/reference/pairwise_superiority.qmd) [compares the methods two at a time](../../docs/explanations/pairwise-superiority.md) on ARI across the six datasets, with the ARI noise floor as the equivalence band. It reports how often one method outperforms another, the effect size next to the significance the mixed-effects intervals give. +[`pairwise_superiority`](../../docs/reference/pairwise_superiority.qmd) [compares the methods two at a time](../../docs/explanations/pairwise-method-comparison.md) on ARI across the six datasets, with the ARI noise floor as the equivalence band. It reports how often one method outperforms another, the effect size next to the significance the mixed-effects intervals give. ```{python} from beam.mcda import pairwise_superiority @@ -423,9 +423,9 @@ print(f"highest standing: {methods[sup.order[0]]} ({sup.standing[sup.order[0]]:. print(f"pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}") ``` -With six datasets the comparison is coarse: most pairs do not reach significance, matching the wide mixed-effects intervals and the leave-one-dataset-out spans above. The pairwise view says the same as the worth panel, as a pairwise record rather than an interval. +With six datasets the comparison is coarse: most pairs do not reach significance, matching the wide mixed-effects intervals and the leave-one-dataset-out spans above. -[`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) reads the same pairwise relation and reports whether one order is [consistent with it](../../docs/explanations/transitivity.md). +[`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) reads the same pairwise relation and reports whether one order is [consistent with it](../../docs/explanations/pairwise-method-comparison.md#transitivity). ```{python} from beam.mcda import pairwise_transitivity @@ -440,7 +440,7 @@ The matrix below orders the methods by how many others they outperform. A transi plot.pairwise_majority(trans) ``` -On the probability scale, [`bayesian_sign_comparison`](../../docs/explanations/bayesian-comparison.md) gives for each pair the posterior probability that one method is practically better. With six datasets most pairs stay inconclusive at the 0.95 threshold. +On the probability scale, [`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) gives for each pair the posterior probability that one method is practically better. With six datasets most pairs stay inconclusive at the 0.95 threshold. ```{python} from beam.mcda import bayesian_sign_comparison @@ -520,8 +520,8 @@ if bt is not None: display(plot.bradley_terry_leaves(bt)) ``` -The pooled ranking has one method at the top, but the leaves disagree: which method ranks first depends on the spatial assay technology. A single ranking does not show this heterogeneity. The 50-dataset task shows it where the 12-dataset Duo benchmark could not. See [the Bradley-Terry explanation](../../docs/explanations/heterogeneity-bradley-terry.md). +The pooled ranking has one method at the top, but the leaves disagree: which method ranks first depends on the spatial assay technology. A single ranking does not show this heterogeneity. The 50-dataset task shows it where the 12-dataset Duo benchmark could not. See [the Bradley-Terry explanation](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees). ## Recommendation -On batch integration, beam re-derives the per-metric normalization from the cards and runs MCDA with sensitivity, which can place a different method first than the platform's mean-of-scores leaderboard. On spatially variable genes, the Bradley-Terry tree shows there is no single method that ranks first across assays. Both cases show the same thing: a benchmark recommendation depends on how the criteria are combined and on which datasets it is read over. +On batch integration, beam re-derives the per-metric normalization from the cards and runs MCDA with sensitivity, which can place a different method first than the platform's mean-of-scores leaderboard. On spatially variable genes, the Bradley-Terry tree shows there is no single method that ranks first across assays. A benchmark recommendation depends on how the criteria are combined and on which datasets it is read over. diff --git a/examples/scenarios/scenarios.qmd b/examples/scenarios/scenarios.qmd index 517f26f..013753f 100644 --- a/examples/scenarios/scenarios.qmd +++ b/examples/scenarios/scenarios.qmd @@ -231,7 +231,7 @@ plot.rank_heatmap( ## Dataset concordance: which dataset is the odd one -[`dataset_concordance`](../../docs/explanations/dataset-concordance.md) ranks the methods within each dataset and reports the Kendall tau-b agreement between every pair, so it names the dataset that orders the methods unlike the rest. This scenario plants one such dataset, and the diagnostic recovers it without being told which one it is. +[`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the methods within each dataset and reports the Kendall tau-b agreement between every pair, so it names the dataset that orders the methods unlike the rest. This scenario plants one such dataset, and the diagnostic recovers it without being told which one it is. ```{python} odd_scores = beam.Scores( @@ -326,9 +326,9 @@ plt.show() The dashed line is the actual runtime in seconds. Under min-max the four fast methods (m0 to m3) sit at almost the same height, so their speed differences vanish and a tiny ARI difference decides the order. Under log_min_max the bars follow the true ordering, so the fastest method ranks first. -## Are the methods separable across datasets? +## Separability across datasets -The MCDA composite gives one ranking, but it does not say whether the differences hold across the datasets. The Demsar (2006) Friedman test and Nemenyi post-hoc answer that. Here they run on the odd-dataset scenario, using the per-dataset ARI as the tool by dataset matrix. See the explanation page on [comparing methods across datasets](../../docs/explanations/comparing-methods-across-datasets.md) for the reasoning. +The MCDA composite gives one ranking, but it does not say whether the differences hold across the datasets. The Demsar (2006) Friedman test and Nemenyi post-hoc answer that. Here they run on the odd-dataset scenario, using the per-dataset ARI as the tool by dataset matrix. See the explanation page on [comparing methods across datasets](../../docs/explanations/critical-difference.md) for the reasoning. ```{python} odd = next(s for s in scenarios if s.kind == "odd_dataset") @@ -350,7 +350,7 @@ plot.critical_difference_band(cd) With only five datasets the critical difference is wide. Every method falls inside the shaded band, and the Friedman test does not reject at alpha 0.05, so the data cannot separate the methods even though the MCDA composite ranks m0 first. The recommendation holds under the stated weighting, but a claim that one method ranks higher across datasets would need more datasets to support it. -[`pairwise_superiority`](../../docs/explanations/pairwise-superiority.md) reads the same data as an effect size: how often each method outperforms another across the five datasets. With so few datasets the sign test has almost no power, so the pairwise records are close and none reaches significance, the same conclusion the critical-difference band gives. +[`pairwise_superiority`](../../docs/explanations/pairwise-method-comparison.md) reads the same data as an effect size: how often each method outperforms another across the five datasets. With so few datasets the sign test has almost no power, so the pairwise records are close and none reaches significance, the same conclusion the critical-difference band gives. ```{python} from beam.mcda import pairwise_superiority @@ -360,7 +360,7 @@ print(f"highest standing: {odd.method_names[sup.order[0]]} ({sup.standing[sup.or print(f"pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}") ``` -[`pairwise_transitivity`](../../docs/explanations/transitivity.md) checks whether those pairwise majorities agree with one order. +[`pairwise_transitivity`](../../docs/explanations/pairwise-method-comparison.md#transitivity) checks whether those pairwise majorities agree with one order. ```{python} from beam.mcda import pairwise_transitivity @@ -375,7 +375,7 @@ The matrix below orders the methods by how many others they outperform. A transi plot.pairwise_majority(trans) ``` -[`bayesian_sign_comparison`](../../docs/explanations/bayesian-comparison.md) gives the posterior probability that one method is practically better than another, the probability-scale companion to the sign test. With five datasets it is mostly inconclusive at the 0.95 threshold, the same limit the sign test and the critical-difference band show. +[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) gives the posterior probability that one method is practically better than another, the probability-scale companion to the sign test. With five datasets it is mostly inconclusive at the 0.95 threshold, the same limit the sign test and the critical-difference band show. ```{python} from beam.mcda import bayesian_sign_comparison @@ -401,7 +401,7 @@ for j in range(8): het[:, j] = np.clip(np.array(means) + rng.normal(0, 0.05, 3), 0, 1) ``` -A [mixed-effects model](../../docs/explanations/heterogeneity-mixed-effects.md) on this metric reports a small dataset intercept and a large residual. With one observation per cell the residual is the method-by-dataset interaction confounded with noise, and here it is the largest share because the methods reorder between the two regimes rather than the datasets differing in difficulty. The fits need the R toolchain, so the chunks run only when it is available. +A [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) on this metric reports a small dataset intercept and a large residual. With one observation per cell the residual is the method-by-dataset interaction confounded with noise, and here it is the largest share because the methods reorder between the two regimes rather than the datasets differing in difficulty. The fits need the R toolchain, so the chunks run only when it is available. ```{python} from beam.heterogeneity import ( @@ -419,7 +419,7 @@ else: print("R with lme4 not available; skipping the mixed-effects fit.") ``` -The [Bradley-Terry tree](../../docs/explanations/heterogeneity-bradley-terry.md) finds the feature behind the interaction. Unlike the small real benchmarks, this synthetic case has enough datasets and a clean signal, so the parameter-stability test splits the datasets on `regime` and gives each leaf a different method ranking first, recovering exactly the reversal built into the construction. +The [Bradley-Terry tree](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) finds the feature behind the interaction. Unlike the small real benchmarks, this synthetic case has enough datasets and a clean signal, so the parameter-stability test splits the datasets on `regime` and gives each leaf a different method ranking first, recovering exactly the reversal built into the construction. ```{python} if bttree_available(): @@ -463,7 +463,7 @@ for finding in flagged.violations: print(" ", finding.severity, finding.message) ``` -## What moves the ranking with a single dataset +## Rank sensitivity with a single dataset The canonical scenarios are single-dataset, so [`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) runs in its two-factor mode: it varies the weighting and the aggregation, and the dataset factor is absent. The constructed scenarios give it a known answer to recover. In the dominant scenario one method scores highest on every metric, so it is rank 1 in every combination and its rank variance is zero, whatever the weighting or aggregation. In the random scenario the two metrics trade off, so the order depends on the choice. @@ -484,7 +484,7 @@ for name in ["dominant", "random"]: f"{curve.n_distinct_top_tools} method(s) reach the top") ``` -The dominant scenario's top method has a rank span of zero and is rank 1 in every combination, as built: with a single dataset and a method that scores highest everywhere, no modeling choice can move it, so [`specification_curve`](../../docs/explanations/specification-curve.md) reports it first in 100 percent of the specifications and only one method reaching the top. In the random scenario the top method's rank moves with the choice, the most-frequent-top fraction drops below 100 percent, and more than one method reaches the top, because the two metrics trade off and there is no dataset factor to carry the variance. +The dominant scenario's top method has a rank span of zero and is rank 1 in every combination, as built: with a single dataset and a method that scores highest everywhere, no modeling choice can move it, so [`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) reports it first in 100 percent of the specifications and only one method reaching the top. In the random scenario the top method's rank moves with the choice, the most-frequent-top fraction drops below 100 percent, and more than one method reaches the top, because the two metrics trade off and there is no dataset factor to carry the variance. The per-method split shows this directly. In the dominant scenario the top method has no rank variance to split, so it is drawn empty with a "rank fixed" note. In the random scenario the methods carry weighting and aggregation shares. diff --git a/examples/transportation/transportation.qmd b/examples/transportation/transportation.qmd index e38a90e..37bb36c 100644 --- a/examples/transportation/transportation.qmd +++ b/examples/transportation/transportation.qmd @@ -25,7 +25,7 @@ The third point is a crossover among the slower modes. Trail running is slower t ## Set-up -The five [aggregation methods](../../docs/explanations/aggregation-methods.md) used below (SAW, TOPSIS, VIKOR, PROMETHEE II, [COMET](../../docs/explanations/comet.md)) are wrapped from pymcdm. beam normalizes the scores, then calls pymcdm on the normalized matrix and keeps the higher-is-better convention. The [weighting schemes](../../docs/explanations/weighting-schemes.md) are beam's own. +The five [aggregation methods](../../docs/explanations/aggregation-methods.md) used below (SAW, TOPSIS, VIKOR, PROMETHEE II, [COMET](../../docs/explanations/aggregation-methods.md#comet)) are wrapped from pymcdm. beam normalizes the scores, then calls pymcdm on the normalized matrix and keeps the higher-is-better convention. The [weighting schemes](../../docs/explanations/weighting-schemes.md) are beam's own. ```{python} %matplotlib inline @@ -227,7 +227,7 @@ Across both comparisons, the choice of weighting changes which mode ranks first ## A critical-difference diagram on a common block -A Demsar [critical-difference diagram](../../docs/explanations/comparing-methods-across-datasets.md) needs a complete tool by dataset table with no missing cells. Because no mode runs on every terrain, the diagram is restricted to a block of modes and the terrains where all of them run. The four ground modes (foot, running, bicycle, motorcycle) are feasible on five common terrains (flat road, mud, uphill, long distance, urban hop); open water is excluded because no ground mode runs there. The helper `common_feasible_block` returns that block and its common terrains. The diagram is built on speed, the metric whose per-terrain ground truth is marked above. +A Demsar [critical-difference diagram](../../docs/explanations/critical-difference.md) needs a complete tool by dataset table with no missing cells. Because no mode runs on every terrain, the diagram is restricted to a block of modes and the terrains where all of them run. The four ground modes (foot, running, bicycle, motorcycle) are feasible on five common terrains (flat road, mud, uphill, long distance, urban hop); open water is excluded because no ground mode runs there. The helper `common_feasible_block` returns that block and its common terrains. The diagram is built on speed, the metric whose per-terrain ground truth is marked above. ```{python} block_modes = ("foot", "running", "bicycle", "motorcycle") @@ -299,7 +299,7 @@ print(f"top rank is fragile: {ts.top_rank_is_fragile}") The most fragile pair is the one that the smallest single-weight nudge can reorder. Here a change of about five thousandths to the speed weight is enough to swap the kayak and the plane, while the equal-weight top rank itself is not fragile under any single-weight change of that size. The pairing tells a user which two modes sit closest together under the current weighting, so a small revision to preferences would reorder them first. -## Leave one dataset out: how much the pooled order leans on one terrain +## Leave one dataset out The sections above keep the analysis per terrain because no mode runs on every terrain. On a complete block, where a set of modes does run on every terrain in the block, a pooled ranking is well defined, and the next question is how much that pooled ranking leans on any single terrain. Leave-one-dataset-out answers it: pool the block, rank it, then drop one terrain, pool the rest, and re-rank. The water modes show this sharply. The kayak, the motorboat and the small plane all run on two terrains, open water and the long distance, so they form a complete two-terrain block. @@ -355,7 +355,7 @@ plt.show() Pooled over both terrains the plane ranks first, because its 750 km/h on the long leg pulls its mean speed far above the others. Drop the long distance and rank on open water alone, and the motorboat ranks first instead: it is the fastest mode on the water, where the plane is slow. Drop open water and the long-distance order stands, plane first. The plane's first place is not a property of water travel; it is an artifact of including the long leg in the pool. The kayak ranks last in every run under speed-heavy weights. Leave-one-dataset-out turns that into a number: the plane and the motorboat each hold their rank in only half the runs, so the pooled recommendation between them depends entirely on whether the long-distance terrain is in the pool. This is the same caution the per-terrain sections raise, now stated as a stability measure on a complete block. -[`pairwise_transitivity`](../../docs/explanations/transitivity.md) reads both complete blocks from the pairwise side, on speed. The four land modes have a fixed speed order, so their pairwise majorities are transitive and give one consistent order, motorcycle down to foot. The water modes have no cycle either, but the plane and the boat tie. The plane is faster on the long distance and the boat on open water, so over the two terrains each outperforms the other once, and neither is the pairwise-majority first choice. The kayak is outperformed by both. The tie is the pairwise reading of the half-and-half stability above. +[`pairwise_transitivity`](../../docs/explanations/pairwise-method-comparison.md#transitivity) reads both complete blocks from the pairwise side, on speed. The four land modes have a fixed speed order, so their pairwise majorities are transitive and give one consistent order, motorcycle down to foot. The water modes have no cycle either, but the plane and the boat tie. The plane is faster on the long distance and the boat on open water, so over the two terrains each outperforms the other once, and neither is the pairwise-majority first choice. The kayak is outperformed by both. The tie is the pairwise reading of the half-and-half stability above. ```{python} from beam.mcda import pairwise_superiority, pairwise_transitivity @@ -404,7 +404,7 @@ fig.tight_layout() plt.show() ``` -[`bayesian_sign_comparison`](../../docs/explanations/bayesian-comparison.md) reads the water block on the probability scale. With two terrains the counts are tiny, so the posterior stays close to the prior and no pair reaches a decisive label. The plane and the boat each score higher on one terrain, so the posterior splits between them. +[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) reads the water block on the probability scale. With two terrains the counts are tiny, so the posterior stays close to the prior and no pair reaches a decisive label. The plane and the boat each score higher on one terrain, so the posterior splits between them. ```{python} from beam.mcda import bayesian_sign_comparison @@ -433,7 +433,7 @@ water_run = beam.rank(water, weights="equal", method="saw") funky_heatmap_from_run(water_run, title="Transport water block: scores and rank robustness") ``` -[`specification_curve`](../../docs/explanations/specification-curve.md) lists the rankings the water block produces under every weighting, aggregation and terrain. The two terrains order the modes differently, so the top mode does not hold across the grid: it is the same instability the leave-one-dataset-out section reports, read as a list of rankings. +[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) lists the rankings the water block produces under every weighting, aggregation and terrain. The two terrains order the modes differently, so the top mode does not hold across the grid: it is the same instability the leave-one-dataset-out section reports, read as a list of rankings. ```{python} from beam.mcda import rank_sensitivity, specification_curve @@ -484,7 +484,7 @@ print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12 ## Dataset concordance: land agrees, water reverses -[`dataset_concordance`](../../docs/explanations/dataset-concordance.md) ranks the modes within each terrain and reports the Kendall tau-b agreement between every pair of terrains. It reads on any complete block. The four ground modes share five land terrains, so they form one complete block; the water modes share two. The two blocks sit at opposite ends of the agreement scale. +[`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the modes within each terrain and reports the Kendall tau-b agreement between every pair of terrains. It reads on any complete block. The four ground modes share five land terrains, so they form one complete block; the water modes share two. The two blocks sit at opposite ends of the agreement scale. ```{python} land_mode_idx = [tb.mode_names.index(m) for m in block_modes] @@ -521,7 +521,7 @@ A single concordance over every mode and terrain at once is not well defined, fo ## Heterogeneity: the interaction is large -The [mixed-effects model](../../docs/explanations/heterogeneity-mixed-effects.md) puts a number on what the per-terrain sections show. Fit on speed, with the mode as a fixed effect and the terrain as a random intercept, it splits the speed variance into a between-terrain shift and the rest. The fit needs R's lme4, so the chunk runs only when it is available. +The [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) puts a number on what the per-terrain sections show. Fit on speed, with the mode as a fixed effect and the terrain as a random intercept, it splits the speed variance into a between-terrain shift and the rest. The fit needs R's lme4, so the chunk runs only when it is available. ```{python} from beam.heterogeneity import mixed_effects_from_matrix, r_available @@ -538,10 +538,10 @@ else: print("R with lme4 not available; skipping the mixed-effects fit.") ``` -The contrast with the other examples is clear. On M4 the band intercept takes most of the variance, so the bands differ mostly in difficulty. Here the terrain intercept takes very little: almost all of the speed variance is the mode-by-terrain interaction, because the fast mode on one terrain is the slow mode on another. That is why a single pooled speed ranking is misleading even where it can be computed. A [Bradley-Terry tree](../../docs/explanations/heterogeneity-bradley-terry.md) on the six terrains has too few datasets to find a stable split, the same small-sample limit the [Duo](../duo2018/duo2018.qmd) and [M4](../m4/m4.qmd) examples hit; the [OpenProblems spatial task](../openproblems/openproblems.qmd) is where a split appears. +The contrast with the other examples is clear. On M4 the band intercept takes most of the variance, so the bands differ mostly in difficulty. Here the terrain intercept takes very little: almost all of the speed variance is the mode-by-terrain interaction, because the fast mode on one terrain is the slow mode on another. That is why a single pooled speed ranking is misleading even where it can be computed. A [Bradley-Terry tree](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) on the six terrains has too few datasets to find a stable split, the same small-sample limit the [Duo](../duo2018/duo2018.qmd) and [M4](../m4/m4.qmd) examples hit; the [OpenProblems spatial task](../openproblems/openproblems.qmd) is where a split appears. ## Why per-terrain is the informative output This example was built so that no mode runs on every terrain and no mode is fastest on every terrain it does run on. Both facts push the analysis to be per-terrain. Partial coverage means a pooled ranking over all modes is not even well defined, since different modes are measured on different terrains. The method-by-dataset interaction means that even where a pooled ranking is computable, it reports a single order that is wrong on at least some terrains: the fast modes rank first on their own terrains and rank lower on others. -This matches method-by-dataset interaction in bioinformatics benchmarks. A clustering method that ranks first on one tissue can rank last on another, and a method that needs a feature one dataset lacks is simply not applicable there, which is the same role the NaN cells play here. The transportation numbers are made up, but the structure of the problem is the same. Report per dataset and mark which method ranks first on each. Then use the cross-dataset tests (the critical-difference diagram, SMAA, weight perturbation) to say how much of the apparent ordering survives a change of conditions or of preferences. +This matches method-by-dataset interaction in bioinformatics benchmarks. A clustering method that ranks first on one tissue can rank last on another, and a method that needs a feature one dataset lacks is simply not applicable there, which is the same role the NaN cells play here. The transportation numbers are made up, but the structure of the problem is the same. The informative output is per dataset: which method ranks first on each. The cross-dataset tests (the critical-difference diagram, SMAA, weight perturbation) then say how much of the apparent ordering survives a change of conditions or of preferences. diff --git a/index.qmd b/index.qmd index 57f3c78..dd3ed09 100644 --- a/index.qmd +++ b/index.qmd @@ -3,17 +3,17 @@ title: "beam" subtitle: "Benchmark Evaluation and Metrics" --- -beam is a metric formalization layer for method comparisons and benchmarks in bioinformatics and beyond. It standardizes performance metrics in metric cards with measurement-theory grounded metadata. It also provides multi-criteria decision making features, and evaluates the driving forces of method-dataset heterogeneity. +beam is a metric formalization layer for method comparisons and benchmarks in bioinformatics. It records each performance metric in a [metric card](docs/explanations/cards-and-pipeline.qmd) with [measurement-theory](docs/explanations/measurement-theory.md) grounded metadata, ranks methods with [multi-criteria decision analysis](docs/explanations/aggregation-methods.md), and decomposes [method-dataset heterogeneity](docs/explanations/method-by-dataset-heterogeneity.md). The metric cards are mapped to STATO, UO, OBI and HuggingFace evaluate where available. The OWL is available at [docs/beam.owl.ttl](docs/beam.owl.ttl) and regenerated from the cards on each release. ## Where to start -- [Quick start](docs/tutorials/quickstart.md). The five-line path from a CSV to an metrics evaluation HTML report. +- [Quick start](docs/tutorials/quickstart.md). From a CSV to a ranking and an HTML report. - [How to run from beam.yaml](docs/how-to/run-from-beam-yaml.md). The declarative beam specification. - [How to use beam from R](docs/how-to/use-beam-from-r.md). The R interface (reticulate-backed; same metric cards and pipeline). -- [Cards and pipeline](docs/explanations/cards-and-pipeline.qmd). What a metric card encodes and how is it processed. -- [Comparing methods across datasets](docs/explanations/comparing-methods-across-datasets.md). Diagnostics on method-dataset performance variability. +- [Cards and pipeline](docs/explanations/cards-and-pipeline.qmd). What a metric card encodes and how it is processed. +- [Critical difference](docs/explanations/critical-difference.md). Whether the methods are separable across datasets (Friedman, Nemenyi, Skillings-Mack). ## Worked vignettes diff --git a/r/beam/vignettes/cell-type-annotation.Rmd b/r/beam/vignettes/cell-type-annotation.Rmd index a8808c3..a284068 100644 --- a/r/beam/vignettes/cell-type-annotation.Rmd +++ b/r/beam/vignettes/cell-type-annotation.Rmd @@ -46,7 +46,7 @@ cat(sprintf("complete block: %d methods, %d datasets, %d metrics\n", ## One ranking, end to end -`beam_rank` pools the two metrics across the datasets, runs the MCDA pipeline, and runs the default sensitivity analysis on the same normalization context. +`beam_rank` pools the two metrics across the datasets, runs the [MCDA pipeline](https://imallona.github.io/beam/docs/explanations/aggregation-methods.html), and runs the default sensitivity analysis on the same normalization context. ```{r rank} scores <- beam$Scores( @@ -71,9 +71,9 @@ GPT-4 ranks first. SingleR, a classical tool, ranks second, and GPT-3.5 ranks th plot(run) ``` -## What moves the ranking: the dataset or the choice? +## Rank sensitivity: dataset or analyst choice -`beam_rank_sensitivity` splits the ranking's variance between the data and the analyst's choices, by ranking the methods on each dataset under every weighting and aggregation. +[`beam_rank_sensitivity`](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html) splits the ranking's variance between the data and the analyst's choices, by ranking the methods on each dataset under every weighting and aggregation. ```{r rank-sensitivity} ctx <- mcda$registry_context(as.list(metrics), "saw") @@ -89,9 +89,9 @@ cat(sprintf("dataset %.2f, weighting %.2f, aggregation %.2f, interaction %.2f\n" The dataset accounts for almost all of the rank variance; the weighting and the aggregation account for very little. Which tissue is annotated decides the per-dataset order far more than the scoring choice. -## Dataset concordance: do the datasets agree on the order? +## Dataset concordance -`dataset_concordance` ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not. +[`dataset_concordance`](https://imallona.github.io/beam/docs/explanations/dataset-concordance-and-discrimination.html) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not. ```{r concordance, fig.height = 4} conc <- run$dataset_concordance @@ -104,7 +104,7 @@ beam_plot(run, "dataset_struggle") ## Specification curve -`beam_specification_curve` lists the rankings the grid produces and reports how often the top method holds. +[`beam_specification_curve`](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html#the-specification-curve) lists the rankings the grid produces and reports how often the top method holds. ```{r specification-curve} curve <- beam_specification_curve(rs) @@ -120,7 +120,7 @@ beam_plot(curve, "specification_curve") ## Blind analysis -GPT-4 is the method under test here, so blinding matters: fixing the pipeline on hidden labels removes the chance to tune the weighting toward it. `beam_blind` hides the names; `beam_unblind` restores them. The ranking does not change. +GPT-4 is the method under test here, so [blinding](https://imallona.github.io/beam/docs/explanations/analysis-blinding.html) matters: fixing the pipeline on hidden labels removes the chance to tune the weighting toward it. `beam_blind` hides the names; `beam_unblind` restores them. The ranking does not change. ```{r blinding} blinded <- beam_blind(scores, seed = 0L) @@ -132,9 +132,9 @@ cat("blinding fingerprint:", substr(blind_run$manifest[["blinding"]][["seal_sha256"]], 1, 12), "\n") ``` -## Are the methods separable on agreement? +## Critical difference on agreement -`beam_critical_difference` runs the Friedman test and the Nemenyi post-hoc on the agreement scores across the complete-block datasets. +[`beam_critical_difference`](https://imallona.github.io/beam/docs/explanations/critical-difference.html) runs the Friedman test and the Nemenyi post-hoc on the agreement scores across the complete-block datasets. ```{r cd} agreement <- block[, , 1] diff --git a/r/beam/vignettes/cross-benchmark.Rmd b/r/beam/vignettes/cross-benchmark.Rmd index 76ed972..7a292f1 100644 --- a/r/beam/vignettes/cross-benchmark.Rmd +++ b/r/beam/vignettes/cross-benchmark.Rmd @@ -75,7 +75,7 @@ Agreement rises from near zero to moderate. Much of the disagreement was in the ## Funky heatmaps per benchmark -The glyph tables show why the reported orders differ: the per-metric pattern over the five methods is not the same across benchmarks. Each circle is the method's standing on that metric within the benchmark (larger is better), coloured by the scIB biological and batch groups. +The [glyph tables](https://imallona.github.io/beam/docs/explanations/funky-heatmaps-and-robustness.html) show why the reported orders differ: the per-metric pattern over the five methods is not the same across benchmarks. Each circle is the method's standing on that metric within the benchmark (larger is better), coloured by the scIB biological and batch groups. ```{r funky, fig.height = 3.6} mean_rank_3d <- tapply(rank_value, list(benchmark, method, metric), mean) @@ -94,7 +94,7 @@ for (b in bench) { } ``` -## A fifth source with a different lens: BatchBench +## A fifth source: BatchBench BatchBench scores the methods with two entropy metrics instead of the scIB family: batch entropy, where a higher value means the batches are better mixed, and cell-type entropy, where a higher value means the cell types are more blurred, so a lower value is better for biology. Ruben Chazarra-Gil provided the scores by personal communication. The two metrics show the batch-mixing against biology tradeoff directly. @@ -121,7 +121,7 @@ A method low and to the right mixes batches well while keeping the cell types se ## One coherent ranking: network meta-analysis -A network meta-analysis pools all five sources into one ranking, the way clinical research ranks treatments never all tried in one trial. Each method is a treatment, each (benchmark, dataset) block is a study. It needs R's netmeta. +A [network meta-analysis](https://imallona.github.io/beam/docs/explanations/network-meta-analysis.html) pools all five sources into one ranking, the way clinical research ranks treatments never all tried in one trial. Each method is a treatment, each (benchmark, dataset) block is a study. It needs R's netmeta. ```{r netmeta, eval = have_netmeta, fig.height = 3.6} arms <- ib$network_arms() @@ -137,7 +137,7 @@ With BatchBench in the pool, harmony no longer leads on its own: it ties with li ## How much of the variance is the benchmark -A cross-source variance decomposition puts a number on the split: the method-by-benchmark component is the disagreement attributable to the benchmark rather than the method. The chunk fits it with the four scIB-family sources and again with BatchBench added, so the effect of the fifth source is visible. It needs R's lme4. +A cross-source [variance decomposition](https://imallona.github.io/beam/docs/explanations/attribution-synthesis.html) puts a number on the split: the method-by-benchmark component is the disagreement attributable to the benchmark rather than the method. The chunk fits it with the four scIB-family sources and again with BatchBench added, so the effect of the fifth source is visible. It needs R's lme4. ```{r source-variance, eval = have_lme4, message = FALSE, warning = FALSE} rec <- ib$mean_rank_records() @@ -179,11 +179,11 @@ ggplot2::ggplot(contrast, ggplot2::aes(method, rank, fill = pipeline)) + ggplot2::theme_minimal(base_size = 11) ``` -Both pipelines agree on the leader on the shared data, but the rest of the order disagrees more than the pooled Spearman would predict. The disagreement is not the data; it is in the pipeline. +Both pipelines agree on the leader on the shared data, but the rest of the order disagrees more than the pooled Spearman would predict. Because the data is held fixed here, that disagreement is the pipeline. ## Specification curve per benchmark -The specification curve lists the rankings every weighting-by-aggregation combination gives on each benchmark's mean-rank matrix, and reports how often the top method holds. On Tran and scIB the top method holds across all combinations; on OpenProblems it does not. +The [specification curve](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html#the-specification-curve) lists the rankings every weighting-by-aggregation combination gives on each benchmark's mean-rank matrix, and reports how often the top method holds. On Tran and scIB the top method holds across all combinations; on OpenProblems it does not. ```{r spec-curve} mcda <- reticulate::import("beam.mcda") @@ -202,7 +202,7 @@ for (b in bench) { ## Blind analysis -A blind analysis hides the method names before the pipeline is fixed. These benchmarks rank on the mean-rank scale rather than through metric cards, so the demonstration relabels the matrix and then restores the names with the seal. `beam_blind` shuffles and relabels the rows; `beam_unblind` maps them back. +A [blind analysis](https://imallona.github.io/beam/docs/explanations/analysis-blinding.html) hides the method names before the pipeline is fixed. These benchmarks rank on the mean-rank scale rather than through metric cards, so the demonstration relabels the matrix and then restores the names with the seal. `beam_blind` shuffles and relabels the rows; `beam_unblind` maps them back. ```{r blind} beam <- reticulate::import("beam") @@ -218,9 +218,9 @@ cat("restored names:", paste(as.character(unlist(restored$tool_names)), collapse cat("seal fingerprint:", substr(bl$seal$fingerprint, 1, 12), "\n") ``` -## A sixth source with disjoint methods: dataset discrimination +## A sixth source: dataset discrimination -Every analysis above compares method orders, so it needs shared methods. The deep-learning benchmark of Shen, He and Guan (2026, [10.1371/journal.pcbi.1014008](https://doi.org/10.1371/journal.pcbi.1014008)) overlaps the classical five only on harmony and scanorama, too few to rank-harmonize. `beam_dataset_discrimination` reports a property defined per dataset: how much each dataset separates the methods it scores, as the spread of the pooled scores (the effect size) and Kendall's W over the method-by-metric matrix (the consistency). beam treats each (dataset, scenario) pair as one unit. +Every analysis above compares method orders, so it needs shared methods. The deep-learning benchmark of Shen, He and Guan (2026, [10.1371/journal.pcbi.1014008](https://doi.org/10.1371/journal.pcbi.1014008)) overlaps the classical five only on harmony and scanorama, too few to rank-harmonize. [`beam_dataset_discrimination`](https://imallona.github.io/beam/docs/explanations/dataset-concordance-and-discrimination.html#dataset-discrimination) reports a property defined per dataset: how much each dataset separates the methods it scores, as the spread of the pooled scores (the effect size) and Kendall's W over the method-by-metric matrix (the consistency). beam treats each (dataset, scenario) pair as one unit. ```{r discrimination} shen <- datasets$load_semisupervised_integration() diff --git a/r/beam/vignettes/heterogeneity.Rmd b/r/beam/vignettes/heterogeneity.Rmd index 60ef2ac..d8c9a2e 100644 --- a/r/beam/vignettes/heterogeneity.Rmd +++ b/r/beam/vignettes/heterogeneity.Rmd @@ -18,7 +18,7 @@ knitr::opts_chunk$set(eval = have_beam, collapse = TRUE, comment = "#>", dpi = 140, out.width = "100%", fig.width = 7, fig.height = 4.5) ``` -A single global ranking hides the more useful question: how much of the score variation is a stable method effect, and how much is the method-by-dataset interaction. beam answers it with three models that fit natively in R, so each section needs its own modelling package (`lme4`, `psychotree`, `PlackettLuce`) and runs only when that package is installed. The benchmark scores come from the Python `beam` core through reticulate; install it once with `install_beam_python()`. The data is the Duo 2018 ARI matrix: 14 clustering methods by 12 datasets. +A single global ranking reports one order; it does not say how much of the score variation is a stable method effect and how much is the method-by-dataset interaction. beam answers that with [three models](https://imallona.github.io/beam/docs/explanations/method-by-dataset-heterogeneity.html) that fit natively in R, so each section needs its own modelling package (`lme4`, `psychotree`, `PlackettLuce`) and runs only when that package is installed. The benchmark scores come from the Python `beam` core through reticulate; install it once with `install_beam_python()`. The data is the Duo 2018 ARI matrix: 14 clustering methods by 12 datasets. ```{r libs} library(rbeam) @@ -32,7 +32,7 @@ ari <- duo$tensor(list("ari"))[, , 1] ## Mixed-effects variance decomposition -`beam_mixed_effects` fits `score ~ method + (1 | dataset)` in lme4. The dataset random intercept absorbs how easy or hard each dataset is for every method alike, and the dataset intraclass correlation is the share of variance that is the dataset rather than the method. +[`beam_mixed_effects`](https://imallona.github.io/beam/docs/explanations/method-by-dataset-heterogeneity.html) fits `score ~ method + (1 | dataset)` in lme4. The dataset random intercept absorbs how easy or hard each dataset is for every method alike, and the dataset intraclass correlation is the share of variance that is the dataset rather than the method. ```{r mixed-effects, eval = have_lme4, fig.height = 4.5} me <- beam_mixed_effects(ari, method_names, dataset_names) @@ -49,7 +49,7 @@ The dataset intercept takes about 0.71 of the ARI variance, so most of the sprea ## Dataset concordance on ARI -The variance decomposition gives one number for the interaction. `beam_dataset_concordance` shows its structure feature-free: it ranks the methods within each dataset on ARI and compares every pair of per-dataset orderings with Kendall tau-b, reporting which datasets order the methods alike and which stands apart. +The variance decomposition gives one number for the interaction. [`beam_dataset_concordance`](https://imallona.github.io/beam/docs/explanations/dataset-concordance-and-discrimination.html) shows its structure feature-free: it ranks the methods within each dataset on ARI and compares every pair of per-dataset orderings with Kendall tau-b, reporting which datasets order the methods alike and which stands apart. ```{r concordance, fig.height = 5} conc <- beam_dataset_concordance( @@ -68,7 +68,7 @@ The datasets fall into families that order the methods alike, the same structure ## Bradley-Terry tree on dataset features -The variance decomposition says how much interaction there is; the Bradley-Terry tree asks which dataset feature drives it. For each dataset the methods are compared pairwise (higher ARI wins), and `psychotree::bttree` splits the datasets by their features so each leaf gets its own Bradley-Terry ranking. +The variance decomposition says how much interaction there is; the [Bradley-Terry tree](https://imallona.github.io/beam/docs/explanations/method-by-dataset-heterogeneity.html#bradley-terry-trees) asks which dataset feature drives it. For each dataset the methods are compared pairwise (higher ARI wins), and `psychotree::bttree` splits the datasets by their features so each leaf gets its own Bradley-Terry ranking. ```{r bt-fit, eval = have_psychotree, include = FALSE} # Fit in a hidden chunk: the per-leaf standard-error step uses a sandwich vcov @@ -92,7 +92,7 @@ On these twelve datasets the parameter-stability test finds no split, so the tre ## Plackett-Luce on per-dataset rankings -`beam_plackett_luce` reads each dataset as a full ranking of the methods and fits a Plackett-Luce model, the generalisation of Bradley-Terry from pairwise wins to full orderings. The worths sum to one. +[`beam_plackett_luce`](https://imallona.github.io/beam/docs/explanations/method-by-dataset-heterogeneity.html#plackett-luce-full-rankings) reads each dataset as a full ranking of the methods and fits a Plackett-Luce model, the generalisation of Bradley-Terry from pairwise wins to full orderings. The worths sum to one. ```{r pl, eval = have_pl, fig.height = 4} pl <- beam_plackett_luce(ari, method_names) @@ -140,7 +140,7 @@ Even at 36 datasets the parameter-stability test finds no split, so the tree red ## Specification curve and blind analysis -The heterogeneity models read the per-dataset variation directly. The MCDA-side specification curve reads the same benchmark from the choice side: every weighting, aggregation and dataset, with how often the top method holds. Here it runs on the Duo ARI, runtime and Shannon entropy difference. +The heterogeneity models read the per-dataset variation directly. The MCDA-side [specification curve](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html#the-specification-curve) reads the same benchmark from the choice side: every weighting, aggregation and dataset, with how often the top method holds. Here it runs on the Duo ARI, runtime and Shannon entropy difference. ```{r spec-curve} mcda <- reticulate::import("beam.mcda") @@ -162,7 +162,7 @@ cat(sprintf("specification curve: %s ranks first in %.0f%% of %d combinations, % beam_plot(curve, "specification_curve") ``` -Blinding fixes the analysis before the method names are seen, which applies to the heterogeneity models above as much as to the ranking. `beam_blind` hides the names; `beam_unblind` restores them; the result does not change. +[Blinding](https://imallona.github.io/beam/docs/explanations/analysis-blinding.html) fixes the analysis before the method names are seen, which applies to the heterogeneity models above as much as to the ranking. `beam_blind` hides the names; `beam_unblind` restores them; the result does not change. ```{r blind} beam <- reticulate::import("beam") diff --git a/r/beam/vignettes/metric-diagnostics.Rmd b/r/beam/vignettes/metric-diagnostics.Rmd index 37e8e53..493d9b5 100644 --- a/r/beam/vignettes/metric-diagnostics.Rmd +++ b/r/beam/vignettes/metric-diagnostics.Rmd @@ -15,7 +15,7 @@ knitr::opts_chunk$set(eval = have_beam, collapse = TRUE, comment = "#>", dpi = 140, out.width = "100%", fig.width = 7, fig.height = 4.5) ``` -A benchmark that reads a group of metrics as one criterion assumes the group behaves as one scale. beam has three measurement-side checks for that assumption, all built on the same oriented Spearman correlations between the metrics. This vignette runs them on the bundled OpenProblems batch integration scores, where the scIB metrics split into biological conservation and batch correction. The R functions forward to the Python `beam` core; install it once with `install_beam_python()`. +A benchmark that reads a group of metrics as one criterion assumes the group behaves as one scale. beam has [three measurement-side checks](https://imallona.github.io/beam/docs/explanations/metric-set-diagnostics.html) for that assumption, all built on the same oriented Spearman correlations between the metrics. This vignette runs them on the bundled OpenProblems batch integration scores, where the scIB metrics split into biological conservation and batch correction. The R functions forward to the Python `beam` core; install it once with `install_beam_python()`. ```{r libs} library(rbeam) @@ -46,7 +46,7 @@ data.frame( ) ``` -## Validity: is the grouping the right split? +## Validity Convergent validity asks whether same-group metrics agree; discriminant validity asks whether different-group metrics agree less (Campbell and Fiske 1959). @@ -60,7 +60,7 @@ beam_plot(v, "metric_correlation") Metrics are ordered by group; a darker within-group block than between-group cells means the split holds. -## Reliability: does each group hold together as one scale? +## Reliability Standardized Cronbach's alpha per group (Cronbach 1951). A group above the conventional 0.7 cutoff reads as one reliable scale. @@ -71,7 +71,7 @@ beam_plot(rel, "metric_reliability_dropped") Each bar is the group alpha recomputed with one metric removed, one facet per group. A bar above the dashed group-alpha line marks a metric whose removal raises reliability; the dotted line is the 0.7 cutoff. -## Dimensionality: how many factors does each group carry? +## Dimensionality Cronbach's alpha reads a group as one scale when it is high, but it cannot test whether the group is a single factor, and it rises with the number of metrics. `metric_dimensionality` counts the factors by parallel analysis (Horn 1965, Glorfeld 1995) on the metric correlation matrix. The scree plot shows the eigenvalues per group: a single large eigenvalue means one factor. @@ -84,7 +84,7 @@ One line per group joins its eigenvalues. The legend gives the factor count para ## Specification curve and blind analysis -The diagnostics above read the metrics. The MCDA-side specification curve reads the methods: every weighting, aggregation and dataset, with how often the top method holds. COMET is left out because it is slow on a metric-rich task. +The diagnostics above read the metrics. The MCDA-side [specification curve](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html#the-specification-curve) reads the methods: every weighting, aggregation and dataset, with how often the top method holds. COMET is left out because it is slow on a metric-rich task. ```{r spec-curve} mcda <- reticulate::import("beam.mcda") @@ -108,7 +108,7 @@ cat(sprintf("specification curve: %s ranks first in %.0f%% of %d combinations, % beam_plot(curve, "specification_curve") ``` -A blind analysis fixes the weighting and the metric set before the method names are seen. `beam_blind` hides the names; `beam_unblind` restores them; the ranking does not change. +A [blind analysis](https://imallona.github.io/beam/docs/explanations/analysis-blinding.html) fixes the weighting and the metric set before the method names are seen. `beam_blind` hides the names; `beam_unblind` restores them; the ranking does not change. ```{r blind} beam <- reticulate::import("beam") @@ -125,9 +125,9 @@ cat("blinding fingerprint:", substr(run$manifest[["blinding"]][["seal_sha256"]], 1, 12), "\n") ``` -## Dataset concordance: do the datasets agree on the method order? +## Dataset concordance -The metric-side checks above ask whether the metrics cohere. The method-side companion asks whether the datasets agree on how they order the methods. `beam_dataset_concordance` ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. +The metric-side checks above ask whether the metrics cohere. The method-side companion asks whether the datasets agree on how they order the methods. [`beam_dataset_concordance`](https://imallona.github.io/beam/docs/explanations/dataset-concordance-and-discrimination.html) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. ```{r concordance, fig.height = 5} conc <- beam_dataset_concordance( @@ -144,4 +144,4 @@ beam_plot(conc, "dataset_concordance") ## Reading -Reliability and dimensionality dissociate on this data. The biological group has the higher alpha (0.85) but carries two factors by parallel analysis, so its alpha is partly the size of the group rather than one underlying quantity. The batch group has the lower alpha (0.61) but is a single factor its metrics track only weakly. A group can be reliable without being one thing, and one thing without being reliable, which is why the checks read together. See `?beam_metric_validity`, `?beam_metric_reliability`, `?beam_metric_dimensionality`. +Reliability and dimensionality dissociate on this data. The biological group has the higher alpha (0.85) but carries two factors by parallel analysis, so its alpha is partly the size of the group rather than one underlying quantity. The batch group has the lower alpha (0.61) but is a single factor its metrics track only weakly. See `?beam_metric_validity`, `?beam_metric_reliability`, `?beam_metric_dimensionality`. diff --git a/r/beam/vignettes/ranking.Rmd b/r/beam/vignettes/ranking.Rmd index 882579e..967c97a 100644 --- a/r/beam/vignettes/ranking.Rmd +++ b/r/beam/vignettes/ranking.Rmd @@ -15,7 +15,7 @@ knitr::opts_chunk$set(eval = have_beam, collapse = TRUE, comment = "#>", dpi = 140, out.width = "100%", fig.width = 7, fig.height = 4.5) ``` -beam ranks methods from a tool by metric score matrix with multi-criteria decision analysis, then qualifies the ranking with sensitivity and statistical checks. This vignette runs the pipeline on the Duo, Robinson and Soneson (2018) single-cell clustering benchmark bundled with the package: 14 methods, 12 datasets, and the three well-populated metrics ARI (higher is better), runtime and Shannon entropy difference (lower is better). The R functions forward to the Python `beam` core through reticulate; install it once with `install_beam_python()`. +beam ranks methods from a tool by metric score matrix with [multi-criteria decision analysis](https://imallona.github.io/beam/docs/explanations/aggregation-methods.html), then qualifies the ranking with sensitivity and statistical checks. This vignette runs the pipeline on the Duo, Robinson and Soneson (2018) single-cell clustering benchmark bundled with the package: 14 methods, 12 datasets, and the three well-populated metrics ARI (higher is better), runtime and Shannon entropy difference (lower is better). The R functions forward to the Python `beam` core through reticulate; install it once with `install_beam_python()`. ```{r libs} library(rbeam) @@ -46,13 +46,13 @@ run <- beam_rank(scores, weights = "equal", method = "saw", seed = 0L) cat("top-ranked under equal weights / SAW:", run$top_tool, "\n") ``` -The funky heatmap shows the run as a glyph table: methods as rows, metrics as circles sized by the card-resolved normalized score, a composite bar, and the rank-robustness columns (leave-one-dataset-out stability and SMAA rank certainty). It is drawn natively in R with the funkyheatmap package. +The [funky heatmap](https://imallona.github.io/beam/docs/explanations/funky-heatmaps-and-robustness.html) shows the run as a glyph table: methods as rows, metrics as circles sized by the card-resolved normalized score, a composite bar, and the rank-robustness columns (leave-one-dataset-out stability and SMAA rank certainty). It is drawn natively in R with the funkyheatmap package. ```{r funky, fig.width = 8, fig.height = 7.5} plot(run, metric_groups = c("accuracy", "cost", "accuracy")) ``` -## SMAA: does the top method hold across weightings? +## SMAA across weightings SMAA samples weight vectors and reports, per method, the share of weightings that rank it first. @@ -60,7 +60,7 @@ SMAA samples weight vectors and reports, per method, the share of weightings tha beam_plot(run, "smaa") ``` -## Leave-one-dataset-out: does the order depend on one dataset? +## Leave-one-dataset-out stability For each dataset dropped, the rest are pooled and re-ranked. The per-method stability is the share of those runs in which the method keeps its base rank. @@ -99,9 +99,9 @@ cat("the top two tools are within the noise floor:", run$noise_floor$top_pair_indistinguishable, "\n") ``` -## Do the scores honour the cards? +## Card and data consistency -`beam_card_data_consistency` reads the raw scores against what each card declares (range, chance baseline, noise floor), before normalization. A value outside its declared range, or a metric on the wrong scale, is named here rather than distorting the ranking silently. The normalization context comes from the registry. +[`beam_card_data_consistency`](https://imallona.github.io/beam/docs/explanations/card-data-consistency.html) reads the raw scores against what each card declares (range, chance baseline, noise floor), before normalization. A value outside its declared range, or a metric on the wrong scale, is named here rather than distorting the ranking silently. The normalization context comes from the registry. ```{r card-consistency} mcda <- reticulate::import("beam.mcda") @@ -114,9 +114,9 @@ audit <- beam_card_data_consistency( cat("pooled scores consistent with the cards:", audit$ok, "\n") ``` -## What moves the ranking: the dataset or the choice? +## Rank sensitivity: dataset or analyst choice -`beam_rank_sensitivity` splits the ranking's variance between the data and the analyst's choices. It ranks the methods on each dataset, under every weighting and aggregation, and attributes each method's rank variance to the dataset, the weighting, the aggregation and their interactions. The shares sum to one. +[`beam_rank_sensitivity`](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html) splits the ranking's variance between the data and the analyst's choices. It ranks the methods on each dataset, under every weighting and aggregation, and attributes each method's rank variance to the dataset, the weighting, the aggregation and their interactions. The shares sum to one. ```{r rank-sensitivity} rs <- beam_rank_sensitivity( @@ -138,9 +138,9 @@ The shares above are pooled over the methods. The `rank_sensitivity_by_tool` plo beam_plot(rs, "rank_sensitivity_by_tool") ``` -## Dataset concordance: do the datasets agree on the order? +## Dataset concordance -`beam_rank` also reports how the datasets agree among themselves. `dataset_concordance` ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not. The standalone wrapper is `beam_dataset_concordance(tensor, polarity)`. +`beam_rank` also reports how the datasets agree among themselves. [`dataset_concordance`](https://imallona.github.io/beam/docs/explanations/dataset-concordance-and-discrimination.html) ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b. A high mean says the pooled ranking stands in for the individual datasets; a low mean says it does not. The standalone wrapper is `beam_dataset_concordance(tensor, polarity)`. ```{r concordance} conc <- run$dataset_concordance @@ -158,7 +158,7 @@ beam_plot(run, "dataset_struggle") ## Specification curve -`beam_specification_curve` reads the same grid as a list of rankings and reports how often the top method holds. It takes the report from `beam_rank_sensitivity`. +[`beam_specification_curve`](https://imallona.github.io/beam/docs/explanations/rank-sensitivity.html#the-specification-curve) reads the same grid as a list of rankings and reports how often the top method holds. It takes the report from `beam_rank_sensitivity`. ```{r specification-curve} curve <- beam_specification_curve(rs) @@ -176,7 +176,7 @@ The top line is the rank of the most-frequent top method across the sorted grid; ## Blind analysis -A blind analysis fixes the weighting, the aggregation and the metric set before the method names are revealed. `beam_blind` hides the names and shuffles the rows; `beam_unblind` restores them. The ranking does not change, and the seal fingerprint is recorded in the manifest. +A [blind analysis](https://imallona.github.io/beam/docs/explanations/analysis-blinding.html) fixes the weighting, the aggregation and the metric set before the method names are revealed. `beam_blind` hides the names and shuffles the rows; `beam_unblind` restores them. The ranking does not change, and the seal fingerprint is recorded in the manifest. ```{r blinding} blinded <- beam_blind(scores, seed = 0L) @@ -188,9 +188,9 @@ cat("blinding fingerprint:", substr(blind_run$manifest[["blinding"]][["seal_sha256"]], 1, 12), "\n") ``` -## Are the methods separable? Critical difference +## Critical difference on ARI -The composite gives one ranking but does not say whether the methods differ beyond chance. `beam_critical_difference` runs the Friedman test and the Nemenyi post-hoc on the ARI block of methods observed on every dataset. Two methods whose average ranks lie within the critical difference are not separable. +The composite gives one ranking but does not say whether the methods differ beyond chance. [`beam_critical_difference`](https://imallona.github.io/beam/docs/explanations/critical-difference.html) runs the Friedman test and the Nemenyi post-hoc on the ARI block of methods observed on every dataset. Two methods whose average ranks lie within the critical difference are not separable. ```{r cd, fig.width = 7, fig.height = 3.6} ari <- duo$tensor(list("ari"))[, , 1] @@ -203,7 +203,7 @@ cat(sprintf("Friedman p = %.1e, Nemenyi CD = %.2f\n", beam_plot(cd, "critical_difference") ``` -When the matrix has gaps, `beam_skillings_mack` is the coverage-aware version of the same test, run on the full ARI matrix. +When the matrix has gaps, [`beam_skillings_mack`](https://imallona.github.io/beam/docs/explanations/critical-difference.html#skillings-mack-for-incomplete-blocks) is the coverage-aware version of the same test, run on the full ARI matrix. ```{r skillings-mack} sm <- beam_skillings_mack(ari, higher_is_better = TRUE, method_names = method_names) @@ -211,9 +211,9 @@ cat(sprintf("Skillings-Mack chi-squared = %.2f on %d df, p = %.1e\n", sm$statistic, as.integer(sm$df), sm$p_value)) ``` -## How often does one method outperform another? +## Pairwise superiority on ARI -The critical-difference test says which methods are separable, not by how much or how often. `beam_pairwise_superiority` compares the methods two at a time across the datasets, with the ARI noise floor as the equivalence band: a method counts as outperforming another on a dataset only when the difference clears that floor. The probability of superiority is the fraction of datasets on which one method outperforms the other. +The critical-difference test says which methods are separable, not by how much or how often. [`beam_pairwise_superiority`](https://imallona.github.io/beam/docs/explanations/pairwise-method-comparison.html) compares the methods two at a time across the datasets, with the ARI noise floor as the equivalence band: a method counts as outperforming another on a dataset only when the difference clears that floor. The probability of superiority is the fraction of datasets on which one method outperforms the other. ```{r pairwise} ari_floor <- cards$properties_for(list("ari"))[[1]]$noise_floor @@ -226,11 +226,11 @@ cat("pairs the sign test cannot separate:", length(sup$equivalent_pairs), "of", length(sup$per_pair), "\n") ``` -Most pairs are not distinguishable: across the twelve datasets the ARI differences are often inside the floor, so neither method outperforms the other decisively. This is the critical-difference reading as an effect size, and it agrees with it. +Most pairs are not distinguishable: across the twelve datasets the ARI differences are often inside the floor, so neither method outperforms the other decisively. This is the critical-difference reading as an effect size. -## Do the pairwise majorities agree with one order? +## Pairwise transitivity -`beam_pairwise_transitivity` reads the same pairwise relation and asks whether one order is consistent with it: the method preferred to every other one, the circular triads, and whether the relation is transitive. +[`beam_pairwise_transitivity`](https://imallona.github.io/beam/docs/explanations/pairwise-method-comparison.html#transitivity) reads the same pairwise relation and asks whether one order is consistent with it: the method preferred to every other one, the circular triads, and whether the relation is transitive. ```{r transitivity} trans <- beam_pairwise_transitivity(sup) @@ -240,9 +240,9 @@ cat("transitive:", trans$is_transitive, "; circular triads:", trans$n_circular_triads, "of", trans$n_triads, "\n") ``` -## How sure are we that one method is better? +## Bayesian sign comparison -The tests above report a p-value. `beam_bayesian_sign_comparison` reads the same pairwise relation on the probability scale. For each pair it gives the posterior probability that one method is practically better, that the two are practically equivalent within the floor, or that the other is practically better (Benavoli et al. 2017). Plot it with `beam_plot(bayes, "bayesian_comparison")`. +The tests above report a p-value. [`beam_bayesian_sign_comparison`](https://imallona.github.io/beam/docs/explanations/pairwise-method-comparison.html#bayesian-sign-comparison) reads the same pairwise relation on the probability scale. For each pair it gives the posterior probability that one method is practically better, that the two are practically equivalent within the floor, or that the other is practically better (Benavoli et al. 2017). Plot it with `beam_plot(bayes, "bayesian_comparison")`. ```{r bayesian, fig.height = 5} bayes <- beam_bayesian_sign_comparison(sup) diff --git a/src/beam/mcda/comet.py b/src/beam/mcda/comet.py index 929c031..5b2954f 100644 --- a/src/beam/mcda/comet.py +++ b/src/beam/mcda/comet.py @@ -120,7 +120,7 @@ def comet( The expert function is the weighted sum of a characteristic object's coordinates, a deterministic stand-in for the human pairwise judgement that COMET assumes. See the module docstring and - docs/explanations/comet.md. + docs/explanations/aggregation-methods.md#comet. References ----------