Documentation

.gitignore

@@ -76,3 +76,6 @@ r/beam/vignettes/*.html
 
 # Generated example reports (built in CI by scripts/generate_example_reports.py)
 docs/reports/*.html
+
+/.quarto/
+**/*.quarto_ipynb

README.md

@@ -82,7 +82,7 @@ beam_run("beam.yaml")
 - [Documentation site](https://imallona.github.io/beam/): vignettes, how-tos, explanations, and the Python API reference.
 - Ontology release: `docs/beam.owl.ttl` (OWL) and `docs/beam.skos.ttl` (SKOS), regenerated from the cards on each release.
 
-## Licence
+## License
 
 - Code: GPL-3.0-or-later (`LICENSE`).
 - Metric cards under `src/beam/metrics/`: CC-BY-4.0 (`src/beam/metrics/LICENSE.md`).

_quarto.yml

@@ -50,54 +50,32 @@ website:
             text: Normalization and scales
           - href: docs/explanations/aggregation-methods.md
             text: Aggregation methods
-          - href: docs/explanations/aggregation-agreement.md
-            text: Aggregation agreement
-          - href: docs/explanations/normalization-agreement.md
-            text: Normalization agreement
+          - href: docs/explanations/choice-agreement.md
+            text: Choice agreement
           - href: docs/explanations/rank-sensitivity.md
-            text: What moves the ranking
-          - href: docs/explanations/specification-curve.md
-            text: Specification curve
+            text: Rank sensitivity and the specification curve
           - href: docs/explanations/analysis-blinding.md
             text: Analysis blinding
           - href: docs/explanations/weighting-schemes.md
             text: Weighting schemes
-          - href: docs/explanations/comparing-methods-across-datasets.md
-            text: Comparing methods across datasets
-          - href: docs/explanations/dataset-concordance.md
-            text: Dataset concordance
-          - href: docs/explanations/dataset-discrimination.md
-            text: Dataset discrimination
+          - href: docs/explanations/critical-difference.md
+            text: Critical difference
+          - href: docs/explanations/dataset-concordance-and-discrimination.md
+            text: Dataset concordance and discrimination
           - href: docs/explanations/attribution-synthesis.md
             text: Attribution synthesis
-          - href: docs/explanations/skillings-mack.md
-            text: Skillings-Mack
-          - href: docs/explanations/pairwise-superiority.md
-            text: Pairwise superiority
-          - href: docs/explanations/transitivity.md
-            text: Pairwise transitivity
-          - href: docs/explanations/bayesian-comparison.md
-            text: Bayesian comparison
+          - href: docs/explanations/pairwise-method-comparison.md
+            text: Pairwise method comparison
           - href: docs/explanations/missing-data.md
             text: Missing data
           - href: docs/explanations/reference-levels.md
             text: Reference levels
           - href: docs/explanations/card-data-consistency.md
             text: Card and data consistency
-          - href: docs/explanations/convergent-discriminant-validity.md
-            text: Metric validity
-          - href: docs/explanations/reliability.md
-            text: Metric reliability
-          - href: docs/explanations/dimensionality.md
-            text: Metric dimensionality
-          - href: docs/explanations/heterogeneity-mixed-effects.md
-            text: Mixed-effects heterogeneity
-          - href: docs/explanations/heterogeneity-bradley-terry.md
-            text: Bradley-Terry trees
-          - href: docs/explanations/full-rankings-and-bounded-metrics.md
-            text: Plackett-Luce and bounded metrics
-          - href: docs/explanations/comet.md
-            text: COMET
+          - href: docs/explanations/metric-set-diagnostics.md
+            text: Metric-set diagnostics
+          - href: docs/explanations/method-by-dataset-heterogeneity.md
+            text: Method-by-dataset heterogeneity
           - href: docs/explanations/funky-heatmaps-and-robustness.md
             text: Funky heatmaps
           - href: docs/explanations/ontology-mappings.md

docs/explanations/aggregation-methods.md

@@ -1,6 +1,6 @@
 # Aggregation methods
 
-After [normalizing](normalization-and-scales.md) the metrics results, beam holds a tool by metric matrix in the unit interval, with every column oriented so higher is better, plus a weight per metric. Aggregation turns that matrix into one preference score per tool, which then becomes a ranking. beam offers four aggregation methods on this page: SAW, TOPSIS, VIKOR and PROMETHEE II ([COMET](comet.md) has its own page). They differ in what they compute and in what they assume about the scale of the normalized scores.
+After [normalizing](normalization-and-scales.md) the metrics results, beam holds a tool by metric matrix in the unit interval, with every column oriented so higher is better, plus a weight per metric. Aggregation turns that matrix into one preference score per tool, which then becomes a ranking. beam offers five aggregation methods: SAW, TOPSIS, VIKOR, PROMETHEE II and COMET. They differ in what they compute and in what they assume about the scale of the normalized scores.
 
 beam wraps pymcdm for these methods rather than reimplementing their algorithms. Each beam function calls the matching pymcdm method with an identity normalization, so pymcdm runs on beam's already normalized matrix, and with every metric typed as positive because the matrix is oriented higher is better. beam keeps the shared input and output contract, applies the higher-is-better convention described below, and handles the corner cases (for example a single tool) that pymcdm does not.
 
@@ -14,7 +14,7 @@ Simple additive weighting (SAW) is the dot product of the normalized scores and
 
 ## TOPSIS
 
-TOPSIS measures each tool's distance to two reference points. It weights the matrix, then finds the ideal solution (the best value in each weighted column) and the anti-ideal (the worst). Each tool's score is its relative closeness: the distance to the anti-ideal divided by the sum of the distances to the ideal and the anti-ideal. A tool near the ideal and far from the anti-ideal scores near 1. Like SAW, TOPSIS uses Euclidean distance on the weighted values, so it assumes an interval scale where differences are comparable across metrics. As a key difference with SAW, TOPSIS rewards a tool for being close to the best on every metric at once, not just for a high total. So a balanced tool can rank above a tool with one very high and one very low score, even when their sums match.
+TOPSIS measures each tool's distance to two reference points. It weights the matrix, then finds the ideal solution (the best value in each weighted column) and the anti-ideal (the worst). Each tool's score is its relative closeness: the distance to the anti-ideal divided by the sum of the distances to the ideal and the anti-ideal. A tool near the ideal and far from the anti-ideal scores near 1. Like SAW, TOPSIS uses Euclidean distance on the weighted values, so it assumes an interval scale where differences are comparable across metrics. Unlike SAW, TOPSIS rewards a tool for being close to the best on every metric at once, not just for a high total. So a balanced tool can rank above a tool with one very high and one very low score, even when their sums match.
 
 ## VIKOR
 
@@ -32,9 +32,17 @@ PROMETHEE II compares every ordered pair of tools rather than scoring each tool
 
 Because the usual preference function looks only at the sign of each difference, PROMETHEE II in this default form is ordinal per metric. That is, it uses the order of the tools on a metric, not the size of the gaps. This makes it the lightest method on scale assumptions of the four. Brans and Vincke define five other preference functions that soften this with indifference and preference thresholds in the metric's own units, which add back the size of the differences. 
 
+## COMET
+
+COMET, the Characteristic Objects Method (Salabun 2015), is the one aggregation whose ranking is rank reversal free: adding or removing a tool cannot change the order of the others. The four methods above score each tool against the other tools present, so the reference they use shifts when the set of tools changes. That matters for a growing leaderboard, such as community benchmarking in hackathons, or a sensitivity analysis that drops one method at a time. COMET instead fits on a fixed grid of reference points, the characteristic objects, and scores the tools against that grid afterward. beam wraps pymcdm for it.
+
+The grid is the Cartesian product of a few characteristic values per metric. beam uses the endpoints of the normalized scale, $0$ and $1$, by default, so with two metrics the objects are the four corners of the unit square; a caller can pass an interior value such as $0.5$ to add anchors. Ordering the objects needs a judgement of which is better, supplied in COMET's original form by a human expert comparing them pairwise. A benchmarking pipeline has none, so beam uses a deterministic rule: one characteristic object outranks another when the weighted sum of its coordinates is larger. Each tool is then scored by triangular fuzzy interpolation between the surrounding objects, a preference in $[0, 1]$ where higher is better. Because the grid and its ordering are fixed before any tool is scored, two tools receive the same scores whatever else is in the table.
+
+COMET costs more than the others. The number of characteristic objects grows as the product of the per-metric value counts, so it is exponential in the number of metrics: ten metrics with two values each is already $2^{10} = 1024$ objects, and the scores are harder to read by hand than a weighted sum. It suits cases where rank stability matters more than simplicity.
+
 ## Choosing an aggregation method
 
-SAW is the easiest choice when transparency matters and the metrics are on a comparable interval scale. TOPSIS and VIKOR both reward balance across metrics, with VIKOR adding explicit control for how much a single weak metric should count. PROMETHEE II is the most conservative about scale, since in its default form it reads only the order of the tools on each metric. Running more than one method and comparing the rankings is itself a sensitivity check: a recommendation that holds across methods is more reliable than one that depends on the choice of aggregation. This is what [aggregation agreement](aggregation-agreement.md) measures. 
+SAW is the easiest choice when transparency matters and the metrics are on a comparable interval scale. TOPSIS and VIKOR both reward balance across metrics, with VIKOR adding explicit control for how much a single weak metric should count. PROMETHEE II is the most conservative about scale, since in its default form it reads only the order of the tools on each metric. COMET is the choice when the tool set changes over time and the order should not shift as tools are added or dropped, at the cost of the exponential grid. Running more than one method and comparing the rankings is itself a sensitivity check: a recommendation that holds across methods is more reliable than one that depends on the choice of aggregation. This is what [aggregation agreement](choice-agreement.md) measures. 
 
 The nature of the input metrics and their redundancy, e.g. speed or memory versus biologically-driven evaluation, and their (lack of) correlation, impact the aggregation method choice.
 
@@ -44,4 +52,5 @@ The nature of the input metrics and their redundancy, e.g. speed or memory versu
 - Opricovic, S. Multicriteria optimization of civil engineering systems. Faculty of Civil Engineering, Belgrade (1998).
 - Opricovic, S. and Tzeng, G.-H. Compromise solution by MCDM methods: a comparative analysis of VIKOR and TOPSIS. European Journal of Operational Research (2004). DOI [10.1016/S0377-2217(03)00020-1](https://doi.org/10.1016/S0377-2217%2803%2900020-1).
 - Brans, J.-P. and Vincke, P. A preference ranking organisation method: the PROMETHEE method for multiple criteria decision-making. Management Science (1985). DOI [10.1287/mnsc.31.6.647](https://doi.org/10.1287/mnsc.31.6.647).
+- Salabun, W. The Characteristic Objects Method: a new distance-based approach to multicriteria decision-making problems. Journal of Multi-Criteria Decision Analysis (2015). DOI [10.1002/mcda.1525](https://doi.org/10.1002/mcda.1525).
 - OECD. Handbook on Constructing Composite Indicators (2008), on weighting and aggregation. DOI [10.1787/9789264043466-en](https://doi.org/10.1787/9789264043466-en).

docs/explanations/analysis-blinding.md

@@ -2,7 +2,7 @@
 
 If a benchmarker can see which method is which while choosing the [weighting](weighting-schemes.md), the [aggregation](aggregation-methods.md) and the metric set, those choices can shift toward a preferred method. This can happen via unconscious biases and without intent. Different choices give different rankings, and it is easy to settle on the one that fits an expected result (Gelman and Loken 2013; Simmons, Nelson and Simonsohn 2011). One way to avoid it is blind analysis: run the analysis on methods whose labels are hidden, then reveal the labels (MacCoun and Perlmutter 2015; Klein and Roodman 2005).
 
-`beam.blind` applies this to a score table. It replaces the tool names with opaque labels and shuffles the rows under a seed, so neither the names nor the order carry the methods' identity. The analyst runs the full beam pipeline on the blinded scores, fixes the configuration, then calls `beam.unblind` with the seal to restore the true names.
+`beam.blind` applies this to a score table. It replaces the tool names with opaque labels and shuffles the rows under a seed, so neither the names nor the order carry the methods' identity. The analyst runs the full beam pipeline on the blinded scores, fixes the configuration, then calls `beam.unblind` with the seal to restore the true names. Unblinding only renames the rows and does not re-rank the methods.
 
 ```python
 import beam
@@ -27,11 +27,7 @@ beam is controlled by users and not the way around. To add an extra layer of aud
 "blinding": {"blinded": true, "seal_sha256": "6ee805357098..."}
 ```
 
-The seal file, kept separately, records that the configuration was fixed before the labels were revealed. A reviewer who has the manifest and the seal can confirm the analysis ran on scores blinded under that seal. This mechanisms suits preregistration well, but does not enforce it.
-
-## Why the blinding does not affect results
-
-Unblinding only renames the rows and does not re-rank methods.
+The seal file, kept separately, records that the configuration was fixed before the labels were revealed. A reviewer who has the manifest and the seal can confirm the analysis ran on scores blinded under that seal. This mechanism suits preregistration well, but does not enforce it.
 
 ## Usage
 
@@ -43,7 +39,7 @@ beam unblind result.json --seal seal.json --out result.unblinded.json
 
 ## See also
 
-- [Specification curve](specification-curve.md)
+- [Specification curve](rank-sensitivity.md#the-specification-curve)
 - [Rank sensitivity](rank-sensitivity.md)
 
 ## References