v0.1.7.1 — system re-scores against certified-translator FLORES refs

Companion to v0.1.7 — finalizes all system evaluations against the new certified-translator FLORES references.

Changes

• Re-ran NLLB-200-distilled-600M for all 8 extension pairs (101 translations, 371 skipped via skip_no_ref); scored against v0.1.7 refs → 16/16 pair coverage
• Re-ran Helsinki-NLP/opus-mt for all 8 extension pairs (101 translations, 1 207 s total) → 16/16 pair coverage
• Re-scored DeepL and identity-baseline against v0.1.7 manifest
• Fixed skip_no_ref bug in run_nllb (parameter declared but unused in loop body)
• Rebuilt leaderboard (chrF); COMET-Kiwi re-score pending

Scores (chrF, oracle-layout, v0.1.7)

System	LTB-100 [95% CI]	chrF	Coverage
DeepL	78.20 [75.2, 81.7]	56.40	6/16
NLLB-200-distilled-600M	73.71 [72.5, 74.9]	47.61	16/16
Helsinki-NLP/opus-mt	68.60 [67.1, 70.0]	37.26	16/16
identity-baseline	50.38 [50.2, 50.6]	0.62	16/16

Notable per-pair findings

• en-uz: chrF ≈ 0 for both NLLB (0.07) and opus-mt (0.04) — Uzbek-Latin is a structural zero for 600M-class open models
• en-ko: NLLB 36.88 chrF vs opus-mt 2.58 — NLLB wins decisively; Helsinki opus-mt-en-ko model is inadequate
• en-id / en-ru: both systems score well (chrF 60–70+), confirming FLORES refs are clean

All scores: 1 000-resample percentile bootstrap, seed 42.

LTB v0.1.7 — certified-translator refs for all 16 LTB pairs

Major dataset milestone: every LTB language pair now has at least 10 certified-translator reference documents. Closes the en-ru gap from v0.1.6.4 (was N=0 after script-validation cleanup) and upgrades all 7 surviving extension pairs from ml-curated to certified-translator quality.

What changed

Released scripts/add_flores_extension_refs.py — a one-shot, idempotent enrichment script that matches each region's English text in FLORES eng_Latn.devtest and pulls the parallel translation from each v0.1.6 extension-pair FLORES file. 408 region-pair refs added across 10 FLORES-derived docs, 0 unmatched.

Per-pair coverage at v0.1.7

Pair	v0.1.6.4	v0.1.7	Δ
en-ru	0	10	+10 (FLORES-only)
en-ko, en-vi, en-id, en-ur, en-uz, en-kk, en-zh-tw	3 each	13 each	+10 each (10 FLORES + 3 rileykim)
Core 8 pairs	20–28	unchanged	—

Reference provenance now

Version	Doc range	Provenance	Grade
v0.1 / v0.1.3	001–010	Author-curated	Native-speaker non-professional
v0.1.4	011–025	rileykim/multilingual-document	ml-curated
v0.1.5	026–035 (core 8)	FLORES-200	certified-translator
v0.1.7	026–035 (extension 8)	FLORES-200	certified-translator
v0.1.6	036–059	rileykim/multilingual-document	ml-curated

Methodology roadmap updates

• Critique #1 (sample size) → ✅ Largely fixed at v0.1.7 (all pairs N≥10 certified)
• Critique #8 (single-author-curated ref) → ⚠️ Partially addressed across v0.1.5 + v0.1.7

See docs/methodology-roadmap.md.

Out of scope for this release

Leaderboard scores remain at v0.1.6 numbers. Re-running NLLB / opus-mt / identity / qwen-vl against the new references is a follow-on scoring milestone.

Citation

@misc{ltbench2026,
  title  = {LayoutTranslateBench: A Benchmark for Document Translation with Layout Preservation},
  year   = {2026},
  url    = {https://github.com/Lawrenzho-bit/LayoutTranslateBench},
  note   = {Version 0.1.7}
}

LTB v0.1.6.4 — 16 language pairs + commercial-safe MT runner

LayoutTranslateBench (LTB) is the first public benchmark for document translation that scores layout fidelity and reading order alongside translation quality. This release is the v0.1.6 series checkpoint — the canonical citeable artifact for the 16-pair coverage tier.

What landed in v0.1.6

Sub-release	Headline
v0.1.6	LangPair extended from 8 to 16. Added: en-ru, en-ko, en-vi, en-id, en-ur, en-uz, en-kk, en-zh-tw. CORE_LANG_PAIRS frozen at the v0.1 core 8 so author-curated docs keep clean coverage. Language-detection gate gets script rules for non-Latin extension pairs. NLLB code map covers all 16.
v0.1.6.1	NLLB-200 extension rerun + macro/micro aggregation consistency fix.
v0.1.6.2	Helsinki-NLP/opus-mt runner — 14 models, full 16/16 coverage, commercial-safe (Apache-2.0 / CC-BY-4.0). Ship-able where NLLB-200's CC-BY-NC-4.0 is not.
v0.1.6.3	Methodology note documenting opus-mt vs NLLB per-pair quality variance — competitive on European pairs, dramatically weaker on Asian / Central Asian.
v0.1.6.4	fugumt en-ja swap (replaced opus-mt-en-jap with staka/fugumt-en-ja). Script-validation gate at ingest + post-hoc scripts/validate_extension_refs.py drops rileykim refs whose tgt_text script doesn't match the expected target-language script. Surfaced en-ru rileykim rows that were actually Chinese.

Coverage after v0.1.6.4

Group	Pairs	N per pair
Core 8	en-es, en-de, en-ar, en-fr, en-th, en-ms	N=20
Core 8 (extended)	en-ja	N=28
Core 8 (extended)	en-zh	N=27
Extension 7 (surviving)	en-ko, en-vi, en-id, en-ur, en-uz, en-kk, en-zh-tw	N=3 each
Extension 1 (dropped)	en-ru	N=0 (rileykim labeling bug, refs in Chinese)

Leaderboard — oracle-layout (COMET-Kiwi-22, higher = better)

System	LTB-100 [95% CI]	Coverage
nllb-text-oracle-nllb-200-distilled-600m	86.95 [85.8, 88.0]	15/16
deepl-text-oracle	84.89 [81.4, 88.0]	6/16
opus-mt-text-oracle	80.46 [78.9, 82.1]	15/16

Leaderboard — end-to-end (COMET-Kiwi-22)

System	LTB-100 [95% CI]	Coverage
identity-baseline	50.66 [50.4, 51.0]	15/16
qwen3-vl-2b-instruct	22.79 [18.6, 27.1]	8/16

Methodology

• Two parallel leaderboards (chrF + COMET-Kiwi-22)
• Bootstrap 95% CIs via 1000 resamples (seed 42)
• Coverage-aware Kendall τ (no free credit for single-region fallbacks)
• Language-detection penalty on chrF (latin bleed-through scores 0)
• Oracle-layout vs end-to-end systems segregated on the leaderboard
• Ingest-time + post-hoc script validation for mined references

Full critique tracker at docs/methodology-roadmap.md.

Citation

@misc{ltbench2026,
  title  = {LayoutTranslateBench: A Benchmark for Document Translation with Layout Preservation},
  year   = {2026},
  url    = {https://github.com/Lawrenzho-bit/LayoutTranslateBench},
  note   = {Version 0.1.6.4}
}

What's next (v0.2 scope)

• Extension pair coverage to N≥10 each (certified or industry-grade refs)
• Multi-reference scoring (2 certified-translator refs per doc on the core 8)
• DA / SQM human evaluation on 50 doc-pair outputs across 4+ systems
• LPIPS visual-fidelity metric (10% weight) + OCR round-trip metric (10% weight)
• Held-out split rotation (20% private, refreshed quarterly)

Links

• Live leaderboard: https://lawrenzho-bit.github.io/LayoutTranslateBench/
• Spec: BENCHMARK.md
• Submission guide: docs/submission.md