Skip to content

docs: retire two post-#707 residuals (stale 0.431 board, dead run_one_job.py)#709

Merged
github-actions[bot] merged 2 commits into
developfrom
docs/residuals-after-707
Jul 10, 2026
Merged

docs: retire two post-#707 residuals (stale 0.431 board, dead run_one_job.py)#709
github-actions[bot] merged 2 commits into
developfrom
docs/residuals-after-707

Conversation

@frapercan

Copy link
Copy Markdown
Owner

Why

PR #707 resealed the results docs on the sealed headline (f_micro_w 0.4063, first in 7 of 9 cells, frame v227 to v230). Two residuals survived it, both flagged in #707's own "known residuals" note.

Defect 1: architecture/evaluation.rst

The stale 0.431 board is removed, not moved, because the record already exists. The full superseded board (the k-ablation, scoring configs, the three re-ranker generations, the external-tool benchmark carrying the 0.431 Fmax, and the leakage analysis) is already preserved verbatim in historical/pre-v227.rst under a dated "superseded, retained for provenance only" banner created by #707. The copy in evaluation.rst was a duplicate of that record. Deleting the duplicate and pointing to the historical page preserves the record while removing the superseded number from a page a reader would read as current. The protocol page keeps the temporal-leakage caveat as a protocol-level point (why external comparisons leak, why references freeze at t0), minus the specific stale figures.

Discrepancy reported: the task described two "Provisional" admonitions on this page. There is only one literal .. admonition:: Provisional results, pending final recompute (and #707's own PR body refers to it in the singular). I resolved that one and also corrected a related stale seealso that framed /results as the "GOA 220 to 229 temporal holdout"; /results is now the sealed v227 to v230 board, so that bullet was retargeted and a link to the historical figures added.

The one admonition read:

Provisional results, pending final recompute (class: warning)
The Fmax values in the tables below were produced before the 2026-04-10 unification of the embedding-backend slicing convention (see operations, section Residue-tensor convention). They will be regenerated end-to-end for the Zenodo deposit accompanying the thesis. The experimental protocol is stable; only the numbers will change. See /results for the full provisional notice.

Resolution: it is no longer "provisional pending recompute". That board is a different metric on a different, superseded frame; it was never going to become the sealed board, so there is nothing to recompute into it. The honest state is superseded, which is exactly how the historical page frames it. The admonition (and its dangling pointer to a "full provisional notice" on /results, which no longer exists) was removed together with the duplicated board, and the page now points to the sealed board and the retained historical record.

Defect 2: complexity/measuring.rst

scripts/run_one_job.py was cited three times as a profiling target. It does not exist. Git history: it was added in the initial commit, migrated in b08f536, and deleted in 80ed10e ("feat: CI/CD, linting, ngrok expose, and test fixes"). Its old body called worker.handle_job(job_id) directly against a hardcoded 3-op registry, bypassing the queue, the exact ad-hoc pattern the house rules now forbid.

There is no single-job CLI wrapper. The task named the internal method execute_single(job_id); that name does not exist anywhere in the repo. The real internal entry point is BaseWorker.handle_job(job_id) in protea/workers/base_worker.py, which is what the page now points at.

What replaced the dead command (all verified to exist):

  • Dispatch one job via POST /jobs with {operation, queue_name, payload} (the only sanctioned dispatch path; the request model and {"id", "status"} response were read from protea/api/routers/jobs.py).
  • Profile the worker that consumes it: poetry run scalene ... scripts/worker.py --queue <queue>. scalene is a real project dependency (pyproject.toml, locked at 2.3.0); scripts/worker.py --queue is the documented worker command; the worker catches KeyboardInterrupt, logs Worker stopped., and exits cleanly (read in scripts/worker.py), so scalene's on-exit report is written after Ctrl+C.
  • Function-level fallback via stdlib cProfile + pstats wrapping the same worker.
  • Removed the pyinstrument and snakeviz commands: neither is a project dependency, so poetry run pyinstrument / snakeviz were unverifiable and I would not print them.
  • Added a Known gap rubric: no CLI runs a single job by id and exits; scripts/run_one_job.py was removed in 80ed10e; the internal entry point is BaseWorker.handle_job(job_id); until a wrapper lands, profile the continuous worker or read per-step timings from the JobEvent log (which needs no profiler).

Validation

  • scripts/check_no_em_dashes.py: OK, 0 violations on both files.
  • scripts/check_doc_refs.py: 199 broken refs, unchanged from the clean-develop baseline. All 5 contributed by evaluation.rst are pre-existing autodoc :module:/:func: roles I did not touch; measuring.rst contributes 0. My new :doc: cross-refs (/results, /historical/pre-v227, /operate/reproduce-0.4063) all resolve.
  • Docs only; no code, no database.

Things I was told that turned out false

  1. "Two Provisional admonitions" in evaluation.rst: there is only one.
  2. "base_worker.py has an internal execute_single(job_id) method": no such method exists; the real entry point is handle_job(job_id). Documented the real one; fabricated nothing.

…_job.py)

Two residuals survived PR #707's results reseal.

architecture/evaluation.rst: the benchmark section still carried a stale
0.431 Fmax board on the superseded GOA 220 to 229 frame under a "Provisional
results, pending final recompute" admonition, plus its leakage tables. That
board is already preserved verbatim in historical/pre-v227.rst under a
superseded banner, so the duplicate is removed from the protocol page and
replaced with a pointer to the sealed board (/results) and the retained
historical record. The admonition was resolved by declaring the frame
superseded (not provisional): the numbers were a different metric and window,
never the sealed board, so there was nothing to "recompute". The temporal
leakage caveat is kept as a protocol-level point (why external comparisons
leak, why references freeze at t0) without the stale figures. The seealso
that framed /results as "GOA 220 to 229" is corrected to the sealed
v227 to v230 board, with a link to the historical figures.

complexity/measuring.rst: dropped three citations of scripts/run_one_job.py,
which does not exist (removed in 80ed10e). Documented the real supported path
to measure a single job: POST /jobs to enqueue, then run scripts/worker.py
for that queue under scalene (the one bundled profiler) or stdlib cProfile,
stopping the worker with Ctrl+C after the job reaches SUCCEEDED (it exits
cleanly, so the report is written). Removed pyinstrument/snakeviz commands
(neither is a project dependency, so they were unverifiable). Added a
"Known gap" note pointing at the internal entry point BaseWorker.handle_job
in protea/workers/base_worker.py and flagging the missing single-job CLI.
@github-actions github-actions Bot enabled auto-merge (squash) July 10, 2026 07:35
check_openapi_refs treats a documented route absent from the OpenAPI
schema as a phantom and fails lint. The page wrote GET /jobs/<job-uuid>
with a literal placeholder; the schema exposes /jobs/{job_id}.
@github-actions github-actions Bot merged commit 70721a4 into develop Jul 10, 2026
14 checks passed
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.85%. Comparing base (b4a9251) to head (8038f46).
⚠️ Report is 19 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #709      +/-   ##
===========================================
- Coverage    88.90%   88.85%   -0.05%     
===========================================
  Files          237      245       +8     
  Lines        17692    18481     +789     
===========================================
+ Hits         15729    16422     +693     
- Misses        1963     2059      +96     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant