Add interface contact probability scores#22
Conversation
|
Pushed review-response commit 85b129d. Main updates: AF2 distogram scoring now uses the lower-bound <8A convention with logsumexp, compressed AF2 pickles are covered by tests, AF3 contact_probs alignment prefers token_res_ids and ignores extra non-residue tokens, summaries only symmetrize the requested chain-pair block, and the benchmark script now uses this checkout explicitly, complete-case metrics, tie-aware AP, checkpointing, and per-model error handling. Benchmark status: the first full CSV in contact_probability_benchmark finished, but it imported the dirty local checkout and AF2 contact scores were present for only 12 rows, so I treat it as preliminary AF3-only signal. Corrected rerun is in /g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark_pr22_fix. Smoke job 58939682 completed; full Slurm job 58939710 is running. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6fce50a27d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Resolves #21.
Summary
Confidence/Interfaceand write four new CSV columns:interface_contact_prob_sourceinterface_contact_prob_maxinterface_contact_prob_top10_meaninterface_expected_contactscontact_probsfromconfidences.json, align token matrices to residue matrices where possible, symmetrize, and expose asaf3_contact_probs.< 8 A, matching the AlphaPulldown diagnostics fallback convention, and expose asaf2_distogram_lt_8A.interface_meta_scorefor now; the benchmark comparison should decide how useful/calibrated they are.scripts/evaluate_contact_probability_scores.pyfor labelled benchmark comparison without mutating prediction directories. It can score all chain pairs, including pairs without coordinate-detected contacts, which is useful for interaction benchmarks.Validation
python -m compileall -q src/alphajudge scripts/evaluate_contact_probability_scores.pypytest -q test/test_parsers_and_runner.py::test_af2_distogram_contact_probs_softmax_cutoff_is_deterministic test/test_parsers_and_runner.py::test_contact_probability_scores_math_is_deterministic test/test_parsers_and_runner.py::test_af3_parser_accepts_alphapulldown_layout test/test_parsers_and_runner.py::test_af3_contact_probability_scores_match_raw_contact_probs->4 passed in 21.38snanaf3_contact_probs, max0.16, top10 mean0.146, expected contacts140.22Slurm benchmark jobs
58905391, human AF3, one pos/one neg per group, output under/g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark/.58905885, dependencyafterok:58905391, output CSVs:/g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark/contact_probability_scores.csv/g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark/contact_probability_metrics.csv