Skip to content

Add interface contact probability scores#22

Merged
DimaMolod merged 6 commits into
mainfrom
codex/issue-21-contact-probability
Jul 10, 2026
Merged

Add interface contact probability scores#22
DimaMolod merged 6 commits into
mainfrom
codex/issue-21-contact-probability

Conversation

@DimaMolod

Copy link
Copy Markdown
Collaborator

Resolves #21.

Summary

  • Add contact-probability support to Confidence / Interface and write four new CSV columns:
    • interface_contact_prob_source
    • interface_contact_prob_max
    • interface_contact_prob_top10_mean
    • interface_expected_contacts
  • AF3: read native contact_probs from confidences.json, align token matrices to residue matrices where possible, symmetrize, and expose as af3_contact_probs.
  • AF2: derive contact probabilities from full result-pickle distogram logits using softmax mass for bins with upper bound < 8 A, matching the AlphaPulldown diagnostics fallback convention, and expose as af2_distogram_lt_8A.
  • Keep these scores out of interface_meta_score for now; the benchmark comparison should decide how useful/calibrated they are.
  • Add scripts/evaluate_contact_probability_scores.py for labelled benchmark comparison without mutating prediction directories. It can score all chain pairs, including pairs without coordinate-detected contacts, which is useful for interaction benchmarks.

Validation

  • python -m compileall -q src/alphajudge scripts/evaluate_contact_probability_scores.py
  • pytest -q test/test_parsers_and_runner.py::test_af2_distogram_contact_probs_softmax_cutoff_is_deterministic test/test_parsers_and_runner.py::test_contact_probability_scores_math_is_deterministic test/test_parsers_and_runner.py::test_af3_parser_accepts_alphapulldown_layout test/test_parsers_and_runner.py::test_af3_contact_probability_scores_match_raw_contact_probs -> 4 passed in 21.38s
  • Runner smoke on copied AF2/AF3 fixtures with PNG/biophysical extras disabled:
    • AF2 fixture without result pickle: contact source blank, contact scores nan
    • AF3 fixture: af3_contact_probs, max 0.16, top10 mean 0.146, expected contacts 140.22
  • I also started a broader five-test run including the existing full schema runner test; the four contact-focused tests passed, and I interrupted the older broad runner path after 3:47 while it was CPU-bound in existing expensive numeric work.

Slurm benchmark jobs

  • Smoke job: 58905391, human AF3, one pos/one neg per group, output under /g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark/.
  • Full benchmark job: 58905885, dependency afterok:58905391, output CSVs:
    • /g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark/contact_probability_scores.csv
    • /g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark/contact_probability_metrics.csv

@DimaMolod

Copy link
Copy Markdown
Collaborator Author

Pushed review-response commit 85b129d. Main updates: AF2 distogram scoring now uses the lower-bound <8A convention with logsumexp, compressed AF2 pickles are covered by tests, AF3 contact_probs alignment prefers token_res_ids and ignores extra non-residue tokens, summaries only symmetrize the requested chain-pair block, and the benchmark script now uses this checkout explicitly, complete-case metrics, tie-aware AP, checkpointing, and per-model error handling.

Benchmark status: the first full CSV in contact_probability_benchmark finished, but it imported the dirty local checkout and AF2 contact scores were present for only 12 rows, so I treat it as preliminary AF3-only signal. Corrected rerun is in /g/transform/kosinski/dima/IntAct_BioGRID_STRING/benchmark_26/contact_probability_benchmark_pr22_fix. Smoke job 58939682 completed; full Slurm job 58939710 is running.

@DimaMolod
DimaMolod marked this pull request as ready for review July 10, 2026 10:25

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6fce50a27d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/alphajudge/contact_probs.py
Comment thread src/alphajudge/runner.py
@DimaMolod
DimaMolod merged commit 0c02cb5 into main Jul 10, 2026
12 checks passed
@DimaMolod
DimaMolod deleted the codex/issue-21-contact-probability branch July 10, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Request to include "contact probability" as measure of protein interaction confidence

1 participant