feat, fix(metada carrying): fix metadata dropping through the pipeline (atomworks, predictors), eliminate the post-fact patchign script. #301
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,17 @@ | ||
| import argparse | ||
| import json | ||
| import subprocess | ||
| import tempfile | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| import joblib | ||
| import pandas as pd | ||
| from biotite.structure.io.pdbx.cif import CIFFile | ||
| from loguru import logger | ||
| from pandas import DataFrame | ||
| from sampleworks.eval.grid_search_eval_utils import parse_eval_args, setup_evaluation_parameters | ||
| from sampleworks.utils.cif_utils import extract_model, model_numbers | ||
|
|
||
|
|
||
| # TODO make more general: https://github.com/diff-use/sampleworks/issues/93 | ||
|
|
@@ -133,12 +136,17 @@ def get_protein_level_z_scores(tortoize_json: dict[str, Any]) -> pd.DataFrame: | |
|
|
||
| def get_stats_for_single_path(path: Path) -> tuple[DataFrame, DataFrame]: | ||
| """ | ||
| Run tortoize per model on a single CIF file and extract residue/protein-level stats. | ||
|
|
||
| tortoize is single-model: it loads zero atoms from a stacked ensemble. Each model is extracted | ||
|
Collaborator
There was a problem hiding this comment. I recall differently. I am 99.9% sure tortoize outputs scores separately for each model. I know this because I have output that shows different values for each model, e.g., /mnt/diffuse-shared/raw/sampleworks/combined_occ_sweep_results/tortoize_residues.csv. You can also see that tortoize iterates over models here: https://github.com/PDB-REDO/tortoize/blob/trunk/src/tortoize.cpp#L1087 You should probably revert this particular change unless you can show that tortoize does not behave as advertised. |
||
| via a pure ``_atom_site`` filter (``extract_model``, which preserves the carried | ||
| ``_entity``/``_entity_poly``/``_entity_poly_seq`` categories), scored on its own, and the | ||
| per-model frames are merged -- each tagged with the source model number. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| path : Path | ||
| Path to a CIF file (single-model or stacked ensemble) to process. | ||
|
|
||
| Returns | ||
| ------- | ||
|
|
@@ -147,20 +155,45 @@ def get_stats_for_single_path(path: Path) -> tuple[DataFrame, DataFrame]: | |
| """ | ||
| logger.info(f"Processing {path}") | ||
| try: | ||
| cif = CIFFile.read(str(path)) | ||
| except Exception as e: # noqa: BLE001 - a bad file must not abort the whole grid-search eval | ||
| logger.error(f"Failed to read {path}: {e}") | ||
| return pd.DataFrame(), pd.DataFrame() | ||
|
|
||
| residue_frames: list[DataFrame] = [] | ||
| protein_frames: list[DataFrame] = [] | ||
| for model_num in model_numbers(cif): | ||
| single = CIFFile.read(str(path)) | ||
| extract_model(single, model_num) | ||
| with tempfile.NamedTemporaryFile(suffix=".cif", delete=False) as handle: | ||
| model_cif = Path(handle.name) | ||
| try: | ||
| single.write(str(model_cif)) | ||
| output = subprocess.check_output(["tortoize", str(model_cif)]) | ||
| result = json.loads(output) | ||
| except (subprocess.CalledProcessError, json.JSONDecodeError, OSError) as e: | ||
| logger.error(f"Failed to process model {model_num} of {path}: {e}") | ||
| continue | ||
| finally: | ||
| model_cif.unlink(missing_ok=True) | ||
|
|
||
| residues = flatten_residues(result) | ||
| protein = get_protein_level_z_scores(result) | ||
| # tortoize labels a lone model "1"; tag rows with the real source model number. | ||
| if not residues.empty: | ||
| residues["model"] = str(model_num) | ||
| residue_frames.append(residues) | ||
| if not protein.empty: | ||
| protein["model"] = str(model_num) | ||
| protein_frames.append(protein) | ||
|
|
||
| if not residue_frames: | ||
| logger.warning(f"No residues found in {path}") | ||
| return pd.DataFrame(), pd.DataFrame() | ||
|
|
||
| residues = pd.concat(residue_frames, ignore_index=True) | ||
| residues["path"] = path | ||
| protein_level_stats = pd.concat(protein_frames, ignore_index=True) | ||
| protein_level_stats["path"] = path | ||
| return residues, protein_level_stats | ||
|
|
||
|
|
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It may confuse users who aren't familiar with the package to say no CIF-patching is needed, since they won't understand it was previously required. You might say "... so the previously-required CIF-patching step is no longer required"