Continued progress#1
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request updates Coreopsis’ federated training and data pipeline to align with new CLIF 2.1 ICU datasets, introduces differential privacy toggles for Flower-based training, refreshes configuration defaults, and replaces older “recipe” scripts with a newer end-to-end workflow (including demographics and updated scoring).
Changes:
- Update default datasets/federation settings and add client/server differential privacy options for federated runs.
- Refresh core YAML configs (collation/tokenization/winnowing/training/scoring) to support new LABEL-based outcomes and updated training behavior.
- Replace legacy preprocessing/processing/postprocessing scripts with new CLIFpy/SOFA scoring helpers and new preprocessing/postprocessing/demographics scripts.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/coreopsis/task.py | Adjusts weight-setting behavior used by server/clients. |
| src/coreopsis/server_app.py | Adds server-side DP wrapper support and configurable fit/eval fractions. |
| src/coreopsis/client_app.py | Switches to DP-capable trainer path and updates training loop accordingly. |
| src/coreopsis/config/winnowing.yaml | Updates outcome token patterns and threshold defaults for new labeling scheme. |
| src/coreopsis/config/training.yaml | Fixes model arg typo, adds new presets, updates training/save/eval cadence, adds privacy params. |
| src/coreopsis/config/tokenization.yaml | Adds LABEL token ordering support. |
| src/coreopsis/config/scoring.yaml | Updates scoring targets toward LABEL-based outcomes. |
| src/coreopsis/config/collation.yaml | Introduces pass-through demographics columns and adds LABEL event definitions (labs/vitals/therapy). |
| recipes/run_training.sh | Adds Slurm helper for standard vs private training runs. |
| recipes/run_federated.sh | Adds Slurm helper for federated runs with optional client-side privacy. |
| recipes/run_me.sh | Adds an end-to-end orchestration script for preprocessing → training → federation → scoring. |
| recipes/run_clifpy.py | Adds CLIFpy orchestration helper (validation, waterfall, dose conversions). |
| recipes/run_sofa_scoring.py | Adds SOFA scoring pipeline helper. |
| recipes/preprocessing.py | Adds new preprocessing for ICU cohort selection and dataset materialization. |
| recipes/postprocessing.py | Adds updated results aggregation and stacked model scoring workflow. |
| recipes/demographics.py | Adds dataset demographics/outcome summarization script. |
| recipes/0-preprocessing.py | Removes outdated preprocessing script tied to older dataset splits. |
| recipes/1-processing.sh | Removes outdated processing orchestration script tied to older dataset splits. |
| recipes/2-postprocessing.py | Removes outdated postprocessing script tied to older scoring approach. |
| README.md | Updates documentation for new datasets, federation defaults, and pipeline instructions. |
| pyproject.toml | Updates dependencies and defaults (datasets/strategy/rounds/federation). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| filter_expr: | ||
| - pl.col("lab_category") == "potassium" | ||
| - pl.col("lab_value_numeric") >= 6.5 | ||
| with_col_expr: pl.lit("hyperK_init").alias("code") |
| filter_expr: | ||
| - pl.col("lab_category") == "potassium" | ||
| - pl.col("lab_value_numeric") < 2.5 | ||
| with_col_expr: pl.lit("hypoK_init").alias("code") |
| filter_expr: | ||
| - pl.col("lab_category") == "sodium" | ||
| - pl.col("lab_value_numeric") < 120. | ||
| with_col_expr: pl.lit("hypoNA_init").alias("code") |
| filter_expr: | ||
| - pl.col("lab_category") == "sodium" | ||
| - pl.col("lab_value_numeric") >= 160. | ||
| with_col_expr: pl.lit("hyperNA_init").alias("code") |
| for tt in grokked_outcome_tokens: | ||
| preds = np.hstack( | ||
| [ | ||
| df.filter(mdl=mdl).select(f"{tt}_rep_score").to_numpy() |
| ] | ||
| ) | ||
| y_qual, y_true = ( | ||
| df.filter(mdl=mdls_to_stack[0]) |
| on="hospitalization_id", | ||
| validate="m:1", | ||
| ).sink_parquet(data_raw / f"{h}-icu" / f.name) | ||
| print(f"Processed {f.name} at hospitalizion-level.") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request makes significant updates to the project’s configuration, documentation, and data processing pipeline to better align with new datasets and training strategies. The most notable changes include updating the default datasets and federated learning strategies, enhancing support for differential privacy, improving documentation, and removing outdated data processing scripts. Additionally, a new script has been added to summarize dataset demographics.
Configuration and Dependency Updates:
pyproject.tomlandREADME.mdto use"mimic-icu","ucmc-icu", and"nu-icu"instead of the previous splits, and changed the default federated learning strategy to"FedAvg"with increased rounds and new differential privacy options. [1] [2] [3]clifpy,rich,scikit-learn) and updated thecotorradependency to the main branch.Documentation Improvements:
README.mdto reflect the new datasets, federated strategies, and configuration options, and clarified the relationship to thecocoaandcotorraprojects. [1] [2] [3] [4]README.md. [1] [2]Data Processing Pipeline:
recipes/0-preprocessing.py,recipes/1-processing.sh,recipes/2-postprocessing.py) that referenced old dataset splits and workflows. [1] [2] [3]recipes/demographics.py, to compute and print demographic and outcome statistics for the updated datasets.Federation Configuration:
"standard"and standardized resource allocations for federations inpyproject.tomlandREADME.md. [1] [2] [3]These changes collectively modernize the project’s configuration, documentation, and data handling to support new datasets and privacy features, while cleaning up legacy workflows.