feat: rolling decision points in the winnower (threshold.every_s)#16
Open
brettbj wants to merge 1 commit into
Open
feat: rolling decision points in the winnower (threshold.every_s)#16brettbj wants to merge 1 commit into
brettbj wants to merge 1 commit into
Conversation
Adds a strided thresholding mode that emits one row per (subject, cutpoint)
at multiples of every_s (capped by max_decision_s, default 7d), each carrying
a decision_s column so downstream evaluation can stratify performance by time
into the stay. This makes 'predict every N hours whether an outcome occurs in
the next H hours' a pure-config task:
threshold:
every_s: 28800 # decide every 8h
max_decision_s: 604800 # over the first 7d
horizon_after_threshold_s: 86400 # outcome within the next 24h
In rolling mode the horizon window is anchored at the cutpoint -- (decision_s,
decision_s + horizon] -- rather than at the last observed event, which is the
correct semantics for a decision made at decision_s. decision_s is carried
through add_outcome_flags to the output frame.
Co-Authored-By: Claude Fable 5 <[email protected]>
Contributor
Author
|
not yet ready for review |
Contributor
|
I'm concerned that predicting on the same hospitalization multiple times might juice our metrics in ways we don't intend. This is just a hunch though, and I'm happy to reconsider / discuss |
Contributor
Author
I share the concern but it's also a realistic deployment setting. Part of why I didn't finish this yet is it would also require overhaul of our metrics and some safeguards (e.g., bootstrapping loses IID assumptions) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a strided / rolling thresholding mode to the winnower. Instead of one decision point per subject,
threshold.every_semits one row per (subject, cutpoint) at multiples ofevery_s(capped bymax_decision_s, default 7d), each strictly inside the stay. Each row carries a newdecision_scolumn (the cutpoint, in seconds since timeline start) so downstream evaluation can stratify performance by time into the stay.This makes "predict every N hours whether outcome Y occurs in the next H hours" a pure-config task:
Why
The existing modes (
duration_s,first_occurrence,uniform_random) each produce a single decision point per subject, so a rolling/repeated-measures evaluation ("score this patient every 8h") was not expressible in config and required out-of-process windowing.Horizon semantics fix for rolling mode
For a decision made at
decision_s, the outcome window must be(decision_s, decision_s + horizon]— anchored at the cutpoint. The existing horizon code anchors at the last observed event before the threshold, which (for a cutpoint that falls between events) silently shrinks the window. Example: decision at 8h, event at 30h, 24h horizon — 30h is 22h after the decision and must count as future, but last-event anchoring excluded it. Rolling mode therefore uses cutpoint-anchored counting; the existing single-point modes are unchanged.Verification
Synthetic timeline (events at 0/4/10/30/50h,
AKI//stage_2at 30h),every_s=8h,horizon=24h:Notes
decision_sis only added in rolling mode; other modes' output schema is unchanged.🤖 Generated with Claude Code