Skip to content

feat: rolling decision points in the winnower (threshold.every_s)#16

Open
brettbj wants to merge 1 commit into
masterfrom
feat/strided-winnowing
Open

feat: rolling decision points in the winnower (threshold.every_s)#16
brettbj wants to merge 1 commit into
masterfrom
feat/strided-winnowing

Conversation

@brettbj

@brettbj brettbj commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

What

Adds a strided / rolling thresholding mode to the winnower. Instead of one decision point per subject, threshold.every_s emits one row per (subject, cutpoint) at multiples of every_s (capped by max_decision_s, default 7d), each strictly inside the stay. Each row carries a new decision_s column (the cutpoint, in seconds since timeline start) so downstream evaluation can stratify performance by time into the stay.

This makes "predict every N hours whether outcome Y occurs in the next H hours" a pure-config task:

threshold:
  every_s: 28800          # decide every 8h
  max_decision_s: 604800  # over the first 7 days of the stay
horizon_after_threshold_s: 86400   # outcome within the next 24h
outcome_tokens: [ AKI//stage_2, XFR-IN//icu, DSCG//expired ]

Why

The existing modes (duration_s, first_occurrence, uniform_random) each produce a single decision point per subject, so a rolling/repeated-measures evaluation ("score this patient every 8h") was not expressible in config and required out-of-process windowing.

Horizon semantics fix for rolling mode

For a decision made at decision_s, the outcome window must be (decision_s, decision_s + horizon] — anchored at the cutpoint. The existing horizon code anchors at the last observed event before the threshold, which (for a cutpoint that falls between events) silently shrinks the window. Example: decision at 8h, event at 30h, 24h horizon — 30h is 22h after the decision and must count as future, but last-event anchoring excluded it. Rolling mode therefore uses cutpoint-anchored counting; the existing single-point modes are unchanged.

Verification

Synthetic timeline (events at 0/4/10/30/50h, AKI//stage_2 at 30h), every_s=8h, horizon=24h:

decision (h) past future
8, 16, 24 F T (AKI within 24h)
32, 40, 48 T F (AKI now in the past)

Notes

  • decision_s is only added in rolling mode; other modes' output schema is unchanged.
  • Repeated rows per subject are correlated — evaluation should use subject-clustered resampling for valid CIs (separate concern from this PR).

🤖 Generated with Claude Code

Adds a strided thresholding mode that emits one row per (subject, cutpoint)
at multiples of every_s (capped by max_decision_s, default 7d), each carrying
a decision_s column so downstream evaluation can stratify performance by time
into the stay. This makes 'predict every N hours whether an outcome occurs in
the next H hours' a pure-config task:

  threshold:
    every_s: 28800            # decide every 8h
    max_decision_s: 604800    # over the first 7d
  horizon_after_threshold_s: 86400   # outcome within the next 24h

In rolling mode the horizon window is anchored at the cutpoint -- (decision_s,
decision_s + horizon] -- rather than at the last observed event, which is the
correct semantics for a decision made at decision_s. decision_s is carried
through add_outcome_flags to the output frame.

Co-Authored-By: Claude Fable 5 <[email protected]>
@brettbj

brettbj commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

not yet ready for review

@burkh4rt

Copy link
Copy Markdown
Contributor

I'm concerned that predicting on the same hospitalization multiple times might juice our metrics in ways we don't intend. This is just a hunch though, and I'm happy to reconsider / discuss

@burkh4rt burkh4rt added the enhancement New feature or request label Jun 10, 2026
@brettbj

brettbj commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

I'm concerned that predicting on the same hospitalization multiple times might juice our metrics in ways we don't intend. This is just a hunch though, and I'm happy to reconsider / discuss

I share the concern but it's also a realistic deployment setting. Part of why I didn't finish this yet is it would also require overhaul of our metrics and some safeguards (e.g., bootstrapping loses IID assumptions)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants