Skip to content

fix(examples): compute the discrete-label threshold from the training split only - #1584

Open
Amir Fathi (AmirF194) wants to merge 1 commit into
microsoft:mainfrom
AmirF194:fix/1175-time-series-discrete-label-leakage
Open

fix(examples): compute the discrete-label threshold from the training split only#1584
Amir Fathi (AmirF194) wants to merge 1 commit into
microsoft:mainfrom
AmirF194:fix/1175-time-series-discrete-label-leakage

Conversation

@AmirF194

Copy link
Copy Markdown

Why are these changes needed?

In the "Forecasting Discrete Values" example, above_mean_sales is thresholded on df["Sales"].mean() computed over the full 180-day series, before the train/test split happens a few lines later. That mean is derived in part from the 30-day period held out as discrete_test_df, so the training labels carry information from the data the model is later evaluated against.

This PR splits into discrete_train_df / discrete_test_df first, computes the threshold from discrete_train_df["Sales"] only, and applies that single value to both frames. Same fix applied to both the notebook and its mirrored markdown doc under website/docs/Examples/.

Verification:

  • Reproduced the leak with the notebook's own data generator (hcrystalball.utils.get_sales_data, same call the cell uses): the full-series mean (17020.14) differs from the train-only mean (16833.40) and changes at least one training row's label.
  • Ran the fixed cell logic standalone (same library calls, outside the notebook) and confirmed discrete_X_train/discrete_X_test keep their original shapes (150/30 rows) and discrete_train_df keeps its original column set.
  • Did not re-run the notebook's downstream AutoML.fit/predict cells (no stored outputs exist for the cell I changed, and no CI test executes this notebook), so I can't speak to any effect on the reported accuracy number further down, only that the label computation itself no longer touches the test period.
  • Checked the docs mirror (website/docs/Examples/AutoML-Time series forecast.md) for the same pattern and fixed it identically; ran pre-commit on both changed files (mdformat reflowed two lines in the doc, otherwise clean).

Related issue number

Fixes #1175

Checks

  • I've used pre-commit to lint the changes in this PR.
  • I've included the doc change needed (the markdown mirror of this notebook cell had the same leak).
  • I've added tests (no test in this repo executes automl_time_series_forecast.ipynb; the fix was verified by running the cell's logic standalone, see above).
  • I've made sure all auto checks have passed.

… split only

`above_mean_sales` in the "Forecasting Discrete Values" example was
thresholded on `df["Sales"].mean()` over the full 180-day series, before the
train/test split. That mean includes the 30-day held-out test period, so the
training labels are partly derived from data the model is supposed to be
evaluated against later, a train/test leak.

Fix: split into `discrete_train_df` / `discrete_test_df` first, compute the
mean from `discrete_train_df["Sales"]` only, and threshold both frames with
that single training-derived value. Applied to both the notebook and its
mirrored markdown doc.

Fixes microsoft#1175
@AmirF194

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

leaking data in 4. Forecasting Discrete Values in automl_time_series_forecast

1 participant