fix(examples): compute the discrete-label threshold from the training split only - #1584
Open
Amir Fathi (AmirF194) wants to merge 1 commit into
Open
Conversation
… split only `above_mean_sales` in the "Forecasting Discrete Values" example was thresholded on `df["Sales"].mean()` over the full 180-day series, before the train/test split. That mean includes the 30-day held-out test period, so the training labels are partly derived from data the model is supposed to be evaluated against later, a train/test leak. Fix: split into `discrete_train_df` / `discrete_test_df` first, compute the mean from `discrete_train_df["Sales"]` only, and threshold both frames with that single training-derived value. Applied to both the notebook and its mirrored markdown doc. Fixes microsoft#1175
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
In the "Forecasting Discrete Values" example,
above_mean_salesis thresholded ondf["Sales"].mean()computed over the full 180-day series, before the train/test split happens a few lines later. That mean is derived in part from the 30-day period held out asdiscrete_test_df, so the training labels carry information from the data the model is later evaluated against.This PR splits into
discrete_train_df/discrete_test_dffirst, computes the threshold fromdiscrete_train_df["Sales"]only, and applies that single value to both frames. Same fix applied to both the notebook and its mirrored markdown doc underwebsite/docs/Examples/.Verification:
hcrystalball.utils.get_sales_data, same call the cell uses): the full-series mean (17020.14) differs from the train-only mean (16833.40) and changes at least one training row's label.discrete_X_train/discrete_X_testkeep their original shapes (150/30 rows) anddiscrete_train_dfkeeps its original column set.AutoML.fit/predictcells (no stored outputs exist for the cell I changed, and no CI test executes this notebook), so I can't speak to any effect on the reported accuracy number further down, only that the label computation itself no longer touches the test period.website/docs/Examples/AutoML-Time series forecast.md) for the same pattern and fixed it identically; ranpre-commiton both changed files (mdformatreflowed two lines in the doc, otherwise clean).Related issue number
Fixes #1175
Checks
automl_time_series_forecast.ipynb; the fix was verified by running the cell's logic standalone, see above).