Skip to content

Filter invalid SIPP imputation weights#1108

Merged
MaxGhenis merged 2 commits into
mainfrom
codex/filter-sipp-tip-weights
May 22, 2026
Merged

Filter invalid SIPP imputation weights#1108
MaxGhenis merged 2 commits into
mainfrom
codex/filter-sipp-tip-weights

Conversation

@MaxGhenis
Copy link
Copy Markdown
Contributor

Summary

  • filter non-positive/non-finite SIPP donor weights before QRF fits with microimpute target filters
  • apply the guard to legacy SIPP tip/asset/vehicle models and calibration source-impute SIPP models
  • add regression coverage for target-filter reindexing and the SIPP tip failure seen in Modal

Failure addressed

Latest publication after #1103 failed in Modal Stage 1:

  • GitHub candidate run: 26255866421
  • GitHub pipeline run: 26255880509
  • Modal run ID: usdata-gha26255866421-a1
  • Modal app: us-data-1-115-5-patch-usdata-gha26255866421-a1
  • Source SHA: e6fdcaf
  • Error: policyengine_us_data/datasets/cps/cps.py -> train_tip_model() -> microimpute ValueError: Weights must be positive and finite; found 8 non-positive or NaN weight(s)

Tests

  • uv run ruff format policyengine_us_data/utils/source_quality.py policyengine_us_data/datasets/sipp/sipp.py policyengine_us_data/calibration/source_impute.py tests/unit/test_source_quality.py tests/unit/datasets/test_sipp_tip_columns.py
  • uv run ruff check policyengine_us_data/utils/source_quality.py policyengine_us_data/datasets/sipp/sipp.py policyengine_us_data/calibration/source_impute.py tests/unit/test_source_quality.py tests/unit/datasets/test_sipp_tip_columns.py
  • uv run pytest tests/unit/test_source_quality.py tests/unit/datasets/test_sipp_tip_columns.py tests/unit/calibration/test_source_impute.py -q
  • uv run python - <<'PY'
    from policyengine_us_data.datasets.sipp.sipp import train_tip_model
    model = train_tip_model()
    print(type(model).name)
    PY

@MaxGhenis MaxGhenis force-pushed the codex/filter-sipp-tip-weights branch from 119ecb1 to 902257c Compare May 21, 2026 23:00
@MaxGhenis MaxGhenis enabled auto-merge (squash) May 22, 2026 00:54
@MaxGhenis MaxGhenis merged commit 18de498 into main May 22, 2026
13 checks passed
@MaxGhenis MaxGhenis deleted the codex/filter-sipp-tip-weights branch May 22, 2026 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant