Skip to content

Fix: Incremental logic for VP trip/stop/day grouping#5237

Open
lauriemerrell wants to merge 1 commit intomainfrom
fix-vp-trip-stop-day-group
Open

Fix: Incremental logic for VP trip/stop/day grouping#5237
lauriemerrell wants to merge 1 commit intomainfrom
fix-vp-trip-stop-day-group

Conversation

@lauriemerrell
Copy link
Copy Markdown
Contributor

Description

Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.

Additional follow up on #5137 post merge of #5092, which may not have fully fixed the issue -- see #5137 (comment)

  • Genericizes the additional service date buffer -- previous logic only added extra lookback if the run was based on lookback days, didn't handle if an explicit date range was being passed (I don't think that's what caused the issue because currently prod is running with lookbacks, but just for completeness)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Include commands/logs/screenshots as relevant.

If making changes to dbt models, make sure they were created or update on Staging. Please run the command uv run dbt run -s CHANGED_MODEL --target staging and uv run dbt test -s CHANGED_MODEL --target staging, then include the output in this section of the PR.

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer-dags["models/intermediate/gtfs/int_gtfs_rt__vehicle_positions_trip_stop_day_map_grouping.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "XIvw4g==" -> (known after apply)
!~      detect_md5hash      = "9yV7+UmE329X7qKulZcCyg==" -> "different hash"
!~      generation          = 1777669782362208 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/intermediate/gtfs/int_gtfs_rt__vehicle_positions_trip_stop_day_map_grouping.sql"
!~      md5hash             = "9yV7+UmE329X7qKulZcCyg==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/gtfs/int_gtfs_rt__vehicle_positions_trip_stop_day_map_grouping.sql"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Deploy dbt #1817

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 5, 2026

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 3 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "7vbSEg==" -> (known after apply)
!~      detect_md5hash      = "gzQlzyAjYlTGiWPOSPmt/Q==" -> "different hash"
!~      generation          = 1777921775322636 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "gzQlzyAjYlTGiWPOSPmt/Q==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/gtfs/int_gtfs_rt__vehicle_positions_trip_stop_day_map_grouping.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "XIvw4g==" -> (known after apply)
!~      detect_md5hash      = "9yV7+UmE329X7qKulZcCyg==" -> "different hash"
!~      generation          = 1777669800588439 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/intermediate/gtfs/int_gtfs_rt__vehicle_positions_trip_stop_day_map_grouping.sql"
!~      md5hash             = "9yV7+UmE329X7qKulZcCyg==" -> (known after apply)
        name                = "data/warehouse/models/intermediate/gtfs/int_gtfs_rt__vehicle_positions_trip_stop_day_map_grouping.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "ruSOBg==" -> (known after apply)
!~      detect_md5hash      = "Mw4Cul2QM1zWeUWwGhMlmw==" -> "different hash"
!~      generation          = 1777921776550660 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "Mw4Cul2QM1zWeUWwGhMlmw==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

📝 Plan generated in Deploy dbt #1817

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant