Fix payments failing tests (#5087)#5145
Conversation
|
Warehouse report: Failed to add ci-report to a comment. Review the ci-report in the Summary. |
|
Terraform plan in iac/cal-itp-data-infra-staging/airflow/us Plan: 0 to add, 11 to change, 0 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~ update in-place
Terraform will perform the following actions:
# google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~ content = (sensitive value)
!~ crc32c = "F2BSlQ==" -> (known after apply)
!~ detect_md5hash = "E9Rs3RD2sBrXN4Bh03TsIg==" -> "different hash"
!~ generation = 1777910056926243 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~ md5hash = "E9Rs3RD2sBrXN4Bh03TsIg==" -> (known after apply)
name = "data/warehouse/target/catalog.json"
# (16 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/intermediate/payments/_int_payments.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "kSTsmw==" -> (known after apply)
!~ detect_md5hash = "Z9SD7LPi1IkjqsOxKpXSvw==" -> "different hash"
!~ generation = 1777327344602368 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/intermediate/payments/_int_payments.yml"
!~ md5hash = "Z9SD7LPi1IkjqsOxKpXSvw==" -> (known after apply)
name = "data/warehouse/models/intermediate/payments/_int_payments.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/mart/payments/_payments.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "v6H6oA==" -> (known after apply)
!~ detect_md5hash = "vEj8VtRuc2qvXocvbXFVGg==" -> "different hash"
!~ generation = 1777672367450522 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/mart/payments/_payments.yml"
!~ md5hash = "vEj8VtRuc2qvXocvbXFVGg==" -> (known after apply)
name = "data/warehouse/models/mart/payments/_payments.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/elavon/_elavon.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "LnZMsA==" -> (known after apply)
!~ detect_md5hash = "+1O+VLnd0FlNSOvQB1udIQ==" -> "different hash"
!~ generation = 1776453637740705 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/elavon/_elavon.yml"
!~ md5hash = "+1O+VLnd0FlNSOvQB1udIQ==" -> (known after apply)
name = "data/warehouse/models/staging/payments/elavon/_elavon.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/littlepay/_stg_littlepay.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "VWF6UA==" -> (known after apply)
!~ detect_md5hash = "ih/ufAm4aAN98+sVyVKA7Q==" -> "different hash"
!~ generation = 1776453637606395 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/littlepay/_stg_littlepay.yml"
!~ md5hash = "ih/ufAm4aAN98+sVyVKA7Q==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay/_stg_littlepay.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/littlepay/stg_littlepay__product_data.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "YFuv2w==" -> (known after apply)
!~ detect_md5hash = "FAiaEkeqMZapsR5HsHH3uw==" -> "different hash"
!~ generation = 1773071356367933 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/littlepay/stg_littlepay__product_data.sql"
!~ md5hash = "FAiaEkeqMZapsR5HsHH3uw==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay/stg_littlepay__product_data.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/littlepay/stg_littlepay__settlements.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "+G4B3g==" -> (known after apply)
!~ detect_md5hash = "XbWfLmUmN6hAXrVVgr586Q==" -> "different hash"
!~ generation = 1749663117562225 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/littlepay/stg_littlepay__settlements.sql"
!~ md5hash = "XbWfLmUmN6hAXrVVgr586Q==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay/stg_littlepay__settlements.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/littlepay_v3/_stg_littlepay_v3.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "CwK/EQ==" -> (known after apply)
!~ detect_md5hash = "obhzXusyeKsZLaIGjXg7pQ==" -> "different hash"
!~ generation = 1776453637126759 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/littlepay_v3/_stg_littlepay_v3.yml"
!~ md5hash = "obhzXusyeKsZLaIGjXg7pQ==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay_v3/_stg_littlepay_v3.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/littlepay_v3/stg_littlepay__product_data_v3.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "+XwabQ==" -> (known after apply)
!~ detect_md5hash = "j+vkGXYRdlJz9z5xMCm8SA==" -> "different hash"
!~ generation = 1773071356829345 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__product_data_v3.sql"
!~ md5hash = "j+vkGXYRdlJz9z5xMCm8SA==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__product_data_v3.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-dags["models/staging/payments/littlepay_v3/stg_littlepay__settlements_v3.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~ crc32c = "gC0w/Q==" -> (known after apply)
!~ detect_md5hash = "6o7M3ur2JEClMN6WP1TanA==" -> "different hash"
!~ generation = 1749663118353822 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__settlements_v3.sql"
!~ md5hash = "6o7M3ur2JEClMN6WP1TanA==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__settlements_v3.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~ content = (sensitive value)
!~ crc32c = "I8NYOw==" -> (known after apply)
!~ detect_md5hash = "QnUi1yF7Mz7udvqMk8+61g==" -> "different hash"
!~ generation = 1777910058048216 -> (known after apply)
id = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~ md5hash = "QnUi1yF7Mz7udvqMk8+61g==" -> (known after apply)
name = "data/warehouse/target/manifest.json"
# (16 unchanged attributes hidden)
}
Plan: 0 to add, 11 to change, 0 to destroy.📝 Plan generated in Deploy dbt #1808 |
|
Terraform plan in iac/cal-itp-data-infra/airflow/us Plan: 0 to add, 9 to change, 0 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~ update in-place
Terraform will perform the following actions:
# google_storage_bucket_object.calitp-composer-dags["models/intermediate/payments/_int_payments.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "kSTsmw==" -> (known after apply)
!~ detect_md5hash = "Z9SD7LPi1IkjqsOxKpXSvw==" -> "different hash"
!~ generation = 1777327338151290 -> (known after apply)
id = "calitp-composer-data/warehouse/models/intermediate/payments/_int_payments.yml"
!~ md5hash = "Z9SD7LPi1IkjqsOxKpXSvw==" -> (known after apply)
name = "data/warehouse/models/intermediate/payments/_int_payments.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/mart/payments/_payments.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "v6H6oA==" -> (known after apply)
!~ detect_md5hash = "vEj8VtRuc2qvXocvbXFVGg==" -> "different hash"
!~ generation = 1777672366114543 -> (known after apply)
id = "calitp-composer-data/warehouse/models/mart/payments/_payments.yml"
!~ md5hash = "vEj8VtRuc2qvXocvbXFVGg==" -> (known after apply)
name = "data/warehouse/models/mart/payments/_payments.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/elavon/_elavon.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "LnZMsA==" -> (known after apply)
!~ detect_md5hash = "+1O+VLnd0FlNSOvQB1udIQ==" -> "different hash"
!~ generation = 1776457910258943 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/elavon/_elavon.yml"
!~ md5hash = "+1O+VLnd0FlNSOvQB1udIQ==" -> (known after apply)
name = "data/warehouse/models/staging/payments/elavon/_elavon.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/littlepay/_stg_littlepay.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "VWF6UA==" -> (known after apply)
!~ detect_md5hash = "ih/ufAm4aAN98+sVyVKA7Q==" -> "different hash"
!~ generation = 1776457911776212 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/littlepay/_stg_littlepay.yml"
!~ md5hash = "ih/ufAm4aAN98+sVyVKA7Q==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay/_stg_littlepay.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/littlepay/stg_littlepay__product_data.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "YFuv2w==" -> (known after apply)
!~ detect_md5hash = "FAiaEkeqMZapsR5HsHH3uw==" -> "different hash"
!~ generation = 1773071359948870 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/littlepay/stg_littlepay__product_data.sql"
!~ md5hash = "FAiaEkeqMZapsR5HsHH3uw==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay/stg_littlepay__product_data.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/littlepay/stg_littlepay__settlements.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "+G4B3g==" -> (known after apply)
!~ detect_md5hash = "XbWfLmUmN6hAXrVVgr586Q==" -> "different hash"
!~ generation = 1751416663836105 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/littlepay/stg_littlepay__settlements.sql"
!~ md5hash = "XbWfLmUmN6hAXrVVgr586Q==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay/stg_littlepay__settlements.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/littlepay_v3/_stg_littlepay_v3.yml"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "CwK/EQ==" -> (known after apply)
!~ detect_md5hash = "obhzXusyeKsZLaIGjXg7pQ==" -> "different hash"
!~ generation = 1776457910865287 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/littlepay_v3/_stg_littlepay_v3.yml"
!~ md5hash = "obhzXusyeKsZLaIGjXg7pQ==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay_v3/_stg_littlepay_v3.yml"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/littlepay_v3/stg_littlepay__product_data_v3.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "+XwabQ==" -> (known after apply)
!~ detect_md5hash = "j+vkGXYRdlJz9z5xMCm8SA==" -> "different hash"
!~ generation = 1773071359946926 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__product_data_v3.sql"
!~ md5hash = "j+vkGXYRdlJz9z5xMCm8SA==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__product_data_v3.sql"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-dags["models/staging/payments/littlepay_v3/stg_littlepay__settlements_v3.sql"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-dags" {
!~ crc32c = "gC0w/Q==" -> (known after apply)
!~ detect_md5hash = "6o7M3ur2JEClMN6WP1TanA==" -> "different hash"
!~ generation = 1751416667316327 -> (known after apply)
id = "calitp-composer-data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__settlements_v3.sql"
!~ md5hash = "6o7M3ur2JEClMN6WP1TanA==" -> (known after apply)
name = "data/warehouse/models/staging/payments/littlepay_v3/stg_littlepay__settlements_v3.sql"
# (17 unchanged attributes hidden)
}
Plan: 0 to add, 9 to change, 0 to destroy.📝 Plan generated in Deploy dbt #1808 |
b66c2ce to
51f286a
Compare
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
ece7c46 to
dc6b2c1
Compare
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
dc6b2c1 to
de426f7
Compare
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
a1341b4 to
eb6d7be
Compare
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
1 similar comment
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
4493e54 to
08e6879
Compare
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
… actually unique & this is expected
…cepted categories
08e6879 to
3f6571e
Compare
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
Impacted ExposuresNo exposures are impacted by the changes in this PR. Changed models
|
Description
Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.
Resolves #5087
Individual failing tests updated here:
trn_ref_numper discussionRemaining failing tests after these changes:
dbt_utils_expression_is_true_int_payments__micropayments_to_aggregations_total_nominal_amount_dollars_net_micropayment_amount_dollars;fct_payments_aggregationsnot null end of month value and unique and not nullaggregation_id,not_null_int_payments__micropayments_to_aggregations_aggregation_id, -- these are all caused by Bug: Redwood Coast Transit transactions missingaggregation_idand other key columns #5168 (empty columns for some Redwood rows), leaving the tests on and failing pending how that gets addresseddbt_utils_unique_combination_of_columns_int_payments__customers_participant_id__customer_idis caused by Bug (Payments-Littlepay): Some customers have more than one principal customer ID #4997 which should be addressedType of change
How has this been tested?
Include commands/logs/screenshots as relevant.
If making changes to dbt models, make sure they were created or update on Staging. Please run the command
uv run dbt run -s CHANGED_MODEL --target staginganduv run dbt test -s CHANGED_MODEL --target staging, then include the output in this section of the PR.Post-merge follow-ups
Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.