Skip to content

Add unit tests to GTFS-RT parse and validate Airflow operators#5179

Draft
ohrite wants to merge 1 commit intomainfrom
mov/airflow-operator-gtfs-rt
Draft

Add unit tests to GTFS-RT parse and validate Airflow operators#5179
ohrite wants to merge 1 commit intomainfrom
mov/airflow-operator-gtfs-rt

Conversation

@ohrite
Copy link
Copy Markdown
Contributor

@ohrite ohrite commented Apr 28, 2026

Description

This PR adds unit test coverage to the GTFS-RT DAG by way of extracting operators and porting behavioral tests.

Relates to #4488

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

pytest and local Airflow

Post-merge follow-ups

  • No action required
  • Actions required (specified below)

Monitor staging Airflow

@ohrite ohrite self-assigned this Apr 28, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 28, 2026

Terraform plan in iac/cal-itp-data-infra/composer/us

No changes. Your infrastructure matches the configuration.
No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1917

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 28, 2026

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 2 to add, 2 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/parse_and_validate_gtfs_rt.py"] will be created
+   resource "google_storage_bucket_object" "calitp-staging-composer" {
+       bucket         = "calitp-staging-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "dags/parse_and_validate_gtfs_rt.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/dags/parse_and_validate_gtfs_rt.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-staging-composer["plugins/operators/gtfs_rt_feed_to_jsonl_operator.py"] will be created
+   resource "google_storage_bucket_object" "calitp-staging-composer" {
+       bucket         = "calitp-staging-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "plugins/operators/gtfs_rt_feed_to_jsonl_operator.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/plugins/operators/gtfs_rt_feed_to_jsonl_operator.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-staging-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "n61rBA==" -> (known after apply)
!~      detect_md5hash      = "zk9XqSQyX8VMcQoviaMv2Q==" -> "different hash"
!~      generation          = 1777671557648398 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "zk9XqSQyX8VMcQoviaMv2Q==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "nSDRww==" -> (known after apply)
!~      detect_md5hash      = "+GI7CKcGvoTWgbI8/Su3MQ==" -> "different hash"
!~      generation          = 1777671558808840 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "+GI7CKcGvoTWgbI8/Su3MQ==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 2 to add, 2 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1917

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 28, 2026

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 2 to add, 2 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+   create
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/parse_and_validate_gtfs_rt.py"] will be created
+   resource "google_storage_bucket_object" "calitp-composer" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "dags/parse_and_validate_gtfs_rt.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/dags/parse_and_validate_gtfs_rt.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-composer["plugins/operators/gtfs_rt_feed_to_jsonl_operator.py"] will be created
+   resource "google_storage_bucket_object" "calitp-composer" {
+       bucket         = "calitp-composer"
+       content        = (sensitive value)
+       content_type   = (known after apply)
+       crc32c         = (known after apply)
+       detect_md5hash = "different hash"
+       generation     = (known after apply)
+       id             = (known after apply)
+       kms_key_name   = (known after apply)
+       md5hash        = (known after apply)
+       md5hexhash     = (known after apply)
+       media_link     = (known after apply)
+       name           = "plugins/operators/gtfs_rt_feed_to_jsonl_operator.py"
+       output_name    = (known after apply)
+       self_link      = (known after apply)
+       source         = "../../../../airflow/plugins/operators/gtfs_rt_feed_to_jsonl_operator.py"
+       storage_class  = (known after apply)
    }

  # google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "Ku/hRQ==" -> (known after apply)
!~      detect_md5hash      = "J/vXFF0Zm0pGSz3pVH1Qwg==" -> "different hash"
!~      generation          = 1777671559668369 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "J/vXFF0Zm0pGSz3pVH1Qwg==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "WTgTJg==" -> (known after apply)
!~      detect_md5hash      = "cPI7QSTY+2+HBdabV3Mh2w==" -> "different hash"
!~      generation          = 1777671561135128 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "cPI7QSTY+2+HBdabV3Mh2w==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 2 to add, 2 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #1917

@ohrite ohrite force-pushed the mov/airflow-operator-gtfs-rt branch from 1dcb614 to 007a3d4 Compare April 28, 2026 20:58
@ohrite ohrite force-pushed the mov/airflow-operator-gtfs-rt branch from 007a3d4 to 5af570e Compare May 3, 2026 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant