Skip to content

Conversation

erikamov
Copy link
Contributor

Description

This PR changes the data type from NUMERIC to BIGNUMERIC to fix Invalid NUMERIC value error accepting longer values on external_ntd__annual_reporting.multi_year__track_and_roadway_by_agency.sum_total_track_miles and external_ntd__annual_reporting.multi_year__track_and_roadway_guideway_age_distribution fields: _1940s, _1950s, _1960s, _1970s, _1980s, _1990s, _2000s, _2010s, _2020s, pre1940s and downstream tables.

Resolves [#4276]

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Tested running all the pipeline on Airflow Staging: external tables DAG and specific tables in dbt_all.

image image image

Post-merge follow-ups

  • No action required
  • Actions required (specified below)

Confirm if dbt_all is able to run successfully fct_track_and_roadway_guideway_age_distribution and fct_track_and_roadway_by_agency.

Copy link

github-actions bot commented Sep 20, 2025

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 6 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_by_agency.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "KHlMmg==" -> (known after apply)
!~      detect_md5hash      = "uRPAddNpBxu55bSEcsJXUw==" -> "different hash"
!~      generation          = 1751416670797784 -> (known after apply)
        id                  = "calitp-composer-dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_by_agency.yml"
!~      md5hash             = "uRPAddNpBxu55bSEcsJXUw==" -> (known after apply)
        name                = "dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_by_agency.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_guideway_age_distribution.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "P/3SfA==" -> (known after apply)
!~      detect_md5hash      = "JOT0WP0nNhmu8B3uo+Pjjg==" -> "different hash"
!~      generation          = 1751416671712936 -> (known after apply)
        id                  = "calitp-composer-dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_guideway_age_distribution.yml"
!~      md5hash             = "JOT0WP0nNhmu8B3uo+Pjjg==" -> (known after apply)
        name                = "dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_guideway_age_distribution.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "jt477Q==" -> (known after apply)
!~      detect_md5hash      = "Z2vrtASxHLJuhHV3ixiaHQ==" -> "different hash"
!~      generation          = 1758311274485789 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "Z2vrtASxHLJuhHV3ixiaHQ==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_by_agency.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "NC5+jw==" -> (known after apply)
!~      detect_md5hash      = "R4tcH7NdmHfuZrzmf+vOcA==" -> "different hash"
!~      generation          = 1751997660679903 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_by_agency.sql"
!~      md5hash             = "R4tcH7NdmHfuZrzmf+vOcA==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_by_agency.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_guideway_age_distribution.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-dags" {
!~      crc32c              = "kqY7Pg==" -> (known after apply)
!~      detect_md5hash      = "nMv4jZ15Vi/7caFto0d1Ew==" -> "different hash"
!~      generation          = 1751997659670018 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_guideway_age_distribution.sql"
!~      md5hash             = "nMv4jZ15Vi/7caFto0d1Ew==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_guideway_age_distribution.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "qWGFow==" -> (known after apply)
!~      detect_md5hash      = "9++V7UQTvHJxJ90vsnG18A==" -> "different hash"
!~      generation          = 1758311276222460 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "9++V7UQTvHJxJ90vsnG18A==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 6 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #680

Copy link

github-actions bot commented Sep 20, 2025

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 4 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_by_agency.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "KHlMmg==" -> (known after apply)
!~      detect_md5hash      = "uRPAddNpBxu55bSEcsJXUw==" -> "different hash"
!~      generation          = 1749661093934904 -> (known after apply)
        id                  = "calitp-staging-composer-dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_by_agency.yml"
!~      md5hash             = "uRPAddNpBxu55bSEcsJXUw==" -> (known after apply)
        name                = "dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_by_agency.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer["dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_guideway_age_distribution.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "P/3SfA==" -> (known after apply)
!~      detect_md5hash      = "JOT0WP0nNhmu8B3uo+Pjjg==" -> "different hash"
!~      generation          = 1758573915692947 -> (known after apply)
        id                  = "calitp-staging-composer-dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_guideway_age_distribution.yml"
!~      md5hash             = "JOT0WP0nNhmu8B3uo+Pjjg==" -> (known after apply)
        name                = "dags/create_external_tables/ntd_data_products/multi_year__track_and_roadway_guideway_age_distribution.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_by_agency.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "NC5+jw==" -> (known after apply)
!~      detect_md5hash      = "R4tcH7NdmHfuZrzmf+vOcA==" -> "different hash"
!~      generation          = 1751997660074376 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_by_agency.sql"
!~      md5hash             = "R4tcH7NdmHfuZrzmf+vOcA==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_by_agency.sql"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-staging-composer-dags["models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_guideway_age_distribution.sql"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer-dags" {
!~      crc32c              = "kqY7Pg==" -> (known after apply)
!~      detect_md5hash      = "nMv4jZ15Vi/7caFto0d1Ew==" -> "different hash"
!~      generation          = 1758573915686918 -> (known after apply)
        id                  = "calitp-staging-composer-data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_guideway_age_distribution.sql"
!~      md5hash             = "nMv4jZ15Vi/7caFto0d1Ew==" -> (known after apply)
        name                = "data/warehouse/models/staging/ntd_annual_reporting/stg_ntd__track_and_roadway_guideway_age_distribution.sql"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 4 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #680

@ohrite ohrite force-pushed the mov/4276-bignumeric branch from ada95a9 to 7f9547f Compare September 22, 2025 22:27
Copy link
Contributor

@ohrite ohrite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🍐 w/ @erikamov

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants