refactor(pydantic): remove Pydantic v1 legacy code and migrate fully to v2 #15261

sgomezvillamor · 2025-11-10T15:32:33Z

Context

This PR removes all Pydantic v1 compatibility code and completes the migration to Pydantic v2. This is now safe because:

All Python projects in the repo have pydantic>=2.0 as a dependency
- metadata-ingestion: pydantic>=2.4.0,<3.0.0
- datahub-actions: pydantic>=2.0.0,<3.0.0
- smoke-test: uses metadata-ingestion (Pydantic v2)
- Note: The codebase maintains compatibility with Pydantic 2.0+ (some CI environments use older constraint files with Pydantic 2.0-2.3)
The codebase no longer needs to support Pydantic v1 users
Pydantic v2 has been stable since June 2023 (over a year and a half ago)
This simplifies the codebase by removing 254 lines of compatibility/branching code

What Changed

Deleted compatibility layer files:

metadata-ingestion/src/datahub/configuration/pydantic_migration_helpers.py (provided v1_* aliases)
metadata-ingestion/src/datahub/pydantic/compat.py (version detection utilities)

Migrated to native Pydantic v2 APIs (38 files):

v1_ConfigModel → BaseModel
v1_Field → Field
v1_validator → @field_validator
v1_root_validator → @model_validator
Extra.forbid → model_config = {"extra": "forbid"}
__fields__ → model_fields
parse_obj() → model_validate()
_init_private_attributes → model_post_init

Removed all version checks:

Deleted conditional code for PYDANTIC_VERSION, PYDANTIC_VERSION_2, PYDANTIC_SUPPORTS_CALLABLE_DISCRIMINATOR
Kept only Pydantic v2 code paths

Updated discriminated unions:

Leverages Pydantic v2's smart union matching for assertion types and search filters
Avoids Discriminator/Tag (Pydantic 2.4+) to maintain compatibility with Pydantic 2.0-2.3 used in some Airflow constraint files
Added comments explaining where smart union matching is used

Updated configurations:

Removed pydantic.v1.mypy from mypy plugins
Removed Pydantic v1 deprecation warnings from pytest config

Breaking Changes

None for users - all projects already require Pydantic v2. This only affects:

Internal development environments still on Pydantic v1 (shouldn't exist given dependencies)
Any external code importing private compatibility utilities (were never meant to be public APIs)

Testing

✅ All 4109 unit tests pass (8 skipped)
✅ All mypy type checks pass (1651 source files)
✅ All linting checks pass (ruff format + ruff check)
✅ Updated test fixtures for Pydantic v2 error message formats
✅ Updated golden files for test outputs
✅ Compatible with Pydantic 2.0+ (tested in CI with various Airflow constraint files)

Files Changed

38 files changed: +1132 insertions, -1386 deletions (net reduction of 254 lines)

Key areas:

Assertion API models (8 concrete assertion types)
Data contract models
Configuration models
Search filters (discriminated unions)
Test files and fixtures

alwaysmeticulous · 2025-11-10T15:35:59Z

🔴 Meticulous spotted visual differences in 2 of 939 screens tested: view and approve differences detected.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Last updated for commit 30532e7. This comment will update as new commits are pushed.}

codecov · 2025-11-10T15:36:31Z

Codecov Report

❌ Patch coverage is 76.47059% with 124 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...c/datahub/ingestion/source/dremio/dremio_source.py	82.11%	39 Missing ⚠️
...estion/src/datahub/api/entities/dataset/dataset.py	9.67%	28 Missing ⚠️
...data-ingestion/src/datahub/configuration/common.py	21.42%	11 Missing ⚠️
...datahub/api/entities/external/external_entities.py	0.00%	8 Missing ⚠️
...ahub/api/entities/assertion/freshness_assertion.py	71.42%	6 Missing ⚠️
...rc/datahub/api/entities/assertion/sql_assertion.py	66.66%	5 Missing ⚠️
...datahub/api/entities/assertion/volume_assertion.py	73.68%	5 Missing ⚠️
.../datahub/api/entities/assertion/field_assertion.py	80.95%	4 Missing ⚠️
...pi/entities/datacontract/data_quality_assertion.py	55.55%	4 Missing ⚠️
.../datahub/api/entities/datacontract/datacontract.py	84.61%	2 Missing ⚠️
... and 11 more

📢 Thoughts on this report? Let us know!

…to v2

treff7es · 2025-11-11T10:39:12Z

metadata-ingestion/src/datahub/api/entities/assertion/assertion_operator.py


-class InOperator(v1_ConfigModel):
+class InOperator(BaseModel):
+    model_config = {"extra": "forbid"}


Is there any reason we are not using a v2 ConfigModel where we set extra forbiddens by default, and all the models inherit it?

really good catch
addressed in commit 3a360ce

treff7es · 2025-11-11T10:42:41Z

metadata-ingestion/src/datahub/api/entities/assertion/field_assertion.py

+    model_config = {"extra": "forbid"}
+
+    type: Literal["count", "percentage"] = Field(default="count")
+    value: int = Field(default=0)


I know it is not part of this pr, but at line 42 the comparison is not correct ->
It is comparing self.type (which is a string at runtime) to Literal["count"] (which is a type annotation, not a value). This comparison will always be False.

treff7es · 2025-11-11T10:48:11Z

metadata-ingestion/src/datahub/api/entities/assertion/sql_assertion.py

 )


 class SqlMetricAssertion(BaseEntityAssertion):


Shouldn't we explicity specify it implements BaseAssertionProtocol which makes explicit to implement get_id?

The problem is BaseAssertionProtocol is not being used at all in the model hierarchy. Not before the PR and so not now.

- Migrate ConfigModel from Pydantic v1 style (class Config) to v2 style (model_config = ConfigDict) - Replace 35 instances of manual 'model_config = {"extra": "forbid"}' with ConfigModel inheritance across 9 files - Simplify Filter union in search_filters.py by removing TYPE_CHECKING conditional - Fix redshift.py to use parse_obj_allow_extras() instead of direct Config.extra manipulation This eliminates code duplication and centralizes Pydantic configuration policy.

codecov · 2025-11-11T11:35:16Z

Bundle Report

Changes will increase total bundle size by 31 bytes (0.0%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.63MB	31 bytes (0.0%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
`assets/index-*.js`	31 bytes	19.0MB	0.0%

This commit completes the migration of all Pydantic v1 legacy syntax to v2 across the entire DataHub Python codebase. **Configuration Migration (18 instances):** - Migrate `class Config:` → `model_config = ConfigDict(...)` - Updated ConfigModel, PermissiveConfigModel, ConnectionModel base classes - Migrated 13 additional model classes across multiple files **Method Migrations (76 instances):** - `.parse_obj()` → `.model_validate()` (38 instances) - `.parse_raw()` → `.model_validate_json()` (2 instances) - `.dict()` → `.model_dump()` (27 instances) - `.json()` → `.model_dump_json()` (4 instances) - `.update_forward_refs()` → `.model_rebuild()` (3 instances) - `.copy()` and `.schema()` - all false positives (dicts/lists/HTTP responses) **Scope of Changes:** - metadata-ingestion/src: 50 instances - metadata-ingestion/tests: 30 instances - datahub-actions: 12 instances - smoke-test: 2 instances **Total: 94 Pydantic v1 calls migrated** **Key Files Updated:** - common.py: Base ConfigModel classes - Multiple source files: sql_queries.py, datahub_classifier.py, schema_assertion.py, etc. - Multiple CLI files: structuredproperties_cli.py, forms_cli.py, dataset_cli.py, etc. - Test files: RDS IAM tests, Unity Catalog tests, assertion tests, etc. - datahub-actions: propagation_action.py, filter tests, consumer offsets - smoke-test: stateful ingestion tests All Pydantic v2 deprecation warnings have been resolved. The codebase is now fully compliant with Pydantic v2 with no remaining v1 syntax.

github-actions bot added ingestion PR or Issue related to the ingestion of metadata smoke_test Contains changes related to smoke tests labels Nov 10, 2025

github-actions bot deployed to datahub-wheels (Preview) November 10, 2025 15:34 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) November 10, 2025 15:35 View deployment

vercel bot deployed to Preview November 10, 2025 15:48 View deployment

refactor(pydantic): remove Pydantic v1 legacy code and migrate fully …

9a68796

…to v2

sgomezvillamor force-pushed the remove-pydantic-v1-legacy-code branch from f906c45 to 9a68796 Compare November 11, 2025 08:03

github-actions bot deployed to datahub-wheels (Preview) November 11, 2025 08:05 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) November 11, 2025 08:08 View deployment

vercel bot deployed to Preview November 11, 2025 08:20 View deployment

sgomezvillamor marked this pull request as ready for review November 11, 2025 09:59

sgomezvillamor self-assigned this Nov 11, 2025

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 11, 2025

treff7es approved these changes Nov 11, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Nov 11, 2025

github-actions bot deployed to datahub-wheels (Preview) November 11, 2025 11:29 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) November 11, 2025 11:32 View deployment

vercel bot deployed to Preview November 11, 2025 11:45 View deployment

github-actions bot deployed to datahub-wheels (Preview) November 11, 2025 14:23 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) November 11, 2025 14:26 View deployment

vercel bot deployed to Preview November 11, 2025 14:39 View deployment

sgomezvillamor force-pushed the remove-pydantic-v1-legacy-code branch from e08616b to 30532e7 Compare November 11, 2025 18:09

github-actions bot deployed to datahub-wheels (Preview) November 11, 2025 18:11 View deployment

github-actions bot deployed to datahub-project-web-react (Preview) November 11, 2025 18:14 View deployment

vercel bot deployed to Preview November 11, 2025 18:27 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(pydantic): remove Pydantic v1 legacy code and migrate fully to v2 #15261

refactor(pydantic): remove Pydantic v1 legacy code and migrate fully to v2 #15261

sgomezvillamor commented Nov 10, 2025 •

edited

Loading

Uh oh!

alwaysmeticulous bot commented Nov 10, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 10, 2025 •

edited

Loading

Uh oh!

treff7es Nov 11, 2025

Uh oh!

sgomezvillamor Nov 11, 2025

Uh oh!

treff7es Nov 11, 2025

Uh oh!

sgomezvillamor Nov 11, 2025

Uh oh!

treff7es Nov 11, 2025

Uh oh!

sgomezvillamor Nov 11, 2025

Uh oh!

codecov bot commented Nov 11, 2025

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

refactor(pydantic): remove Pydantic v1 legacy code and migrate fully to v2 #15261

Are you sure you want to change the base?

refactor(pydantic): remove Pydantic v1 legacy code and migrate fully to v2 #15261

Conversation

sgomezvillamor commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

What Changed

Deleted compatibility layer files:

Migrated to native Pydantic v2 APIs (38 files):

Removed all version checks:

Updated discriminated unions:

Updated configurations:

Breaking Changes

Testing

Files Changed

Uh oh!

alwaysmeticulous bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

treff7es Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

treff7es Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

treff7es Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

sgomezvillamor Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 11, 2025

Bundle Report

Affected Assets, Files, and Routes:

Assets Changed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sgomezvillamor commented Nov 10, 2025 •

edited

Loading

alwaysmeticulous bot commented Nov 10, 2025 •

edited

Loading

codecov bot commented Nov 10, 2025 •

edited

Loading