Skip to content

Troubleshooting Column Description Propagation #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Troubleshooting Column Description Propagation #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5c75872
Update docs-propagation.md
Apr 18, 2025
5d422c0
Update lineage.md
Apr 18, 2025
ff63334
Update datahub-apis.md
Apr 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/api/datahub-apis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,24 @@ Here's an overview of what each API can do.
| Remove Owner from a Dataset | ✅ [[Guide]](/docs/api/tutorials/owners.md#remove-owners) | ✅ [[Guide]](/docs/api/tutorials/owners.md) | ✅ |
| Add Lineage | ✅ [[Guide]](/docs/api/tutorials/lineage.md) | ✅ [[Guide]](/docs/api/tutorials/lineage.md#add-lineage) | ✅ |
| Add Column Level (Fine Grained) Lineage | 🚫 | ✅ [[Guide]](docs/api/tutorials/lineage.md#add-column-level-lineage) | ✅ |

## Troubleshooting Column Description Propagation

### Verify Propagation Settings
Ensure that column-level documentation propagation is enabled in your DataHub settings. This feature is enabled by default in Open Source DataHub.

### Check Lineage Configuration
- **DataHub UI**: Navigate to the dataset and check the "Lineage" tab to ensure column-level lineage is correctly configured.
- **Code**: Verify `FineGrainedLineage` objects are correctly defined in your code.

### Review Ingestion Logs
- **Docker**: Use `docker logs <container_id>` to check logs.
- **Kubernetes**: Use `kubectl logs <pod_name>` to check logs.
- **CLI**: Use `--debug` flag for detailed logs.

### Manual Verification
- **DataHub UI**: Check the schema details to ensure the column description is set.
- **Python SDK**: Use the SDK to manually emit metadata if necessary.
| Add Documentation (Description) to a Column of a Dataset | ✅ [[Guide]](/docs/api/tutorials/descriptions.md#add-description-on-column) | ✅ [[Guide]](/docs/api/tutorials/descriptions.md#add-description-on-column) | ✅ |
| Add Documentation (Description) to a Dataset | ✅ [[Guide]](/docs/api/tutorials/descriptions.md#add-description-on-dataset) | ✅ [[Guide]](/docs/api/tutorials/descriptions.md#add-description-on-dataset) | ✅ |
| Add / Remove / Replace Custom Properties on a Dataset | 🚫 | ✅ [[Guide]](/docs/api/tutorials/custom-properties.md) | ✅ |
Expand Down
19 changes: 19 additions & 0 deletions docs/api/tutorials/lineage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,25 @@ You can now see the column-level lineage between datasets. Note that you have to
<img width="70%" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/apis/tutorials/column-level-lineage-added.png"/>
</p>

## Troubleshooting Column Description Propagation

### Verify Propagation Settings
Ensure that column-level documentation propagation is enabled in your DataHub settings. This feature is enabled by default in Open Source DataHub.

### Check Lineage Configuration
- **DataHub UI**: Navigate to the DataHub UI and locate the dataset. Go to the "Lineage" tab and enable the "Show Columns" option to visualize column-level lineage.
- **Lineage Configuration in Code**: Review your code to ensure that the `FineGrainedLineage` objects are correctly defined.

### Review Ingestion Logs
- **Docker Logs**: Use `docker logs <container_id>` to check for errors or warnings.
- **Kubernetes Logs**: Use `kubectl logs <pod_name>` to check logs of the DataHub pods.
- **CLI Debug Logs**: Use the `--debug` flag with the DataHub CLI to enable detailed logging.

### Manual Verification
- **Inspect DBT Artifacts**: Check `manifest.json` and `catalog.json` files for correct descriptions.
- **DataHub UI**: Verify the schema details in the DataHub UI.
- **Manual Metadata Emission**: Use the DataHub Python SDK to manually emit metadata if necessary.

## Add Lineage to Non-Dataset Entities

You can also add lineage to non-dataset entities, such as DataJobs, Charts, and Dashboards.
Expand Down
18 changes: 18 additions & 0 deletions docs/automations/docs-propagation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,24 @@ and then click "Rollback".

This one-time step will remove all propagated tags and glossary terms from Snowflake. To simply stop propagating new tags, you can disable the automation.

## Troubleshooting Column Description Propagation

### Verify Propagation Settings
Ensure that column-level documentation propagation is enabled. In Open Source DataHub, this feature is enabled by default. Navigate to the 'Settings' and 'Features' tab to confirm.

### Check Lineage Configuration
- **DataHub UI**: Navigate to the dataset and check the 'Lineage' tab with 'Show Columns' enabled to verify column-level lineage.
- **Code Verification**: Ensure `FineGrainedLineage` objects are correctly defined in your code.

### Review Ingestion Logs
- **Docker**: Use `docker logs <container_id>` to check for errors.
- **Kubernetes**: Use `kubectl logs <pod_name>` for relevant pods like GMS or ingestion.
- **CLI**: Use `--debug` flag for detailed logs during ingestion.

### Manual Verification
- **DataHub UI**: Check the schema details of the source dataset to ensure the description is set.
- **Python SDK**: Use the SDK to manually emit metadata if necessary.

## Viewing Propagated Descriptions

Once the automation is enabled, you'll be able to recognize propagated descriptions as those with the thunderbolt icon next to them:
Expand Down