Skip to content

Ingestion Pipeline Failure Criteria #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Ingestion Pipeline Failure Criteria #77

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions docs/ui-ingestion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,17 +283,23 @@ Once cancelled, you can view the output of the ingestion run by clicking **Detai

A variety of things can cause an ingestion run to fail. Common reasons for failure include:

1. **Recipe Misconfiguration**: A recipe has not provided the required or expected configurations for the ingestion source. You can refer
1. **Failures in Source or Sink**: If any failures are reported by the source or sink during the ingestion process, the pipeline can be marked as failed.

2. **Uncaught Exceptions**: Uncaught exceptions during pipeline execution result in a failure status.

3. **Commit Policy**: The commit policy affects pipeline status, particularly if errors or warnings prevent meeting commit policy requirements.

4. **Recipe Misconfiguration**: A recipe has not provided the required or expected configurations for the ingestion source. You can refer
to the [Metadata Ingestion Framework](https://datahubproject.io/docs/metadata-ingestion) source docs to learn more about the configurations required for your source type.

2. **Failure to resolve Secrets**: If DataHub is unable to find secrets that were referenced by your Recipe configuration, the ingestion run will fail.
5. **Failure to resolve Secrets**: If DataHub is unable to find secrets that were referenced by your Recipe configuration, the ingestion run will fail.
Verify that the names of the secrets referenced in your recipe match those which have been created.

3. **Connectivity / Network Reachability**: If DataHub is unable to reach a data source, for example due to DNS resolution
6. **Connectivity / Network Reachability**: If DataHub is unable to reach a data source, for example due to DNS resolution
failures, metadata ingestion will fail. Ensure that the network where DataHub is deployed has access to the data source which
you are trying to reach.

4. **Authentication**: If you've enabled [Metadata Service Authentication](authentication/introducing-metadata-service-authentication.md), you'll need to provide a Personal Access Token
7. **Authentication**: If you've enabled [Metadata Service Authentication](authentication/introducing-metadata-service-authentication.md), you'll need to provide a Personal Access Token
in your Recipe Configuration. To so this, set the 'token' field of the sink configuration to contain a Personal Access Token:

<p align="center">
Expand Down