feat(docs): Adding models + APIs for context base V1 #15191

jjoyce0510 · 2025-11-04T02:50:00Z

Introducing Documents in DataHub (Context)

This PR introduces a new Document entity to DataHub, enabling users to create, manage, and organize first-party knowledge base content directly within the platform. Documents can be hierarchically organized, linked to data assets, and managed through a complete lifecycle including draft/publish workflows.

Core Data Models

Introduces comprehensive metadata models for the Document entity in DataHub:

Entity Definition

New document entity with key aspect documentKey and search capabilities
Full support for standard DataHub aspects: ownership, domains, tags, glossary terms, structured properties, institutional memory

Core Aspects (PDL Models)

DocumentKey - Unique identifier for documents
DocumentInfo - Primary aspect containing:
- Title and text contents
- Document status (PUBLISHED/UNPUBLISHED)
- Source information (distinguishes first-party vs third-party ingested documents)
- Audit stamps (created/lastModified with actor and timestamp)
- Hierarchical parent-child relationships
- Related assets (datasets, dashboards, etc.) and related documents
- Draft workflow support via draftOf field
DocumentContents - Text content storage
DocumentStatus & DocumentState - Publication state management
DocumentSource - Tracking external sources for third-party integrations
ParentDocument, RelatedAsset, RelatedDocument - Relationship models
DraftOf - Draft-to-published document linking

GraphQL APIs

Comprehensive GraphQL API surface in knowledge.graphql:

Mutations

createDocument - Create new documents with content, relationships, and hierarchy
- Supports custom IDs or auto-generated UUIDs
- Can create as draft or published
- Automatic ownership assignment to creator
updateDocumentContents - Update document text and title
updateDocumentRelatedEntities - Manage relationships to assets and other documents
moveDocument - Relocate documents within the hierarchy
deleteDocument - Remove documents and their references
updateDocumentStatus - Toggle between PUBLISHED/UNPUBLISHED states
mergeDraft - Merge draft content into published document with optional draft deletion

Queries

document(urn) - Fetch document by URN with full metadata
searchDocuments - Hybrid semantic search with rich filtering:
- Semantic query support
- Filter by parent document (hierarchical browsing)
- Filter by types, domains, states
- Option to include/exclude drafts
- Faceted search support

Special Features

drafts field - Lists all draft versions of a published document
changeHistory field - Chronological audit log of document modifications with support for: Content changes, Parent changes (moves), Relationship changes, State changes, etc.

Authorization & Privileges

New Platform Privilege

MANAGE_DOCUMENTS - Platform-level privilege for managing all documents

Entity-Level Privileges

Documents support standard DataHub entity privileges:

VIEW_ENTITY_PAGE / GET_ENTITY - View document
EDIT_ENTITY_DOCS / EDIT_ENTITY - Edit document content
CREATE_ENTITY - Create documents
EDIT_ENTITY_OWNERS - Manage ownership
EDIT_ENTITY_DOMAINS - Assign domains
SHARE_ENTITY - Share documents
EDIT_ENTITY_PROPERTIES - Edit structured properties

Authorization Logic

canCreateDocument() - Requires CREATE_ENTITY for documents or MANAGE_DOCUMENTS
canEditDocument() - Requires EDIT_ENTITY_DOCS, EDIT_ENTITY, or MANAGE_DOCUMENTS
canGetDocument() - Requires VIEW_ENTITY_PAGE or MANAGE_DOCUMENTS
canDeleteDocument() - Requires delete authorization or MANAGE_DOCUMENTS

Backend Services

DocumentService

Complete service layer implementation in metadata-service/services:

CRUD operations with validation
Draft workflow management (create, merge, track)
Hierarchical structure management (move operations)
Relationship management (assets and documents)
Ownership management
State transition handling
Full audit trail via lastModified timestamps

Timeline Support

DocumentInfoChangeEventGenerator - Generates change events for audit history
Tracks all modifications to document aspects
Integrates with DataHub's timeline service

Factory Beans

DocumentServiceFactory - Spring factory for service instantiation
Integration with GraphQL engine

Test Coverage

Smoke Tests

document_test.py (410 lines) - End-to-end document lifecycle tests
document_draft_test.py (326 lines) - Draft creation, merging, and workflows
document_change_history_test.py (281 lines) - Timeline and change tracking

Unit Tests

DocumentServiceTest.java (486 lines) - Service layer business logic
GraphQL resolver tests for all mutations and queries
DocumentMapperTest.java - Type mapping validation
DocumentInfoChangeEventGeneratorTest.java - Timeline event generation

Key Features & Use Cases

Knowledge Base Management - Create and organize internal documentation, FAQs, tutorials, and runbooks
Asset Documentation - Link documents to data assets for enriched context
Draft Workflows - Work on document updates without publishing immediately
Hierarchical Organization - Structure documents in parent-child relationships
Semantic Search - Find relevant documents through hybrid search
Change Tracking - Full audit history of all document modifications
Third-Party Integration Ready - Source field supports ingesting external docs (Confluence, Notion, etc.)

This PR lays the foundation for DataHub to become a central knowledge hub, combining first-party documentation with data asset management in a unified platform.

Coming in a followup PR:

Add a browse paths for docs, enabling us to replicate hierarchical structure from other places.
Add the "container" story for docs. One option is to define a parent container type as a Dataset entity (e.g. Dataset = Collection of Documents) which is then itself within a container.
Models for document-level lineage, and UI support for creating document level lineage links.
Support Document Tags, Glossary Terms, and inclusion in Data Products

Status

Ready for review.

codecov · 2025-11-06T19:30:07Z

Codecov Report

❌ Patch coverage is 70.14170% with 295 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...com/linkedin/metadata/service/DocumentService.java	61.97%	102 Missing and 25 partials ⚠️
...graphql/resolvers/knowledge/DocumentResolvers.java	33.96%	35 Missing ⚠️
...atahub/graphql/types/knowledge/DocumentMapper.java	72.80%	26 Missing and 8 partials ⚠️
...lvers/knowledge/DocumentChangeHistoryResolver.java	67.74%	12 Missing and 18 partials ⚠️
...l/resolvers/knowledge/SearchDocumentsResolver.java	69.87%	15 Missing and 10 partials ⚠️
...ql/resolvers/knowledge/CreateDocumentResolver.java	77.77%	9 Missing and 7 partials ⚠️
...com/linkedin/datahub/graphql/GmsGraphQLEngine.java	0.00%	8 Missing ⚠️
.../datahub/graphql/types/knowledge/DocumentType.java	78.37%	8 Missing ⚠️
...ahub/graphql/authorization/AuthorizationUtils.java	76.00%	6 Missing ⚠️
...raphql/types/common/mappers/UrnToEntityMapper.java	0.00%	3 Missing and 1 partial ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

abedatahub

Looks good to land; a few minor comments.

abedatahub · 2025-11-10T21:44:14Z

metadata-models/src/main/pegasus/com/linkedin/knowledge/DocumentInfo.pdl

+    /**
+     * Information about the external source of this document. 
+     * Only populated for third-party documents ingested from external systems.
+     * If null, the document is first-party (created directly in DataHub).


If this is the convention then you can remove the sourceType field in DocumentSource

abedatahub · 2025-11-10T21:59:04Z

...raphql-core/src/main/java/com/linkedin/datahub/graphql/authorization/AuthorizationUtils.java


+  /**
+   * Returns true if the current user is able to create Knowledge Articles. This is true if the user
+   * has the 'Create Entity' privilege for Knowledge Articles or 'Manage Knowledge Articles'


Are there plans for fine grained (per-document view, create-child, edit) privileges?

abedatahub · 2025-11-10T22:00:53Z

...e/src/main/java/com/linkedin/datahub/graphql/resolvers/knowledge/DeleteDocumentResolver.java

+
+            return true;
+          } catch (Exception e) {
+            log.error(


No need to log here since the caller can log the exception which has all the needed debugging info.

abedatahub · 2025-11-10T22:25:22Z

...ain/java/com/linkedin/datahub/graphql/resolvers/knowledge/DocumentChangeHistoryResolver.java

+    implements DataFetcher<CompletableFuture<List<DocumentChange>>> {
+
+  private final TimelineService _timelineService;
+  private static final long DEFAULT_LOOKBACK_MILLIS = 30L * 24 * 60 * 60 * 1000; // 30 days


TimeUnit.DAYS.toMillis(30)

abedatahub · 2025-11-10T22:33:34Z

...ain/java/com/linkedin/datahub/graphql/resolvers/knowledge/DocumentChangeHistoryResolver.java

+            long endTime = endTimeMillis != null ? endTimeMillis : System.currentTimeMillis();
+            long startTime =
+                startTimeMillis != null ? startTimeMillis : (endTime - DEFAULT_LOOKBACK_MILLIS);
+            int maxResults = limit != null ? limit : DEFAULT_LIMIT;


You should use the java.time APIs to do time math.

abedatahub · 2025-11-10T22:39:51Z

metadata-service/services/src/main/java/com/linkedin/metadata/service/DocumentService.java

+    // Batch ingest all proposals
+    entityClient.batchIngestProposals(opContext, mcps, false);
+
+    log.info("Updated contents for document {}", documentUrn);


This looks more like a debug log.

abedatahub · 2025-11-10T22:40:45Z

metadata-service/services/src/main/java/com/linkedin/metadata/service/DocumentService.java

+      log.error(
+          "Failed to clear entity references for Document with URN {}: {}",
+          documentUrn,
+          e.getMessage());


I think we want the stacktrace?

abedatahub · 2025-11-10T22:46:21Z

metadata-service/services/src/main/java/com/linkedin/metadata/service/DocumentService.java

+      }
+    } catch (Exception e) {
+      // If we can't get parent info, assume no cycle for safety
+      log.warn("Failed to check parent info for {}: {}", currentParent, e.getMessage());


Log the exception object

abedatahub · 2025-11-10T22:47:57Z

metadata-service/services/src/main/java/com/linkedin/metadata/service/DocumentService.java

+    infoProposal.setAspect(GenericRecordUtils.serializeAspect(publishedInfo));
+    entityClient.ingestProposal(opContext, infoProposal, false);
+
+    log.info("Merged draft {} into published document {}", draftUrn, publishedUrn);


This and others look like they should be debug level. And we can turn up log levels for specific modules at runtime

Adding models + APIs for context base V1

2e3ca2c

github-actions bot added product PR or Issue related to the DataHub UI/UX devops PR or Issue related to DataHub backend & deployment smoke_test Contains changes related to smoke tests labels Nov 4, 2025

vercel bot deployed to Preview November 4, 2025 03:07 View deployment

add tags and glossary terms

7d7ae01

vercel bot deployed to Preview November 4, 2025 03:54 View deployment

spotless apply

0dd1a86

vercel bot deployed to Preview November 5, 2025 02:17 View deployment

Fix failing document service test

aa22d2d

jjoyce0510 marked this pull request as ready for review November 5, 2025 22:06

datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 5, 2025

abedatahub self-requested a review November 6, 2025 16:48

Adding type changes

fca0f3d

vercel bot deployed to Preview November 6, 2025 18:59 View deployment

abedatahub approved these changes Nov 10, 2025

View reviewed changes

datahub-cyborg bot added pending-submitter-merge and removed needs-review Label for PRs that need review from a maintainer. labels Nov 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(docs): Adding models + APIs for context base V1 #15191

feat(docs): Adding models + APIs for context base V1 #15191

Uh oh!

jjoyce0510 commented Nov 4, 2025 •

edited

Loading

Uh oh!

codecov bot commented Nov 6, 2025 •

edited

Loading

Uh oh!

abedatahub left a comment

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

abedatahub Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(docs): Adding models + APIs for context base V1 #15191

Are you sure you want to change the base?

feat(docs): Adding models + APIs for context base V1 #15191

Uh oh!

Conversation

jjoyce0510 commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introducing Documents in DataHub (Context)

Core Data Models

Entity Definition

Core Aspects (PDL Models)

GraphQL APIs

Mutations

Queries

Special Features

Authorization & Privileges

New Platform Privilege

Entity-Level Privileges

Authorization Logic

Backend Services

DocumentService

Timeline Support

Factory Beans

Test Coverage

Smoke Tests

Unit Tests

Key Features & Use Cases

Coming in a followup PR:

Status

Uh oh!

codecov bot commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

abedatahub left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jjoyce0510 commented Nov 4, 2025 •

edited

Loading

codecov bot commented Nov 6, 2025 •

edited

Loading