Implement inline citation #1821
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment
|
… chains refactor: enhance citation utilities and improve source handling in chain runners - Introduced new utility functions for citation management, including sanitization and formatting of source catalogs. - Updated CopilotPlusChainRunner and VaultQAChainRunner to utilize the new citation utilities for improved citation handling. - Enhanced ChatSingleMessage component to consolidate duplicate sources and ensure accurate citation numbering. - Added comprehensive tests for citation utilities to ensure functionality and edge case handling. refactor: dedupe chunks for unique titles in search tool sources refactor: enhance citation handling in CopilotPlus and VaultQA chain runners - Introduced a sanitization function to remove pre-existing citation markers from document content, preventing number leakage. - Updated the formatting of retrieved documents in VaultQA to utilize the new sanitization function for cleaner output. - Improved footnote renumbering logic in ChatSingleMessage to prioritize first-mention order, ensuring accurate citation mapping. refactor: streamline source citation handling in ChatSingleMessage component and enhance web search tool instructions - Simplified the processing of source links in ChatSingleMessage to use a more efficient mapping approach. - Updated the web search tool's instruction to clarify the use of footnote-style citations, ensuring compatibility with Markdown formatting.
- Introduced a new setting to enable inline citations in AI-generated responses, allowing for footnote-style references within the text. - Updated citation utilities to manage fallback sources and improve citation instructions based on user settings. - Enhanced CopilotPlusChainRunner and VaultQAChainRunner to utilize the new inline citation feature, ensuring accurate source representation. - Modified ChatSingleMessage component to process inline citations based on the new setting, improving the display of sources in chat messages. - Added tests for new citation functionalities to ensure reliability and correctness.
- Introduced the `normalizeCitations` function to improve citation formatting by removing periods after citations and handling footnote references. - Updated `processInlineCitations` to ensure correct mapping of non-sequential citations and consolidate duplicate sources. - Added comprehensive tests for the new citation functionalities, ensuring accurate citation processing and representation in AI-generated responses. - Modified QASettings description for clarity regarding the inline citations feature.
…tions - Enhanced the `normalizeCitations` function to correctly handle consecutive citations without spaces, ensuring accurate mapping and representation in output. - Added a test case to replicate and validate the fix for the identified bug, confirming that both citations are processed and displayed correctly. - Updated existing citation handling logic to improve robustness and prevent unconverted citations in the output.
logancyang
force-pushed
the
citation
branch
from
e5ae8af to
abf8c9c
Compare
![korbit-ai[bot]](https://avatars.githubusercontent.com/in/322216?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review by Korbit AI
Korbit automatically attempts to detect when you fix issues in new commits.
| Category | Issue | Status |
|---|---|---|
 |
Scattered Settings Management ▹ view | |
 |
Citation ID mismatch due to filtered array indexing ▹ view | |
 |
Unnecessary content sanitization when citations disabled ▹ view | |
|
Quadratic time complexity in duplicate source detection ▹ view | |
|
Inefficient multi-pass citation normalization ▹ view |
Files scanned
| File Path | Reviewed |
|---|---|
| src/search/vectorStoreManager.ts | ✅ |
| src/LLMProviders/chainRunner/utils/searchResultUtils.ts | ✅ |
| src/LLMProviders/chainRunner/VaultQAChainRunner.ts | ✅ |
| src/tools/SearchTools.ts | ✅ |
| src/settings/model.ts | ✅ |
| src/main.ts | ✅ |
| src/settings/v2/components/QASettings.tsx | ✅ |
| src/tools/ToolResultFormatter.ts | ✅ |
| src/constants.ts | ✅ |
| src/LLMProviders/chainRunner/utils/citationUtils.ts | ✅ |
| src/components/chat-components/ChatSingleMessage.tsx | ✅ |
| src/LLMProviders/chainRunner/CopilotPlusChainRunner.ts | ✅ |
Explore our documentation to understand the languages and file types we support and the files we ignore.
Check out our docs on how you can make Korbit work best for you and your team.
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
- Refactored `CopilotPlusChainRunner` to streamline citation source management, replacing `lastCitationLines` with `lastCitationSources` for better clarity and structure. - Updated the `addFallbackSources` function to handle citation formatting more effectively, ensuring proper display of titles and paths. - Enhanced the `normalizeCitations` function to support multiple citations in brackets, ensuring correct sorting and mapping of citation numbers. - Added new test cases to validate the handling of alternate headings, HTML summaries, and multiple citations, improving overall robustness of citation processing. - Introduced `getWebSearchCitationInstructions` for web-specific citation guidance, enhancing user experience with clear instructions for inline citations.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
- Introduced `normalizeFootnoteRendering` function to streamline footnote display by removing separators and backreferences, and fixing numbering artifacts. - Updated `ChatSingleMessage` to utilize the new normalization function, enhancing the clarity and presentation of footnotes in chat messages. - Added unit tests for `normalizeFootnoteRendering` to ensure proper functionality and handling of various footnote scenarios.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
… search tools - Updated `ChatSingleMessage.test.tsx` to include `TooltipProvider` for improved rendering context. - Refactored mock implementation for `obsidian` to streamline testing setup. - Introduced new test file `SearchTools.dedupe.test.ts` to validate the deduplication logic in search tool sources, ensuring correct handling of duplicate entries while preserving the highest score. - Added tests to verify deduplication behavior for various scenarios, enhancing overall test coverage and reliability.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description by Korbit AI
What change is being made?
Implement inline citation support across the codebase, including new citation utilities, integration points in CopilotPlus and VaultQA chain runners, UI/settings changes to enable the feature by default, content sanitization, fallback sources handling, and extensive tests for citation processing and formatting.
Why are these changes being made?
To enable stable, readable inline citations and a consistent sources section, improve how sources are tracked and displayed (including fallback behavior when citations are missing), and provide end-to-end support (including UI settings and test coverage) for this experimental feature.