Support returning ContentToolResultImage from a tool with ChatAnthropic
#118
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for returning tool results as ContentToolResultImage or ContentToolResultResource, implements image serialization for Anthropic, and raises errors for unsupported image/resource results in OpenAI and Google providers.
- Switched the Anthropiс test helper to return
ContentToolResultImage - Updated OpenAI/Google providers to detect and reject image/resource results
- Extended Anthropic provider to serialize images and stub resource support
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_provider_anthropic.py | Update test to use ContentToolResultImage instead of generic ContentToolResult. |
| chatlas/_openai.py | Added imports and NotImplementedError for image/resource tool results. |
| chatlas/_google.py | Added imports and NotImplementedError for image/resource tool results (with a matching check). |
| chatlas/_content.py | Introduced get_model_value stub for new content types. |
| chatlas/_anthropic.py | Implement image serialization and stub resource support. |
| chatlas/init.py | Re-export ContentToolResultImage. |
Comments suppressed due to low confidence (3)
chatlas/_openai.py:489
- Add tests for the OpenAI provider to verify that
NotImplementedErroris raised when a tool returns an image or resource result.
):
chatlas/_google.py:434
- Add tests for the Google provider to ensure it raises
NotImplementedErrorfor image or resource tool results.
)
chatlas/init.py:5
- Consider also re-exporting
ContentToolResultResourceto keep the public API consistent with the newly supported content types.
from ._content import ContentToolRequest, ContentToolResult, ContentToolResultImage
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
cpsievert
changed the title
ContentToolResultImage/ContentToolResultResource where we canContentToolResultImage from a tool with ChatAnthropic
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ContentToolResultImageandContentToolResultResoucewere first added in #39 (since MCP tools can return images and resources), but that PR didn't add any logic to convert these tool results into a provider-specific content type for response generationUnfortunately, Anthropic is currently the only provider to support non-text tool results. More specifically, it only supports images. This PR adds support for
ContentToolResultImagefrom a tool forChatAnthropic, but throws ifContentToolResultImageandContentToolResultResouceare used in any other context.Perhaps there is a trick we can apply to include these results in a clever way for some providers, but I'm punting on that for now.