Skip to content

feat: Add video message part type support for Alibaba Cloud vision models #254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ding-ding666
Copy link

Summary

This PR adds support for a new video message part type in the eino schema to enable direct video frame transmission compatible with Alibaba Cloud's vision model services.

Background

Alibaba Cloud's Model Studio vision models support video content processing through a specific format that requires video frames to be sent as an array of base64-encoded images. This enhancement enables eino framework to work seamlessly with Alibaba Cloud's multimodal capabilities.

Reference: https://help.aliyun.com/zh/model-studio/vision#80dbf6ca8fh6s

Changes

  • Added ChatMessagePartTypeVideo = "video" constant for video message parts
  • Added ChatMessageVideo struct to handle video frames with base64-encoded data
  • Updated ChatMessagePart struct to include Video *ChatMessageVideo field
  • Updated message concatenation logic to handle video fields

API Changes

New Types

// ChatMessagePartTypeVideo represents video content with multiple frames
const ChatMessagePartTypeVideo ChatMessagePartType = "video"

// ChatMessageVideo handles video frames in base64 format for Alibaba Cloud compatibility
type ChatMessageVideo struct {
    Video    []string          `json:"video,omitempty"`    // Array of base64-encoded frames
    MIMEType string            `json:"mime_type,omitempty"` // Frame format (e.g., "image/jpeg")
    Extra    map[string]any    `json:"extra,omitempty"`     // Additional metadata
}

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ding-ding666
Copy link
Author

The following three links are a set of modifications:

#254
meguminnnnnnnnn/go-openai#3
cloudwego/eino-ext#280

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants