Make sentence_transformers an optional dependency

[theparthgupta]

Summary

This PR makes sentence_transformers an optional dependency as requested in #636. The dependency is heavy and not needed for all use cases, so users can now choose whether to install it.

Changes Made

• pyproject.toml: Moved sentence_transformers from required dependencies to optional dependencies under the sentence-transformers extra
• python/cocoindex/features.py: Added conditional import logic to only define SentenceTransformerEmbed when the dependency is available
• python/cocoindex/init.py: Updated to conditionally export SentenceTransformerEmbed only when available
• examples/: Added installation instructions to all example files that use SentenceTransformerEmbed

Installation Options

Users can now install with:

# With sentence-transformers support
pip install 'cocoindex[sentence-transformers]'

# Or install the dependency separately
pip install sentence-transformers

Fixes #636

[badmonster0]

Can you also help to update the documentation for the SentenceTransformerEmbed function?

[lemorage]

Hey, personally, I feel that for the extra name, sentence-transformers is a bit long and does not quite make sense. Following PEP 621, Extras collect specific subsets of optional dependencies based on features. Maybe we can change it to some descriptive names like ml or embeddings. This way, it is also quite extensible for future changes.

[theparthgupta]

I've updated all 7 examples that use SentenceTransformerEmbed with the new optional dependency approach.

I noticed that the main README.md and several docs files also reference SentenceTransformerEmbed. Should I update those files with the new installation instructions as well, or would you prefer to handle the documentation updates yourself?

Files that might need updates:

• README.md (main example uses SentenceTransformerEmbed)
• docs/quickstart.md
• docs/basics.md
• docs/functions.md
• docs/custom_function.mdx

[badmonster0]

Hey, personally, I feel that for the extra name, sentence-transformers is a bit long and does not quite make sense. Following PEP 621, Extras collect specific subsets of optional dependencies based on features. Maybe we can change it to some descriptive names like ml or embeddings. This way, it is also quite extensible for future changes.

Agree, sentence-transformers is indeed too long and probably too specific. +1 for using embedding as the feature name.

[badmonster0]

I've updated all 7 examples that use SentenceTransformerEmbed with the new optional dependency approach.

I noticed that the main README.md and several docs files also reference SentenceTransformerEmbed. Should I update those files with the new installation instructions as well, or would you prefer to handle the documentation updates yourself?

Files that might need updates:

• README.md (main example uses SentenceTransformerEmbed)
• docs/quickstart.md
• docs/basics.md
• docs/functions.md
• docs/custom_function.mdx

It'll be great if you can help to update these docs in the current PR:

• docs/quickstart.md (in the instruction for pip install cocoindex, we need to add this new optional feature)
• docs/functions.md (we need to tell users this SentenceTransformerEmbed function needs this new optional feature)

Other docs may be just mentioning SentenceTransformerEmbed. Seems they don't need to be updated.

README.md usually doesn't need to be updated either - after the dependency is added into pyproject.toml, the dependency will be automatically installed in pip install.

Thanks!

[theparthgupta]

I've addressed all the feedback regarding documentation updates:
Changes made:

• ✅ docs/quickstart.md: Updated pip install command to pip install -U 'cocoindex[sentence-transformers]'
• ✅ docs/functions.md: Added optional dependency note for SentenceTransformerEmbed function with clear installation instructions
• ✅ All 7 examples: Removed manual installation instructions from README.md and main.py files as requested

[badmonster0]

Hey, personally, I feel that for the extra name, sentence-transformers is a bit long and does not quite make sense. Following PEP 621, Extras collect specific subsets of optional dependencies based on features. Maybe we can change it to some descriptive names like ml or embeddings. This way, it is also quite extensible for future changes.

Agree, sentence-transformers is indeed too long and probably too specific. +1 for using embedding as the feature name.

@theparthgupta what do you think about this? Would you also change the feature name to embedding? Thanks!

[theparthgupta]

Hey, personally, I feel that for the extra name, sentence-transformers is a bit long and does not quite make sense. Following PEP 621, Extras collect specific subsets of optional dependencies based on features. Maybe we can change it to some descriptive names like ml or embeddings. This way, it is also quite extensible for future changes.

Agree, sentence-transformers is indeed too long and probably too specific. +1 for using embedding as the feature name.

@theparthgupta what do you think about this? Would you also change the feature name to embedding? Thanks!

Agreed on changing it! I'd lean toward embeddings (plural) since it typically handles multiple embeddings, but either embedding or embeddings works well. Much better than the current long name. If you agree i can work on this too

[badmonster0]

embeddings

embeddings looks good. Thanks!

[theparthgupta]

embeddings

embeddings looks good. Thanks!

Sounds good! I'll get started on the implementation.

[theparthgupta]

I've updated the optional dependency group name from sentence-transformers to embeddings as discussed. Let me know if you'd like any other naming adjustments or additions

[badmonster0]

Thanks for this change!

[theparthgupta]

Thanks for this change!

thanks for merging the PR!

I noticed CI is failing on main after my PR was merged — looks like mypy can't find sentence_transformers or numpy. I think it's due to the optional import. Should I open a quick fix to update mypy.ini or CI deps?

[lemorage]

@theparthgupta The CI failure is largely due to the missing numpy dep, after making sentence_transformers optional. I've made the fix, and feel free to check out #676.

[theparthgupta]

@theparthgupta The CI failure is largely due to the missing numpy dep, after making sentence_transformers optional. I've made the fix, and feel free to check out #676.

thanks for doing it