Skip to content

update hffilesystem docstring #3177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

update hffilesystem docstring #3177

wants to merge 1 commit into from

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Jun 24, 2025

it was referring to overheads that don't exist anymore

I think it was about file streaming and copying files but that have been optimized some time ago:

Regarding the point on reliability, I think it was related to the caching mechanism mentioned here: #2643 but lmk what you think

@lhoestq lhoestq requested a review from hanouticelina June 24, 2025 15:47
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Wauplin
Copy link
Contributor

Wauplin commented Jun 25, 2025

Even though we've fixed some issues on the most important aspects (hf_transfer, efficient copies, etc.) it is still true that using fsspec adds some overhead since there is the abstraction layer on top of HfApi. It's not that I want to discourage people to use it when they need fsspec API compatibility but for day to day usage I still think it's best to push for HfApi.

(basically with HfApi we "own" all the Python stack, with HfFileSystem we delegate part of it to an abstraction layer which necessarily adds complexity when debugging)

@lhoestq
Copy link
Member Author

lhoestq commented Jun 26, 2025

I'm a bit hesitant in keeping "overhead" as is since it's pretty vague. Maybe it would be best to clarify which HfFileSystem operations are different with HfApi and why. For instance HfFileSystem is better suited than HfApi to stream files or remote random access (since HfApi doesn't have such a feature IIRC), but is less suited for git operations (it does one commit per upload). What do you think ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants