Skip to content

[Xet] Basic shard creation #1633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 52 commits into from
Aug 26, 2025
Merged

[Xet] Basic shard creation #1633

merged 52 commits into from
Aug 26, 2025

Conversation

coyotte508
Copy link
Member

@coyotte508 coyotte508 commented Jul 16, 2025 •
edited
Loading

cc @Kakulukian @assafvayner for viz, follow up to #1616

Based on https://github.com/huggingface/xet-core/blob/7e41fb0dd7cfb276222b9668d0b97a984647721e/spec/shard.md

Need to handle:

  • split into multiple shards when xorb or file info grows too big
  • uploading xorbs & shards (and we need to upload xorbs before shards referencing them)

coyotte508
coyotte508 commented Jul 16, 2025
View reviewed changes
coyotte508 added 25 commits July 16, 2025 21:38
@coyotte508
remove shard key expiry
6676431
@coyotte508
actually make API calls to xet backend to upload shards/xorbs
3fc715b
@coyotte508
fix prefix for shard upload
7a38fc7
@coyotte508
fix verificaiton data
947a926
@coyotte508
update wasm bindings
9cd2e66
@coyotte508
fixup! update wasm bindings
bf8ae1c
@coyotte508
no need to compute shard hash client-side
70f0a0e
@coyotte508
no need for shard hash
431949b
@coyotte508
Merge remote-tracking branch 'origin/main' into shard-creation
515cae6
@coyotte508
add va prefix to xorb too
adbe363
@coyotte508
progress events for uploading xorbs
2f9d4a0
@coyotte508
commit leftover xorb after all chunks have been processed
a13da79
@coyotte508
integrate xet upload in commit function
3cbde5e
@coyotte508
add local dedup for xet uploads
651f6c0
@coyotte508
move chunk caching to its own class/file
f3e190f
@coyotte508
Make sure to not OOB when writing shards
06dcd6d
@coyotte508
dedup boolean when loading chunks from wasm
c093c64
@coyotte508
global dedup! (just need hmac algorithm)
2e27699
@coyotte508
delay file events until matching xorb is emitted
7c397dd
@coyotte508
add dedup ratio to information
5f3a61b
@coyotte508
fixup! add dedup ratio to information
66299a3
@coyotte508
use hmac function from wasm
988d85b
@coyotte508
add bench script
433284e
@coyotte508
fix wasm instantiation
74dfb71
@coyotte508
fix api calls
b19209f
coyotte508 and others added 23 commits August 1, 2025 14:11
@coyotte508
Copy link
Member Author

coyotte508 commented Aug 26, 2025 •
edited
Loading

Merging, still experimental

Optimizations remaining:

@coyotte508 coyotte508 merged commit 754b069 into main Aug 26, 2025
3 of 6 checks passed
@coyotte508 coyotte508 deleted the shard-creation branch August 26, 2025 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@assafvayner assafvayner assafvayner left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@coyotte508 @assafvayner