Skip to content

Conversation

clefourrier
Copy link
Member

This PR does 2 things:

  1. add IFBench (process of comparing the numbers with the paper ongoing)
  2. refacto the registry to make it simpler
  • the successive functions which must be called in a specific order are now called in the init, functions have also been merged to make the flow easier
  • the registry stores the configs once
  • last but not least, the registry no longer tries to load all functions (including multilingual and extended and community), but uses the suites requested in the kwargs to know which to try.

@@ -42,7 +42,7 @@
from langcodes import Language as LangCodeLanguage
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in community tasks, changes are unrelated, it's just a bug fix

@HuggingFaceDocBuilderDev
Copy link
Collaborator

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@clefourrier
Copy link
Member Author

Ok good to review, sorry about the monster PR

@clefourrier clefourrier requested a review from NathanHB August 29, 2025 00:13
@@ -90,7 +90,7 @@ Here’s a quick command to evaluate using the Accelerate backend:
```shell
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the tasks edit are removing the truncating few shots, since it's no longer used

@clefourrier
Copy link
Member Author

Main question at this point @NathanHB is : do you want me to split the registry edit into its own PR, and put ifbench in another one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants