Created FAQ page first draft #1896
Conversation
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
Summary of ChangesHello @cajeonrh, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new Frequently Asked Questions (FAQ) page within the "Getting Started" section of the documentation. The primary goal is to centralize answers to common user queries about the LLM Compressor, thereby enhancing user self-service and clarity on topics such as model performance post-compression, integration with other tools like sglang, and practical guidance on compression strategies and memory requirements. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a new FAQ page, which is a great addition to the documentation. The content is relevant and covers important user questions. I've identified a few areas for improvement, mainly related to Markdown link formatting, content clarity, and consistency. There are several instances of incorrect link syntax that need to be fixed across the document. I've also suggested consolidating a couple of redundant questions and using relative paths for internal links to improve maintainability.
cajeonrh
force-pushed
the
INFERENG-1867
branch
from
21cb4f0 to
34edf9d
Compare
|
Thanks Fynn! I've incorporated your feedback. |
In the sidebar we're getting this title for the page. Can we simplify this to just "Frequently Asked Questions" or maybe even "FAQ". I believe this is being set by the |
Also on the "Getting Started" page we have these boxes for "Installation", "Compress Your Model" and "Deploy on vLLM". Could we add a box for FAQ? |
There was a problem hiding this comment.
The other common question we get asked is about multi-gpu support.
Can we add the following?
- LLM Compresor handles all gpu movement for you.
- For data-free pathways, we leverage all available gpus and offload anything that doesnt fit onto the allocated gpus. If using pathways that require data, we sequentially onload model layers onto a single gpu. This is the case for LLM Compressor 0.6-0.8.
|
I've incorporated feedback, added more questions, and also added a FAQ box on the Getting Started page. Please let me know if I missed anything. |
|
Looks like you need to fix DCO though. There are some instructions here: https://github.com/vllm-project/llm-compressor/pull/1896/checks?check_run_id=52066401360. |
There was a problem hiding this comment.
I'm really not a fan of using casual pronouns like "us", "we", "my". This may sound pedantic, but speaking from personal experience contributing to other OSS repos, words like "we" have the effect of alienating open source contributors. LLM Compressor is owned by everyone, the RedHat/ LLM Compressor team helps to maintain and shepherd it.
There was a problem hiding this comment.
We should add a section titled "Where can I learn more about LLM Compressor?" which links to talks we've given.
https://www.youtube.com/watch?v=caLYSZMVQ1c
https://www.youtube.com/watch?v=GrhuqQDmBk8
https://www.youtube.com/watch?v=WVenRmF4dPY
https://www.youtube.com/watch?v=G1WNlLxPLSE
|
|
||
| **7. Does LLM Compressor have multi-GPU support?** | ||
|
|
||
| LLM Compressor handles all GPU movement for you. For data-free pathways, we leverage all available GPUs and offload anything that doesn't fit onto the allocated GPUs. If you are using pathways that require data, we sequentially onload model layers onto a single GPU. This is the case for LLM Compressor 0.6-0.8. |
There was a problem hiding this comment.
We essentially do not have mult-GPU support right now.
There was a problem hiding this comment.
dispatch_for_generation is used for data-free atm?
There was a problem hiding this comment.
Is this FAQ still relevant then? Should I remove it or does it need to be changed?
There was a problem hiding this comment.
** 7. Does LLM Compressor support compressing large models? **
LLM Compressor enables the compression of large models via sequential onloading, whereby layers of the model are jointly onloaded to a single GPU, optimized, then offloaded back to the CPU. While this is the default data pipeline for most cases, you can override this setting for slightly better runtime by using oneshot(..., pipeline="basic") if you model can fit across all GPU resources. Multi-GPU parallel optimization is currently in development and being tracked here.
There was a problem hiding this comment.
I think we need to add a point re: data free pathways for FP8 quantization @kylesayrs
There was a problem hiding this comment.
Also, we need to keep the original question as most users ask about mutli-gpu support specifically / are confused when they only see one active gpu
There was a problem hiding this comment.
Hm, maybe something like this
** 7. Does LLM Compressor have multi-GPU support? **
LLM Compressor enables the compression of large models via sequential onloading, whereby layers of the model are jointly onloaded to a single GPU, optimized, then offloaded back to the CPU. This is why, in most cases, only one GPU is used at a time.
In cases where no calibration data is needed, the model is dispatched to all GPUs, although only one GPU is used at a time for compression.
Multi-GPU parallel optimization is currently in development and being tracked here.
Signed-off-by: Cassie Jeon <cajeon@redhat.com>
cajeonrh
force-pushed
the
INFERENG-1867
branch
from
286d259 to
8108e51
Compare
@kylesayrs |
|
@cajeonrh That’s fine, we can table the discussion for now |
|
|
||
| **5. What layers should be quantized?** | ||
|
|
||
| Typically, all linear layers are quantized except the `lm_head` layer. This is because the `lm_head` layer is the last layer of the model and sensitive to quantization, which will impact the model's accuracy. For example, [this code snippet shows how to ignore the lm_head layer](https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/llama3_example.py#L18). |
There was a problem hiding this comment.
This link and the one below might become stale if those files are ever changed, since they point to a file on main that can be changed. Should we use a tagged version instead?
5 participants
SUMMARY:
Created a FAQ page under the "Getting Started" section
TEST PLAN:
Requesting review of content