init #2852
init #2852
Conversation
diffusers-quantization.md
Outdated
|
|
||
| **BF16:** | ||
|
|
||
|  |
There was a problem hiding this comment.
Let's also provide an actual caption for the figure.
diffusers-quantization.md
Outdated
| **BnB 4-bit:** | ||
|
|
||
|  | ||
|
|
||
| **BnB 8-bit:** | ||
|  |
There was a problem hiding this comment.
Can we combine three images here?
- BF16
- 4bit
- 8bit
Along with the caption?
diffusers-quantization.md
Outdated
| **BnB 8-bit:** | ||
|  | ||
|
|
||
| | BnB Precision | Memory after loading | Peak memory | Inference time | |
There was a problem hiding this comment.
Let's include the Bf16 numbers too.
| | Q8_0 | 21.502 GB | 25.973 GB | 15 seconds | | ||
| | Q2_k | 13.264 GB | 17.752 GB | 26 seconds | | ||
|
|
||
| **Example (Flux-dev with GGUF Q4_1)** |
There was a problem hiding this comment.
I don't think we have to be exhaustive about showing snippets for every configuration unless they vary significantly from one another.
|
|
||
| For more information check out the [GGUF docs](https://huggingface.co/docs/diffusers/quantization/gguf). | ||
|
|
||
| ### FP8 Layerwise Casting (`enable_layerwise_casting`) |
There was a problem hiding this comment.
Could also write that can be combined with group_offloading:
https://huggingface.co/docs/diffusers/en/optimization/memory#group-offloading
diffusers-quantization.md
Outdated
|
|
||
| We created a setup where you can provide a prompt, and we generate results using both the original, high-precision model (e.g., Flux-dev in BF16) and several quantized versions (BnB 4-bit, BnB 8-bit). The generated images are then presented to you and your challenge is to identify which ones came from the quantized models. | ||
|
|
||
| Try it out [here](https://huggingface.co/spaces/derekl35/flux-quant)! |
There was a problem hiding this comment.
Let's embed the space inside the blog post.
diffusers-quantization.md
Outdated
| Here's a quick guide to choosing a quantization backend: | ||
|
|
||
| * **Easiest Memory Savings (NVIDIA):** Start with `bitsandbytes` 4/8-bit. | ||
| * **Prioritize Inference Speed:** `torchao` + `torch.compile` offers the best performance potential. |
There was a problem hiding this comment.
GGUF also supports torch.compile(). So does bitsandbytes. I think we should mention that.
diffusers-quantization.md
Outdated
| * **Simplicity (Hopper/Ada):** Explore FP8 Layerwise Casting (`enable_layerwise_casting`). | ||
| * **For Using Existing GGUF Models:** Use GGUF loading (`from_single_file`). | ||
|
|
||
| Quantization significantly lowers the barrier to entry for using large diffusion models. Experiment with these backends to find the best balance of memory, speed, and quality for your needs. |
There was a problem hiding this comment.
Should we hint the readers that they can expect a follow-up blog around training with quantization?
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Hey there, here is the thumbnail suggestion! cc @sayakpaul
|
|
@ChunTeLee possible to reduce the size of the middle object a bit so that "exploring" and "quantization" words are clear? |
Here you go! |
diffusers-quantization.md
Outdated
| ## Combining with Memory Optimizations | ||
|
|
||
| Most of these quantization backends can be combined with the memory optimization techniques offered in Diffusers. For example, using `enable_model_cpu_offload()` with `bitsandbytes` cuts the memory further, giving a reasonable trade-off between memory and latency. You can learn more about these techniques in the [Diffusers documentation](https://huggingface.co/docs/diffusers/main/en/optimization/memory). |
There was a problem hiding this comment.
What would be nice would be to show some results here with enable_model_cpu_offload and torch.compile ! Maybe we can test these with bnb 4-bit as it is compatible with the main version.
Then we can also combine all benchmark results and put it into a dataset so that users can easily compare.
There was a problem hiding this comment.
Having a central dataset sounds good, and it enables systematic exploration.
There was a problem hiding this comment.
Sounds good to me. Should I include code too? Not sure how to do torch.compile with pipeline level quant config
There was a problem hiding this comment.
This would be a cool feature to add @sayakpaul in pipeline level quant config actually. Maybe you can open a PR @DerekLiu35 in diffusers to support this scenario ?
There was a problem hiding this comment.
@SunMarc which scenario you're talking about? To maybe better focus on the blog post, it'd be good to file that request on our repo or discuss on Slack.
Sounds good to me. Should I include code too? Not sure how to do torch.compile with pipeline level quant config
We can individually compile the components of a pipeline. Like this:
https://github.com/sayakpaul/diffusers-torchao/blob/9b9f2383dccb8eb73a4c6a8ffe736dd9610c26d2/inference/benchmark_image.py#L40
| ></script> | ||
|
|
||
| Building on our previous post, "[Memory-efficient Diffusion Transformers with Quanto and Diffusers](https://huggingface.co/blog/quanto-diffusers)", this post explores the diverse quantization backends integrated directly into Hugging Face Diffusers. We'll examine how bitsandbytes, GGUF, torchao, and native FP8 support make large and powerful models more accessible, demonstrating their use with Flux (a flow-based text-to-image generation model). | ||
|
|
There was a problem hiding this comment.
let's do a quick introduction of the pipeline that we will be quantizing, share about what each components do, the memory it takes and what might be interesting to quantize.
There was a problem hiding this comment.
I believe it could be done in bulleted points.
diffusers-quantization.md
Outdated
| from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig | ||
| from diffusers.quantizers import PipelineQuantizationConfig | ||
| from transformers import T5EncoderModel | ||
| from transformers import BitsAndBytesConfig as TransformersBitsAndBytesConfig |
There was a problem hiding this comment.
Maybe explain somewhere that we need to be careful when they import the configs DiffusersBitsAndBytesConfig and TransformersBitsAndBytesConfig since the components comes from different libraries and that if they don't want to deal with that, they can use this instead
|
|
||
| | torchao Precision | Memory after loading | Peak memory | Inference time | | ||
| |-------------------------------|----------------------|-------------|----------------| | ||
| | int4_weight_only | 10.635 GB | 14.654 GB | 109 seconds | |
There was a problem hiding this comment.
any idea why it takes so much time ?
There was a problem hiding this comment.
I remember I faced something while working on https://github.com/sayakpaul/diffusers-torchao. IIUC it was a combination of unavailability of a good kernel, shape constraints, etc.
diffusers-quantization.md
Outdated
|
|
||
| | quanto Precision | Memory after loading | Peak memory | Inference time | | ||
| |------------------|----------------------|-------------|----------------| | ||
| | int4 | 12.254 GB | 16.139 GB | 109 seconds | |
diffusers-quantization.md
Outdated
| ) | ||
| ``` | ||
|
|
||
| > **Note:** At the time of writing, for float8 support with Quanto, you'll need `optimum-quanto<0.2.5` and use quanto directly. |
There was a problem hiding this comment.
this is why you didn't put the results for float8 ?
There was a problem hiding this comment.
Yeah, would need to add code to show how to quantize with float8:
import torch
from diffusers import AutoModel, FluxPipeline
from transformers import T5EncoderModel
from optimum.quanto import freeze, qfloat8, quantize
model_id = "black-forest-labs/FLUX.1-dev"
text_encoder_2 = T5EncoderModel.from_pretrained(
model_id,
subfolder="text_encoder_2",
torch_dtype=torch.bfloat16,
)
quantize(text_encoder_2, weights=qfloat8)
freeze(text_encoder_2)
transformer = AutoModel.from_pretrained(
model_id,
subfolder="transformer",
torch_dtype=torch.bfloat16,
)
quantize(transformer, weights=qfloat8)
freeze(transformer)
pipe = FluxPipeline.from_pretrained(
model_id,
transformer=transformer,
text_encoder_2=text_encoder_2,
torch_dtype=torch.bfloat16
).to("cuda")
pipe_kwargs = {
"prompt": "Ghibli style, a fantasy landscape with a grey castle with multiple tall, blue-roofed towers, beside a clear, flowing river in a lush green valley.",
"height": 1024,
"width": 1024,
"guidance_scale": 3.5,
"num_inference_steps": 50,
"max_sequence_length": 512,
}
print(f"Pipeline memory usage: {torch.cuda.max_memory_reserved() / 1024**3:.3f} GB")
image = pipe(**pipe_kwargs, generator=torch.manual_seed(0),).images[0]
print(f"Pipeline memory usage: {torch.cuda.max_memory_reserved() / 1024**3:.3f} GB")
image.save("flux-dev_quanto_fp8.png")There was a problem hiding this comment.
yeah let's add the code but make sure to say that we will be working on fixing this.
diffusers-quantization.md
Outdated
| ## Spot The Quantized Model | ||
|
|
||
| Quantization sounds great for saving memory, but how much does it *really* affect the final image? Can you even spot the difference? We invite you to test your perception! | ||
|
|
||
| We created a setup where you can provide a prompt, and we generate results using both the original, high-precision model (e.g., Flux-dev in BF16) and several quantized versions (BnB 4-bit, BnB 8-bit). The generated images are then presented to you and your challenge is to identify which ones came from the quantized models. | ||
|
|
||
| Try it out here! | ||
|
|
||
| <gradio-app theme_mode="light" space="derekl35/flux-quant"></gradio-app> | ||
|
|
||
| Often, especially with 8-bit quantization, the differences are subtle and may not be noticeable without close inspection. More aggressive quantization like 4-bit or lower might be more noticeable, but the results can still be good, especially considering the massive memory savings. |
There was a problem hiding this comment.
Maybe we can put that at the top since users will be definitely super interested in reading the rest to the blogpost after playing with the space !
There was a problem hiding this comment.
NF4 often gives the best trade-off though. At least for image models and saves considerable amount of memory. Also see this tip here:
diffusers-quantization.md
Outdated
| * **Simplicity (Hopper/Ada):** Explore FP8 Layerwise Casting (`enable_layerwise_casting`). | ||
| * **For Using Existing GGUF Models:** Use GGUF loading (`from_single_file`). | ||
|
|
||
| Quantization significantly lowers the barrier to entry for using large diffusion models. Experiment with these backends to find the best balance of memory, speed, and quality for your needs. |
diffusers-quantization.md
Outdated
| * **For Using Existing GGUF Models:** Use GGUF loading (`from_single_file`). | ||
| * **Curious about training with quantization?** Stay tuned for a follow-up blog post on that topic! | ||
|
|
||
| Quantization significantly lowers the barrier to entry for using large diffusion models. Experiment with these backends to find the best balance of memory, speed, and quality for your needs. |
There was a problem hiding this comment.
Let's make sure acknowledge @ChunTeLee for providing a nice thumbnail.
diffusers-quantization.md
Outdated
| Before diving into the quantization backends, let's introduce the FluxPipeline and its components, which we'll be quantizing: | ||
|
|
||
| * **Text Encoders (CLIP and T5):** | ||
| * **Function:** Process input text prompts. FLUX-dev uses CLIP for initial understanding and a larger T5 for nuanced comprehension and better text rendering. | ||
| * **Memory:** T5 - 9.52 GB; CLIP - 246 MB (in BF16) | ||
| * **Transformer (Main Model - MMDiT):** | ||
| * **Function:** Core generative part (Multimodal Diffusion Transformer). Generates images in latent space from text embeddings. | ||
| * **Memory:** 23.8 GB (in BF16) | ||
| * **Variational Auto-Encoder (VAE):** | ||
| * **Function:** Translates images between pixel and latent space. Decodes generated latent representation to a pixel-based image. | ||
| * **Memory:** 168 MB (in BF16) | ||
| * **Focus of Quantization:** Examples will primarily focus on the `transformer` and `text_encoder_2` (T5) for the most substantial memory savings. | ||
|
|
There was a problem hiding this comment.
Nice ! Can you add a link to https://huggingface.co/black-forest-labs/FLUX.1-dev. Also you can precise the memory needed to load the model
diffusers-quantization.md
Outdated
| from diffusers import AutoModel, FluxPipeline | ||
| from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig | ||
| from diffusers.quantizers import PipelineQuantizationConfig | ||
| from transformers import T5EncoderModel |
There was a problem hiding this comment.
clean some of the import since we don't use them, same for the other snippets
Preparing the Article
mdfile. You can also specifyguestororgfor the authors.