Performance of llama.cpp with Vulkan #10879
Replies: 101 comments 155 replies
-
|
AMD FirePro W8100
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
AMD RX 470
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
ubuntu 24.04, vulkan and cuda installed from official APT packages.
build: 4da69d1 (4351) vs CUDA on the same build/setup
build: 4da69d1 (4351) |
Beta Was this translation helpful? Give feedback.
-
|
Macbook Air M2 on Asahi Linux ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
For the record I think this is slow on the HoneyKrisp side rather than llama.cpp |
Beta Was this translation helpful? Give feedback.
-
|
Can you share how you got vulkan to build on Asahi? I can't seem to get cmake to notice it. |
Beta Was this translation helpful? Give feedback.
-
|
Spoke too soon, got it working! |
Beta Was this translation helpful? Give feedback.
-
|
Gentoo Linux on ROG Ally (2023) Ryzen Z1 Extreme ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: Found 4 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
Cool setup! Could you also post the result of 1, 2 and 3 7900 XTX GPUs? You can use only the first GPU with |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
For this multi GPU case getting Vulkan to support #6017 pipeline parallelism might help improve the prompt processing speed. |
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve I updated the commit id in all my result. |
Beta Was this translation helpful? Give feedback.
-
|
build: 0d52a69 (4439) NVIDIA GeForce RTX 3090 (NVIDIA)
AMD Radeon RX 6800 XT (RADV NAVI21) (radv)
AMD Radeon (TM) Pro VII (RADV VEGA20) (radv)
Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver)
|
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve Some of the tg results here are a little low, I think they might be debug builds. The cmake step (at least on Linux) might require |
Beta Was this translation helpful? Give feedback.
-
|
I've added My release numbers for the RX 470 are basically identical to the ones I posted earlier without the flag.
|
Beta Was this translation helpful? Give feedback.
-
|
Maybe not in your case, but some other results are suspiciously low in tg (for example the RTX 3080) |
Beta Was this translation helpful? Give feedback.
-
|
Build: 8d59d91 (4450)
Lack of proper Xe coopmat support in the ANV driver is a setback honestly.
edit: retested both with the default batch size. |
Beta Was this translation helpful? Give feedback.
-
I put my A770 into a Windows PC and gave Intel GPA and vtune a shot: GPA just crashes most of the time, I couldn't get it to trace anything useful. vtune works, but does not support Vulkan. It just shows some high-level metrics in that case, not really useful sadly. |
Beta Was this translation helpful? Give feedback.
-
I did build it with cmake with build type Release. |
Beta Was this translation helpful? Give feedback.
-
|
In that case it's something else, cause it should be performing similarly to my A770. I suspect the mesa version, there was something in newer mesa versions that slowed down tg on Intel. |
Beta Was this translation helpful? Give feedback.
-
|
A750 has 448 CUs, A770 has 512 CUs I think. Personally, I am not worried about tg. I am worried about pp here. The gemm batch quickly saturates my GPU. |
Beta Was this translation helpful? Give feedback.
-
|
@0cc4m https://gitlab.freedesktop.org/mesa/mesa/-/issues/12585 |
Beta Was this translation helpful? Give feedback.
-
|
Here's something exotic: An AMD FirePro S10000 dual GPU from 2012 with 2x 3GB GDDR5. build: 914a82d (4452)
|
Beta Was this translation helpful? Give feedback.
-
|
Very interesting, and looks like it's pretty close to the W8100 in tg despite being a dual GPU card. Your backend scales pretty well with layer splitting which is why I find it worthwhile to run my RX470 and W8100 together (I end up getting results that are close to the average of both cards).
|
Beta Was this translation helpful? Give feedback.
-
|
Latest arch with For the sake of consistency I run every bit in a script and also build every target from scratch (for some reason kill -STOP -1
timeout 240s $COMMAND
kill -CONT -1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none
build: ff3fcab (4459)
This bit seems to underutilise both GPU and CPU in real conditions based on
|
Beta Was this translation helpful? Give feedback.
-
|
Unless you reduce the number of GPU layers, threads and openblas/non-openblas is not gonna make any difference. Try it with ngl 0, then only prompt processing is accelerated using Vulkan, the rest runs on CPU. This is often a good setting for integrated GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
That's something I didn't think about, with
build: ba8a1f9 (4460)
|
Beta Was this translation helpful? Give feedback.
-
|
It seems latest patches has improved the results a bit: ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
A few months later and I get:
build: f3a4b16 (5568) I run it on Linux (Arch with lama.cpp-vulkan-git package compiled by GCC 15). From my tests, only Vulkan backend ( I'm curious why I cannot go over 6 t/s. Is this an issue with the newer llama.cpp version or with my OS configuration? |
Beta Was this translation helpful? Give feedback.
-
|
Intel ARC A770 on Windows:
build: ba8a1f9 (4460) |
Beta Was this translation helpful? Give feedback.
-
Single GPU VulkanRadeon Instinct MI25 ggml_vulkan: 0 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Radeon PRO VII ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Multi GPU Vulkanggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Single GPU RocmDevice 0: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no
build: 2739a71 (4461) Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Multi GPU RocmDevice 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Layer split
build: 2739a71 (4461) Row split
build: 2739a71 (4461) Single GPU speed is decent, but multi GPU trails Rocm by a wide margin, especially with large models due to the lack of row split. |
Beta Was this translation helpful? Give feedback.
-
|
What is the power profile for this MI25? Mine is 110W but its running slower than yours on git from today. |
Beta Was this translation helpful? Give feedback.
-
|
Mine defaults to 220w. |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX 5700 XT on Arch using mesa-git and setting a higher GPU power limit compared to the stock card.
I also think it could be interesting adding the flash attention results to the scoreboard (even if the support for it still isn't as mature as CUDA's).
|
Beta Was this translation helpful? Give feedback.
-
|
There is no Vulkan flash attention support (except with coopmat2 on very new nvidia drivers). What you're measuring here is a CPU fallback. |
Beta Was this translation helpful? Give feedback.
-
|
I see, I was sure about the CPU fallback but didn't know there was no flash attention support at all. |
Beta Was this translation helpful? Give feedback.
-
|
I tried but there's nothing after 1 hrs , ok, might be 40 mins... Anyway I run the llama_cli for a sample eval...
Meanwhile OpenBLAS |
Beta Was this translation helpful? Give feedback.
-
|
Even at below 1t/s |
Beta Was this translation helpful? Give feedback.
-
|
Truth is ... (0.79 tokens per second), 3788.63 ms / 3 tokens So it's not even...it just slower... |
Beta Was this translation helpful? Give feedback.
-
|
Lunar Lake 258V with 140V iGPU and Coopmat (Windows 11) PS C:\Users\julia\Downloads\llama-b5598-bin-win-vulkan-x64> .\llama-bench.exe -m ..\llama-2-7b.Q4_0.gguf -ngl 99
build: 669c13e (5598) |
Beta Was this translation helpful? Give feedback.
-
|
OS: Arch Linux x86_64 ggml_vulkan: Found 1 Vulkan devices:
build: 228f34c (5604) OS: Arch Linux x86_64 ggml_vulkan: Found 1 Vulkan devices:
build: 228f34c (5604) |
Beta Was this translation helpful? Give feedback.
-
|
OS: Arch Linux x86_64 vulkaninfo:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: 0 = Intel(R) Graphics (Intel Corporation) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
build: 2e89f76 (5640) |
Beta Was this translation helpful? Give feedback.
-
|
Lenovo ThinkPad t14s wsl2 ubuntu 24.04
build: ed52f36 (5648) windows ggml_vulkan: 0 = AMD Radeon(TM) Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 0 | matrix cores: none
build: ed52f36 (5648) windows ggml_vulkan: 0 = AMD Radeon(TM) Graphics (AMD proprietary driver) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 0 | matrix cores: none
build: ed52f36 (5648) |
Beta Was this translation helpful? Give feedback.
-
|
FYI the WSL run is actually using the CPU as it can't find a Vulkan device. Maybe WSL doesn't support Vulkan 😮? |
Beta Was this translation helpful? Give feedback.
-
|
395+ on arch
b7cc774 (5656) |
Beta Was this translation helpful? Give feedback.
-
|
As far as I know, NPUs are mostly for low-power AI inference, for example for background AI tasks. They're pretty new, so support isn't good yet. ROCm and CUDA are focused on high-power GPU compute, so it's just a different purpose. |
Beta Was this translation helpful? Give feedback.
-
|
Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32
here is the rocm version, 1/4 the speed in pp, and tg in margin of error. this is with ROCBLAS_USE_HIPLASLT=1 |
Beta Was this translation helpful? Give feedback.
-
This is really interesting! It is consistent with the rumors that ROCm support for Strix Halo is "meh". Maybe it'll get better in subsequent ROCm releases. |
Beta Was this translation helpful? Give feedback.
-
might be interesting to just put the embeddings model on the npu while the llm is on the apu. |
Beta Was this translation helpful? Give feedback.
-
|
Any more details about your setup? Is this a laptop or mini-pc? What is the linux kernel (uname -r)? |
Beta Was this translation helpful? Give feedback.
-
|
Operating System: Microsoft Windows 11 24H2 (Build 26100.4061) AMD Radeon RX 6600 With
build: ed52f36 (5648) AMD Radeon RX 9060 XT (16 GB) With
Without
build: ed52f36 (5648) |
Beta Was this translation helpful? Give feedback.
-
|
RTX 3060 (asus phoenix single fan) ggml_vulkan: Found 1 Vulkan devices:
build: 860a9e4 (5688) better than existing result but still lower than cuda ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
build: 860a9e4 (5688) for some reason, only 1 GPU was detected using vulkan. running vulkaninfo displayed 2 gpu but each with different vulkan api version
|
Beta Was this translation helpful? Give feedback.
-
|
It only uses the RTX 3060 because by default we don't use iGPUs unless no dedicated GPU is available. If you wanna give it a shot, you can override this by setting |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! I haven't tried the options but i just start the machine without the eGPU, and tried running with iGPU. will try the options next time i put the eGPU again. CPU: AMD Ryzen 5 PRO 3400G ggml_vulkan: Found 1 Vulkan devices:
build: 860a9e4 (5688) |
Beta Was this translation helpful? Give feedback.
-
|
after ram change from 2666 to 3200 (2933MT/s)
build: 860a9e4 (5688) |
Beta Was this translation helpful? Give feedback.
-
|
A dump of most of my AI-capable hardware All OS used is Arch Linux x86_64 LCPP build: 860a9e4 (5688) NVIDIA GeForce RTX 3090 (PCIe 4x16) ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1 -mg 1 -sm none -r 30
build: 860a9e4 (5688) NVIDIA GeForce GTX 980 ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1
build: 860a9e4 (5688) NVIDIA GeForce GTX 1650 Mobile ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 30 -fa 0,1 -sm none -mg 1
build: 860a9e4 (5688) NVIDIA GeForce MX 330 ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 15 -fa 0,1 -mg 0 -sm none
build: 860a9e4 (5688) AMD Radeon Pro W5500 (PCIe 3x2) ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1 -mg 0 -sm none -r 30
build: 860a9e4 (5688) AMD Radeon Pro WX 4100 ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1
build: 860a9e4 (5688) AMD Radeon HD 7790 ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 7 -fa 0,1
build: 860a9e4 (5688) AMD Radeon HD 7750 ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 7 -fa 0,1
build: 860a9e4 (5688) AMD Ryzen 7 5800HS (DDR4 3200 @ 2ch) ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1 -sm none -mg 0
build: 860a9e4 (5688) Intel Core i7-1165G7 (DDR4 2666 @ 1ch) ./llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 -fa 0,1 -mg 1 -sm none
build: 860a9e4 (5688) Intel Core i5-6300U ./llama-bench -m ../../llama-2-7b.Q4_0.gguf -ngl 31 -fa 0,1
build: 860a9e4 (5688) |
Beta Was this translation helpful? Give feedback.
-
|
That's a lot of GPUs 😁. It looks like something went wrong with the FA W5500 run? |
Beta Was this translation helpful? Give feedback.
-
|
For i7-1185G7 with DDR4 3200 @ 2ch I got the same numbers for prompt processing and 2x more for text generations. I also get similar results on newer generation of i7 with 2ch DDR4. The Vulkan backend seems to mostly impact the pp. There is a little difference between linux and windows builds of current versions of llama.cpp. @netrunnereve The takeaway for processors with Intel Iris Xe is:
|
Beta Was this translation helpful? Give feedback.
-
|
Update for Radeon RX 9070 XT (16GB) w/ Ryzen 9 5900X:
build: fa4a9f2 (5740) |
Beta Was this translation helpful? Give feedback.
-
|
i would like to add mining gpu (Nvidia P106-100, Pascal arch) to the benchmark since it can run llama-bench with cuda with around 410-420 t/s. unfortunately, it is not detected by vulkan. In windows it is listed as Display adapters but not as GPU in task managers. I looked into nvidia's vulkan driver and it seems that it is not listed as supported gpu. |
Beta Was this translation helpful? Give feedback.
-
|
NVIDIA GeForce GTX 1070 Ti
build: 860a9e4 (5688) on cuda it can reach over 700 t/s
build: 860a9e4 (5688) |
Beta Was this translation helpful? Give feedback.
-
|
Again in a RX 6600 And comparing with my previous bench in march, the pp512 test improved like 60% The previous pp512 380.87 ± 0.21
build: 72babea (5767) |
Beta Was this translation helpful? Give feedback.
-
|
Okay gonna try submitting my first benchmarks. My understanding is that the performance might vary a lot depending on the exact kernel drivers as well as user-space implementations and versions of the Linux AMD ecosystem so will try to capture those details as well including how to capture the info for others to reproduce if so desired. (Is there a script to do all this already or something as this is rather ad-hoc and I'm not sure it accurately describes all the moving parts in play sufficiently to recreate such an environment? Haha the AMD GPU stack is clear as mud to me...) Hardware Details
Driver Details
BenchmarksVulkan
ROCm/HIP$ export HIPCXX="$(hipconfig -l)/clang"
$ export HIP_PATH="$(hipconfig -R)"
$ cmake -B build -DGGML_VULKAN=0 -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DGGML_HIP_ROCWMMA_FATTN=ON -DCMAKE_BUILD_TYPE=Release
$ cmake --build build --config Release -j $(nproc)
# run as sudo because permissions are borked despite adding user to group etc...
$ sudo ./build/bin/llama-bench \
-m /models/TheBloke/Llama-2-7B-GGUF/llama-2-7b.Q4_0.gguf \
-ngl 100 \
-fa 0,1 \
-t 1
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
sweep-benchTo get a better understanding of the performance across the kv-cache size curve I've run the same test model both for Vulkan and ROCm/HIP backends as compiled above. I maintain a branch ported from ik_llama.cpp of llama-sweep-bench for mainline llama.cpp. This tools gives the full view across a wider context length rather than sampling a single point along the curve (e.g. PP512 TG128). Also useful for checking various ./build/bin/llama-sweep-bench \
-m /models/TheBloke/Llama-2-7B-GGUF/llama-2-7b.Q4_0.gguf \
-fa \
-c 18432 \
-ngl 99 \
--threads 1
The winner to me of this specific benchmark roundup would be ROCm/HIP with flash attention enabled despite vulkan FA=1 having slight advantage in token generation. At longer contexts FA=1 is a must for both vulkan and rocm/hip otherwise performance falls off rapidly despite similar TG128 scores. For baseline I've added in my home rig 3090TI FE 24GB VRAM running at default 450W cap. While it gets much warmer, you can see the performance disparity. I've not tried vulkan backend with NVIDIA GPU, but some recent nvidia slides for some models vulkan may be faster than native CUDA interestingly.
Thanks much for this informative thread and all the help figuring out the best combination of kernel/userspace drivers and api/backend/compiler libraries to get the best performance out of this GPU on llama.cpp. Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the great visualizations! The pp behaviour with flash attention on Vulkan seems unexpected, maybe there's some optimization we can do for that (FYI @jeffbolznv). The only thing that would make it even better are the RTX 3090 Vulkan results. |
Beta Was this translation helpful? Give feedback.
-
|
I made a reasonable attempt to optimize the KHR_coopmat FA path on my RTX 4070, and IIRC it was slightly better than break-even. I think if somebody tuned it on an AMD card they might be able to squeeze out a bit more. |
Beta Was this translation helpful? Give feedback.
-
|
I don't currently have the required hardware either, but maybe in the future. |
Beta Was this translation helpful? Give feedback.
-
|
@0cc4m had to suffer through some updates but got Vulkan 1.4.303 installed on my local ARCH rig with latest nvidia drivers and CUDA 12.9. Here is a look at just the 3090TI FE 24GB VRAM running uncap'd at 450 Watts with both CUDA and Vulkan backend as well as ik_llama.cpp fork's CUDA implementation.
The Vulkan FA=1 implementation with I did however give a newer slightly larger dense model a quick try too with a "pure" Qwen3-14B-Q4_0 and interestingly here Vulkan FA=1 takes the lead over native CUDA implementations with sufficient n_kv depth! (I didn't publish this model on hf as i only made it for internal benchmarking purposes).
Very interesting! Now I'm curious about a bunch of other models hah.... |
Beta Was this translation helpful? Give feedback.
-
|
Curious that with DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M, DeepSeek-R1-Distill-Qwen-7B-Q4_K_M or Devstral-Small-2505-Q4_K_M i see nothing but 5% degradation in speed when running RADV vs AMDVLK in prompt processing/text generation but with llama-2-7b.Q4_0 prompt processing is twice as fast with AMDVLK
build: 27208bf (5774) |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Apple Silicon benchmark thread, but for Vulkan! Many improvements have been made to the Vulkan backend and I think it's good to consolidate and discuss our results here.
We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our Vulkan releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and Vulkan info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same device newer commits with substantial Vulkan updates are prioritized, alternatively the one with the highest tg128 score will be used. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!
Vulkan Scoreboard for Llama 2 7B, Q4_0 (no FA)
Vulkan Scoreboard for Llama 2 7B, Q4_0 (with FA)
Beta Was this translation helpful? Give feedback.
All reactions