Add a no-gpu CLI arg to llama-cli

[WilliamTambellini]

Add a no-gpu CLI arg to llama-cli.
Disable using the gpu (at least cuda), load and, run only using cpu resources.
Similar to whisper-cpp 'no-gpu':
./whisper-cpp/main --help
...
-ng, --no-gpu [false ] disable GPU
ref
ggml-org/whisper.cpp#1762

[WilliamTambellini]

@ggerganov would you mind if I add such option to llamacli?

[Allan-Luu]

I know there currently isn't an option to disable GPU on llama-cli at the moment, but if you build without any backend then it'll default to running on CPU resources only, without any option to offload to GPU. This could be a workaround for now.

The first section this link gives a tutorial on how to do this. Hope that helps!

[ggerganov]

You can also set CUDA_VISIBLE_DEVICES to empty and it won't use any CUDA GPUs. I believe @slaren has plans for improving the device selection in llama.cpp, so it's better to not make changes to this logic before that to avoid conflicts.

[slaren]

I don't plan to work on this on the immediate future. If you are willing to work on this, adding a list of GPU devices to use to llama_model_params should be a straightforward change and likely to remain in the future. Don't just add a no-gpu parameter however, add the ability to select the list of devices to use with a model. Possibly remove rpc_servers from llama_model_params as well, and instead have the caller add the RPC servers to use to the list of devices.

[WilliamTambellini]

Tks all.
CUDA_VISIBLE_DEVICES="" indeed does the job although totally prevents the process to use any cuda devices later on.
@slaren solution is indeed the most sustainable one. Not high prio for us anymore though.
Tks.

[sonnybox]

Tks all. CUDA_VISIBLE_DEVICES="" indeed does the job although totally prevents the process to use any cuda devices later on. @slaren solution is indeed the most sustainable one. Not high prio for us anymore though. Tks.

This actually helps a lot. I was trying to load a multimodal model and it kept loading the mmproj on the GPU when I wanted CPU only inference (I set -ngl 0). Setting this flag made it all run on CPU.