Why llama-server is so limited to concurrent requests even using -cb and -np ?

[celsowm]

Hi
Why llama-server is so limited to concurrent requests even using -cb and -np ?
Sglang and vllm servers are not so limited like llama.cpp when you have even hundreds of requests

o3 response:

vLLM and SGLang were designed from the ground-up for token-level scheduling and smart KV-cache management, so one copy of the model can keep a GPU busy while dozens of users stream tokens concurrently.
llama-cpp’s server, by contrast, simply round-robins request-level contexts that each own a large, contiguous cache, so every extra user slices throughput and memory almost linearly.

[besrym]

Hi, this is mainly due to memory management (kv cache). vLLM uses paged attention, which reduces the wasted portion of the kv cache to as close to zero.

At the moment llama.cpp does not yet use this strategy, and I cannot tell you whether this is planned for the future. (See: 1955)

[ExtReMLapin]

Also see #10860