Why is my latency getting worse over time?

[besrym]

Hello there,

I am evaluating different LLM Inference Serving Backends on an NVIDIA A40.

I start the llama-server with:

./build/bin/llama-server \
      --model /srv/models/F16/Meta-Llama-3-8B-Instruct.f16.gguf \
      --host 0.0.0.0 \
      --port 8000 \
      --metrics \
      --threads 48 \
      --flash-attn \
      --n-gpu-layers 99 \
      --ctx-size 212992 \
      --parallel 26

What I now noticed: In my tests without concurrency (one request at a time), the Latency of my requests rises over time:

End-to-End Request Latency (s) :
9.257249126210809,
9.676895394921303,
10.249773621093482,
10.394722724333405,
10.799425629898906,
11.164135189261287,
11.47115689329803,
12.026003962848336,
12.227484032977372,
12.712236621417105,
13.03553528804332,
13.4061823268421,
14.022667570970953,
14.032732385676354,
14.548057908657938,
14.751412217039615,
15.324861367233098,
15.491500550881028,
16.062391933053732,
16.225378068629652,
16.745809122920036,
17.18366448674351,
14.789957445580512,
17.591574397869408,
18.021023420151323,
18.411880071740597,
18.39220358012244,
18.796500200871378,
18.596912099979818,
18.9200197160244,
18.730387301184237,
...

Then I came across the payload option "cache_prompt", which is true by default. I set it to False in my benchmark client, but that didn't help. It seems like the system keeps caching the prompts:

The first requests are in the expected latency range (9- 10s). Any ideas why the latency increases?

Please let me know if there is any important information you would like me to provide! Thank you in advance.

[besrym]

As I am working on one request after the other, I have adapted the deployment. With --ctx-size 8192 and --parallel 1, I get stable 9-10 sec Latencies, nevertheless, I would be interested in where the problem with my initial deployment comes from.

[ExtReMLapin]

Cause : #10860

[besrym]

thank you!