Endpoints created with custom image using create_inference_endpoint from huggingface_hub returns inconsistent and hallucinatory outputs

[Sanahm]

Issue Description

Context:
I am using the Hugging Face Inference API to deploy models on a leaderboard using the huggingface_hub library with a custom image for text generation. Everything was working fine until recently. Now, I am experiencing issues where the model either hallucinates or the inference API returns a sequence of random words in the "generated_text" field.

Problem Description

1. Hallucination or Random Words:

   • When using a custom image, the model returns hallucinatory or random text.
   • Without the custom image, the output is more reasonable but still inconsistent.

2. Inconsistency Between requests and InferenceClient:

   • There is an inconsistency in the output when using the requests library and the InferenceClient.

Code to Reproduce the Error

Setting Up the Endpoint

from huggingface_hub import create_inference_endpoint

endpoint = create_inference_endpoint(
    "test-qwenz",
    repository="Qwen/Qwen2.5-0.5B-Instruct",
    framework="pytorch",
    namespace="<xxxxxxxx>",
    task="text-generation",
    accelerator="gpu",
    vendor="aws",
    region="us-east-1",
    type="protected",
    instance_size="x1",
    instance_type="nvidia-a10g",
    custom_image={
        "health_route": "/health",
        "env": {
            "MAX_BATCH_PREFILL_TOKENS": "2048",
            "MAX_INPUT_LENGTH": "1024",
            "MAX_TOTAL_TOKENS": "1512",
            "MODEL_ID": "/repository"
        },
        "url": "ghcr.io/huggingface/text-generation-inference:latest",
    },
)

Test Code Using requests

import requests

# Define the input data
data = {
    "inputs": "What is 4*2 ?"
}

# Define the headers, including the Authorization token
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

# Send the POST request
response = requests.post(endpoint_url, json=data, headers=headers)

# Print the response
print(response.json())

Output with Custom Image

[
    {
        'generated_text': 'What is 4*2 ? The integral is not 0 for ν ≤ 3/2. Any such τ will allow us to express (7b) in a form of a quadratic function of the twists τ x : Z z → Z y , which turns out to be a constant ...'
    }
]

Output without Custom Image

[
    {
        'generated_text': 'What is 4*2 ? To solve the problem of multiplying two numbers, you simply multiply them together. In this case, we...'
    }
]

Inconsistency Between requests and InferenceClient

from huggingface_hub import InferenceClient

client = InferenceClient(endpoint_url, token="<token>")

output = client.text_generation(
    prompt="What is 4*2 ?", details=True
)

print(output)

Output:

' - Brainly.com\\nprofile\\njessie1078\\njessie10'

Additional Information

• Models Tested:

  • Phi4
  • Qwen7B-Instruct

• Custom Image URLs Tested:

  • ghcr.io/huggingface/text-generation-inference:1.1.1
  • ghcr.io/huggingface/text-generation-inference:3.3.1
  • ghcr.io/huggingface/text-generation-inference:latest

Expected Behavior

• The model should return a consistent and accurate response to the input prompt.
• There should be no significant difference in the output when using requests and InferenceClient.

Environment

• Python Version: 3.10.16
• huggingface_hub Version: 0.27.1 (tested also on the 0.33.1)
• requests Version: 2.32.3

Steps to Reproduce

1. Set up the inference endpoint with the provided code.
2. Use the requests library to send a POST request to the endpoint.
3. Use the InferenceClient to send a request to the endpoint.
4. Observe the inconsistent and hallucinatory outputs.

Additional Notes

• The issue persists across different models and custom image versions (Qwen-1.5B-Instruct, Qwen-7B-Instruct, Phi4).
• The problem started recently, suggesting a potential change in the inference API or custom image configuration.

Request

Could you please investigate this issue and provide a solution or workaround to ensure consistent and accurate model outputs?

Thank you for your attention and support.

System info

- **Python Version:** 3.10.16
- **huggingface_hub Version:** 0.27.1 (tested also on the 0.33.1)
- **requests Version:** 2.32.3

[hanouticelina]

hi @Sanahm,

Inconsistency Between requests and InferenceClient:
There is an inconsistency in the output when using the requests library and the InferenceClient.

I wasn't able to reproduce a consistent discrepancy between requests and InferenceClient. In both cases, the outputs vary between runs, sometimes I get hallucinations, and other times the answers are a bit more reasonable. If you haven't already, I wouldd recommend setting a low temperature in both requests and InferenceClient snippets. This should make the outputs easier to compare.

That said, if you use the model as a chat completion model, the responses tend to be more coherent and accurate. So depending on the use case, switching to the chat completion API might give you better results:

import os
import requests

# Define the input payload, setting temperature low makes outputs easier to compare with InferenceClient requests.
data = {"messages": [{"role": "user", "content": "What is 4*2 ?"}], "temperature": 0.2}

# Headers with auth token
headers = {
    "Authorization": f"Bearer {os.environ['HF_TOKEN']}",
    "Content-Type": "application/json",
}

response = requests.post(IE_URL, json=data, headers=headers)
print(response.json()["choices"][0]["message"]["content"])

This returns:

4 * 2 equals 8.

And here's the equivalent using huggingface_hub.InferenceClient:

from huggingface_hub import InferenceClient

client = InferenceClient(API_URL)

response = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "What is 4*2 ?"}],
    temperature=0.2,  # setting this low makes outputs easier to compare with raw `requests`
)

print(response.choices[0].message.content)

Which also returns:

4 * 2 equals 8.

Hallucination or Random Words:

When using a custom image, the model returns hallucinatory or random text.
Without the custom image, the output is more reasonable but still inconsistent.

I wasn't able either to consistently reproduce a discrepancy between TGI images (tested 3.3.1 and latest (which is 3.3.4?))). but pinging @ErikKaum here if you have more insights regarding this.

[ErikKaum]

No revolutionary insights from my end but a few sanity checks so we know what we're testing:

• remember to set the seed when calling the model (both for requests and InferenceClient), that way you know that you're really testing the client and not getting variance in output due to the model
• I'd recommend not using the TGI latest tag. Generally the latesttag in docker containers doesn't literally have to point to the latest version of a container. It's just a string. And I think TGI doesn't always create a latest based what's really the latest version. So always use versioned containers 👍

[Sanahm]

hi @hanouticelina @ErikKaum

Thank you for your responses. In my use case, I deploy local models for evaluation against a set of benchmarks, similar to a leaderboard. The primary issue I'm encountering is that the client.text_generation method is no longer working for models created using create_inference_endpoint from the huggingface_hub.

I have tested this with different container versions: 1.1.1, 3.3.1 (which I was previously using), and the latest version, 3.3.4. None of these versions have been successful.

Is it possible that recent changes have been introduced to Hugging Face inference endpoints that could be causing this issue? I suspect that the submission of create_inference_endpoint is now handled differently at the HF inference server side.

[hanouticelina]

hi @Sanahm,

Is it possible that recent changes have been introduced to Hugging Face inference endpoints that could be causing this issue? I suspect that the submission of create_inference_endpoint is now handled differently at the HF inference server side.

As far as the client-side is concerned, there have been no recent changes that would explain the shift in behavior for endpoints created via create_inference_endpoint, or how InferenceClient.text_generation interacts with them.

However, I opened this PR to remove some unnecessary default values in the client, This ensures that:

from huggingface_hub import InferenceClient

API_URL = ...
client = InferenceClient(API_URL, token=os.environ["HF_TOKEN"])
response = client.text_generation("What is 4*2 ?")
print(response)

will now be strictly equivalent to:

import requests

API_URL = ...
data = {"inputs": "What is 4*2 ?"}
headers = {
    "Authorization": f"Bearer {os.environ['HF_TOKEN']}",
    "Content-Type": "application/json",
}

response = requests.post(
    API_URL,
    json=data,
    headers=headers,
)
print(response.json())

That said, if your use case allows for it, we strongly recommend switching to the chat completion API (client.chat.completions.create(...)) as mentioned in my previous message. It provides more stable and coherent outputs.

Finally, if you're still seeing some behavior discrepancy between TGI container versions, I'd recommend opening an issue directly on the TGI repo.