[Bug]: batch_completion not working, sometimes? #13139
Description
I get all kinds of different errors for some of the requests when perfoming batch completion.
Some of the requests are successful, sometimes all of them sometimes none. Errors vary, but I seem to mostly see
File ".../python3.12/site-packages/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py", line 502, in convert_to_model_response_object
assert response_object["choices"] is not None and isinstance(
~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'choices'
This code should reproduce, sometimes at least 🙃
import litellm
countries = ["France", "Germany", "Italy", "Spain", "United Kingdom"]
batch_messages = []
for country in countries:
messages = [
{"role": "system", "content": "Answer the following question:"},
{"role": "user", "content": f"What is the capital of {country}"},
]
batch_messages.append(messages)
batch_responses = litellm.batch_completion(
model="azure/my-model-deployment-name",
messages=batch_messages,
)I was running on litellm version 1.72.0, tried to upgrade to 1.74.9 but the error persists.