Skip to content

Commit 3c7b980

Browse files
chensuyuepre-commit-ci[bot]
authored and
alexsin368
committed
Update LLM model workaround for the vllm issues (opea-project#2182)
Signed-off-by: chensuyue <suyue.chen@intel.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: alexsin368 <alex.sin@intel.com>
1 parent 5c06cd6 commit 3c7b980

File tree

10 files changed

+17
-18
lines changed

10 files changed

+17
-18
lines changed

‎CodeTrans/README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,12 @@ The table below lists currently available deployment options. They outline in de
3434

3535
## Validated Configurations
3636

37-
| **Deploy Method** | **LLM Engine** | **LLM Model** | **Hardware** |
38-
| ----------------- | -------------- | ---------------------------------- | ------------ |
39-
| Docker Compose | vLLM, TGI | mistralai/Mistral-7B-Instruct-v0.3 | Intel Gaudi |
40-
| Docker Compose | vLLM, TGI | mistralai/Mistral-7B-Instruct-v0.3 | Intel Xeon |
41-
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD EPYC |
42-
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |
43-
| Helm Charts | vLLM, TGI | mistralai/Mistral-7B-Instruct-v0.3 | Intel Gaudi |
44-
| Helm Charts | vLLM, TGI | mistralai/Mistral-7B-Instruct-v0.3 | Intel Xeon |
45-
| Helm Charts | vLLM, TGI | mistralai/Mistral-7B-Instruct-v0.3 | AMD ROCm |
37+
| **Deploy Method** | **LLM Engine** | **LLM Model** | **Hardware** |
38+
| ----------------- | -------------- | ------------------------------ | ------------ |
39+
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi |
40+
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon |
41+
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD EPYC |
42+
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |
43+
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi |
44+
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon |
45+
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |

‎CodeTrans/code_translation.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
# Copyright (C) 2024 Intel Corporation
22
# SPDX-License-Identifier: Apache-2.0
33

4-
import asyncio
54
import os
65

76
from comps import MegaServiceEndpoint, MicroService, ServiceOrchestrator, ServiceRoleType, ServiceType

‎CodeTrans/docker_compose/amd/cpu/epyc/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,7 @@ Key parameters are configured via environment variables set before running `dock
158158
| :-------------------------------------- | :-------------------------------------------------------------------------------------------------------------------- | :------------------------------------ |
159159
| `HOST_IP` | External IP address of the host machine. **Required.** | `your_external_ip_address` |
160160
| `HF_TOKEN` | Your Hugging Face Hub token for model access. **Required.** | `your_huggingface_token` |
161-
| `LLM_MODEL_ID` | Hugging Face model ID for the CodeTrans LLM (used by TGI/vLLM service). Configured within `compose.yaml` environment. | `mistralai/Mistral-7B-Instruct-v0.3` |
161+
| `LLM_MODEL_ID` | Hugging Face model ID for the CodeTrans LLM (used by TGI/vLLM service). Configured within `compose.yaml` environment. | `Qwen/Qwen2.5-Coder-7B-Instruct` |
162162
| `LLM_ENDPOINT` | Internal URL for the LLM serving endpoint (used by `codetrans-epyc-llm-server`). Configured in `compose.yaml`. | `http://${HOST_IP}:8008` |
163163
| `LLM_COMPONENT_NAME` | LLM component name for the LLM Microservice. | `OpeaTextGenService` |
164164
| `BACKEND_SERVICE_ENDPOINT` | External URL for the CodeTrans Gateway (MegaService). Derived from `HOST_IP` and port `7778`. | `http://${HOST_IP}:7777/v1/codetrans` |

‎CodeTrans/docker_compose/amd/cpu/epyc/set_env.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
host_ip=$(hostname -I | awk '{print $1}')
88
export host_ip
99

10-
export LLM_MODEL_ID="mistralai/Mistral-7B-Instruct-v0.3"
10+
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
1111
export LLM_ENDPOINT="http://${host_ip}:8008"
1212
export LLM_COMPONENT_NAME="OpeaTextGenService"
1313
export MODEL_CACHE=${model_cache:-"./data"}

‎CodeTrans/docker_compose/intel/cpu/xeon/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ Key parameters are configured via environment variables set before running `dock
125125
| :-------------------------------------- | :-------------------------------------------------------------------------------------------------------------------- | :------------------------------------ |
126126
| `HOST_IP` | External IP address of the host machine. **Required.** | `your_external_ip_address` |
127127
| `HF_TOKEN` | Your Hugging Face Hub token for model access. **Required.** | `your_huggingface_token` |
128-
| `LLM_MODEL_ID` | Hugging Face model ID for the CodeTrans LLM (used by TGI/vLLM service). Configured within `compose.yaml` environment. | `mistralai/Mistral-7B-Instruct-v0.3` |
128+
| `LLM_MODEL_ID` | Hugging Face model ID for the CodeTrans LLM (used by TGI/vLLM service). Configured within `compose.yaml` environment. | `Qwen/Qwen2.5-Coder-7B-Instruct` |
129129
| `LLM_ENDPOINT` | Internal URL for the LLM serving endpoint (used by `codetrans-xeon-llm-server`). Configured in `compose.yaml`. | `http://${HOST_IP}:8008` |
130130
| `LLM_COMPONENT_NAME` | LLM component name for the LLM Microservice. | `OpeaTextGenService` |
131131
| `BACKEND_SERVICE_ENDPOINT` | External URL for the CodeTrans Gateway (MegaService). Derived from `HOST_IP` and port `7778`. | `http://${HOST_IP}:7777/v1/codetrans` |

‎CodeTrans/docker_compose/intel/hpu/gaudi/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ Key parameters are configured via environment variables set before running `dock
125125
| :-------------------------------------- | :-------------------------------------------------------------------------------------------------------------------- | :------------------------------------ |
126126
| `HOST_IP` | External IP address of the host machine. **Required.** | `your_external_ip_address` |
127127
| `HF_TOKEN` | Your Hugging Face Hub token for model access. **Required.** | `your_huggingface_token` |
128-
| `LLM_MODEL_ID` | Hugging Face model ID for the CodeTrans LLM (used by TGI/vLLM service). Configured within `compose.yaml` environment. | `mistralai/Mistral-7B-Instruct-v0.3` |
128+
| `LLM_MODEL_ID` | Hugging Face model ID for the CodeTrans LLM (used by TGI/vLLM service). Configured within `compose.yaml` environment. | `Qwen/Qwen2.5-Coder-7B-Instruct` |
129129
| `LLM_ENDPOINT` | Internal URL for the LLM serving endpoint (used by `codetrans-gaudi-llm-server`). Configured in `compose.yaml`. | `http://${HOST_IP}:8008` |
130130
| `LLM_COMPONENT_NAME` | LLM component name for the LLM Microservice. | `OpeaTextGenService` |
131131
| `BACKEND_SERVICE_ENDPOINT` | External URL for the CodeTrans Gateway (MegaService). Derived from `HOST_IP` and port `7778`. | `http://${HOST_IP}:7777/v1/codetrans` |

‎CodeTrans/docker_compose/intel/set_env.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ source .set_env.sh
99
popd > /dev/null
1010

1111
export host_ip=$(hostname -I | awk '{print $1}')
12-
export LLM_MODEL_ID="mistralai/Mistral-7B-Instruct-v0.3"
12+
export LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct"
1313
export LLM_ENDPOINT="http://${host_ip}:8008"
1414
export LLM_COMPONENT_NAME="OpeaTextGenService"
1515
export NUM_CARDS=1

‎CodeTrans/kubernetes/gmc/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ By default, the LLM model is set to a default value as listed below:
1313

1414
|Service |Model |
1515
|---------|-------------------------|
16-
|LLM |mistralai/Mistral-7B-Instruct-v0.3|
16+
|LLM |Qwen/Qwen2.5-Coder-7B-Instruct|
1717

1818
Change the `MODEL_ID` in `codetrans_xeon.yaml` for your needs.
1919

‎CodeTrans/kubernetes/gmc/codetrans_gaudi.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,6 @@ spec:
2929
internalService:
3030
serviceName: tgi-gaudi-svc
3131
config:
32-
MODEL_ID: mistralai/Mistral-7B-Instruct-v0.3
32+
MODEL_ID: Qwen/Qwen2.5-Coder-7B-Instruct
3333
endpoint: /generate
3434
isDownstreamService: true

‎CodeTrans/kubernetes/gmc/codetrans_xeon.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,6 @@ spec:
2929
internalService:
3030
serviceName: tgi-service
3131
config:
32-
MODEL_ID: mistralai/Mistral-7B-Instruct-v0.3
32+
MODEL_ID: Qwen/Qwen2.5-Coder-7B-Instruct
3333
endpoint: /generate
3434
isDownstreamService: true

0 commit comments

Comments
 (0)