Skip to content

Nixl optimization for llama4 local attention #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 59 commits into
base: pd-launch-branch
Choose a base branch
from
Open

Nixl optimization for llama4 local attention #87

wants to merge 59 commits into from

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented May 15, 2025 •
edited by github-actions bot
Loading

No description provided.

ekagra-ranjan and others added 30 commits May 14, 2025 12:31
@ekagra-ranjan @WoosukKwon
[V1][Spec Decode] Share input embedding of target model with EAGLE dr…
418d2f8 
…aft model to free ~1GB for llama 3 model (vllm-project#17326)

Co-authored-by: root <root@ekagra-8xh100.us-east5-a.c.serving-efficiency-poc.internal>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
@bnellnm
Modularize fused experts and integrate PPLX kernels (vllm-project#15956)
f9c069c
@robertgshaw2-redhat
[CI] Disable Failing Tests (vllm-project#18165)
8568650
@mgoin
Local attention optimization for NIXL
7e55a34 
Signed-off-by: mgoin <mgoin64@gmail.com>
@mgoin
Clean up a lot!
8ea467d 
Signed-off-by: mgoin <mgoin64@gmail.com>
@mgoin
Small opt
73a8272 
Signed-off-by: mgoin <mgoin64@gmail.com>
@davidxia @aarnphm
[Frontend] decrease import time of vllm.multimodal (vllm-project#18031)
749f792 
Co-authored-by: Aaron Pham <Aaronpham0103@gmail.com>
@LucasWilkinson
[Kernel] Have rotary embeddings support tensors (vllm-project#18046)
d93c976 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@aarnphm @russellb
[V1] Structured Outputs + Thinking compatibility (vllm-project#16577)
2fc9075 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
@mgoin
Fix mypy
17cc4c9 
Signed-off-by: mgoin <mgoin64@gmail.com>
@jerryzh168
Add support for loading torchao models with AOPerModuleConfig (vllm…
7974736 
…-project#17826)

Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
@russellb
[CI] Fix race condition in test_kv_cache_events test (vllm-project#18169
78aa341 
)

Signed-off-by: Russell Bryant <rbryant@redhat.com>
@mgoin @njhill
[V1] Support multiple kv connectors (vllm-project#17564)
2142035 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
@atalman
Upload vllm index for the rc builds (vllm-project#18173)
09f106a
@davidxia
[Bugfix]: make most of test_openai_schema.py pass (vllm-project#17664)
f25e0d1
@heheda12345
[v1] Support multiple KV cache groups in GPU model runner (vllm-proje…
e60f550 
…ct#17945)

Signed-off-by: Chen Zhang <zhangch99@outlook.com>
@markmc
[V1][Metrics] Remove unused code (vllm-project#18158)
65334ef 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
@aarnphm
[Chore] astral's ty (vllm-project#18116)
afe3236 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
@reidliu41
[Misc] add lobe-chat support (vllm-project#18177)
2dff093 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
@ProExpertProg
[Fix][ROCm] Enforce eager for all encoder-decoder models on ROCm (vll…
83f74c6 
…m-project#18154)

Signed-off-by: Luka Govedič <lgovedic@redhat.com>
@hmellor
Update deprecated type hinting in models (vllm-project#18132)
26d0419 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@tdoublep @LucasWilkinson
[Bugfix] Fix fp8 tests for triton_unified_attention for Triton 3.3 (v…
e6b8e65 
…llm-project#18013)

Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@huachenheli
Support custom implementations of VideoLoader backends. (vllm-project…
4f07a64 
…#18091)
@andyxning
[UT] Add ut for none hash (vllm-project#17892)
420caf7 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
@inkcherry
[Model] Allow the use of sliding window in Qwen2 (vllm-project#17772)
dd2a945 
Signed-off-by: inkcherry <mingzhi.liu@intel.com>
@MengqingCao
[Bugfix] Fix FusedMoEPrepareAndFinalize for cuda-disalike backends (v…
70f8b96 
…llm-project#18178)

Signed-off-by: Mengqing Cao <cmq0113@163.com>
@davidxia
[CI] don't skip fixed test_kv_cache_events() (vllm-project#18183)
de71fec 
Signed-off-by: David Xia <david@davidxia.com>
@russellb
[V1] Update zmq socket creation in nixl connector (vllm-project#18148)
a8f5aec 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
@omahs
fix: typos (vllm-project#18151)
a9944aa 
Signed-off-by: omahs <73983677+omahs@users.noreply.github.com>
@hmellor
Update deprecated type hinting in model_loader (vllm-project#18130)
07ad271 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
tjtanaa and others added 23 commits May 15, 2025 09:53
@tjtanaa
[Bugfix] [ROCm]: Remove assertion logic when using AITER fused moe in…
9254052 
… unquantizedMethod to reenable LLama4 BF16 (vllm-project#18205)

Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
@NickLucche
[Misc] Avoid cuda graph log when sizes still match (vllm-project#18202)
e3f3aee 
Signed-off-by: NickLucche <nlucches@redhat.com>
@Alexei-V-Ivanov-AMD
Adding "AMD: Tensorizer Test" to amdproduction. (vllm-project#18216)
0b34593
@luccafong
[Bugfix] Fix test_eagle test (vllm-project#18223)
8795eb9 
Signed-off-by: Lucia Fang <fanglu@fb.com>
@LucasWilkinson
[Build] Allow shipping PTX on a per-file basis (vllm-project#18155)
c7852a6 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
[Bugfix] fix rotary embedding test for _get_padded_tensor_shape (vllm…
4e1c6a0 
…-project#18229)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@kliuae
[Bugfix][ROCm] Use chunked_prefill_paged_decode as fallback for V1 …
ee659e3 
…attention on ROCm (vllm-project#18093)

Signed-off-by: kf <kuanfu.liu@embeddedllm.com>
@skylee-01
[Model] vLLM v1 supports Medusa (vllm-project#17956)
f4937a5 
Signed-off-by: lisiqi23 <lisiqi23@xiaomi.com>
Signed-off-by: skylee-01 <497627264@qq.com>
Co-authored-by: lisiqi23 <lisiqi23@xiaomi.com>
@hmellor
Allow users to pass arbitrary JSON keys from CLI (vllm-project#18208)
b18201f 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@wseaton
Throw better error for when running into k8s service discovery issue (v…
6b31c84 
…llm-project#18209)

Signed-off-by: Will Eaton <weaton@redhat.com>
@luccafong
[Feature] Support Pipeline Parallism in torchrun SPMD offline inferen…
3d2779c 
…ce for V1 (vllm-project#17827)

Signed-off-by: Lucia Fang <fanglu@fb.com>
@davidxia
[doc] fix multimodal example script (vllm-project#18089)
5c04bb8 
Signed-off-by: David Xia <david@davidxia.com>
@vadiklyutiy
[PERF] Speed up Qwen2.5-VL model by speed up rotary position embedding (
67da572 
vllm-project#17973)

Signed-off-by: Vadim Gimpelson <vadim.gimpelson@centml.ai>
@eicherseiji
[Misc] Add Ray Prometheus logger to V1 (vllm-project#17925)
5418176 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@Isotr0py
[Misc] Consolidate Audio tests into multimodal common generation tests (
390ec88 
vllm-project#18214)

Signed-off-by: Isotr0py <2037008807@qq.com>
@IwakuraRein
use ceil_div in cutlass block scaling shape check (vllm-project#17918)
e23564c
@fxmarty-amd
[Fix] Fix typo in resolve_hf_chat_template (vllm-project#18259)
a5f8c11 
Signed-off-by: Felix Marty <felmarty@amd.com>
@learner0810
[Model] Use autoweightloader for dbrx (vllm-project#18251)
87d8714 
Signed-off-by: learner0810 <zhongjun.li@daocloud.io>
@reidliu41
[Misc][MacOS] fix bfloat16 error (vllm-project#18249)
d3d91b6 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
@njhill
[BugFix] Fix multi async save in MultiConnector (vllm-project#18246)
1db4f47 
Signed-off-by: Nick Hill <nhill@redhat.com>
@mgoin
Add TODO to remove
af2f264 
Signed-off-by: mgoin <mgoin64@gmail.com>
@mgoin
Merge branch 'main' into nixl-l4-opt
e8fd2f1
@mgoin
Bug fixes
7f0ef82 
Signed-off-by: mgoin <mgoin64@gmail.com>
@github-actions GitHub Actions
Copy link

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!

@github-actions github-actions bot added the stale label Aug 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.