[BUG]: 用64卡或者80卡训练 loss 总是nan

[xujin1184104394]

Is there an existing issue for this bug?

• I have searched the existing issues

The bug has not been fixed in the latest main branch

• I have checked the latest main branch

Do you feel comfortable sharing a concise (minimal) script that reproduces the error? :)

Yes, I will share a minimal reproducible script.

🐛 Describe the bug

使用的公开库https://github.com/flymin/MagicDriveDiT
按照项目里的按照方式安装colossal的pt分支，（如果直接安装会自动卸载我的新版本pytorch）

Environment

No response

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Title: [BUG]: Use 64 cards or 80 cards to train loss is always nan

[xujin1184104394]

机器用的h20,cuda12.4
环境复用的公开项目的：https://github.com/flymin/MagicDriveDiT/blob/main/requirements/h20_cu124.yaml

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

h20 for machine, cuda12.4
Public project for environmental reuse: https://github.com/flymin/MagicDriveDiT/blob/main/requirements/h20_cu124.yaml

[xujin1184104394]

当初修改从bf16换成fp16 有所缓解，到是训后后面还是炸nan了。

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The original modification changed from bf16 to fp16 to ease, but it was still a blow-up after training.

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Whether you are developing an app or a website, the free AI code tool https://code.mayoubang.cn can help you!