Mistral: Add support for interleaved attention

[manueldeprada]

Adds support for interleaved attention masks to the Mistral model.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[manueldeprada]

run-slow: mistral

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/mistral']
quantizations: [] ...

[Cyrilvallez]

A few comments to make it fit our standard way of doing 🤗

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: minimax, mistral, mixtral, phi3, phi4_multimodal, starcoder2

[manueldeprada]

This check is necessary due to test_modeling_mistral.py::Mask4DTestHard::test_stacked_causal_mask passing a raw 4d mask.

Maybe should I change the test to pass a dict instead? It would break BC...

[ArthurZucker]

this does not make sense, the model is sliding only

[manueldeprada]

run-slow: mistral

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/mistral']
quantizations: [] ...

[ArthurZucker]

Mistral model was never interleaved, so my first review is to go towards no.
We can maybe define this in the config but hardcode it to be sliding. Our philosophy is to not add abstraction when it is not relevant. Same for mixtral: modeling code should be unchanged

[ArthurZucker]

this does not make sense, the model is sliding only

[manueldeprada]

Mistral model was never interleaved, so my first review is to go towards no.
We can maybe define this in the config but hardcode it to be sliding. Our philosophy is to not add abstraction when it is not relevant.

mistralai/Ministral-8B-Instruct-2410 is interleaved, as reported by @hmellor, quoting model's readme "Trained with a 128k context window with interleaved sliding-window attention"

I had understood that we wanted to support this 😅 But if Ministral isn’t relevant enough to justify the modeling change, sure just close the PR 🤗

We can maybe define this in the config but hardcode it to be sliding.

The PR already does this: defaults are set to sliding or full attention unless the model's config says otherwise:

transformers/src/transformers/models/mistral/configuration_mistral.py

Lines 164 to 167 in 2166534

	if self.layer_types is None:
	self.layer_types = [
	"sliding_attention" if self.sliding_window is not None else "full_attention"
	] * num_hidden_layers

(several Mistral models use full attention, example here)

Same for mixtral: modeling code should be unchanged.

This PR doesn’t change Mixtral’s modeling, it only moves Attention’s sliding_window to __init__ so inheriting models can modify it.

[ArthurZucker]

I don't mind the config, but the modeling code is not longer mistral, but qwen2. We need to check because that architecture already exists -> does not make sense for me to do an exception here ! So as much as possible let's check which model is thus the closest

[manueldeprada]

I don't mind the config, but the modeling code is not longer mistral, but qwen2. We need to check because that architecture already exists -> does not make sense for me to do an exception here ! So as much as possible let's check which model is thus the closest

🤯 wow, you have every model on your head! I made a quick Ministral modular from Qwen2 and it matches exactly this PR's outputs with just the bias removal:

class MinistralAttention(Qwen2Attention):
    def __init__(self, config, layer_idx: int):
        super().__init__(config, layer_idx)
        self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)

So I understand now! Whats the preferred approach?:

1. Give up on Ministral and only do the config change for external libraries (and maybe warn that Ministral may be incorrect on long context windows?).
2. Add a new Ministral model with the slim modular from Qwen2.

[hmellor]

https://huggingface.co/mistralai/Ministral-8B-Instruct-2410/blob/main/params.json (Mistral format config) shows that Ministral was supposed to be interleaved.

From what I understand about the history, when this model was originally contributed to Transformers, interleaved sliding attention was not supported, so the model was capped to use all sliding attention.

[ArthurZucker]

2. Add a new Ministral model with the slim modular from Qwen2. would be the best IMO! We can open a PR to support or just do a hardcode case for mistral + interleaved -> load Ministral

[manueldeprada]

2. Add a new Ministral model with the slim modular from Qwen2. would be the best IMO! We can open a PR to support or just do a hardcode case for mistral + interleaved -> load Ministral

I am interested in learning about model bring-up, so I’ll open the PR! Closing this