Decoder lora feature

[YonghaoZhao722]

Hi @anwai98, I have something to discuss.

Summary

Add LoRA support for SAM decoder transformer layers with decoder_lora=True parameter.

Changes

• Add SimpleLoRALayer for individual linear layer LoRA adaptation
• Add decoder_lora parameter to PEFT_Sam.__init__()
• Apply LoRA to decoder: self-attention, cross-attention, MLP layers

SimpleLoRALayer

Why I added SimpleLoRALayer instead of adapting LoRASurgery:

• Encoder: Uses combined qkv = nn.Linear(dim, dim*3) → splits output into q,k,v
• Decoder: Uses separate q_proj, k_proj, v_proj layers

Current LoRASurgery/AttentionLoRA (encoder):

# One projection layer for all Q, K, V
qkv = self.qkv_proj(x)
qkv = torch.cat([
    qkv[:, :, :, :self.dim] + new_q,
    qkv[:, :, :, self.dim:-self.dim] + new_k,
    qkv[:, :, :, -self.dim:] + new_v
], dim=-1)

Decoder needs (separate projections):

q = self.q_proj(x) + lora_q(x)
k = self.k_proj(x) + lora_k(x)
v = self.v_proj(x) + lora_v(x)

That's why I add a SimpleLoRALayer. What do you think?

[anwai98]

Hi @YonghaoZhao722,

Thanks for the PR. I had a brief look at the changes. Overall, the changes on a higher level looks good!

I'll take over from here on and refactor the design a bit, to make this modular to support for other PEFT methods in future and clean up a few more things here and there. However, I would only have time in a few days to get back to this.

I'll keep you posted how this goes!

[codecov]

Codecov Report

❌ Patch coverage is 15.68627% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.04%. Comparing base (71181d4) to head (dd51307).
⚠️ Report is 628 commits behind head on dev.

Files with missing lines	Patch %	Lines
micro_sam/models/peft_sam.py	16.32%	41 Missing ⚠️
micro_sam/training/util.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #1095      +/-   ##
==========================================
- Coverage   39.72%   36.04%   -3.68%     
==========================================
  Files          33       54      +21     
  Lines        4179     8450    +4271     
==========================================
+ Hits         1660     3046    +1386     
- Misses       2519     5404    +2885     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.