diffllama

Model type	diffllama
Class	DiffLlamaCausalLMModel
Task	text-generation
Source	models/diffllama.py

Description

DiffLlama: differential-attention variant of Llama.

Replaces each decoder layer’s standard Attention with DiffLlamaAttention, which computes a weighted difference between two attention streams to reduce attention noise.

All other components (MLP, LayerNorm, embedding, LM head) are identical to the base Llama model.

Replicates HuggingFace’s DiffLlamaForCausalLM.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")