modernbert-decoder

Model type

modernbert-decoder

Class

ModernBertDecoderModel

Task

text-generation

Source

models/modernbert.py

Description

ModernBERT decoder (causal LM variant).

Extends CausalLMModel with:

  • Embedding LayerNorm (via _ModernBertDecoderTextModel)

  • Three-component LM head: dense → LayerNorm → decoder (with bias)

Replicates HuggingFace ModernBertForCausalLM.

HF weight naming differences vs standard CausalLMModel:

  • Separate q_proj/k_proj/v_proj (not fused Wqkv)

  • Fused MLP Wi → split into gate_proj + up_proj

  • Custom layer-norm names (attn_norm / mlp_norm / final_norm)

  • Embedding LayerNorm at model.embeddings.norm

  • Three-part LM head (lm_head.dense + lm_head.norm + decoder)

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")