persimmon¶
Model type |
|
Class |
|
Task |
|
Source |
|
Description¶
Persimmon causal language model with FCMLP and QK LayerNorm.
Differences from the Llama-style CausalLMModel:
Sequential architecture with LayerNorm (not RMSNorm)
Fused QKV projection (split in
preprocess_weights)Per-head QK LayerNorm in attention (
attn_qk_norm=True)Non-gated FCMLP instead of gated MLP
Final norm is
model.final_layernorm(matches HF attribute)
Persimmon only supports MHA (no GQA), so num_key_value_heads is
forced to match num_attention_heads.
Replicates HuggingFace’s PersimmonForCausalLM.
Usage¶
mobius build --model <MODEL_ID> output_dir/
from mobius import build
model = build("<MODEL_ID>")