mpt

Model type

mpt

Class

MPTCausalLMModel

Task

text-generation

Source

models/falcon.py

Description

MPT (MosaicML Pretrain Transformer) causal language model.

MPT uses a sequential pre-norm architecture with ALiBi positional encoding:

norm_1 → attention → residual → norm_2 → MLP → residual

This is the standard pre-norm transformer (not parallel attention), with two separate LayerNorms per block.

Weight naming differences from Falcon:

  • transformer.blocks.N.* instead of transformer.h.N.*

  • norm_1 / norm_2 instead of ln_attn / ln_mlp

  • attn.Wqkv (fused QKV) instead of self_attention.query_key_value

  • attn.out_proj instead of self_attention.dense

  • ffn.up_proj / ffn.down_proj (already matches our naming!)

  • transformer.wte instead of transformer.word_embeddings

  • transformer.norm_f instead of transformer.ln_f

  • Uses ALiBi positional encoding (no RoPE)

MPT only supports MHA (no GQA), so num_key_value_heads is forced to match num_attention_heads.

Replicates HuggingFace’s MptForCausalLM.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")