gpt-sw3

Model type

gpt-sw3

Class

GPT2CausalLMModel

Task

text-generation

Source

models/gpt2.py

Description

GPT-2 causal language model.

Differences from Llama-style:

  • Pre-norm with LayerNorm (not RMSNorm)

  • Absolute positional embeddings (not RoPE)

  • Combined QKV projection in HF weights (split during preprocess)

  • Conv1D weights are transposed vs Linear

  • Tied word embeddings (lm_head = wte)

Replicates HuggingFace’s GPT2LMHeadModel.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")