phi

Model type

phi

Class

PhiCausalLMModel

Task

text-generation

Source

models/phi.py

Description

Phi-1/2 causal language model with parallel attention.

Differences from the Llama-style CausalLMModel:

  • Single-norm parallel residual (like GPT-J, not sequential like Llama)

  • Non-gated FCMLP instead of gated MLP

  • Full LayerNorm throughout (not RMSNorm)

  • LM head has a bias term (lm_head.bias)

  • Final norm is model.final_layernorm (matches HF attribute)

Replicates HuggingFace’s PhiForCausalLM.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")