gptj¶


Model type	`gptj`
Class	`GPTJCausalLMModel`
Task	`text-generation`
Source	`models/gptj_codegen.py`

Description¶

GPT-J causal language model.

Uses single-norm parallel residual: one LayerNorm feeds both the attention and MLP branches whose outputs are summed with the residual. GPT-J weights use separate q_proj, k_proj, v_proj projections (no fused QKV).

GPT-J is MHA only (no GQA), so num_key_value_heads is forced to match num_attention_heads.

Replicates HuggingFace’s GPTJForCausalLM.

Usage¶

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")