gpt_oss

Model type

gpt_oss

Class

GPTOSSCausalLMModel

Task

text-generation

Source

models/gptoss.py

Description

GPT-OSS causal language model with MoE FFN and attention sinks.

Architecture highlights:

  • Alternating sliding/full attention layers (config.layer_types)

  • Attention sinks: learned per-head scalar extends softmax denominator

  • GQA with YaRN RoPE and attention projection biases

  • MoE FFN: top-k routing with softmax scores and additive router bias

  • Custom activation: (up+1) * silu_alpha(gate), alpha=1.702

HuggingFace model_type: gpt_oss.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")