xglm

Model type	xglm
Class	ScaledEmbeddingGPT2CausalLMModel
Task	text-generation
Source	models/gpt2.py

Description

GPT-2-family model whose input embedding activation is scaled.

BioGPT and XGLM multiply token embeddings by sqrt(hidden_size) but use the unscaled tied table for their output projection. Their ONNX graphs therefore need two physical parameters even when the logical HF config says that word embeddings are tied.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")