gpt_neox_japanese

Model type

gpt_neox_japanese

Class

GPTNeoXJapaneseCausalLMModel

Task

text-generation

Source

models/gpt_neox.py

Description

GPT-NeoX-Japanese causal language model.

GPT-NeoX-Japanese uses sequential pre-norm rather than GPT-NeoX’s parallel residual. The backbone, embedding, and LM-head attribute names are otherwise identical to GPTNeoXCausalLMModel.

Additional weight-name differences vs GPT-NeoX:

  • gpt_neox_japanese.* top-level prefix (normalized in preprocess_weights)

  • attention.dense_bias (separate bias tensor, not part of the Linear) is renamed to attention.o_proj.bias during preprocessing

  • mlp.dense_h_to_4h.* / mlp.dense_4h_to_h.* → up/down proj (same renaming as GPT-NeoX)

  • intermediate_multiple_size in HF config determines MLP width: set intermediate_size = intermediate_multiple_size * hidden_size in the test config to match

Replicates HuggingFace’s GPTNeoXJapaneseForCausalLM.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")