hunyuan_v1_moe

Model type

hunyuan_v1_moe

Class

HunYuanMoEV1CausalLMModel

Task

text-generation

Source

models/moe.py

Description

HunYuan MoE V1 causal language model.

Extends the generic MoE model with:

  • QK-norm: per-head RMSNorm on Q and K projections after linear projection.

  • Ungated shared expert (shared_mlp): runs on every token, added to routed expert output without sigmoid gating.

  • Custom gate naming: HF uses gate.wg.weight; we use gate.weight.

HuggingFace class: HunYuanMoEV1ForCausalLM

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")