bailing_moe

Model type

bailing_moe

Class

Ernie45MoECausalLMModel

Task

text-generation

Source

models/moe.py

Description

Ernie 4.5 Mixture of Experts causal language model.

Uses SoftmaxTopKGate routing with an always-active shared expert per layer (no sigmoid gate on the shared contribution), matching Ernie4_5_MoeForCausalLM.

Key differences from :class:MoECausalLMModel:

  • Each MoE layer adds a shared_expert MLP summed unconditionally.

  • Expert weights stored as fused 3D tensors in HF; split in preprocess_weights.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")