glm4v_moe_text

Model type	glm4v_moe_text
Class	Glm4MoECausalLMModel
Task	text-generation
Source	models/moe.py

Description

GLM4 Mixture of Experts causal language model.

Uses SigmoidTopKGate routing (sigmoid over logits, not softmax) with an always-active shared expert per layer, matching Glm4MoeMoE.

Key differences from :class:MoECausalLMModel:

• Gate applies sigmoid instead of softmax before top-k selection.

• Each MoE layer adds a shared_expert MLP summed unconditionally.

• Expert weights stored as fused 3D tensors in HF; split in preprocess_weights.

• For GLM4’s default config (n_group=1), group routing collapses to standard top-k.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")