granitemoehybrid

Model type	granitemoehybrid
Class	GraniteMoeHybridCausalLMModel
Task	hybrid-text-generation
Source	models/granitemoehybrid.py

Description

GraniteMoeHybrid hybrid Mamba2+Attention causal language model.

Routed-MoE checkpoints run block_sparse_moe plus shared_mlp; dense checkpoints run shared_mlp alone. Mamba2 layers use the SSD selective scan, while attention layers use RoPE or NoPE according to the config.

Uses HybridCausalLMTask with mixed "mamba2" and "full_attention" layer types for the KV/SSM cache.

HuggingFace reference: GraniteMoeHybridForCausalLM.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")