nanochat¶
Model type |
|
Class |
|
Task |
|
Source |
|
Description¶
NanoChat model: FCMLP with relu2 + parameter-free RMSNorm + QK-norm.
HF NanoChat applies RoPE then QK-norm (reversed from standard Llama).
We replace every layer’s self_attn with NanoChatAttention which
implements the correct RoPE → QK-norm ordering.
HF NanoChat also applies norm before AND after the transformer layers
(via NanoChatTextModel).
NanoChat also applies final logit soft-capping:
logits = tanh(logits / cap) * cap where cap = 15.0 by default.
Usage¶
mobius build --model <MODEL_ID> output_dir/
from mobius import build
model = build("<MODEL_ID>")