nemotron¶
Model type |
|
Class |
|
Task |
|
Source |
|
Description¶
Nemotron model with offset LayerNorm and non-gated MLP.
Replicates HuggingFace’s NemotronForCausalLM. Key differences
from the base CausalLMModel:
NemotronLayerNorm1P: Full LayerNorm (with bias) using a +1 weight offset —
LayerNorm(x, weight + 1, bias, eps).Non-gated MLP:
down_proj(act_fn(up_proj(x)))instead of the GLU-style gated MLP.relu2 activation: Squared ReLU
max(x, 0)².partial_rotary_factor = 0.5: Only half of head dimensions receive RoPE positional encoding.
Usage¶
mobius build --model <MODEL_ID> output_dir/
from mobius import build
model = build("<MODEL_ID>")