cohere2

Model type

cohere2

Class

CohereCausalLMModel

Task

text-generation

Source

models/cohere.py

Description

Cohere and Cohere2 causal language model.

Differences from CausalLMModel:

  • Uses LayerNorm (not RMSNorm) for all normalizations.

  • Parallel pre-norm decoder: a single input_layernorm is shared by both the attention and MLP branches; their outputs are summed together before the residual addition.

  • logit_scale: the final logits are multiplied by a scalar from the HuggingFace config (default 0.0625 = 1/16).

Replicates HuggingFace’s CohereForCausalLM and Cohere2ForCausalLM.

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")