distilbert

Model type	distilbert
Class	DistilBertModel
Task	feature-extraction
Source	models/distilbert.py

Description

DistilBERT encoder-only model for feature extraction.

DistilBERT is a distilled version of BERT with:

• No token type embeddings

• Different weight naming (transformer.layer instead of encoder.layer)

• Pre-norm style attention (sa_layer_norm, output_layer_norm)

Usage

mobius build --model <MODEL_ID> output_dir/

from mobius import build

model = build("<MODEL_ID>")