# moonshine_streaming

| | |
|---|---|
| **Model type** | `moonshine_streaming` |
| **Class** | `MoonshineStreamingForConditionalGeneration` |
| **Task** | `speech-to-text` |
| **Source** | `models/moonshine_streaming.py` |

## Description

Moonshine Streaming encoder-decoder model for low-latency ASR.

Replicates Hugging Face ``MoonshineStreamingForConditionalGeneration``: a
causal framing/convolution front end feeding a windowed (bounded-lookahead)
transformer encoder, and a cached Moonshine decoder that cross-attends to the
position-adapted encoder context.

## Caller contract

The export contains an ``encoder`` and a cached ``decoder``. It supports
FP16 CUDA builds (``execution_provider="cuda"``) and CPU builds.

The encoder consumes a rolling raw-audio window, not precomputed features:

| Tensor | Type | Shape | Meaning |
|---|---|---|---|
| ``input_values`` | graph dtype (``float32`` or ``float16``) | ``(batch, audio_samples)`` | Raw 16 kHz waveform. The sample dimension must be a multiple of 80: the model frames 80 samples (5 ms) at a time. |
| ``attention_mask`` | ``int64`` | ``(batch, audio_samples)`` | ``1`` for real samples and ``0`` for right padding. It must match ``input_values``. |
| ``encoder_hidden_states`` | graph dtype | ``(batch, encoder_frames, hidden_size)`` | Encoder output for the decoder. |
| ``encoder_attention_mask`` | ``int64`` | ``(batch, encoder_frames)`` | Downsampled validity mask for encoder frames. |

The causal convolution front end downsamples four 5-ms input frames into each
encoder frame. Per-layer ``sliding_windows`` gives encoder attention bounded
right lookahead, so frames near a window's right edge can change after more
audio arrives. The pinned tiny checkpoint has four layers with four
right-lookahead encoder frames, giving a maximum composed 16-frame (320-ms)
lookahead. The export has no stable-frame-count output; callers must retain
that tail when displaying a provisional result.

This is a streaming *architecture*, not a stateful streaming inference API.
Neither this export nor Hugging Face's
``MoonshineStreamingForConditionalGeneration`` provides encoder cache,
encoder offset, ``predict``, ``commit``, or ``finalize`` inputs/outputs.
Each encoder invocation is independent, so callers cannot faithfully commit
newly stable tokens by chaining ONNX encoder state. The supported protocol is
to retain the raw waveform, periodically re-run the encoder and regular
seq2seq decoder for an uncommitted hypothesis, and publish the final
transcript after end-of-stream. At end-of-stream, pad the final raw window to
an 80-sample boundary, set its padding-mask entries to zero, re-run on the
complete buffered waveform, and decode normally; there is no separate flush
operation or finalization token.

To decode, pass ``decoder_input_ids``, the current encoder outputs, and
``position_ids``, with zero-length ``past_key_values.*.{key,value}`` tensors
for the first token. Feed each ``present.*.{key,value}`` output back as its
corresponding ``past_key_values.*.{key,value}`` input for later tokens while
keeping the encoder outputs and mask fixed. If an application displays
partial hypotheses, it must compare and de-duplicate those ordinary decoded
token sequences at the application layer; they are not model commit events.

## Usage

```bash
mobius build --model <MODEL_ID> output_dir/
```

```python
from mobius import build

model = build("<MODEL_ID>")
```