vibevoice¶
Model type |
|
Class |
|
Task |
|
Source |
|
Description¶
Eight-stage VibeVoice 1.5B continuous-token TTS pipeline.
flowchart LR
Ref[Reference audio] --> AE[Audio encoder]
AE --> AP[Acoustic projector]
Text[Text tokens] --> Emb[Embedding mixer]
AP --> Emb
Emb --> Qwen[Qwen2 decoder]
Qwen --> PosCond[Positive decoder condition]
Qwen --> NegCond[Negative decoder condition]
PosCond --> Diff[DPM-Solver diffusion head]
NegCond --> Diff
Qwen --> PosKV[Positive KV cache]
Qwen --> NegKV[Negative CFG KV cache]
PosKV --> Qwen
NegKV --> Qwen
Diff --> Latent[64-D acoustic latent]
Latent --> AD[Streaming audio decoder]
AD --> Wave[3200-sample waveform chunk]
Wave --> SE[Semantic encoder]
SE --> SP[Semantic projector]
SP --> Qwen
The exported package contains the audio encoder, acoustic projector, embedding mixer, Qwen2 decoder, diffusion head, audio decoder, semantic encoder, and semantic projector. For every generated audio token, the host runs the positive and negative CFG decoder branches, samples one 64-D acoustic latent with DPM-Solver, emits a 3200-sample waveform chunk, and feeds its semantic embedding into the next decoder step.
The decoder owns dual Qwen KV caches. The streaming acoustic decoder and
semantic encoder each expose their convolution histories, for 34 explicit
cache slots per tokenizer stack. The negative CFG branch resets to a valid
suffix after audio BOS; because GroupQueryAttention only represents
prefix-valid lengths through seqlens_k, this decoder declares an
arbitrary-mask contract and retains standard ONNX Attention. The host,
rather than ONNX Runtime GenAI, owns this multi-stage orchestration; the
package is therefore exportable but not an OGA runnable model.
Usage¶
mobius build --model <MODEL_ID> output_dir/
from mobius import build
model = build("<MODEL_ID>")