Qwen4-Exp multimodal pipeline

Created: 2026-08-27 · Last updated: 2026-10-19

Mobius implements both the qwen4_exp_text decoder and the Qwen4ExpForConditionalGeneration composite published as Qwen/Qwen3.8-Flash-Next. The composite exports a standard three-model package: decoder, vision_encoder, and embedding. text_only=True selects the same decoder without the vision stages.

The implementation evidence was collected from:

• Qwen/Qwen3.8-Flash-Next@f5d08274bafd880402bd16f5e3e6c514136ec06c

• unsloth/Qwen3.8-Flash-Next-FP8@41cc25fe32cc20053a59c89716196897580cddf6

• unsloth/Qwen3.8-Flash-Next-GGUF@d3bc75ee6ccef3efc1e228ec00a6cc2cdb1e2249

• huggingface/transformers@598d8ba8baaec7fec5a22da0e2844c7bf4ea20e1

Exported models record the semantic reference as mobius.semantic_reference_revision and record the caller’s requested checkpoint revision separately as mobius.source_revision (unpinned when no revision was supplied).

Exported architecture

The ONNX graph includes the repeating three-linear/one-QSA attention schedule, full-kernel Gated-DeltaNet convolution state, recurrent delta-rule state, four-stream gated residual hyper-connections, exact softmax-first top-k routed MoE plus the sigmoid-gated shared expert, QSA block pooling and token selection, and PLE hashed n-gram embeddings with their dilated convolution and token-context states. The evidenced BF16 checkpoint keeps DeltaNet recurrent math and recurrent cache state in float32, while convolution state, projections, sparse-attention caches, and logits remain in model dtype. Official safetensors are loaded through a bounded-memory package transaction: decoder, embedding, and vision bindings are all validated from one shard index before any graph is mutated; parameter payloads remain lazy and no source state dict is retained. The PLE table is allocated once when serialized, then populated one checkpoint shard at a time.

The flattened cache ABI is:

past_position_ids -> present_position_ids
linear layer:
  conv_state, recurrent_state
PLE linear layer:
  conv_state, recurrent_state, ple_conv_state, ple_context
QSA layer:
  key, value, index_key

The multimodal decoder takes fused inputs_embeds and the original lexical ple_input_ids as independent inputs. Position state has shape [4, B, S]: channel 0 is the text/causal sequence axis, while channels 1–3 carry temporal, height, and width M-RoPE positions used by both sparse attention and QSA.

Vision and embedding reuse

The checkpoint’s vision config proves identity with the no-DeepStack Qwen3.5 tower: 27 blocks, hidden size 1152, intermediate size 4304, 16 heads, patch size 16, temporal patch size 2, spatial merge size 2, and 2304 learned position embeddings. Mobius reuses the Qwen3 vision implementation with DeepStack disabled and the merger projected to the decoder width of 2560.

The embedding graph scatters image_features at token 248056 while preserving the original token IDs for PLE. With an empty image-feature stream, it preserves all lexical embeddings, including an image token emitted during cached decode, matching the HF forward without new pixels. The processor contract follows the evidenced Qwen3 image processor with vision start/end tokens 248053/248054. This package is explicitly image-only: config extraction validates the checkpoint’s video token but removes it from runtime metadata, the embedding graph exposes no video feature input, and direct configs that request video support fail closed. The embedding graph also publishes mobius.unsupported_token_ids and carries a dynamic ONNX Reshape guard: any source video token is sanitized before vocabulary lookup and then requests two output elements from a one-element tensor. ONNX’s element-count invariant is enforced before every ORT EP executes, so direct graph execution cannot silently treat <|video_pad|> as ordinary text. This deliberately avoids out-of-range Gather, which CUDA zero-fills.

QSA uses standard ONNX operators to reproduce the selected-token mask, then runs ordinary dense attention under that mask. This is numerically faithful, including contiguous left padding, but it does not provide the memory savings of a dedicated sparse-attention runtime kernel.

Standard-ONNX-only export

Select the existing execution_provider="onnx-standard" to export the text decoder or all three image-text components without custom-domain operators. Other EPs retain the existing fused com.microsoft::MoE route, parameter layouts, tasks, and cache ABI. No additional model alias or configuration flag is needed.

BF16 image-text export with onnx-standard is rejected at the Qwen4-Exp vision component, including when that component is built independently. Official-weight production-size vision parity failed all four tested cases at rtol=atol=1e-2, with maximum absolute error 0.04052734375; the cause remains unresolved. Use FP32/FP16 for image-text export, or text_only=True for a BF16 decoder. BF16 text-only exports and other EPs are unchanged. This guard does not establish numerical qualification for those alternatives.

The standard expert dispatch selects TopK from float32 router probabilities with a one-dimensional INT64 K tensor. When requested, it divides the selected probabilities by their float32 sum (not a second softmax), then casts route weights to model dtype. A single Scan traverses ascending expert IDs and carries only the [T,H] output accumulator, with no scan-output history. NonZero finds token/route pairs for each expert; an If skips empty experts. The nonempty branch scalar-gathers the expert’s contiguous [2I,H] gate/up and [H,I] down matrices from lexically captured parent banks, applies SwiGLU, multiplies by route weights, and replaces the affected accumulator rows with their old values plus the contribution. SiLU evaluates gate / (1 + exp(-gate)) in FP32 and casts its result back before multiplying by up, preserving HF’s activation rounding without prematurely materializing a low-precision sigmoid. Expert projections, the up product, and accumulation stay in model dtype; optional routed scaling is applied after accumulation. The shared sigmoid-gated expert is unchanged.

The compute graph is independent of expert count: E=2 and E=512 tiny fixtures have the same recursive node count and exactly two MatMuls in the expert body. Weight storage remains O(EHI); per-expert activation workspace is O(RI+RH), where R<=T, plus routing [T,K] and accumulator [T,H]. There are no [T,K,H,I] gathered banks or dense [T,E,I] activations. Scan still visits all E experts and may incur control-flow, NonZero, and scatter overhead; this is not a sparse-kernel performance claim. Existing FP8 QDQ bank reconstruction can remain O(E) in runtime memory and is not optimized by this dispatch.

Standard-only validation runs after optimization, after package opset lowering, and after Transformers dense/streaming/QDQ binding. It recursively checks main graphs, GRAPH/GRAPHS attributes, and remaining function signatures and bodies against standard domains and applicable ONNX schemas. Unresolved custom ops fail with component/path/domain/op diagnostics instead of a warning or EP fallback. This global guard may expose previously unresolved exports in other architectures; it does not change other EPs.

Local evidence uses tiny synthetic text/image-text graphs, ONNX full checker validation in FP32/FP16 (plus BF16 text-only), independent FP32 expert parity (K=1/intermediate/E, ties, near ties, saturated logits, normalization, scaling, mixed/unused experts, batches and single tokens), FP16 routing execution, complete cache/logit prefill/decode comparisons across the QSA boundary, left padding, multimodal decoder parity, a nonzero vision→embedding→decoder pipeline, and reduced dense and storage-preserving FP8 QDQ streaming fixtures. The existing ORT 1.30.0.dev20260823001 CPU runtime executes the FP32 tests and the tiny FP16 expert fixture; its BF16 expert fixture stops at the missing Expand(13) kernel. Schema validity is not proof of kernel support for every dtype, efficient sparse execution, or full-checkpoint generation parity. No full checkpoint generation or performance benchmark is claimed.

An independent torch.nn.functional.silu boundary fixture checks CPU FP32 and CUDA FP32/FP16/BF16 with graph optimization both disabled and enabled. It covers negative tails amplified by large up values, ordinary values, and large finite inputs through the actual Scan/If dispatch, and profiles CUDA expert MatMuls. CPU FP16 numerical parity is not an acceptance target; its routing smoke test does not establish HF SiLU parity.

This change does not enable video, MTP execution, GGUF payload loading, GenAI engine integration, or a paged-cache ABI.

Independent reduced-model parity

The focused integration tests use randomly initialized HF models, not outputs from another Mobius graph. The reference source is Transformers commit 598d8ba8baaec7fec5a22da0e2844c7bf4ea20e1; it requires tokenizers>=0.23.1,<0.24.0. Install or expose that reference in an isolated environment before running:

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest \
  src/mobius/models/qwen4_exp_test.py -q -m integration \
  -k "reduced_random_weight_huggingface or reduced_standard_huggingface"

The primary reduced-model acceptance paths are CPU FP32 and CUDA FP16. The standard-ONNX text cases also retain CUDA FP32/BF16 coverage. They compare full prefill and per-step logits, greedy token choices, linear convolution/recurrent state, PLE convolution/context, QSA key/value/indexer caches, and position history. Text prefill spans the sparse selection boundary and is followed by six cached decode steps. The image-text cases run CPU FP32 and CUDA FP32/FP16 and compare two nonzero, differently shaped image grids through the actual ONNX vision-to-embedding-to-decoder pipeline, independently checking vision features and HF-fused embeddings, then three decode steps with no new images. BF16 image-text regression tests verify export rejection rather than running the unqualified pipeline. Shapes and dtypes are checked alongside numerical values; tolerances are rtol=atol=1e-3 for FP32 and 1e-2 for FP16/BF16. Recurrent state remains FP32 even when weights, embeddings, and convolution/KV caches are FP16/BF16. The expert fixture separately checks FP16/BF16 CUDA dispatch with an empty expert branch and near-tie float32 routing probabilities.

The hyper-connection mean retains the rounded model-dtype products, reduces them with an FP32 accumulator, then casts the result back to model dtype, matching the HF mean without requiring an ORT BF16 ReduceMean kernel. QSA compares the mask bias to zero in FP32, avoiding a BF16 Equal kernel without changing visibility.

This is independent reduced-model evidence, not official checkpoint parity, semantic generation quality, full-size-model reduced-precision parity, or GenAI driver compatibility. Tests skip when the experimental HF classes are absent; CUDA cases additionally require an available CUDA device and ORT CUDA provider. Regression tests simulate either backend being unavailable, including CUDA PyTorch paired with CPU-only ORT, and verify that all three affected test entry points skip before constructing a model or session.

On October 16, 2026, all seven standard reduced-model HF cases passed on ORT 1.30.0 (f1d07f0d1) and 1.32.0.dev20261016001 (f33c1744e). CUDA FP16 profiles recorded 370 decoder matrix executions for text, and 6 vision plus 226 decoder matrix executions for image-text, on each runtime. These are actual CUDA execution events; ordinary CPU metadata/control-flow handling is allowed. This is not full-checkpoint or performance qualification.

Guarded features

The evidenced ordinary Transformers forward preserves MTP metadata but does not execute its mtp.* sidecar. Mobius mirrors that next-token route and does not publish an MTP task. Dedicated MTP embeddings fail closed because no flattened NextN cache ABI exists. Alternative vision geometries and nonempty DeepStack configurations also fail closed.

GGUF header support and payload guard

The GGUF evidence artifact is a text-only general.architecture=qwen4exp split set:

Shard	Tensors	Bytes	LFS SHA-256
UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf	0	10,946,624	88a1420825a9304063e882ada29d438263617f51ac8923d438d927496693bafd
UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00002-of-00003.gguf	595	49,990,818,368	3a62e35bbf9add4733bd1438ebd3a67649d5edd6cb0e72bb78e33c913992b2b6
UD-IQ1_S/Qwen3.8-Flash-Next-UD-IQ1_S-00003-of-00003.gguf	629	22,544,696,352	0e25ceaeb89b8a80aa973c6c0c7448943682f7408c2855b2ebd016b7643a861a

Shard 0 owns all model/tokenizer metadata and no tensors in that evidence set. Production routing is not bound to its repository, revision, filenames, byte sizes, hashes, or shard distribution. Bounded header inspection identifies general.architecture=qwen4exp, while validation uses the model metadata and complete 1,224-name tensor shape/qtype contract. GGUF’s split indexer query/key matrices are concatenated row-wise into Hugging Face’s fused index_qk_proj; they are not Q/K-permuted.

Payload conversion deliberately fails before Hub download. The combined PLE table is an enormous IQ4_NL embedding for which the graph has no compatible native gather ABI. Routed experts are rank-3 banks with IQ1_S gate/up and IQ4_NL down tensors, while the released runtime has neither a mixed-format sparse native-block MoE ABI nor real-weight execution evidence. Treating these as ordinary affine MatMulNBits would be incorrect. Explicit float dequantization is also rejected because the PLE table alone expands beyond the bounded single-tensor materialization policy. The exact header/config/mapping support is therefore a fail-closed foundation for future runtime ABI work, not a quantized execution claim.

Released onnxruntime-genai and the current ONNX GenAI workflow schema cannot represent Qwen4-Exp’s ple_input_ids, four-axis position state, and heterogeneous per-layer PLE/QSA membership. Both metadata exporters therefore fail closed instead of emitting missing bindings, unsupported semantic keys, or lossy %d cache templates. The decoder graph carries a separate mobius.state_manifest metadata document with explicit role-to-layer membership for direct ONNX Runtime orchestration.

FP8 checkpoint evidence

The committed evidence for unsloth/Qwen3.8-Flash-Next-FP8 was collected at immutable revision 41cc25fe32cc20053a59c89716196897580cddf6. Library builds do not lock to that revision: omitting revision follows the Hugging Face default, and an explicit branch, tag, or SHA is forwarded unchanged to config and weight loading. Its 131 safetensors headers were range-read without downloading the 185.5 GB tensor payload. The schema evidence records the config/index hashes, complete tensor census, and canonical header-schema hash.

The checkpoint uses three text-weight paths:

• 73,728 routed-expert matrices use F8_E4M3 values plus BF16 inverse-scale grids. Every grid is validated as exactly [ceil(rows / 128), ceil(cols / 128)].

• 128 PLE embedding shards use F8_E4M3 storage and one shared BF16 ngram_embedding.weight_scale scalar. The evidenced payload is BF16 bits 0x3951 (0.00019931793212890625 as float32). Mobius requires that exact scalar and reconstructs each shard lazily as shard.astype(target_dtype) * weight_scale. The shards remain separate in the ONNX graph instead of concatenating a roughly 95 GiB dense table during export.

• Remaining text weights are ordinary BF16 tensors. The 943-entry modules_to_not_convert list resolves completely against the evidenced header.

By default Mobius preserves every FP8 code tensor and BF16 scale tensor as an external-data initializer. Standard ONNX QDQ reconstructs the logical weights:

• a 2-D block weight is padded and transformed [R,C] -> [Br,128,Bc,128] -> [Br,Bc,128,128] -> [Br*Bc,16384];

• its [Br,Bc] scale grid becomes [Br*Bc] and feeds DequantizeLinear(axis=0);

• inverse reshape/transpose plus a final slice restores [R,C];

• each PLE shard is scalar-dequantized independently before its Gather (standard ONNX Gather does not accept FLOAT8); a dependency chain prevents the next shard DQ from running before the previous token-sized Gather completes. Masked outputs accumulate without a full-table Concat or a roughly 95 GiB destination, so peak runtime storage is one code shard plus one dense shard plus its BF16 scale and any explicit output-dtype cast.

The transform is invertible for source codes, and the external data keeps their exact bytes. weight-loading-report.json records output_weight_format: fp8_qdq, storage_preserving: true, and native_fp8: false: QDQ storage is faithful, but no current ORT execution or fusion capability is claimed. Stock ORT may reject the float8 DQ kernel while the ONNX package remains schema-valid and round-trippable.

Because ONNX IR buffers one output shard before flushing it, streaming packages default to 1 GiB output shards, reject shard limits above 5 GB, and force external-data serialization to one worker. Missing scales, wrong grids, changed deterministic PLE buffers, orphan scales, duplicate source names, unknown tensors, and missing graph targets all fail closed.

The report separates excluded checkpoint families instead of hiding them in a single aggregate: all 3,101 mtp.* tensors are marked mtp_exported: false with the missing-forward/cache-ABI reason, and all 333 model.visual.* tensors are identified as belonging to the dependent multimodal PR.

mobius build \
  --model unsloth/Qwen3.8-Flash-Next-FP8 \
  --revision 41cc25fe32cc20053a59c89716196897580cddf6 \
  --text-only \
  --external-data safetensors \
  --max-shard-size 5GB \
  output/

Pass --dequantize (or API keep_quantized=False) only when an explicitly dense BF16 reconstruction is required.

The FP8 loader currently targets the text-only component and reports visual tensors separately. The branch includes the multimodal graph implementation, but FP8 visual-package loading is not claimed until every visual tensor is classified through the same strict streaming contract. The MTP sidecar remains excluded for the same authoritative-forward/cache-ABI reason documented above. NVFP4 checkpoint lowering remains out of scope.