Qwen3.8-27B standard-only multimodal export qualification

Created: · Last updated:

Source and scope

Qwen/Qwen3.8-27B uses the existing qwen3_5 multimodal architecture and qwen3_5_text decoder, not Flash-Next’s qwen4_exp architecture. Its text decoder has 64 layers: 48 Gated DeltaNet layers and 16 full-attention layers, with a dense MLP in every layer.

The offline fixture testdata/configs/qwen3_8-27b.json preserves the official config values from revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0. The upstream raw config.json has SHA256 191e0af232104ed8b65258cf3fb2b842e288008baca7633c11b82a1ac7203aab and size 4312 bytes. The checked-in fixture is reformatted, so that raw-byte hash is provenance, not the fixture’s byte hash.

The target is the complete, unquantized standard ONNX vision-language package, not just its text decoder. Text-only export remains an optional subset: removing vision does not remove any hybrid decoder layers. The checked-in tests cover full-size graph construction and reduced random-weight text/image/video parity. They do not download model weights or establish official-checkpoint accuracy. Separate bounded native ORT GenAI execution and AI2D first-200 evidence are recorded below; they do not establish full numerical parity or production readiness. ModelBuilder equivalence remains deferred, not an acceptance gate for this export.

Operators and state contract

execution_provider="onnx-standard" disables custom-op fusion and inlines the CausalConvWithState and LinearAttention function bodies. The resulting graph uses standard convolution, per-head normalization, gate arithmetic, Scan recurrence, standard attention, and dense MLP operations. Tests require standard domains and schemas recursively, including the Scan bodies, and no remaining local functions.

The public Transformers builder supports text_only=True for the official qwen3_5 composite and its qwen3_5_text sibling. The resulting package has one model entry rather than a vision/embedding/decoder pipeline.

With text_only=False, the package contains vision_encoder, embedding, and decoder. The full decoder still contains all 48 DeltaNet and 16 full-attention layers; the vision encoder contains all 27 layers.

Visual feature contract

The vision graph accepts processor-native FLOAT32 pixel_values and casts once to model dtype internally. Run image and video packed patches through the same vision graph, supplying each stream’s grid as image_grid_thw.

The embedding graph keeps its existing input_ids and image_features input names. Despite the latter’s name, it accepts both image and video features. Pack feature rows in the order their placeholder tokens occur in flattened, row-major input_ids, including across batch rows. Do not concatenate all image rows before all video rows unless the placeholders actually follow that order. Feature indices do not restart at each batch row. For text-only input or decode without new media, supply an empty [0, hidden_size] feature tensor of model dtype.

The same global indices apply to shared Qwen3-VL DeepStack feature maps. Qwen3.8-27B has no DeepStack maps. A separate embedding-only CPU sentinel test covers two maps, two-row mixed image/video prompts with reversed order, and zero-media input; full DeepStack decoder parity remains unqualified. This fixes shared Qwen3/Qwen3.5 split components; existing saved graphs must be re-exported to receive the fixes. GenAI’s ability to assemble the packed image/video stream is not established here.

For each DeltaNet layer:

  • Convolution state is model dtype, [B, conv_dim, K-1].

  • Recurrent state is [B, value_heads, key_dim, value_dim].

  • The official mamba_ssm_dtype="float32" is preserved through config extraction, task inputs, recurrence function registration, and outputs, even when weights/activations are FP16.

  • Configs without an explicit state-dtype policy (or with auto) retain the prior model-dtype behavior. Invalid state dtypes are rejected.

  • Qwen3.5 per-head Q/K normalization uses the reference epsilon 1e-6, rather than the generic component’s zero-epsilon default.

The epsilon is fixed by the Qwen3.5 reference architecture, not a configurable Qwen3.5 export option. The shared decoder change also applies to DeltaNet layers constructed by Qwen3.5 MoE/VL variants. Their graph/weight regressions are included, but this qualification does not establish official-checkpoint parity for those other variants. The existing MTP decoder is full-attention and does not construct DeltaNet. FP16/BF16/FP64 state policies can be represented by the config; only the official FP32 policy is numerically qualified here.

Learned offset RMS normalization now computes input normalization, the 1 + weight offset and scale multiplication in FP32 before casting the output back to model dtype. The prior implementation rounded the offset in FP16/BF16. A nonzero learned-weight unit test reproduces that difference; zero-initialized reference norms do not exercise it. This corrects the shared component contract but does not resolve full-checkpoint parity alone.

The HF reference caches K convolution positions; Mobius needs the K-1 preceding positions. Comparisons explicitly select the final K-1 HF entries. The recurrent-state comparison uses nonsquare key/value dimensions in the reduced fixture to detect an accidental transpose. This is a dense invocation ABI, not the native packed/paged contract in PR #738.

Compatibility: re-exporting a hybrid checkpoint that requests FP32 state changes its FP16/BF16 recurrent slots to FP32. Existing saved models do not change. Callers must allocate/rebind cache using the graph’s per-slot types, not assume that every state has model dtype. The raw ORT parity tests exercise that contract; compatibility of a particular GenAI cache allocator is not established by the standard export alone.

Native ORT-GenAI bounded qualification

The extension preserves the exact three standard graphs and native Qwen3.5 multimodal dispatch; it does not substitute a generic decoder, ModelBuilder, manual Python cache loop, PagedAttention, or fused MoE. The ordinary CLI route and existing --runtime ort-genai route are covered by tiny offline VLM tests in integrations/ort_genai/qwen38_standard_test.py. No new Mobius CLI flag was added. --ep onnx-standard remains the build policy; execution selects CUDA with og.Config.clear_providers(), append_provider("CUDAExecutionProvider"), and og.Model(config).

Two reproduced metadata gaps were corrected:

  • Local --config assets reached tokenizer/generation metadata but not the vision processor writer. The writer now uses the explicit HF source or, if absent, local_config_dir, retaining checkpoint normalization and smart-resize bounds.

  • The VLM’s unwrapped qwen3_5_text configuration was missing from the Qwen vision-family sets, dropping PatchImage and Qwen vision metadata.

The VLM writer already bypasses the text-only _inspect_decoder_abi; its inputs_embeds mapping and graph-derived highest global cache slot did not require an inspector rewrite. Standard hybrid packaging keeps sharing false and capture off. Generic fast CPU e2e tests remain pinned to GenAI 0.15.2.

Native runtime evidence and provider-routing diagnosis (2026-10-08)

The isolated checkout at 477380e595bc75fcdc6a87939d5c2246741bedc1 imports Mobius from its own src. Installed ORT is 1.30.0 and GenAI reports 0.16.0-dev; its build identity has not been established.

tests/ort_genai_hybrid_native_test.py is opt-in through MOBIUS_QWEN_NATIVE_PROBE=1, independent of the generic fast CPU lane. Its standard-only synthetic three-graph pipeline has eight global slots, sparse KV at 3 and 7, FP16 convolution state, FP32 recurrent state, and 3D positions. The final probe uses the real per-slot geometry: hidden width 5120, KV [B,4,pastS,256], conv [B,10240,3], and recurrent [B,48,128,128], but only eight slots and a 32-token synthetic vocabulary, without checkpoint weights. Expected tokens [0, 4, 8] depend on both state families and past KV length. Every state output’s dtype, shape, and progression is checked; a fresh generator must reproduce the same sequence. Model construction alone is not a pass.

On GPU GPU-630e9db2-c115-0a2d-789f-4a76ccb92ecc:

  • The generic native CUDA control passed, generating [0, 1, 2].

  • The hybrid native model and bundled multimodal processor were constructed successfully, and native text inputs were produced. Thus absence of the Python onnxruntime_extensions package is not evidence of missing bundled processor support.

  • The initial mixed-provider probe segfaulted during generator.set_inputs(inputs) / native prefill, not generator construction. The earlier attribution was corrected after explicit stage markers and GDB backtraces. CPU ConstantOfShape::Compute, and then CPU GatherCopyData<long> after replacing the synthetic embedding with Gather, were called through MultiModalPipelineState::Run and OgaGenerator_SetInputs. Recurrent-cache incompatibility was not established.

  • With empty auxiliary provider lists, the execution override did not route vision/embedding to CUDA. Config.overlay() patches specifying auxiliary or all three providers did not resolve the fault in either tested call order.

  • Rendering CUDA runtime session metadata for all three standard graphs through the existing writer, keeping capture off, resolved native prefill. The required clear_providers() / append_provider("CUDAExecutionProvider") / Model(config) calls remain. This does not select CUDA graph construction or alter/export any ONNX graph.

  • The synthetic native probe then passed all state-dependent token, dtype, shape, multi-step state progression, and fresh-generator reset assertions: [[0,4,8], [0,4,8]]. The installed dev runtime therefore demonstrated sparse global indexing, mixed cache types, and non-shared swaps; published source capability is no longer being used as a substitute for execution evidence.

Run the passing opt-in probe with one bounded command (fresh absolute basetemp):

env PYTHONPATH=/home/titaiwang/workspace/mobius/.worktrees/qwen27b-genai-9a3ab8ed/src \
  TMPDIR=/datadisks/disk5/titaiwang/mobius-qwen38-27b-standard-9a3ab8ed/genai-integration/tmp \
  HF_HOME=/datadisks/disk5/titaiwang/mobius-qwen38-27b-standard-9a3ab8ed/genai-integration/hf \
  HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTHONDONTWRITEBYTECODE=1 \
  CUDA_VISIBLE_DEVICES=GPU-630e9db2-c115-0a2d-789f-4a76ccb92ecc \
  MOBIUS_QWEN_NATIVE_PROBE=1 timeout 120 \
  /home/titaiwang/miniconda3/envs/dev/bin/python -m pytest \
  /home/titaiwang/workspace/mobius/.worktrees/qwen27b-genai-9a3ab8ed/tests/ort_genai_hybrid_native_test.py::test_native_sparse_mixed_state_progression_and_reset \
  -q -s --tb=long \
  -o cache_dir=/datadisks/disk5/titaiwang/mobius-qwen38-27b-standard-9a3ab8ed/genai-integration/native-control-cache \
  --basetemp=/datadisks/disk5/titaiwang/mobius-qwen38-27b-standard-9a3ab8ed/genai-integration/native-probe-next

Replacing/installing a runtime or rebuilding native C++ requires separate approval. Do not relax parity tolerances or substitute raw ORT to declare this gate passed.

Metadata-only real-artifact overlay

examples/qwen38_standard_genai.py accepts absolute source, HF-assets, and fresh output paths. It loads graph metadata lazily with onnx_ir, reconstructs ArchitectureConfig from the pinned HF text config and parent, and uses the repository’s config writer. It never calls ModelPackage.load/save, loads an HF model, or serializes/copies external weights. Component directories are symlinked, preserving their sidecars; small tokenizer/metadata files are copied before potentially mutable asset patches. Existing destinations are refused.

CUDA preparation and bounded generation passed for the immutable onnx-rmsfix artifact and produced:

/datadisks/disk5/titaiwang/mobius-qwen38-27b-standard-9a3ab8ed/genai-integration/runtime-overlay-cuda-02/overlay-receipt.json

and native-results.json in that same overlay.

The audit confirmed zero functions/custom-domain nodes in all three graphs, global KV indices 3,7,...,63, FP16 convolution and FP32 recurrent ports, and unchanged original HF-asset hashes. The decoder, vision, and embedding graph metadata sizes are respectively 3,421,999, 1,065,742, and 6,663 bytes. No 54GB weight duplication occurred. Transformers emitted documentation errors for min_frames/max_frames during processor import, but preparation exited successfully. These warnings were not generation failures.

Use --prepare-only for metadata preparation. Real generation requires --native-proof /absolute/.../native-proof.json from a successful synthetic test with the same reported installed runtime version. The example selects CUDA through og.Config, delegates preprocessing/positions/caches/tokens to GenAI, and bounds generation to 1–16 new tokens. It checks text, a fresh- generator text reset, and an optional single image. A programmatic second_image argument additionally checks two different images without a new CLI flag. It releases native objects after use and has no success-shaped fallback.

The proof is a local operator attestation, not a runtime binary identity check: development builds can share the same version string. Rerun the native proof after any runtime binary change. Matching installed binary SHA256 fingerprints for this qualification are stored in genai-integration/qualification-receipt.json on the qualification artifact disk; the example does not automatically verify them.

Bounded real native results

These results used historical runtime-overlay-cuda-02 metadata before the RGB normalization correction below. They establish execution only; fresh metadata and a real-image rerun are required to validate the corrected inputs.

The 420-second-bounded command completed successfully with four new greedy tokens per invocation, thinking disabled, and no HF model loaded alongside GenAI:

Invocation

Native-expanded prefill

Tokens

Decoded

Text

17

[760,6511,314,9338]

The capital of France

Fresh-generator text reset

17

Same four tokens

Same text

Single pipeline-cat-chonk.jpeg image

649

[1919,2099,4774,264]

This image shows a

Cat + mage-vl-dog.jpg images

2700

[760,1118,2099,4774]

The first image shows

GenAI owned image preprocessing, 3D positions, all cache slots, and token generation. This establishes bounded native execution and text reset, not independent numerical correctness of native preprocessing/positions or full-checkpoint accuracy. The native model was released on completion. The immutable source remains 54,779,987,399 bytes; only small metadata/tokenizer files and symlinks were added. No weights were duplicated or reserialized.

The first real command covered text/reset/single-image. A second 420-second- bounded invocation reused the same CUDA metadata overlay and immutable weights to cover the two-image case as well; it again reproduced the same text/reset and single-image tokens. It did not create a second model export or load two native models simultaneously.

Bounded phase status: executed. Synthetic progression/reset and real text/ single-/two-image few-token generation passed. Full AI2D200 evaluation was not run in this phase. The prior FP16 logit-parity failures and unchanged 1e-2 tolerances remain unresolved. No performance, CPU/DML, video, serving, or capture promise is made.

RGB metadata producer correction

The native DecodeImage(color_space="RGB") operation already emits RGB. In the upstream ORT-Extensions producer contract (operators/vision/decode_image.hpp and shared/api/image_transforms.hpp), nonzero Normalize.qwen2_5_vl or Normalize.qwen3_vl enables a BGR-to-RGB swap, not patch interleaving. Injecting that flag after RGB decode therefore reversed red and blue. The writer now leaves ordinary Normalize mean/std attributes unchanged; PatchImage separately owns temporal/spatial patch layout. Resize, rescale, all graphs, cache policy, and model tolerances are unchanged. Regenerate processor metadata; historical overlays are not patched in place.

The lead’s before-prefill diagnostic found matching image/grid/IDs but direct pixel max-absolute difference 2.0. Red/blue reversal alone, without row reorder, reduced it to approximately 0.0156862. That residual resize/rounding difference remains unresolved and is not bitwise pixel parity.

An opt-in toy native regression uses a lossless uniform RGB [255,128,0] image, only three added image-marker tokens, and no checkpoint weights. It checks the native processor’s pixel values against analytical per-channel normalization before any generator/prefill call. No manual flips, row reorders, graph changes, or tensor substitutions are used. This independently reproduces the old producer’s swapped channels and guards the corrected RGB contract; it does not qualify real-image resize behavior or model parity.

Verification

Corrected-RGB native AI2D first-200 evidence

The corrected metadata-only runtime-overlay-cuda-03 completed all 200 pinned AI2D examples through installed ORT GenAI 0.16.0-dev on CUDA, using native image processing, positions, caches, and greedy generation (16 new tokens, thinking disabled). HF processing supplied only the identical chat template and independent diagnostics, never tensors to the native pipeline.

Metric

Corrected native GenAI

Prior HF Torch

Prior raw ORT CUDA

Correct answers

174/200

175/200

174/200

Invalid native answers

3

—

—

Exact native generated tokens

—

199/200

198/200

This is completed execution/task evidence, not preprocessing or full-logit parity. Native IDs, image grids, and pixel shapes matched HF for 172/200 examples. The other 28 have a diagnosed native smart-resize rounding difference: ORT-Extensions uses std::round, whereas the installed HF processor uses Python’s ties-to-even round. Both rounding models predict their respective observed grids for all 200 examples. For example, the 400-pixel width at zero-based index 130 becomes native 416 versus HF 384 at factor 32; its native answer is B versus the HF/raw-ORT answer D. This coincidence does not establish that resizing alone caused the answer change. Index 124 instead matches HF’s C, while prior raw ORT answered A.

For the 172 shape-comparable examples, maximum pixel absolute difference was 0.0156863928 and minimum correlation was 0.9999995187. Pixel comparisons for the 28 different-shape examples were not counted as passes. The inspected upstream Resize attribute contract exposes no rounding-mode override. Runtime-side alignment remains outside this Mobius change; no tensor flip, pre-resize workaround, new CLI flag, or ONNX weight rebuild was used.

The results and rounding diagnosis are retained in genai-integration/ai2d-native-rgb-fixed-first200-01/ and genai-integration/native-smart-resize-diagnosis.json on the qualification artifact disk. Prior Torch/raw-ORT results are frozen references, not fresh reruns. The existing official FP16 full-logit parity blocker remains open.

Input-size and repeated-inference memory findings

The native AI2D200 runner used one Python process and one Model for the entire ordered subset, creating and deleting a Generator per question. Its preserved runner SHA256 is e7f0306d524f04d1a527d9470929a447a54edd626e08c8e8afd56012fe0b5541. Observed native prompt lengths (including image tokens) have median 314.5 and maximum 1,705; image patch rows have median 1,064 and maximum 6,580. Generation was capped at 16 tokens. These are observed subset bounds, not dataset-wide maxima or a deployment-safe input limit.

A separate full-resolution large-image diagnostic used 4,875 prompt tokens, 19,200 patch rows, and a 128-token generation cap. With the same Model kept alive and a fresh Generator per call, the first call completed and exactly matched the standalone tokens, text, processor tensors, and prompt. The second call failed at decoder layer 3’s Attention_node_335, requesting 4,563,000,320 bytes. GPU usage rose from 57,947 MiB after Model loading to 81,133 MiB after the first call, leaving 20 MiB free; deleting the Generator and collecting garbage did not reduce that usage while the Model remained alive. Process exit restored 81,150 MiB free and 4 MiB used.

This diagnostic used an isolated GenAI build at source commit 265917c09ffe68198a026e086d21e694cc938298 with a one-line last-token logits allocation correction and package-local $ORIGIN RPATH. It retained the installed ORT library (SHA256 5a2e81ad05259bce596e29d6361fe76750b6c2c9569b8cdddf709ab0d944b0ee), unchanged ONNX graphs, full-resolution images, and generation policy. It is not the installed GenAI binary used for AI2D200; that binary’s originating source commit is unknown. The external GenAI correction is not included in this Mobius PR and did not resolve repeated-inference OOM.

The exported decoder contains 16 standard Attention nodes with explicit FP16 masks and dynamic past/present KV ports, and no TensorScatter nodes. For the failing 4,875-token case, the local ORT unfused workspace formula matches the requested bytes exactly: 2 * align256(24 * 4875 * 4875 * sizeof(float)) = 4,563,000,320. The inspected CUDA dispatch rejects Flash with an explicit mask and rejects MEA when a masked KV row stride is not a multiple of four; 4875 % 4 = 3. This is strong source-and-size evidence for unfused Attention workspace, not a captured installed-binary dispatch trace. Exact correspondence of the inspected ORT source to the installed binary remains unverified. Other large-image attempts also failed in the vision encoder, so decoder KV-cache changes alone are not established as a complete remedy.

Repeated-inference failure is confirmed, but live tensor ownership, arena retention, and fragmentation have not been distinguished. It is not a proven KV-cache leak, graph incompatibility, or numerical-parity fix. TensorScatter static-cache A/B testing has not been performed. Deployment needs validated post-processor input bounds, generation/concurrency budgets, and repeated mixed-size request testing; AI2D200 success alone does not qualify those. No MMMU accuracy result is claimed here.

Evidence remains under genai-runtime-fix-01/probes/candidate-case56-repeat01/ (receipt.json, execution.json) on the qualification artifact disk, alongside the immutable AI2D records. No additional benchmark run or graph change is part of this documentation update.

Native numerical diagnostics and reset

Actual native ORT GenAI was compared with the pinned HF FP16 reference using native processor snapshots and a native-selected token prefix. HF maintained its own cache; no HF tensors or states were injected into GenAI. This isolates model numerics from independent preprocessing and generation, and is not a replacement for the independently generated AI2D results above.

The completed cases cover text and AI2D indices 0, 124, and 130, plus repeated text/image and a fresh-generator text reset. Each completed case compares prefill and four cached decode forwards: all 248,320 last-position logits and all 128 convolution, recurrent, and KV outputs. Raw graph logits are FP16; the GenAI logits API’s FP32 promotion was verified separately.

At unchanged rtol=atol=1e-2, all 35 paired forward comparisons failed full-vocabulary logit tolerance, while all 35 argmax predictions matched. These include repeated cases, not 35 independent examples. Of 4,480 state comparisons, 2,907 passed and 1,573 failed. Maximum absolute errors were 0.236328125 for logits and 85.25 for states. There were no shape, dtype, or nonfinite failures in those numerical comparisons. The first failing cache layer for text was layer 31; image cases could fail at earlier layers. Neither this observation nor output-token agreement identifies the root cause or clears the numerical gate.

In the repeat run, all 30 directly captured native position_ids and attention_mask comparisons exactly matched HF-derived shapes, int64 dtypes, and values. Recreating the generator after an intervening image case reproduced all five text logits bitwise, all 640 state hashes, and the selected tokens. This establishes reset repeatability and input agreement only for those tested cases.

The two-image native processor emits two maximum-length patch slots: 1,064 and 864 valid patches become 2,128 stored rows, including 200 zero padding rows, while the grids describe 1,928 valid patches. Native prefill executed with finite logits and all expected state schemas, but HF could not consume that unchanged padded snapshot. Its paired numerical case remains blocked; this is not proof that the native pipeline is incompatible. No unpadding, repacking, resizing, or tolerance change was used.

Receipts, full-vocabulary vectors, input captures, and state hashes are in genai-integration/native-numerical-validation-01/attempt-03/, including final-consolidated-receipt.json and reset-receipt.json. The preceding attempt-02 evidence is preserved unchanged. Official FP16 numerical qualification remains failed; task-level validation does not waive it.

CI disposition for the implementation head

At implementation head 9e8d6aa88ee16a8db77df1dbe0b1a1e585d99d61, CI run 37867805012 did not complete its L4/L5 golden suites. Both check-run annotations explicitly state that the jobs exceeded the one-hour execution limit. Cancellation is not a successful numerical or generation test, and the partial logs also contain failed test markers.

The exact base is 26a712b210a0a7cd895347f601680e82bbf5e452. Its main CI run 37681522335 and scheduled L4/L5 runs 37882646916 and 37886682810 also ended without completing those suites. All 24 named L4 failures observed on the PR head have failed markers on that exact base. For L5, 19 of 20 named head failures have corresponding base failed markers. This is overlap of observed failures, not proof that their root causes or complete test outcomes are identical.

The remaining L5 case is test_generation_matches_golden[text-generation/granitemoe-1b]. It failed on the PR head, but neither inspected base L5 job reached it before cancellation. Its baseline disposition is therefore unresolved; absence from the base failure list is not a base pass.

The pinned SmolLM FP16 CPU ORT GenAI job was skipped by the PR workflow’s existing configuration: the caller passes run_real: false, and the real lane requires that input or a scheduled/manual event. This is not evidence that Qwen or the pinned released runtime passed that job.

The filtered logs, annotations, exact-base comparisons, and unresolved case coverage are retained in genai-integration/ as ci-disposition-before-mmmu200.json, ci-golden-direct-api-log-proof.json, ci-golden-node-comparison.json, and ci-granitemoe-l5-disposition.json. This historical disposition must not be presented as the status of a later documentation/publication head. Any merge acceptance of incomplete qualification requires an explicit maintainer decision; this evidence does not declare the PR merge-ready.

Reproducing graph checks

Run from the repository root with Transformers and CUDA ORT available:

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 NVIDIA_TF32_OVERRIDE=0 \
  python -m pytest src/mobius/models/qwen38_standard_test.py -q -rs

The tests separate these confidence levels:

  1. Config compatibility: assert the pinned full text config’s layer schedule, dimensions, interleaved mRoPE sections, and FP32 state policy.

  2. Full-size graph construction: build the 64-layer decoder and the complete 64+27-layer VLM in FP32 and FP16, without allocating/loading checkpoint weights; recursively verify all components’ standard-only nodes and the 48 recurrent/16 KV cache groups.

  3. Reduced random-weight numerical parity: compare an independently constructed HF Qwen3_5ForCausalLM with the actual exported/saved/reloaded ONNX package on CUDA FP32 and FP16. Cover single-token and six-token prefill, batch sizes one and two, then four greedy decode steps. Compare full logits, exact next-token IDs, and every convolution/recurrent/KV output, checking shapes and dtypes as well as values.

  4. Reduced VLM numerical parity: run actual saved/reloaded ONNX vision outputs through the embedding graph and hybrid decoder. Cover text, multiple images, video, and mixed image/video prompts in FP32 and FP16, including two-row media batches with reversed placeholder ordering. Compare vision features, fused embeddings, full logits, exact greedy IDs, and all carried states over prefill and four decode steps.

The focused suite passed 34 tests with no skips in the qualification environment. The related offline regression selection passed 102 tests with 16 existing skips. This is a different selection from the later 102-test affected-component run below; their counts are not additive. The VLM patches are synthetic, nonzero packed processor-shaped inputs; HF supplies explicit MRoPE positions. This is not independent qualification of processor preprocessing or host position-ID generation.

The reduced model retains one complete 3-DeltaNet/1-attention cycle, the original full-attention Q/KV head ratio (6:1), the DeltaNet value/key head ratio (3:1), convolution kernel 4, partial rotary factor 0.25, and interleaved mRoPE. Widths, vocabulary, context length, and mRoPE sections are reduced; weights are synthetic. This is L3 evidence, not L4/L5 checkpoint evidence.

Numerical tolerances are rtol=atol=1e-3 for FP32 and 1e-2 for FP16. CUDA profiling must contain actual CUDA matrix execution; ordinary CPU metadata/control-flow handling remains allowed. Provider registration alone is not considered proof of GPU computation. CUDA tests explicitly skip when CUDA ORT/PyTorch is unavailable; such skips are not successful CUDA evidence.

The test that inserts a custom op inside a Scan body ensures the recursive standard-only check is exercised, rather than merely checking top-level nodes. This is a verification assertion, not a claim that main has a generic fail-closed standard-only production validator.

Remaining qualification

The official 18-shard checkpoint has been downloaded at the pinned revision (55,562,855,904 tensor bytes). The complete FP16 VLM package, including the learned-norm correction, was exported and reloaded successfully: 54,779,987,399 bytes, with all three components recursively standard-only and no remaining local functions. CUDA sessions for all three components loaded successfully. This establishes a real-weight artifact, not numerical qualification.

The first export timed out while writing external tensor data. A production-matrix probe traced slow writes to non-C-contiguous folded weights and verified byte-identical output after storage conversion. Operational export preparation copies only strided storage and verifies every bit, shape and dtype; it does not change graph operators or add a generic serializer change to Mobius.

Official FP16 runtime parity remains failed at the unchanged rtol=atol=1e-2, using Transformers 5.15.0 and ORT 1.30.0:

  • The image vision comparison exceeded tolerance at 90 of 327,680 values (0.0275%). The norm correction does not change the vision graph.

  • Text embeddings matched exactly. Corrected-source text prefill logits exceeded tolerance at 79,551 of 5,711,360 values (1.39%), with maximum absolute error 0.091796875. Validation stopped before greedy-token, state and reused-cache comparisons.

  • Visual-only diagnosis verified all 333 visual tensors and isolated rounding differences in FP16 projection-plus-bias and rotary arithmetic. Standard Gemm matched the isolated HF QKV projection exactly, unlike separate MatMul/Add; FP32 rotary arithmetic also matched locally. These are candidate remedies, not a completed full-vision fix.

After the norm correction, the combined affected component and reduced qualification selection passed 102 tests with no skips. Neither those passes nor the serialized standard-only audit clear the official failures. The two additional DeepStack sentinel cases also passed.

Before declaring the official model production-qualified, compare real-weight vision/fusion, prefill and multiple reused-state decode steps, full logits and state, and multi-token generation. Native ORT GenAI driver execution and AI2D first-200 evaluation have completed, as documented above; this bounded execution/task evidence does not clear the official FP16 numerical parity failures. Full-dataset AI2D accuracy and real-processor video/mixed-media remain unqualified. Full-size runtime memory and throughput require deployment-scale qualification beyond graph build, reduced-model correctness, and the recorded bounded runs. Repeated large-image inference still fails with OOM, as documented below. ModelBuilder equivalence remains deferred.

In particular, mixed FP16 convolution/FP32 recurrent state must be qualified against the intended GenAI runtime separately. Quantized text-only component plans and their module exclusions are outside this unquantized qualification. BF16, MTP, and performance qualification are also outside the current evidence.

Input-size and repeated-inference findings (October 11, 2026)

The completed native AI2D200 result remains 174/200 (87%), compared with frozen HF 175/200 (87.5%) and raw ORT 174/200 (87%). Native/HF tokens matched 199/200 and native/raw tokens 198/200. AI2D used one process and one Model for all 200 questions, with a fresh Generator per question; it was not a per-question process-isolation run.

Observed native AI2D200 input

Median

Maximum

Prompt tokens, including image tokens

314.5

1,705

Image patch rows

1,064

6,580

Generation was capped at 16 tokens. These are bounds of the tested subset, not dataset-wide maxima or a validated deployment-safe limit.

A separate large-image memory diagnostic used 4,875 prompt tokens, 19,200 patch rows, and a 128-token cap. The first call completed; a second identical call on the same Model with a new Generator failed at decoder layer 3 Attention_node_335, requesting 4,563,000,320 bytes. After the first call, GPU usage was 81,133 MiB with only 20 MiB free; deleting the Generator and GC did not reduce usage while the Model stayed alive. Process exit restored 81,150 MiB free / 4 MiB used. The first call exactly matched the standalone tokens, text, processor tensors, and prompt.

This diagnostic used an isolated GenAI build, source 265917c09ffe68198a026e086d21e694cc938298, with a one-line last-token logits allocation correction and package-local $ORIGIN RPATH; actual loaded installed ORT SHA256 remained 5a2e81ad05259bce596e29d6361fe76750b6c2c9569b8cdddf709ab0d944b0ee. Graphs, weights, images, and generation policy were unchanged. It is not the installed GenAI binary used for AI2D200; the installed GenAI source origin is unknown. The external GenAI correction is not included in this Mobius PR and does not resolve repeated-inference OOM.

The decoder export has 16 standard Attention nodes, explicit FP16 masks, dynamic KV ports, and no TensorScatter. The inspected ORT unfused workspace formula exactly matches the allocation: 2 * align256(24 * 4875 * 4875 * 4) = 4,563,000,320. The local dispatch excludes Flash with an explicit mask and excludes MEA for a masked KV stride not divisible by four (4875 % 4 = 3). This strongly supports an unfused-workspace attribution, not a directly captured installed-binary dispatch trace; the inspected ORT source-to-binary correspondence remains unverified. Other large-image attempts also failed in the vision encoder.

The repeated-inference memory blocker is reproducible, but live tensor ownership, arena retention, and fragmentation are not yet distinguished. A KV-cache leak is not proven, and static TensorScatter cache has not been A/B tested. Deployment requires validated processor-output limits, generation/concurrency budgets, and repeated mixed-size request testing, rather than assuming a successful isolated inference or AI2D subset qualifies serving stability.

Per the narrowed publication scope, no MMMU accuracy result is claimed. Existing official FP16 numerical parity failures, native/HF resize differences, and incomplete historical L4/L5 CI remain open; this update does not declare merge/deployment readiness. Detailed evidence and historical CI disposition are recorded in docs/design/qwen38-27b-standard-onnx.md. Qualification-disk receipts include genai-runtime-fix-01/probes/candidate-case56-repeat01/{receipt,execution}.json and the unchanged AI2D200 artifacts.