Phi4MM ORT-GenAI Integration Spec¶

Created: 2026-03-18 · Last updated: 2026-03-18

This document consolidates the ORT-GenAI integration plan for Phi4MM: the official reference config, I/O contract comparison, GenaiConfigGenerator extension design, target configs, and implementation plan.

1. Official Reference Config¶

Source: https://huggingface.co/microsoft/Phi-4-multimodal-instruct-onnx/tree/main/gpu/gpu-int4-rtn-block-32

The official ONNX export uses a 4-model split:

{
    "model": {
        "type": "phi4mm",
        "decoder": {
            "filename": "phi-4-mm-text.onnx",
            "inputs": {
                "inputs_embeds": "inputs_embeds",
                "attention_mask": "attention_mask",
                "past_key_names": "past_key_values.%d.key",
                "past_value_names": "past_key_values.%d.value"
            },
            "outputs": {
                "logits": "logits",
                "present_key_names": "present.%d.key",
                "present_value_names": "present.%d.value"
            }
        },
        "vision": {
            "filename": "phi-4-mm-vision.onnx",
            "config_filename": "vision_processor.json",
            "adapter_filename": "phi-4-mm-vision.onnx_adapter",
            "inputs": {
                "pixel_values": "pixel_values",
                "attention_mask": "image_attention_mask",
                "image_sizes": "image_sizes"
            },
            "outputs": {
                "image_features": "image_features"
            }
        },
        "speech": {
            "filename": "phi-4-mm-speech.onnx",
            "config_filename": "speech_processor.json",
            "adapter_filename": "phi-4-mm-speech.onnx_adapter",
            "inputs": {
                "audio_embeds": "audio_embeds",
                "attention_mask": "audio_attention_mask",
                "audio_sizes": "audio_sizes",
                "audio_projection_mode": "audio_projection_mode"
            },
            "outputs": {
                "audio_features": "audio_features"
            }
        },
        "embedding": {
            "filename": "phi-4-mm-embedding.onnx",
            "inputs": {
                "input_ids": "input_ids",
                "image_features": "image_features",
                "audio_features": "audio_features"
            },
            "outputs": {
                "inputs_embeds": "inputs_embeds"
            }
        }
    }
}

Key Observations¶

Speech model has audio_projection_mode input — integer selector between speech and vision projection branches. More efficient than dual outputs. Values likely: 0 = speech-only, 1 = combined vision+audio.
Vision model has additional inputs — image_attention_mask and image_sizes for HD dynamic resolution handling.
Speech model input is audio_embeds (not audio_features), with audio_sizes and audio_attention_mask inputs.
LoRA adapters are separate files — phi-4-mm-vision.onnx_adapter and phi-4-mm-speech.onnx_adapter. Our design bakes LoRA into the decoder as a deliberate simplification.
Decoder uses inputs_embeds (not input_ids), confirming the 4-model split where the embedding model handles token lookup.
No position_ids in decoder config — may be computed internally.

Processor Configs (Official)¶

vision_processor.json: DecodeImage → Phi4VisionDynamicPreprocess (dynamic_hd=36, base=448) → Rescale → Normalize(mean=0.5, std=0.5) → Phi4VisionProcessor
speech_processor.json: AudioDecoderEx (8kHz + 16kHz) → Phi4AudioEmbed (audio_compression_rate=8, n_mel=80, various STFT params)

2. I/O Contract Comparison¶

Official vs Our Implementation¶

Decoder¶

Dir	Official	Ours	Status
In	`inputs_embeds`	`inputs_embeds`	✅ Match
In	`attention_mask`	`attention_mask`	✅ Match
In	—	`position_ids [B, S]`	⚠️ We add this (codebase convention)
In	`past_key_values.%d.key/value`	`past_key_values.{i}.key/value`	✅ Match
Out	`logits`	`logits`	✅ Match
Out	`present.%d.key/value`	`present.{i}.key/value`	✅ Match

Vision¶

Dir	Official	Ours	Status
In	`pixel_values`	`pixel_values`	✅ Match
In	`image_attention_mask`	—	❌ Missing
In	`image_sizes`	—	❌ Missing
Out	`image_features`	`image_features`	✅ Match

Speech¶

Dir	Official	Ours	Status
In	`audio_embeds`	`audio_features`	⚠️ Name differs
In	`audio_attention_mask`	—	❌ Missing
In	`audio_sizes`	—	❌ Missing
In	`audio_projection_mode`	—	❌ We use dual outputs
Out	`audio_features` (single)	`speech_features` + `speech_features_for_vision`	⚠️ Dual outputs

Embedding¶

Dir	Official	Ours	Status
In	`input_ids`	`input_ids`	✅ Match
In	`image_features`	`image_features`	✅ Match
In	`audio_features`	`speech_features`	⚠️ Name differs
Out	`inputs_embeds`	`inputs_embeds`	✅ Match

Codebase Comparison: How Other Models Handle Similar Inputs¶

Model	Vision Extra Inputs	Audio Extra Inputs	Decoder position_ids
Gemma3	None	N/A	2D `[B, S]`
Qwen2.5-VL	`image_grid_thw [N, 3]`	N/A	3D `[3, B, S]` MRoPE
Qwen3-ASR	N/A	None	3D `[3, B, S]` MRoPE
Official Phi4MM	`image_attention_mask`, `image_sizes`	`audio_attention_mask`, `audio_sizes`, `audio_projection_mode`	None
Our Phi4MM	None	None	2D `[B, S]`

Gap Analysis¶

GAP 1: image_sizes / image_attention_mask on Vision Needed for HD dynamic crop with variable sub-images per input. Our vision forward only takes pixel_values. Works for fixed-size test inputs. Decision: DEFER to follow-up PR. Document as known limitation.

GAP 2: audio_sizes / audio_attention_mask on Speech Needed for variable-length audio with batched sequences. Decision: DEFER to follow-up PR. Same rationale.

GAP 3: audio_projection_mode vs dual outputs Official uses integer input to select branch (single output). Ours runs both branches (two outputs). Dual outputs are simpler (no conditional ops) but slightly wasteful. Decision: Keep dual outputs. Switch to mode input if ORT-GenAI requires.

GAP 4: position_ids on decoder Official has no position_ids. All other models in our codebase use explicit position_ids. Decision: KEEP position_ids. The genai_config handles the mapping.

GAP 5: Input naming differences Cosmetic differences (audio_features vs audio_embeds, speech_features vs audio_features). Handled by genai_config input/output name mapping. Decision: Keep our names for codebase consistency.

Intentional Divergences Summary¶

Aspect	Official	Ours	Reason
`position_ids`	Not in decoder	Explicit input	Codebase convention
Projection branch	`audio_projection_mode` input	Dual outputs	Simpler graph
LoRA	Separate adapter files	Baked into decoder	Single file per model
HD inputs	`image_sizes`, masks	Not yet	Deferred
Audio metadata	`audio_sizes`, masks	Not yet	Deferred

3. GenaiConfigGenerator Extension¶

Problem Statement¶

GenaiConfigGenerator supports LLM (decoder-only) and VLM (vision + embedding + decoder via with_vision()). Phi4MM requires a 4-model multimodal config with vision, speech, embedding, and decoder sections.

Design: `with_speech()` + extended `with_vision()`¶

Follow the same builder pattern as with_vision(). Keep the generator model-agnostic so future audio/multimodal models reuse it.

New `with_speech()` method¶

def with_speech(
    self,
    *,
    audio_token_id: int | None = None,
    filename: str = "speech/model.onnx",
    config_filename: str = "speech_processor.json",
    input_names: dict[str, str] | None = None,
    output_names: dict[str, str] | None = None,
) -> GenaiConfigGenerator:
    """Add speech/audio model section for multimodal models."""
    if input_names is None:
        input_names = {
            "audio_embeds": "audio_embeds",
            "attention_mask": "audio_attention_mask",
            "audio_sizes": "audio_sizes",
            "audio_projection_mode": "audio_projection_mode",
        }
    if output_names is None:
        output_names = {"audio_features": "audio_features"}

    self._speech = {
        "filename": filename,
        "config_filename": config_filename,
        "inputs": input_names,
        "outputs": output_names,
    }
    if audio_token_id is not None:
        self._vlm_token_ids["audio_token_id"] = audio_token_id
    return self

Extended `with_vision()` — optional overrides¶

Add optional input_names/output_names/config_filename parameters. If not provided, existing defaults apply. Zero breakage for current callers.

def with_vision(
    self,
    *,
    image_token_id: int,
    filename: str = "vision/model.onnx",
    embedding_filename: str = "embedding/model.onnx",
    spatial_merge_size: int = 2,
    config_filename: str = "processor_config.json",
    input_names: dict[str, str] | None = None,   # NEW
    output_names: dict[str, str] | None = None,   # NEW
    embedding_input_names: dict[str, str] | None = None,  # NEW
    embedding_output_names: dict[str, str] | None = None,  # NEW
) -> GenaiConfigGenerator:

Updated `generate()` method¶

def generate(self) -> dict[str, Any]:
    # ... existing code ...
    if self._vision is not None:
        model["vision"] = self._vision
    if self._embedding is not None:
        model["embedding"] = self._embedding
    if self._speech is not None:
        model["speech"] = self._speech
    model.update(self._vlm_token_ids)
    return {"model": model, "search": _default_search_params()}

Embedding construction for multimodal¶

Defer embedding construction to generate() based on enabled modalities:

if self._vision is not None or self._speech is not None:
    emb_inputs = {"input_ids": "input_ids"}
    if self._vision is not None:
        emb_inputs["image_features"] = "image_features"
    if self._speech is not None:
        emb_inputs["audio_features"] = "audio_features"
    model["embedding"] = {
        "filename": self._embedding_filename or "embedding/model.onnx",
        "inputs": emb_inputs,
        "outputs": {"inputs_embeds": "inputs_embeds"},
    }

Decoder inputs for multimodal¶

Multimodal decoders use inputs_embeds (like VLMs):

if is_vlm or is_multimodal:
    inputs["inputs_embeds"] = "inputs_embeds"
else:
    inputs["input_ids"] = "input_ids"

Differences from Existing VLM Pattern¶

Aspect	VLM (Qwen2.5-VL)	Phi4MM Multimodal
Model sections	decoder, vision, embedding	decoder, vision, speech, embedding
Vision inputs	pixel_values, image_grid_thw	pixel_values, image_attention_mask, image_sizes
Vision extras	spatial_merge_size	config_filename: vision_processor.json
Speech section	N/A	filename, config_filename, inputs (4), outputs (1)
Embedding inputs	input_ids, image_features	input_ids, image_features, audio_features
Processor configs	processor_config.json (unified)	vision_processor.json + speech_processor.json
search.past_present_share_buffer	false	true

Backward compatibility: Zero breaking changes. All new parameters are optional with defaults matching current behavior.

Detection and Trigger Logic (auto_export.py)¶

# Current:
is_vlm = "vision" in pkg and "embedding" in pkg

# Extended:
is_vlm = "vision" in pkg and "embedding" in pkg
is_multimodal = is_vlm and "speech" in pkg

# Add to _ORT_GENAI_MODEL_TYPE:
"phi4mm": "phi4mm",
"phi4_multimodal": "phi4mm",

4. Target genai_config.json¶

The complete target config for Phi4MM, mapping between ORT-GenAI expected names (left side) and ONNX tensor names (right side):

{
    "model": {
        "bos_token_id": 199999,
        "context_length": 131072,
        "decoder": {
            "session_options": {
                "log_id": "onnxruntime-genai",
                "provider_options": []
            },
            "filename": "model/model.onnx",
            "head_size": 128,
            "hidden_size": 3072,
            "inputs": {
                "inputs_embeds": "inputs_embeds",
                "attention_mask": "attention_mask",
                "past_key_names": "past_key_values.%d.key",
                "past_value_names": "past_key_values.%d.value"
            },
            "outputs": {
                "logits": "logits",
                "present_key_names": "present.%d.key",
                "present_value_names": "present.%d.value"
            },
            "num_attention_heads": 24,
            "num_hidden_layers": 32,
            "num_key_value_heads": 8
        },
        "vision": {
            "filename": "vision/model.onnx",
            "config_filename": "vision_processor.json",
            "inputs": {
                "pixel_values": "pixel_values",
                "attention_mask": "image_attention_mask",
                "image_sizes": "image_sizes"
            },
            "outputs": {
                "image_features": "image_features"
            }
        },
        "speech": {
            "filename": "speech/model.onnx",
            "config_filename": "speech_processor.json",
            "inputs": {
                "audio_embeds": "audio_embeds",
                "attention_mask": "audio_attention_mask",
                "audio_sizes": "audio_sizes",
                "audio_projection_mode": "audio_projection_mode"
            },
            "outputs": {
                "audio_features": "audio_features"
            }
        },
        "embedding": {
            "filename": "embedding/model.onnx",
            "inputs": {
                "input_ids": "input_ids",
                "image_features": "image_features",
                "audio_features": "audio_features"
            },
            "outputs": {
                "inputs_embeds": "inputs_embeds"
            }
        },
        "eos_token_id": [200020, 199999],
        "pad_token_id": 199999,
        "type": "phi4mm",
        "vocab_size": 200064
    },
    "search": {
        "diversity_penalty": 0.0,
        "do_sample": false,
        "early_stopping": true,
        "length_penalty": 1.0,
        "max_length": 131072,
        "min_length": 0,
        "no_repeat_ngram_size": 0,
        "num_beams": 1,
        "num_return_sequences": 1,
        "past_present_share_buffer": true,
        "repetition_penalty": 1.0,
        "temperature": 1.0,
        "top_k": 1,
        "top_p": 1.0
    }
}

5. Processor Configs¶

vision_processor.json¶

{
    "processor": {
        "name": "phi_4_vision_processor",
        "transforms": [
            {
                "operation": {
                    "name": "decode_image",
                    "type": "DecodeImage",
                    "attrs": { "color_space": "RGB" }
                }
            },
            {
                "operation": {
                    "name": "phi4_vision_dynamic_preprocess",
                    "type": "Phi4VisionDynamicPreprocess",
                    "attrs": {
                        "dynamic_hd": 36,
                        "dyhd_base_resolution": 448
                    }
                }
            },
            {
                "operation": {
                    "name": "rescale",
                    "type": "Rescale",
                    "inputs": [":0"]
                }
            },
            {
                "operation": {
                    "name": "normalize",
                    "type": "Normalize",
                    "attrs": {
                        "mean": [0.5, 0.5, 0.5],
                        "std": [0.5, 0.5, 0.5]
                    }
                }
            },
            {
                "operation": {
                    "name": "phi4_vision_processor",
                    "type": "Phi4VisionProcessor",
                    "inputs": [":0", "phi4_vision_dynamic_preprocess:1"],
                    "attrs": {
                        "dyhd_base_resolution": 448,
                        "interpolation": "CUBIC"
                    }
                }
            }
        ],
        "output_aligner": "phi4-vision-aligner"
    }
}

Key parameters: dynamic_hd: 36 (max HD crops), dyhd_base_resolution: 448, normalization mean/std=[0.5, 0.5, 0.5] (SigLIP standard).

speech_processor.json¶

{
    "feature_extraction": {
        "sequence": [
            {
                "operation": {
                    "name": "audio_decoder",
                    "type": "AudioDecoderEx",
                    "attrs": {
                        "target_sample_rates": [8000, 16000]
                    }
                }
            },
            {
                "operation": {
                    "name": "phi_4_audio_embed",
                    "type": "Phi4AudioEmbed",
                    "attrs": {
                        "audio_compression_rate": 8,
                        "stft_normal/n_fft": 512,
                        "stft_normal/frame_length": 400,
                        "stft_normal/hop_length": 160,
                        "stft_normal/win_fn": "hamming",
                        "logmel/chunk_size": 30,
                        "logmel/hop_length": 160,
                        "logmel/n_fft": 512,
                        "logmel/n_mel": 80,
                        "logmel/feature_first": 0,
                        "logmel/no_padding": 1,
                        "stft_normal_8k/n_fft": 256,
                        "stft_normal_8k/frame_length": 200,
                        "stft_normal_8k/hop_length": 80,
                        "stft_normal_8k/win_fn": "hamming",
                        "logmel_8k/chunk_size": 30,
                        "logmel_8k/hop_length": 80,
                        "logmel_8k/n_fft": 512,
                        "logmel_8k/n_mel": 80,
                        "logmel_8k/feature_first": 0,
                        "logmel_8k/no_padding": 1
                    }
                }
            }
        ],
        "output_aligner": "phi4-audio-aligner"
    }
}

Key parameters: audio_compression_rate: 8, n_mel: 80, supports 8kHz and 16kHz sample rates, hamming window STFT.

Generation approach: Static templates in examples (recommended). Auto-generation via auto_export.py is a follow-up.

6. Implementation Plan¶

Phase 1: Example with static config (this PR)¶

Static genai_config dict in phi4mm_ort_genai.py example
Static _write_vision_processor_config() in example
Static _write_speech_processor_config() in example

Phase 2: GenaiConfigGenerator extension (follow-up)¶

genai_config.py changes:

Add self._speech: dict | None = None to __init__()
Add with_speech() method
Extend with_vision() with optional I/O name overrides
Update generate() to include speech section
Handle embedding construction with audio_features

genai_config_test.py changes:

Test with_speech() produces speech section
Test with_vision().with_speech() chaining produces all 4 sections
Test embedding inputs include audio_features when speech enabled
Test backward compat: existing VLM tests still pass
Test Phi4MM config matches official reference

Phase 3: auto_export.py integration (follow-up)¶

Add phi4mm / phi4_multimodal to _ORT_GENAI_MODEL_TYPE
Extend multimodal detection
Call with_speech() when speech model detected in package
Extend _write_processor_config() for Phi4MM

Deferred Items¶

I/O name alignment with official export — audio_embeds vs audio_features, missing metadata inputs, position_ids in decoder
LoRA adapter files — official uses separate .onnx_adapter files; our approach bakes LoRA in
past_present_share_buffer: true — official uses this for KV cache efficiency; our default is false
Additional search params — diversity_penalty, length_penalty, no_repeat_ngram_size (all at no-op values, safe to omit)