ONNX GenAI Models - Test Infrastructure Comprehensive Map

Created: · Last updated:

Last Updated: 2025-03-09

EXECUTIVE SUMMARY

Aspect

Count

Location

Test Files

37

/tests/ (18) + src/**/*_test.py (19+)

Test Classes

248

33 in /tests/, 204 in src/

Test Methods

237

in /tests/ (79) + src/ (1003)

Test Fixtures

1

conftest.py (deterministic_seed)

Pytest Markers

7

integration, integration_fast, integration_slow, arch_validation, golden, generation, benchmark

CI Workflows

7

main, gpu_tests, nightly_l2, golden_regen, arch_diff, benchmark, pages

Test Dependencies

10

pytest, pytest-xdist, pytest-timeout, onnxruntime-easy, transformers, safetensors, gguf, Pillow, librosa, onnxruntime-genai

1. TEST DIRECTORIES & FILES

1.1 /tests/ Directory Structure

Main Test Files (18 total):

File

Classes

Methods

Purpose

Markers

build_graph_test.py

33

100

L1 Unit Tests: Build ONNX graphs for all supported architectures without weights or network access

parametrize

arch_validation_test.py

1

4

L2 Architecture Validation: Downloads config.json from HF for registered models, builds full-size ONNX graphs (no weights), validates structure

arch_validation, parametrize

synthetic_parity_test.py

0

1

L3 Synthetic Parity: Builds tiny random-weight models (PyTorch + ONNX), compares logits to detect op-level bugs

parametrize

e2e_golden_test.py

2

2

L4/L5 Golden Comparison: Data-driven tests from YAML files in testdata/cases/; L4 compares single-forward logits, L5 compares generation

golden, generation, integration, parametrize

integration_test.py

15

34

Integration Tests: Downloads real model weights, compares logits and greedy generation vs HF PyTorch reference

integration, integration_fast, integration_slow, parametrize

weight_alignment_test.py

5

5

Weight Alignment Tests: Verifies preprocess_weights() doesn’t corrupt ONNX names; catches renaming bugs

parametrize

onnx_checker_test.py

1

1

ONNX Checker Tests: Runs CheckerPass on all built models; detects op errors, malformed protos (~30s)

parametrize

cli_test.py

3

8

CLI Tests: Invokes main() directly; no network access, all build tests use --no-weights

None

seq2seq_integration_test.py

8

9

Seq2Seq Integration: T5/BART encoder-decoder vs HF PyTorch

integration, parametrize

vision_integration_test.py

2

2

Vision Integration: ViT/CLIP with random weights vs PyTorch

integration, integration_fast

multimodal_integration_test.py

1

2

Multimodal Integration: Vision-language models (Gemma3, etc.)

integration, integration_slow

moe_integration_test.py

2

3

MoE Integration: Deprecated (superseded by integration_test.py); MoE models prefill/decode

integration, integration_fast

whisper_integration_test.py

3

4

Whisper Integration: Audio encoder-decoder vs HF

integration, integration_fast

ort_genai_test.py

2

3

ONNX Runtime GenAI: End-to-end with onnxruntime-genai inference

integration, integration_slow, parametrize

mamba2_integration_test.py

1

2

Mamba2 Integration: SSM state carry, single-token decode

integration

phi4mm_integration_test.py

1

4

Phi4-MM Integration: 4-model pipeline (vision, speech, embedding, decoder) with text+image+audio

integration, integration_slow

quantization_integration_test.py

2

6

Quantization Integration: Tiny quantized Llama, synthetic weights, MatMulNBits execution

integration

benchmark_build.py

1

1

Build Performance Benchmark: Measures task.build() time and memory for key architectures

benchmark, parametrize

Support Files:

  • conftest.py (141 lines) — Shared fixtures, marker registration, CLI flags, test filtering

  • _test_configs.py (38KB) — Config data for parametrized tests

1.2 src/ Test Files (54 total)

Test Coverage by Module:

Module

Classes

Methods

Key Test Files

Core Config

27

189

_configs_test.py (62M), _config_resolver_test.py (40M), _registry_test.py (27M)

Building & Export

22

113

_exporter_test.py (32M), _model_package_test.py (29M), _diffusers_builder_test.py (21M)

Components

67

379

23 test files (rotary_embedding 28M, quantized_linear 22M, audio 16M, attention 17M, vision 16M, whisper 20M, etc.)

Weight Handling

17

114

_weight_utils_test.py (58M), _weight_loading_test.py (27M), _graph_diff_test.py (24M)

GGUF Integration

19

91

_builder_test.py, _reader_test.py (26M), _repacker_test.py (30M), _tensor_mapping_test.py (18M)

Testing Utilities

9

55

golden_test.py (28M), parity_test.py (11M), code_paths_test.py (16M)

Tasks Framework

15

81

_task_test.py (15 classes, 81 methods — core model building logic)

Other

27

61

ORT GenAI auto_export_test (integration marker), rewrite rules (bias/gelu/layer_norm fusion, etc.)

Total: 204 test classes, 1003 test methods in src/

2. CONFTEST.PY FILES

/tests/conftest.py (141 lines)

Marker Registrations (via pytest_configure):

markers = [
    "golden: L4/L5 golden reference comparison tests",
    "generation: L5 end-to-end generation tests",
    "integration: tests that require real model weights (network)",
    "integration_fast: fast integration tests with small models",
    "arch_validation: L2 architecture validation tests",
]

CLI Flags (via pytest_addoption):

  • --fast: Skip non-representative model configs (uses FAST_*_CONFIGS)

  • --models <comma-separated-list>: Filter by model_type parameter

Collection Hook (pytest_collection_modifyitems):

  • Filters parametrized tests based on --fast and --models flags

  • Respects model_type parameter in test callspecs

  • Always runs non-parametrized tests

Fixtures:

@pytest.fixture()
def deterministic_seed():
    """Set numpy and python random seeds for reproducibility."""
    seed = 42
    random.seed(seed)
    np.random.seed(seed)
    return seed

Note: No conftest.py files found in src/ directory (tests are colocated with modules).

3. TEST UTILITIES

/src/mobius/_testing/ (12 files)

Exported Classes & Functions:

Module

Key Classes/Functions

__init__.py

Public API: GoldenRef, TestCase, Tolerances, discover_test_cases(), load_golden_ref(), save_golden_ref(), load_test_case(), load_tolerances(), compare_golden(), compare_synthetic(), detect_code_paths(), CODE_PATH_INDICATORS, etc.

golden.py

GoldenRef — stores L4/L5 reference outputs; TestCase — YAML test case (model_id, task, seed); Tolerances — L4/L5 comparison thresholds; discover_test_cases(), load_golden_ref(), save_golden_ref(), load_test_case(), load_tolerances(), golden_path_for_case(), has_golden()

parity.py

ParityReport, ParityResult — numerical comparison results; compare_golden() — L4 logit comparison; compare_synthetic() — L3 synthetic parity check

code_paths.py

CodePathIndicator — config→code-path mapping; detect_code_paths(), detect_code_paths_from_config(), get_all_code_path_labels() — implicit code path detection from config values

generation.py

OnnxGenerator — greedy autoregressive text generation with KV cache; generate() method

ort_inference.py

OnnxModelSession — ONNX Runtime inference wrapper

torch_reference.py

TorchReference — PyTorch reference model wrapper for parity testing

comparison.py

compare_onnx_and_torch() — numerical comparison utilities

code_paths_test.py

Tests for code-path detection (2 classes, 16 methods)

golden_test.py

Tests for golden-file loading/saving (5 classes, 28 methods)

parity_test.py

Tests for parity utilities (2 classes, 11 methods)

Test Infrastructure Design:

  • Golden files stored in testdata/golden/ (YAML + JSON)

  • Test cases in testdata/cases/ (YAML definitions)

  • Tolerances in testdata/default_tolerances.yaml (L4/L5 thresholds)

  • Data-driven approach: Adding test coverage = adding a YAML file (no code changes needed)

4. CI CONFIGURATION & WORKFLOWS

.github/workflows/ (7 files)

File

Trigger

Level

Test Command

Purpose

main.yml

PR + push to main

L1–L3, selective L4

pytest tests/build_graph_test.py -n auto (L1); pytest tests/synthetic_parity_test.py -n auto (L3); pytest -m 'not integration' (coverage); selective L4 + L5

Fast PR feedback; runs on every push

gpu_tests.yml

push to main + weekly (Mon 4am UTC)

L4–L5

pytest tests/e2e_golden_test.py -m golden (L4); pytest tests/e2e_golden_test.py -m generation (L5)

GPU-only golden/generation tests (requires checkpoint verification)

nightly_l2.yml

nightly (3am UTC) + l2-* tags

L2

pytest tests/arch_validation_test.py -m arch_validation

Architecture validation with real HF configs

golden_regen.yml

weekly (Sun 2am UTC) + manual

—

Regenerates golden files

Updates L4/L5 reference data (before nightly_l2)

arch_diff.yml

Manual dispatch

—

Detects architecture differences

Custom diff detection (details TBD)

benchmark.yml

Manual dispatch + scheduled

—

Benchmark reporting

Performance tracking

pages.yml

Push to main

—

Docs build

GitHub Pages deployment

CI Test Levels:

L1 (Smoke)           → build_graph_test.py            (fast, no weights, no network)
L2 (Architecture)    → arch_validation_test.py        (real HF configs, no weights, nightly)
L3 (Synthetic)       → synthetic_parity_test.py       (tiny random weights, detects op bugs)
L4 (Golden Logits)   → e2e_golden_test.py -m golden   (checkpoint-verified single-forward)
L5 (Generation)      → e2e_golden_test.py -m generation (checkpoint-verified generation)

Caching Strategy:

  • pip-smoke-*, pip-parity-*, pip-gpu-*, pip-nightly-l2-* — pip package cache

  • hf-gpu-*, hf-l2-configs-* — HuggingFace model/config cache

5. PYTEST CONFIGURATION

pyproject.toml [tool.pytest.ini_options]

[tool.pytest.ini_options]
addopts = "--tb=short --color=yes"
testpaths = ["src", "tests"]
python_files = ["*_test.py"]
markers = [
    "integration: tests requiring model downloads and GPU/CPU inference",
    "integration_fast: fast integration tests (text, encoder, audio)",
    "integration_slow: slow integration tests (vision-language, multimodal)",
    "benchmark: graph construction performance benchmarks",
    "arch_validation: L2 architecture validation (real HF configs, no weights)",
    "golden: L4 checkpoint-verified golden comparison tests",
    "generation: L5 end-to-end golden generation tests",
]

pytest Options:

  • --tb=short — Short traceback format

  • --color=yes — Colorized output

  • testpaths — Test discovery paths: src/**/*_test.py + tests/**/*_test.py

  • python_files — Only files matching *_test.py

6. TEST DEPENDENCIES

pyproject.toml [project.optional-dependencies]

testing = [
    "onnxruntime-easy",           # CPU inference
    "torch",                       # PyTorch reference
    "transformers>=5.0",          # HuggingFace models
    "safetensors",                # Weight loading
    "onnxruntime-genai",          # GenAI runtime
    "pytest-xdist",               # Parallel test execution (-n auto)
    "pytest-timeout",             # Timeout management
    "gguf>=0.10.0",               # GGUF format
    "Pillow",                      # Image processing
    "librosa",                     # Audio processing
]

requirements/ci/requirements.txt

pytest
pytest-cov
pytest-xdist
coverage[toml]
gguf>=0.10.0

7. PYTEST MARKERS USAGE

Marker Hierarchy

Marker

Level

When to Use

Deselection

(none)

L0–L1

Unit tests, fast integration

Always run

@pytest.mark.parametrize

All

Model/config parametrization

Run as needed

@pytest.mark.arch_validation

L2

Architecture validation

-m "not arch_validation"

@pytest.mark.golden

L4

Golden logit comparison

-m "not golden"

@pytest.mark.generation

L5

Golden generation E2E

-m "not generation"

@pytest.mark.integration

L3–L5

Real weights required

-m "not integration"

@pytest.mark.integration_fast

L3

Small models (text, encoder, audio)

Subset of integration

@pytest.mark.integration_slow

L3

Large models (multimodal, VL)

Subset of integration

@pytest.mark.benchmark

—

Performance benchmarking

-m "not benchmark"

@pytest.mark.skip

—

Conditional skips

N/A

Marker Usage Counts (in /tests/*.py)

@pytest.mark.integration        56 usages
@pytest.mark.integration_fast   36 usages
@pytest.mark.integration_slow    9 usages
@pytest.mark.golden              1 usage (e2e_golden_test.py)
@pytest.mark.generation          1 usage (e2e_golden_test.py)
@pytest.mark.arch_validation     1 usage (arch_validation_test.py)
@pytest.mark.benchmark           1 usage (benchmark_build.py)
@pytest.mark.parametrize        Multiple (config/model parametrization)

8. FIXTURES

Fixture Definitions

/tests/conftest.py:

@pytest.fixture()
def deterministic_seed():
    """Set numpy and python random seeds for reproducibility."""
    seed = 42
    random.seed(seed)
    np.random.seed(seed)
    return seed

src/mobius/integrations/gguf/_builder_test.py:

@pytest.fixture
def [fixture_name]  # Implementation details TBD

src/mobius/integrations/gguf/_reader_test.py:

@pytest.fixture
def [fixture_name_1]  # Implementation details TBD

@pytest.fixture
def [fixture_name_2]  # Implementation details TBD

Total Fixture Count: 4 (1 in tests/conftest.py, 3 in src/gguf/)

9. TEST ORGANIZATION SUMMARY

Test Distribution by Type

Type

Count

Location

Purpose

Unit (L0/L1)

~100 methods

src/**/*_test.py, tests/build_graph_test.py

Fast, no network

Component

~379 methods

src/components/*_test.py

Individual operator validation

Integration (L3)

~52 methods

tests/*_integration_test.py, tests/synthetic_parity_test.py

Real weights, network required

Golden (L4/L5)

~2–10 methods

tests/e2e_golden_test.py, parametrized via YAML

Data-driven, checkpoint-verified

Architecture (L2)

4 methods

tests/arch_validation_test.py

Real HF configs, no weights

Infrastructure

~55 methods

src/*_testing*/*_test.py

Golden/parity utils testing

Special

~17 methods

CLI, benchmarks, weight alignment

Auxiliary validation

Execution Strategy

  1. PR CI (main.yml): L1 (smoke) + L3 (synthetic parity) + selective L4 (fast golden)

  2. Weekly GPU (gpu_tests.yml): L4 (golden) + L5 (generation) on GPU

  3. Nightly (nightly_l2.yml): L2 (architecture validation)

  4. Weekly Regen (golden_regen.yml): Update L4/L5 golden data

10. KEY TESTING PATTERNS

Pattern 1: Parametrized Model Testing

@pytest.mark.parametrize("model_type,config_overrides", _MODEL_PARAMS)
def test_build_model(model_type, config_overrides):
    # Runs for each model_type in _MODEL_PARAMS
    # Filter with --fast flag or --models option

Pattern 2: Integration Testing with Real Weights

@pytest.mark.integration
@pytest.mark.integration_fast
@pytest.mark.parametrize("model_id,trust_remote_code", _TEXT_MODELS)
def test_export_and_run(model_id, trust_remote_code):
    # Downloads weights, compares logits

Pattern 3: Golden-File-Based Testing (Data-Driven)

@pytest.mark.golden
@pytest.mark.parametrize("case", _L4_CASES)
def test_golden_logits(case):
    # case = TestCase(model_id, task, seed, ...)
    # Loads from testdata/golden/*.json

Pattern 4: Synthetic Parity Testing

def test_parity_causal_lm():
    # Build PyTorch + ONNX with same seed
    # Compare logits for op-level bugs

11. TEST DATA LOCATIONS

Path

Contents

Purpose

testdata/cases/

YAML test case definitions

L4/L5 test data (model_id, task, seed)

testdata/golden/

JSON golden reference outputs

L4/L5 expected results

testdata/default_tolerances.yaml

L4/L5 comparison thresholds

Numerical tolerance config

tests/perf_baseline.json

Baseline performance metrics

Benchmark regression guard

tests/_test_configs.py

Parametrization config data

FAST_*_CONFIGS, model lists, overrides

12. QUICK REFERENCE COMMANDS

# L1 Smoke (all architectures)
pytest tests/build_graph_test.py -v

# L1 Fast (representative only)
pytest tests/build_graph_test.py -v --fast

# L2 Architecture Validation
pytest tests/arch_validation_test.py -m arch_validation -v

# L3 Synthetic Parity
pytest tests/synthetic_parity_test.py -v --tb=short -n 0

# L4 Golden Logits
pytest tests/e2e_golden_test.py -m golden -v

# L5 Generation E2E
pytest tests/e2e_golden_test.py -m generation -v

# Integration (small models only)
pytest tests/integration_test.py -m integration_fast -v -k "smollm or albert"

# All non-integration tests with coverage
pytest -m 'not integration' --cov=src --cov-report=html

# Filter by model type
pytest tests/build_graph_test.py -v --models "qwen2,llama"

# Run specific test
pytest tests/build_graph_test.py::TestQwen2::test_qwen2 -v

13. DESIGN NOTES

Data-Driven Golden Testing

  • No hardcoded test code for L4/L5 — each YAML file = one test case

  • Tolerances centralized in testdata/default_tolerances.yaml

  • Golden data stored as JSON (single-forward logits, generation tokens)

  • Benefits: Easy to add new models without code changes, consistent thresholds

Parametrization Strategy

  • Config-based: Model parameters derived from _test_configs.py

  • Model-based: Real HF model IDs parametrized via @pytest.mark.parametrize

  • Filtering: --fast skips non-representative configs; --models filters by type

Test Levels & CI Integration

  • L1 (Smoke): Always runs, fastest feedback

  • L2 (Architecture): Nightly, validates real HF configs

  • L3 (Parity): Catches op bugs early, before L4/L5

  • L4/L5 (Golden): Weekly on GPU, checkpoint-verified

  • Pyramid approach: Fast tests first, expensive tests later

Fixture Philosophy

  • Minimal fixtures (just deterministic_seed)

  • Most setup inline or in helper functions (e.g., make_config())

  • Avoids fixture complexity; easier to understand test flow