Generator Contract Specification

Status: Canonical Reference
Scope: All generator files in generators/
Last Updated: {{ git_revision_date_localized }}
Created: {{ git_creation_date_localized }}

This document defines the contract for test case generator files. Generators enable stress testing, edge case discovery, and reproducible test generation.

File Structure
generate() Function
Generator Design Patterns
Complexity Estimation Generator
Input Format Specifications
JUDGE_FUNC Requirement
Running Generated Tests
Best Practices
Quick Reference

File Structure

Naming Convention

generators/{problem_id}_{slug}.py

Component	Format	Example
`problem_id`	4-digit zero-padded LeetCode ID	`0001`, `0004`, `0051`
`slug`	snake_case problem name	`two_sum`, `median_of_two_sorted_arrays`

Examples:

generators/0001_two_sum.py
generators/0004_median_of_two_sorted_arrays.py
generators/0051_n_queens.py

Required Elements

Every generator file MUST contain:

Element	Required	Description
`generate()` function	✅	Main entry point for test generation
Docstring with constraints	✅	LeetCode constraints documentation
Edge cases	✅	Known edge cases yielded first

Optional Elements

Element	Optional	Description
`generate_for_complexity()`	⭕	For time complexity estimation
Helper functions	⭕	Internal `_generate_case()` etc.
Custom generators	⭕	`generate_all_sizes()` etc.

generate() Function

Function Signature

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    """
    Generate random test case inputs.
    
    Args:
        count: Number of test cases to generate
        seed: Random seed for reproducibility (optional)
    
    Yields:
        str: Test input in the same format as .in files
    """

Contract Rules

Rule	Requirement	Rationale
Yield format	Must match `.in` file format	Runner passes to `solve()` via stdin
Reproducibility	Same seed → same output	Enables failure reproduction
Edge cases first	Yield known edge cases before random	Catch corner-case bugs early
Constraint compliance	Respect LeetCode constraints	Ensure valid test cases
JUDGE_FUNC required	Solution must have `JUDGE_FUNC`	No `.out` file for generated cases

📖 See JUDGE_FUNC Specification for validation details.

Minimal Example

# generators/0001_two_sum.py
import json
import random
from typing import Iterator, Optional

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    """Generate test cases for Two Sum."""
    if seed is not None:
        random.seed(seed)
    
    # Edge cases first (using canonical JSON format)
    edge_cases = [
        ([2, 7, 11, 15], 9),   # Classic example
        ([3, 2, 4], 6),        # Answer not first element
        ([3, 3], 6),           # Duplicate values
    ]
    
    for nums, target in edge_cases:
        yield f"{json.dumps(nums, separators=(',', ':'))}\n{target}"
        count -= 1
        if count <= 0:
            return
    
    # Random cases
    for _ in range(count):
        yield _generate_case()

def _generate_case() -> str:
    size = random.randint(2, 5000)
    nums = [random.randint(-10**6, 10**6) for _ in range(size)]
    i, j = random.sample(range(size), 2)
    target = nums[i] + nums[j]
    return f"{json.dumps(nums, separators=(',', ':'))}\n{target}"

Generator Design Patterns

Standard Template

# generators/{problem_id}_{slug}.py
"""
Test Case Generator for Problem {ID} - {Title}

LeetCode Constraints:
- {constraint_1}
- {constraint_2}
- ...

Time Complexity: O(?)
"""
import random
from typing import Iterator, Optional


# ============================================
# Random Test Generation (for functional testing)
# ============================================

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    """
    Generate random test case inputs.
    
    Args:
        count: Number of test cases to generate
        seed: Random seed for reproducibility (optional)
    
    Yields:
        str: Test input in the same format as .in files
    """
    if seed is not None:
        random.seed(seed)
    
    # Edge cases first
    edge_cases = [
        # Known important test cases
    ]
    
    for edge in edge_cases:
        yield edge
        count -= 1
        if count <= 0:
            return
    
    # Random cases
    for _ in range(count):
        yield _generate_case()


def _generate_case() -> str:
    """Generate a single random test case."""
    # Implementation here
    pass


# ============================================
# Complexity Estimation (controlled size)
# ============================================

def generate_for_complexity(n: int) -> str:
    """
    Generate test case with specific input size for complexity estimation.
    
    Args:
        n: Target input size
    
    Returns:
        str: Test input with size approximately n
    """
    pass

Edge Case Design

Edge cases should cover:

Category	Examples
Boundary values	Min/max constraints, empty inputs
Special cases	Single element, all same values
Negative cases	Negative numbers, zero
Classic examples	LeetCode example inputs

Example (Median of Two Sorted Arrays):

import json

# Store as data structures, not strings
edge_cases = [
    ([], [1]),                    # nums1 is empty
    ([1], []),                    # nums2 is empty
    ([1, 3], [2]),                # Classic odd total length
    ([1, 2], [3, 4]),             # Classic even total length
    ([-5, -3, -1], [2, 4, 6]),    # Negative and positive
    ([1], [1]),                   # Same single element
]

for nums1, nums2 in edge_cases:
    yield f"{json.dumps(nums1, separators=(',', ':'))}\n{json.dumps(nums2, separators=(',', ':'))}"

Guaranteed Valid Input

For problems requiring valid solutions exist, ensure generated inputs are solvable:

def _generate_case(size: int) -> str:
    """
    Generate a Two Sum case with guaranteed solution.
    
    Strategy:
    1. Generate random array
    2. Pick two random indices
    3. Set target = nums[i] + nums[j]
    """
    nums = [random.randint(-10**6, 10**6) for _ in range(size)]
    i, j = random.sample(range(size), 2)
    target = nums[i] + nums[j]  # Guaranteed to have solution
    
    return f"{','.join(map(str, nums))}\n{target}"

Weighted Random Distribution

For more thorough testing, weight towards challenging cases:

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    # ...
    
    for _ in range(count):
        # Weight towards larger n (more thorough testing)
        n = random.choices(
            population=range(1, 10),
            weights=[1, 1, 2, 3, 4, 5, 6, 7, 8],  # Higher weight for larger
            k=1
        )[0]
        yield str(n)

Complexity Estimation Generator

Function Signature

def generate_for_complexity(n: int) -> str:
    """
    Generate test case with specific input size for complexity estimation.
    
    Args:
        n: Target input size
    
    Returns:
        str: Test input with size approximately n
    """

Purpose

The --estimate flag uses this function to:

Generate test cases of increasing sizes
Measure execution time for each size
Fit curve to estimate Big-O complexity

📖 See Test Runner § Complexity Estimation for usage.

Example

def generate_for_complexity(n: int) -> str:
    """
    Generate test case with specific input size.
    
    For Two Sum:
    - n is the length of nums array
    - Expected complexity: O(n) with hash map
    """
    n = max(2, n)  # Ensure minimum valid size
    return _generate_case(n)

Size Semantics

Define what "n" means for your problem:

Problem Type	n Meaning
Array problems	Array length
String problems	String length
Two-array problems	Total elements (m + n)
Matrix problems	Total cells (rows × cols)
Graph problems	Number of nodes or edges

Input Format Specifications

Format Rules (Canonical JSON)

Rule	Requirement
1 line = 1 parameter	Follow function signature order
JSON literal	Each line is a valid JSON value
No spaces after `,`	`[1,2,3]` not `[1, 2, 3]`
Double quotes only	`"abc"` not `'abc'`
Lowercase booleans	`true`/`false` not `True`/`False`

📖 See Test File Format for complete format specification.

Common Formats

Single array:

# One parameter: nums
"[2,7,11,15]"

Array + target:

# Two parameters: nums (line 1), target (line 2)
"[2,7,11,15]\n9"

Two arrays:

# Two parameters: nums1 (line 1), nums2 (line 2)
"[1,3]\n[2]"

Matrix (grid):

# One parameter: 2D array as single line
"[[1,2,3],[4,5,6],[7,8,9]]"

String parameter:

# String must be JSON double-quoted
"\"hello\""

Single integer:

# Plain number
"4"

Using json.dumps for Serialization

import json

def _generate_case() -> str:
    nums = [3, 2, 2, 3]
    val = 3
    # Use separators to avoid spaces
    return f"{json.dumps(nums, separators=(',', ':'))}\n{val}"
    # Output: "[3,2,2,3]\n3"

⚠️ Avoid manual string formatting:

# Wrong: has spaces
f"{nums}"  # -> "[1, 2, 3]"

# Correct: use json.dumps
json.dumps(nums, separators=(',', ':'))  # -> "[1,2,3]"

JUDGE_FUNC Requirement

Why JUDGE_FUNC is Required

Generated test cases have no expected output (.out file). The solution MUST validate correctness using JUDGE_FUNC:

Generator → Input only → Solution → Output → JUDGE_FUNC validates

📖 See JUDGE_FUNC Specification for complete documentation.

Generator-Specific Considerations

When JUDGE_FUNC is used with generators (judge-only mode):

Parameter	Value	Implication
`actual`	Solution output	Parsed via `ast.literal_eval()` if valid
`expected`	`None`	No `.out` file exists
`input_data`	Raw input string	Use to validate correctness

The JUDGE_FUNC MUST be able to validate using only actual and input_data when expected is None.

Example Pattern

def judge(actual, expected, input_data: str) -> bool:
    # Parse input to understand problem constraints
    n = int(input_data.strip())
    
    # Validate actual output against problem requirements
    if not _is_valid_output(actual, n):
        return False
    
    # For judge-only mode: use known answers or algorithmic validation
    if expected is None:
        return _validate_without_expected(actual, n)
    
    # For static tests: compare with expected
    return actual == expected

JUDGE_FUNC = judge

Running Generated Tests

Command Line Usage

# Static tests + N generated tests
python runner/test_runner.py {problem} --generate N

# Only generated tests (skip static tests)
python runner/test_runner.py {problem} --generate-only N

# Reproducible with seed
python runner/test_runner.py {problem} --generate N --seed 12345

# Save failing cases to tests/
python runner/test_runner.py {problem} --generate N --save-failed

# Complexity estimation
python runner/test_runner.py {problem} --estimate

📖 See Test Runner Specification for full CLI reference.

Output Format

============================================================
🧪 Test Results: 0001_two_sum
============================================================

📁 Static Tests:
0001_two_sum_1: ✅ PASS [judge]          0.12ms
0001_two_sum_2: ✅ PASS [judge]          0.08ms

🎲 Generated Tests (seed=12345):
gen_1: ✅ PASS [judge-only]              0.45ms
gen_2: ✅ PASS [judge-only]              0.52ms
gen_3: ❌ FAIL [judge-only]              0.38ms
   ┌─ Input ─────────────────────────────────
   │ 5,3,8,1,2
   │ 11
   └─────────────────────────────────────────

💡 To reproduce: python runner/test_runner.py 0001 --generate 3 --seed 12345
============================================================

Failure Reproduction

When a generated test fails:

Re-run with same seed:

python runner/test_runner.py 0001 --generate 10 --seed 12345

Save failed case:

python runner/test_runner.py 0001 --generate 10 --save-failed
# Creates: tests/0001_failed_1.in

Debug specific case:

python runner/case_runner.py 0001 failed_1

Best Practices

Generator Checklist

Docstring with LeetCode constraints
seed parameter for reproducibility
Edge cases yielded first
Random cases respect constraints
Input format matches .in files
Solution has JUDGE_FUNC defined
generate_for_complexity() if using --estimate

Performance Considerations

Consideration	Recommendation
Generation speed	Keep generators fast (< 1ms per case)
Constraint limits	Use LeetCode max constraints for stress tests
Practical limits	Don't exceed O(N!) or exponential complexity bounds

Testing Your Generator

# Manual test
from generators.{problem} import generate

for i, test_input in enumerate(generate(count=5, seed=42)):
    print(f"--- Case {i+1} ---")
    print(test_input)
    print()

Quick Reference

Generator Template

# generators/{problem_id}_{slug}.py
"""
Test Case Generator for Problem {ID} - {Title}

LeetCode Constraints:
- {constraints}
"""
import json
import random
from typing import Iterator, Optional


def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    if seed is not None:
        random.seed(seed)
    
    # Edge cases (store as data structures)
    edge_cases = [
        ([1, 2, 3], 4),  # example: (nums, target)
    ]
    for nums, target in edge_cases:
        yield f"{json.dumps(nums, separators=(',', ':'))}\n{target}"
        count -= 1
        if count <= 0:
            return
    
    # Random cases
    for _ in range(count):
        yield _generate_case()


def _generate_case() -> str:
    # Generate valid input using json.dumps
    nums = [random.randint(1, 100) for _ in range(10)]
    target = random.randint(1, 200)
    return f"{json.dumps(nums, separators=(',', ':'))}\n{target}"


def generate_for_complexity(n: int) -> str:
    return _generate_case_with_size(n)

CLI Reference

# Run with generation
python runner/test_runner.py {problem} --generate N
python runner/test_runner.py {problem} --generate-only N
python runner/test_runner.py {problem} --generate N --seed S
python runner/test_runner.py {problem} --generate N --save-failed

# Complexity estimation
python runner/test_runner.py {problem} --estimate

📖 See Test Runner Specification for full CLI reference.

Document	Content
Test File Format	Canonical `.in`/`.out` format specification
Solution Contract	`SOLUTIONS`, `JUDGE_FUNC`, `COMPARE_MODE`, file structure
Test Runner Specification	CLI options, output format, troubleshooting
Architecture Migration	Polymorphic pattern migration guide

FilesExpand file tree

generator-contract.md

Latest commit

History

generator-contract.md

File metadata and controls

Generator Contract Specification

Table of Contents

File Structure

Naming Convention

Required Elements

Optional Elements

generate() Function

Function Signature

Contract Rules

Minimal Example

Generator Design Patterns

Standard Template

Edge Case Design

Guaranteed Valid Input

Weighted Random Distribution

Complexity Estimation Generator

Function Signature

Purpose

Example

Size Semantics

Input Format Specifications

Format Rules (Canonical JSON)

Common Formats

Using json.dumps for Serialization

JUDGE_FUNC Requirement

Why JUDGE_FUNC is Required

Generator-Specific Considerations

Example Pattern

Running Generated Tests

Command Line Usage

Output Format

Failure Reproduction

Best Practices

Generator Checklist

Performance Considerations

Testing Your Generator

Quick Reference

Generator Template

CLI Reference

Related Documentation