Testing Strategy

Current Status (2026-02-05)

✅ Container infrastructure validated. All 4 lecture repositories build successfully on both containers.

Validation Results

Repository	quantecon-build	quantecon	Status
lecture-python-programming.myst	4.6 min	5.4 min	✅ Pass
lecture-python-intro	11.5 min	13.4 min	✅ Pass
lecture-python-advanced.myst	33.6 min	35.9 min	✅ Pass
lecture-python.myst	60.0 min	60.0 min	✅ Pass

Key Findings:

Lean container (quantecon-build) performs equal or better than full container
No Jupyter Book warnings (builds pass with -W warnings-as-errors)
Container pulls + setup: ~2-3 min (vs 7-8 min for ubuntu-latest)

Action-Level PR Harness (`test-actions.yml`)

.github/workflows/test-actions.yml tests the composite actions themselves — via uses: ./<action> local paths, so it exercises the code on the PR, not a released ref. The workflow runs on every PR, on pushes to main, and on manual dispatch; a gate job then decides whether the jobs themselves apply, skipping them for changes that cannot affect the actions. It works that way because a workflow suppressed by a paths: filter never publishes a check run at all, which would leave a required Action harness: all checks pending forever. This is stage 1 of the two-part design in issue #100; it is the permanent form of the throwaway harness that verified the #104 fix.

Fixtures are salted with run_id-run_attempt so cache keys are unique per run and miss assertions cannot be polluted by earlier runs. The committed fixture (.github/fixtures/mini-lectures/) executes a real code cell, so builds populate a genuine _build/.jupyter_cache.

The container fixture at containers/quantecon/tests/minimal-jupyter-book/ also executes cells, but with force + raise_on_error and against the image's own stack. It tests the image; this one tests the action logic (#108).

Action	Coverage
`restore-jupyter-cache`	Both cache types × both modes: seed/miss, read-only restore, save-mode restore (the #104 scenario), `fail-on-miss` firing and staying quiet, payload round-trip
`setup-environment`	Standard-mode conda cache, two-run miss→hit chain (the #33/#78 path, previously never confirmed by CI)
`build-lectures`	Real HTML build of the fixture on the cache-restored conda env, executed-cell output asserted; plus the negative direction — a page whose code cell raises must fail the step, so a regression that swallows the build status cannot pass silently
`build-jupyter-cache`	Smoke build + outputs, then a full round-trip into `restore-jupyter-cache` for both cache types. ⚠️ Its internal `setup-environment`/`build-lectures` calls run `@v0`, not the PR — GitHub forbids expressions in `uses:`, so that chain is untestable pre-release by construction (see #100)
`deploy-cloudflare`	Everything short of a real deploy: the Access gate probe against a local stand-in for every response shape (exact team domain passes; wrong org, public `2xx`, same-host redirect, missing `Location`, a `Location` hiding its host behind userinfo, `404`, `5xx`, unreachable, an empty team domain, and a public asset behind a gated root all fail); the action refusing an ungated Worker before wrangler is even installed, asserted on the filesystem, with a positive control proving the same inputs reach wrangler once the gate is off; every invalid input failing in validation, with the gate off so a dropped rule cannot hide behind it; and the pinned wrangler accepting the generated config (`--dry-run` deploy and alias upload, with no `Unexpected fields` warning), which is what a Dependabot bump of the lockfile runs against. ❌ A real deploy needs a gated Worker and a token, so the first consumer deploy is its only proof
`preview-netlify`, `preview-cloudflare`	CLI install only: `npm ci` from each action's lockfile on Node 24, then the CLI must start and report the pinned version, so a Dependabot CLI bump gets some signal (#105). The actions themselves, deploys included, are ❌ not covered — they need real deploy targets and secrets; owned by the post-release canary repo (stage 2+ of #100)
`publish-gh-pages`	❌ Not covered — needs real deploy targets and secrets; owned by the post-release canary repo (stage 2+ of #100)

Local Test Fixtures

`tests/local/` (git-ignored)

Scratch space for local clones of real lecture repos, used to test workflows on your own machine against something larger than the committed fixture. Ignored via tests/local/ in .gitignore, so it exists only in your working tree — never committed, and read by no workflow. See tests/README.md for the wider layout.

Not to be confused with QuantEcon/test-actions-lecture-intro, the canary repo that runs the actions in real CI (#100 stage 2). This is a throwaway directory; that is a live repo.

Local clones used to be documented at the repo root as test-lecture-python-intro/, one character away from the canary's old name — close enough that the two were repeatedly confused. A path under tests/local/ cannot collide with a repo name. The old test-*/ ignore pattern is retained, so existing local clones stay ignored where they are.

Purpose:

Test container builds locally
Validate action changes before pushing
Debug workflow issues

Setup:

# from the repo root
git clone https://github.com/QuantEcon/lecture-python-intro.git tests/local/lecture-python-intro

Usage with local scripts:

# Test container builds
cd containers/quantecon/tests
./run-local-tests.sh

Approach

Test in isolation using test fixtures before production rollout.

Testing Principles

Isolated testing - Use test directories, not production repos
Compare outputs - Build artifacts must match current approach
Measure performance - Document actual vs expected timing
Validate deployment - Test Netlify and build artifacts
Staged rollout - One repository at a time after validation

Testing Workflow

Phase 1: Container Validation

1.1 Build Container

Trigger container build workflow:

# Manual trigger
gh workflow run build-containers.yml

# Or push to main (auto-triggers)
git push origin main

1.2 Verify Container Image

# Pull and inspect
docker pull ghcr.io/quantecon/quantecon:latest
docker run --rm ghcr.io/quantecon/quantecon:latest python --version
docker run --rm ghcr.io/quantecon/quantecon:latest conda list
docker run --rm ghcr.io/quantecon/quantecon:latest pdflatex --version

Verify:

Python 3.13 installed
Anaconda base packages available (numpy, scipy, pandas, matplotlib)
Jupyter Book tools installed
LaTeX commands work

Phase 2: Test Repository Workflow

2.1 Update test-actions-lecture-intro

Create container-based workflow:

name: Build with Container
on:
  workflow_dispatch:
  push:
    branches: [test-container]

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/quantecon/quantecon:latest
    steps:
      - uses: actions/checkout@v7
      
      # Install lecture-specific packages
      - name: Install dependencies
        run: conda env update -f environment.yml
      
      - name: Build lectures
        run: jupyter-book build lectures/
      
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: html
          path: lectures/_build/html/

2.2 Compare Builds

Run both container and ubuntu-latest workflows:

Timing - Measure setup time, build time, total time
Artifacts - Download and diff HTML outputs
Errors - Check for differences in warnings/errors
Performance - Verify 60-70% faster setup time

Phase 3: Validation Checklist

Before production rollout:

Validation Tests

Test 1: Container Pull Performance

Goal: Verify container pulls quickly from GHCR

Check workflow logs:

First pull: ~1-2 min (downloads 3.33 GB compressed for the full image, 2.93 GB for the lean one)
Subsequent pulls: ~10-20 sec (runner cache)

Test 2: Environment Validation

Goal: Verify all required packages available

# In workflow, add diagnostic step:
- name: Validate environment
  run: |
    python --version
    conda list | grep -E "(numpy|scipy|pandas|matplotlib|jupyter)"
    jupyter-book --version
    pdflatex --version

Test 3: Lecture Package Installation

Goal: Verify lecture-specific packages install correctly

Check workflow logs:

conda env update -f environment.yml completes
Packages like quantecon, cvxpy installed
Installation time: ~1-2 min

Test 4: Build Output Comparison

Goal: Verify builds produce identical output

Steps:

Build with container workflow
Build with current ubuntu-latest workflow
Download both HTML artifacts
Compare:
```
diff -r container-html/ ubuntu-html/
```

Expected: No differences (or only timestamp differences)

Test 5: Performance Measurement

Goal: Document actual performance improvements

Measure:

Setup time (container pull + package install)
Build time (jupyter-book build)
Total workflow time

Compare to baseline:

ubuntu-latest setup: 7-8 min
Container setup target: 2-3 min
Improvement: 60-70%

Rollout Plan

Stage 1: Test Repository

Pick a test repository or create a fork:

# Option 1: Use lecture-python-programming.myst (lower traffic)
# Option 2: Create a fork for testing

Merge test workflow into main
Monitor for 1 week
Collect metrics (build times, success rate)

Complete testing with QuantEcon/test-actions-lecture-intro, validate all metrics.

Stage 2: CPU Lecture Repositories

After successful testing, migrate CPU-based lecture repositories:

lecture-python-intro (Netlify, similar to test repo)
lecture-python-programming.myst (GitHub Pages)
lecture-python-advanced.myst (GitHub Pages)
lecture-python.myst - CPU builds only (defer GPU workflows)

Stage 3: Production Validation

For each repository:

Create migration PR with container workflow
Test in PR (manual trigger)
Compare outputs with current workflow
Merge after validation
Monitor first production build

Rollback: Revert workflow commit if issues occur.

Success Metrics

Track before/after migration:

Metric	Current (ubuntu-latest)	Target (Container)
Setup time	7-8 min	2-3 min
Build time	8-10 min	8-10 min (same)
Total time	15-18 min	10-13 min
LaTeX install	2-3 min	0 min (pre-installed)
Base packages	3-4 min	0 min (pre-installed)

Success criteria:

Build output matches current production (100%)
Setup time reduced by 60-70%
No new errors or warnings

Troubleshooting

Container not found

Issue: Error: failed to pull image

Solution:

Verify image name: ghcr.io/quantecon/quantecon:latest
Check container build workflow ran successfully
Ensure image is public (no auth required)

Package conflicts

Issue: Conda solve fails or package conflicts

Debug:

echo "Cache key: conda-${{ runner.os }}-${{ hashFiles('environment.yml') }}-v1"
ls -la ~/.conda/pkgs || echo "No conda cache"

Solution: Check cache key format, verify paths

Issue: Build output differs

Symptom: HTML/PDF differs from production

Debug:

Download both artifacts
Compare with diff -r
Check for version differences in packages

Solution: Pin package versions in environment.yml

Issue: Workflow timeout

Symptom: Workflow exceeds time limit

Debug: Check if cache is working, verify runner specs

Solution: Increase timeout or investigate slow steps

- name: Debug environment
  run: |
    conda list
    python -c "import quantecon; print(quantecon.__version__)"

Check:

Verify lecture's environment.yml doesn't conflict with container base
Try updating conda: conda update -n base conda

Build differences

Issue: Container build differs from ubuntu-latest

Debug:

Compare Python versions
Check package versions: conda list > versions.txt
Diff HTML outputs to identify specific differences
Look for timestamp-only changes vs content changes

Navigation Menu

Uh oh!

Testing Strategy

Current Status (2026-02-05)

Validation Results

Action-Level PR Harness (`test-actions.yml`)

Local Test Fixtures

`tests/local/` (git-ignored)

Approach

Testing Principles

Testing Workflow

Phase 1: Container Validation

1.1 Build Container

1.2 Verify Container Image

Phase 2: Test Repository Workflow

2.1 Update test-actions-lecture-intro

2.2 Compare Builds

Phase 3: Validation Checklist

Validation Tests

Test 1: Container Pull Performance

Test 2: Environment Validation

Test 3: Lecture Package Installation

Test 4: Build Output Comparison

Test 5: Performance Measurement

Rollout Plan

Stage 1: Test Repository

Stage 2: CPU Lecture Repositories

Stage 3: Production Validation

Success Metrics

Troubleshooting

Container not found

Package conflicts

Issue: Build output differs

Issue: Workflow timeout

Build differences

See Also

Uh oh!

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing Strategy

Current Status (2026-02-05)

Validation Results

Action-Level PR Harness (test-actions.yml)

Local Test Fixtures

tests/local/ (git-ignored)

Approach

Testing Principles

Testing Workflow

Phase 1: Container Validation

1.1 Build Container

1.2 Verify Container Image

Phase 2: Test Repository Workflow

2.1 Update test-actions-lecture-intro

2.2 Compare Builds

Phase 3: Validation Checklist

Validation Tests

Test 1: Container Pull Performance

Test 2: Environment Validation

Test 3: Lecture Package Installation

Test 4: Build Output Comparison

Test 5: Performance Measurement

Rollout Plan

Stage 1: Test Repository

Stage 2: CPU Lecture Repositories

Stage 3: Production Validation

Success Metrics

Troubleshooting

Container not found

Package conflicts

Issue: Build output differs

Issue: Workflow timeout

Build differences

See Also

Action-Level PR Harness (`test-actions.yml`)

`tests/local/` (git-ignored)