OAD Stress Test Protocol

Paper type: benchmark / evaluation paper. This repository implements a feature-level stress-test protocol for Online Action Detection (OAD) under resource-constrained streaming recognition.

The first engineering goal is deliberately modest: run a causal evaluation loop on pre-extracted features and reproduce simple budget-aware baselines. This codebase should not be presented as a SOTA method repository.

Core question

When should online action detectors predict, wait, or abstain under feature-level observation / inference budgets, event transitions, and distribution shift?

Known-Horizon Causal Budget Setting

The first benchmark version uses a known-horizon feature-level observation / inference budget setting. A policy may know the current video's total length T when allocating its budget. At timestep t, however, the policy may only use observed features x_1...x_t and its own past state. It must not access future features x_{t+1...T}, future labels, or future prediction results.

Unknown-horizon streaming is a future extension, not a first-version promise. The resource constraint should be described as a feature-level budget, not as real device or hardware deployment. Reference wait/abstain wrappers are baselines for the benchmark, not SOTA methods.

What this repo provides

• Causal streaming evaluator: at timestep t, a policy only receives information from x_1...x_t.
• Feature-level datasets and a dummy data generator.
• Prototype and linear-probe sanity classifiers for first-pass baselines.
• Classifier fitting cache under .cache/classifiers/ by default; pass --no-cache to force refitting.
• Budget wrappers: uniform, random, confidence threshold, and uncertainty wait/abstain.
• Metrics:
  • transition delay distribution;
  • budget degradation curve;
  • selective risk / coverage / latency summary;
  • failure taxonomy diagnostics.
• Plotting scripts for first-stage figures.
• Basic tests that guard against future leakage in the evaluator.

Quick start with dummy data

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .

python scripts/prepare_dummy_data.py --config configs/dummy.yaml
python scripts/run_baseline.py --config configs/dummy.yaml --policy uniform --budgets 0.10 0.25 0.50 1.00 --overwrite
python scripts/run_baseline.py --config configs/dummy.yaml --policy random --budgets 0.10 0.25 0.50 1.00 --overwrite

python scripts/evaluate.py --results-dir results/dummy --policies uniform random --budgets 0.10 0.25 0.50 1.00 --clean-summary
python scripts/plot_budget_curve.py --summary results/dummy/summary.csv --out figures/dummy_budget_curve.png
python scripts/plot_transition_delay.py --summary results/dummy/summary.csv --out figures/dummy_transition_delay.png
python scripts/plot_selective_frontier.py --summary results/dummy/summary.csv --out figures/dummy_selective_frontier.png

To inspect additional reference wrappers, run their baselines explicitly and include their names in --policies.

Using THUMOS14 / TVSeries features

The project expects pre-extracted features. A minimal dataset layout is:

data/thumos14/
  features/
    video_0001.npz
    video_0002.npz
  splits/
    train.txt
    test.txt

Each .npz file should contain:

• features: shape [T, D], float array;
• labels: shape [T], integer frame/timestep labels;
• optional timestamps: shape [T].

Then edit configs/thumos14.yaml to point to the feature and split directories.

Important discipline

1. This is an evaluation codebase, not a new SOTA method.
2. The budget is feature-level observation / inference budget, not real device or hardware deployment.
3. The stress-test protocol is the contribution. Datasets are instantiations.
4. The first version uses a known-horizon setting: a policy may know T for budget allocation.
5. Keep all baselines causal. Never use future frames, future labels, or future prediction results for online decisions.
6. Treat uncertainty wait/abstain as a reference baseline, not a SOTA method.

Linear-Probe Sanity Note

The linear_probe backend is a CPU sanity classifier, not an OAD-native temporal model. The v0.3 convergence smoke compared linear_max_iter=1000 and 3000 on a 5-video THUMOS14 summary-only run: both removed the ConvergenceWarning and produced matching smoke metrics, while 3000 was much slower. linear_max_iter=5000 was not run because the 3000 smoke already exceeded the intended engineering-check cost. These checks support using linear_max_iter=1000 for convergence-sane sanity runs, with cache enabled.

Recommended first milestone

By the end of the first month, the only required deliverable is:

• THUMOS14 feature loader runs;
• uniform / random baselines run;
• budget degradation curves are generated;
• evaluator tests pass;
• logs include per-timestep decisions.