Feat: Adding test framework with benchmarks - #36

Open

Lei Jiang (lionelc) wants to merge 54 commits into

lionelc:import-cosmos-test-framework

Commits on Jul 6, 2026

Create secret-scan.yml for Gitleaks integration
Lei Jiang (lionelc)
authored
2cddfaf
Add GITHUB_TOKEN
Lei Jiang (lionelc)
authored
e20ec9c
Add workflow_dispatch trigger to secret scan
Lei Jiang (lionelc)
authored
e93e26d
Add pinned commit SHA
Lei Jiang (lionelc)
authored
c292c4a
Update Gitleaks action to use version v2
Lei Jiang (lionelc)
authored
b317896
Merge pull request #1 from lionelc/gitleak
Lei Jiang (lionelc)
authored
a41d3aa

Commits on Jul 13, 2026

Add SECURITY.md for compliance
Lei Jiang
committed
035a745
Fix link for Coordinated Vulnerability Disclosure
Lei Jiang (lionelc)
authored
4e05a9a

Commits on Aug 3, 2026

Merge branch 'Azure:main' into main
Lei Jiang (lionelc)
authored
5108b66

Commits on Aug 11, 2026

Add the two-loop test architecture: testing/ (deterministic) + evals/ (skills)

Lei Jiang
and
Copilot
committed
70d687d
testing: fail fast with a clear message when the DB password is wrong

Lei Jiang
and
Copilot
committed
7bd3560

Commits on Aug 13, 2026

Fix four measured nondeterminism bugs in the diagnostic scripts

Lei Jiang
and
Copilot
committed
52f540e
Add task-level token / AI-credit accounting for Loop B

Lei Jiang
and
Copilot
committed
421f3ad
Add before/after remediation grading — objective, no LLM judge

Lei Jiang
and
Copilot
committed
c2f69c4
Add the Loop B cross-model experiment matrix and static config guards

Lei Jiang
and
Copilot
committed
9279f9b
Add CI for both loops, and fix a latent break in the container setup

Lei Jiang
and
Copilot
committed
b1a2cb9
Add docs/TESTING.md — reproduction guide for both loops

Lei Jiang
and
Copilot
committed
b64c433
Add the MSBench skill-efficacy benchmark (Loop C), with per-task token cost

Lei Jiang
and
Copilot
committed
78340ac
Build and verify the MSBench benchmark end-to-end: oracle passes, naive fails

Lei Jiang
and
Copilot
committed
473612f
Correct the grading-criteria language: fit, not a hierarchy of legitimacy

Lei Jiang
and
Copilot
committed
5659218
Add the effectiveness report generator, and state plainly that no MSBench run exists

Lei Jiang
and
Copilot
committed
97f7e52
Define "treatment" and "control" — the terms were used but never explained

Lei Jiang
and
Copilot
committed
8edab2f
Fix an intermittent determinism failure: settle planner statistics after seeding

Lei Jiang
and
Copilot
committed
88c3e7d
Document what "quality" the benchmark actually measures — and the gap

Lei Jiang
and
Copilot
committed
e4949ad

Commits on Aug 14, 2026

Add the guidance-quality judge: the kit's one qualitative measurement

Lei Jiang
and
Copilot
committed
b0560aa
Make every reported number reproducible, and document how

Lei Jiang
and
Copilot
committed
df35964

Commits on Aug 17, 2026

Complete the reproduction commands in REPORT.md, and verify them from a clean clone

Lei Jiang
and
Copilot
committed
3707cb1
Check results into the repo, with provenance and paired arms enforced

Lei Jiang
and
Copilot
committed
1ab5b4c
Rebuild the token study as a measured benchmark: route cost AND parity

Lei Jiang
and
Copilot
committed
80787da
Price cross-model cost in USD, and unblock the matrix with subagents

Lei Jiang
and
Copilot
committed
5fb6199

Commits on Aug 18, 2026

Retract the "superseded" label on token-tests: the objection was wrong

Lei Jiang
and
Copilot
committed
7d40401
Decouple the testing suite and route benchmark from token-tests/

Lei Jiang
and
Copilot
committed
3cef893
Clean up (no token-tests dependencies for now)
Lei Jiang (lionelc)
committed
9496c73

Commits on Aug 19, 2026

Fix shellcheck failures in Loop C — including a real bug in the route harness

Lei Jiang
and
Copilot
committed
86b0f97
Fix workflow issues
Lei Jiang (lionelc)
committed
72e7c72
cleanup
Lei Jiang
committed
1da37fa
Stop config validation depending on generated results

Lei Jiang
and
Copilot
committed
32b2e8c
Further cleanup to remove unnecessary dependency for Loop C benchmark
Lei Jiang (lionelc)
committed
e0f488d

Commits on Aug 28, 2026

Clarify remediation structural redundancy guard
Lei Jiang
committed
606441e

Commits on Aug 31, 2026

Add MongoDB-to-DocumentDB ecommerce compatibility quickstart
Lei Jiang
committed
16975d2

Commits on Sep 1, 2026

Move the compatibility quickstart to the repository root
Lei Jiang
committed
11965ce
Make the compatibility project DocumentDB-only
Lei Jiang
committed
3469663
Add deterministic advanced ecommerce compatibility data
Lei Jiang
committed
5b05166

Commits on Sep 15, 2026

cross-platform compatibility for scripts
Lei Jiang
committed
31a65b0
cross-platform compatibility
Lei Jiang (lionelc)
committed
e0d06f8

Commits on Sep 21, 2026

adding missing files for testing framework
Lei Jiang
committed
5285d4e
use commit hash in yml
Lei Jiang
committed
19c69f8
consolidate doc
Lei Jiang
committed
226677d

Commits on Sep 22, 2026

sort out mongosh installation with a check
Lei Jiang
committed
de79c89
permission to 755
Lei Jiang
committed
400325b
addressed model selection issue
Lei Jiang
committed
17f50cd
fix invalid json in perf-advisor.sh
Lei Jiang
committed
65ac7fb
fix conftest.py to add a gate
Lei Jiang
committed
f3408d2
fix coll concat in several shell scripts
Lei Jiang
committed
bff3d6a