Feat: Adding test framework with benchmarks - #36
Open
Lei Jiang (lionelc) wants to merge 54 commits into
Open
Lei Jiang (lionelc) wants to merge 54 commits into
Lei Jiang (lionelc) wants to merge 54 commits into
Conversation
Add GitHub Actions workflow for secret scanning using Gitleaks.
Create secret-scan.yml for Gitleaks integration
… (skills)
Phase 0 + Phase 1 of the testing plan. The kit now tests two different things,
with opposite success criteria, in two folders:
testing/ Loop A — the diagnostic scripts are TOOLS: same DB state must give
the same answer. Free, fast, every PR.
evals/ Loop B — the text skills should make an AGENT better: measured on
quality and token cost. Costs AI credits, run deliberately.
Loop A — testing/scenarios/determinism/
- Runs each diagnostic script 3x against one untouched database and asserts the
canonicalised --json is identical.
- harness/canonicalise.py normalises only what is provably volatile: live
measurements, lists ordered by a volatile metric, $sample-derived counts, and
sampled size strings. The allowlist is measured, not guessed, and each entry
documents the drift that justifies it.
- Three traps are explicitly defended: false determinism (must_find asserts the
output is non-empty), over-broad normalisation, and a test that cannot fail
(a negative control, verified by temporarily emitting $RANDOM from a script).
- The fixture is itself reproducible (fixed-seed LCG, never Math.random) and
uses incompressible text so it actually reaches TOAST.
It found three real volatility sources on first use, all documented rather than
silently normalised away:
1. data-integrity-check.sh uses $sample -> type COUNTS wobble (87/13 vs 85/15);
the finding (amount holds numbers AND strings) is still asserted.
2. document-bloat-advisor.sh samples dominant_fields -> sizes wobble
(title:13B vs 12B); the field ranking / recommended_split_field is asserted.
3. perf-advisor.sh builds slow_queries by TIMING against a threshold, so
membership changes with cache warmth — it passed in isolation and failed in
the full run. Its structural findings (collscans, index_health) still hold.
Loop B — evals/ on Vally (@microsoft/vally-cli, MIT, pinned)
- First eval is skill ROUTING, chosen because it is the cheapest useful signal,
is graded objectively (skill-invocation is binary — no LLM judge), and is
otherwise only ever checked by hand: 3 positive triggers + 2 anti-triggers.
- Skills are made discoverable but never hinted, so this measures the skill's
organic effect (protocol borrowed from the MSBench runner).
- README documents that the mock executor invokes no skills, so anti-triggers
pass VACUOUSLY there — mock validates plumbing only, never signal.
Full suite: 51 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
DocumentDB reports a bad SCRAM credential as `MongoServerError: Invalid key`. That surfaced only later, as a "fixture seed failed" error on every scenario, so a simple password mismatch produced ~30 cryptic failures that look like a broken fixture rather than an auth problem. - kit.can_authenticate(): one-shot credential preflight (ping against admin). - conftest.require_container(): if a password is set but does not authenticate, abort the session once via pytest.exit with an actionable message that names the real cause, shows how to recover the container's password via `docker inspect`, and explains the misleading "Invalid key" wording. Behaviour now: no password -> skip (unchanged) wrong password -> single, explicit auth error (was ~30 cryptic failures) correct password -> 51 passed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
The determinism suite (Loop A) found four real defects. All four are fixed at
the source rather than normalised away in the volatile-field allowlist.
1. data-integrity-check.sh used $sample for type-consistency, so the reported
counts changed between runs on an unchanged database
({"number":87,"string":13} vs {"number":85,"string":15}).
2. document-bloat-advisor.sh used $sample for dominant_fields, so average field
sizes wobbled ("title:13B" vs "title:12B").
Both now use a deterministic sampler: half the documents from each end of _id
order. Head+tail rather than head-only is deliberate — mixed types usually come
from schema drift over time, so the oldest and newest documents are where the
disagreement lives. Sampling only the head would systematically miss a type
change introduced after the collection was created.
3. perf-advisor.sh decided slow_queries MEMBERSHIP with a wall-clock threshold,
so which queries qualified changed with cache warmth. Split into
query_timings (every probe; deterministic membership and results, volatile
ms) and slow_queries (the >threshold subset, kept for compatibility and
correctly treated as a measurement).
4. perf-advisor.sh built COLLSCAN probe values from findOne(), i.e. an
arbitrary document. A numeric probe of val/2 therefore scanned a different
fraction of the collection each run, changing docs_scanned and sometimes the
plan PostgreSQL chose. Now probes the first document by _id.
Bug 4 was invisible until bug 3 was fixed: the timing noise being normalised
away was masking a genuine logic defect underneath. That is the argument for
keeping the volatile-field allowlist as small as possible.
sampled_count_maps and sampled_size_strings are now empty but retained, so
re-introducing a normalisation is a visible, reviewable change.
Verified: 51 passed; determinism suite green across 3 consecutive invocations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
evals/harness/token_usage.py reads the Copilot CLI's local session store (assistant_usage_events) and emits per-task cost artifacts, then aggregates them into the model x arm comparison table Loop B is meant to produce. Stdlib only, read-only, and it works today without the Vally auth token. The design is built around two measurement traps: 1. cache_read_tokens is a SUBSET of input_tokens (verified against the real store: 0 of 6,348 rows had cache_read > input). Skills are sent once and then read from cache, so raw input_tokens overstates their marginal cost by roughly 20x. We report fresh_input = input - cache_read as the tokens actually billed at the uncached rate, plus cache_read_share to show the amortisation. 2. "Skills use more tokens" is trivially true and meaningless. The honest comparison is cost-to-OUTCOME, so the artifact carries credits_to_green / turns_to_green / tokens_to_green, and these are null for a run that never went green - otherwise giving up early would look like efficiency. Other deliberate choices: - credits_per_pass divides by PASSES, not runs: failed attempts are part of the price of a passing result. - Cells with n < 3 are flagged in the rendered table; a mean over one non-deterministic agent run is not a result. - Subagent spend is broken out rather than dropped, so a skill that delegates cannot hide its cost. - The store is snapshotted with its -wal/-shm sidecars before reading; copying only the .db would silently lose transactions still in the write-ahead log. testing/scenarios/token-accounting/ regression-guards all of the above. It overrides the root require_container fixture with a no-op, so it needs no Docker, no credentials and no network - it runs in CI on every PR. Two mutations were injected to confirm the suite can actually fail (removing the cache subtraction: 3 failures; dividing credits by runs instead of passes: 1). Verified: 67 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Implements the strongest rung of the grading ladder for diagnostics. Instead of
asking a model whether the kit's advice is good, this scenario APPLIES the
advice and measures whether the database improved. The database is the oracle.
The chain, every link asserted:
the tool detects a defect -> the fix is DERIVED FROM THE TOOL'S OWN OUTPUT ->
the plan measurably improves -> results are unchanged -> no new redundancy ->
the tool confirms the defect is gone.
Deriving the fix from perf-advisor's collscans[] rather than hardcoding
createIndex({customer_id:1}) is what makes this a test of the kit rather than a
test of PostgreSQL: delete the collscans reporting and this suite can no longer
derive a fix, and fails.
The metric is SCAN AMPLIFICATION (totalDocsExamined / nReturned), not raw
documents examined. Measured on this container, DocumentDB's planner picks an
index scan even for very unselective predicates:
amount > 490 (selective) IXSCAN examined=760 returned=760
amount > 10 (unselective) IXSCAN examined=19960 returned=19960
Both are healthy - they read only what they return - so raw "examined" would
mostly measure selectivity, not index quality. Amplification captures the real
property independently of selectivity (2000x before, 1x after) and is the same
ratio the query-performance-tuning skill teaches users to read from explain().
Anti-gaming, because an improvement metric alone is trivially gamed by indexing
everything: index-redundancy-finder.sh runs as a paired regression guard,
result sets must be byte-identical before and after, and the before-state must
be genuinely slow or every later assertion passes for free.
Verified by mutation, since a green suite that cannot fail proves nothing:
- skipping the remediation -> 3 tests fail
- indexing everything to game it -> anti-gaming guard fails with
PREFIX_REDUNDANT + WRITE_TAX
Deliberately NOT graded: index-backed sort and covered queries. The gateway
pins the experimental GUCs that enable them off per session, so the local
container cannot demonstrate them. Grading advice we cannot demonstrate would
be worse than not grading it.
Verified: 74 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
evals/documentdb-skills.experiment.yaml declares the 3-model x 2-arm matrix using Vally's matrix block: claude-opus-5, gpt-5.6-sol and gemini-3.1-pro-preview, each run with the kit installed and with no skills at all, 5 runs per cell. Verified with --dry-run: 6 cells resolve, each with a distinct config hash and an identical eval hash. Two design points that make the result meaningful rather than decorative: - The control arm is the whole point. An absolute score is uninterpretable - "90% of stimuli passed" says nothing without knowing what a bare agent scores on the same stimuli. Any published number must be a skills-vs-control delta. - The agent is never told to use the skills. They are made discoverable exactly as a user would have them installed, and no prompt mentions them. This measures the kit's organic effect rather than a hinted best case. (Protocol borrowed from the Cosmos MSBench runner.) testing/scenarios/evals-config/ statically validates both configs, and exists because of a live bug found while writing them: the experiment referenced ../skills/query-performance-tuning, which only exists on another branch, and `vally experiment run --dry-run` PASSED. Vally validates matrix structure but not skill paths. That is the worst failure mode an A/B test can have. A treatment arm whose skills silently fail to load is just a second control arm, and it would have produced a confident, published, entirely wrong "the kit makes no difference". The guard reproduces and catches exactly that case. It also pins the properties that keep the comparison honest: the control arm must be empty, the baseline must sit in the control arm, prompts may not hint at the skills, anti-trigger stimuli must exist (an eval with only positive triggers rewards a kit that fires on everything), runs>=3, and graders must come from an objective allowlist so adding an LLM judge is a deliberate, reviewed change. Needs no container, credentials or network - runs on every PR. Verified: 86 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Two workflows, split by cost: loop-a-tests.yml push + PR + dispatch. Fast, free. loop-b-evals.yml dispatch + monthly schedule ONLY. Spends AI credits. Loop A runs a `static` job first: the token-accounting and evals-config scenarios override the container fixture with a no-op, so they need no Docker, no credentials and no network. Verified on a clean clone: 28 tests in 0.41s. A broken eval config or a wrong cost metric is now caught before the expensive job starts. The important fix: the previous (uncommitted) workflow could never have passed. The documentdb-local image ships psql but NO mongosh - verified against the pristine image - while every diagnostic script drives the database with `docker exec <container> mongosh`. Every scenario would have failed with "executable file not found", which reads like a broken test rather than a missing dependency. loop-a-tests.yml now installs mongosh 2.3.8 into the container before waiting for the gateway. Verified end-to-end rather than assumed: a fresh container was started with the exact CI command, mongosh installed by the exact CI step, and the full suite run against it - 86 passed. Loop B never triggers on push or pull_request; the full matrix is 30 trials per stimulus. Its free `validate` job (static guards + vally lint + matrix resolve) always runs, so a bad skill path is caught before credits are spent - which matters because `vally --dry-run` does not validate skill paths. The paid job checks for COPILOT_SDK_AUTH_TOKEN up front and stops with an explanation rather than failing obscurely halfway through, sets both Vally telemetry opt-outs, and its summary tells readers to read the deltas rather than the absolute scores. Supersedes regression-tests.yml, which was never committed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Documents the two-loop architecture, every command verified by running it rather than written from memory: - Prerequisites, including the mongosh install step, with an explicit warning that it is not optional (the image ships psql but no mongosh) and that a wrong password surfaces as the misleading "MongoServerError: Invalid key". - Loop A: full suite, the infrastructure-free subset, per-scenario and per-marker selection (marker list cross-checked against pytest.ini), and how to reproduce a determinism check by hand. - Loop B: free commands first (lint, plan, mock) with a prominent warning that the mock executor invokes no skills, so anti-triggers pass vacuously and mock results must never be quoted. Paid commands second, with the single-cell --variant form verified against the CLI. - The two rules for reading Loop B output: only deltas are publishable, and the agent is never told to use the skills. - Cost accounting, with the three ways the numbers mislead if read naively. - The grading ladder, and why improvement metrics are paired with regression guards. Also hardens the Loop B workflow's --variant handling to use a bash array: variant selectors contain commas and '=', so the unquoted form would have word-split. README now points at docs/TESTING.md and lists evals/ in the repo structure. Verified: 86 passed; all internal links resolve; both workflows parse. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
…n cost Phases 3b-3e of the plan. Structure mirrors cosmosdb-agent-kit's benchmarks/cosmos-sdk-skills: contract-driven verifier, shared check_* library, Harbor task layout, ces runner, msbench-registration. TWO ARMS, ALWAYS documentdb-sdk-skills (kit installed, discoverable, NEVER mentioned) and documentdb-sdk-skills-noskills (control). The control is a separately REGISTERED benchmark, not a runtime flag - that is the convention the live platform already uses for skillsbench / skillsbenchnoskills. Only the delta is publishable; an absolute pass@k cannot distinguish a good kit from an easy task. STRONGER THAN ITS COSMOS ANCESTOR The Cosmos benchmark concedes that client-side properties a single-node emulator "cannot prove behaviorally" must fall back to static source regex - its weakest grader, ~520 lines of it. DocumentDB is MongoDB-compatible on top and PostgreSQL underneath, and the verifier talks to both, so check_engine.py proves index usage from explain() plans and pg_stat_user_indexes instead of grepping for create_index(. An agent can fake the source; it cannot fake the engine's statistics. Our check_source.py is ~130 lines because nearly everything moved to a stronger rung. The metric is scan amplification (examined/returned), not raw documents examined: measured on this engine the planner picks an index scan even for unselective predicates, so raw "examined" mostly measures selectivity. Engine checks run against 20k filler documents plus ANALYZE - on a 4-row collection a sequential scan genuinely is cheaper and the oracle itself would fail. PER-TASK TOKEN COST (new capability) MSBench's reward is binary and cannot say what a run cost, or which kind of task is expensive. harvest_metrics.py writes MSBench's first-class custom_metrics.json hook with token usage read from the Copilot CLI's own session store, plus per-check-category pass/fail counts so an expensive run can be traced to the rung it was failing. Honesty properties, each pinned by a test: - tokens_fresh_input = input - cache_read. cache_read is a SUBSET of input and ~91% of input on a real session, so quoting raw input overstates skill cost by roughly 20x. - credits_to_green is emitted ONLY for a passing run; otherwise giving up early looks like the cheapest strategy. - tokens_available=0 when harvesting fails, so a gap is never read as free. - All values numeric, because MSBench infers a numeric schema and validates it across instances. - Harvest failure can never change the reward (advisory, runs after grading). The harvester cannot import the Loop B cost module (it must run inside a minimal task container), so consistency is pinned by test instead of by import path - mutation-verified that drift is caught. CORRECTIONS FROM THE LIVE PLATFORM (Phase 3a findings) - registry.json is REQUIRED; the Cosmos registration predates it and would be rejected today. Both arms have one. - [tasks.skip-verification] is deliberately EMPTY. The platform's own skillsbench config skips several checks as "Non-deterministic"; under a binary reward a flaky check corrupts the whole signal, so a check that cannot be made deterministic is removed rather than skipped. A test enforces this. - task_style harbor-native, so --backend local can iterate the whole benchmark without pushing to the internal ACR. TESTING (34 new tests, no container/credentials/network, every PR) testing/scenarios/benchmark-config/ (19) and benchmark-metrics/ (15). Guards verified by mutation: adding "compound index"/"singleton" to instruction.md fails the no-hints check; adding a skip-verification entry fails; breaking the cost definition fails 3 tests. NOT YET VERIFIED - the Docker daemon (Docker Desktop/WSL) became unavailable mid-session, so the images have not been built and the oracle has not been run. Both controls (oracle must score 1, empty /app must score 0) are wired into loop-c-benchmark.yml on workflow_dispatch, and the benchmark README states this gap explicitly rather than implying it works. Verified: 62 infra-free tests pass; all shell entrypoints parse; all doc links resolve. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
…ve fails
Completes 3b-3e. The images now build and BOTH controls run, which turned up
three real bugs that inspection had missed.
VERIFIED BY RUNNING
oracle (reference impl) -> REWARD=1, 31 checks passed
empty /app -> REWARD=0
working-but-NAIVE app -> REWARD=0, 10 checks failed
custom_metrics.json -> emitted with per-category check counts
The naive control is the one that matters. An empty /app failing only proves
the harness notices missing files. The naive app is fully functional - correct
HTTP behaviour, correct persistence - and passes every API and behavioural
check, yet still scores 0, failing exactly the ten skill-specific checks (type
discriminator, schemaVersion, queryable timestamp, secondary index, indexed
query field, no-full-scan, scan amplification, explicit connection options, TLS
not hardcoded off, per-request client). The benchmark measures best practices,
not basic competence.
THREE REAL BUGS THE CONTROLS FOUND
1. check_source.py's regex had catastrophic backtracking (nested quantifiers
over lines). It never returned on a 6 KB file - the verifier hung for
minutes and the run had to be killed. Replaced with ast: exact, and ~1000x
faster (0.3ms vs never). Guarded by a CI test that bans a quantified group
followed by another quantifier.
2. The credential check matched an f-string TEMPLATE,
mongodb://{quote_plus(user)}:{quote_plus(password)}@... - i.e. the CORRECT
pattern of reading credentials from the environment. It failed the reference
implementation. A false positive here would fail every well-written
submission, which is worse than having no check. Guarded by a test asserting
env-driven URIs pass and real secrets still fail.
3. The singleton-client check only looked inside route-decorated functions, so
the naive app slipped through by hiding its per-request MongoClient in a
plain helper. Now flags construction inside ANY function, exempting memoised
factories (@lru_cache) and start-up hooks. Found by the negative control -
exactly what negative controls are for.
BUILD FIXES (each verified against the real image, not assumed)
- `which` is not a package on this base; asking for it failed the whole apt
transaction. The binary already exists at /usr/bin/which.
- pip 22.x on Ubuntu 22.04 has no --break-system-packages (that arrived in pip
23 for PEP 668), and jammy is not externally-managed.
- files.pythonhosted.org is unreachable from Docker's VM on this network
(blocked by IT) though the host can reach it. Wheels are now vendored on the
host by vendor-wheels.sh and installed with --no-index. This is better than a
workaround: the task promises the agent NO INTERNET during grading, and a
pinned wheel set makes the image reproducible. --no-index makes that
enforceable - a missing wheel fails the build instead of silently reaching
the network.
- pip evaluates `python_version` markers against the interpreter running the
download, not --python-version, so tomli/exceptiongroup silently resolved to
"not needed" on a 3.12 host and the image build failed. Pinned explicitly.
- The base image reset ENTRYPOINT and switched to root, but PostgreSQL refuses
to run as root. The database is now started back under the `documentdb` user
while the verifier stays root - the same split the Cosmos benchmark makes.
- The password was generated inside start-documentdb, a child process, so it
could never propagate to the caller. Generated in runner.sh instead.
- /tests/checks.py could not see /verifier/conftest.py fixtures (pytest only
applies a conftest at or below its own directory). It is copied into
/verifier before the run.
- A 3.12-only nested f-string quoting parsed on the dev machine and raised
SyntaxError inside the 3.10 image. Now guarded by a CI test that parses every
verifier module with feature_version=(3, 10).
NEW CI GUARDS (all infra-free, every PR)
Python 3.10 compatibility, catastrophic-backtracking patterns, credential
false positives, and the singleton detector's three cases.
Verified: 128 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
…macy The docs overstated the case against LLM-as-a-judge, calling it "the weakest grader available", "the last resort" and "easily fooled". That is not accurate: a judge is a normal, widely used grader, and this kit deliberately uses one in Loop B for qualities that are genuinely matters of degree - clarity, targeting, whether an explanation would actually help a user. The real reason remediation-effect measures instead of judging is narrower and more defensible: it lives in the DETERMINISTIC loop, and its job is to show the diagnostic scripts return a stable, reproducible answer. A judge cannot demonstrate repeatability, because the same input can score differently across runs. Where an outcome is a fact about the system - a query plan, a counter, a result set, an index shape - measuring it is simply the better fit. Reframed the ladder from "strongest -> weakest" to "most reproducible -> most qualitative", and made the selection rule about fit rather than avoidance. Updated: remediation-effect SCENARIO.md and its test module docstring, docs/TESTING.md, evals/README.md, and the evals-config grader test (which now explains that the allowlist is a speed bump for a deliberate choice, not a prohibition - Phase 1 stimuli are skill-TRIGGERING checks with a definite right answer, so a judge would add cost and variance for nothing). Also softened the comparison with the Cosmos benchmark. Describing their static source checks as "its weakest grader" was needlessly dismissive of a reasonable call: given a single-node emulator that cannot observe client configuration at runtime, a static check beats no coverage. The accurate point is narrower - a static check is easier to satisfy by accident than a runtime observation, and DocumentDB's PostgreSQL layer lets us avoid that trade-off rather than making us better engineers. No behaviour change. Verified: 128 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
…ench run exists The report was requested with "the most recent msbench results". There are none: no MSBench run has been executed, no agent has attempted the task, and there is no skills-on vs skills-off data or agent token usage. Producing a report with numbers would have meant inventing them, so docs/REPORT.md leads with that fact and separates what HAS been measured from what has not. WHAT IS REAL IN THE REPORT Grader validation, re-measured rather than recalled (the first draft said 21/31 from memory; the actual figures are below): oracle reward 1, 31/31 empty /app reward 0 working-but-naive reward 0, 20/30, 10 failed category naive oracle api 6/6 6/6 behavior 6/6 6/6 documentdb 0/5 6/6 engine 2/4 4/4 source 1/4 4/4 skills 4/4 4/4 Basic competence is perfect (12/12) while best practices collapse (3/13) - exactly the separation a skills benchmark must produce. It is offered as a prior for what the control arm may look like, explicitly not as a prediction. The token pipeline was proven end-to-end by mounting a real Copilot session store into the task container: custom_metrics.json came back with tokens_available=1 and populated fields. Those token values are labelled a plumbing demonstration, NOT a result - the oracle is a file copy, not an agent. The one generalisable figure is that 91.6% of input was cache reads, which is why the report leads with fresh (uncached) input. report.py turns two `msbench-cli report --output` files into the headline table. Written against the real msbench.report 2.0.0 schema read from the CLI source, not guessed. Design choices that keep the number honest: - It REFUSES to run on one arm. An absolute pass rate cannot distinguish an effective kit from an easy task. - Only `resolved is True` counts as a pass; the string "error" sentinel does not, or the published rate would be silently inflated. - Credits are divided by PASSES, not runs, so failed attempts are charged to the successes they paid for. - Instances with tokens_available=0 are excluded from cost means and reported separately, so a harvest failure cannot drag an average toward zero. - Arms with fewer than 3 instances are flagged as indicative. REPORT.md also documents where every number originates in the code, with commands to inspect each layer. 5 new tests pin the above. Verified: 133 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
…ained The reports label columns "treatment" and "control" without anywhere saying what the treatment IS. That is experimental-design jargon, and the audience for these numbers is a GTM one; a reader could reasonably assume the treatment was a different model, a different prompt, or a hinted run. It is none of those. The intervention is exactly one thing: THE AGENT KIT'S SKILLS ARE INSTALLED AND DISCOVERABLE. Task, prompt, API contract, model, agent, container and verifier are all identical between arms, which is what makes the difference attributable to the kit rather than to anything else. Implemented in shared/ces/runner.sh: SKILLS_ARM=kit copies the skills into the agent's skills directory, SKILLS_ARM=control installs nothing and deletes the directory defensively so a stale image layer cannot leak skills into the control. For the TOKEN comparison specifically, which is what prompted the question: the control has no skill files, so there is nothing to discover or read - its input is the prompt and its own work. The treatment reads a SKILL.md when it judges one relevant, and those bytes become input tokens. So the fresh-input delta is literally the price of the kit being available and used. Two caveats are now stated where the numbers appear, not buried: - Skills are CACHED (~92% of input was cache reads), so raw tokens_input badly overstates marginal cost; the report leads with fresh input. - "Skills add context, therefore skills cost more" is trivially true and uninteresting. The question is whether that context buys fewer attempts and more successes, which is why the headline is credits per PASSING RESULT. Added REPORT.md section 0 with the arm-by-arm table, made report.py's own output self-explanatory (a reader who skips the doc still sees what the columns mean), and mirrored the definition into the Loop B experiment config and docs/TESTING.md, which use the same terms. Documentation only. Verified: 133 passed; experiment matrix still resolves. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
…ter seeding The determinism suite failed roughly 2 full runs in 7 while passing every time in isolation. That pattern is the worst kind of test failure - it trains people to re-run until green - so it was chased to root cause rather than retried. Two distinct causes, and only one of them was a bug in our code. 1. index_health[].unused reads $indexStats access counters. Those are LIVE, the engine updates them ASYNCHRONOUSLY, and perf-advisor's own probe queries touch the very indexes it is reporting on. Whether run N observes run N-1's probes is therefore a race. Treated as a measurement and allowlisted; index_health[].redundant - a structural fact - is still asserted. 2. The bigger one, and NOT a bug in the scripts at all. The planner chose IXSCAN in one run and COLLSCAN in the next for the identical probe, so `collscans` membership and `collscan_patterns` moved (11 vs 12). perf-advisor was reporting exactly what the engine told it; the engine's answer legitimately changed as PostgreSQL's cost estimates improved. Immediately after a bulk seed the planner works from stale statistics, so a probe on a NON-LEADING index column (status, inside tenant_id_1_status_1) was costed as an index scan once autovacuum had analysed and as a sequential scan before that. Confirmed rather than assumed: on a freshly analysed database the same probe planned as IXSCAN 8 times out of 8. The fix belongs in the harness, not the tool. kit.seed() now runs ANALYZE after seeding, so every scenario measures against settled statistics - the same reason the MSBench verifier ANALYZEs after its bulk filler load. Best-effort: a failed ANALYZE is a missed optimisation, not a test failure. Verified by running the FULL suite 6 consecutive times with no failures, against 2 failures in 7 before the fix. The general lesson, now recorded in SCENARIO.md: when a determinism test flakes, first ask whether the TOOL is unstable or whether the ENVIRONMENT IT OBSERVES is. Only the former is a bug to fix in the tool. Verified: 133 passed x6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Asked what the judge is for the control/treatment quality comparison. Checked
rather than recalled, and the answer is worth writing down precisely.
THE JUDGE IS NOT A MODEL. The MSBench verifier makes zero model calls; the
judge is 30 deterministic pytest checks, and the arm-vs-arm quality signal is
the per-category pass rate (checks_<category>_passed / _total). That is a real
and reproducible quality measure - the naive-vs-oracle run shows it separating
12/12 on basic competence from 3/13 on best practice.
But it measures CONFORMANCE TO A RUBRIC WE WROTE, and the claim has to match:
supported "agents with the kit installed follow Azure DocumentDB best
practices more often" - precisely what the rubric encodes
NOT supported "the kit produces better solutions"
Three blind spots, now stated in the report rather than left implicit:
1. Ties are invisible - two submissions both at 31/31 may differ a lot in
readability, structure or error handling.
2. Unanticipated merit scores zero - the rubric rewards only what it names, so
a smarter approach we did not encode earns no credit.
3. The rubric is our opinion frozen into code. If a rule is wrong or
incomplete the benchmark measures the wrong thing, confidently, with a
reassuring number attached.
THE GAP THIS EXPOSED: there is currently NO qualitative assessment anywhere in
the kit. Loop A measures determinism, Loop C measures rubric conformance, and
Loop B - which I had designated as the place for judged quality - is
configured with skill-invocation graders only. It verifies that the right skill
FIRES, never whether the guidance was any good. A skill could trigger perfectly
and give poor advice and the eval would report a pass.
A judge does not belong in the verifier (offline container, no model access,
binary reward). It belongs in Loop B, which drives real models and supports
panel/prompt graders natively. Not yet configured; both files now say so
explicitly so no one quotes a routing number as an answer-quality number.
Documentation only. Verified: 133 passed; eval.yaml still lints.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Closes the gap identified last turn. Until now every measurement in the kit was
a conformance check - Loop A that scripts are stable, Loop B that the right
skill FIRES, Loop C that produced code satisfies 30 fixed rules. None could tell
good advice from bad. A skill could trigger perfectly, give poor guidance, and
report a pass.
evals/documentdb-quality/quality-eval.yaml scores the guidance itself with an
LLM judge panel. This is the one place in the kit where a judge is the right
instrument, because "was this explanation clear and correct" is genuinely a
matter of degree rather than a fact about the system.
A judge is easy to do badly, and a badly-built one produces confident numbers
that are wrong. Five guards, every one enforced by a test so it cannot quietly
regress:
3 judges, 3 vendors models favour their own output; no model may be the
sole judge of a run it could have produced
median, not mean one miscalibrated judge must not swing the verdict
blinded prompt a judge told which arm it is grading will find reasons
to agree; the prompt also may not leak experiment
vocabulary
anchored criteria 5 named dimensions with written descriptions and
weights, plus a per-stimulus rubric. "Rate 1-5" is not
a measurement.
correctness gates technical_correctness and documentdb_specificity are
`required`, and correctness outweighs every
presentational criterion, so a fluent, confident,
WRONG answer fails however well it reads
quality.experiment.yaml runs it treatment vs control. That control arm matters
more here than anywhere else: frontier models already answer MongoDB questions
well, so a treatment-only score of 4.2/5 proves nothing at all. Only the delta
against an unskilled agent supports "the kit makes the guidance better". If the
delta turns out small, that is a real finding - it would mean the kit's value is
in DocumentDB-specific correctness (which the documentdb_specificity gate
isolates) rather than in general answer quality.
Prompts deliberately do not name the solution - no "compound index", "singleton",
"TTL index" - or the judge would be grading instruction-following and both arms
would score alike. Enforced by test.
Verified: both specs lint; both experiment matrices resolve; the panel executes
end-to-end on the mock executor. Three mutations confirm the guards bite -
collapsing the panel to one vendor, un-gating correctness, and removing the
blinding instruction each fail their test.
Noted in the README that the mock executor is near-useless for THIS eval
specifically: it invokes no skills and cannot call judge models, so everything
scores 0 with zero tokens. It validates plumbing only.
Not yet run against real models. 10 new tests; 143 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Asked for reproduction instructions. Writing them surfaced a real gap first:
the naive control - whose 20/30 result is cited in the report as the headline
evidence that the grader discriminates - existed only in /tmp. Nobody else
could have reproduced it. Evidence that cannot be re-run is an assertion, not a
measurement.
Committed as controls/naive-python/ with a README explaining why a
working-but-wrong submission is the control that matters. An empty /app scoring
0 proves only that missing files are noticed; a grader that rejects nothing
else would be useless. The naive app implements every endpoint correctly,
persists to DocumentDB, rejects duplicates, and returns the right rows - it
passes 12/12 on competence and 3/13 on best practice. That gap is the
benchmark's entire reason to exist.
That fixture has already earned its keep: its per-request MongoClient, hidden
inside a coll() helper, is what exposed the blind spot in the first version of
the singleton check, which only inspected route-decorated functions.
verify-controls.sh reproduces all three in one command and exits non-zero if
any deviates. Verified by running it: oracle 31/31 reward 1, empty reward 0,
naive 20/30 reward 0 with the same per-category split the report publishes.
REPORT.md section 4 is now a staged guide rather than three commands:
1-2 Docker only build, then reproduce the grader validation
3 + msbench-cli[harbor] run through MSBench locally, NO ACR push, using
the harbor-native task_style
4-5 internal access publish, register both arms, run with --pass_at_k 5
6 - generate the effectiveness report
7 Copilot SDK auth guidance quality (Loop B, not MSBench)
CI nothing config guards on every PR
Each stage states what it needs and what it establishes, so a reader can stop
at the last one they have access to. Stages 1-2 are the ones worth running on
every verifier change: they are the only ones that answer "is this instrument
still trustworthy", and they cost nothing but time.
All four documented msbench-cli flags (--backend, --pass_at_k, --dataset,
--runner) were verified against the installed CLI rather than assumed.
3 new tests pin reproducibility, including one that fails if someone "fixes"
the naive control into following best practice - which would quietly destroy
the fixture's purpose.
Verified: 146 passed; all three controls reproduce exactly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
… a clean clone
Three gaps, found by auditing every claim in the document against the commands
provided for it.
1. A DUPLICATED HEADING. "## 5. Reading the numbers honestly" appeared twice on
one line - damage from a string replacement two commits ago that I did not
notice. Fixed, and the doc is now checked for duplicate headings.
2. THE TOKEN NUMBERS HAD NO REPRODUCTION COMMAND. Section 1 publishes
tokens_fresh_input, cache_read_share_pct and ai_credits from a session store
mounted into the task container, but never said how. Those figures were
therefore unverifiable by a reader. Both routes are now documented: the
in-container mount (which is what produced the published values) and the
standalone harvester. Verified: the standalone run reports 91.61% cache
share, consistent with the 91.6% in the report.
The note that omitting the mount yields tokens_available=0 with NO token
keys is included deliberately - that is the correct behaviour, and a reader
should be able to confirm a missing measurement is never reported as zero
cost.
3. NO SINGLE COPY-PASTE PATH. The staged guide is right for someone deciding
how far to go, but wrong for someone who just wants the numbers. Added a
TL;DR block covering everything that needs no internal access.
The TL;DR is not aspirational. I ran it from a fresh clone into /tmp:
git clone -> venv -> 50 config tests pass (no Docker, no credentials, ~1s)
-> bash build.sh -> both images built offline from vendored wheels
-> bash verify-controls.sh -> oracle 31/31 reward 1,
empty reward 0,
naive 20/30 reward 0
Every number in section 1 came back identical from a checkout that shared
nothing with my working tree.
REPORT.md now carries 13 runnable command blocks, all links resolve, and no
heading is duplicated.
Verified: 146 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Asked whether results should be committed, and how cosmosdb-agent-kit does it.
Checked their repo rather than assuming:
- They DO commit results: curated .md + .json batch summaries under
testing-v2/scenarios/*/batch-results/, carrying date, scenario, language,
skills-yes/no, iteration count and the originating issue/PR numbers.
- Raw run output is gitignored (results/).
- They commit NO MSBench results at all - benchmarks/cosmos-sdk-skills/ holds
only registration and task definition.
- Every committed batch file is named *-skills.*, i.e. TREATMENT ONLY. There
is no control counterpart, so a delta cannot be computed from what is in
that repo.
Adopted the curated-artifact pattern, and deliberately diverged on the last
point.
results/ now holds committed artifacts with two rules, both enforced by tests
rather than by convention:
1. ARMS COME IN PAIRS. A *-treatment.json with no matching *-control.json fails
the build. A treatment score with no control is not a result: 80% resolved
could mean an excellent kit or an easy task, and the number cannot tell you
which. Committing one arm alone is exactly how that gets quoted anyway.
2. PROVENANCE IS MANDATORY. Effectiveness results must carry run_id, date,
benchmark, image_tag, dataset_version, kit_commit, model and pass_at_k.
Without kit_commit you cannot say which skills the agent had; without model
you cannot compare like-for-like. A number missing these is a rumour with a
decimal point.
Also enforced: every .json has a readable .md companion, and the committed
controls-validation artifact must not drift from the numbers quoted in
REPORT.md - if someone updates one and forgets the other, the test fails.
verify-controls.sh gained --output, so the artifact is GENERATED by the run
rather than hand-written. The first one is committed:
results/2026-08-17-controls-validation.{json,md} - oracle 31/31 reward 1, empty
reward 0, naive 20/30 reward 0, kit_commit 3707cb1.
Raw MSBench dumps (msbench-runs/, *-raw.json, extracted-logs/) are gitignored;
only curated artifacts are checked in.
Mutation-verified: committing a lone treatment result fails the pairing test,
and a .json without its .md companion fails too.
Verified: 152 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
token-tests/ never calls a model. It counts bytes with wc -c and converts with
tokens ~= bytes/4, so its headline (52-97% saving, median 77%) is payload
arithmetic, not consumption - and it never checks whether either route reaches
the correct answer. benchmarks/documentdb-route-efficiency/ replaces it.
THE ARMS
control route-text the kit's text skills; scripts/ and knowledge-base/
REMOVED. Must run its own queries and interpret raw
output.
treatment route-script scripts/ + knowledge-base/; skills/ REMOVED. Routes
with kb-route.sh and reads a compact verdict.
The arm is expressed purely by what is on disk, the prompt is identical, and
neither arm is told which route it has. install-arm.sh deletes the other route
and ASSERTS it is gone - a leftover file would silently turn one arm into the
other and still produce a number, i.e. a confident comparison of an arm against
itself.
LEAKAGE CONTROLS (the part that most needed care)
fresh database per run $indexStats/idx_scan counters accumulate and the
redundancy finding depends on them. Running the
script arm first warms those counters and CHANGES
THE CORRECT ANSWER for the text arm. Each run gets
its own database, seeded from the same deterministic
fixture.
ANALYZE after seed stale statistics flip query plans; this caused a
real 2-in-7 flake in the deterministic suite.
randomised arm order prompt-cache warmth is not controllable, so it is
randomised (fixed seed) and cannot systematically
favour one arm.
fresh agent session no conversation carries between runs.
cleared output dir a stale finding.json would be graded as this run's
answer.
PARITY IS GRADED FIRST AND GATES COST
Both arms must reach the same finding. A route that is cheaper because it
answered worse has saved nothing, and the failing run is typically the cheapest,
so averaging it in would manufacture a saving out of a failure. summarize.py
computes cost means over parity-passing runs only, and warns if the cheaper
route is the less correct one.
Parity is structural, not judged: which indexes are redundant is a fact about
the database. The grader rejects missed findings, false positives, listing every
index (the obvious way to "win"), and right-names-with-no-mechanism, while
tolerating formatting differences so it grades correctness rather than prose.
The harness REFUSES to run without AGENT_CMD rather than falling back to
anything - numbers that look like measurements and are not would be worse than
no data.
token-tests/RESULTS.md now carries a SUPERSEDED banner explaining the three
effects a static proxy cannot see (caching, turn count, output/reasoning
tokens), which pull in different directions - so the measured result may well
disagree with the published 77%.
Also fixed a real defect: README linked to token-tests/RESULTS.md, which is
untracked and therefore missing from any clone, and described it as "measured"
when it is an estimate. Now points at the new benchmark.
NOT YET RUN - an agent-driven run needs model credentials. 14 new tests cover
the arm design, leakage controls, all six parity cases, and the
cost-excludes-failures rule. Verified: 166 passed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Two corrections and one unblock, both from questions I had not asked myself. 1. TOKENS ARE NOT COMPARABLE ACROSS MODELS Measured against published rates, output differs 2.5x between Gemini 3.1 Pro ($12/MTok) and GPT-5.6 Sol ($30/MTok), and input 2.5x ($2 vs $5). A cross-model table in raw tokens ranks tokenisers and verbosity, not cost. pricing.py converts each run at its own model's rate, itemising fresh input, cache reads and output, and REFUSES to price an unknown model rather than applying a default - a guessed rate produces a plausible dollar figure that nothing downstream could identify as invented. Rates retrieved 2026-08-17 from each provider's own documentation, not an aggregator, with the source URL and retrieval date attached to every priced result. Cache reads are ~90% cheaper across all three, which is precisely why the fresh/cached split matters: billing all input at the full rate overstates a cached skill payload by ~10x. 2. A CORRECTION I HAD TO MAKE MID-TASK I first concluded Copilot's AI credits were a premium-request metric only weakly tied to tokens (correlations 0.15-0.46) and therefore unusable as cost. That was a measurement error: I correlated RAW token counts with cost, but cached tokens bill ~10x cheaper, so tokens and cost are not proportional by construction. Pricing fresh and cached input separately, published rates and credits agree within ~3% (gemini 1.000, opus-5 0.995, opus-4.8 0.985, sol 0.968) - credits ARE token-based per-model billing at 1 credit = $0.01. The wrong explanation was already written into pricing.py's docstring; it is now corrected there rather than quietly deleted, because the error is instructive. A test cross-checks the two sources on live data and fails if they diverge, which would mean either a stale rate table or a billing change. 3. SUBAGENTS REMOVE THE CREDENTIAL BLOCKER The cross-model matrix was believed to need COPILOT_SDK_AUTH_TOKEN. It does not. Copilot CLI subagents give per-run isolation (own context window - clean context structurally, not by cleanup), model pinning, and exact attribution: usage lands in assistant_usage_events with a distinct non-null agent_id and exactly one model each, while the orchestrating parent's rows carry agent_id NULL and are excluded. Verified with a probe pinned to gemini-3.1-pro-preview: one row, agent_id=toolu_01MYbS4..., 6867 in / 3 out, $0.01377 computed - matching the 1.377 credits recorded. attribute.py implements snapshot/collect around that. Stated plainly in the README: a subagent is NOT the MSBench or Vally executor, so absolute numbers are not comparable to an MSBench run. They are comparable within this harness, which is all a cross-model or cross-arm comparison needs. 6 new tests: pricing coverage for every matrix model, the cached-rate discount, that models are genuinely not interchangeable on cost, long-context repricing (both Sol and Gemini reprice the WHOLE request above threshold), retrieval-date provenance, and the live credits cross-check. Verified: 172 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Lei Jiang (lionelc)
requested review from
Ritvik Jayaswal (Ritvik-Jayaswal),
Khelan Modi (khelanmodi) and
German Eichberger (xgerman)
August 19, 2026 02:19
Removing benchmarks/documentdb-sdk-skills/results/ broke the Loop C validate
job. The dependency was mine and it was backwards.
results/ holds DATA — the output of running the benchmark.
scenarios/benchmark-config validates CONFIGURATION — registration files, task
layout, verifier wiring. Coupling "is this benchmark correctly configured" to
"has someone run it and kept the output" is wrong:
- a fresh clone should validate. Confirmed by cloning: results/ is not in
git, so this test would have failed EVERY CI run, not just after cleanup.
- results are dated artifacts; pruning stale ones is housekeeping, not a
regression
- a PR touching only the verifier should not fail because results were tidied
Not every results/ test was wrong, and the distinction matters:
improper test_results_directory_documents_its_contract hard-required
results/README.md to exist
proper arm-pairing, provenance, and .md-companion checks only assert
things about results that DO exist, so they pass vacuously on an
empty or absent directory. Those are unchanged.
The README check now applies only once there are results to interpret, which
preserves its intent without making absence a failure.
The naming and provenance contract has moved into the benchmark README, so the
convention survives deletion of the data directory rather than living inside
it. Also fixed four links in docs/REPORT.md that pointed at the removed
artifacts; the grader-validation section now gives the regeneration command
instead of citing a specific dated file.
Verified: 172 passed, 2 skipped (both correctly skipping on absent results).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Collaborator
|
P2 — restore executable modes. This PR changes |
Collaborator
|
Suggestion (non-blocking) — consider more descriptive loop names. "Loop A / B / C" isn't a standard industry term, and the docs already have to gloss each one every time it's used (
Not a correctness issue, just readability for anyone new to the repo who hits "Loop A/B/C" without the surrounding context. |
Distinguishes structural redundancy from transient WRITE_TAX findings caused by asynchronous live usage counters on a newly created index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Adds a portable Node.js project, reusable generation prompt, deterministic ecommerce seed, one find query and three relational-style lookup aggregations. The unchanged project was run against MongoDB 7 and a fresh DocumentDB Local endpoint by changing only MONGODB_URI. Both passed four exact-result checks and produced byte-identical 2,420-byte output with SHA-256 c6d56dddb823ada7db9d2b47fbba97adb890dbed7dee2bff0c2d0a75f1dbad30. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Renames scenarios/ecommerce/mongodb-documentdb-compat to compat so the MongoDB-to-DocumentDB workflow is immediately visible and removes MongoDB from the folder name. The relocated workflow was rerun against MongoDB 7 and DocumentDB. Both passed four checks and produced byte-identical output with SHA-256 c6d56dddb823ada7db9d2b47fbba97adb890dbed7dee2bff0c2d0a75f1dbad30. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Removes the MongoDB server workflow and comparison from compat/. The project remains MongoDB-compatible through the official mongodb driver and standard find/aggregation syntax, but the full stack now runs only on DocumentDB. Simplifies PROMPT.md to six requirements and retains the exact DocumentDB result: four checks pass, including three lookup aggregations, with SHA-256 c6d56dddb823ada7db9d2b47fbba97adb890dbed7dee2bff0c2d0a75f1dbad30. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Adds a 2.2 MB generated dataset for transaction, change-stream, and advanced aggregation compatibility testing: 200 customers, 100 products, 300 inventory rows, 2,000 orders, 6,000 order items, 2,000 payments, 120 returns, a gapped 90-day sales series, and change-stream seed documents. The committed generation process includes four transaction edge cases, four ordered change-stream operations, seven advanced aggregation cases, index definitions, file hashes, invariant validation, a clean-state loader, and a two-generation byte-identity check. Validated by loading all files into DocumentDB Local: counts matched, date fields were BSON dates, declared indexes were created, and stale merge/change-stream output collections were removed before reload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86722878-7b4c-4d57-933c-3d4188317703
Add ps1 scripts and use python for routing
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
added 2 commits
September 21, 2026 18:34
Collaborator
Author
Names changed now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This one makes the DocumentDB agent-kit in parity with CosmosDB agent-kit by adding a test framework with benchmarks.
No token-tests yet -- will hold it for a separate PR.