WIP: test(docs): run every tutorial end to end against a live backend - #63
Open
samuelhel wants to merge 3 commits into
Open
WIP: test(docs): run every tutorial end to end against a live backend#63samuelhel wants to merge 3 commits into
samuelhel wants to merge 3 commits into
Conversation
The doc pages had drifted out of sync with SDK 0.2.0 and nothing caught it, because nothing ran them. This adds a suite that does: it assembles each page's own fenced code into a program, runs it against a real stack, and then asks the backend whether the page's promise came true. The code always comes from the page, never a copy, so there is no second copy to drift and nowhere to "fix" a red test except the documentation. Covers all 71 pages that contain runnable code (Python; Java and Rust runners are implemented and selectable with --langs). 205 of 232 checks pass; the 27 failures are real defects, each verified by reproducing it with plain SDK calls outside the harness before it was believed. What makes it hold up: - Block-count pinning. Plans address blocks by index, so an edited page fails the plan instead of silently re-pointing it at different code. - Bounded-run substitutions that must keep matching, so a rewritten loop fails loudly rather than hanging CI. - A coverage gate: a new page with runnable code is red until someone writes a plan or records why it cannot run. UNTRIAGED.toml is empty and should stay so. - Assertions on the backend, not on exit codes. Several tutorials catch their own exceptions by design, so exiting 0 proves nothing about whether data landed. - Sweeps before and after each run, so the suite is repeatable rather than passing only against a virgin backend. - 18 harness self-tests (test_harness.py) that prove those guards actually fire. They need no backend, so CI can run them anywhere. Live listeners are bounded and fed real traffic rather than skipped, graph reads are settled before assertions, and pages needing credentials or optional packages skip with the reason instead of failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoqhJA3haV5BqiqHvhPBZf
Adds the workflow and the two backend-free tiers it leans on, so the suite is useful before anyone has stood up a stack for CI. - test_api_surface.py: checks every SDK name and service method the docs use against the built SDK. Needs a build, not a backend. This is the tier that would have caught the 0.2.0 rename and the removal of BasicEventFilter in the pull request that did them, rather than months later. It reads the service classes off a locally-constructed client, since they are not exported at module level, and it treats a namespace package as missing — a stale empty `datahub_sdk/` directory answers `find_spec` while importing nothing, which is precisely how the rename stayed invisible. - doc-tutorials.yml: three jobs. `structure` needs only pytest and runs in under a second. `api-surface` builds the SDK from the ref under test. `tutorials` runs the live suite when DOCTEST_BASE_URL is set, and posts a notice instead of failing when it is not — a missing backend is not a broken tutorial. The workflow accepts `repository_dispatch` (type `sdk-updated`) carrying an `sdk_ref`, so the SDK repo can have its changes checked against the docs. Also fixes run.sh, which prepended the whole suite even when given a specific file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoqhJA3haV5BqiqHvhPBZf
Review pass over the suite. Three kinds of repetition, none of them harmless:
- conftest's seeding fixture had its own copy of the walk from plan to runnable
program — resolve the chain, read the blocks, compose. It had already drifted:
only the tutorial path validated the plan against the page it was about to run,
so a stale selection in a seeding page would have gone unnoticed. Both now go
through scenario.build(), which validates once and is the only place that knows
how a plan becomes a program.
- The same two calls answered "did the page do what it promised" in two places.
Now backend.unmet_expectations().
- The two eventual-consistency waits in tutorial_support were the same poll loop
written twice.
And 444 lines out of the plans. Sixty-five carried
[java]
disabled = "Java scenario not written yet."
which means precisely what omitting the section already means — Plan.lang() falls
back to a disabled scenario — while burying the `disabled` reasons that are
specific. The [blocks] table still pins each language's fence count, so nothing is
lost. Twenty-seven repeated a three-line explanation of what a Recorder is, which
belongs in the skill and is there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoqhJA3haV5BqiqHvhPBZf
samuelhel
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The doc pages had drifted out of sync with SDK 0.2.0 and nothing caught it, because nothing ran them. This adds a suite that does: it assembles each page's own fenced code into a program, runs it against a real stack, and then asks the backend whether the page's promise came true.
The code always comes from the page, never a copy, so there is no second copy to drift and nowhere to "fix" a red test except the documentation.
Covers all 71 pages that contain runnable code (Python; Java and Rust runners are implemented and selectable with --langs). 205 of 232 checks pass; the 27 failures are real defects, each verified by reproducing it with plain SDK calls outside the harness before it was believed.
What makes it hold up:
Live listeners are bounded and fed real traffic rather than skipped, graph reads are settled before assertions, and pages needing credentials or optional packages skip with the reason instead of failing.
Signed-off-by: samuel samuel@intellistream.ai