From 1b9540a78e036e4b0dd76cb627151df78610f612 Mon Sep 17 00:00:00 2001 From: Igor Gentil Date: Mon, 25 May 2026 10:17:55 +1000 Subject: [PATCH] chore(openspec): archive add-testing-helpers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #24 (commit f0f9507) shipped the implementation: the `cfn_handler.testing` module with `replay()`, `Replay`, `make_event` / `make_context`, the three assertion helpers, and auto-discovered pytest fixtures, plus the soft-deprecation of `test_mode` / `last_response`. This archive: * Moves the change to `openspec/changes/archive/2026-05-25-add-testing-helpers/`, preserving proposal / design / tasks / spec for historical reference. * Promotes the delta spec to a new top-level capability at `openspec/specs/testing-helpers/spec.md` (verbatim copy of the ADDED requirements + a fresh Purpose section). `openspec validate testing-helpers --type spec --strict` passes. `openspec list` is now empty — no active changes. --- .../.openspec.yaml | 0 .../2026-05-25-add-testing-helpers}/design.md | 0 .../proposal.md | 0 .../specs/testing-helpers/spec.md | 0 .../2026-05-25-add-testing-helpers}/tasks.md | 34 +-- openspec/specs/testing-helpers/spec.md | 207 ++++++++++++++++++ 6 files changed, 218 insertions(+), 23 deletions(-) rename openspec/changes/{add-testing-helpers => archive/2026-05-25-add-testing-helpers}/.openspec.yaml (100%) rename openspec/changes/{add-testing-helpers => archive/2026-05-25-add-testing-helpers}/design.md (100%) rename openspec/changes/{add-testing-helpers => archive/2026-05-25-add-testing-helpers}/proposal.md (100%) rename openspec/changes/{add-testing-helpers => archive/2026-05-25-add-testing-helpers}/specs/testing-helpers/spec.md (100%) rename openspec/changes/{add-testing-helpers => archive/2026-05-25-add-testing-helpers}/tasks.md (92%) create mode 100644 openspec/specs/testing-helpers/spec.md diff --git a/openspec/changes/add-testing-helpers/.openspec.yaml b/openspec/changes/archive/2026-05-25-add-testing-helpers/.openspec.yaml similarity index 100% rename from openspec/changes/add-testing-helpers/.openspec.yaml rename to openspec/changes/archive/2026-05-25-add-testing-helpers/.openspec.yaml diff --git a/openspec/changes/add-testing-helpers/design.md b/openspec/changes/archive/2026-05-25-add-testing-helpers/design.md similarity index 100% rename from openspec/changes/add-testing-helpers/design.md rename to openspec/changes/archive/2026-05-25-add-testing-helpers/design.md diff --git a/openspec/changes/add-testing-helpers/proposal.md b/openspec/changes/archive/2026-05-25-add-testing-helpers/proposal.md similarity index 100% rename from openspec/changes/add-testing-helpers/proposal.md rename to openspec/changes/archive/2026-05-25-add-testing-helpers/proposal.md diff --git a/openspec/changes/add-testing-helpers/specs/testing-helpers/spec.md b/openspec/changes/archive/2026-05-25-add-testing-helpers/specs/testing-helpers/spec.md similarity index 100% rename from openspec/changes/add-testing-helpers/specs/testing-helpers/spec.md rename to openspec/changes/archive/2026-05-25-add-testing-helpers/specs/testing-helpers/spec.md diff --git a/openspec/changes/add-testing-helpers/tasks.md b/openspec/changes/archive/2026-05-25-add-testing-helpers/tasks.md similarity index 92% rename from openspec/changes/add-testing-helpers/tasks.md rename to openspec/changes/archive/2026-05-25-add-testing-helpers/tasks.md index b610833..39f66d2 100644 --- a/openspec/changes/add-testing-helpers/tasks.md +++ b/openspec/changes/archive/2026-05-25-add-testing-helpers/tasks.md @@ -236,33 +236,21 @@ during pytest plugin collection (before `--cov` instrumentation attaches), making module-level lines look unhit. Documented in the recipe comment. -- [ ] 12.2 Run `just gha-pre-release` to replay every CI gating - workflow locally. All green required before merge. **Defer to - pre-merge step.** +- [x] 12.2 Run `just gha-pre-release` to replay every CI gating + workflow locally. All green required before merge. **Done: + ran on the PR branch; CodeQL flagged as a false-failure under + act (post-analysis REST API call to a synthesized run id 404s), + fixed by removing CodeQL from the recipe with rationale (commit + `5df0b1f`). Real GH Actions CodeQL on PR #24 passed.** - [x] 12.3 Verify the built wheel includes `cfn_handler/testing/`: `uv build && unzip -l dist/*.whl | grep testing`. Verified — 9 files including `_internal/` modules and `py.typed`. -- [ ] 12.4 Verify the conventional-commit message for the squash-merge +- [x] 12.4 Verify the conventional-commit message for the squash-merge starts with `feat(testing):` so release-please bumps minor - (target: `1.3.0`). **Squash-merge commit message guidance:** - ``` - feat(testing): add cfn_handler.testing module with replay() helpers - - Adds the new `cfn_handler.testing` public surface: - - `CustomResource.replay(event, context=None)` — in-process dispatch - returning a structured `Replay` (no HTTP, no boto3). - - `Replay` frozen dataclass. - - `make_event` / `make_context` factories with safe defaults. - - `assert_success` / `assert_failed` / `assert_deferred` helpers. - - pytest fixtures (`cfn_create_event`, `cfn_update_event`, - `cfn_delete_event`, `cfn_lambda_context`) auto-discovered via - the `pytest11` entry point. - - DEPRECATED: `CustomResource(test_mode=True)` and `last_response` - now emit a DeprecationWarning. They continue to work in v1.x; - removal scheduled for v2.0. - ``` - **Defer to merge step.** + (target: `1.3.0`). **Done: PR #24 merged as + `feat(testing): add cfn_handler.testing module with replay() and helpers (#24)` + on commit `f0f9507`. release-please will open the v1.3.0 + release PR on next workflow trigger.** ## 13. Validation diff --git a/openspec/specs/testing-helpers/spec.md b/openspec/specs/testing-helpers/spec.md new file mode 100644 index 0000000..53e9ca6 --- /dev/null +++ b/openspec/specs/testing-helpers/spec.md @@ -0,0 +1,207 @@ +# testing-helpers Specification + +## Purpose + +Provide a public testing surface (`cfn_handler.testing`) that lets users +unit-test custom-resource handlers in-process — without HTTP, without +`boto3`, without `moto` — and without reaching into the +`cfn_handler._internal` namespace whose contract is explicitly unstable. + +The capability is built around `CustomResource.replay(event, context)`, +which executes the full dispatch pipeline (handler resolution, handler +invocation, polling deferral) and returns a structured `Replay` value +capturing the response payload that *would* have been sent. Polling is +stubbed so a deferred replay mutates the event with the same marker +keys real polling adds, allowing a follow-up `replay()` to drive the +poll handler without provisioning EventBridge rules. + +Adjacent surfaces — event/context factories, assertion helpers, and +auto-discovered pytest fixtures — round out the kit so users can write +TDD-shaped handler tests with minimal boilerplate. + +This capability is intentionally additive to the `lifecycle-handler` +and `polling` capabilities: production behaviour is unchanged. The +seams that make replay possible are private implementation details +documented in the change archive (`openspec/changes/archive/`). + +## Requirements + +### Requirement: Public testing module is importable + +The library SHALL expose a `cfn_handler.testing` module importable in any Python environment where `cfn_handler` itself imports cleanly, without requiring `pytest`, `boto3`, or any other optional dependency. + +#### Scenario: Module imports without pytest installed +- **WHEN** a user runs `import cfn_handler.testing` in an environment + where pytest is not installed +- **THEN** the import succeeds and the public names (`Replay`, + `make_event`, `assert_success`, `assert_failed`, `assert_deferred`) + are available + +#### Scenario: Module imports without boto3 installed +- **WHEN** a user runs `import cfn_handler.testing` in an environment + where boto3 is not installed +- **THEN** the import succeeds and `Replay` / `make_event` / assertion + helpers are available + +### Requirement: In-process replay of the dispatch flow + +`CustomResource` SHALL expose a `replay(event, context=None)` method that executes the full dispatch pipeline in-process and returns a `Replay` object capturing the outcome, without issuing HTTP requests, importing `boto3`, or mutating the registered handler functions. + +#### Scenario: Successful create handler is replayed +- **WHEN** a `CustomResource` has a CREATE handler registered that + returns `{"Endpoint": "https://x"}`, and `replay(create_event)` is + invoked +- **THEN** the returned `Replay` has `status="SUCCESS"`, + `data={"Endpoint": "https://x"}`, and `payload` is the rendered + CFN response payload that would have been PUT to the response URL + +#### Scenario: Handler raises during replay +- **WHEN** a CREATE handler raises `RuntimeError("boom")` during + replay +- **THEN** the returned `Replay` has `status="FAILED"` and `reason` + contains `"boom"` + +#### Scenario: Replay does not perform HTTP I/O +- **WHEN** `replay()` is invoked with a valid event whose `ResponseURL` + is `https://example.invalid/cfn-response` +- **THEN** no HTTP request is made to any URL during the call + +#### Scenario: Replay does not import boto3 +- **WHEN** `replay()` is invoked in an environment without boto3 + installed AND no poll handler is registered +- **THEN** the call completes successfully without raising + `PollingDependencyError` or `ImportError` + +### Requirement: Replay produces a structured result + +The `Replay` type SHALL be a frozen, immutable dataclass with the fields `status` (literal `"SUCCESS" | "FAILED" | "DEFERRED"`), `physical_resource_id` (`str | None`), `data` (`dict[str, Any]`), `reason` (`str`), `no_echo` (`bool`), `payload` (`dict[str, Any]`), and `request_type` (literal `"Create" | "Update" | "Delete"`). + +#### Scenario: Replay result is immutable +- **WHEN** a user attempts to mutate `replay.status = "FAILED"` after + a SUCCESS replay +- **THEN** `dataclasses.FrozenInstanceError` is raised + +#### Scenario: Replay payload matches what would be sent +- **WHEN** `replay()` returns a `Replay` with `status="SUCCESS"` and + `data={"Endpoint": "x"}` +- **THEN** `replay.payload["Status"] == "SUCCESS"`, + `replay.payload["Data"] == {"Endpoint": "x"}`, and the payload + conforms to the CFN custom-resource response schema + +### Requirement: Replay supports the polling-deferral case + +`replay()` SHALL handle the polling-deferral path without invoking any AWS API or importing `boto3`: when a matching poll handler is registered, it MUST return a `Replay` with `status="DEFERRED"` and an empty `payload` dict, and MUST mutate the input event to add the polling marker keys (`CfnHandlerPoll`, `CfnHandlerRule`, `CfnHandlerPermission`) so a subsequent `replay()` call resumes into the poll handler path. + +#### Scenario: Create with poller defers +- **WHEN** a `CustomResource` has both `@create` and `@poll_create` + handlers registered, and `replay(create_event)` is invoked +- **THEN** the returned `Replay` has `status="DEFERRED"`, no AWS API + call is made, and the input event has been mutated to include + `event["CfnHandlerPoll"] is True` + +#### Scenario: Poll re-invocation completes the flow +- **WHEN** a deferred event is replayed a second time, and the + registered poll handler returns response data +- **THEN** the returned `Replay` has `status="SUCCESS"` and the + data the poll handler provided + +### Requirement: Event factory produces canonical CFN events + +The library SHALL expose a `make_event` callable in `cfn_handler.testing` that returns a dict matching the documented CloudFormation custom-resource event shape, with keyword overrides for every documented field, and MUST require a non-`None` `physical_resource_id` argument when `RequestType` is `"Update"` or `"Delete"` (raising `ValueError` if not supplied). + +#### Scenario: Default Create event is well-formed +- **WHEN** `make_event()` is called with no arguments +- **THEN** the returned dict has `RequestType="Create"`, + syntactically valid `StackId`, `RequestId`, `LogicalResourceId`, + `ResourceType`, `ResourceProperties`, `ResponseURL`, `ServiceToken` + fields, and no `PhysicalResourceId` + +#### Scenario: Update event requires PhysicalResourceId +- **WHEN** `make_event(request_type="Update")` is called without + passing `physical_resource_id` +- **THEN** `ValueError` is raised with a message identifying the + missing argument + +#### Scenario: Field overrides are applied +- **WHEN** `make_event(resource_properties={"Foo": "bar"})` is + called +- **THEN** the returned dict has `ResourceProperties == {"Foo": "bar"}` + +#### Scenario: Defaults use safe placeholder values +- **WHEN** `make_event()` is called with no overrides +- **THEN** the `ResponseURL` host is `example.invalid` (RFC 6761 + reserved name guaranteed not to resolve) and the account ID portion + of `StackId` is `111111111111` (AWS-reserved example account) + +### Requirement: Lambda context factory satisfies the protocol + +The library SHALL expose a `make_context` callable in `cfn_handler.testing` that returns an object satisfying the existing `LambdaContext` protocol used by `CustomResource.__call__`, exposing `aws_request_id`, `function_name`, `invoked_function_arn`, `log_group_name`, `log_stream_name`, and `get_remaining_time_in_millis()`. + +#### Scenario: Context satisfies the protocol +- **WHEN** `ctx = make_context()` is called and used in + `resource.replay(event, ctx)` +- **THEN** the call succeeds and `ctx.get_remaining_time_in_millis()` + returns a positive integer + +#### Scenario: Remaining-time override is honoured +- **WHEN** `make_context(remaining_time_ms=5000)` is called +- **THEN** `ctx.get_remaining_time_in_millis()` returns `5000` + +### Requirement: Assertion helpers raise informative AssertionError + +The library SHALL expose `assert_success`, `assert_failed`, and `assert_deferred` helpers in `cfn_handler.testing`, each of which MUST raise `AssertionError` with a message identifying both the expected and actual values when the assertion fails. + +#### Scenario: assert_success on a SUCCESS replay passes +- **WHEN** `assert_success(replay, data={"x": 1})` is called and + `replay.status == "SUCCESS"` and `replay.data == {"x": 1}` +- **THEN** the call returns `None` (no exception) + +#### Scenario: assert_success on a FAILED replay raises +- **WHEN** `assert_success(replay)` is called and + `replay.status == "FAILED"` with `reason="boom"` +- **THEN** `AssertionError` is raised and the message contains both + `"FAILED"` and `"boom"` + +#### Scenario: assert_failed with reason_contains matches a substring +- **WHEN** `assert_failed(replay, reason_contains="boom")` is called + and `replay.status == "FAILED"` with `reason="something boom happened"` +- **THEN** the call returns `None` + +#### Scenario: assert_deferred on a SUCCESS replay raises +- **WHEN** `assert_deferred(replay)` is called and + `replay.status == "SUCCESS"` +- **THEN** `AssertionError` is raised + +### Requirement: pytest fixtures auto-register via entry point + +The library's `pyproject.toml` SHALL declare a `pytest11` entry point named `cfn_handler` pointing at the fixtures module so that the fixtures `cfn_create_event`, `cfn_update_event`, `cfn_delete_event`, and `cfn_lambda_context` are available without any user-side `pytest_plugins` declaration. + +#### Scenario: Fixture is auto-discovered +- **WHEN** a user with `cfn_handler` installed writes a test + `def test_x(cfn_create_event): ...` in a fresh pytest project + with no `conftest.py` configuration +- **THEN** pytest resolves the fixture without error and passes a + Create-shaped event dict + +#### Scenario: Each invocation gets a fresh event +- **WHEN** two tests both consume `cfn_create_event` and one mutates + the event dict +- **THEN** the second test sees the unmutated default event (no + cross-test leak) + +### Requirement: Replay never sends a real CFN response + +`CustomResource.replay` SHALL NOT, under any code path, send an HTTP request to the event's `ResponseURL` or any other URL, and the production HTTP transport MUST be replaced by an in-memory capture for the duration of the replay call and restored when the call returns or raises. + +#### Scenario: Replay catches a handler exception without sending HTTP +- **WHEN** `replay()` is invoked with a handler that raises during + execution +- **THEN** the returned `Replay` has `status="FAILED"` AND no HTTP + request was issued (verified via mock or instrumentation) + +#### Scenario: Replay restores transport after exception +- **WHEN** `replay()` raises an unexpected internal exception (not a + handler exception) and the same `CustomResource` instance is then + invoked normally via `__call__` (with the production HTTP transport) +- **THEN** the production invocation correctly issues an HTTP PUT to + the event's `ResponseURL`