Skip to content

Document month-keyed YAML test outputs - #81

Merged
hua7450 merged 1 commit into
mainfrom
yaml-month-keyed-outputs
Aug 8, 2026
Merged

Document month-keyed YAML test outputs#81
hua7450 merged 1 commit into
mainfrom
yaml-month-keyed-outputs

Conversation

@hua7450

@hua7450 hua7450 commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

The testing guidance says the YAML runner supports only YYYY and YYYY-01, and concludes a mid-year month cannot be tested. The restriction is real but applies to the case period:; keying the output by month inside a whole-year period asserts any month.

- name: Case 41, PA waiver starts in September 2024.
  period: 2024
  input:
    state_code: PA
    county_fips: "42005"
  output:
    is_in_snap_abawd_waived_area:
      2024-08: false
      2024-09: true

Fixes #80

Why this matters

The old advice does not just omit a technique, it recommends a worse test. references/tests.md said to "test with the first month of a year where it's fully active"; test-creator said to use the next January after the effective date. Both yield a case that passes whether or not the mid-year boundary is modeled — a silently vacuous test rather than a failing one. Asserting the months on either side of a change is what proves the boundary sits where you think it does.

It also pushed authors toward hand-built Simulation objects for something the runner handles natively. That happened in PolicyEngine/policyengine-us#9243: a Python test file was written for an April 1 boundary, and the PR description asserted on the strength of this guidance that the boundary "cannot be expressed there." Both have since been corrected.

Mid-year boundaries are common in benefit rules — state BBCE standards re-basing off the federal October cycle, ABAWD waiver windows, mid-year rate changes — so this is not a rare corner.

Changes

  • references/tests.md — scopes the restriction to period:, adds an "Asserting a specific month" section with the pattern, when to prefer it, and the month-naming failure message. Keeps the existing (correct) statement about which period: values fail.
  • test-creator — same correction at the point of use, and clarifies that month keys under output: are not what its period: self-check targets, so the agent does not "fix" valid tests.

The mid-year January fallback is retained, rescoped to cases that are not asserting the boundary.

Verification

Per authoring rule 1, confirmed by execution rather than memory. On tanf_non_cash_gross_income_limit in policyengine-us: 14 cases pass with month-keyed outputs, and deliberately corrupting one month fails with

tanf_non_cash_gross_income_limit@2026-04: [2660.] differs from 9999.0 with an absolute margin > 0.01

confirming the assertions evaluate rather than being parsed and ignored — the failure mode that would otherwise make this change worthless.

Test plan

  • uv run pytest --ignore=tests/test_skill_examples.py — 99 passed
  • python3 scripts/build_claude_wrapper.py — builds clean
  • CI passes

The testing guidance said the runner supports only YYYY and YYYY-01,
and concluded that a mid-year month cannot be tested. The first half is
true of the case period; the conclusion is not. Keying the output by
month inside a whole-year period asserts any month, and the pattern is
already used in policyengine-us.

The old advice steered toward a test that passes whether or not the
boundary is modeled -- "use the first January where it's fully active"
never exercises the change -- and toward hand-built Simulation objects
in Python for something the YAML runner handles natively.

Scope the restriction to period:, document the month-keyed form, and
note in test-creator that month keys under output: are not what its
period: self-check targets.

Verified by execution per authoring rule 1: 14 cases pass on
tanf_non_cash_gross_income_limit, and corrupting one month fails naming
that month, so the assertions evaluate rather than being ignored.

Fixes #80

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hua7450
hua7450 merged commit 9185c20 into main Aug 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Testing guidance wrongly says mid-year months cannot be asserted in YAML tests

1 participant