Document month-keyed YAML test outputs - #81
Merged
Conversation
The testing guidance said the runner supports only YYYY and YYYY-01, and concluded that a mid-year month cannot be tested. The first half is true of the case period; the conclusion is not. Keying the output by month inside a whole-year period asserts any month, and the pattern is already used in policyengine-us. The old advice steered toward a test that passes whether or not the boundary is modeled -- "use the first January where it's fully active" never exercises the change -- and toward hand-built Simulation objects in Python for something the YAML runner handles natively. Scope the restriction to period:, document the month-keyed form, and note in test-creator that month keys under output: are not what its period: self-check targets. Verified by execution per authoring rule 1: 14 cases pass on tanf_non_cash_gross_income_limit, and corrupting one month fails naming that month, so the assertions evaluate rather than being ignored. Fixes #80 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The testing guidance says the YAML runner supports only
YYYYandYYYY-01, and concludes a mid-year month cannot be tested. The restriction is real but applies to the caseperiod:; keying the output by month inside a whole-year period asserts any month.Fixes #80
Why this matters
The old advice does not just omit a technique, it recommends a worse test.
references/tests.mdsaid to "test with the first month of a year where it's fully active";test-creatorsaid to use the next January after the effective date. Both yield a case that passes whether or not the mid-year boundary is modeled — a silently vacuous test rather than a failing one. Asserting the months on either side of a change is what proves the boundary sits where you think it does.It also pushed authors toward hand-built
Simulationobjects for something the runner handles natively. That happened in PolicyEngine/policyengine-us#9243: a Python test file was written for an April 1 boundary, and the PR description asserted on the strength of this guidance that the boundary "cannot be expressed there." Both have since been corrected.Mid-year boundaries are common in benefit rules — state BBCE standards re-basing off the federal October cycle, ABAWD waiver windows, mid-year rate changes — so this is not a rare corner.
Changes
references/tests.md— scopes the restriction toperiod:, adds an "Asserting a specific month" section with the pattern, when to prefer it, and the month-naming failure message. Keeps the existing (correct) statement about whichperiod:values fail.test-creator— same correction at the point of use, and clarifies that month keys underoutput:are not what itsperiod:self-check targets, so the agent does not "fix" valid tests.The mid-year January fallback is retained, rescoped to cases that are not asserting the boundary.
Verification
Per authoring rule 1, confirmed by execution rather than memory. On
tanf_non_cash_gross_income_limitin policyengine-us: 14 cases pass with month-keyed outputs, and deliberately corrupting one month fails withconfirming the assertions evaluate rather than being parsed and ignored — the failure mode that would otherwise make this change worthless.
Test plan
uv run pytest --ignore=tests/test_skill_examples.py— 99 passedpython3 scripts/build_claude_wrapper.py— builds clean