Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 35 additions & 3 deletions skills/policyengine-model-development/references/tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,44 @@ mid-file and renumber (noisy diffs).

## Period restrictions

Only two period formats are supported:
The case-level `period:` accepts only two formats:
- `2024` — whole year
- `2024-01` — first month **only**

`2024-04`, `2024-10`, and full dates like `2024-01-01` all fail. For a mid-year policy change, test
with the first month of a year where it's fully active (e.g. `2025-01`).
`2024-04`, `2024-10`, and full dates like `2024-01-01` all fail **as the case period**.

### Asserting a specific month

That restriction applies to `period:`, not to what you can assert. To check a mid-year month, use a
whole-year period and key the **output** by month:

```yaml
- name: Case 41, PA waiver starts in September 2024.
period: 2024
input:
state_code: PA
county_fips: "42005"
output:
is_in_snap_abawd_waived_area:
2024-08: false
2024-09: true
```

Any month works here, and inputs stay plain scalars because the period itself is still a year. This
is the right tool whenever a MONTH-defined variable changes mid-year — a benefit standard that
re-bases in April, a waiver that starts in September, a rate that changes on a state's own schedule.
Asserting the months on either side of the change is what proves the boundary is where you think it
is; a single January case would pass just as happily if the change were ignored entirely.

Reach for a hand-built `Simulation` in a Python test only when you need something the YAML runner
genuinely cannot express. A mid-year boundary is not that — writing one is a sign this pattern was
overlooked.

Failures name the month, so a broken expectation is easy to place:

```
tanf_non_cash_gross_income_limit@2026-04: [2660.] differs from 9999.0 with an absolute margin > 0.01
```

## Error margins by output type

Expand Down
22 changes: 19 additions & 3 deletions targets/claude/agents/country-models/test-creator.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,29 @@ Creates comprehensive integration tests for government benefit programs based on

## CRITICAL: Test Period Format

**Use `YYYY-01` or `YYYY` ONLY.** PolicyEngine's YAML test system does not support any other month or date-with-day format — tests using them WILL fail. This applies regardless of the variable's `definition_period` (YEAR or MONTH).
**Use `YYYY-01` or `YYYY` ONLY** as the case `period:`. PolicyEngine's YAML test system does not support any other month or date-with-day format there — tests using them WILL fail. This applies regardless of the variable's `definition_period` (YEAR or MONTH).

- ✅ `2024-01` (first month) or `2024` (whole year)
- ❌ `2024-02`, `2024-03`, `2024-04`, `2024-05`, `2024-06`, `2024-07`, `2024-08`, `2024-09`, `2024-10`, `2024-11`, `2024-12` — **WILL FAIL**
- ❌ `2024-01-15` or any date-with-day format — **WILL FAIL**

**When the policy is effective mid-year — use the NEXT January AFTER the effective date.**
**This restricts `period:`, not what you can assert.** To check a specific month, use a whole-year period and key the output by month:

```yaml
- name: Case 41, PA waiver starts in September 2024.
period: 2024
input:
state_code: PA
county_fips: "42005"
output:
is_in_snap_abawd_waived_area:
2024-08: false
2024-09: true
```

Any month works, and inputs stay plain scalars because the period is still a year. Prefer this whenever a MONTH-defined variable changes mid-year: asserting the months on either side is what proves the boundary sits where you think it does. A January-only case would pass just as happily if the change were ignored.

**When the policy is effective mid-year and you are NOT asserting the boundary — use the NEXT January AFTER the effective date.**

PolicyEngine resolves annual parameters at the START of the period. `period: 2024` and `period: 2024-01` both look up at 2024-01-01. If the policy only becomes effective later in 2024, those periods return the OLD pre-effective value — your test will silently assert against the wrong parameter, or pressure the implementer to backdate the new value just to make the test pass.

Expand All @@ -30,7 +46,7 @@ Always pick the first January **on or after** the effective date:
| July 1, 2024 | `2025-01` | `2024` resolves to 2024-01-01 (pre-July) |
| October 1, 2023 | `2024-01` or `2024` | Lookup at 2024-01-01 is post-October |

**Self-check before saving every test file:** search for any `period:` value that is not `YYYY` or `YYYY-01`. Fix before writing.
**Self-check before saving every test file:** search for any `period:` value that is not `YYYY` or `YYYY-01`. Fix before writing. Month keys under `output:` are fine and are not what this check is looking for.

## Load the consolidated skill first

Expand Down