Fix managed population simulations

[anth-volk]

Fixes #266
Fixes #264

Summary

• update policyengine.py from 5.2.0 to 5.3.0;
• let policyengine.py select and authenticate its certified default UK dataset, while UK Chat supplies only the presentation title Enhanced FRS 2024-25 and persists typed dataset provenance without exposing the private URI;
• validate each population aggregate's structure and value ranges, then give those validated outputs directly to the conversational model without whole-response numerical verification;
• use ordinary household_net_income consistently for both decile impacts and winner/loser groups by default, while retaining explicit equivalised-HBAI and wealth-decile options across initial calculations and follow-up reruns;
• retain equivalised HBAI household net income for poverty and inequality calculations and deterministically add the applicable income-definition disclosures when the model omits them;
• retain count-valued aggregates for calculation validation but instruct the conversational model not to report or derive any absolute person or household counts; gain/loss incidence is reported only as person-weighted percentages within named deciles, excluding the overall winner/loser row;
• scan completed calculation text for potentially political, normative, or evaluative keywords and request at most one text-only contextual review, with permission to retain technical or literal uses;
• constrain reform semantic targets to catalogue-returned parameter paths and treat generic phrases such as societal_impact as requests for the default population output profile;
• add a live population-simulation smoke test, three SQL-backed ten-user-turn population chat regressions including a household-to-population transition, focused derivative tests, and offline response evaluations;
• document the managed dataset, income-concept, population-validation, disclosure, and response-review contracts.

Production failures addressed

UK Chat previously passed an explicit Hugging Face dataset URI to policyengine.py's managed-data API. That API rejects explicit URIs unless unmanaged loading is deliberately enabled, so the population calculation stopped before dataset materialization. UK Chat now omits the dataset selector and lets policyengine.py select, verify, and materialize its certified default dataset. Unmanaged loading remains disabled.

The prior winner/loser calculation measured changes in ordinary household net income while grouping households with an HBAI-equivalised income-decile variable. The default calculation now measures and forms groups from ordinary household net income for both distributional outputs. Poverty and inequality keep their established equivalised HBAI definitions and responses state those definitions explicitly.

Verification

• strict backend mypy check — passed;
• focused population, derivative, response-finalization, and evaluation tests — 115 passed, 3 skipped;
• full backend suite under policyengine.py 5.3.0 — 477 passed, 19 skipped, 83.36% coverage;
• complete offline AI evaluation suite — 71 passed, 0 failed, 24 expected data/live-model skips;
• authenticated fresh-cache managed-data integration tests on policyengine.py 5.3.0 — 3 passed in 94.84 seconds, including native dataset selection and authentication, a real baseline/reform population calculation, and every population derivative adapter.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-uk-chat	Ready	Preview	Sep 2, 2026 12:34pm UTC

[github-actions]

Beta preview has been cleaned up because this PR was closed.

[codecov]

Codecov Report

❌ Patch coverage is 84.37500% with 75 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.08%. Comparing base (dbfd9f0) to head (ecf0477).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
backend/capabilities/society_outputs.py	82.66%	24 Missing and 19 partials ⚠️
backend/capabilities/policy_reform.py	68.51%	9 Missing and 8 partials ⚠️
backend/engine/py_runtime.py	80.64%	5 Missing and 1 partial ⚠️
backend/capabilities/follow_up.py	68.75%	4 Missing and 1 partial ⚠️
backend/capabilities/society.py	92.30%	1 Missing and 2 partials ⚠️
backend/chat/artifact_context.py	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #265      +/-   ##
==========================================
+ Coverage   73.41%   75.08%   +1.67%     
==========================================
  Files         152      154       +2     
  Lines       11403    11834     +431     
  Branches     1963     2020      +57     
==========================================
+ Hits         8371     8886     +515     
+ Misses       2309     2193     -116     
- Partials      723      755      +32     

Flag	Coverage Δ
backend	82.18% <84.37%> (+1.68%)	⬆️
frontend	37.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
backend/capabilities/artifacts.py	100.00% <100.00%> (ø)
backend/chat/capability_service.py	85.77% <100.00%> (+3.35%)	⬆️
backend/chat/model_port.py	84.61% <100.00%> (+1.63%)	⬆️
backend/chat/narration.py	85.93% <100.00%> (+2.00%)	⬆️
backend/conversation_context/tools.py	93.00% <ø> (ø)
backend/engine/constants.py	100.00% <ø> (+10.52%)	⬆️
backend/engine/derivatives.py	96.20% <100.00%> (ø)
backend/tools/analysis_support.py	95.08% <100.00%> (+0.08%)	⬆️
backend/tools/definitions.py	100.00% <ø> (ø)
backend/tools/dispatch.py	78.60% <ø> (+1.60%)	⬆️
... and 7 more

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.