Add the UK transport and energy source packages issue 257 asks for

[juaristi22]

Closes #257.

Every gap PR #255 declared, plus the bus-use incidence family requested in #257 item 6. (Earlier revisions of this body cited "the Microcosm #890 plan" for that family; #890's issue body does not mention incidence and the plan is an uncommitted working file, so #257 item 6 is the request of record.) Six new packages, 6,012 new facts, plus a restructure of the ONS 07CN record sets.

1. Ofgem cap history

ofgem-energy-price-cap-levels-2024-2026, 3,636 facts, from the Annex 9 levelised cap-rates model (v1.11). Every charge restriction period from 2024-Q1 to 2026-Q4, the fourteen charge restriction regions plus the GB average, three payment methods, electricity single-rate and multi-register and gas, at nil and benchmark consumption. The landed ofgem-energy-price-cap-q1-2024 package stays as it is — four GB-average direct-debit rates from a news page — and this one carries the matrix.

Ofgem does not publish a p/kWh unit rate or a p/day standing charge at this grain, so this package does not claim to. Its cap-level annexes state the cap as a benchmark maximum charge in pounds per customer per year, at two consumption levels: nil consumption, which is the annual standing charge (Ofgem labels the same regional values "Annual … Standing Charge" in its monthly levelisation-rates workbook), and benchmark consumption. p/kWh and p/day exist only as GB averages for the current and previous cap period, on the consumer page and in the quarterly news release. Converting the levels is a consumer transformation, so the levels are emitted as published:

standing charge (p/day) = nil / 365 * 100
unit rate (p/kWh)       = (benchmark - nil) / k * 100

The subtraction is the step to get right — the benchmark level already contains the standing charge, so dividing it by k alone gives 34.90 p/kWh for 2024-Q1 GB direct-debit electricity against the 28.62 Ofgem published. A test reconstructs all four rates in the earlier ofgem-energy-price-cap-q1-2024 package from these levels, to the rounding the news page printed.

The benchmark consumption is not constant across the window, which is the trap in that arithmetic: 3,100 kWh single-rate electricity / 4,200 multi-register / 12,000 gas to December 2025, then 2,700 / 3,900 / 11,500, then 2,500 / 3,400 / 9,500 from July 2026. The workbook's Notes sheet publishes that schedule with its effective dates, so it is emitted as 36 facts of its own under ofgem.price_cap.benchmark_consumption, one per cap period and fuel. It also means these levels are not the headline bill Ofgem quotes in press releases, which uses the Typical Domestic Consumption Values — a different basis from the licence benchmark.

Levelised, not pre-levelised, because from April 2024 Ofgem levelises standing charges and the levelised figures are what its cap tables carry. Levelisation equalises direct debit and prepayment and leaves standard credit its own allowance — the three are not interchangeable. From 2024-Q2 the GB electricity single-rate nil level is £208.91 excluding VAT for both direct debit and prepayment, against £229.00 for standard credit.

Electricity VAT is zero for October 2026 to March 2027, so the 2026-Q4 GB electricity rows carry an including-VAT level equal to the excluding-VAT one, and dual fuel sits at 1.017–1.025 because only its gas leg is taxed. 270 of the 288 GB VAT pairs are at 1.05 and the 18 that are not are exactly those cells. That is Ofgem reporting the government's VAT cut, which its 26 August 2026 summary of changes states in terms ("reduce the level of VAT from all electricity bills from 5% to zero … from 1 October 2026 to 31 March 2027 … only applies to electricity bills"), not a defect. A test pins it, and a consumer differencing the treatments to recover a VAT rate must read it per period and fuel. Four cells are pinned in the test file against the published PDFs for January to March 2024 (period 11b) and October to December 2026 (period 17a).

Charge restriction regions are not ONS geographies, so they are statistical_scope with an ofgem: prefix, following the dft-bus05i-revenue-support-2025 precedent. Cap levels carry aggregation: rate rather than mean: a cap level is a regulated level per customer per year, and Microcosm refuses mean facts as sum-type targets. The benchmark-consumption facts keep mean, being a typical-consumption value.

This package and ofgem-energy-price-cap-q1-2024 sit on different VAT bases — the news-page rates include VAT and carry no vat_treatment filter, these regional levels exclude it, and only the GB rows carry both. Both package comments now say so.

2. Wales bus finance

welshgov-transport-revenue-outturn-2024-25, 828 facts. The Public Service Vehicles release reports journeys and has no fare revenue or public support, so the publisher that carries it is StatsWales, on the roads and transport revenue outturn dataset. Concessionary fares, support to operators and total public transport, on gross expenditure / total income / net current expenditure / grants, for 22 unitary authorities and the Wales total, FY2022-23 to FY2024-25.

Scope, and the checklist row says this too: these lines are the support that flows through local authorities under the RO2 return. Welsh Government funding paid directly to operators sits outside it, and no Wales bus fare-receipts figure is published anywhere. The Wales leg is a support cell, not a fare cell.

FY2024-25 Wales: concessionary fares £62.4m gross and £61.1m net; support to operators £69.0m gross and £57.2m net.

The artifact is the publisher's own unfiltered CSV export of the whole dataset, 132,238 rows, with selected_rows picking the public transport lines so source-cell lineage stays proportionate to the facts emitted. Signs follow the publisher: income and grant lines are negative because the outturn forms record them as deductions.

3. Northern Ireland bus support

nithc-annual-report-accounts-2024-25, 8 facts, from note 23 (Related Party Transactions) of the Northern Ireland Transport Holding Company's audited 2024/25 accounts. Capital grants, Public Service Obligation compensation, concessionary fare compensation and other revenue funding, FY2023-24 and FY2024-25. Concessionary fare compensation was £49.9m in FY2024-25 (£43.7m in FY2023-24); PSO compensation £61.8m (£55.5m).

Two calls the issue left to Chronicle:

• A company report, yes, for a statutory audited public-corporation account laid before the Assembly under the Transport Act (Northern Ireland) 1967. DfI's own annual report and accounts for 2024-25 is not published, and the DfI Public Transport Statistics release has no support series.
• The figures are group-wide across bus and rail. NITHC holds Ulsterbus, Metro, Goldline and NI Railways, and the note has no segmental split, so every fact carries mode_scope: all_public_transport and no bus-only NI figure is claimed. There is none published.

4. Scotland 2025 edition

scotgov-bus-coach-statistics-2024-25, 10 facts, mirroring the 2023-24 package on the 2025 edition of chapter 2 with the measure columns shifted. FY2024-25 passenger journeys (334m) and concessionary journeys (183.6m), passenger revenue £391m, government support £499m, total £890m, the four Table 2.9 support components (local authority bus support £63m, concessionary fares £392m, Network Support Grant £44m, all government support £499m), and the CY2025 fare index (214). Scotland is no longer a year behind the England facts.

The two Table 2.8 rows the 2024-edition package did not take, government support and the total row, are added here because the follow-up asks for the support components alongside passenger revenue.

The publisher labels that total row "Total passenger revenue current prices", but its Note 33 makes it passenger revenue plus government support: 391 + 499 = 890. A reader taking the label at face value reads fares. The publisher's label stays as the row-header guard and the value id is total_operator_revenue, so the fact key says what the number is.

5. ONS 07CN coicop

The 07CN record sets are restructured so the coicop is a row dimension, one record set per class, the shape 04CN already had. Fact count and values are unchanged (93). What changes is that layout.groupby_value_id now names the class on every fact instead of saying all_households on 07CN, and table_record_kind is detail throughout, so a consumer selects 04.5, 07.2.2 and 07.3.2 the same way across both sheets.

Key disclosure: the restructure re-mints the aggregate_fact_key of all 62 07CN facts, because the record set id, the measure id (personal_transport_fuels → expenditure) and the layout all move. Every value, concept, period and semantic_fact_key is identical to ca727a29, and no consumer selects on these today, but anyone holding 07CN aggregate keys must re-pin.

6. NTS frequency of use

dft-nts0313-mode-use-frequency-2025 (1,332 facts) is the whole NTS0313 table: the seven publisher frequency bands plus total and unweighted sample size, every mode, 2003 to 2025. The latest published year is 2025, not the 2024 the issue assumed.

dft-nts0621-local-bus-use-frequency-2025 (198 facts) is NTS0621, the same bands for local bus among people aged 60 and over — the age breakdown DfT publishes, and the closest published read on concessionary-age bus use.

Publisher-limited: DfT publishes no frequency-of-use breakdown by region, rural-urban area type or household income. The NTS99 area-type series and the NTS07 income series report trips and distance, not use frequency. The package comments record that, and the NTS0621 comment records how to join the two tables, since NTS0313 carries its mode label verbatim as Local bus [note 5] while NTS0621 states its mode in the sheet title.

Housekeeping

nithc is a new publisher prefix. Bundle source packages go from 185 to 191 and UK bundle sources from 116 to 122. Bundle coverage counts in tests/test_chronicle_bundle.py are updated for the new facts. docs/pe-uk-source-checklist.md records what each row now binds and what stays publisher-limited.

Review round 1

Addressed from @juaristi22: Scotland total_passenger_revenue renamed to total_operator_revenue with the Note 33 composition in the label and comment (1); the levelisation wording now says direct debit and prepayment equalised, standard credit separate (2); cap levels moved from mean to rate (3); the differing VAT bases of the two Ofgem packages noted on both packages and in the checklist row (4); the Wales checklist row qualified to local-authority-routed support with fare revenue not published (5).

Addressed from @vahid-ahmadi: the unit-rate arithmetic now states the subtraction, with a test reconstructing the four published Q1-2024 rates (2); the 07CN key re-mint disclosed above and in the checklist (3); the #890 pointer corrected to #257 item 6.

On his item 1 — the 18 GB including-VAT cells at 2026-Q4 — his arithmetic is exactly right (12 electricity at ratio 1.0, 6 dual fuel at 1.017–1.025, gas at 1.05, 270 of 288 pairs unaffected) but these are not a ported defect. Electricity VAT is zero for that window by government decision, which Ofgem states in its 26 August 2026 summary of changes, so the publisher is right and so is the port. Nothing is skipped; a test pins the pattern and the comment explains it. This also corrects my own review note that the ratio is 1.0500 on every period — it is not, for the same reason.

Not changed, with reasons: the dft-nts0621 doubled year in source_record_id is the row axis genuinely being the survey year, and a generic row value id would make layout.groupby_value_id useless; the ruff I001 on the new test file is the two-blank-lines-after-imports house style that tests/test_issue_254_transport_energy_sources.py shares, and ISC is not selected in this repo's config, so both would be changes away from house style rather than toward it.

🤖 Generated with Claude Code

[juaristi22]

Reviewed against a rebuild of the facts from the PR head (a362d1e) and the publisher pages; nothing here blocks. Verified: tests/test_issue_257_transport_energy_followups.py passes (25); the Ofgem cap levels reproduce the earlier package's news-page rates exactly when divided as the package comment says — 2024-Q1 GB direct debit gives 28.62 / 7.42 p/kWh and 53.35 / 29.60 p/day, and 2026-Q2 gives 24.67 / 5.74 p/kWh; the GB rows' including/excluding VAT ratio is 1.0500 on every period; the Scotland 2024-25 facts match the published chapter (334m journeys, £391m passenger revenue, £499m government support of which £392m concessionary, £890m total operator revenue); the ONS 07CN restructure leaves all 93 (concept, period, value) triples identical to ca727a29.

Notes, in order of usefulness before consumers pin these keys:

1. Scotland operator_revenue value id total_passenger_revenue carries £890m, which is the publisher's total operator revenue (passenger revenue £391m plus government support £499m), not passenger revenue. The id is a fact key, so renaming it to total_operator_revenue now is cheaper than after a consumer pins it.

2. Ofgem levelisation note. The package comment and PR body say standing charges are equalised across payment methods from April 2024. The data show direct debit and prepayment equal from 2024-Q2 (electricity nil level £208.91 excl. VAT for both) with standard credit higher (£229.00), which is what Ofgem's levelisation actually did. Worth stating as "direct debit and prepayment equalised; standard credit keeps its own allowance" so a consumer does not treat the three as interchangeable.

3. Ofgem aggregation: mean on cap levels. A cap level is a level, and the DESNZ duty-rate package used rate for a per-unit regulated figure. It is harmless for a consumer that vendors these as stage anchors, but Microcosm's compiler refuses mean facts as sum-type targets unless the reference declares a time mean, so anyone binding a cap level as a target later would trip on it. If the vocabulary has rate or level, one of those reads truer.

4. VAT treatment differs between the two Ofgem packages. The 2024-Q1 news-page package carries rates including VAT with no vat_treatment filter; this package's regional levels exclude VAT and only the GB rows carry both. A one-line note on the earlier package (or in the checklist row that lists both) would stop a consumer mixing them.

5. Wales scope. The outturn lines capture support that flows through local authorities (FY2024-25 concessionary fares £62.4m gross and operator support £69.0m gross, with grants of £51.4m and £44.7m against them). Welsh Government funding paid directly to operators sits outside the RO2 return, and there is still no Wales fare-receipts fact. The checklist row now reads "ported" for Wales fare revenue and public support; suggest qualifying it to "public support routed through local authorities; fare revenue not published".

6. Already declared and fine as written: NITHC support is group-wide across bus and rail; NTS0313 and NTS0621 word the same frequency band differently; NTS0313 is the latest year 2025 rather than the 2024 the follow-up assumed (the table carries 2024 too, so the base-year read is available).

🤖 Posted with Claude Code

[vahid-ahmadi]

Review at a362d1e8 (Claude Code, high effort; worktree checkout; full suite 949 passed, 1 skipped; all six packages rebuilt from their artifacts with validate_source_package, validate_facts, validate_consumer_fact_contract and validate_source_cells, zero errors or warnings; ruff at the repo-pinned 0.14.10 clean; CI green).

The substance is right. All six manifests' sha256 match the committed artifact bytes, and all six source_urls re-fetch live to the same digest — including the StatsWales token URL, as the comment says. Rebuilt counts are 1,332 / 198 / 8 / 3,636 / 10 / 828 = 6,012, bundle 185 → 191, UK sources 116 → 122, fact count 196,578 → 202,590. I checked cells against the artifacts themselves for every package: the NTS0313 2025 local-bus row and the NTS0621 2025 row, NITHC note 23 on page 39, the Scottish Tables 2.2a, 2.8, 2.9 and 2.5 cells, the Wales concessionary and operator-support lines gross and net plus Cardiff, and Ofgem regional cells across 2a/2b/2c and the Notes-sheet benchmark schedule. All match. Every #257 item is closed or explicitly declared publisher-limited, and the NTS local-bus rows are bindable in microcosm by source_concept plus layout_groupby_value_id: local_bus without touching the [note 5] string. Two should-fixes on Ofgem, one disclosure item on 07CN, then questions.

1. Should-fix — the GB including-VAT electricity cells for 2026-Q4 carry the excluding-VAT value. In sheet 1b Historical level tables, the "Total inc VAT" row for October–December 2026 equals the GB-average row for electricity (200.144 both, where the July–September column is 198.807 / 208.748). Of the 288 GB (excluding, including) pairs in the rebuilt facts, 270 sit at exactly 1.0500; the 18 exceptions are all 2026-Q4 — the twelve electricity single-rate and multi-register cells across three payment methods at nil and benchmark at ratio 1.0, and the six dual-fuel cells at 1.017–1.025 because they inherit the un-VATed electricity leg. Gas 2026-Q4 is correctly 1.05. The self-review's "1.0500 on every period" is therefore not true, and the package comment says the GB rows carry both treatments with no caveat. Either skip those 18 cells, as the package already does for unset cells, or keep them with a declared-defect note in the comment and the checklist row. Not a blocker today — nothing in microcosm binds GB including-VAT 2026-Q4 — but it is a publisher defect ported silently.

2. Should-fix — the comment's unit-rate arithmetic omits a step. source_package.yaml:22-24 says the levels turn into a daily standing charge and a unit rate by division by 365 and by the benchmark consumption. The standing charge is right: 194.7152 / 365 = 53.35 p/day, 108.034 / 365 = 29.60. The unit rate is not: 1081.8161 / 3100 gives 34.90 p/kWh, and only (benchmark − nil) / k gives the 28.62 and 7.42 in the landed Q1-2024 package. Since the paragraph exists to warn consumers off a trap in this exact arithmetic, it should state the subtraction.

3. Disclosure — the 07CN restructure re-mints 62 of 93 fact keys. Values, concepts and periods are identical on both sides and every semantic_fact_key survives, but aggregate_fact_key changes for all 62 07CN facts because record_set_id, measure_id (personal_transport_fuels → expenditure), source_record_id and the layout all move. The body's "fact count and values are unchanged" is true; "bundle delta = new facts only" holds for counts, not keys. Microcosm has no selector on these facts, so nothing breaks; one sentence in the body or the checklist row is enough.

Questions.

• Scotland total_passenger_revenue = £890m mirrors the publisher's row label, but Note 33 makes that row passenger revenue plus government support. Keeping the publisher's id is defensible; a consumer will read it as fares. The label or concept could carry the clarification rather than renaming the id.
• aggregation: mean on the 3,600 cap-level facts, pinned by test. rate is in ALLOWED_AGGREGATIONS; no functional effect in Chronicle, but a cap level is a level, not a mean of anything.
• The body cites "the Microcosm #890 plan" for the incidence family. #890's issue body does not mention bus-use incidence and I could not find a plan file at microcosm HEAD; the ask lives only in #257 item 6. Fine, but the pointer should go where the request actually is.

Nits. dft-nts0621 source_record_ids double the year (…cy2025.cy2025…) because the row axis is the survey year. Newer ruff (0.16.4) flags an import-sort and two implicit string concatenations in the new test file; the pinned ruff is clean.

Could not verify

The four Ofgem test pins against the published cap-level PDFs (checked against the workbook only), and whether any live consumer reads the 18 cells in item 1.

---

Approve once 1 is decided either way; 2 is a wording fix and 3 a sentence. The rest of the package work is careful and every number I checked reproduces.

[juaristi22]

Both reviews addressed in 0f447f7. Thank you — between them they caught a
mislabelled fact key, a formula that would have sent consumers 22% wrong on the unit
rate, and a key re-mint I had described as "unchanged".

@juaristi22

1. Renamed. total_passenger_revenue → total_operator_revenue, with the composition in
   the label ("passenger revenue plus government support") and Note 33 spelled out in the
   package comment. The publisher's own row label stays as the row-header guard.
2. Reworded, and I verified it rather than taking it on trust: 2024-Q1 has all three
   payment methods apart (185.44 / 208.60 / 209.47 excl. VAT), and from 2024-Q2 direct
   debit and prepayment are equal at 208.91 with standard credit at 229.00. The comment,
   the PR body and the checklist row now say "direct debit and prepayment equalised;
   standard credit keeps its own allowance".
3. Moved to rate — all 3,600 cap-level facts, the 216 regional and 288 GB record sets.
   level is not in ALLOWED_AGGREGATIONS; rate is, and it is what the DESNZ duty-rate
   package uses for the same shape of regulated per-unit figure. The 36
   benchmark-consumption facts keep mean, being a typical-consumption value. The test
   pin moved with it.
4. Noted on both packages, not just the new one, plus the checklist row: the news-page
   rates include VAT and carry no vat_treatment, the regional levels exclude it, only
   the GB rows carry both.
5. Qualified. The checklist row now reads "ported where published; Wales support is the
   local-authority route only and Wales fare revenue is not published", with the RO2
   scope, the direct-to-operator funding that sits outside it, and the absent fare-receipts
   fact written out.

@vahid-ahmadi

2. Fixed, and it was the right catch — the paragraph existed to warn about that arithmetic
   and then got it wrong. It now gives both formulas explicitly, including
   (benchmark - nil) / k, and says why: 1081.8161 / 3100 = 34.90 against the published
   28.62. There is a new test that reconstructs all four rates in
   ofgem-energy-price-cap-q1-2024 from these levels — 53.35 and 29.60 p/day, 28.62 and
   7.42 p/kWh, to the rounding the news page printed — and asserts the un-subtracted form
   does not reproduce them.
3. Disclosed, in the PR body and the checklist row: all 62 07CN facts get a new
   aggregate_fact_key while every value, concept, period and semantic_fact_key
   survives. My "fact count and values are unchanged" was true and incomplete.

• The #890 pointer is corrected. You are right that it does not live there: #890's body
  does not mention incidence and the plan is an uncommitted working file. UK transport and energy facts, follow-up to #254/#255: Ofgem cap history, Wales and NI bus finance, Scotland 2025 edition, ONS 07CN coicop dimension, NTS bus-use frequency #257 item 6 is
  the request of record and the body now says so.

On item 1, I think the conclusion is wrong although every number in it is right.

Your arithmetic reproduces exactly: 288 GB pairs, 270 at 1.0500, 18 exceptions all at
2026-Q4, twelve electricity cells at ratio 1.0 and six dual-fuel at 1.017–1.025 from the
un-VATed electricity leg, gas correctly at 1.05.

Those cells are not a publisher defect. Electricity VAT is zero for that window. Ofgem's
26 August 2026 summary of changes says so in terms: "The government has decided to reduce
the level of VAT from all electricity bills from 5% to zero. This change will apply to all
bills, including those under the price cap, from 1 October 2026 to 31 March 2027. This
change only applies to electricity bills and VAT will remain unchanged for gas bills at 5%
for this period." Its Annex 1 cost breakdown for the period carries VAT (electricity) at
0% and VAT (gas) at 5%. Sheet 1a Levelised DTC agrees with 1b: the current-period GB
average and GB average inc VAT rows are identical for electricity and differ for gas. And
the dual-fuel ratio lands between the two because only its gas leg is taxed — 858.20 +
823.96 = 1682.16 excluding, 858.20 + 865.16 = 1723.36 including, which is the £1,723
headline Ofgem published.

So nothing is skipped and nothing is flagged as defective. Instead the package comment now
explains the holiday, and a test pins the whole pattern — 288 pairs, 18 exceptions, all
2026-Q4, electricity at 1.0, dual fuel strictly between 1.0 and 1.05, gas at 1.05
throughout — so the next reader meets an assertion rather than an anomaly. The comment also
warns anyone differencing the two treatments to recover a VAT rate to read it per period
and per fuel.

This also corrects my own review note that the ratio is 1.0500 on every period. It is not,
and for the same reason.

Not changed

• dft-nts0621 doubling the year in source_record_id: the row axis really is the survey
  year, so the row value id is cy2025. A generic id would remove the doubling at the cost
  of making layout.groupby_value_id carry nothing.
• The ruff I001: it is the two-blank-lines-after-imports style that
  tests/test_issue_254_transport_energy_sources.py and the rest of tests/ use — the
  same rule fires on that file too. ISC is not selected in this repo's config, and the
  alternative to the implicit concatenations is lines over the configured 88. Both changes
  would move away from house style rather than toward it, so they seem better handled
  repo-wide than in one file.

Full suite green locally on 0f447f7: 951 passed, 1 skipped, including test_chronicle_bundle.py (the two extra against Vahid's 949 are the new tests above). CI on the commit is still running at the time of writing.

🤖 Posted with Claude Code