docs(nowcast): the 7 September rebuild — all eight years, and what the stack moved

[WesIngwersen]

Top of the stack. Records the 7 September rebuild in the progress report, and summarises everything below it.

Every table was rebuilt from cleared artifacts on the methods in this stack. All eight years balance at T11 max |residual| 0.0 $M, 20 outer iterations, nothing skipped; Step 6 produced the quartet for all eight; Step 7 applied the redefinitions across the span; 160 files uploaded to GCS.

What changed in the stack

Trade — the 1:m splits were weighted on the wrong thing

• fix(nowcast): split export families by exports, not by supply (#701) #865 — export families were split by same-year total supply T007. A supply weight answers who produced it; an export split needs who exported it, and those differ by a factor of eight inside aerospace (engines export 83% of output, guided missiles 11%). Graded on two benchmark holdouts, T007 misplaced 13.7% and 20.9% of family export dollars against published exports' 3.2% and 4.7%.
• fix(nowcast): split export families by exports, not by supply (#701) #865 — Census names a leaf for only 10.7% of 2017 aerospace exports and that 10.7% has a mix 39.6% off the published export mix, so the family is now folded onto 33641X before splitting.
• fix(nowcast): wire the import within-family re-split #763 validated #868 — 62 import families were mapped 1:1 or m:1, so no attribution weight was ever applied and Census's own within-family split stood. Wiring the construction Step 4b: validate a selection rule for the import within-family split on the 2012 holdout #763 validated and never connected closed the mix column.
• fix(nowcast): wire the import within-family re-split #763 validated #868 — guarded, after review caught it smearing one contaminated leaf over healthy siblings: 3399 had 88% of its excess on 339910 jewelry at 2.93×, and an unguarded re-split dragged five leaves from within 9% of published to 1.50×.
• feat(nowcast): grade the courier trade mover on the published summary row (#701) #864 — couriers graded on the published summary row: 1.68% mean absolute error 2018–2024, against 17.8% for the next best mover.

Value added and gross output

• fix(nowcast): hold the 336414/33641A see-saw the 2022 census invents (#862) #874 — the 2022 Economic Census invents a 336414/33641A see-saw: it agrees with BEA on the pair's combined output to 0.03% and moves 5.1bn between the two members, with QCEW payroll opposed in both halves. Both are now held.
• this PR's parent — the value-added residual was routed to gross operating surplus for every industry, including two that have no surplus row at all.

Margins

• feat(nowcast): anchor final-demand margins on purchasers' value (#872) #873 — final-demand margins now anchor on purchasers' value, as BEA's own process does. Graded 2017→2012: gross error 714.5 → 408.7 bn, systematic over-prediction −53%.
• docs(nowcast): record how BEA actually distributes margins, and check ours #871 — the cascading-base question was tested and is empty: the cascade telescopes and is algebraically identical to BEA's simultaneous distribution. What the test found instead is that retail margin rates moved +37.6% between 2012 and 2017, which is the real exposure.

Tooling

• stale_artifacts cannot see a method change, so an edited FBS method silently returns the previous build #867 — stale_artifacts could not see a method change: nothing upstream moves, the cache key does not move, and a rerun silently returns the previous parquet. It found 10 stale artifacts before, 290 after.
• After the rebuild it reports 386, and none is a product built in this cycle: 108 are superseded earlier versions of the nowcast quartet still on disk, the rest upstream extract artifacts deliberately left alone.

Impact on the result, against the previous build

⚠️ Grand totals are preserved. The Use SUT differs by 3 $M at 2017 on 56.7 trillion, and 0.007% at worst across the span. Every change this cycle is a redistribution, so the interior moves and the totals do not.

What this column measures: the gross cell-by-cell difference — the sum of |new − old| over all 406 × 421 = 170,926 cells of the balanced Use SUT — divided by the previous build's grand total for the same cells. Two consequences worth stating: it is gross, not net, so offsetting moves add rather than cancel (the controls hold, so net is near zero by construction); and it counts each relocated dollar twice, once where it leaves and once where it arrives, so 762,736 $M at 2017 is roughly 381bn of value actually changing position. It also covers the whole table — value-added rows and final-demand columns included, not just the commodity-by-industry interior — which is why those are broken out.

year	Use SUT, gross |Δ|	intermediate	value added	final demand	Supply	MUT Use
2017	762,736 $M (1.3%)	59%	28%	13%	120,033 $M (0.3%)	1.4%
2018	1,688,403 $M (2.8%)	52%	40%	8%	164,282 $M (0.4%)	2.8%
2019	1,965,366 $M (3.2%)	50%	42%	7%	181,344 $M (0.4%)	3.2%
2020	3,637,855 $M (6.0%)	49%	43%	8%	615,046 $M (1.6%)	5.9%
2021	4,781,009 $M (7.0%)	48%	42%	10%	1,296,471 $M (2.9%)	6.8%
2022	5,423,197 $M (7.1%)	50%	44%	7%	648,615 $M (1.3%)	7.0%
2023	5,409,242 $M (6.8%)	50%	45%	5%	336,857 $M (0.6%)	6.6%
2024	6,131,087 $M (7.3%)	50%	45%	5%	423,500 $M (0.8%)	7.2%

The three middle columns are shares of that year's gross movement. The value-added share nearly doubles across the span — 28% at 2017 to 45% at 2024, which is 2,766,328 $M of the 2024 total. At the far end of the span nearly half of this column is the value-added reconciliation rather than the commodity-by-industry interior.

The growth with distance from the benchmark is the expected shape, not drift: 2017 is anchored, so only the trade re-splits and the aerospace conditioning reach it. Later years additionally carry the value-added reconciliation, whose gap against BEA's published series grows across the span. A rebuild that moved 2024 as little as 2017 would mean the new methods were doing nothing.

Cell-level, on the Use SUT: 16,068 cells move by more than $1M at 2017, rising to 33,290 at 2024, out of 170,926. Counting any non-zero movement the figure is far wider — 46,959 cells at 2017.

Named outcomes, 2017 against published:

• import MATCH cells 109 → 123, EXTRA 2 → 0, Pearson excluding specials 0.976 → 0.989
• aerospace exports: five leaves at 1.17/0.60/1.05/4.94/4.23× → a uniform 1.06×, gross error 35,446 → 6,434 $M (which equals its net, so the split error is gone rather than reduced)
• vehicle exports 336111/336112: 0.31× and 1.52× → 1.01× both, MATCH
• 336414 gross operating surplus: −2,957 → +311 $M at 2022, −3,693 → −22 at 2024
• gross import commodity error 445,999 → 309,342 $M, and the mix component 34% → 5%

Source lineage for the Step 5-7 products

The rebuild exposed that these products carry no source lineage. They are written by bespoke savers rather than the FBS framework, so nothing populated primary_source_meta, and the artifact check could not tell when their inputs moved — it answered "No stale cached artifacts" about balanced SUTs sitting on trade inputs rebuilt hours earlier. A silent pass on something that cannot be checked reads as a clean bill of health.

Fixed for Steps 6 and 7. source_lineage() in utils/metadata takes the artifact names a saver read and returns their sidecars in the shape stale_artifacts._walk_sources already follows. It is wired into both savers that know their inputs — save_mut() from MutTables.sources, and save_after_redef() from the before_inputs it already accepted and had only recorded as bare filenames. A rebuilt Step 5 product now invalidates Step 6, and a rebuilt Step 6 invalidates Step 7. A name with no readable sidecar is recorded as a lineage_error rather than dropped, for the same reason a silent pass is worse than a noisy flag.

⚠️ Step 5 itself is still not covered, deliberately. save_balance() cannot name its inputs: the seed comes from fit_interior, derive_initial_Y_pur, derive_initial_value_added and assemble_supply_seed, none of which report the artifacts they read, and each globs for its own newest file. Giving Step 5 real lineage means making those four report — a change across several modules rather than a saver edit, and not one to land inside a docs PR. It keeps the builder surface from #867 in the meantime, so a change to the assembly module still flags it; what it cannot see is an upstream FBS rebuild.

Not retroactive: products already on disk keep the sidecars they were written with, so the lineage appears on the next build.

Which files on GCS correspond to this stack

Newest per product and year, gs://cornerstone-default/. 80 parquet + 80 sidecars = the 160 files reported above. Two commits cover the set: d2e2112 for Step 5 and the before-redefinitions quartet, 4276083 for the after-redefinitions quartet — both in this stack.

Step 5 — balanced SUT — gs://cornerstone-default/flowsa/BalancedSUT/

product	stage	years	files	version + commit	size	uploaded (UTC)
Balanced_Detail_Supply	—	2017-2024	8	v0.3.0_d2e2112	2.3 MB	2026-09-08 01:46
Balanced_Detail_Use_SUT	—	2017-2024	8	v0.3.0_d2e2112	5.5 MB	2026-09-08 01:46

Steps 6-7 — MUT quartet — gs://cornerstone-default/flowsa/NowcastMUT/

product	stage	years	files	version + commit	size	uploaded (UTC)
Nowcast_Detail_Import_after_redef	after	2017-2024	8	v0.3.0_4276083	4.3 MB	2026-09-08 02:32
Nowcast_Detail_Make_after_redef	after	2017-2024	8	v0.3.0_4276083	2.0 MB	2026-09-08 02:32
Nowcast_Detail_Margins_after_redef	after	2017-2024	8	v0.3.0_4276083	39.5 MB	2026-09-08 02:32
Nowcast_Detail_Use_after_redef	after	2017-2024	8	v0.3.0_4276083	5.8 MB	2026-09-08 02:32
Nowcast_Detail_Import_before_redef	before	2017-2024	8	v0.3.0_d2e2112	4.4 MB	2026-09-08 02:24
Nowcast_Detail_Make_before_redef	before	2017-2024	8	v0.3.0_d2e2112	2.1 MB	2026-09-08 02:24
Nowcast_Detail_Margins_before_redef	before	2017-2024	8	v0.3.0_d2e2112	39.8 MB	2026-09-08 02:24
Nowcast_Detail_Use_before_redef	before	2017-2024	8	v0.3.0_d2e2112	5.9 MB	2026-09-08 02:24

The bucket holds 248 data blobs in these two prefixes in total; the rows above are the newest per product and year, which is what the detail loaders resolve.

The comparison figures no longer have a threshold in them

The report was dominated by a statistic with no grounding: the share of cells landing within 1% of the published value. The 1% was a choice, and the report read as though it were a property of the data — two cells 0.9% and 1.1% apart were drawn in different colours and counted on opposite sides of a headline number.

Cells both sides populate are now coloured on one continuous ramp, green at no difference through to red at 100% or more, and each figure carries a median and a value-weighted relative difference in place of "accuracy".

The pair earns its keep immediately. Summary Use at 2024 has a median cell 22.0% off but only 6.7% of its dollars misplaced — the large cells are far better than the typical one, which no threshold count could show. Its old headline was "7.5% accuracy", which reads as catastrophic and says nothing; its old picture was near-uniform amber. It now shows a green interior with identifiable hot rows and columns.

Tolerance survives where a boundary is genuinely needed — check() has to answer pass or fail, so it keeps rtol/atol and the MATCH/PARTIAL split. Nothing in the report depends on it any more.

⚠️ The ramp is green-to-red, but not on hue alone — that is the one gradient protanopes and deuteranopes cannot read. Lightness falls monotonically along it (L* 81/73/64/48/33), so the two ends separate by dE 45.3 under simulated deuteranopia, almost entirely from lightness. The anchors were searched, not picked: the green end is squeezed between white (ABSENT) and the blue of EXTRA, which under tritanopia both sit near a light green. Best achievable worst-pair separation is dE 26.0 against the floor of 25 that palette_separation already enforced — it passes with less headroom than the old palette, and the binding pair is white-against-green, the same one that bound the old one.

All 30 committed figures re-rendered; every caption and both summary tables rewritten from the measured values, so they now reflect the 7 September rebuild rather than the older numbers they had been carrying.

⚠️ PNG figures moved to Git LFS — this repo now needs git-lfs installed

.gitattributes had declined LFS on a measurement and named the condition for revisiting it: "if these figures start churning per-commit rather than per-milestone." That happened — 1.16 MB of incompressible blobs per re-render, twice in three days. All 42 committed PNGs are now pointers.

This does not shrink the repo: blobs committed before the change stay in history, so it stops growing at this rate rather than getting smaller. Converting the history too would rewrite main and all ten branches in this stack, and is deliberately not done here. The nine committed .parquet fixtures are 3.1 MB — larger than the images — and are left as ordinary blobs, since they are static reference artifacts rather than regenerated output.

Verification

• 8/8 years balance at T11 = 0.0 $M; Step 6 quartet for 8/8; Step 7 for 8/8 — 32 + 64 + 64 = 160 files uploaded to GCS
• Step 7's 2017 replay gate held before the run: max Make cell gap $19.0M, Use interior $6.1M, value added $15.9M
• nothing built today is flagged stale by stale_artifacts
• V00300 × 814000 is 0.0 in every year through Step 6, with compensation carrying the whole 18.7 → 27.4 bn
• Census_USATrade FBA regenerated and the trade totals verified identical, so its staleness flag is true but benign
• black, ruff and mypy clean; --check-palette passes at dE 26.0 against its floor of 25
• previous build preserved off-tree throughout, so every figure above is a measured diff rather than a recollection
• Stack brought onto current main by merge, not rebase. Main added four commits (the MECS energy wiring, the GHG fix for the hardcoded 2018 vintage, the v0.3.2 snapshot bump) and none of them touches a file this stack changes, so all ten merges were clean with nothing to resolve. Rebase is the wrong tool here: the stack still carries the pre-squash nowcast history that main holds as the single squashed commit for Add nowcasted SUTs and MUTs 2017-2024 and associated CS models and analysis #854, so git rebase replays 180 commits main already has and conflicts on August history rather than on anything in the stack.

🤖 Generated with Claude Code

[bl-young]

Seven of eight years failed at first. The value-added reconciliation put 1,967 → 5,104 $M of household gross operating surplus into a cell the mask holds as a structural zero. BEA publishes V00300 of exactly zero for 814000 in 2007, 2012 and 2017, with VAPRO equal to compensation to the dollar — the sector's output is the compensation it pays. The balanced products on disk had been built four days before that reconciliation landed, so it had never run on those years. Fixed in the parent commit; the routing now follows the row each industry actually carries, and exactly two industries take the exception.

In your description, this is listed under " Two things this rebuild exposed that are not fixed here". But this sounds like it was fixed?

[bl-young]

Thank you this is a helpful summary! Looks like some good improvements across the board, but especially in trade. One comment on the description above. Also might be good to clarify where or which specific files were uploaded to GCS that correspond to this stack so we have a easily trackable record.

[WesIngwersen]

Both addressed.

You were right that the first item was fixed — it was under the wrong heading. That whole "not fixed here" section is gone: the household V00300 item was fixed in the parent commit and is already described under the value-added changes, so restating it as an open problem was just wrong.

The second item is now fixed too, for Steps 6 and 7. source_lineage() in utils/metadata records the sidecars of what a saver actually read, in the shape stale_artifacts._walk_sources already follows, wired into save_mut() and save_after_redef(). A rebuilt Step 5 product now invalidates Step 6, and a rebuilt Step 6 invalidates Step 7. Step 5 itself is still uncovered and I have said so explicitly rather than quietly — save_balance() cannot name its inputs without making four seed derivations report what they read, which is a multi-module change I did not want to land inside this PR.

On the GCS record: added as its own section with a table of the newest file per product and year, which reconciles to the 160 figure (80 parquet + 80 sidecars). Two commits cover the set — d2e2112 for Step 5 and the before-redefinitions quartet, 4276083 for the after-redefinitions quartet.

[bl-young]

On the GCS record: added as its own section with a table of the newest file per product and year, which reconciles to the 160 figure (80 parquet + 80 sidecars).

And the GHG FBS too?

[WesIngwersen]

On the GCS record: added as its own section with a table of the newest file per product and year, which reconciles to the 160 figure (80 parquet + 80 sidecars).

And the GHG FBS too?

Not yet will do that after merging