bea_Vnorm_scrap_corrected aligns scrap on the wrong axis and returns a non-square A

[WesIngwersen]

What happens

bea_Vnorm_scrap_corrected in bedrock/transform/eeio/cornerstone_bea_intermediates.py:51 applies the CEDA scrap correction like this:

V = load_detail_V_usa()
q = bea_q()
Vnorm = compute_Vnorm_matrix(V=V, q=q)
scrap = V.loc[:, 'S00401']
return Vnorm.divide((1.0 - (scrap / q).fillna(0.0)))

scrap is indexed by industry (the rows of the Make matrix) and q is indexed by commodity (its columns). The two axes share 398 codes but are not the same set, so pandas aligns scrap / q on the union of 406 labels rather than on the 402 commodities. The divisor then carries four labels that are not commodities at all, and DataFrame.divide creates a column for each.

Measured on current code, 2024 nowcast configuration

• bea_Vnorm_scrap_corrected() returns 402 x 406 where the Make matrix is 402 x 402.
• The four extra columns are the industry-only codes 331314, S00101, S00201, S00202, and each is entirely NaN.
• bea_Aq() at line 71 consumes it directly, so it returns Adom and Aimp at 402 x 406 with 1,608 NaN cells. A direct-requirements matrix is meant to be square, and this one is not.
• The NaN cells are confined to those four columns; nothing leaks into the rest.
• On the 398 codes that exist on both axes the correction factor is exactly right, matching 1 - scrap_j / q_j to 1.1e-16. The formula is correct; only the alignment is wrong.

Why nothing is broken today

derive_cornerstone_Aq expands the result onto CS_COMMODITY_LIST, and none of the four spurious codes is in that list, so all four columns are dropped before anything downstream sees them. The published 405-sector A matrix has zero NaN cells and its axis matches the Cornerstone list exactly. The commodity-only codes on the other side of the mismatch (S00300, S00401, S00402, S00900) come out of .fillna(0.0) with a divisor of 1, meaning no scrap correction, which is the right answer for them since no industry shares those codes.

So this is latent, not a live error, and no published number needs revisiting.

Why it is still worth fixing

Two ways it could stop being latent:

1. A new consumer inherits it. bea_Aq advertises a square commodity-by-commodity matrix and returns something else. Any caller that does not happen to reindex picks up both the wrong shape and the NaN cells. This is not hypothetical: writing the A-matrix leverage analysis this week, calling compute_L_matrix on the result failed with a shape error, and the workaround was to reindex to the Make matrix's own commodity axis.
2. The safety depends on a list that changes. The four columns are dropped only because they are absent from CS_COMMODITY_LIST. A future disaggregation that introduces one of them as a commodity, government electric utilities being the plausible one, would silently promote an all-NaN column into the model.

Suggested fix

Align the divisor to the commodity axis explicitly, so the intent stated in the docstring, "the scrap output of the industry sharing code j", is what the code does:

scrap = V.loc[:, 'S00401'].reindex(q.index).fillna(0.0)
return Vnorm.divide(1.0 - (scrap / q).fillna(0.0))

That keeps every one of the 398 shared codes at its current value, gives the four commodity-only codes the same divisor of 1 they already get, and returns a 402 x 402 matrix. bea_Aq then returns a square A.

Worth adding alongside it: an assertion in bea_Aq that Adom.index.equals(Adom.columns), so a future axis mismatch fails at the point it is introduced rather than being absorbed by the expansion.

Verification

The change should be a no-op on published output. Confirm by checking that the 405-sector A matrix is unchanged cell for cell before and after, and that bea_Aq comes back square with no NaN.