generate_models.sh crashes, then silently drops content, against the 2026-08-25 spec reorg

[vishkaty]

Summary

Running npm run generate -- <2026-08-25 ucp checkout> fails outright, and
once that failure is fixed, produces an incomplete src/spec_generated.ts
with no error at all. Four independent hardcoded assumptions in the
generator baked in the 2026-04-08 layout; the 2026-08-25 reorg (schemas
moved from schemas/shopping/ into a new schemas/common/ tree, several
capabilities renamed, a canonical discovery profile document published)
breaks each of them differently. Two further gaps only show up once the
first four are fixed and the real spec is regenerated end to end.

All line references below are against upstream/main at 44c448d.

Root cause 1: AP2 mandate path hardcode

scripts/project-current-ucp-schemas.mjs:1044-1045:

function writeCompatibilityAp2Schema(schemaCache) {
  const sourceRel = "shopping/ap2_mandate.json";

The AP2 mandate extension moved AND was renamed in the reorg
(shopping/ap2_mandate.json -> common/payment_ap2_mandate.json;
dev.ucp.shopping.ap2_mandate -> dev.ucp.common.payment.ap2_mandate). The
hardcoded path resolves to nothing, and the function falls through to a
malformed intermediate value, crashing at clone():

SyntaxError: "undefined" is not valid JSON
    at clone (scripts/project-current-ucp-schemas.mjs:176)
    at renameExtensionCheckoutDef (...:998)
    at writeCompatibilityAp2Schema (...:1076)

This is the first, blocking failure -- nothing downstream runs until this
is fixed.

Root cause 2: common/ root blindness

scripts/project-current-ucp-schemas.mjs:435,449:

function loadSchemaCache() {
  ...
  for (const fileName of fs.readdirSync(sourceShoppingRoot)) {

loadSchemaCache walks schemas/shopping/ and schemas/common/types/ but
never the schemas/common/ root. Every root-level file the reorg introduced
there (payment_terms.json, payment_authentication.json,
location_lookup.json, payment_ap2_mandate.json, and more) is invisible
to the generator, not
merely unmodeled -- resolveSourceSchema cannot find them by basename or by
declared name either, since the cache never contains them in the first
place.

Root cause 3: capability/transport allowlist

generate_models.sh:69-108 (the QUICKTYPE_ARGS array) hand-enumerates the
2026-04-08 set of extension/lookup/search capabilities
(buyer_consent/discount/fulfillment/cart/catalog_lookup/
catalog_search) as explicit --src flags. Any capability the reorg adds
beyond that fixed set -- payment_terms, location_lookup, loyalty,
payment_split_payments, payment_authentication, and new transport
envelopes -- never reaches quicktype, no matter how loudly it announces
itself in the spec (each has a proper dev.ucp.* name and a recognizable
lookup/search/attachment shape).

Root cause 4: core response_*_schema hand-list

scripts/project-current-ucp-schemas.mjs:900-909:

function writeCompatibilityCoreSchemas() {
  ...
      response_checkout_schema: {
        ...
      },
      response_order_schema: { $ref: "../discovery/ucp_response.json" },
      response_cart_schema: { $ref: "../discovery/ucp_response.json" },
      response_catalog_schema: { $ref: "../discovery/ucp_response.json" },

The response envelope derivation hand-lists exactly four
response_*_schema keys. 2026-08-25 adds a fifth
(response_location_schema, for the new location lookup/search
capability); the hand-list silently excludes it from the derived envelope
with no error.

What this looks like against the real spec

PR #61 regeneration output against the real 2026-08-25 spec shows part of
the same footprint this issue describes: no PaymentTerms or Loyalty
models, no payment_authentication, and UcpDiscoveryProfileSchema still
publishing the retired signing_keys field where the spec now defines a
canonical keys field (RFC 7517 JWK Set,
profile.json#/$defs/base/properties/keys).

Two further gaps, found while proving a fix complete

A fix for the four root causes above must be proven against the REAL
2026-08-25 spec end to end, not just a synthetic fixture -- doing so
surfaces two more generator-level gaps:

A fragment-name collision. quicktype names an untitled schema-mode
fragment from its $ref fragment name (e.g. lookup_request). Once
capability discovery is fixed generically, a second capability reusing an
untitled fragment name already in use (location_lookup.json alongside
the pre-existing catalog_lookup.json, both defining bare
$defs.lookup_request/lookup_response) collides and silently drops the
fields of one shape.

A genuinely recursive schema. 2026-08-25 adds
common/types/constraint_expression.json, an Object Constraint schema that
is deliberately self-referential ("$ref": "#" inside its own
properties.additionalProperties.oneOf and anyOf.items, letting a
constraint expression nest inside itself). The quicktype typescript-zod
target cannot represent a type like this at all -- verified in total
isolation (this one schema, zero other content): it exhausts its internal
type-ordering pass and silently emits nothing, with only a warning
("Exceeded maximum number of passes when determining output order, output
may contain forward references") and no
non-zero exit. Worse, this loss is not confined to the recursive type
itself: when it shares an invocation with other schemas (as it does here,
via the constraints property on available_payment_instrument.json, which the
discovery profile payment.handlers[] response reaches), quicktype
silently drops OTHER, unrelated types from the SAME invocation too --
including the entire discovery-profile envelope
(UcpDiscoveryProfileSchema, which is where the new keys field actually
lives) and the UcpResponseSchema/UcpCheckoutResponseSchema family. A
generator fix that only unblocks discovery still ships with keys
silently missing unless this is also handled.

Suggested fix shape

• Resolve the AP2 extension (and any future move/rename) by basename, then
  by declared-name-suffix, rather than a hardcoded path.
• Walk the whole schemas/ tree, not just schemas/shopping/.
• Discover capabilities and transport envelopes by SHAPE (a declared
  dev.ucp.* name plus a recognizable lookup/search/attachment/envelope
  shape) rather than a hand-enumerated list, so a future spec release
  needs no script edit.
• Derive the response envelope response_*_schema set from the real
  ucp.json instead of a fixed list.
• Title its own copy of the fragment for the newly discovered capability
  when the fragment name is already in use to avoid the collision, without touching the pre-existing,
  already-working untitled fragments.
• Treat quicktype failures on a per-family basis (isolate each discovered
  capability into its own bounded invocation) so one unrepresentable shape
  cannot corrupt the rest of the regeneration, and never let such a failure
  pass silently -- gate it behind a short, explicit, reviewed exclusion
  list, not an unconditional log-and-continue.

[vishkaty]

Update against the revised #61 head 7e11e3e, which overtakes most of this issue. Root causes 1 and 2 are fixed there: the AP2 mandate path is now resolved through the schema cache, and loadSchemaCache walks the common root. Root causes 3 and 4 are resolved for the 2026-08-25 shapes by extending the hand lists, so generate_models.sh still enumerates sources by hand and writeCompatibilityCoreSchemas still names response schemas one by one, which means the next capability the spec adds is dropped silently again. The observations under What this looks like against the real spec no longer hold either, apart from payment_authentication.

[vishkaty]

Narrowing this issue now that #65 is closed in favor of #61.

Measured 2026-09-02 on 688b627: regeneration against release/2026-08-25
completes byte for byte against the committed output, so nothing at the current
pin is dropped except payment_authentication.
common/payment_authentication.json declares
dev.ucp.common.payment.authentication, appears zero times in
generate_models.sh, and generates zero models, with no note recording the
exclusion.

The silent drop class itself remains. Adding a synthetic next release capability
file with a declared dev.ucp name and a lookup_request and lookup_response pair
to a spec copy, generate_models.sh exits 0 and the new capability appears
nowhere in the output or the log.

So this issue now tracks two things: the hand enumerated source lists that admit
those silent drops, and the undocumented payment_authentication exclusion. The
declaration shaped half has moved to #66 and #67 and is not part of this issue
any more.