fix: regenerate against the 2026-08-25 UCP spec reorg - #65
Closed
vishkaty wants to merge 1 commit into
Closed
Conversation
Running generate_models.sh against a 2026-08-25-shaped spec tree crashed
immediately (SyntaxError: "undefined" is not valid JSON) and, once past
that, silently produced an incomplete src/spec_generated.ts with no error
at all. Root-caused this to four independent hardcoded assumptions the
2026-04-08 pin baked into the generator, plus two further generator-level
gaps found while proving the fix complete against the real spec:
1. AP2 mandate path hardcode. writeCompatibilityAp2Schema looked for the
AP2 extension at its 2026-04-08 path (shopping/ap2_mandate.json). The
reorg moved AND renamed it (common/payment_ap2_mandate.json, capability
name dev.ucp.shopping.ap2_mandate -> dev.ucp.common.payment.ap2_mandate).
Fixed by resolving it with a basename fallback and, failing that, a
declared-name-suffix fallback, so a further move or rename does not
require another hardcoded path.
2. common/ root blindness. loadSchemaCache walked schemas/shopping/ and
schemas/common/types/ but never the schemas/common/ root, so every
root-level file the reorg introduced there (payment_terms.json,
payment_authentication.json, location_lookup.json, and more) was invisible to the generator, not just
unmodeled. Fixed by walking the whole schemas/ tree.
3. Capability/transport allowlist. The 2026-04-08 set of extension/lookup/
search capabilities and transport envelopes was hand-enumerated as
--src flags. Any capability the reorg added beyond that fixed set never
reached quicktype no matter how loudly it announced itself. Fixed with
discoverAdditionalCapabilities/discoverTransportEnvelopes: capabilities
are found by SHAPE (a declared dev.ucp.* name plus a lookup_request/
lookup_response pair, a search_request/search_response pair, or a
$defs key ending in .checkout/.order/.cart) rather than by name, so a
future spec release adding another one needs no script edit. A
capability matching none of these known shapes (identity_linking,
permalink) is left alone and reported, not silently dropped.
identity_linking exists unmodeled in BOTH the 2026-04-08 and 2026-08-25
trees (a pre-existing residual); permalink is new in 2026-08-25 and
matches no modelable shape, so modeling either is out of scope for a
root-cause-the-generator fix, not a regression.
4. Core response_*_schema hand-list. writeCompatibilityCoreSchemas
hand-listed exactly four response_*_schema keys when deriving the
response envelope; 2026-08-25 added a fifth (response_location_schema)
that the hand-list silently excluded. Fixed by deriving the full set
from the real ucp.json instead of a fixed list.
Alongside the four root causes:
- signing_keys -> keys authoring fix. profile.json now publishes a
canonical `keys` field (a JWK Set per RFC 7517); the previous
hand-authored `signing_keys` guess in the generator is now derived from profile.json
when it exists, falling back to the 2026-04-08 hand-authored shape
otherwise, so the pinned output does not move.
- Fragment name collision. quicktype names an untitled schema-mode
fragment from its $ref fragment name (e.g. lookup_request); a second,
newly-discovered capability reusing that same untitled fragment name
(location_lookup.json alongside the pre-existing catalog_lookup.json)
collided and silently lost the fields of one shape. Fixed by titling
only its own copy of the fragment for the newly discovered capability.
- Per-family isolation never silently drops. quicktype cannot represent
every JSON Schema shape, and its type-ordering pass has a fixed
iteration budget it can silently exceed (dropping types with no
non-zero exit). Each discovered capability now generates in its own
bounded quicktype invocation and merges into the main output via
scripts/merge-generated-fragment.mjs, so one unrepresentable family
cannot corrupt the rest of the regeneration. A family that still fails
must be on the short, reviewed KNOWN_UNREPRESENTABLE_FAMILIES allowlist
in scripts/check-generation-completeness.mjs (each entry citing the
exact quicktype failure) or the whole run now exits non-zero -- the
isolation mechanism was itself found to be a fail-open (log and
continue, exit 0 regardless) during this work, and is now a hard gate
instead. common/payment_authentication is the one current entry: its
"actions" attachment puts named Action keys plus a schema-typed
additionalProperties catch-all on the same object, which the quicktype
typescript-zod target cannot represent at all ("Internal error: .",
minimal-repro-verified, no allOf involved).
- Self-referential properties. 2026-08-25 added
common/types/constraint_expression.json, a genuinely recursive schema
($ref: "#" pointing at its own whole document). quicktype cannot place
a type like this at all (verified in total isolation: zero exports,
"Exceeded maximum number of passes"), and that failure was not confined
to the recursive type itself -- it silently deleted unrelated types
sharing the same invocation, including the entire discovery-profile
envelope (UcpDiscoveryProfileSchema, carrying the keys field above) and
the UcpResponseSchema/UcpCheckoutResponseSchema family. Fixed by
dropping any property whose value is a $ref to a schema containing a
bare "$ref": "#" before quicktype ever sees it, logged loudly
(deliberately NOT matching an ordinary "#/$defs/..." fragment
reference, ordinary and already working). The fix engages only on a
property that is solely a $ref to such a schema; the 2026-04-08 run
logs zero such drops and its output stays byte-identical.
Verification: the 2026-04-08 regeneration is confirmed byte-identical to
the committed src/spec_generated.ts (re-run and diffed against the actual
git-committed baseline, not a working-tree copy, to rule out contamination
from an in-progress 2026-08-25 run). The real 2026-08-25 regeneration
completes (242 exported schema constants; the committed 2026-04-08 output
has 151) with only the one reviewed exclusion
above. A kill test (reverting both scripts to their pre-fix upstream/main
versions with the tests kept) reproduces the original crash: the fixture
suite goes red (all 11 tests in generate-0825-projection.test.js fail;
the 9 allowlist-gate tests stay green by design) and the real 2026-08-25
regeneration crashes with the exact "undefined is not valid JSON" error
this fix resolves.
Scope: only generate_models.sh, scripts/, and tests/ change.
src/spec_generated.ts is untouched -- the 2026-08-25 regeneration output
is deliberately not committed here (out of scope for this generator fix);
whichever PR lands second rebases these scripts and regenerates; once
this discovery mechanism is in the tree, any 2026-08-25 regeneration
completes with every family present or explicitly excluded.
damaz91
added
the
status:needs-triage
vishkaty
pushed a commit
to vishkaty/js-sdk
that referenced
this pull request
Aug 31, 2026
…ally fix/generate-0825-projection (js-sdk#65) fixed the generation pipeline itself but left two residuals: (1) common/types/* files with no INBOUND $ref from anything reachable are projected to disk but never handed to quicktype -- 2026-08-25's four payment-credential subtypes (card/pan/ network_token/token_credential.json, which only allOf-ref BACK to their shared base) are the real-world instance; (2) a capability whose $defs is its own platform/business config declaration -- neither a checkout/order/ cart attachment nor a lookup/search pair -- matched no known shape and was reported as skipped; identity_linking.json (both spec pins) and permalink.json (2026-08-25 only) are the real-world instances. ptiper's js-sdk#61 review asked for exactly these to be projected as a follow-up. Both closed by shape, not by naming the affected files: - discoverCommonTypeFiles adds EVERY common/types/*.json file to a new `types` array in generated-src-manifest.json, which generate_models.sh's existing per-family isolated-invocation loop (built for discovered capabilities) already generically consumes unchanged (one added spread, `...manifest.types`, in two places). A file already reachable transitively regenerates a byte-identical duplicate that merge-generated-fragment.mjs already skips as "already present"; a file quicktype cannot represent at all (2026-08-25's constraint_expression.json, genuinely self-referential, already documented) fails its own isolated invocation and must clear check-generation-completeness.mjs's reviewed KNOWN_UNREPRESENTABLE_FAMILIES gate or the whole run fails loudly -- the same safety net a discovered capability's own unrepresentable shape already goes through. - classifyCapability gains a fourth shape, "declaration": a $defs key equal to the capability's own `name`, itself holding platform_schema/ business_schema (optionally response_schema) -- the exact roles capability.json's own generic base defines, redeclared per-capability so it can layer required/config fields on top. permalink.json uses the identical shape and was also previously reported "matched no known shape" -- catching it too (unprompted by the review, which only named identity_linking) is the proof the shape rule generalizes rather than being tailored to one file. buildDeclarationVariantSchema flattens each variant with the existing flattenAllOf helper (already used by buildEntityResponseSchema), inlining the generic capability.json/ ucp.json boilerplate so no cross-file $ref into capability.json is needed -- capability.json is never projected into the output tree (only its flattened discovery/capability.json compat form is), which is why the naive "point quicktype at the nested fragment" approach fails with a generic "Internal error: ." (root-caused as a missing file, not a representability limit, by running the identical fragment against the raw unprojected source, where it resolves cleanly). Two narrow fixups on top of flattenAllOf's existing behavior, both needed only because declaration variants keep properties as-is (unlike toCompatLeaf callers, which sidestep both by collapsing every $ref property to a plain string): ucp.json's bare "#/$defs/version" is collapsed to a plain string (the constraint is reattached later by inject-schema-constraints.mjs, same as every other quicktype-dropped constraint); capability.json's "extends" property (the one file-valued $ref anywhere in the boilerplate) is rewritten with the existing rewriteRef helper, scoped to that one property so a capability's own delta properties keep their own, already-correct file refs untouched. - Collateral fix required to make the common/types inclusion succeed: common/types/error_response.json (previously orphaned, so this never surfaced) $refs ucp.json#/$defs/error, which the projected compat ucp.json did not carry. writeCompatibilityCoreSchemas now also derives "success"/"error" (the same allOf-base shape every response_*_schema key is already derived from) by name, not by broadening the shape test alone: ucp.json's own platform_schema/business_schema share the identical allOf[0] shape but are a different, fuller request-side form, so mapping them to the generic response envelope would be semantically wrong, not merely incomplete. Both pins verified (evidence-only; src/spec_generated.ts, pinned to 2026-04-08, is untouched -- see the report for full evidence, model counts, and the byte-identity diff against the committed baseline). New tests/project-credential-identity-defs.test.js (red-first, kill-tested against e177d27) plus fixture additions (identity_linking.json and three previously-missing common/types fixture files needed once the fixture resolves a full discovery/*.json + declaration-capability invocation for the first time). Updates one now-obsolete assertion in generate-0825-projection.test.js that documented identity_linking as "out of scope" -- true when written, superseded by this change. 134/134 tests green, tsc pretest clean, build clean, pre-commit clean (ShellCheck could not run in this environment -- Docker unavailable, same environment limitation already flagged on fix/generate-0825-projection -- flagged for CI verification, not silently skipped). Stacked on fix/generate-0825-projection (js-sdk#65, unmerged upstream): if Universal-Commerce-Protocol#65 merges first this branch rebases trivially onto main; if the generation mechanism changes materially before then, this branch re-derives against the new mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vishkaty
pushed a commit
to vishkaty/js-sdk
that referenced
this pull request
Aug 31, 2026
…ally fix/generate-0825-projection (js-sdk#65) fixed the generation pipeline itself but left two residuals: (1) common/types/* files with no INBOUND $ref from anything reachable are projected to disk but never handed to quicktype at all, since nothing in generate_models.sh adds common/types/*.json to any --src set. The real-world instance is 2026-08-25's four payment-credential subtypes: card_credential.json, pan_credential.json, network_token_credential.json, token_credential.json. Each only allOf-refs BACK to their shared base (payment_credential.json, which is already reachable and generates fine as PaymentCredentialSchema); nothing refs any of the four subtypes forward, so none of them ever reach quicktype. (2) A capability whose $defs is its own platform/business config declaration, not a checkout/order/cart attachment and not a lookup/search pair, matches none of the shapes discoverAdditionalCapabilities currently recognizes and is reported as a skipped, unmodeled gap. common/identity_linking.json (present at both the 2026-04-08 and 2026-08-25 pins) and shopping/permalink.json (2026-08-25 only) are both instances of this exact shape: a $defs key equal to the capability's own `name`, itself holding platform_schema/business_schema/response_schema sub-keys. Thank you to ptiper for calling both of these out explicitly in js-sdk#61's review and asking that they land as a follow-up right after 0.5.0. Both closed by shape, not by naming the affected files: - discoverCommonTypeFiles adds every common/types/*.json file to a new `types` array in generated-src-manifest.json, which generate_models.sh's existing per-family isolated-invocation loop (built for discovered capabilities) already generically consumes -- one added spread, `...(manifest.types ?? [])`, in the two places that loop reads the manifest. A file already reachable transitively regenerates a byte-identical duplicate that merge-generated-fragment.mjs already skips as "already present"; a file quicktype cannot represent at all fails its own isolated invocation (or, like constraint_expression, exits 0 while emitting nothing, which the loop now also treats as a failure -- see below) and must clear check-generation-completeness.mjs's reviewed KNOWN_UNREPRESENTABLE_FAMILIES gate or the whole run fails loudly, the same safety net a discovered capability's own unrepresentable shape already goes through. - classifyCapability gains a fourth shape (alongside lookup, search, and extension), "declaration": a $defs key equal to the capability's own `name`, itself holding platform_schema/business_schema. permalink.json uses the identical shape and was also previously reported "matched no known shape" -- catching it too (unprompted by the ticket, which only named identity_linking) is the proof the shape rule generalizes rather than being tailored to one file. buildDeclarationVariantSchema flattens each variant with the existing flattenAllOf helper (already used by buildEntityResponseSchema), inlining the generic capability.json/ucp.json boilerplate so no cross-file $ref into capability.json is needed -- capability.json is never projected into the output tree (only its flattened discovery/capability.json compat form is), which is why the naive "point quicktype at the nested fragment" approach fails with a generic "Internal error: ." (root-caused as a missing file, not a representability limit, by running the identical fragment against the raw unprojected source, where it resolves cleanly). Two narrow fixups on top of flattenAllOf's existing behavior, both needed only because declaration variants keep properties as-is (unlike toCompatLeaf callers, which sidestep both by collapsing every $ref property to a plain string): ucp.json's bare "#/$defs/version" is collapsed to a plain string (the constraint is reattached later by inject-schema-constraints.mjs, same as every other quicktype-dropped constraint); capability.json's "extends" property (the one file-valued $ref anywhere in the boilerplate) is rewritten with the existing rewriteRef helper, scoped to that one property so a capability's own delta properties keep their own, already-correct file refs untouched. - Collateral fix required to make the common/types inclusion succeed: common/types/error_response.json (previously orphaned, so this never surfaced) $refs ucp.json#/$defs/error, which the projected compat ucp.json did not carry. writeCompatibilityCoreSchemas now also derives "success"/"error" (the same allOf-base shape every response_*_schema key is already derived from) by name, not by broadening the shape test alone: ucp.json's own platform_schema/business_schema share the identical allOf[0] shape but are a different, fuller request-side form, so mapping them to the generic response envelope would be semantically wrong, not merely incomplete. Fable-gate follow-up folded into this same commit: generate_models.sh's per-family isolated invocation treated "quicktype exited 0" as success, but every invocation bundles --src discovery/*.json alongside the family's own --src (so quicktype can resolve $refs into it) -- a family whose own type quicktype cannot represent (constraint_expression.json) still exits 0 and still emits the discovery-only content successfully, contributing literally nothing of its own: silently indistinguishable from success by exit code alone. scripts/count-new-exports.mjs (new, directly unit-tested, same pattern as check-generation-completeness.mjs) computes the set of export names a family's fragment contributes beyond a plain discovery-only baseline. Verified directly against the real 2026-08-25 tree that "zero new exports" ALONE is not a safe failure signal: several genuinely successful families also produce it (a bare-scalar common/types root has nothing to name; a common/types file already reachable transitively from discovery/*.json alone nets zero new names on a complete success). generate_models.sh gates on the conjunction instead: zero new exports AND the family's own quicktype log shows the specific "Exceeded maximum number of passes" warning that failure mode emits. Two families genuinely hit this, both added to KNOWN_UNREPRESENTABLE_FAMILIES with the exact evidence: the already-known common/types/constraint_expression, and a second, newly surfaced instance, common/types/request_constraints -- it $refs constraint_expression.json too, but nested (properties.additionalProperties/anyOf.items), not through a bare $ref property, so the existing dropSelfReferentialProperties never caught it; this new loop is what actually found it. Both pins verified (evidence-only; src/spec_generated.ts, pinned to 2026-04-08, is untouched -- see the report for full evidence, model counts, and the byte-identity diff against the committed baseline). 2026-08-25 exports 273 (byte-identical across every re-verification pass in this lane); 2026-04-08 diff against the committed baseline stays exactly 58 pure insertions, 0 removals. New tests/project-credential-identity-defs.test.js (5 tests, 4 red-first against the fixture, the fifth driving quicktype directly on a manifest entry so it passes either way) plus fixture additions (identity_linking.json and three previously-missing common/types fixture files needed once the fixture resolves a full discovery/*.json + declaration-capability invocation for the first time). New tests/count-new-exports.test.js (6 tests, red-first) for the zero-new-exports-plus-warning rule. Updates one now-obsolete assertion in generate-0825-projection.test.js that documented identity_linking as "out of scope" -- true when written, superseded by this change. Fixes a comment miscount along the way: classifyCapability's "declaration" shape is a fourth shape, not a third. 140/140 tests green, tsc pretest clean, build clean, pre-commit clean (ShellCheck could not run in this environment -- Docker unavailable, same environment limitation already flagged on fix/generate-0825-projection -- flagged for CI verification, not silently skipped). Stacked on fix/generate-0825-projection (js-sdk#65, unmerged upstream): if Universal-Commerce-Protocol#65 merges first this branch rebases trivially onto main; if the generation mechanism changes materially before then, this branch re-derives against the new mechanism. Fixes #N Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vishkaty
pushed a commit
to vishkaty/js-sdk
that referenced
this pull request
Aug 31, 2026
…ally fix/generate-0825-projection (js-sdk#65) fixed the generation pipeline itself but left two residuals: (1) common/types/* files with no INBOUND $ref from anything reachable are projected to disk but never handed to quicktype at all, since nothing in generate_models.sh adds common/types/*.json to any --src set. The real-world instance is 2026-08-25's four payment-credential subtypes: card_credential.json, pan_credential.json, network_token_credential.json, token_credential.json. Each only allOf-refs BACK to their shared base (payment_credential.json, which is already reachable and generates fine as PaymentCredentialSchema); nothing refs any of the four subtypes forward, so none of them ever reach quicktype. (2) A capability whose $defs is its own platform/business config declaration, not a checkout/order/cart attachment and not a lookup/search pair, matches none of the shapes discoverAdditionalCapabilities currently recognizes and is reported as a skipped, unmodeled gap. common/identity_linking.json (present at both the 2026-04-08 and 2026-08-25 pins) and shopping/permalink.json (2026-08-25 only) are both instances of this exact shape: a $defs key equal to the capability's own `name`, itself holding platform_schema/business_schema/response_schema sub-keys. Thank you to ptiper for calling both of these out explicitly in js-sdk#61's review and asking that they land as a follow-up right after 0.5.0. Both closed by shape, not by naming the affected files: - discoverCommonTypeFiles adds every common/types/*.json file to a new `types` array in generated-src-manifest.json, which generate_models.sh's existing per-family isolated-invocation loop (built for discovered capabilities) already generically consumes -- one added spread, `...(manifest.types ?? [])`, in the two places that loop reads the manifest. A file already reachable transitively regenerates a byte-identical duplicate that merge-generated-fragment.mjs already skips as "already present"; a file quicktype cannot represent at all fails its own isolated invocation (or, like constraint_expression, exits 0 while emitting nothing, which the loop now also treats as a failure -- see below) and must clear check-generation-completeness.mjs's reviewed KNOWN_UNREPRESENTABLE_FAMILIES gate or the whole run fails loudly, the same safety net a discovered capability's own unrepresentable shape already goes through. - classifyCapability gains a fourth shape (alongside lookup, search, and extension), "declaration": a $defs key equal to the capability's own `name`, itself holding platform_schema/business_schema. permalink.json uses the identical shape and was also previously reported "matched no known shape" -- catching it too (unprompted by the ticket, which only named identity_linking) is the proof the shape rule generalizes rather than being tailored to one file. buildDeclarationVariantSchema flattens each variant with the existing flattenAllOf helper (already used by buildEntityResponseSchema), inlining the generic capability.json/ucp.json boilerplate so no cross-file $ref into capability.json is needed -- capability.json is never projected into the output tree (only its flattened discovery/capability.json compat form is), which is why the naive "point quicktype at the nested fragment" approach fails with a generic "Internal error: ." (root-caused as a missing file, not a representability limit, by running the identical fragment against the raw unprojected source, where it resolves cleanly). Two narrow fixups on top of flattenAllOf's existing behavior, both needed only because declaration variants keep properties as-is (unlike toCompatLeaf callers, which sidestep both by collapsing every $ref property to a plain string): ucp.json's bare "#/$defs/version" is collapsed to a plain string (the constraint is reattached later by inject-schema-constraints.mjs, same as every other quicktype-dropped constraint); capability.json's "extends" property (the one file-valued $ref anywhere in the boilerplate) is rewritten with the existing rewriteRef helper, scoped to that one property so a capability's own delta properties keep their own, already-correct file refs untouched. - Collateral fix required to make the common/types inclusion succeed: common/types/error_response.json (previously orphaned, so this never surfaced) $refs ucp.json#/$defs/error, which the projected compat ucp.json did not carry. writeCompatibilityCoreSchemas now also derives "success"/"error" (the same allOf-base shape every response_*_schema key is already derived from) by name, not by broadening the shape test alone: ucp.json's own platform_schema/business_schema share the identical allOf[0] shape but are a different, fuller request-side form, so mapping them to the generic response envelope would be semantically wrong, not merely incomplete. Fable-gate follow-up folded into this same commit: generate_models.sh's per-family isolated invocation treated "quicktype exited 0" as success, but every invocation bundles --src discovery/*.json alongside the family's own --src (so quicktype can resolve $refs into it) -- a family whose own type quicktype cannot represent (constraint_expression.json) still exits 0 and still emits the discovery-only content successfully, contributing literally nothing of its own: silently indistinguishable from success by exit code alone. scripts/count-new-exports.mjs (new, directly unit-tested, same pattern as check-generation-completeness.mjs) computes the set of export names a family's fragment contributes beyond a plain discovery-only baseline. Verified directly against the real 2026-08-25 tree that "zero new exports" ALONE is not a safe failure signal: several genuinely successful families also produce it (a bare-scalar common/types root has nothing to name; a common/types file already reachable transitively from discovery/*.json alone nets zero new names on a complete success). generate_models.sh gates on the conjunction instead: zero new exports AND the family's own quicktype log shows the specific "Exceeded maximum number of passes" warning that failure mode emits. Two families genuinely hit this, both added to KNOWN_UNREPRESENTABLE_FAMILIES with the exact evidence: the already-known common/types/constraint_expression, and a second, newly surfaced instance, common/types/request_constraints -- it $refs constraint_expression.json too, but nested (properties.additionalProperties/anyOf.items), not through a bare $ref property, so the existing dropSelfReferentialProperties never caught it; this new loop is what actually found it. Both pins verified (evidence-only; src/spec_generated.ts, pinned to 2026-04-08, is untouched -- see the report for full evidence, model counts, and the byte-identity diff against the committed baseline). 2026-08-25 exports 273 (byte-identical across every re-verification pass in this lane); 2026-04-08 diff against the committed baseline stays exactly 58 pure insertions, 0 removals. New tests/project-credential-identity-defs.test.js (5 tests, 4 red-first against the fixture, the fifth driving quicktype directly on a manifest entry so it passes either way) plus fixture additions (identity_linking.json and three previously-missing common/types fixture files needed once the fixture resolves a full discovery/*.json + declaration-capability invocation for the first time). New tests/count-new-exports.test.js (6 tests, red-first) for the zero-new-exports-plus-warning rule. Updates one now-obsolete assertion in generate-0825-projection.test.js that documented identity_linking as "out of scope" -- true when written, superseded by this change. Fixes a comment miscount along the way: classifyCapability's "declaration" shape is a fourth shape, not a third. 140/140 tests green, tsc pretest clean, build clean, pre-commit clean (ShellCheck could not run in this environment -- Docker unavailable, same environment limitation already flagged on fix/generate-0825-projection -- flagged for CI verification, not silently skipped). Stacked on fix/generate-0825-projection (js-sdk#65, unmerged upstream): if Universal-Commerce-Protocol#65 merges first this branch rebases trivially onto main; if the generation mechanism changes materially before then, this branch re-derives against the new mechanism.
carolinerg1
added
status:under-review
and removed
status:needs-triage
Contributor
Author
|
Closing in favor of #61, and thank you to @segiodongo. Re-measured 2026-09-02 on current main 688b627 (v0.5.1): generate_models.sh I said in this PR that whichever landed second would rebase, so I tried. Merging What remains is tracked where it belongs: the generator hand lists that can drop |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
npm run generateagainst a 2026-08-25-shaped spec tree crashesimmediately, and once that crash is fixed, silently produces an incomplete
src/spec_generated.tswith no error at all. This fixes both, generically(no hand lists, no new hardcoded paths), and proves the regeneration
completes end to end against the real 2026-08-25 spec with every exclusion
explicit and reviewed, not just a synthetic fixture.
Root causes (all against upstream/main at 44c448d)
AP2 mandate path hardcode
(
scripts/project-current-ucp-schemas.mjs:1044-1045).writeCompatibilityAp2SchemahardcodessourceRel = "shopping/ap2_mandate.json". The reorg moved AND renamedthis extension (
common/payment_ap2_mandate.json,dev.ucp.shopping.ap2_mandate->dev.ucp.common.payment.ap2_mandate),so the hardcoded path resolves to nothing and the function crashes:
SyntaxError: "undefined" is not valid JSONatclone(). Fixed byresolving the extension with a basename fallback, then a
declared-name-suffix fallback (
resolveSourceSchema), so a furthermove or rename does not need another hardcoded path.
common/ root blindness
(
scripts/project-current-ucp-schemas.mjs:435,449).loadSchemaCachewalks
schemas/shopping/andschemas/common/types/but never theschemas/common/root. Every root-level file the reorg introduced therewas invisible to the generator, not merely unmodeled. Fixed by walking the whole
schemas/tree.Capability/transport allowlist (
generate_models.sh:69-108). The2026-04-08 set of extension/lookup/search capabilities and transport
envelopes was hand-enumerated as
--srcflags. Any capability thereorg added beyond that fixed set never reached quicktype. Fixed with
discoverAdditionalCapabilities/discoverTransportEnvelopes:capabilities are found by SHAPE (a declared
dev.ucp.*name plus alookup_request/lookup_responsepair, asearch_request/search_responsepair, or a$defskey ending in.checkout/.order/.cart) rather than by name, so a future specrelease adding another one needs no script edit. A capability matching
none of these shapes (
identity_linking,permalink) is left aloneand reported.
identity_linkingexists in BOTH the 2026-04-08 and2026-08-25 trees and was never modeled at either pin, so it is a
pre-existing, documented residual;
permalinkis new in 2026-08-25 andmatches no modelable shape, so it is reported rather than dropped.
Core response_*_schema hand-list
(
scripts/project-current-ucp-schemas.mjs:900-909).writeCompatibilityCoreSchemashand-listed exactly fourresponse_*_schemakeys; 2026-08-25 added a fifth(
response_location_schema) that the hand-list silently excluded.Fixed by deriving the full set from the real
ucp.json.Discovery-driven fix design
The common thread across all four root causes is the same failure mode:
the 2026-04-08 spec specific SET of capabilities, files, and keys got
hand-enumerated somewhere in the generator, so anything the next spec
release adds is invisible until someone edits the generator by hand. The
fix is not four unrelated patches -- it is one design applied four times:
discover by shape, from the schema tree itself, never from a fixed
list. A capability schema declares its own identity (a
dev.ucp.*name)and its own shape (lookup/search/attachment/envelope); the generator now
reads that instead of assuming it already knows the full set. This is why
the fix generalizes forward: a 2026-11 spec release adding a sixth
capability needs no generator change at all, as long as it uses one of the
shapes this generator already recognizes.
One further derivation joined the same design: the discovery profile
signing_keysfield was a hand-authored guess in the projection script;it is now derived from
profile.jsonitself (property name and itemschema both), with the 2026-04-08 hand-authored shape as the fallback, so
the pinned output does not move while a spec that publishes the canonical
keysfield (RFC 7517) gets it derived rather than guessed.Two more gaps surfaced only once this discovery mechanism ran against the
REAL spec end to end (not the synthetic fixture, which cannot reproduce
scale-dependent quicktype behavior):
Fragment name collision. quicktype names an untitled schema-mode
fragment from its
$reffragment name.location_lookup.jsonreusesthe same untitled
lookup_request/lookup_responsefragment names asthe pre-existing
catalog_lookup.json, colliding and silently droppingthe fields of one shape. Fixed by titling only its own copy of the
fragment for the newly discovered capability, leaving the pre-existing,
already-working untitled fragments untouched.
A per-family isolation gap, and a genuinely recursive schema.
quicktype cannot represent every JSON Schema shape, and its
type-ordering pass has a fixed iteration budget it can silently exceed
(dropping types with no non-zero exit -- not a crash, just a hole).
Each discovered capability now generates in its own bounded quicktype
invocation, merged into the main output via
scripts/merge-generated-fragment.mjs, so one unrepresentable familycannot corrupt the rest of the regeneration. That isolation mechanism
was ITSELF found to be a fail-open during this work (log a failure and
continue, exit 0 regardless) -- fixed with
scripts/check-generation-completeness.mjs: a family that fails must beon a short, reviewed
KNOWN_UNREPRESENTABLE_FAMILIESallowlist (eachentry citing the exact quicktype failure) or the whole run now exits
non-zero. Separately, 2026-08-25 adds
common/types/constraint_expression.json, a genuinely self-referentialschema (
"$ref": "#"pointing at its own whole document) thatquicktype cannot place at all, verified in total isolation. That failure
silently deleted UNRELATED types sharing the same invocation, including
the entire discovery-profile envelope carrying the new
keysfield (seethe signing_keys derivation above). Fixed by dropping any property whose value is a
$refto a schema containing a bare
"$ref": "#"before quicktype ever seesit, logged loudly -- deliberately NOT matching an ordinary
"#/$defs/..."fragment reference, which is a normal, already-workingJSON Schema idiom.
Both-pin verification
2026-04-08 stays byte-identical. Re-run and diffed against the
actual git-committed
src/spec_generated.ts(viagit show HEAD:src/spec_generated.ts, not a working-tree copy): clean diff,confirmed twice.
None of the fixes above can move it -- the AP2/response-envelope/
discovery fixes are additive fallbacks that only engage when the
2026-04-08 shape is absent; the fragment-titling fix only engages on a
real name collision (none exists in the 2026-04-08 tree); the
self-referential-property fix only engages on a property that is solely
a
$refto a schema containing a bare"$ref": "#"-- the 2026-04-08run logs zero such drops and its output stays byte-identical.
2026-08-25 completes. The real regeneration (not the fixture) now
succeeds end to end: 242 exported schema constants (the committed
2026-04-08 output has 151). Family-count evidence:
PaymentTerms21 references,Loyalty31,PaymentSplit18,Location*32,signing_keys0 (fully replaced by the newkeysfield, present on
UcpDiscoveryProfileSchema).identity_linkingandthe four credential subtype schemas (card/PAN/network-token/token) are
confirmed absent in BOTH pins for the same reason (an out-of-scope
capability shape, and orphaned/unreferenced type files respectively) --
not something this PR introduces or is expected to fix.
payment_authenticationis the one reviewed exclusion: its namedAction keys plus a schema-typed additionalProperties catch-all on one
object hit a quicktype typescript-zod internal error
(
Error: Internal error: .), verified with a minimal isolated repro.Kill test
Reverted
generate_models.shandscripts/project-current-ucp-schemas.mjsto their pre-fix
upstream/mainversions, keeping the tests from thisPR: all 11 tests in
tests/generate-0825-projection.test.jsfail, the 9allowlist-gate tests stay green (they exercise the untouched new script
by design), and the real
2026-08-25 regeneration crashes with the exact original error
(
SyntaxError: "undefined" is not valid JSON). Restoring the fix returnsboth to green. This confirms the tests actually exercise the fix, not just
its surface.
Tests
129/129 (
npm run pretest && npm test),npm run build, andnpm run build:noEmitall green. New coverage: 11 tests intests/generate-0825-projection.test.js(each root cause, the fragmentcollision, and the self-referential-property drop, all against the
fixture spec tree) plus 9 tests in
tests/check-generation-completeness.test.js(the allowlist gate, directand via CLI, no quicktype or spec tree needed).
Coexistence with #61
#61 also targets the 2026-08-25 spec: it extends the generator hand lists
far enough to produce an output and commits that output. The two PRs
overlap in two places:
src/spec_generated.tsitself: this PR does not touch it (the2026-08-25 regeneration output is deliberately left uncommitted here,
out of scope for a generator fix) while feat: update sdk models for ucp 2026-08-25 #61 does. No conflict on this
file either way.
generate_models.sh/scripts/project-current-ucp-schemas.mjs: feat: update sdk models for ucp 2026-08-25 #61edits the same regions this PR rewrites (the QUICKTYPE_ARGS block and
the schema-cache/type-list areas), so whichever PR lands second needs a
rebase of these two scripts plus a regeneration. The substance is
complementary rather than competing: once the discovery mechanism from
this PR is in the tree, a 2026-08-25 regeneration completes with every
family present or explicitly excluded, which is the property the
checked-in output from feat: update sdk models for ucp 2026-08-25 #61 needs in order to be complete on the next
regeneration.
Fixes #64