We ran the suite against an independent merchant implementation (the example server in agentic-commerce-alliance/ucp-php-sdk, a PHP UCP SDK) at the pinned commit fdbdafd. Unmodified, it reports 1 passed / 63 failed / 13 skipped — and almost none of those failures are about the merchant.
Two defects account for it. Both are cases where the suite depends on the reference server's specific behaviour rather than on the protocol.
1. The mock agent profile declares one capability, the tests exercise seven
shopping-agent-test.json declares only dev.ucp.shopping.order, while the tests drive catalog, cart, checkout, discount and fulfillment.
A Business is required to negotiate capabilities against the Platform profile it is given, and to refuse operations outside the intersection. So a conformant merchant answers capabilities_incompatible to almost everything — correctly. 59 of the 63 failures share that one cause, most of them in tests asserting something else entirely that merely needed a checkout to exist first.
2. protocol_test.py sends a literal profile="..."
The UCP-Agent header is sent with the placeholder text rather than a resolvable profile URL, so a merchant that actually fetches the profile — as the spec requires — cannot.
Effect
|
Passed |
Failed |
| Unmodified |
1 |
63 |
| With both fixed |
5 |
59 |
| With all our fixes (see below) |
58 |
6 |
The first row is what any non-reference implementation sees today.
Patches
We carry these as patches and apply them in our CI lane. Happy to open PRs for any of them if that's easier — say the word and I'll send them.
Related issues follow for the other defects we hit. Context and per-defect reasoning:
docs/upstream/conformance-suite-defects.md.
Worth saying plainly: the suite found five genuine interop bugs in our SDK that our own test suite could not — discount codes read in a shape no schema defines, an idempotent replay that wasn't byte-identical, order webhooks never dispatched, an ignored UCP-Agent version parameter, and a mis-typed error code. It is doing its job. These reports are meant to help it do that for implementations other than the reference one.
We ran the suite against an independent merchant implementation (the example server in
agentic-commerce-alliance/ucp-php-sdk, a PHP UCP SDK) at the pinned commitfdbdafd. Unmodified, it reports 1 passed / 63 failed / 13 skipped — and almost none of those failures are about the merchant.Two defects account for it. Both are cases where the suite depends on the reference server's specific behaviour rather than on the protocol.
1. The mock agent profile declares one capability, the tests exercise seven
shopping-agent-test.jsondeclares onlydev.ucp.shopping.order, while the tests drive catalog, cart, checkout, discount and fulfillment.A Business is required to negotiate capabilities against the Platform profile it is given, and to refuse operations outside the intersection. So a conformant merchant answers
capabilities_incompatibleto almost everything — correctly. 59 of the 63 failures share that one cause, most of them in tests asserting something else entirely that merely needed a checkout to exist first.2.
protocol_test.pysends a literalprofile="..."The
UCP-Agentheader is sent with the placeholder text rather than a resolvable profile URL, so a merchant that actually fetches the profile — as the spec requires — cannot.Effect
The first row is what any non-reference implementation sees today.
Patches
We carry these as patches and apply them in our CI lane. Happy to open PRs for any of them if that's easier — say the word and I'll send them.
0001-protocol-test-real-profile-url.patch0002-agent-profile-declare-exercised-capabilities.patchRelated issues follow for the other defects we hit. Context and per-defect reasoning:
docs/upstream/conformance-suite-defects.md.Worth saying plainly: the suite found five genuine interop bugs in our SDK that our own test suite could not — discount codes read in a shape no schema defines, an idempotent replay that wasn't byte-identical, order webhooks never dispatched, an ignored
UCP-Agentversion parameter, and a mis-typed error code. It is doing its job. These reports are meant to help it do that for implementations other than the reference one.