Publishing site Sun Nov 23 16:44:46 CET 2025

Author: lbroudoux

documentation/explanations/conformance-testing/index.html

@@ -44,7 +44,7 @@
 <i class="fas fa-chevron-down toggle-icon"></i></a><li data-nav-id=https://microcks.io/documentation/references/apis/ title="API Reference" class=sidelist-mobile><a href=https://microcks.io/documentation/references/apis/ class="d-flex align-items-center justify-content-between"><span><i class="fa-solid fa-caret-right"></i> &nbsp; API Reference</span>
 <i class="fas fa-chevron-down toggle-icon"></i></a></ul></li></ul></div></div></div></div><div class="col-xl-8 col-lg-9 col-md-8"><div class="px-lg-5 px-sm-4 py-5"><h2 class="mb-4 fw-bold">Conformance testing</h2><span class="text-muted fw-300">🗓️ Last updated on March 11, 2025
 <span class="mx-1 text-dark-50 fw-300">|</span> <i class="far fa-clock text-dark me-1"></i> 3
-<span class="mx-1 text-dark-50 fw-300">|</span> <a href=https://github.com/microcks/microcks.io/edit/master/content/documentation/explanations/conformance-testing.md class="f6 ph3 pv1 br2 dib tc ttu mv3 bg-primary-color white hover-bg-green link">Improve this page</a></span><div class="doc-content mt-4 pt-4" data-bs-spy=scroll data-bs-target=#TableOfContents data-bs-offset=20><h2 id=introduction>Introduction</h2><p>You may have experienced the frustration of deploying to production, only to discover that an API or service you integrate with has broken its contract. How can we effectively ensure this does not happen?</p><p>As introduced in <a href=/documentation/overview/main-concepts>Main Concepts</a>, Microcks can be used for <strong>Contract conformance testing</strong> of API or services under development. You spend a lot of time describing request/response pairs and matching rules: it would be a shame not to use this sample as test cases once development is underway!</p><p>On the internet, you can find many different representations of how the different testing techniques relate to one another and should ideally be combined into a robust testing pipeline. At Microcks, we particularly like the Watirmelon representation below. Microcks enables you to automate <strong>Automated API Tests</strong> and focus specifically on <strong>Contract conformance testing</strong>.</p><br><div align=center><img alt="Ideal Software Testing Pyrami" src=https://miro.medium.com/max/1400/0*f2vFclaitRRo1w2i.jpg style=max-width:70%;border-color:#ddd;border-style:solid!important></div><br><p>The purpose of Microcks tests is precisely to check that the <strong>Interaction Contract</strong> - as represented by an OpenAPI or AsyncAPI specification, a Postman collection or whatever <a href=/documentation/references/artifacts>supported Artifact</a> - consumer and producer agreed upon is actually respected by the API provider. In other words, to check that an implementation of the API is conformant to its contract.</p><blockquote><p>💡 If you want to learn more about this topic and get into the details on <strong>how Microcks is different from other contract-testing or conformance testing solutions</strong>, we&rsquo;ve got you covered! We recommend having a read of this two articles: <a href=https://medium.com/@lbroudoux/microcks-and-pact-for-api-contract-testing-3e0e7d4516ca target=_blank>Microcks and Pact for API contract testing</a> and <a href=https://medium.com/@lbroudoux/different-levels-of-api-contract-testing-with-microcks-ccc0847f8c97 target=_blank>Different levels of API contract testing with Microcks</a></p></blockquote><h2 id=conformance-metrics>Conformance metrics</h2><p>In order to help you get confidence in your implementations, we developed the <strong>Conformance index</strong> and <strong>Conformance score</strong> metrics that you can see on the top right of each API | Service details page:</p><br><a href=/images/documentation/test-conformance.png style=display:block class=glightbox><img title loading=lazy decoding=async class=img width=2802 height=846 src=/images/documentation/test-conformance_hue4ee572963f6328ddf8c45d038e52e92_171737_2802x846_resize_q90_h2_lanczos_3.webp alt=image onerror='this.onerror="null",this.src="/images/documentation/test-conformance_hue4ee572963f6328ddf8c45d038e52e92_171737_2802x846_resize_lanczos_3.png"'>
+<span class="mx-1 text-dark-50 fw-300">|</span> <a href=https://github.com/microcks/microcks.io/edit/master/content/documentation/explanations/conformance-testing.md class="f6 ph3 pv1 br2 dib tc ttu mv3 bg-primary-color white hover-bg-green link">Improve this page</a></span><div class="doc-content mt-4 pt-4" data-bs-spy=scroll data-bs-target=#TableOfContents data-bs-offset=20><h2 id=introduction>Introduction</h2><p>You may have experienced the frustration of deploying to production, only to discover that an API or service you integrate with has broken its contract. How can we effectively ensure this does not happen?</p><p>As introduced in <a href=/documentation/overview/main-concepts>Main Concepts</a>, Microcks can be used for <strong>Contract conformance testing</strong> of API or services under development. You spend a lot of time describing request/response pairs and matching rules: it would be a shame not to use this sample as test cases once development is underway!</p><p>On the internet, you can find many different representations of how the different testing techniques relate to one another and should ideally be combined into a robust testing pipeline. At Microcks, we particularly like the Watermelon representation below. Microcks enables you to automate <strong>Automated API Tests</strong> and focus specifically on <strong>Contract conformance testing</strong>.</p><br><div align=center><img alt="Ideal Software Testing Pyrami" src=https://miro.medium.com/max/1400/0*f2vFclaitRRo1w2i.jpg style=max-width:70%;border-color:#ddd;border-style:solid!important></div><br><p>The purpose of Microcks tests is precisely to check that the <strong>Interaction Contract</strong> - as represented by an OpenAPI or AsyncAPI specification, a Postman collection or whatever <a href=/documentation/references/artifacts>supported Artifact</a> - consumer and producer agreed upon is actually respected by the API provider. In other words, to check that an implementation of the API is conformant to its contract.</p><blockquote><p>💡 If you want to learn more about this topic and get into the details on <strong>how Microcks is different from other contract-testing or conformance testing solutions</strong>, we&rsquo;ve got you covered! We recommend having a read of this two articles: <a href=https://medium.com/@lbroudoux/microcks-and-pact-for-api-contract-testing-3e0e7d4516ca target=_blank>Microcks and Pact for API contract testing</a> and <a href=https://medium.com/@lbroudoux/different-levels-of-api-contract-testing-with-microcks-ccc0847f8c97 target=_blank>Different levels of API contract testing with Microcks</a></p></blockquote><h2 id=conformance-metrics>Conformance metrics</h2><p>In order to help you get confidence in your implementations, we developed the <strong>Conformance index</strong> and <strong>Conformance score</strong> metrics that you can see on the top right of each API | Service details page:</p><br><a href=/images/documentation/test-conformance.png style=display:block class=glightbox><img title loading=lazy decoding=async class=img width=2802 height=846 src=/images/documentation/test-conformance_hue4ee572963f6328ddf8c45d038e52e92_171737_2802x846_resize_q90_h2_lanczos_3.webp alt=image onerror='this.onerror="null",this.src="/images/documentation/test-conformance_hue4ee572963f6328ddf8c45d038e52e92_171737_2802x846_resize_lanczos_3.png"'>
 </a><script>window.addEventListener("load",e=>{const t=GLightbox()})</script><br><p>The <strong>Conformance index</strong> is a kind of grade that estimates how your API contract is actually covered by the samples you&rsquo;ve attached to it. We compute this index based on the number of samples you&rsquo;ve got for each operation, the complexity of the dispatching rules of these operations and so on&mldr; It represents the maximum possible conformance score you may achieve if all your tests are successful.</p><p>The <strong>Conformance score</strong> is the current score computed during your last test execution. We also added a trend computation to show whether things are going better or worse compared to your history of tests on this API.</p><p>Once you have activated <a href=/documentation/guides/administration/organizing-repository/#1-applying-labels>labels filtering</a> on your repository and have run a few tests, Microcks is also able to give you an aggregated view of your API patrimony in terms of <strong>Conformance Risks</strong>. The tree map below is displayed on the <em>Dashboard</em> page and represents risks in terms of average score per group of APIs (depending on the concept you chose, it could be per domain, per application, per team, &mldr;)</p><div align=center><br><figure><img src=/images/documentation/test-conformance-risks.png alt=image width=90%></figure><br></div><p>This visualization allows you to have a clear understanding of your conformance risks at first glance!</p><h2 id=tests-history-and-details>Tests history and details</h2><p>/mocks/#mocks-info
 The test history for an API/Service is easily accessible from the API | Service <a href=/documentation/tutorials/getting-started/#viewing-an-api>summary page</a>. Microcks keeps a history of all the launched tests on an API/Service version. Successes and failures are kept in a database with a unique identifier and test number to allow you to compare cases of success and failure.</p><a href=/images/documentation/test-history.png style=display:block class=glightbox><img title loading=lazy decoding=async class=img width=1666 height=1078 src=/images/documentation/test-history_hu3bb808b66fdb93f8f1832a5d0c6220f2_273441_1666x1078_resize_q90_h2_lanczos_3.webp alt=image onerror='this.onerror="null",this.src="/images/documentation/test-history_hu3bb808b66fdb93f8f1832a5d0c6220f2_273441_1666x1078_resize_lanczos_3.png"'></a><p>Specific test details can be visualized: Microcks also records the request and response pairs exchanged with the tested endpoint so that you&rsquo;ll be able to access payload content as well as headers. Failures are tracked, and violated assertions messages are displayed as shown in the screenshot below :</p><a href=/images/documentation/test-details.png style=display:block class=glightbox><img title loading=lazy decoding=async class=img width=1666 height=1101 src=/images/documentation/test-details_hu02771827a8c02e1e25583a36d0cc44fe_277912_1666x1101_resize_q90_h2_lanczos_3.webp alt=image onerror='this.onerror="null",this.src="/images/documentation/test-details_hu02771827a8c02e1e25583a36d0cc44fe_277912_1666x1101_resize_lanczos_3.png"'></a></div><nav class=doc-pagination><a class="nav nav-prev nav-group" href=/documentation/explanations/mcp-endpoints/><i class="fas fa-chevron-left me-2"></i>
 <span class="d-none d-sm-block">MCP endpoints support</span> <span class="d-block d-sm-none">Prev</span>