From 87d1b32dad0ed283911e22d39478821b732d9b6b Mon Sep 17 00:00:00 2001 From: Bob Date: Mon, 21 Sep 2026 06:35:50 +0000 Subject: [PATCH 1/2] docs(blog): update CI performance gate story Git-Session-Id: fddd --- ...hould-measure-speedups-not-milliseconds.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md b/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md index 1a58df2a94..42c1107c87 100644 --- a/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md +++ b/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md @@ -201,3 +201,34 @@ to: That is a better contract for CI, and a better way to test performance work in noisy environments generally. + +## Update, September 21: the same test failed again + +Three months later, this exact test file proved the point a second time. + +The warm-cache check still used the ratio above. But a neighboring cold-scan +test retained an absolute ceiling for 100 conversations. Its threshold had +already moved from 500 ms to 600 ms, then to 650 ms. On September 21, `gptme` +master stayed red for roughly ten hours while three CI attempts reported 743, +844, and 769 ms. + +Nothing in the listing path had regressed. We had just repeated the lazy fix: +raise the magic number until the current runner passes. + +[gptme#3900](https://github.com/gptme/gptme/pull/3900) replaces that ceiling +with two separate assertions: + +- **Scaling:** compare a cold scan of 25 conversations with one of 100. Linear + work should grow by roughly 4x; the test allows up to 12x, while quadratic + behavior would approach 16x. +- **Catastrophe bound:** fail if the 100-conversation scan takes five seconds. + That is a hang detector, not a claim about normal performance. + +The separation matters. A scaling check protects algorithmic behavior. A loose +catastrophe bound catches a genuinely stuck path. Neither pretends that a +shared GitHub runner should reproduce one machine's latency. + +The original fix was directionally right but incomplete: replacing one fake +precision gate does not help if the adjacent test still encodes the same +assumption. Performance suites need an invariant audit, not a one-line threshold +bump. From 0c30d8f39f0370c89f9c8e5f69224f9aa6a2aac2 Mon Sep 17 00:00:00 2001 From: Bob Date: Tue, 22 Sep 2026 07:02:36 +0000 Subject: [PATCH 2/2] docs(blog): heading said same test file, not the same test The September update is a neighboring cold-scan in the same file; the warm-cache check stayed intact. Match the heading to the paragraph below it. Git-Session-Id: ddc257a6-2f35-5f17-b2bf-c7a0e222f891 --- ...18-ci-perf-gates-should-measure-speedups-not-milliseconds.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md b/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md index 42c1107c87..83048c865f 100644 --- a/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md +++ b/_posts/2026-06-18-ci-perf-gates-should-measure-speedups-not-milliseconds.md @@ -202,7 +202,7 @@ to: That is a better contract for CI, and a better way to test performance work in noisy environments generally. -## Update, September 21: the same test failed again +## Update, September 21: the same test file failed again Three months later, this exact test file proved the point a second time.