Skip to content

fix: compute list percentiles over the filtered rows (tasks, AI traces, endpoints) - #319

Open
FrameAutomata wants to merge 4 commits into
mainfrom
fix/task-ai-trace-percentile-parity
Open

fix: compute list percentiles over the filtered rows (tasks, AI traces, endpoints)#319
FrameAutomata wants to merge 4 commits into
mainfrom
fix/task-ai-trace-percentile-parity

Conversation

@FrameAutomata

@FrameAutomata FrameAutomata commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Closes #313. Rebased onto current main (which now carries 95a65d24), and absorbs what was left of #318 — that PR is closed.

The bug

Three list views computed their percentiles from a second query that carried no filter, while the grouping query that selected the rows applied a search/root filter. The percentile therefore described a different population than the rows the user asked for.

This is worse than a mislabelled column. All three lists sort and paginate on those percentiles — tasks via the needsGoSort path for orderBy=p50_duration|p95_duration|impact, AI traces via orderByMap directly, endpoints via sortEndpointStats and the chart's top-5 ranking. So the page was ordered by the population the user had just filtered out. The count column was already correct, which is exactly why it stayed invisible.

SQLite-only: DuckDB and ClickHouse compute the percentiles inline in the filtered aggregate. So it was also a backend parity break — identical data, different numbers and different row order.

What is fixed here vs already on main

95a65d24 fixed the endpoints case while this work was in review, threading rootFilter into fetchSortedDurations. It did not touch tasks or AI traces.

entity state
tasks fixed here — live bug
AI traces fixed here — live bug
endpoints fixed on main; this PR adds forward-compat + the missing test

The fix

Each file routes the selecting query and the duration query through one clause buildertaskFilterClause / aiTraceFilterClause / the existing endpointFilterClause — so the two cannot drift apart, and a predicate added there is picked up by both.

The endpoint change carries no behaviour change: with the endpoint pinned by endpoint = :endpoint, the search and method legs cannot alter the row set, so rootFilter is the only load-bearing one. It buys that a future row-level predicate reaches the percentile query automatically. Picking one leg out by hand is how the original split happened, and all three entities now read the same way — which is why it belongs here rather than in a PR of its own.

Also drops a second parameter map in FindGroupedByTaskName: lit's ParseNamedQuery ignores keys the query never references, and the count query consumes params before pagination is added, so one map serves both.

Verification

Four regression tests. The three that cover a live bug fail on the pre-fix code with exactly these values:

--- FAIL: TestTaskRepository_FindGroupedByTaskName_PercentilesUseFilteredRows
    P50 computed over unfiltered rows: got 5.05s, want 100ms
    P95 computed over unfiltered rows: got 10s, want 100ms
--- FAIL: TestTaskRepository_FindGroupedByTaskName_RanksOnFilteredRows
    page ordered on unfiltered percentiles: got "task.fast-root" first, want "task.slow-root"
--- FAIL: TestAiTraceRepository_FindGroupedByTraceName_PercentilesUseFilteredRows
    P50 computed over unfiltered rows: got 15.1s, want 200ms
--- FAIL: TestAiTraceRepository_FindGroupedByTraceName_RanksOnFilteredRows
    page ordered on unfiltered percentiles: got "trace.fast-root" first, want "trace.slow-root"

The fifth — TestEndpointRepository_GetEndpointStackedChart_RanksOnFilteredRowspasses on main. 95a65d24 fixed that path but only tested the grouped list, so this guards the existing fix rather than reporting a live bug. Stating that so a green run is not mistaken for "reproduced and fixed."

All tests live in the untagged telemetry package, so they run against all three backends and pass on DuckDB and ClickHouse unchanged — that contrast is the parity assertion.

Also reproduced end-to-end, not just in tests. Two binaries on the same seeded SQLite database, populated over real OTLP ingest (root and non-root CONSUMER spans for tasks, gen_ai.* spans for AI traces), queried over HTTP:

main this branch
tasks, rootFilter=all count=8 p50=5050ms p95=10000ms same
tasks, rootFilter=root count=4 p50=5050ms p95=10000ms count=4 p50=100ms p95=100ms
ai-traces, rootFilter=root count=4 p50=15100ms p95=30000ms count=4 p50=200ms p95=200ms

Local: go test -race -count=1 ./app/..., the DuckDB-tagged suites, go vet ./app/..., gofmt — all clean.

Follow-ups, not in scope

  • idx_endpoints_project_endpoint is (project_id, endpoint) with no recorded_at, so per-group duration queries scan a group's whole retained history to answer a short window.
  • The structural fix: SQLite splits the percentile query only because it has no percentile aggregate. One filtered query ordered by (group, duration) partitioned in Go — the shape getStackedChartWithPercentiles already uses — removes the drift and the per-group N+1. The tests here protect that refactor.

🤖 Generated with Claude Code

@FrameAutomata FrameAutomata added the ci Run CI on this PR (remove and re-add to re-validate after a push) label Aug 26, 2026
FrameAutomata and others added 4 commits August 26, 2026 16:32
The same split #318 fixes for endpoints was live in two sibling
repositories on the SQLite telemetry backend.

`FindGroupedByTaskName` and `FindGroupedByTraceName` both apply their
search + root filter to the grouping query, then compute percentiles
from a second query that carried no filter at all. So a filtered `count`
was reported next to unfiltered P50/P95 -- and because both lists sort
and paginate on those values (`orderBy=p50_duration|p95_duration|impact`
takes the Go-sort path in tasks, and the AI-traces list orders on them
directly), the page was ordered by the population the user had just
filtered out, not only mislabelled.

DuckDB and ClickHouse compute the percentiles inline in the filtered
query and were already correct, so this was also a storage-backend
parity break: identical data, different numbers and different row order.

Both files now route the selecting query and the duration query through
one clause builder -- `taskFilterClause` / `aiTraceFilterClause`,
mirroring `endpointFilterClause` -- so the two cannot drift apart again,
and a predicate added there is picked up by both. In tasks that also
removes a second, hand-maintained copy of the search parameter: the
group query's parameter map is now copied from the filtered one rather
than re-deriving one leg of it.

Verified end-to-end, not just in tests: two binaries on the same seeded
database populated over real OTLP ingest, queried over HTTP.

  tasks     rootFilter=root   main: count=4 p50=5050ms  p95=10000ms
                            branch: count=4 p50=100ms   p95=100ms
  ai-traces rootFilter=root   main: count=4 p50=15100ms p95=30000ms
                            branch: count=4 p50=200ms   p95=200ms

`rootFilter=all` is identical on both.

Refs #313

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review follow-ups on the task/AI-trace percentile fix.

FindGroupedByTaskName kept two parameter maps, the second a copy of the
first plus pagination. The copy is unnecessary: lit's ParseNamedQuery
resolves :names against the map and ignores keys the query never
references, and the count query consumes params and returns before
limit/offset are added, so one map serves both. That also matches the
AI-trace path in this same change and the DuckDB twin, and it means a
future filter leg reaches the group query without anyone remembering to
propagate it.

The AI-trace duration query now names its filter clause instead of
splicing a params-mutating call into the middle of the argument list
that also passes those params -- the form the rest of the package uses.

Comments: the new helpers claimed to be "the single definition" of each
list's filter, which is not true across backends (the DuckDB twins still
inline the same construction). They now describe what they actually do,
including the params mutation the signature does not advertise. The
comment on fetchSortedTaskDurations restated its own signature and is
gone; the tests are the durable record.

Also adds the reordering coverage the percentile test's comment claimed
but never exercised. orderBy=p95_duration takes the Go-sort path, so
unfiltered percentiles reorder the page and not just its numbers; the
new test has two tasks that rank one way on all rows and the other way
on root rows alone, and it fails on main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ts caller

Review follow-ups.

FindGroupedByTraceName sorts and paginates in Go on P50/P95, exactly
like the task list, so unfiltered percentiles reorder the page and not
just its numbers -- but the AI test only covered the reporting half
under orderBy=count, where ordering cannot fail. A regression that
reverted only the ai_trace change would have passed. The new test
mirrors the task one and fails on main.

taskFilterClause sat ~350 lines below its only two callers, while the
helpers it mirrors (aiTraceFilterClause, endpointFilterClause) both sit
directly above the grouped-list function they serve. Moved, no other
change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds on 95a65d2, which fixed the endpoint percentile/filter split
while this branch was in review. That commit threads rootFilter into
fetchSortedDurations and splices shared.RootFilterClause directly.

Two follow-ups.

First, the duration query now goes through endpointFilterClause like the
queries that select the endpoint, instead of picking out the one leg
that constrains rows today. There is no behaviour change: with the
endpoint pinned by `endpoint = :endpoint`, the search and method legs
cannot alter the row set, so only rootFilter is load-bearing. What it
buys is that a row-level predicate added to endpointFilterClause reaches
the percentile query automatically. Splicing one leg by hand is how the
original split happened, and it is the shape the task and AI-trace
repositories now share.

Second, a regression test for the chart ranking. 95a65d2 fixed
getTopEndpointsByMetric along with the grouped list, but only the
grouped list got a test. The chart ranks its top 5 on percentiles and
then plots only the filtered rows, so ranking on the unfiltered
population puts endpoints on the chart that are not slow under the
filter. This test pins that; it passes on main, so it guards the fix
rather than reporting a live bug.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@FrameAutomata
FrameAutomata force-pushed the fix/task-ai-trace-percentile-parity branch from 2dad44b to 772a089 Compare August 26, 2026 21:33
@FrameAutomata FrameAutomata changed the title fix: compute task and AI-trace percentiles over the filtered rows fix: compute list percentiles over the filtered rows (tasks, AI traces, endpoints) Aug 26, 2026
@FrameAutomata FrameAutomata added ci Run CI on this PR (remove and re-add to re-validate after a push) and removed ci Run CI on this PR (remove and re-add to re-validate after a push) labels Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci Run CI on this PR (remove and re-add to re-validate after a push)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLite/DuckDB endpoint-chart parity: top-5 ranked on unfiltered percentiles, and the method filter silently became case-sensitive

1 participant