Skip to content

Publish what the background scheduler has been doing - #291

Merged
haksungjang merged 1 commit into
mainfrom
feat/task-run-metrics
Sep 3, 2026
Merged

Publish what the background scheduler has been doing#291
haksungjang merged 1 commit into
mainfrom
feat/task-run-metrics

Conversation

@haksungjang

Copy link
Copy Markdown
Contributor

O5, on top of the task_runs table #285 added.

The gap

The endpoint counted things that exist (projects, scans, findings) and said nothing about work that runs on a schedule. A task failing every night for a week looked identical to one that has never run: neither appeared at all.

Four series

Series Labels
trusca_task_runs_24h task, outcome runs in the last day; outcome=running are runs that started and never reported an end
trusca_task_run_duration_seconds_p50_24h task median duration of runs that finished
trusca_task_run_duration_seconds_p95_24h task same at the tail
trusca_task_runs_last_recorded_timestamp_seconds unix time of the newest row

Why gauges and not counters

task_runs has a retention sweep. A cumulative count goes down when the sweep runs, and a collector reads a falling counter as a process restart, so rate() is wrong across that point.

The same reasoning rules out histogram buckets. _bucket promises a cumulative counter and collectors rate() it before histogram_quantile(); a sliding window does not satisfy that, and the result is numbers that render fine and mean nothing.

Why the window is in the name

Changing the window changes what the series means, and a series whose definition moved silently cannot be compared with its own past. That is also why it is a constant rather than a setting, plus one more reason: deployments with different windows cannot be read on one dashboard.

p99 is deliberately absent. Schedules here run from every five minutes to weekly, so a day holds one or two runs of the sparse ones, and a 99th percentile over two samples is the maximum wearing a statistic's name.

The one that watches the watcher

trusca_task_runs_last_recorded_timestamp_seconds exists because of a fault found while building #285. Recording is designed never to fail a task, so a missing grant or an unrun migration produces no error anywhere: tasks keep succeeding, the table stays empty, and everything looks healthy. This value going stale is the only outward sign.

No threshold ships with it. Which schedules a deployment enables is a local fact, so the judgement belongs to whatever scrapes this.

Verification

6 integration tests against a real database, since the aggregates are percentile_cont and a grouped count and would otherwise assert the shape of a query rather than its result. Two of them carry the design: unfinished runs are counted rather than dropped, and the window actually bounds the aggregate.

Contract fixture updated (the test asserts both the series list and its order). ruff clean, mypy clean across 823 files, existing metrics and contract tests green. Admin guide updated in both locales, including why these are gauges and why the last-recorded series deserves its own alert.

The metrics endpoint counted things that exist (projects, scans, findings)
but said nothing about work that runs on a schedule. A task that has been
failing every night for a week looked identical to one that has never run,
because neither appeared at all.

Four series over task_runs close that. Runs by task and outcome, duration at
p50 and p95, and the unix time of the newest row.

All four are gauges over a fixed 24-hour window rather than counters. The
table has a retention sweep, so a cumulative count goes down when the sweep
runs and a collector reads a falling counter as a process restart, which
makes rate() wrong across that point. The same reasoning rules out histogram
buckets: _bucket promises a cumulative counter and collectors rate() it
before taking a quantile, which a sliding window does not satisfy.

The window is in the metric name because changing it changes what the series
means, and a series whose definition moved silently cannot be compared with
its own past. It is a constant rather than a setting for the same reason,
plus one more: deployments with different windows cannot be read on one
dashboard.

p99 is left out. Schedules here range from every five minutes to weekly, so a
day holds one or two runs of the sparse ones, and a 99th percentile of two
samples is the maximum wearing a statistic's name.

Runs with a NULL outcome are counted as running rather than dropped. That is
the shape a killed worker leaves, and an absent series reads on a dashboard
as nothing being wrong.

The last-recorded timestamp watches the recorder itself. Recording is designed
never to fail a task, which means a missing grant or an unrun migration
produces no error anywhere: tasks keep succeeding and the table quietly stays
empty. This value going stale is the only outward sign. No threshold ships
with it, because which schedules a deployment enables is a local fact.
@haksungjang
haksungjang merged commit 812754d into main Sep 3, 2026
30 checks passed
@haksungjang
haksungjang deleted the feat/task-run-metrics branch September 3, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant