Publish what the background scheduler has been doing - #291
Merged
Conversation
The metrics endpoint counted things that exist (projects, scans, findings) but said nothing about work that runs on a schedule. A task that has been failing every night for a week looked identical to one that has never run, because neither appeared at all. Four series over task_runs close that. Runs by task and outcome, duration at p50 and p95, and the unix time of the newest row. All four are gauges over a fixed 24-hour window rather than counters. The table has a retention sweep, so a cumulative count goes down when the sweep runs and a collector reads a falling counter as a process restart, which makes rate() wrong across that point. The same reasoning rules out histogram buckets: _bucket promises a cumulative counter and collectors rate() it before taking a quantile, which a sliding window does not satisfy. The window is in the metric name because changing it changes what the series means, and a series whose definition moved silently cannot be compared with its own past. It is a constant rather than a setting for the same reason, plus one more: deployments with different windows cannot be read on one dashboard. p99 is left out. Schedules here range from every five minutes to weekly, so a day holds one or two runs of the sparse ones, and a 99th percentile of two samples is the maximum wearing a statistic's name. Runs with a NULL outcome are counted as running rather than dropped. That is the shape a killed worker leaves, and an absent series reads on a dashboard as nothing being wrong. The last-recorded timestamp watches the recorder itself. Recording is designed never to fail a task, which means a missing grant or an unrun migration produces no error anywhere: tasks keep succeeding and the table quietly stays empty. This value going stale is the only outward sign. No threshold ships with it, because which schedules a deployment enables is a local fact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
O5, on top of the
task_runstable #285 added.The gap
The endpoint counted things that exist (projects, scans, findings) and said nothing about work that runs on a schedule. A task failing every night for a week looked identical to one that has never run: neither appeared at all.
Four series
trusca_task_runs_24htask,outcomeoutcome=runningare runs that started and never reported an endtrusca_task_run_duration_seconds_p50_24htasktrusca_task_run_duration_seconds_p95_24htasktrusca_task_runs_last_recorded_timestamp_secondsWhy gauges and not counters
task_runshas a retention sweep. A cumulative count goes down when the sweep runs, and a collector reads a falling counter as a process restart, sorate()is wrong across that point.The same reasoning rules out histogram buckets.
_bucketpromises a cumulative counter and collectorsrate()it beforehistogram_quantile(); a sliding window does not satisfy that, and the result is numbers that render fine and mean nothing.Why the window is in the name
Changing the window changes what the series means, and a series whose definition moved silently cannot be compared with its own past. That is also why it is a constant rather than a setting, plus one more reason: deployments with different windows cannot be read on one dashboard.
p99 is deliberately absent. Schedules here run from every five minutes to weekly, so a day holds one or two runs of the sparse ones, and a 99th percentile over two samples is the maximum wearing a statistic's name.
The one that watches the watcher
trusca_task_runs_last_recorded_timestamp_secondsexists because of a fault found while building #285. Recording is designed never to fail a task, so a missing grant or an unrun migration produces no error anywhere: tasks keep succeeding, the table stays empty, and everything looks healthy. This value going stale is the only outward sign.No threshold ships with it. Which schedules a deployment enables is a local fact, so the judgement belongs to whatever scrapes this.
Verification
6 integration tests against a real database, since the aggregates are
percentile_contand a grouped count and would otherwise assert the shape of a query rather than its result. Two of them carry the design: unfinished runs are counted rather than dropped, and the window actually bounds the aggregate.Contract fixture updated (the test asserts both the series list and its order).
ruffclean,mypyclean across 823 files, existing metrics and contract tests green. Admin guide updated in both locales, including why these are gauges and why the last-recorded series deserves its own alert.