feat(dashboard): tokens and cache visualizations (input/output, cache hit ratio) - #15
Open
desek wants to merge 4 commits into
Open
feat(dashboard): tokens and cache visualizations (input/output, cache hit ratio)#15desek wants to merge 4 commits into
desek wants to merge 4 commits into
Conversation
Add a stat panel that shows average token throughput over the selected range, split by token type (input, output, cache creation, cache read). Each type is a separate, independently-scaled tile, so input and output stay readable next to the much larger cache-read figure. The per-second figure is the summed token total (last_over_time of each series) divided by the range length in seconds, not rate(): each session is a short-lived cumulative series, so rate() returns no value over an idle window. The agent, repository, branch, and model variables filter the panel, matching the other panels. Cost by repository is narrowed from full width to make room. Verified in Grafana: the four type tiles render with data (input 3.46/s, output 277/s, cacheCreation 1.61K/s, cacheRead 88.9K/s over a 6h range).
Replace the 'Tokens per second by type' stat with a 'Tokens and cache' section holding three panels, per design discussion: - Input and output tokens: a diverging timeseries, output drawn upward and input downward from a centered zero baseline (input negated in the query), each point increase() over the bucket so it counts tokens generated in that interval. - Cache hit ratio: a gauge over the dashboard range, cacheRead / (cacheRead + cacheCreation + input) = share of prompt tokens served from cache. - Cache hit ratio over time: the same ratio per bucket from increase(). The time series use increase() per bucket rather than last_over_time, a deliberate choice: it counts tokens per interval but is blank on idle intervals, because increase() needs more than one sample in the bucket on these short-lived per-session counters. Cost by repository is restored to full width and the lower sections shift down to make room. Verified in Grafana over a 6h range: output/input bars render around the active window, the gauge reads 97.9%, and the ratio line tracks ~95-100% during activity.
desek
changed the title
Add a total-throughput timeseries to the Tokens and cache section for monitoring tokens/sec by time of day. Each point is increase() over the bucket converted to a per-second rate (increase * 1000 / $__interval_ms); idle intervals are filled with 0 via 'or vector(0)', because no tokens flowing is a true zero rate, not missing data, which keeps the daily curve continuous. Counts all token types, so cache reads dominate the height; the bucket grows with the range so a wider window smooths the line. The section is rearranged into a 2x2 grid so the new panel fits without pushing the lower sections further down: input/output and cache-ratio-over-time on top, tokens/sec and the cache-ratio gauge below. Verified in Grafana over 24h: a continuous line with a clear daily pattern, mean ~9K tokens/sec and a ~95K/sec peak.
Scope the throughput monitor to the output token type only, so it tracks the generation rate rather than total token movement dominated by cache reads. Retitle to 'Output tokens per second'. Idle intervals still fill with 0 to keep the daily curve continuous. Verified in Grafana over 24h: mean ~47 output tokens/sec, peak ~800, with a clear active-hours pattern and a zero baseline overnight.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Tokens and cache section to the dashboard (between "Cost and tokens" and "Activity and outcomes") with three panels, designed against this stack's data model. Supersedes the earlier "Tokens per second by type" stat, which is removed.
1. Input and output tokens (diverging timeseries)
Output drawn upward, input downward from a centered zero baseline (input is negated in the query). Each point is
increase()over the bucket, so it counts tokens generated in that interval.2. Cache hit ratio (gauge, over the dashboard range)
cacheRead / (cacheRead + cacheCreation + input)— the share of all prompt (input-side) tokens served from cache. Useslast_over_timeover$__range, consistent with the other range-scoped panels. Reads ~97.9% currently.3. Cache hit ratio over time (timeseries)
The same ratio per bucket from
increase().Data-model note (deliberate design choice)
The two time series use
increase()per bucket, notrate()/last_over_time. Measured on this stack,rate()/increase()return a value only where a session emitted more than one sample in the bucket, so idle intervals are blank on purpose. This was chosen over the always-continuouslast_over_timebecause it counts true per-bucket tokens; the gauge, which only needs a range total, useslast_over_time.Layout
"Cost by repository" is restored to full width; the "Activity and outcomes" and "Conversation and traces" sections shift down to make room. All panels honor the agent, repository, branch, and model variables.
Verification
Rendered in Grafana over a 6h range: output/input bars render around the active window (with honest idle gaps), the gauge reads 97.9%, and the ratio line tracks ~95–100% during activity.