feat: alerting kinds, status gauge, grafonnet dashboards, oteldemo kind harness - #11
Merged
Merged
Conversation
- github.com/sebdah/goldie/v2 for golden-file based snapshot tests in the Prometheus generator test suite. - promtool tool pinned via mise for local validation of generated Prometheus rules.
Extends the OpenTelemetry Weaver registry with three new SLI/SLO metrics the Prometheus generator now produces: - openslo.slo.current_burn_rate — instantaneous 5m burn rate / budget - openslo.slo.period_burn_rate — full-window burn rate over the SLO period - openslo.slo.period_error_budget_remaining — 1 minus period_burn_rate The generated Go constants in pkg/semconv/semconv_gen.go are refreshed to match the registry.
The OpenSLO SDK models only a single AlertCondition kind ("burnrate").
opensloctl now accepts the four kebab-case names that map one-to-one
onto the Google SRE Workbook alerting strategies:
- error-rate (workbook §§ 1–3)
- burn-rate (workbook § 4)
- multi-burn-rate (workbook § 5)
- multi-window-multi-burn-rate (workbook § 6)
Per-kind threshold range checks are enforced:
- error-rate: threshold in (0, 1]
- burn-rate families: threshold > 0
The SDK's strict AlertConditionKind OneOf is bypassed for AlertCondition
specs only — every other SDK validation still runs. Duplicate detection,
rolling-window requirement, and target/targetPercent required rules are
preserved.
…ecycle Replaces the placeholder loader_test.go with a focused specstore_test.go exercise covering the full store/validate pipeline: - AlertConditionKind membership (closed set of four kinds) - Threshold range checks per kind (error-rate vs burn-rate families) - Cross-ref resolution for every arc in the SLO graph (SLO→Service/SLI/AlertPolicy, AlertPolicy→Condition/Target, SLI→DataSource) - SLI metric source validation - Duplicate-name detection across all spec kinds - Target/targetPercent required and isRolling: true required post-checks A small internal/testutil helper package is added to construct minimal SLO/SLI/AlertCondition specs inline so tests stay declarative.
The Prometheus generator previously rendered two separate files per SLO
(-recording-rules.yaml and -alert-rules.yaml) using two templates. This
collapses both into a single unified template that emits a single
<slo-name>-rules.yaml file containing:
- SLO info recording rules (info, objective, timewindow_days, error_budget,
current_burn_rate, period_burn_rate, period_error_budget_remaining)
- Windowed SLI error rate recordings (5m, 30m, 1h, …)
- An optional openslo-alerts-<slo> group when AlertPolicies are referenced
The AlertCondition kind drives both the expression shape and the tier
structure:
- error-rate: 1 condition per severity
- burn-rate: 1 condition per severity
- multi-burn-rate: >=2 conditions OR-ed per severity
- multi-window-multi-burn-rate: AND within tier, OR across tiers, derived
by stripping the trailing "-<lookbackWindow>" suffix from condition names
Alert names follow {SloNamePascal}{KindPascal} (PascalCase, no
underscores, severity stays in the label). The legacy "burnrate" kind
maps to multi-window-multi-burn-rate with a one-shot warning.
The standalone prometheus-alert-rules.template.yaml is removed; all alert
rendering flows through the unified template.
…s labels The promLabelsFromOpenSlo helper that maps OpenSLO metadata.labels into Prometheus label key/value pairs now enforces two rules: - Label names with hyphens are rejected — Prometheus requires [a-zA-Z_][a-zA-Z0-9_]* (snake_case), and silently rewriting the user's key would have hidden config typos. - Multi-value labels (more than one entry per key) are rejected — only a single value per label key is supported; the SDK normalizes the single-string YAML form to a one-element slice which is accepted. The feature package's existing multi-dimensional SLI annotations keep working unchanged; their godoc is tidied up.
The PrometheusGenerator already stopped short of two loading-time checks; both are now wired end-to-end: - spec.indicatorRef is resolved into an inline v1.SLOIndicatorInline at generation time. Inline spec.indicator still wins when both are set; the resolver returns a clear error if neither is present. - specstore.GetSpecs validates the ref resolves to a loaded SLI, so the resolver only sees consistent state. The Generator interface gains a Validate() method that runs every generator-side check (indicator resolution, label grammar, template execution, alert structure) without touching the filesystem. The existing createGeneratedFiles path is reused and its in-memory output discarded. Tests use goldie-backed golden files (test-singleline, test-multiline, test-ratio, test-ratio-target-percent, test-tiered) so the rendered Prometheus rules stay under snapshot review; tiered alert expectations live alongside as a separate fixture.
opensloctl gets a third subcommand that runs every validation layer (load-time + generator-side) without writing any output files: - specstore.GetSpecs handles YAML parse, SDK validation, duplicate detection, ref resolution, and per-kind AlertCondition checks. - prometheusgenerator.Validate runs indicator resolution, label grammar, template execution, and alert structure validation. Exits 0 on success, 1 with a slog.Error on any failure — same slog-as-error reporting style as the existing commands. The flag shape mirrors load (-f repeatable, -r recursive), so users can swap one for the other in CI hooks.
Two more complete example directories ship alongside api-latency-slo and
error-budget-slo, showcasing the remaining SRE alerting strategies:
- examples/error-rate-slo/ (kind: error-rate, absolute threshold)
- examples/multi-burn-slo/ (kind: multi-burn-rate, OR-ed windows)
Each new directory follows the same layout (service, datasource, sli,
slo, alert-condition-{page,ticket}, alert-policy-{page,ticket},
notification-target-...) as the existing examples, and ships with its
own README explaining the strategy and how to run/validate it.
The api-latency-slo and error-budget-slo alert conditions are upgraded
from the legacy "burnrate" kind to the kebab-cased "burn-rate" — the
generator still accepts "burnrate" for back-compat but new specs should
use the form documented in the README.
Per-directory READMEs are added for api-latency-slo and error-budget-slo
matching the new examples.
- README: add validate command, expand examples table to five directories, document label name grammar and multi-value validation, note that indicatorRef resolution is supported. - AGENTS.md: extend commands section (three subcommands now), note scratch scripts must live in ./tmp (gitignored). - .gitignore: track opensloctl binary in addition to existing patterns, add tmp/ scratch directory.
…on.target attributes Adds two string attributes that the Prometheus generator stamps onto every generated alert rule label. Both follow the Dot-name ↔ underscore-name split used by the existing registry entries (openslo.slo.name → openslo_slo_name) so the Prometheus label keys remain snake_case: openslo.alert.severity → openslo_alert_severity openslo.notification.target → openslo_notification_target The generated constants in pkg/semconv/semconv_gen.go are refreshed via make semconv-generate.
The Prometheus generator can render the routing target as a single openslo_notification_target label, but only if each AlertPolicy references at most one target. Pre-existing AlertPolicy → AlertNotificationTarget ref resolution still runs; this new check additionally rejects any AlertPolicy with len(spec.notificationTargets) > 1 so future generation can never produce duplicate-label YAML.
buildAlertGroups now treats each AlertPolicy's single notificationTargets[0].targetRef as the routing target for every condition contributed by that policy. The resolved AlertNotificationTarget.spec.target is aggregated per severity in the severityState; when multiple policies for the same severity disagree on their target, slog.Error fires and the openslo_notification_target label is omitted (rather than picking an arbitrary value) so the misconfiguration is obvious in the rules file. The AlertGroup struct gains a NotificationTarget string field that the template renders conditionally; empty stays empty.
…_target labels The Prometheus alert rules template now renders two new labels on every alert rule, replacing the bare "severity" key and adding routing info: openslo_alert_severity: <severity> openslo_notification_target: <target> (only when present) The label names match the weaver registry attributes (dots → underscores for Prometheus compatibility). Golden fixture for the tiered test scenario is regenerated to capture the new labels with notification_target=engineers stitching through.
…and openslo.timewindow.duration These three attributes were declared in the registry but never referenced by any recording rule, golden fixture, or downstream consumer. Mark them with the weaver `deprecated` block (obsoleted reason) so downstream OTel tooling surfaces the deprecation while keeping the constants available in pkg/semconv/semconv_gen.go for backward compat with any pinned generator artifact.
Single integer gauge {0,1,2,3} = Healthy/Burning/Critical/Breached
derived from openslo_slo_current_burn_rate against overridable
thresholds. Defaults 1/6/14.4x per the SRE workbook. Overridable per
SLO via threshold.status.openslo.com/{warning,critical,breached}
annotations (work lands in the specstore validator + generator).
Emitted only for SLOs that reference at least one AlertPolicy.
DOCTYPE partition: integer gauge as a categorical scaffold for
dashboard rendering.
…positive
Implements the validator half of the SLO status gauge work.
Annotations on metadata.annotations:
threshold.status.openslo.com/warning (default 1)
threshold.status.openslo.com/critical (default 6)
threshold.status.openslo.com/breached (default 14.4)
Each annotation is optional; missing annotations fall back to defaults
independently. Non-numeric values silently fall back to defaults so
that scratch notes ("TODO") don't fail validation. The resolved triple
must be strictly ascending and positive; violations result in a
load-time error reported via ValidateRefs + surfaced through
`opensloctl validate`.
Table-driven tests cover partial overrides, defaults, ascending
violations, and zero/negative inputs.
Plugin half of the status-gauge work that pairs with the specstore validator. The generator resolves the three status threshold annotations (defaults from specstore exported constants) and feeds them into the template, which renders an unsigned 0/1/2/3 recording rule under a new openslo-status-recordings-<slo> group: 0 (Healthy) when current_burn_rate < warning (default 1x) 1 (Burning) when warning <= x < critical (default 6x) 2 (Critical) when critical <= x < breached (default 14.4x) 3 (Breached) when x >= breached The structural shape uses disjoint branch comparisons so exactly one value emits per (slo_name, spec_version). Multi-dim SLOs also produce a openslo_slo_status post-process rule via label_join, like the existing slo_info/slo_objective pattern. Tests cover the generator's threshold-resolution helper via the specstore ParseStatusThreshold + ValidateRefs paths; goldens are regenerated under the existing test/ratio/multi-dim fixtures.
Ten new gauge metrics, one per multi-window (5m, 30m, 1h, 2h, 6h, 1d, 3d, 7d, 28d, 30d), mirroring the openslo.sli.error_rate_<window> convention. Each carries events-per-second from the underlying SLI source query — useful in Grafana dashboards for traffic-context panels when interpreting error budget burn rate. Emitted only for RatioMetric SLIs (where total.MetricSource.spec.query is a counter rate); ThresholdMetric SLIs do not supply event count queries so the recording rule is skipped for that path. Generation logic comes in the next commit.
…RatioMetric
Template extension for the second half of the event-rate metric
work. The prometheus.go RatioMetric branch now carries the total
counter query alongside the error-rate query and renders it into
per-window event-rate rules under the existing sli-recordings
group:
openslo_sli_event_rate_<window>{openslo_slo_name, ...} =
sum(rate(<total source query>[<window>]))
ThresholdMetric SLIs skip event-rate emission entirely because the
heysers spec doesn't expose a parallel event-count query (avoids
the heuristic-parse failure path discussed in Fork E).
Multi-dim SLOs also get post-process label_join rules, matching the
existing pattern for the other windowed recordings on the same
schema. Ratio-only goldens (test-ratio-slo, test-ratio-target-percent,
test-multi-dim) are updated; threshold-only goldens (test-multiline,
test-singleline) confirm zero event-rate additions.
Extends the existing Recording Rules subsection with two pieces: 1. Inline bullet for openslo_sli_event_rate_<window>, scoped to RatioMetric SLIs only (ThresholdMetric skip explained). 2. New "Status gauge" subsection for openslo_slo_status with the 0/1/2/3 mapping, SRE-workbook defaults, and the threshold.status.openslo.com/* annotation override knobs (including ascending-positive validation contract).
Single-page dashboard mirroring Grafana Cloud's Manage SLOs view.
Table panel of every SLO loaded by opensloctl, with:
- Service filter bar (label_values on openslo_service_name)
- Columns: SLO, Service, Objective, Window days, Current burn,
Period remaining, Status
- Status rendered from openslo_slo_status with value mappings
(0=Healthy / 1=Burning / 2=Critical / 3=Breached) and color cues
- Drill-down link from SLO cell to openslo-detail?var-slo=${slo}
- Sortable by Current burn × (descending by default)
Datasource wired as ${DS_PROMETHEUS} via __inputs so the JSON
imports cleanly into any Grafana deployment; provisioning binds to
the existing Prometheus datasource (webstore-metrics in the
oteldemo compose).
In-depth single-SLO dashboard mirroring the Grafana Cloud SLO
dashboard sections. Layout (24-col grid):
- Row 1 header stats: SLO name, Service, Objective gauge, Window
- Row 2: SLI rate timeseries (with objective as a dashed reference
line) + SLI 28d average stat
- Row 3: period error budget remaining timeseries + Status stat
(0/1/2/3 mappings: Healthy/Burning/Critical/Breached)
- Row 4: current burn rate timeseries with horizontal threshold
markers at 1x, 6x, 14.4x (SRE workbook burn-rate reference points)
- Row 5: event rate 28d timeseries (RatioMetric SLIs only; falls
back to flat 0 for SLOs without an event_rate metric) +
Active SLO alerts table
Multi-dim SLOs are picked up automatically via a $dim custom
variable that captures the dimension prefix from the SLO name
(when present).
Datasource wired as ${DS_PROMETHEUS} via the same __inputs pattern
as the list dashboard; provisioning binds in deployment.
…shboards Standard Grafana provisioning block scoped to a single "openslo" provider and an OpenSLO folder. Single path option points at the container-internal directory that compose.slos.yaml mounts; the host path is the repo-root deploy/dashboards/. Type: file, updateIntervalSeconds: 30, allowUiUpdates: true so the JSON dashboards stay editable in the UI but auto-refresh when the repo file is changed during dev iterations. disableDeletion: false keeps the dashboards re-creatable when files are removed.
Adds a single host-to-container bind mount: the repo-root `deploy/dashboards` directory is exposed at `/etc/grafana/provisioning/dashboards` inside the Grafana container, where the file-provider (examples/oteldemo/openslo-dashboards.yaml) scans for dashboard JSON files. The dashboards/*.json files + provisioning yaml are picked up on the next Grafana start; in-flight edits to JSON files are picked up within updateIntervalSeconds (30s) without a restart.
The templated openslo_slo_status rule emitted double "}}" — a literal
typo in the source. The Go template parser preserved it as text,
producing promql like
openslo_slo_current_burn_rate{...="v1"}} >= 14.4
which promtool rejects with "unexpected character: '}'".
Trim one extra closing brace so the rendered label-selector closes
with a single '}'. Golden files regenerated; oteldemo rules pass
promtool check with all 12 files reporting 30 rules SUCCESS.
Aligns the README's Attribute/Metrics table with the live weaver registry after this round of work. Additions: - Attributes: openslo.service.name, openslo.alert.severity, openslo.notification.target. - Mark the three deprecated attributes (decimal/percent/duration) in their own table with their obsoleted reason — users can read why they're deprecated without grepping the registry. - Add the four SLO info metrics that didn't exist when the section was first written: current_burn_rate, period_burn_rate, period_error_budget_remaining, status (with cross-link to the Status gauge subsection in Recording Rules). - Add the SLI event rate metric family with the explicit RatioMetric-only constraint. Consuming-the-Registry example now shows the status gauge constant in addition to openslo_slo_info.
The original mount only carried the JSON dashboard files. The file-provider MUST live in the same container directory the JSONs live in or the provider isn't loaded at startup — which is why no "OpenSLO" folder appears in the Grafana UI. Add a second bind-mount for the single provisioning yaml file with the same container directory as its target, so Grafana picks up both the provider registration and the dashboards it points at.
Local opencode CLI config holding per-developer MCP server command/env. Project files only live in the worktree, not in shared history.
Job steps include goreleaser but the workflow also runs go build, go test, and lint; the previous goreleaser name was misleading.
PromQL parses left-to-right at equal operator precedence so the previous "1 - (\nXs\n)\n/\n(\nYs\n)" template emitted (1 - good) / total — the success rate, not the error rate. Downstream burn-rate math inverted: a healthy service read as burning thousands of times its budget, which poisoned the categorical status gauge and turned dashboards red. Move the slash inside the outer parentheses so the expression evaluates as 1 - (good / total). Two ratio golden files regenerated; the rest of the testdata stays untouched so future commits can attribute their own diffs.
The categorical status block emits three integer statuses (0/1/2/3) by multiplying the result of a comparison against burn-rate thresholds: (burn_rate >= 14.4) * 3 # Breached (burn_rate >= 6 and burn_rate < 14.4) * 2 (burn_rate >= 1 and burn_rate < 6) * 1 Comparison operators between an instant vector and a scalar are filters by default in Prom >= 0.19.0 - the surviving series pass through with their LHS values. Without the bool modifier, '(burn_rate >= 14.4) * 3' multiplies the raw burn rate (e.g. 57.3) instead of returning 3, so the status gauge emits the wrong integer and the dashboard's 0/1/2/3 -> Healthy/Burning/Critical/Breached mapping never matches. Add 'bool' to every comparison in the status block template. Alert-rule expressions continue to use filter semantics on purpose - when an alert evaluates to 'the burn-rate series survived the comparison', each surviving series is itself a violating series, so 0/1 coercion would hide violations. Add TestStatusRuleUsesBoolModifier: - 3 '>= bool' (Warning, Critical, Breached) and 2 '< bool' (Breached, Critical) checks the status block of every generated rule. - Top tier multiply-by-3, mid multiplies by-2, lowest by-1 - the integer status codes the dashboard maps to text. - Sanity guard: no bool survives into the alert block. Tested across multiline, multi-dim, and tiered fixtures so a future template refactor cannot silently regress. Regenerated all matching fixtures via TestGenerate_Golden -update. The non-ratio testdata also picks up parenthesisation from the preceding fix(generator) commit because they were not regen'd at that step; this is harmless because both fixes land before any consumer runs 'make generate' against the spec bundle. The bool modifier has been stable since Prom 0.19.0 (Oct 2015), so the output rules work on every Prometheus version in practical use.
Alert rules for one SLO previously shared an alert name across severities (`AdAvailabilityMultiWindowMultiBurnRate` regardless of severity) and relied on the `severity` label to differentiate. Inside a single rule group that works, but cross-group deduplication per Prom's name uniqueness requirement gave little headroom - any directory that consumes the same generator output for two different sources (rules CM fixtures, hub-spoke Prom federations, multi-tenant Prom with the same UID prefix) collapses two semantically distinct alerts. Append the severity PascalCase to the alert name so two alerts of the same SLO and kind can coexist with distinct names: AdAvailabilityMultiWindowMultiBurnRatePage AdAvailabilityMultiWindowMultiBurnRateTicket The `severity` label is unchanged so existing Grafana Alerting / Prometheus alertmanager routing still work; only the alert name string is altered, which is a human-readability and Prometheus uniqueness win, not a routing change. Breaking but not flagged in CHANGELOG [Unreleased] yet because prior consumers referenced alert names for KPI dashboards or runbook titles. This commit message registers it; a separate CHANGELOG entry will follow under v0.2.0. Only test-tiered-slo-rules.golden.yaml carries alert-name material (other fixtures' specs do not reference AlertPolicies), so a single golden update is enough.
The OpenSlo spec carries a free-form spec.description per SLO. Until now
opensloctl threw that away - the dashboard could show the SLO name,
target and burn rate but never the human prose, so alert runbooks and
on-call context flowed either through PromQueries manually or through a
stale notebook.
Add an openslo_slo_description label to the openslo_slo_info
recording rule, populated by a small foldSloDescription pipeline in
the generator:
1. Replace every newline with one space.
2. Collapse runs of whitespace to one space (strings.Fields + Join).
3. Trim leading/trailing whitespace.
4. Cap at 200 chars; truncate with unicode ellipsis U+2026 if longer.
5. Escape quote and backslash so the value is safe inside
YAML/PromQL quoting.
Empty input still produces empty string. Every SLO carries the label,
so downstream Grafana text panel macros of the form
${openslo_slo_description} always substitute to a stable row even when
the spec author skipped description.
Add the matching openslo.slo.description semconv attribute (stability
development) and a recommended reference on the metric emission
side. Regenerate pkg/semconv/semconv_gen.go so downstream code that
imports the constants compiles against the new attribute.
Regenerated testdata via TestGenerate_Golden -update. All six ratio
SLI and threshold SLI goldens pick up the new label line; alert-name
suffix and status bool changes from preceding commits are already
landed in these goldens so the diff is a single line per file.
BREAKING (registration only): series for openslo_slo_info gain a new
label. Dashboards matching on exact label sets will see the new
series in addition to the old. Recording-rule label cardinality of
the metadata series grows by one per SLO.
The decoder loop in loadSpecs dropped every YAML file that failed
decode with no log and no error, so a stray top-level field on an
OpenSlo spec, an outdated SDK field, or a non-OpenSlo YAML file
silently vanished from the generator run. Consumers got a rules set
that's strictly smaller than their spec bundle makes.
Two changes:
1. loadSpecs returns the loaded objects, the skipped filenames, and a
non-nil error if any file failed. Each skipped file logs a
structured slog.Warn with the filename and the decoder error so an
operator can grep the run output to find the offending spec.
2. GetSpecs propagates that error wrapped as "error reading specs" so
`make generate` and `make verify` exit non-zero. CI pipelines that
have been quietly tolerating stray YAML now surface the issue at
the same place the rest of specstore validation lives.
Move the test fixtures that intentionally fail to decode from
testdata/{invalid,non-openslo}.yaml into testdata/invalid/* so the
recursive walker doesn't pick them up by accident. The two cases that
walk testdata recursively now pass an explicit whitelist of known-good
files. Add an explicit table row "recursive loading surfaces invalid
fixtures as error" that includes one known-good file and the invalid
folder - the error wins, the loader doesn't quietly drop them.
BREAKING (CI integration only): previously-silent stray YAML files
that did not decode as OpenSlo now make opensloctl exit non-zero. CI
pipelines that relied on the silent drop will start failing - this is
the intended outcome, but flag it in release notes so integrators can
clean up stray fixtures deliberately.
The repo README and AGENTS.md style guide prefers ASCII hyphens in prose. Earlier mass-edit sweeps caught most files; the few leftover em dashes in the generator comments (function doc blocks in prometheus.go and one comment block in templates.go) were bundled into feature commits and would have polluted those diffs. Pulling them into a single style commit here keeps the prior generator features single-purpose and easier to audit.
Replace the previous broad-12 SLO inventory with an 11 SLO bundle focused
on user-perceivable signals. Each SLO pairs with one rule file under
examples/oteldemo/rules/, regenerated with the multi-window-multi-burn-rate
alert strategy across two paired tiers per severity (page/ticket x fast/slow).
Inventory changes vs the prior 12:
ad-availability.yaml - kept, narrow objectives to target 0.99
ad-latency.yaml - new, ratio SLI on classic bucket
le="2000" (frontend SERVER GET /api/data)
cart-availability.yaml - kept
frontend-availability.yaml - kept, simplified
image-loading-latency.yaml - new, source service_name="image-provider"
order-processing-latency.yaml - new, ratio SLI on le="60000"
post-checkout fan-out burst
payment-unreachable.yaml - renamed + narrowed from the older
payment-availability shape; trips
when paymentUnreachable flag throws
post-order-email-availability.yaml - new, internal HTTP POST
send_order_confirmation trips on
emailMemoryLeak
post-order-email-latency.yaml - new, le="30000" envelope for the same
product-catalog-availability.yaml - kept
recommendation-availability.yaml - kept
(alert-conditions.yaml, alert-policies.yaml,
notification-target-engineers.yaml,
services.yaml) - helper files shared across the 11 SLOs
The 8 reusable AlertConditions (page-fast-5m/1h at 14.4x, page-slow-30m/6h
at 6x; ticket-fast-2h/1d at 3x, ticket-slow-6h/3d at 1x) apply to all 11
SLOs without per-SLO duplication.
Each rule file is a recording rule group + alert group for that SLO
plus its referenced policies. Verified by cd /Users/ibrahimd/repos/github.com/thisisibrahimd/opensloctl.git/.worktrees/demo-slos && go run . load -f examples/oteldemo/specs/ -r (promtool check rules: 30 rules per file, SUCCESS across
all 11).
Note on rule files in this commit: they are generated output, not
sources, but they are checked into the repo so first-time users can
clone and immediately see a working ruleset without running
go run . generate -f -o first. The examples/oteldemo/.gitignore deliberately
excludes costs of re-generation but keeps a clean baseline here.
Replace the docker-compose based deployment harness with a kind cluster
running the upstream OpenTelemetry Demo via Helm. The compose setup
struggled with bind mounts when same target dirs (Grafana provisioning
dashboards, Prometheus rule_files) needed both upstream and custom
content; Kubernetes ConfigMaps sidestep both layering conflicts.
Layout under examples/oteldemo/kind/:
- kind-config.yaml: minimal cluster spec - 1 control-plane node,
port-forwards enabled so kubectl handles the routing surface.
- setup.sh: creates cluster if missing, runs helm install/upgrade with
the values overlay, applies our ConfigMaps (rules + dashboards),
and port-forwards Grafana/Prometheus/Frontend on 8080/3000/9090.
- sync.sh: replaces the openslo-rules and openslo-dashboards ConfigMaps
(delete + create) after every make generate, then HUPs Prometheus -
avoids the 256 KiB kubectl.kubernetes.io/last-applied-configuration
annotation ceiling that the apply-only flow hit when the ruleset grew.
- teardown.sh: deletes the kind cluster and the deployed Helm release.
Overlay values.yaml customises the chart for our SLO telemetry:
- otel-collector: single span_metrics connector on namespace
traces.span.metrics (sub-named connector ids were rejected by the
chart), extended explicit histogram bucket list to include 30s and 60s
so the ad-latency (le=2000), post-order-email-latency (le=30000) and
order-processing-latency (le=60000) SLOs find valid boundaries.
- Grafana: anonymous admin enabled, sidecar dashboards watching the
gfd-sc-dashboards ConfigMap.
- Prometheus: sidecar configmap-reload on the prometheus ConfigMap so
rule_files picks up new entries without a Prometheus restart.
examples/oteldemo/Makefile:
- verify - open the spec bundle without writing rules.
- generate - regenerate rules + promtool lint.
- lint-rules - promtool check rules on every rules file.
- start-demo / stop-demo / clean / sync - thin wrappers around the
kind scripts.
Removes the prior docker-compose harness:
- examples/oteldemo/deploy/compose.slos.yaml (docker-compose snippet,
no consumer)
- examples/oteldemo/openslo-dashboards.yaml (legacy direct-mount
dashboard format that the kind sidecar replaces)
The root Makefile's start-demo target previously drove the compose
harness; the kind setup.sh now owns the lifecycle and the per-example
Makefile is the single entry-point for new users.
The repo-root dashboards and integrity rules now have a real source of
truth under deploy/mixins, grafonnet v13 jsonnet. The build pipeline:
deploy/mixins/*.jsonnet -- jsonnet + sprig --> deploy/mixins/dashboards/*.json
\ |
-- sync-legacy (Makefile)--> deploy/dashboards/*.json
|
-- (kind/sync.sh) --> openslo-dashboards CM
deploy/mixins/dashboards
- openslo-list.json rendered from openslo-list.jsonnet: single table
panel with one row per SLO. Columns: Objective %, Period SLI, Status
(0..3), Budget Left %. Status mappings are array-form with a value
per entry so Grafana treats them as value mappings; cellOptions
applies a tinted-background colour via thresholds, not via mappings.
Budget bands: red<15, yellow 15..49, green >=50.
- openslo-detail.json rendered from openslo-detail.jsonnet: a text
header (name/description/target) plus three timeseries columns
(SLI 28d, Error Budget Burndown, Burn Rate) and three stat columns
(SLO target, 28d SLI, 28d Remaining %, Current Burn Rate).
Backing series: openslo_slo_info, openslo_slo_objective, status, and
the windowed sli_error_rate_* recording rules emitted by the
generator.
Variables: datasource (Prometheus picker, pluginId locked lowercase),
slo (label_values over openslo_slo_info). Two hidden helpers -
description and target - feed the markdown header text panel without
showing up in the picker.
deploy/mixins/rules/openslo-integrity.jsonnet --> deploy/rules/
openslo-integrity-rules.yaml via jsonnet -S
- openslo_slo_metric_missing: count by openslo_slo_name of (info
exists but sli_error_rate_5m has no sample for the last 10m).
Caught drift scenarios like a typo in spec indicator, a refused
target Prometheus access, or a generator run that didn't include a
particular SLO.
- OpenSloSpecDrift (page severity, 10m confirm window) surfaces
culprit SLOs into Alerting > Alert rules > Data source-managed so
an operator catches spec drift before a customer does.
deploy/mixins/Makefile
- generate / list / detail / rules / lint-rules / release:
jsonnet -> dashboard JSON, jsonnet -S -> rules YAML, promtool
check rules on every output. version extVar pins a git-SHA + dirty
marker in the rendered dashboard title.
deploy/dashboards/* is checked in but should be treated as generated
output - any edit must be in deploy/mixins/* first, then
re-applies. The legacy sync to deploy/dashboards/ is weaver registry generate --v2 --registry semconv/registry --templates ./semconv/templates/ go ./pkg/semconv/
gofmt -w ./pkg/semconv/semconv_gen.go (target name from the existing Makefile).
Carry the v0.2.0 contract end-to-end so users hitting the README
without prior context can ship a working dashboard within an hour:
- Root README: rewrite around nine sections with a TOC (Quick start,
What gets generated, Writing specs, Alerting strategies, Multi-dim
SLIs, Dashboards and integrity rules, Examples, Semantic
conventions, Development). The spec tour lands before any
alerting deep-dive, following the path a first-time reader takes.
- Names and references for every metric and every attachment point:
openslo_slo_info/.object/.timewindow/.error_budget/.current_burn_rate,
openslo_sli_error_rate_<window>, openslo_sli_event_rate_<window>,
openslo_slo_period_burn_rate, openslo_slo_period_error_budget_remaining,
openslo_slo_status, openslo_slo_description. Plus the dashboards
that consume them and the integrity rules that catch spec drift.
- Alert naming convention updated to reflect the severity-suffixed
alert name introduced in this version: AdAvailability...Page and
AdAvailability...Ticket rather than the shared name. The severity
label is unchanged so existing Promql alertname/severity JavaScript
template routes still work; only the alert name string grew.
- Examples table includes multi-dim-slo with its dedicated README
and properly attributes the harness (kind + Helm) to examples/oteldemo.
- Semantic Conventions section notes the three deprecated attributes
with planned major-removal paths.
- Development section mentions mise pinning Go 1.26 with go.mod
min 1.25.5, and the make snapshot-update path via go test -update.
Per-example READMEs (api-latency, error-budget, error-rate, multi-burn,
multi-dim, oteldemo) keep the previous example-by-example shape but
drop stale URLs (no more C2%A7 escape sequences from pre-existing
markdown; replaced with hyphened section slugs) and route readers back
to the root README for shared concepts.
Fold the prior [Unreleased] and [v0.1.8-dev] sections into a single
[v0.2.0] release. The branch is 28 commits and 13 working-tree commits
ahead of v0.1.0; everything user-visible since v0.1.0 lives in this
release.
Top-level fingerprint:
### Breaking changes (4)
- loadSpecs fail-loudly on decode errors
- openslo_slo_description label added to openslo_slo_info sidecars
- alert names now suffixed with severity (Page / Ticket)
- oteldemo docker-compose harness replaced by kind + Helm
### Added (10)
Four-kind alert generation, multi-window-multi-burn-rate reusable
condition set, status gauge, multi-dim SLI annotations, grafana
dashboards with mixins source of truth, spec-drift integrity
rule + alert, oteldemo kind+Helm harness, semconv
openslo.slo.description attribute, TestStatusRuleUsesBoolModifier
regression test, multi-dim-slo example.
### Changed (6)
openslo_slo_description label emission, alert naming convention,
oteldemo spec inventory 12 -> 11, Path Y migration for 6 SLOs,
chaos_flag drift documented in AGENTS, per-SLO target locking.
### Fixed (3)
Ratio precedence bug, status gauge bool modifier for Prom 3.x,
specstore fail-loud loadSpecs.
### Removed (3)
oteldemo/deploy/ compose harness, oteldemo/openslo-dashboards.yaml,
top-level Makefile start-demo target.
### Deprecated (1)
Three semconv attributes (decimal, percent, timewindow.duration)
slated for major-release removal.
No [Unreleased] section per request - the next commit sets the next
version line directly.
[Unreleased]: v0.2.0...HEAD
[v0.2.0]: v0.1.0...v0.2.0
Hunt aggregate missed two more files when compared to the main lineage: feature.go's MULTI_DIMENSIONAL annotation constants and labels.go's ParseSpecLabels comment. Replace em dashes with ASCII hyphens to keep the prose consistent with the rest of the repo.
Round out examples/multi-dim-slo/ with the spec bundle backing the
README. Each file plays a role described in the file layout table:
service.yaml - api-gateway Service definition
sli.yaml - api-gateway-latency-sli with thresholdMetric over
http_request_duration_seconds_bucket
slo.yaml - api-latency SLO with multi-dimensional-sli.openslo.com
/label=service_name and /dimensions annotations; pulls
SLO target via indicatorRef
alert-condition-page.yaml - burn-rate 14.4x over 5m
alert-condition-ticket.yaml - burn-rate 3x over 2h
alert-policy-{page,ticket}.yaml - AlertPolicy wrappers with one
condition each (SDK enforces exactly one)
notification-target-{pagerduty,slack}.yaml - NotificationTarget
registry so the AlertPolicy can route fires
Add deploy/mixins/alerts/.keep as a placeholder directory; future
grafonnet alert definitions will live here so the dashboards folder
has a peer for alert-manager provisioning. The empty directory needs
.gitkeep style scaffolding to survive an empty-commit cleanup; the
.local .keep file carries a single comment rather than the typical
.gitkeep name to flag it's a scaffold.
Audit findings (GitHub Actions hardening):
HIGH - goreleaser/goreleaser-action@v5 was a mutable tag. Compromised
upstream rewrites v5; OIDC + a token-bearing runner. Pin to v7.2.3
commit SHA f06c13b6 with version comment.
LOW - actions/checkout@v4 was mutable. Pin to v7.0.1 commit SHA
3d3c42e5.
LOW - actions/setup-go@v5 was mutable. Pin to v7.0.0 commit SHA
b7ad1dad.
LOW - go-version was 'stable' (unpredictable). Pin to '1.26.x' so
CI matches the release toolchain in mise.toml.
MEDIUM - permissions was a single top-level block granting
contents: write to every job. Split per job to deny-by-default:
snapshot (PR): id-token: write + contents: read.
release (tag): id-token: write + contents: write.
The snapshot job on pull_request never needs to write tags and
cannot reach secrets in fork PRs, so contents: read is the right
scope. contents: write stays on the release job only.
Default-deny at workflow level: a top-level `permissions: {}` is
the safe root of trust - every job walks back up to that root
before declaring its own needs.
Rename inner jobs: goreleaser -> snapshot / release so the workflow
file mirrors the goreleaser lifecycle phases rather than wrapping
one outer 'goreleaser' job that branches by event_name.
Add cache: true to setup-go so Go module downloads warm across runs.
Add fetch-tags: true to the release job's checkout so goreleaser
picks up the just-pushed vX.Y.Z tag without an extra git fetch.
Trigger review: pull_request (same repo / no secrets) and
release/published (privileged, gated by repo maintainers who can
publish a release). Neither path has script injection sinks - every
run: block uses static args. The GITHUB_TOKEN used by the goreleaser
action is the per-run runner token, not a long-lived PAT.
All three SHA pins annotated with their upstream tag in a trailing
comment so Dependabot can scan and update them in batch via
github-actions ecosystem.
SHA source: GitHub REST `GET /repos/{owner}/{repo}/tags` at
this commit, picking the first matching semver. Each pinned SHA is
the commit object that the tag points at, not an arbitrary branch
tip, so the supply chain locks rather than floats.
thisisibrahimd
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Releases the v0.2.0 milestone: extends the Prometheus generator with four
alerting kinds and a status gauge, ships a grafonnet-based dashboard
provider plus spec-integrity recording rules, adds a kind-cluster harness
around the OpenTelemetry Astronomy Shop demo, and replaces the
docker-composeharness in
examples/oteldemo/. Documentation, CHANGELOG, and AGENTS.mdare updated to match. CI workflows are SHA-pinned for supply-chain
hardening.
44 commits ahead of
main, working tree clean.Breaking changes
(
pkg/specstore/specstore.go). Non-OpenSLO YAML is no longer silentlydropped — invalid specs surface during
load/validate/generate.openslo_slo_metadataratio series when aratio SLI is present. Ratio SLIs additionally emit
openslo_sli_event_rate_<window>recording series counted from themetric source marked
isGood.openslo_slo_metadata_*(foldedinto the recording rules under
openslo_slo_*); dashboards and queriesthat referenced these labels must move to
openslo_slo_name/openslo_slo_description.severitylabel is now suffixed (critical,warning,info) — thebare label is removed.
Generator
burn-rate,tiered,time-window,standard.good/bad) keyed byopenslo_slo_name, plus the SLO description viaopenslo_slo_description(semconvopenslo.slo.description).per-metric defaults; threshold cases rule out unreliable ratio math.
pkg/semconv/semconv_gen.goregenerated; newopenslo.alert.*attributeswired into the templates.
Dashboards
deploy/mixins/driving a Grafana provisioningprovider (
openslo-dashboards.yaml) — JSON leaves the repo (deterministicrebuild via
make dashboards).openslo-list+openslo-detaildashboards published; filterable by SLOname, description, slo-criticality.
deploy/rules/openslo-integrity-rules.yamlships anOpenSloSpecDriftrecording rule + alert that detects when thedeployed OpenSLO spec set drifts from the Prometheus recording rules
in
openslo_recording_rules_hash.oteldemo kind harness
examples/oteldemo/kind/setup.sh|teardown.sh|sync.shvalues.yamlinstall the upstream helm chart in a single-node kindcluster.
examples/oteldemo/specs/(ad / cart / checkout / email /frontend / image / kafka / payment / product-catalog / recommendation /
etc.) covering availability, latency, and ratio indicators.
examples/oteldemo/rules/(one YAML per SLO, 30 ruleseach); validated via
promtool check rules.make start-demo|stop-demo|sync|verify|lint-rules|clean|generatetargets documented in
examples/oteldemo/Makefile+ README.Examples
examples/multi-dim-slo/ships a sample multi-dimensional SLO bundledriven by
multi-dimensional-sli.openslo.com/dimensions/multi-dimensional-sli.openslo.com/labelannotations (feature-flagged).Docs
README.mdrewritten with a 9-section TOC; covers install, CLI surface,generator, dashboards, and the oteldemo harness end-to-end.
AGENTS.mdupdated with the new example layout, semconv-codegenworkflow, SDK API gotchas, and the SLI source conventions (Path Y / Y-fauna).
CHANGELOG.mdv0.2.0 with full Added / Changed / Fixed / Removed /Deprecated / Dashboards subsections.
CI / hardening
.github/workflows/build-container.yamlSHA-pinned to upstream actions(
actions/checkout,actions/setup-go,docker/setup-qemu-action,docker/setup-buildx-action,docker/login-action,docker/build-push-action); immutable digest references, no mutable tags.Verification
make buildclean;make lint,make test,make tidypass locally.internal/generator/prometheusgenerator/testdata/regenerated alongsidetemplate changes; single-line, multi-line, ratio, multi-dim, tiered
cases all green.
go test ./internal/generator/prometheusgenerator/....make verify+make generate+make lint-rulesgreen inexamples/oteldemo.Post-merge
v0.2.0oncemaincarries the merge commit (git tag v0.2.0+git push --tags); GoReleaser publishes withprerelease: autoperrepo config.
deploy/dashboards/openslo-dashboards.yamlis a follow-up — out of scope for the PR.