diff --git a/CLAUDE.md b/CLAUDE.md index 8005b8b1..62679e4f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,7 @@ Migrated from the dobby memory repo (`encryption4all/dobby`). This file is the h - The oasdiff gate's settings are **not** self-evident and `--fail-on ERR` alone fails open. `fail-on: WARN` is deliberate: oasdiff rates removing or renaming an *optional* response property, and removing a request parameter, as WARN, and this spec marks only `status` as `required`, so at ERR the gate silently passed a removed `key` (the IBE user secret key `/v2/request/key` exists to return), a renamed `proofStatus` and a dropped `timestamp` parameter. Two more, a changed non-success status (401 to 403) and a dropped response enum value, rate ERR but are **opt-in**, so they only run when named in `include-checks`. Of the 31 WARN checks, `response-property-enum-value-added` is the only one that fires on something `COMPATIBILITY.md` does not already forbid, which is why that document now names a new response enum value as non-additive too. Reproduce a verdict with the exact flags the action's entrypoint builds (`oasdiff/oasdiff-action/breaking@v0.1.10` is `FROM tufin/oasdiff:v1.26.1`, so the pinned tag is what makes a local run authoritative): `git show origin/main:pg-pkg/api-description.yaml > /tmp/base.yaml && oasdiff breaking /tmp/base.yaml pg-pkg/api-description.yaml --allow-external-refs=false --composed=false --fail-on WARN --include-checks response-non-success-status-removed,response-property-enum-value-removed`. Two traps: `oasdiff --version` prints `oasdiff version main` after a `go install` of a tag because the version comes from release ldflags (the code is still the tag), and `--fail-on` takes `ERR`/`WARN` while `oasdiff checks --severity` takes `error`/`warn`/`info`. `--severity ERR` is a usage error, so `oasdiff checks --severity ERR | wc -l` counts the help text instead and badly undercounts the tier (it is 213 error checks, 31 warn, 265 info). The spec has no external `$ref`s, so `allow-external-refs` stays at its safe (SSRF-guarding) default. The gate only sees paths the spec documents, and the spec documents canonical paths only, so dropping the `/v2/irma/...` alias handlers (#257) passes it. - **Importing another repo's history here silently closes issues here. Two vectors fire it, each sufficient on its own, so rule out neither.** Imported commits carry their original messages verbatim, closing keywords included, and GitHub resolves those against the **destination** repo's numbering. **Vector 1, the squash body.** This repo's `squash_merge_commit_message` is `COMMIT_MESSAGES` (the same setting the release-plz bullet above turns on its head — there it is why a `BREAKING CHANGE:` footer in a PR body never reaches the commit), so squash-merging the import PR concatenates every imported commit message into the merge commit's body. `ba380a1`'s body is 1678 lines and carries all 17 refs (`Closes #38 #45 #52 #54 #123 #125 #134 #142 #146 #153 #155 #157 #159 #167 #186 #191 #194`); 16 pointed at numbers already closed here, and one — #146, a live unimplemented feature request — was closed four seconds after it landed, with nothing warning, and stood four days before being found and reopened. Note what this means: a plain squash merge is enough by itself, *because* squashing concatenates the messages even as it throws the history away. **Vector 2, the history itself.** Here GitHub never reads the merge commit's message — it attributes the closes to the **imported commits**, once they become reachable from the default branch. This vector went untested in this repo rather than disproven: `9887e1a` carries no keywords of its own, and by the time it landed thirteen minutes later all 17 targets were already closed, and GitHub does not re-close a closed issue. It is measured in encryption4all/postguard-js#139, which is also the guard: import PR #137 merged with a real two-parent merge commit (`b1bb2ee`, a three-line message, no keywords, no squash body anywhere), and js#128/#129 closed six seconds later attributed to imported commits `a0ce27f`/`2acf42f` — single-parent, committed 2026-06-04 — then reopened fifteen minutes later. **So the audit is the only step that covers both.** Before merging *and* after, from the import branch: `git log origin/main..HEAD --pretty=%B | grep -oiE '(close[sd]?|fix(e[sd])?|resolve[sd]?) +#[0-9]+' | sort -u`, then check each number here. Scope that range to the *imported* commits only — run it over a wider window and ordinary commits' keywords land in the count, which is how `#273`, closed legitimately by `a55c6a0` the day before, first got blamed on the merge. Editing the squash body in the merge dialog defuses vector 1 and only vector 1; on vector 2 there is no body to edit, and rewriting the keywords out of the imported commits costs every imported SHA (`git filter-repo`). Do **not** reach for `commit_message` on `PUT /pulls/N/merge` as the scripted version of that: the REST reference words it "Extra detail to append to automatic commit message" and documents nothing about `merge_method=squash`, so whether it replaces the concatenated body or appends to it is unverified — and if it appends, the defusal silently no-ops and every keyword still fires. Confirm it on a throwaway repo and record the answer here before relying on it. Reading close *state* cannot tell you which happened, so read the close **event** — and read it correctly, because the obvious rule is wrong. A keyword in a *commit message* is attributed to the commit and carries a `commit_id`; a keyword in the *PR body* is attributed to the PR and carries `commit_id: null`. #146 shows `ba380a1464…` because PR #277's body said only `Closes #255`, while #273 shows `null` because PR #274's body said `Closes #273`. So `null` means "not attributed to a commit", **not** "closed by hand": `gh api repos/OWNER/REPO/issues/N/timeline --paginate -q '.[] | select(.event=="closed") | .created_at+" "+(.commit_id//"-")'`. Same failure class as the unapplied-workflow half below (#272). - The `dobby-coder` GitHub App lacks `workflows: write` on this repo; any push touching `.github/workflows/*.yml` is rejected at the remote. Before treating a fix as blocked, check whether the same effect can be achieved in a pushable file (crate manifest, source, committed script); if a fix genuinely can only live in a workflow file, ship the pushable half and hand the maintainer ready-to-paste YAML in the PR body. The block covers *merge* commits too, which is easy to miss: once a branch carries its own `build.yml` change (typically a maintainer applying such a patch onto it), a later `git merge origin/main` that has to touch `build.yml` produces a commit updating a workflow file, and the push is rejected even when the resolution is only "keep both new jobs". Nothing can be split out of a merge commit, so that merge has to be landed by a maintainer, or the App needs `workflows: write`. Measured exception, worth trying before handing the sync over: the App pushed `ce0fc59` on this branch, a merge whose diff against its first parent added main's 64 new `build.yml` lines. That merge needed no resolution inside `build.yml` — it took main's side whole, so the blob it committed already existed in the repo. Try the merge and read the remote's answer; only escalate on an actual rejection. +- `gh pr edit` does not work on this repo and **fails in the direction that looks like success**: a classic project is attached, so the mutation `gh` sends requests `repository.pullRequest.projectCards` and the whole call dies with `GraphQL: Projects (classic) is being deprecated ... (repository.pullRequest.projectCards)`. That message reads as a deprecation warning, `gh` still exits 0, and the edit silently does not land — so a `--body-file` update looks applied and isn't. Use REST instead, which touches no project fields: `gh api -X PATCH repos/encryption4all/postguard/pulls/N -f body="$(cat body.md)"`. Verify by reading the body back (`gh pr view N --json body`), because the failure is invisible otherwise. Same trap for `--title`/`--add-label` through `gh pr edit`. - **Consolidating a repo into a monorepo means transfer its open issues first, then archive — not archive-then-orphan.** `postguard-website`, `postguard-outlook-addon`, `postguard-tb-addon` and `postguard-examples` were archived (read-only) after folding into `postguard-js`, and the "open issues transfer here" step was silently skipped on all four: 62 open issues sat stranded, unworkable (`HTTP 403: Repository was archived so is read-only` on label/assign/comment/close). GitHub will not transfer an issue out of an already-archived repo, so recovering from the skip costs an unarchive → transfer → re-archive round trip per repo instead of a single transfer before archiving (decided in postguard#282). Do the transfer as part of the same change that archives the repo, and leave a "development moved to ``, see ``" banner at the top of the archived repo's README before re-archiving, so an old link still finds the new home. ## Dependencies diff --git a/cryptify/CLAUDE.md b/cryptify/CLAUDE.md index 073a7878..dca983ac 100644 --- a/cryptify/CLAUDE.md +++ b/cryptify/CLAUDE.md @@ -203,8 +203,9 @@ works fine. The regex is anchored (`^...$`), so there's no subdomain/wildcard bypass. ## Metrics -- `GET /metrics`: Prometheus text format, unauthenticated by design. Lock down at - the firewall, not the endpoint. +- `GET /metrics`: Prometheus text format. Gated by a Bearer token when + `metrics_token` is set; with the key unset the endpoint is open and startup + logs a warning. Lock it down at the firewall either way. - Channel label derived in priority: `X-Cryptify-Source`, then `Authorization: Bearer` / `X-Api-Key` (-> `api`), then `Origin` (-> `website` / `staging-website`), then `User-Agent` (-> `outlook` / `thunderbird`), then @@ -213,6 +214,18 @@ bypass. `metrics_scan_interval_secs`). - `FileState.source_channel` is populated at `upload_init` from request headers; populate it in any new test fixtures too. +- **No metric here knows which deployment it runs in.** There is no `env` label + in the exporter and adding one would be wrong: staging and Procolix production + are separate scrape targets, so `env` belongs in the Prometheus job's static + `labels:`. `docs/grafana/README.md` has the scrape config. +- The reference dashboard is `docs/grafana/cryptify-usage.json`, pinned to the + exporter by `mod dashboard_tests` in `src/metrics.rs`. It reads the `# TYPE` + lines out of a real `Metrics::render()` and compares them against the metric + names in the committed panel queries, both directions, so renaming a metric + fails `cargo test` rather than silently emptying a graph. It also requires + every panel query to carry `env=~"$env"`. Adding a metric therefore means + adding a panel in the same PR. That coupling is deliberate; don't loosen the + test to avoid it. ## Integration test harness - `build_rocket(figment, vk)` is the injection point. `#[launch] rocket()` wraps it diff --git a/cryptify/docs/grafana/README.md b/cryptify/docs/grafana/README.md new file mode 100644 index 00000000..00601fe8 --- /dev/null +++ b/cryptify/docs/grafana/README.md @@ -0,0 +1,85 @@ +# Grafana dashboard for cryptify usage + +`cryptify-usage.json` is the reference dashboard behind +[postguard#305](https://github.com/encryption4all/postguard/issues/305): messages +sent per channel, and cryptify storage in use per environment. + +It reads only metrics that `GET /metrics` already exports (see +`cryptify/src/metrics.rs`). No exporter change is needed to import it. + +## Metrics it uses + +| Metric | Type | Labels | Panel | +| --- | --- | --- | --- | +| `cryptify_uploads_total` | counter | `channel` | Messages sent per channel | +| `cryptify_upload_bytes_total` | counter | `channel` | Bytes uploaded per channel | +| `cryptify_uploads_by_app_total` | counter | `app` | Messages per client app | +| `cryptify_storage_bytes` | gauge | none | Storage in use | +| `cryptify_active_files` | gauge | none | Files on disk | +| `cryptify_expired_files_total` | counter | none | Uploads expired before finalize | + +`channel` is `website`, `staging-website`, `outlook`, `thunderbird`, `api` or +`unknown`; `app` is `pg-js`, `pg-dotnet`, `pg4ol`, `pg4tb` or `unknown`. Both +label sets are seeded at 0 on startup, so a channel with no traffic still shows +as a zero line instead of vanishing from the legend. + +The counters are per process. A cryptify restart resets them to 0, which is why +every panel goes through `increase()` rather than reading the raw counter. + +## The `env` label is supplied by Prometheus, not by cryptify + +Nothing in the exporter knows which deployment it is running in. Splitting +staging from Procolix production is the scrape config's job: attach a static +`env` label per job, and the dashboard's `Environment` variable picks it up. + +```yaml +scrape_configs: + - job_name: cryptify + metrics_path: /metrics + scheme: https + authorization: + type: Bearer + credentials_file: /etc/prometheus/cryptify-metrics-token + static_configs: + - targets: ["cryptify.staging.postguard.eu"] + labels: + env: staging + - targets: ["cryptify.postguard.eu"] + labels: + env: production +``` + +Set the same token as `metrics_token` in each deployment's `conf/config.toml` +(or `ROCKET_METRICS_TOKEN` in its environment). With no token configured the +endpoint answers unauthenticated and logs a warning at startup, so keep it +restricted to the Prometheus segment at the firewall as well. + +Adjust the target hostnames to whatever the two deployments actually resolve to. +The dashboard does not care about the hostnames, only that `env` is present. + +The two gauges carry no labels of their own, so a raw select distinguishes series +only by `instance` and `job`. The storage and file-count panels therefore go +through `max by (env)`: with one target per environment it changes nothing, and +if an environment ever gets a second target sharing the volume it reports the +volume once instead of twice. Switch those three panels to `sum by (env)` if the +targets get separate volumes. + +## Importing + +Grafana, Dashboards, New, Import, upload `cryptify-usage.json`, pick the +Prometheus data source. The datasource is a dashboard variable rather than a +baked-in UID, so the same file imports into any Grafana instance. + +Storage panels are sampled from `data_dir` on a background task every +`metrics_scan_interval_secs` (60 by default), so they trail a burst of uploads +by up to one interval. The dashboard refreshes every 5 minutes and opens on a +30-day window, which suits monthly usage reporting; shorten both if you are +watching a deploy. + +## Keeping it honest + +`mod dashboard_tests` in `cryptify/src/metrics.rs` reads this JSON and checks it +against `Metrics::render()`: every metric the exporter emits appears on the +dashboard, every `cryptify_*` name the dashboard queries is one the exporter +actually emits, and every panel filters on `env=~"$env"`. Renaming a metric or +adding a panel that ignores the environment filter fails `cargo test`. diff --git a/cryptify/docs/grafana/cryptify-usage.json b/cryptify/docs/grafana/cryptify-usage.json new file mode 100644 index 00000000..4c010ae2 --- /dev/null +++ b/cryptify/docs/grafana/cryptify-usage.json @@ -0,0 +1,299 @@ +{ + "title": "PostGuard usage (cryptify)", + "uid": "postguard-cryptify-usage", + "description": "Messages sent per channel and cryptify storage in use, per environment. Scrapes the cryptify /metrics endpoint; see docs/grafana/README.md for the Prometheus job.", + "tags": ["postguard", "cryptify", "usage"], + "timezone": "browser", + "editable": true, + "schemaVersion": 39, + "version": 1, + "refresh": "5m", + "time": { + "from": "now-30d", + "to": "now" + }, + "templating": { + "list": [ + { + "name": "datasource", + "label": "Data source", + "type": "datasource", + "query": "prometheus", + "current": {}, + "hide": 0 + }, + { + "name": "env", + "label": "Environment", + "type": "query", + "datasource": { + "type": "prometheus", + "uid": "${datasource}" + }, + "query": "label_values(cryptify_storage_bytes, env)", + "refresh": 1, + "includeAll": true, + "multi": true, + "current": {}, + "hide": 0 + } + ] + }, + "panels": [ + { + "id": 1, + "type": "row", + "title": "Messages sent", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, + "panels": [] + }, + { + "id": 2, + "type": "stat", + "title": "Messages sent (selected range)", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 6, "x": 0, "y": 1 }, + "fieldConfig": { + "defaults": { "unit": "short", "decimals": 0 }, + "overrides": [] + }, + "options": { + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "colorMode": "none", + "graphMode": "none", + "textMode": "value" + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum(increase(cryptify_uploads_total{env=~\"$env\"}[$__range]))", + "instant": true, + "legendFormat": "messages" + } + ] + }, + { + "id": 3, + "type": "timeseries", + "title": "Messages sent per channel", + "description": "Channel comes from X-Cryptify-Source, then API auth, then Origin, then User-Agent. A channel with no traffic reports 0 rather than dropping out.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 18, "x": 6, "y": 1 }, + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 10, "showPoints": "auto" } + }, + "overrides": [] + }, + "options": { + "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (channel) (increase(cryptify_uploads_total{env=~\"$env\"}[$__rate_interval]))", + "legendFormat": "{{channel}}" + } + ] + }, + { + "id": 4, + "type": "bargauge", + "title": "Messages per channel (selected range)", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 }, + "fieldConfig": { + "defaults": { "unit": "short", "decimals": 0 }, + "overrides": [] + }, + "options": { + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "displayMode": "basic", + "orientation": "horizontal", + "showUnfilled": true + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (channel) (increase(cryptify_uploads_total{env=~\"$env\"}[$__range]))", + "instant": true, + "legendFormat": "{{channel}}" + } + ] + }, + { + "id": 5, + "type": "timeseries", + "title": "Bytes uploaded per channel", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 }, + "fieldConfig": { + "defaults": { + "unit": "bytes", + "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 10, "showPoints": "auto" } + }, + "overrides": [] + }, + "options": { + "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (channel) (increase(cryptify_upload_bytes_total{env=~\"$env\"}[$__rate_interval]))", + "legendFormat": "{{channel}}" + } + ] + }, + { + "id": 6, + "type": "bargauge", + "title": "Messages per client app (selected range)", + "description": "From the app field of X-POSTGUARD-CLIENT-VERSION: pg4ol is the Outlook add-in, pg4tb the Thunderbird one. Clients that do not send the header land in unknown.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 17 }, + "fieldConfig": { + "defaults": { "unit": "short", "decimals": 0 }, + "overrides": [] + }, + "options": { + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "displayMode": "basic", + "orientation": "horizontal", + "showUnfilled": true + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (app) (increase(cryptify_uploads_by_app_total{env=~\"$env\"}[$__range]))", + "instant": true, + "legendFormat": "{{app}}" + } + ] + }, + { + "id": 7, + "type": "row", + "title": "Storage", + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 25 }, + "panels": [] + }, + { + "id": 8, + "type": "stat", + "title": "Storage in use", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 6, "x": 0, "y": 26 }, + "fieldConfig": { + "defaults": { "unit": "bytes" }, + "overrides": [] + }, + "options": { + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "colorMode": "none", + "graphMode": "area", + "textMode": "value_and_name" + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "max by (env) (cryptify_storage_bytes{env=~\"$env\"})", + "instant": false, + "legendFormat": "{{env}}" + } + ] + }, + { + "id": 9, + "type": "timeseries", + "title": "Storage in use over time", + "description": "Sampled from data_dir by a background task every metrics_scan_interval_secs (60s by default), so it lags a burst of uploads by up to that interval.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 18, "x": 6, "y": 26 }, + "fieldConfig": { + "defaults": { + "unit": "bytes", + "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 10, "showPoints": "auto" } + }, + "overrides": [] + }, + "options": { + "legend": { "displayMode": "table", "placement": "right", "calcs": ["max"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "max by (env) (cryptify_storage_bytes{env=~\"$env\"})", + "legendFormat": "{{env}}" + } + ] + }, + { + "id": 10, + "type": "timeseries", + "title": "Files on disk", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 34 }, + "fieldConfig": { + "defaults": { + "unit": "short", + "decimals": 0, + "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 10, "showPoints": "auto" } + }, + "overrides": [] + }, + "options": { + "legend": { "displayMode": "table", "placement": "right", "calcs": ["max"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "max by (env) (cryptify_active_files{env=~\"$env\"})", + "legendFormat": "{{env}}" + } + ] + }, + { + "id": 11, + "type": "timeseries", + "title": "Uploads expired before finalize", + "description": "Sessions purged after the idle timeout without ever being finalized. A rise here means senders are starting uploads they do not complete.", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 34 }, + "fieldConfig": { + "defaults": { + "unit": "short", + "custom": { "drawStyle": "bars", "lineWidth": 1, "fillOpacity": 60, "showPoints": "never" } + }, + "overrides": [] + }, + "options": { + "legend": { "displayMode": "table", "placement": "right", "calcs": ["sum"] }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "refId": "A", + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (env) (increase(cryptify_expired_files_total{env=~\"$env\"}[$__rate_interval]))", + "legendFormat": "{{env}}" + } + ] + } + ] +} diff --git a/cryptify/src/metrics.rs b/cryptify/src/metrics.rs index 6d44c81d..2c320aff 100644 --- a/cryptify/src/metrics.rs +++ b/cryptify/src/metrics.rs @@ -366,6 +366,185 @@ pub async fn storage_sampler( } } +/// Pins the committed Grafana dashboard to what this module actually exports. +/// +/// The dashboard is a separate artefact from the exporter, so nothing but a +/// test stops the two drifting: renaming a metric here leaves panels querying +/// a name that no longer exists, and Grafana renders that as an empty graph +/// rather than an error. These tests fail instead. +#[cfg(test)] +mod dashboard_tests { + use super::*; + use std::collections::BTreeSet; + + use serde_json::Value; + + const DASHBOARD: &str = include_str!("../docs/grafana/cryptify-usage.json"); + + /// Label matcher every panel must carry, so that every panel is scoped to + /// the environments selected in the `env` variable. Note that this only + /// filters: panels 2 through 6 all aggregate `env` away, so with All + /// selected they report staging and production as one total. + const ENV_SELECTOR: &str = "env=~\"$env\""; + + fn dashboard() -> Value { + serde_json::from_str(DASHBOARD).expect("cryptify-usage.json must be valid JSON") + } + + /// The metric names the exporter emits, read off the `# TYPE` lines of a + /// real render rather than from a hand-kept list. + fn exported_metrics() -> BTreeSet { + Metrics::new() + .render() + .lines() + .filter_map(|l| l.strip_prefix("# TYPE ")) + .filter_map(|rest| rest.split_whitespace().next()) + .map(str::to_string) + .collect() + } + + /// Every `expr` in the dashboard. Rows nest their panels one level deeper, + /// so this walks the whole tree instead of looping over `panels`. + fn panel_exprs(v: &Value, out: &mut Vec) { + match v { + Value::Object(map) => { + if let Some(Value::String(expr)) = map.get("expr") { + out.push(expr.clone()); + } + for child in map.values() { + panel_exprs(child, out); + } + } + Value::Array(items) => { + for child in items { + panel_exprs(child, out); + } + } + _ => {} + } + } + + fn all_exprs() -> Vec { + let mut out = Vec::new(); + panel_exprs(&dashboard(), &mut out); + assert!(!out.is_empty(), "dashboard has no queries at all"); + out + } + + /// The `query` of each dashboard variable, where it is a plain string. + /// The datasource variable's query is the string `prometheus`, which + /// carries no metric name and so contributes nothing. + fn variable_queries() -> Vec { + dashboard() + .get("templating") + .and_then(|t| t.get("list")) + .and_then(Value::as_array) + .map(|list| { + list.iter() + .filter_map(|v| v.get("query").and_then(Value::as_str)) + .map(str::to_string) + .collect() + }) + .unwrap_or_default() + } + + /// Pull `cryptify_*` identifiers out of a PromQL string. + fn cryptify_idents(promql: &str) -> BTreeSet { + let bytes = promql.as_bytes(); + let mut found = BTreeSet::new(); + let mut i = 0; + while let Some(rel) = promql[i..].find("cryptify_") { + let start = i + rel; + let mut end = start; + while end < bytes.len() { + let c = bytes[end] as char; + if c.is_ascii_alphanumeric() || c == '_' { + end += 1; + } else { + break; + } + } + found.insert(promql[start..end].to_string()); + i = end; + } + found + } + + /// Metrics named by an actual panel query. The "everything exported is on + /// the dashboard" direction must use this rather than `referenced_metrics`: + /// a metric mentioned only in a template variable's `query` is on no graph, + /// so it must not count as covered. + fn panel_metrics() -> BTreeSet { + all_exprs() + .iter() + .flat_map(|q| cryptify_idents(q)) + .collect() + } + + /// Every `cryptify_*` name the dashboard mentions anywhere, panels and + /// template variables alike. Used for the other direction, where a variable + /// querying a metric the exporter dropped is just as broken. + fn referenced_metrics() -> BTreeSet { + panel_metrics() + .into_iter() + .chain(variable_queries().iter().flat_map(|q| cryptify_idents(q))) + .collect() + } + + #[test] + fn every_exported_metric_appears_on_the_dashboard() { + let on_a_panel = panel_metrics(); + for metric in exported_metrics() { + assert!( + on_a_panel.contains(&metric), + "{metric} is exported but no dashboard panel queries it — \ + add a panel to cryptify/docs/grafana/cryptify-usage.json" + ); + } + } + + #[test] + fn every_metric_the_dashboard_queries_is_exported() { + let exported = exported_metrics(); + for metric in referenced_metrics() { + assert!( + exported.contains(&metric), + "the dashboard queries {metric}, which this module does not emit — \ + the panel would render empty" + ); + } + } + + #[test] + fn every_panel_filters_on_the_environment_variable() { + for expr in all_exprs() { + assert!( + expr.contains(ENV_SELECTOR), + "query is missing the {ENV_SELECTOR} matcher, so it ignores the \ + environment the dashboard is scoped to:\n{expr}" + ); + } + } + + #[test] + fn the_environment_variable_is_declared() { + let declared = dashboard() + .get("templating") + .and_then(|t| t.get("list")) + .and_then(Value::as_array) + .map(|list| { + list.iter() + .any(|v| v.get("name").and_then(Value::as_str) == Some("env")) + }) + .unwrap_or(false); + assert!( + declared, + "panels filter on $env but no `env` variable is declared, so every \ + panel resolves to an empty selector" + ); + } +} + #[cfg(test)] mod tests { use super::*;