feat(cryptify): Grafana usage dashboard, pinned to the /metrics exporter

[dobby-coder]

What this is

The dashboard half of #305. The exporter half already landed: GET /metrics
has shipped per-channel upload counters and storage gauges since
encryption4all/cryptify#102, and came into this workspace with the cryptify
merge (#277). So this PR does not touch the exporter.

What was missing is the dashboard itself. cryptify/src/metrics.rs:9 has been
telling readers to "see docs/grafana/ for the reference dashboard JSON" while
that directory did not exist. This adds it.

What is in it

cryptify/docs/grafana/cryptify-usage.json, importable as-is:

• Messages sent per channel, as a time series and as a range total. Channel is
  website, staging-website, outlook, thunderbird, api, unknown.
• Messages per client app (pg4ol, pg4tb, pg-js, pg-dotnet), from the
  app field of X-POSTGUARD-CLIENT-VERSION. This is the more reliable read on
  the two add-ins than the User-Agent sniff.
• Bytes uploaded per channel.
• Storage in use, files on disk, and uploads that expired before finalize.
  The three panels behind these aggregate with max by (env) rather than
  selecting raw, because the two gauges carry no labels of their own.

The datasource is a dashboard variable rather than a baked-in UID, so the file
imports into any Grafana without editing.

Staging vs Procolix production

env is a Prometheus label, not an exporter one. Nothing in a cryptify process
knows which deployment it is, and adding a config key for it would put the same
fact in two places. The scrape job attaches env: staging / env: production
via static labels; cryptify/docs/grafana/README.md has the config, including
the Bearer auth against metrics_token.

Test

mod dashboard_tests in cryptify/src/metrics.rs pins the dashboard to the
exporter, since a dashboard querying a renamed metric renders as an empty graph
rather than an error. It reads the # TYPE lines out of a real
Metrics::render() and compares them with the metric names in the committed
panel queries, both directions, and requires every panel query to carry
env=~"$env".

Verified it catches drift rather than just passing, five mutations, each
reverted after:

Mutation	Result
Renamed a metric in a panel query	both name tests fail, naming the metric each way
Renamed a metric in the exporter, dashboard untouched	both name tests fail
Dropped env=~"$env" from one panel	the env test fails, printing the query
Deleted both storage panels	the exported->dashboard test fails, naming cryptify_storage_bytes
Renamed the metric in the env variable's query only	the dashboard->exporter test fails, so variables stay covered

cargo test --manifest-path cryptify/Cargo.toml --all-targets is 161 passed, 0
failed. cargo fmt --all -- --check and cargo clippy --manifest-path cryptify/Cargo.toml --all-targets -- -D warnings are clean.

Review cycle 1

One bug in the guard rail, found by the reviewer and reproduced here before
fixing: the exported->dashboard test folded the template-variable queries into
the set it checked against, and the env variable is
label_values(cryptify_storage_bytes, env). So cryptify_storage_bytes
certified itself. I deleted both storage panels on the old code and all four
tests stayed green. The metric #305 names twice was the one thing the test did
not cover. The panel-only set is now split out for that direction; the other
direction keeps the combined set, since a variable querying a dropped metric is
broken too.

Also addressed: panel 8 asked for a sparkline but ran an instant query, the
three gauge panels labelled their series {{env}} while selecting unlabelled
metrics raw, and the env-matcher comment and assertion message claimed a matcher
prevents aggregation when it only filters. All four threads have a reply and are
resolved.

What I could not verify

I have no Grafana or Prometheus in this container (no usable docker, no
network for a PromQL parser), so the JSON has not been through an actual
import and the PromQL is eyeballed, not machine-checked. I did check the file
structurally: 11 panels, unique ids, no gridPos overlap, every target has a
refId and the templated datasource. Worth a real import before merge.

Left for a human

This is deliberately Refs, not Closes. Creating the Scaleway Grafana
dashboard and adding the scrape job are ops actions no PR can perform, so #305
should stay open until those are done:

1. Add the scrape config from docs/grafana/README.md to the Scaleway
   Prometheus, with env labels for staging and Procolix production, and set
   metrics_token on both deployments.
2. Import cryptify-usage.json.

@rubenhensen: two questions I could not answer from the repo. What are the real
hostnames of the two cryptify deployments (the README uses placeholders), and
is /metrics reachable from the Scaleway Prometheus at all, or does Procolix
need a firewall rule first?

One gap worth naming: the Outlook and Thunderbird add-ins are separate repos,
and if they do not send X-POSTGUARD-CLIENT-VERSION or X-Cryptify-Source
today, their traffic lands in unknown rather than in their own row. The
dashboard shows that honestly instead of hiding it. Happy to open follow-ups on
those two repos if you want the split to be exact.

Refs #305

[dobby-coder]

VERDICT: request-changes

Rules + review — cycle 1

Rule sweep: no breaches. I selected the 12 rules that plausibly fire on a Rust + Grafana-JSON + docs diff (the Rust fmt/clippy/test gates, digest-pinned-file-needs-gitattributes, tests-required-on-fixes, the docs-claim rules, cross-repo-link-format, no-justification-paragraphs-for-simple-changes) and checked each against the diff. Two I want to record as explicitly cleared, because both look like breaches at a glance:

• The cryptify/CLAUDE.md metrics hunk rewrites "unauthenticated by design" into Bearer-gated-when-metrics_token-is-set. That is not an overclaim: the guard is already on this branch (cryptify/src/main.rs:140–200, cryptify/src/config.rs:143, plus the startup warning at main.rs:1603). The edit corrects stale docs rather than describing something this PR did not build.
• include_str! on a checked-in text file is the shape that normally needs a .gitattributes -text pin, and the repo's existing .gitattributes pins exactly two files for that reason. Not needed here: nothing is byte-exact or a multi-line anchor — the JSON is parsed as JSON, and the only substring check is a single-line contains(). CRLF-safe.

One blocking issue, handed over from review and confirmed here by reproduction, plus three non-blocking nits inline.

The blocker is worth stating plainly: the test module is the entire safety argument for committing a dashboard as a separate artefact, and it does not cover cryptify_storage_bytes — the one metric issue #305 names twice. I deleted panels 8 and 9 on this branch and all four tests stayed green. The fix is small and I have verified it both ways; the inline suggestion is rustfmt's formatting of it, not the shorter chain, because cargo fmt --check is a CI gate here.

Everything else holds up. The mutation table in the PR body is honest, the three claimed mutations do fail as described, and the "what I could not verify" section is the right call rather than a hedge — the PromQL genuinely has not been through a real Grafana.

[dobby-coder]

VERDICT: approve

Rules + review, cycle 2

Cycle 1's blocker is fixed, and I reproduced the fix rather than taking it on trust.

panel_metrics() now feeds the coverage direction while referenced_metrics() (panels plus template variables) feeds the other. The env variable's label_values(cryptify_storage_bytes, env) query is datasource plumbing that puts nothing on a graph, so it no longer lets cryptify_storage_bytes certify itself. Probe: I deleted panels 8 and 9 from cryptify-usage.json and ran the suite.

cryptify_storage_bytes is exported but no dashboard panel queries it —
add a panel to cryptify/docs/grafana/cryptify-usage.json
test result: FAILED. 3 passed; 1 failed

The same deletion left all four tests green before the fix. The guard rail now guards the metric #305 names twice. Restored with the inverse edit, tree clean.

The max by (env) change on panels 8, 9 and 10 is right, and the README paragraph explaining it earns its place: which aggregator to use depends on whether a second target shares the volume, and that is a call the operator has to make. It says so and names the alternative.

Rule sweep on the 50e36a9..53493e5 delta: no breaches. Two worth recording as explicitly cleared:

• include_str! on the new cryptify-usage.json with no .gitattributes pin. Not a finding: nothing is byte-exact or a multi-line anchor. The file is parsed as JSON, and the one substring check is a single-line contains() on env=~"$env", so a CRLF checkout is fine.
• The new root CLAUDE.md bullet on gh pr edit. It fires only in this repo, and this repo has a writable root CLAUDE.md, so that is where it belongs rather than in the shared rules index.

One nit, fixed on the branch

The ENV_SELECTOR caveat named panels 2, 4 and 6 as the ones that aggregate env away. Panels 3 and 5 do it too: both are sum by (channel) (...), and by (channel) drops env just as surely as a bare sum(...) does. Panels 2 through 6 are the whole Messages sent row, so the range says it exactly and is shorter than the list. This came from my own cycle-1 nit, which listed the same incomplete trio; the PR copied it faithfully.

Pushed as 3a88f54, comment text only.

Verification

On 3a88f54, from cryptify/:

Check	Result
cargo fmt --all -- --check	clean
cargo clippy --all-targets -- -D warnings	clean
cargo test --all-targets	161 passed, 0 failed

No blocking issues left. Flipping this out of draft.