Skip to content

Make readiness dashboard audit-honest #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dorkalev merged 1 commit into from
Aug 6, 2026
Merged

Make readiness dashboard audit-honest #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 50 additions & 4 deletions actions/shadow-ci/src/verify.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -728,11 +728,18 @@ fn summary(checks: &[Value]) -> Value {
let mut dimensions: BTreeMap<String, (usize, usize, usize)> = BTreeMap::new();
let mut evidenced = BTreeSet::new();
let mut operating = BTreeSet::new();
let mut observations = BTreeMap::from([
("pass", 0usize),
("fail", 0usize),
("unknown", 0usize),
("not_applicable", 0usize),
]);
for check in checks {
let dimension = check["dimension"].as_str().unwrap_or("unknown").to_string();
let entry = dimensions.entry(dimension.clone()).or_default();
match check["verdict"].as_str().unwrap_or("unknown") {
"pass" => {
observations.entry("pass").and_modify(|count| *count += 1);
entry.0 += 1;
if let Some(criteria) = check["criteria"].as_array() {
for criterion in criteria.iter().filter_map(Value::as_str) {
Expand All @@ -743,9 +750,19 @@ fn summary(checks: &[Value]) -> Value {
}
}
}
"fail" => entry.1 += 1,
"n/a" => {}
_ => entry.2 += 1,
"fail" => {
observations.entry("fail").and_modify(|count| *count += 1);
entry.1 += 1;
}
"n/a" => {
observations
.entry("not_applicable")
.and_modify(|count| *count += 1);
}
_ => {
observations.entry("unknown").and_modify(|count| *count += 1);
entry.2 += 1;
}
}
}
let dimensions: BTreeMap<String, Value> = dimensions
Expand All @@ -763,8 +780,18 @@ fn summary(checks: &[Value]) -> Value {
)
})
.collect();
let total = checks.len();
let not_applicable = observations["not_applicable"];
json!({
"dimensions": dimensions,
"observations": {
"total": total,
"applicable": total - not_applicable,
"pass": observations["pass"],
"fail": observations["fail"],
"unknown": observations["unknown"],
"not_applicable": not_applicable
},
"criteria_with_evidence": evidenced.len(),
"criteria_with_operating_evidence": operating.len()
})
Expand Down Expand Up @@ -1238,9 +1265,24 @@ pub fn run_verify() -> Result<i32, String> {
let unknowns = checks.iter().filter(|c| c["verdict"] == "unknown").count();
let report_summary = summary(&checks);
let report = json!({
"schema_version": 2,
"schema_version": 3,
"subject": {"repository": repo, "gcp_projects": gcp_projects},
"observed_at": observed_at,
"assurance": {
"basis": "automated-point-in-time",
"auditor_opinion": false,
"soc2_type_1": "not-determined",
"soc2_type_2": "not-determined",
"note": "Type II requires an elapsed CPA examination period and auditor-selected operating samples."
},
"provenance": {
"generator": "shadow-ci",
"repository": std::env::var("GITHUB_REPOSITORY").ok(),
"commit": std::env::var("GITHUB_SHA").ok(),
"run_id": std::env::var("GITHUB_RUN_ID").ok(),
"workflow": std::env::var("GITHUB_WORKFLOW").ok(),
"report_signature": "none"
},
"checks": checks,
"summary": report_summary,
"failures": failures,
Expand Down Expand Up @@ -1329,6 +1371,10 @@ mod tests {
let s = summary(&checks);
assert_eq!(s["dimensions"]["design"]["percent"], 100.0);
assert_eq!(s["dimensions"]["technical"]["percent"], 0.0);
assert_eq!(s["observations"]["total"], 2);
assert_eq!(s["observations"]["applicable"], 2);
assert_eq!(s["observations"]["pass"], 1);
assert_eq!(s["observations"]["fail"], 1);
}

#[test]
Expand Down
18 changes: 10 additions & 8 deletions website/SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,9 @@

## Purpose

One page. One number. A fixed gauge from 0 to 100% ("how completely does the current signed readiness snapshot satisfy its declared design, technical, observation, and operating dimensions?") above the full 61-criterion auditor checklist **and the machinery section** — the agents, workflows, webhooks, scanners, and registers installed on the project, sourced from [`../procedures/PROCEDURES.md`](../procedures/PROCEDURES.md). The gauge is the arithmetic mean of the signed report's design, technical, and operating percentages plus observation coverage (the percentage of checks with a known verdict); `unknown` therefore lowers the score. The site imports and renders the deterministic verifier's signed summary; it does not independently reinterpret criterion credit as the official score.
One page with one primary internal metric above the full 61-criterion checklist **and the machinery section** — the agents, workflows, webhooks, scanners, and registers installed on the project, sourced from [`../procedures/PROCEDURES.md`](../procedures/PROCEDURES.md). The gauge is **weighted, in-scope criterion evidence maturity**: `Σ(weight × credit) / Σ(weight)`, where verified is 100% credit, implemented/design-only is 60%, and failing or not-started is 0%. It is not a probability of passing an examination.

Automated observation results are rendered separately with an explicit denominator: pass, fail, unknown, and not-applicable. The page always says that it is an automated point-in-time assessment, not a SOC 2 report or CPA opinion; neither Type I nor Type II status is inferred. Type II requires an elapsed examination period and auditor-selected operating samples. Provenance names the repository, commit, workflow run, generator, and whether the JSON report itself has a cryptographic signature.

Everything renders on one sheet — no tabs, no view flips (they contradicted the one-pager paradigm and were removed):

Expand Down Expand Up @@ -63,9 +65,9 @@ CREATE TABLE attestations ( -- manual evidence for organizational criteria

CREATE TABLE gauge_history ( -- one row per verify run
ts TEXT PRIMARY KEY,
gauge REAL NOT NULL,
cap REAL, -- 79.0 when a hard gate is tripped, else NULL
cap_reason TEXT
gauge REAL NOT NULL, -- weighted in-scope criterion evidence maturity
cap REAL, -- retained for schema compatibility; currently NULL
cap_reason TEXT -- retained for schema compatibility
);

CREATE TABLE procedures ( -- the machinery ledger, seeded from procedures/PROCEDURES.md
Expand All @@ -82,7 +84,7 @@ CREATE TABLE procedures ( -- the machinery ledger, seeded from procedures/P

## Micro board (`/micro`)

The one-pager's dense sibling: one small box per criterion (ID + status glyph, status-colored, category headers, out-of-scope dimmed), the gauge in the corner. **Clicking a box runs that criterion's checks right now**: the click is a form POST (`/run/{id}`, zero JS) that spawns the verifier — the `claude` CLI if found on PATH (built-in single-criterion prompt: execute the criterion file's "Automated shadow checks" table, POST results to `/ingest`), or any command set in `SHADOW_RUNNER` (invoked via `sh -c` with `CRITERION`, `CRITERION_FILE`, `SHADOW_URL` env). While anything runs, boxes pulse ⟳ and the page polls via a meta-refresh. Two honesty rules: single-box runs never write a gauge entry (the official gauge moves only on the full verify), and with no verifier available the board renders read-only and says so. The site still never computes compliance — it triggers the agent that does.
The one-pager's dense sibling: one small box per criterion (ID + status glyph, status-colored, category headers, out-of-scope dimmed), with weighted criterion maturity in the corner. **Clicking a box runs that criterion's checks right now**: the click is a form POST (`/run/{id}`, zero JS) that spawns the verifier — the `claude` CLI if found on PATH (built-in single-criterion prompt: execute the criterion file's "Automated shadow checks" table, POST results to `/ingest`), or any command set in `SHADOW_RUNNER` (invoked via `sh -c` with `CRITERION`, `CRITERION_FILE`, `SHADOW_URL` env). While anything runs, boxes pulse ⟳ and the page polls via a meta-refresh. Two honesty rules: single-box runs never write a gauge entry (the primary metric moves only on the full verify), and with no verifier available the board renders read-only and says so. The site never computes or claims compliance; it records evidence state.

## Routes

Expand All @@ -101,13 +103,13 @@ Responsive: below 900px the sheet goes single-column (cards stack, header stacks

## The page (top to bottom)

1. **Gauge** — fixed semicircular arc, 0–100%, needle at current gauge. Color bands: 0–49 red, 50–79 amber, 80–94 green, 95–100 deep green. If a hard-gate cap is active, the arc beyond the cap renders hatched with the cap reason underneath ("capped at 79% — org 2FA not enforced"). Below the needle: the trend sparkline from `gauge_history`.
2. **Category chips** — Security 33/33 in scope, Availability, Confidentiality, PI, Privacy — with per-category sub-scores; out-of-scope categories greyed with "not in scope".
1. **Criterion maturity gauge** — fixed semicircular arc, 0–100%, needle at weighted in-scope evidence maturity. The formula and credits are printed beside it. Below the needle: the comparable trend from `gauge_history`; a metric-version migration discards incompatible legacy history.
2. **Evidence summary and category chips** — verified/implemented/not-started/failing criterion counts; applicable automated pass/fail/unknown counts with n/a separated; Security, Availability, Confidentiality, PI, and Privacy weighted sub-scores; out-of-scope categories greyed with "not in scope".
3. **The Machinery** — the ten territory cards (see above).
4. **The Criteria** — the 61-cell checkbox matrix (see above), mirroring [CHECKLIST.md](../CHECKLIST.md) content via hover.
5. **Footer** — last verify run time, count of `unknown` checks ("blind spots"), link to `/db`.

Honest-rendering rules: a stale verify run (>48h) banners the whole page ("state is stale — monitor may be dead"); `unknown` never displays as pass and lowers observation coverage; the gauge is always shown with its computation date, never as a timeless fact. A perfect dashboard score means the declared machine-verifiable readiness checks all pass with no blind spots; it is not an auditor's opinion or a substitute for a Type II observation period.
Honest-rendering rules: a stale verify run (>48h) banners the whole page ("state is stale — monitor may be dead"); `unknown` never displays as pass and not-applicable never inflates the applicable denominator; the gauge is always shown with its computation date, formula, and denominator. A perfect maturity score only means every in-scope criterion received full credit under this internal evidence rubric. It is never an auditor's opinion, a prediction of examination outcome, or a substitute for either a Type I CPA evaluation or a Type II observation period.

## Seeding

Expand Down
Loading