One-command HF publish for Cross-dataset evaluation bundles

docs/chronicle-update-workflow.md

@@ -83,7 +83,9 @@ The command refuses to use an integration reviewed against another snapshot.
 It classifies every fact for every registered source, runs Microcosm +
 PolicyEngine-US, Public CPS + Tax-Calculator, and Raw ACS PUMS, incorporates the
 reviewed Yale reconstruction checkpoint, scores the results, and publishes the
-frontend partitions.
+frontend partitions. Upload the resulting bundle to the Hugging Face dataset
+with `scripts/publish_evaluation_bundle_to_hf.py`, described in
+[the Cross-dataset artifact API](cross-dataset-api.md#publish-a-bundle-to-hugging-face).
 
 ## 4. Manual harness verification
 

docs/cross-dataset-api.md

@@ -53,6 +53,65 @@ run before writing. It emits a manifest, summary and group partitions, a fact
 index, and bounded fact pages. Every partition carries the immutable run and
 snapshot IDs and has a SHA-256 recorded in the manifest.
 
+## Publish a bundle to Hugging Face
+
+Publish the verified bundle to the
+[`policyengine/microcosm-evaluation`](https://huggingface.co/datasets/policyengine/microcosm-evaluation)
+dataset with one command. It reproduces the layout of the Belgium bundle
+referenced below:
+
+```bash
+export HF_TOKEN=...   # or HUGGINGFACE_TOKEN; a write token for the dataset
+uv run --extra publish python scripts/publish_evaluation_bundle_to_hf.py \
+  --bundle /path/to/evaluation-run/frontend \
+  --jurisdiction BE
+```
+
+Options: `--repo <owner/name>` targets another dataset, `--dry-run` verifies the
+bundle and prints the upload plan without any network call (and without the
+`publish` extra or a token), and `--no-latest` leaves the country's pointer
+alone. The dataset layout is:
+
+```text
+<cc>/<run_id>/frontend/{manifest,summary,groups,fact-index}.json
+<cc>/<run_id>/frontend/facts/NNNNN.json
+<cc>/latest.json
+```
+
+`<cc>` is the dashboard country code: the lower-case jurisdiction (`us`, `uk`,
+`be`), with `GB`-coded bundles published under `uk`. The command:
+
+1. verifies the bundle locally: `schema_version` must be
+   `cross_dataset.frontend_bundle.v1`, the manifest's `jurisdictions` must
+   contain `--jurisdiction`, every listed partition must match its recorded
+   SHA-256, and every partition must carry the manifest's run and snapshot IDs;
+2. compares the bundle with `<cc>/<run_id>/frontend/` on the Hub. Files that
+   already exist with the same size and content are skipped, so a re-run is a
+   no-op. A file that differs under an existing run ID aborts the publish
+   before anything is uploaded: run directories are immutable;
+3. uploads the missing files and, unless `--no-latest`, writes
+   `<cc>/latest.json` in one commit:
+
+   ```json
+   {
+    "schema_version": 1,
+    "jurisdiction": "BE",
+    "run_id": "evaluation-f28ca06a0b0d2baf13c87f2f",
+    "snapshot_id": "chronicle-82b574e3a8526ce0718ad08d",
+    "base_path": "be/evaluation-f28ca06a0b0d2baf13c87f2f/frontend/"
+   }
+   ```
+
+4. re-downloads `manifest.json` and `summary.json` (and `latest.json` when
+   written) through the public resolve URL, exactly as the dashboard reads
+   them, and checks their hashes against the local bundle;
+5. prints the variable to set for the dashboard, for example
+   `CROSS_DATASET_ARTIFACT_BASE_URL_BE=https://huggingface.co/datasets/policyengine/microcosm-evaluation/resolve/main/be/<run_id>/frontend/`.
+
+The token is read only from `HF_TOKEN` or `HUGGINGFACE_TOKEN`; the script never
+reads a keychain or a cached login. Files in the bundle directory that the
+manifest does not list are reported and never uploaded.
+
 ## Configure the application
 
 Configure at most one local directory or remote base URL for each country. The
@@ -89,7 +148,9 @@ export CROSS_DATASET_ARTIFACT_BASE_URL=https://example.org/evaluation-run/fronte
 make dev
 ```
 
-For example, the published Belgium bundle can be selected with:
+A bundle published with `scripts/publish_evaluation_bundle_to_hf.py` is
+selected with the base URL the command prints; `<cc>/latest.json` on the
+dataset names the current run. For the published Belgium bundle:
 
 ```bash
 export CROSS_DATASET_ARTIFACT_BASE_URL_BE=https://huggingface.co/datasets/policyengine/microcosm-evaluation/resolve/main/be/evaluation-f28ca06a0b0d2baf13c87f2f/frontend/

evaluation_harness/frontend_bundle.py

@@ -3,6 +3,7 @@
 import hashlib
 import json
 import math
+import re
 from collections import Counter
 from decimal import Decimal
 from pathlib import Path
@@ -21,6 +22,10 @@
 FRONTEND_BUNDLE_SCHEMA = "cross_dataset.frontend_bundle.v1"
 WITHIN_BOUNDS_RELATIVE_ERROR = Decimal("0.10")
 FAR_OUTSIDE_BOUNDS_RELATIVE_ERROR = Decimal("0.25")
+# Mirrors the web reader's partition descriptor rules (frontend/lib/cross-dataset/
+# artifact.ts) so a bundle that passes here is one the dashboard will accept.
+_PARTITION_PATH = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._/-]*$")
+_SHA256_HEX = re.compile(r"^[a-f0-9]{64}$")
 
 DEFAULT_SOURCE_LABELS = {
     "census_acs_pums_2024": "Raw ACS PUMS",
@@ -808,3 +813,148 @@ def add_facet(dimension: str, key: str, page: int) -> None:
     }
     (output / "manifest.json").write_bytes(_document(manifest))
     return manifest
+
+
+def _partition_descriptor(value: Any, field: str) -> dict[str, Any]:
+    if not isinstance(value, dict):
+        raise ValueError(f"frontend bundle partition descriptor {field} is missing")
+    path = value.get("path")
+    sha256 = value.get("sha256")
+    if not isinstance(path, str) or not path:
+        raise ValueError(f"frontend bundle partition {field} has no path")
+    if (
+        path.startswith("/")
+        or "\\" in path
+        or any(part in ("", ".", "..") for part in path.split("/"))
+        or not _PARTITION_PATH.match(path)
+    ):
+        raise ValueError(
+            f"frontend bundle partition {field} has an unsafe path: {path!r}"
+        )
+    if not isinstance(sha256, str) or not _SHA256_HEX.match(sha256):
+        raise ValueError(f"frontend bundle partition {field} has no SHA-256")
+    return value
+
+
+def frontend_bundle_partitions(manifest: dict[str, Any]) -> list[dict[str, Any]]:
+    """Return the manifest's partition descriptors in a stable order.
+
+    The order is summary, groups, fact index, then fact pages. Every descriptor
+    carries a safe relative ``path`` and a hex ``sha256``; malformed manifests
+    raise ``ValueError``.
+    """
+
+    partitions = manifest.get("partitions")
+    if not isinstance(partitions, dict):
+        raise ValueError("frontend bundle manifest has no partitions")
+    expected_keys = {"summary", "groups", "fact_index", "facts"}
+    unknown_keys = set(partitions) - expected_keys
+    if unknown_keys:
+        unknown = ", ".join(sorted(str(key) for key in unknown_keys))
+        raise ValueError(f"frontend bundle manifest has unknown partitions: {unknown}")
+    for field, minimum in (("fact_count", 0), ("page_count", 0), ("page_size", 1)):
+        value = manifest.get(field)
+        if not isinstance(value, int) or isinstance(value, bool) or value < minimum:
+            raise ValueError(f"frontend bundle manifest has no valid {field}")
+    descriptors = [
+        _partition_descriptor(partitions.get(key), f"partitions.{key}")
+        for key in ("summary", "groups", "fact_index")
+    ]
+    facts = partitions.get("facts")
+    if not isinstance(facts, list):
+        raise ValueError("frontend bundle manifest has no fact partitions")
+    for index, value in enumerate(facts):
+        descriptor = _partition_descriptor(value, f"partitions.facts[{index}]")
+        page = descriptor.get("page")
+        if not isinstance(page, int) or isinstance(page, bool) or page != index + 1:
+            raise ValueError("frontend bundle fact partition sequence is incomplete")
+        count = descriptor.get("count")
+        if not isinstance(count, int) or isinstance(count, bool) or count < 0:
+            raise ValueError(f"frontend bundle fact partition {index + 1} has no count")
+        descriptors.append(descriptor)
+    if len(facts) != manifest.get("page_count"):
+        raise ValueError("frontend bundle fact partition sequence is incomplete")
+    if sum(descriptor["count"] for descriptor in facts) != manifest.get("fact_count"):
+        raise ValueError("frontend bundle fact partition counts do not reconcile")
+    paths = [descriptor["path"] for descriptor in descriptors]
+    if len(paths) != len(set(paths)):
+        raise ValueError("frontend bundle partition paths must be unique")
+    return descriptors
+
+
+def verify_frontend_bundle(bundle_path: str | Path) -> dict[str, Any]:
+    """Verify a published frontend bundle directory and return its manifest.
+
+    Checks the manifest schema, recomputes the SHA-256 of every partition the
+    manifest lists, and confirms each partition carries the manifest's schema,
+    run ID, and snapshot ID. Raises ``ValueError`` on any mismatch.
+    """
+
+    bundle = Path(bundle_path)
+    manifest_path = bundle / "manifest.json"
+    if not manifest_path.is_file():
+        raise ValueError(f"frontend bundle has no manifest.json: {bundle}")
+    try:
+        manifest = json.loads(manifest_path.read_bytes())
+    except json.JSONDecodeError as error:
+        raise ValueError("frontend bundle manifest.json is not valid JSON") from error
+    if not isinstance(manifest, dict):
+        raise ValueError("frontend bundle manifest.json is not an object")
+    if manifest.get("schema_version") != FRONTEND_BUNDLE_SCHEMA:
+        raise ValueError(
+            f"unsupported frontend bundle schema: {manifest.get('schema_version')!r}"
+        )
+    for field in ("run_id", "snapshot_id"):
+        if not isinstance(manifest.get(field), str) or not manifest[field]:
+            raise ValueError(f"frontend bundle manifest has no {field}")
+    jurisdictions = manifest.get("jurisdictions")
+    if jurisdictions is not None and (
+        not isinstance(jurisdictions, list)
+        or any(not isinstance(value, str) or not value for value in jurisdictions)
+    ):
+        raise ValueError("frontend bundle manifest jurisdictions must be strings")
+    descriptors = frontend_bundle_partitions(manifest)
+    for index, descriptor in enumerate(descriptors):
+        path = bundle / descriptor["path"]
+        if not path.is_file():
+            raise ValueError(f"frontend bundle is missing {descriptor['path']}")
+        content = path.read_bytes()
+        if _sha256(content) != descriptor["sha256"]:
+            raise ValueError(f"frontend bundle hash mismatch for {descriptor['path']}")
+        try:
+            document = json.loads(content)
+        except json.JSONDecodeError as error:
+            raise ValueError(
+                f"frontend bundle partition {descriptor['path']} is not valid JSON"
+            ) from error
+        if not isinstance(document, dict) or any(
+            document.get(field) != manifest[field]
+            for field in ("schema_version", "run_id", "snapshot_id")
+        ):
+            raise ValueError(
+                f"frontend bundle partition {descriptor['path']} belongs to "
+                "another run or snapshot"
+            )
+        if index >= 3:
+            integer_fields = {
+                "page": descriptor["page"],
+                "page_size": manifest["page_size"],
+                "total": manifest["fact_count"],
+            }
+            if any(
+                not isinstance(document.get(field), int)
+                or isinstance(document[field], bool)
+                or document[field] != expected
+                for field, expected in integer_fields.items()
+            ):
+                raise ValueError(
+                    f"frontend bundle fact partition {descriptor['path']} has "
+                    "inconsistent page metadata"
+                )
+            rows = document.get("rows")
+            if not isinstance(rows, list) or len(rows) != descriptor["count"]:
+                raise ValueError(
+                    f"frontend bundle fact partition {descriptor['path']} has "
+                    "an inconsistent row count"
+                )
+    return manifest

pyproject.toml

@@ -12,6 +12,7 @@ microcosm = [
   "tables>=3.10,<4",
 ]
 taxcalc-cps = ["taxcalc==6.7.1"]
+publish = ["huggingface-hub>=0.27,<2"]
 
 [project.scripts]
 evaluation-harness = "evaluation_harness.cli:main"