Proposal: optional audit manifest for reproducible MLCube runs

[mindbomber]

Proposal

Would MLCube be open to an optional run audit manifest for MLCube executions?

MLCube already focuses on portability and reproducibility. A small sidecar manifest could make benchmark runs easier to review, compare, cite, and publish safely without changing the core MLCube task interface.

Suggested manifest shape

{
  "schema_version": "mlcube.run_audit.v1",
  "mlcube_task": "run",
  "runner": "docker",
  "image": "...",
  "mlcube_config_hash": "...",
  "benchmark": "...",
  "dataset_refs": [
    {
      "source_id": "...",
      "kind": "dataset",
      "provenance": "...",
      "redaction_status": "safe_for_public_log"
    }
  ],
  "result_paths": ["..."],
  "provenance": {
    "repo": "...",
    "commit": "...",
    "created_at": "..."
  },
  "claim_status": "diagnostic",
  "redaction_status": "safe_for_public_log"
}

Why this may help

• makes it clearer which MLCube/config/image produced a benchmark result
• preserves provenance such as runner, image, config hash, repo commit, dataset refs, and result paths
• separates diagnostic/internal runs from results intended for public reports or model cards
• gives downstream benchmark users a standard place for audit-safe metadata
• avoids storing raw secrets, private paths, tokens, or full sensitive arguments in public logs

Scope I would keep small

If maintainers think this is useful, I can prepare a follow-up PR that:

• documents an optional manifest schema
• adds a minimal example manifest under docs/examples or docs/getting-started
• keeps the manifest opt-in and backward compatible
• does not change existing runner behavior by default
• does not add external service dependencies

This is motivated by work in the AANA project around audit-safe AI evaluation artifacts, but the contribution would be generic to MLCube and would not require AANA as a dependency.

[HarperZ9]

I like keeping this optional and sidecar-based. One refinement that may make it easier to implement without changing the runner contract is splitting the shape into a local run_receipt and a shareable public_attestation.

A minimal receipt could be:

{
  "schema": "mlcube.run_receipt.v1",
  "run_id": "...",
  "created_at": "...",
  "mlcube": {
    "task": "run",
    "mlcube_yaml_hash": "sha256:...",
    "parameters_hash": "sha256:...",
    "redacted_parameter_keys": ["..."]
  },
  "runtime": {
    "runner": "docker|singularity|host",
    "image_ref": "...",
    "image_digest": "sha256:...",
    "container_runtime_version": "...",
    "os": "...",
    "python": "...",
    "cuda": "...",
    "package_lock_hash": "sha256:...",
    "env_hash": "sha256:..."
  },
  "code": {
    "repo": "...",
    "commit": "...",
    "dirty_tree": false
  },
  "data": [
    {
      "dataset_id": "...",
      "dataset_version": "...",
      "manifest_hash": "sha256:...",
      "access_policy_ref": "..."
    }
  ],
  "outputs": [
    {
      "path": "metrics.json",
      "kind": "metrics|model|log|artifact",
      "hash": "sha256:..."
    }
  ],
  "execution": {
    "started_at": "...",
    "finished_at": "...",
    "exit_code": 0
  },
  "redaction_profile": "public-log-safe",
  "claim_status": "diagnostic|candidate|published"
}

Then public_attestation can be a strict subset/digest-only projection of the receipt for papers, model cards, benchmark dashboards, or GitHub issues. That keeps private paths, tokens, raw arguments, and restricted dataset details out of public artifacts while still making the run auditable.

Suggested CI tests for a first PR:

• sample cube emits a receipt only when explicitly requested, e.g. --audit-manifest out.json
• receipt validates against JSON Schema
• image digest, mlcube.yaml hash, dataset manifest hash, and output hashes are present for the sample run
• secret-looking env vars / bearer tokens / private absolute paths are redacted or rejected from the public attestation
• rerunning the same sample keeps stable identity hashes for config/data/runtime while allowing timestamps and duration to differ
• a deliberately modified metrics file fails receipt verification

That pattern maps well to Project Telos/Crucible-style receipts: keep the private source payload local, publish hashes/refs/verdicts, and make claims re-checkable without turning the manifest into another data leak surface.