Add DeepMind-style progressive references and complete scientific runtime v4

[eightmm]

Summary

This PR turns Codex Science into a progressively disclosed, local-first scientific runtime. SKILL.md remains a thin decision router; exact commands, arguments, schemas, failure handling, common mistakes, and evidence boundaries live in indexed references/*.md files and are loaded only for the selected route.

Progressive references and skill quality

• machine-readable reference indexes with stable IDs, purpose, read_when, mandatory required_before, search patterns, authority, version, and evidence boundaries;
• hash-covered reference-use ledgers;
• detailed DeepMind-style usage references for artifact inspection, project evidence memory, local/Slurm jobs, pipeline promotion, next-experiment planning, FAIR export, independent review packets, and authenticated connector packs;
• machine-readable quality.json declarations;
• deterministic L0-L4 maturity computation from actual instruction sections, progressive references, output schemas, checked fixtures, seeded failures, and tests;
• repository policy for coordinator, provenance, reviewer, literature review, docking validation, and remote compute;
• reference and maturity audits are part of candidate/install and CI contract gates.

Connector transport and large artifacts

• bounded Connector Transport v3 with retries, response-size caps, ETag/Last-Modified/rate-limit capture, real page receipts, replay, and drift classification;
• typed operations for initial high-value scientific sources while preserving legacy MCP and v2 contracts;
• streaming SHA-256, chunk descriptors, directory Merkle roots, immutable external references, content-addressed storage, and symlink/path-traversal rejection;
• manifest schema 1 remains backward compatible through optional artifact metadata.

Native scientific artifact runtime

• bounded non-mutating views for PDB/mmCIF, SDF/MOL/SMILES, VCF/BED/genome tracks, H5AD/Loom metadata, CSV/TSV/Parquet metadata, figures, evidence graphs, trajectories, directories, text, and binary files;
• hard byte, record, selection, proposal, and reason limits;
• typed atom/residue/chain, molecular, locus/interval, single-cell, table, figure, graph, trajectory, line, and byte selections;
• every descriptor and selection is bound to artifact path and SHA-256;
• stale anchors remain explicit;
• annotations become non-executing transform proposals with affected steps, expected outputs, and approval boundaries;
• offline escaped HTML view;
• validated sidecars: artifact-runtime-descriptor, artifact-selection, and transform-proposal.

Project evidence memory and experiment forks

• local SQLite project index over immutable, validated run bundles;
• project, run, parent, branch-base, and branch-head lineage;
• quantitative evidence assertions with effect measure, interval, unit, sample size, population, independence group, risk-of-bias reference, and exact hash-bound artifact locator;
• run comparison and review invalidation;
• non-executing three-way scientific merge plans with claim and artifact conflicts;
• SHA-256 chained project event ledger;
• manifests and artifact bytes remain authoritative.

Durable compute runtime

• common job spec, preflight, approval, state, cancellation, checkpoint, and collection receipts;
• durable local backend with detached worker, atomic state, logs, timeout, status, wait, cancel, and output file/directory hashing;
• input file SHA-256 and directory Merkle validation before execution;
• approval-gated Slurm backend with rendered script preview, sbatch, sacct/squeue, scancel, and shared-filesystem collection;
• distinct failure classes for nonzero exit, wall time, worker loss, user cancellation, scheduler OOM, node failure, preemption, and timeout;
• process exit zero is never treated as scientific acceptance.

Reviewed-run pipeline promotion

• converts a currently passed run into a non-active skill draft;
• generates SKILL.md, progressive reference index, exact pipeline reference, input/output schemas, and promotion receipt;
• preserves source manifest identity and argv contracts without copying source conclusions;
• raw private input values are not copied;
• secret-like environment fields are redacted;
• credential-bearing argv and URLs are rejected;
• no automatic activation, catalog insertion, receipt reuse, or generalization claim.

Next-experiment planning

• deterministic eligibility filtering, Pareto fronts, weighted objectives, uncertainty bonus, explicit diversity grouping, required controls, budget, batch size, and group caps;
• selected and rejected candidates retain declared properties and reasons;
• proposal is non-executing, approval-required, and fingerprinted;
• no inferred chemical similarity, calibrated information-gain claim, or experiment execution.

FAIR-oriented export and scientific BOM

• validated run export to RO-Crate-oriented metadata, W3C PROV-oriented lineage, scientific dependency BOM, and hash-covered export receipt;
• records code, packages, containers, lockfiles, models, weights, databases, datasets, inputs, and licenses when present in source provenance;
• does not overwrite a non-empty export directory;
• explicitly does not certify formal standard conformance, licensing, regulatory compliance, provenance completeness, or scientific validity.

Independent reviewer packet runtime

• deterministic, blinded, hash-bound packets for separate human or agent review;
• excludes producer-only rationale fields and private chain-of-thought fields;
• redacts secret-like keys and credential-bearing URL query values;
• binds review modes, claims, artifact hashes, graph, studies, queries, lanes, models, and review questions;
• finalizes separate responses into review receipts only after validating packet identity, claim coverage, artifact hashes, findings, limitations, and independence attestation;
• a producer second pass cannot be labeled independent;
• reproduction requires a separate execution.

Optional authenticated connector-pack framework

• core remains public/read-only and stores no provider credentials;
• optional packs implement a common adapter contract with out-of-band credentials;
• action spec, provider preview, exact scopes, optimistic before-state hash, explicit approval, idempotency ledger, before/after hashes, provider IDs, compensation, and action receipt;
• provider records with credential-like fields are rejected;
• write actions cannot disable approval;
• no provider is claimed supported until a separate pack with pinned API references and fixtures exists.

Tests and validation

Added deterministic tests for:

• artifact views, selections, proposals, and sidecar staleness;
• project imports, assertions, forks, comparisons, conflicts, and immutable manifest identity;
• durable local jobs, Slurm rendering, approvals, timeouts, input hash preflight, and secret boundaries;
• native skill maturity and progressive reference policy;
• pipeline draft generation and secret/private-input protection;
• next-experiment controls, Pareto ranking, diversity, budget, determinism, and infeasibility;
• FAIR exports and changed-byte rejection;
• blinded reviewer packets, Unicode-safe review receipts, coverage, independence, and artifact substitution rejection;
• authenticated action preview, approval, idempotency, concurrency, provider-secret rejection, and receipt validation;
• absolute resource ceilings and boolean/integer contract hardening.

The clean branch contains reports/native-runtime-v4-validation.json, generated only after:

• permanent source compilation;
• focused runtime suites;
• complete Python 3.11 scripts/check.sh fast gate;
• complete Python 3.12 unit suite;
• inventory and wrapper determinism;
• progressive reference and maturity audits;
• release, connector, artifact, reviewer, model, and SBDD contracts.

CodeQL and hosted public-source/live-provider checks remain separate workflows. A passed software contract does not establish scientific truth, empirical model performance, provider availability, or regulatory compliance.