Skip to content

feat: add native prospective instrumentation - #7

Draft
jimmlucas wants to merge 1 commit into
feat/esdp-lightfrom
feat/prospective-instrumentation
Draft

feat: add native prospective instrumentation#7
jimmlucas wants to merge 1 commit into
feat/esdp-lightfrom
feat/prospective-instrumentation

Conversation

@jimmlucas

Copy link
Copy Markdown
Owner

What changed

  • adds a native esdp init command for prospective long-read instrumentation projects
  • adds esdp record-round to collect one light observation and rebuild its causal history as one immutable round transaction
  • exposes the same workflow through Python init_project and record_round functions
  • records sequencing technology, chemistry, ONT basecaller/model, assembler, polisher, versions, and frozen parameters
  • generates a strict offline-QC JSON Schema for BUSCO counts/provenance, independent k-mer QV, separated consensus errors, homopolymer indels, and coding-integrity outcomes
  • binds observations, external QC, reads, and project contracts through SHA-256 identities
  • rejects skipped/overwritten rounds, unsafe identifiers, mismatched assembly QC, pre-polish alignment references, duplicate reads, and project-contract drift
  • keeps every project and round permanently at decision_enabled: false
  • packages the instrumentation API in the wheel and Docker runtime

Why

ESDP needs prospectively collected, provenance-complete long-read trajectories before another stopping model can be scientifically selected. This layer lets ESDP collect those data directly, without nf-core/Nextflow and without enabling experimental STOP/CONTINUE decisions.

Expensive evaluators remain external for now. ESDP validates and archives their results rather than silently introducing BUSCO, Merqury, alignment, or gene-prediction runtime dependencies.

Immutable project layout

Each trajectory stores independent R1...Rn directories. A round includes its observation, cumulative causal history, transaction record, and optional offline QC. Round directories are committed by atomic rename; recording R2 never modifies R1.

Every round records the SHA-256 of esdp-project.json. Changing technology or tool provenance after R1 blocks subsequent rounds.

User interfaces

esdp init --project-directory study ...
esdp record-round --project-directory study --sample-id isolate-1 --round 1 ...

Equivalent Python functions are documented in docs/PROSPECTIVE_INSTRUMENTATION.md.

Scientific boundaries

  • no model is loaded or trained
  • no stopping decision is produced
  • expensive QC outcomes never enter the light feature history
  • the held-out scientific test set remains untouched
  • future collectors can execute external tools only after their commands, containers, cost, and portability contracts are frozen

Validation

  • full suite: 201 passed
  • instrumentation/CLI/integration focused suite: 63 passed
  • wheel built successfully without build isolation
  • wheel contains esdp_instrumentation.py, CLI, metrics, and history modules
  • wheel installed into an external virtual environment
  • installed esdp init successfully initialized a PacBio HiFi project outside the repository
  • git diff --cached --check: clean

This is a stacked draft PR targeting feat/esdp-light so its diff contains only prospective instrumentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant