Extraction concurrency + maintenance by jimador · Pull Request #48 · embabel/dice

jimador · 2026-06-19T12:18:25Z

Extraction concurrency + maintenance

Second of the four-PR stack. #47 (store layer + trust authority) has merged and this branch is rebased onto main — the diff below is this PR's own changes only. #49 and #50 stack on top of this one.

What's in it

Extract many documents at once without breaking reconciliation (Extract many documents at once without breaking reconciliation #36) — splits extraction into
a concurrent extract stage and a serial reconcile stage behind a pluggable execution strategy,
so concurrency is a swappable policy rather than baked in.
Reclaim stale and duplicate facts instead of growing forever (Reclaim stale and duplicate facts instead of growing forever #37) — a GC-style mark-and-sweep
collector: pluggable mark strategies, a sweep policy (transition / delete / skip), and an audit
record for every reclamation.
Tidy and consolidate what DICE knows between sessions (Tidy and consolidate what DICE knows between sessions #38) — a background "dream loop" of
composable consolidation passes (abstract related, resolve contradictions, sweep decayed) and an
orchestrator that runs them as repeatable, threshold-gated cycles, wired into existing memory
maintenance.
Check new facts before they reach the store (Check new facts before they reach the store #39) — an ExtractionGatePipeline that runs
right after extraction and emits an observable decision per proposition (persist, route to review,
reject, skip projection, or demote below the evidence floor).

Resolves pre-existing design issues

Post-extraction evaluation pipeline #15 Post-extraction evaluation pipeline — ExtractionGatePipeline / StandardGates
implements the configurable gate chain the issue proposed (pass / reject / route / transform,
cheap gates first), sitting between extraction and persistence.
Proposition pinning and eviction immunity #9 Proposition pinning and eviction immunity — Proposition.pinned is now a real "must
retain" guarantee: pinned propositions are immune to decay marking, sweep reclamation, and
automatic contradiction (at both ingest and consolidation), with pin/unpin/findPinned
store helpers and a pinned query filter. Budget-priority injection stays with Proposition provenance metadata #7.

Related but not closed here: #10 (decay sweeping advances it; content-aware decay curves remain
open).

Hardening & quality

After the feature work the branch went through an adversarial review pass and follow-up hardening:

Correctness — the REST extract endpoints now persist revised/contradicted originals (not just
the new propositions); contradiction resolution resolves each pair once with a corrected tie-break;
the concurrent extraction strategies are AutoCloseable so a self-created pool can't leak; gates
expose a shouldSave guard so Demote/SkipProjection aren't dropped; dream-loop cycle totals count
new vs. transitioned correctly; dry-run sweeps record MARKED rather than looking like real sweeps.
Concurrency — the dream loop locks cycles per contextId (no overlapping same-context cycles;
different contexts still run in parallel), the pipeline's per-chunk failure map is a
ConcurrentHashMap, and the thread-safety contracts across the collector, maintenance, and
resolver are documented.
Observability — debug/trace logging across extraction, maintenance, the consolidation passes,
and per sweep decision.
API placement — the sweep-policy cluster (SweepPolicy/StatusTransitionSweepPolicy plus its
SweepAction/PropositionMark/MarkReason vocabulary) lives in com.embabel.dice.spi alongside
the other lifecycle/trust policies. (Moving these public types is source-incompatible for any
external importer of the old package — acceptable pre-1.0.)
Docs — four design notes under docs/design/ with mermaid diagrams (extraction pipeline;
knowledge hygiene; consolidation & the dream loop; reclamation & the collector), and AGENTS
navigation updated for every new subsystem.
Tests — per-context-lock concurrency regression tests, a DecayStatusPolicy unit test, and a
no-mocks end-to-end reclamation test, on top of the per-feature suites.

Verification

mvn verify is green across all modules: dice 935 unit tests + dice-storage 27 (including the Neo4j
Testcontainers integration tests).

Closes #36
Closes #37
Closes #38
Closes #39
Closes #15
Closes #9

Splits chunk extraction into a parallelizable phase and a serial one. Phase 1 pulls propositions from a chunk and builds suggested entities with no shared state, so it can run concurrently across chunks; phase 2 resolves entities against the shared cross-chunk resolver and stays serial, keeping entity identity consistent. Single-chunk behavior is unchanged. - ExtractionExecutionStrategy is the pluggable seam — serial or concurrent - ChunkPropositionResult becomes a sealed type: Success carries the resolver output; a failed chunk yields an empty result instead of aborting the batch - the extract controller reads entityResolutions from Success results Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Adds a garbage-collector-style maintenance pass: mark candidates, then sweep. It runs dry by default and records what each run did, so a real run's effect is visible before anything is removed. - CollectorStrategy with DecayCollectorStrategy and DuplicateCollectorStrategy; CollectorRunner / DefaultCollectorRunner drive mark then sweep - SweepPolicy and SweepAction decide what happens to a marked proposition; MarkReason and PropositionMark record why it was marked - CollectorRun / CollectorRecord / CollectorOutcome give every run an audit trail, held by a CollectorRecordStore Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…orchestrator Adds offline passes that tidy stored memory the way sleep consolidates the day, and an orchestrator that runs them. Opt-in — nothing runs by default. - ConsolidationPass with SessionConsolidationPass, AbstractionPass, ContradictionResolutionPass, and DecaySweepPass (which reuses the collector, so consolidation and collection share one mechanism) - DreamLoopOrchestrator runs the passes and emits a DreamLoopReport per run - MemoryMaintenanceOrchestrator ties the consolidation passes together with the collector pass; MemoryConsolidator reaches its final form here Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Adds a pipeline that inspects each freshly extracted proposition and decides keep / reject / route-to-review / demote before it is stored. Opt-in — the default keeps current behavior. - ExtractionGate and ExtractionGatePipeline; StandardGates are the built-in checks - ObservableGate wraps a gate and emits its decision as an event, so a rejection or demotion is never silent - EvidenceFloor demotes a relation that lacks enough supporting evidence to a weaker one (PropositionDemoted) rather than rejecting it outright Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Add docs/design/knowledge-hygiene.md — the design decisions behind keeping the store healthy: admitting deliberately at post-extraction gates, reclaiming via mark-and-sweep with an audit trail, and consolidating between sessions with the dream loop. Frames the three as separate, pluggable, conservative-by-default interventions tied to the proposition lifecycle. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…enance, and store paths Wire SLF4J loggers through the store, trust, conflict, event, gate, and memory-maintenance seams so a consuming application can see the decision and persistence paths: - gates: log each gate's decision outcome, pipeline short-circuits, and every observable gate event - maintenance: log collector-run and dream-loop summaries, decay/duplicate strategy mark counts, and retirement counts - trust/events/store: resolved authority tier and trust score, conflict verdict, emitted lifecycle and projection events, query match/return counts, JSON-file load count and flush-failure rollbacks, and no-embedder vector skips Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…common layering Move the policy extension points — TrustScorer, AuthorityResolver/AuthorityTier, AuthorityWeightedTrustScorer, ConflictDetector, and ConflictType (plus their shipped defaults) — out of common and proposition.revision into a dedicated com.embabel.dice.spi package, giving the library's extension points one home. This also fixes a layering wrinkle: common no longer imports proposition.revision.ConflictType. ConflictType now lives in spi alongside the policies that share it, and proposition.revision depends downward on spi. Tests mirror the move into the spi package; no behaviour change. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

The extract endpoints saved only the freshly extracted propositions, so when revision contradicted an existing proposition the retired original (reduced confidence, CONTRADICTED status) was never written — it stayed ACTIVE at full confidence. Persist propositionsToPersist() instead, which includes revised originals. Also: parse and apply the knownEntities part on file uploads (it was bound then dropped), surface failed chunk ids in the file response instead of an empty list, and reject blank extract text with 400 rather than calling the LLM. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…ct tie-break A symmetric reviser reports the same conflict from both sides, so the pass retired both members and left no survivor. Resolve each unordered pair only once. Also change the tie comparison from <= to <, so a tie genuinely favors the proposition being classified (as the KDoc states) rather than retiring it. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…ir pool A default-constructed ParallelExtractionStrategy/BatchedExtractionStrategy created a non-daemon thread pool in a private field with no way to shut it down, so the threads leaked. Both strategies are now AutoCloseable: close() shuts down a pool they created themselves and leaves a caller-supplied executor untouched. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

shouldPersist is true only for a plain Persist decision, so a consumer guarding save() with it silently dropped Demote and SkipProjection results, which both mean keep the proposition. Add shouldSave covering every non-reject, non-review decision. Also fix the KDoc example to read effectiveConfidence() rather than the raw confidence, matching the real ConfidenceGate. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

A dry run recorded its preview rows with TRANSITIONED/HARD_DELETED outcomes — indistinguishable from a real sweep when querying records by outcome. Record them as MARKED (the purpose-built preview outcome). Also flag the pure-read collect() result as dryRun, since it applies nothing. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

totalNewPropositions counted re-saved snapshot members (e.g. SUPERSEDED sources) as new, and totalTransitioned missed decay transitions entirely because the decay pass writes through its own runner and returns empty save/delete lists. Count a proposition as new only when its id was absent from the examined snapshot, and let a pass report transitions it applied itself via a new Changed.externallyApplied count that the orchestrator folds into the transitioned total. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Documents the part of the substrate that had no design note yet: the extraction pipeline. Covers why extraction is split into two phases, why only the resolver-free first phase may run concurrently while resolution stays serial, the execution-strategy SPI and its order-preserving / null-slot-on-failure contract, per-chunk failure isolation, and why the pipeline returns unsaved results for the caller to persist. Includes architecture, two-phase sequence, strategy, failure, and persistence diagrams, cross-linked to the existing lifecycle, hygiene, and events notes. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

The processChunk KDoc ended with an internal planning-item tag that shouldn't ship in committed code. The sentence reads the same without it. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…sign notes Reword the extraction pipeline note to describe its two stages by name (extraction and resolution) rather than numbered phases, so it reads as a self-contained architecture overview instead of a sequence of delivery steps. Also surface the design notes from the README, which previously linked none of them, so the conceptual docs are discoverable from the entry point. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Replace numbered "Phase 1/2/3/4" comments and N-phase pipeline phrasing with descriptive stage wording across the extraction pipeline and maintenance orchestrator, and rename the pipeline's private extractPhase/resolvePhase/ ExtractionPhaseResult to the stage spelling the design note already uses. The mark-and-sweep collector keeps its canonical mark phase / sweep phase terms, which name an algorithm rather than a delivery step. No behaviour change. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…ad-safety contracts The change-volume gate in DefaultDreamLoopOrchestrator did a check-then-act on lastActiveCount, so two triggers for the same context could both pass the gate on the same stale count and run overlapping cycles. consolidate/consolidateNow now hold a per-context lock (reentrant; different contexts still run in parallel). The pipeline's per-chunk failure map is a ConcurrentHashMap rather than a HashMap guarded by an external lock with an undocumented post-join read. Also document the thread-safety contracts surfaced by an audit: InMemoryEntityResolver is single-run-scoped and not thread-safe; DefaultCollectorRunner is safe across contexts but double-sweeps a context called concurrently; the maintenance orchestrator's per-step persistence is non-transactional but self-heals on the next idempotent run; and dream-loop copy() isolation comes from the field initializer. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

The abstraction, contradiction-resolution, and session-consolidation passes did real work but emitted nothing of their own, and the default sweep policy skipped pinned/unmarked propositions silently. Add debug logging per pass summarizing what it decided (counts, per context) and trace logging for each sweep decision, so a maintenance cycle is observable end to end rather than only at the orchestrator. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…d strategy subsystems The AGENTS navigation guides predated the extraction-strategy, gate, collector, dream-loop, and consolidation-pass subsystems. Add the missing package rows and a knowledge-hygiene orientation section (admission -> reclamation -> consolidation), and point both guides at the design notes, so the pipelines are navigable end to end. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…deep dives The knowledge-hygiene note explains why admission, reclamation, and consolidation exist; these two new notes explain how the latter two are built so a newcomer can follow them end to end. Consolidation-and-dream-loop covers the ConsolidationPass abstraction, the four passes, cycle composition, and threshold-gated/locked triggering. Reclamation-and-collector covers the mark strategies (including the union-find duplicate detection), the sweep policy, collect/dry-run/live entry points, and the audit trail. Cross-linked from knowledge-hygiene and the README; all diagrams parse. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…amation flow Add the gaps the subsystems lacked: two concurrency regression tests proving the dream-loop per-context lock serializes same-context cycles while letting different contexts run in parallel; a direct unit test for DecayStatusPolicy.evaluate (the hysteresis band and the importance/reinforce utility weights, which only config defaults exercised before); and a no-mocks end-to-end test that runs a real decay sweep through the orchestrator, collector, store, events, and audit trail together. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

SweepPolicy/StatusTransitionSweepPolicy and their SweepAction/PropositionMark/ MarkReason vocabulary move from projection.memory into com.embabel.dice.spi, sitting alongside StatusTransitionPolicy. Deciding a marked proposition's lifecycle fate is a policy seam, the same kind of plug-point as the trust, authority, conflict, and status policies already in spi. The cluster depends only on the core proposition model, so spi keeps its 'depends on core only' layering; the collector in projection.memory now imports these from spi (the correct feature -> spi direction). Pure move, no behaviour change. Also relocate StatusTransitionSweepPolicyTest into the spi test package to match the DecayStatusPolicyTest convention. Note: moving these public types is a source-incompatible change for any external importer of the old package (acceptable pre-1.0). Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

…'t replace Sources were marked SUPERSEDED unconditionally. Two cases lost facts: when every candidate abstraction exceeded maxLevel (no replacement at all), and when a source was pinned (eviction-immune). Now skip a group whose abstractions are all over the cap, and never supersede a pinned source — keep it ACTIVE alongside the abstraction. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Cover MarkReason.Custom (consumer key/description flowing through a mark), CollectorRecordStore.findRun (hit and miss), and the DreamLoopReport data-class contract (totals carry through, cycleCompleted defaults to construction time). Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

Make Proposition.pinned a real 'must retain' guarantee end to end: the decay collector strategy never marks a pinned proposition, contradiction resolution never auto-retires one, and the reviser keeps a pinned original intact on contradiction (leaving the clash for explicit resolution) instead of demoting it. Adds a pinned filter to PropositionQuery and pin/unpin/findPinned helpers on PropositionStore. Decay-status and sweep-policy already exempted pinned; this closes the remaining eviction paths. Budget-priority injection of pinned propositions remains with the token-budget work (embabel#7). Closes embabel#9 Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

jasperblues

Reviewed the maintenance + extraction subsystems closely (consolidation/dream-loop, mark-sweep reclamation, gates, the pinning lifecycle). Excellent structure overall — the pluggable-policy decomposition is clean and the test coverage is broad. A few findings cluster around one theme worth calling out: several "skip / protect / keep" branches drop information with no durable trace, which makes them hard to observe and, in two cases, means a feature silently doesn't run. Inline comments below; five come with a ready-to-run failing test (each encodes the intended contract, so it flips green once fixed). Happy to push these as a single Pr48FindingsTest.kt commit if useful — just say the word.

Tighten the "skip / protect / keep" branches the review flagged for dropping information without a durable trace: - Dream-loop trigger: stop advancing the change-volume baseline on a skipped call so growth accumulates since the last cycle, not the last call (a steadily-growing context was never consolidating). - Reconcile cross-pass saves by id with a defined status precedence, so a proposition retired by two passes in one cycle has a deterministic final status instead of one decided by saveAll ordering. - Count deletes as transitions only when allowHardDelete actually applies them. - Dedupe abstraction's superseded sources by id (pass and maintenance twin) so a source mentioning two entities is retired once. - Collector: persist the partial audit trail on a mid-run failure, and reorder to persist -> record -> emit so a throwing listener can't leave a transition recorded-but-not, persisted-but-unrecorded. - Surface a PropositionRoutedToReview signal when a contradiction lands on a pinned fact, on both the ingest and consolidation paths, instead of swallowing it silently. - Route the REST file-extract endpoint through process() so one bad chunk yields partial results instead of failing the whole upload and the execution strategies are actually exercised. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

jasperblues

Re-ran the seven findings against 1f1e8d2 — all addressed and verified green: the five correctness contracts (threshold accumulation, cross-pass reconciliation, transition counting, abstraction dedup, mid-run audit durability) now pass, the REST handlers route through process(), and pinned contradictions surface a PropositionRoutedToReview signal instead of being silently dropped. Clean, well-tested fixes. 🚀

jimador mentioned this pull request Jun 19, 2026

Graph backend + retrieval #49

Merged

jimador requested review from jasperblues and johnsonr June 19, 2026 12:26

jimador force-pushed the feat/extraction-and-maintenance branch 6 times, most recently from 2ec057b to eb0b481 Compare June 19, 2026 19:53

jimador added 7 commits June 22, 2026 22:14

jimador force-pushed the feat/extraction-and-maintenance branch from 6961019 to 8c5421b Compare June 23, 2026 02:24

jimador marked this pull request as ready for review June 23, 2026 02:28

jimador added 12 commits June 22, 2026 22:58

docs(pipeline): drop a stray planning-item reference from a KDoc

e3f507b

The processChunk KDoc ended with an internal planning-item tag that shouldn't ship in committed code. The sentence reads the same without it. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>

jimador added 7 commits June 22, 2026 23:48

jasperblues reviewed Jun 24, 2026

View reviewed changes

jimador requested a review from jasperblues June 24, 2026 06:06

jasperblues approved these changes Jun 24, 2026

View reviewed changes

jimador merged commit 13a6068 into embabel:main Jun 24, 2026
4 checks passed

jimador deleted the feat/extraction-and-maintenance branch June 24, 2026 06:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Extraction concurrency + maintenance - #48

Extraction concurrency + maintenance#48
jimador merged 27 commits into
embabel:mainfrom
jimador:feat/extraction-and-maintenance

jimador commented Jun 19, 2026 •

edited

Loading

Uh oh!

jasperblues left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasperblues left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jimador commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!