Extraction concurrency + maintenance

AGENTS.md

@@ -48,12 +48,14 @@ The `dice` module is organized by responsibility:
 | `com.embabel.dice.proposition.extraction` | `PropositionExtractor` SPI, `LlmPropositionExtractor` |
 | `com.embabel.dice.proposition.revision` | `PropositionReviser` SPI, `LlmPropositionReviser` |
 | `com.embabel.dice.proposition.store` | In-memory and JSON-file implementations of `PropositionRepository` and `DecayManager` |
-| `com.embabel.dice.pipeline` | `PropositionPipeline` — the orchestrating entry point that chains extractor → entity resolver → reviser |
-| `com.embabel.dice.spi` | Policy extension points consumers plug into: `TrustScorer`, `AuthorityResolver`/`AuthorityTier`, `AuthorityWeightedTrustScorer`, `ConflictDetector`, `ConflictType`, `StatusTransitionPolicy`, and their shipped defaults |
+| `com.embabel.dice.proposition.gate` | Admission gates that route pipeline output before persistence (`ExtractionGate`, `ExtractionGatePipeline`, `StandardGates`, `EvidenceFloorGate`) |
+| `com.embabel.dice.pipeline` | `PropositionPipeline` — the two-stage entry point (concurrent extraction → serial entity resolution); pluggable `ExtractionExecutionStrategy` (serial / parallel / batched) |
+| `com.embabel.dice.spi` | Policy extension points consumers plug into: `TrustScorer`, `AuthorityResolver`/`AuthorityTier`, `AuthorityWeightedTrustScorer`, `ConflictDetector`, `ConflictType`, `StatusTransitionPolicy`, the mark-and-sweep `SweepPolicy`/`StatusTransitionSweepPolicy` (with its `SweepAction`/`PropositionMark`/`MarkReason` vocabulary), and their shipped defaults |
 | `com.embabel.dice.common` | Shared types: `SourceAnalysisContext`, `EntityResolver`, `Relations`, `ContentHasher`, `KnowledgeType`, events, validation rules, mention filters, resolver implementations |
 | `com.embabel.dice.entity` | Entity-only pipeline (`EntityPipeline`, `EntityIncrementalAnalyzer`, `EntityResolutionService`, `EntityResolutionTools`) |
 | `com.embabel.dice.incremental` | `AbstractIncrementalAnalyzer`, `ChunkHistoryStore`, `ConversationSource`, windowed processing |
-| `com.embabel.dice.projection` | Projectors: `GraphProjector`, `PrologProjector`, `MemoryProjector`; lineage tracking (`ProjectionRecordStore`, `Reconciler`) |
+| `com.embabel.dice.projection` | Projectors (`GraphProjector`, `PrologProjector`, `MemoryProjector`); memory maintenance and the dream loop (`MemoryMaintenanceOrchestrator`, `DreamLoopOrchestrator`); the mark-and-sweep collector (`CollectorRunner`, `CollectorStrategy`; the `SweepPolicy` lives in `spi`) and its audit trail (`CollectorRecordStore`); lineage tracking (`ProjectionRecordStore`, `Reconciler`) |
+| `com.embabel.dice.operations` | Higher-level proposition operations (`PropositionAbstractor`, `PropositionContraster`) and, under `operations.consolidation`, the passes that compose the dream loop (`SessionConsolidationPass`, `AbstractionPass`, `ContradictionResolutionPass`, `DecaySweepPass`) |
 | `com.embabel.dice.text2graph` | Text-to-graph builder pipeline (`KnowledgeGraphBuilder`, `SourceAnalyzer`, entity merge/relationship resolution policies) |
 | `com.embabel.dice.provenance` | `ProvenanceEntry`, `SourceLocator` — rich evidence links from propositions back to source material |
 | `com.embabel.dice.query.oracle` | `Oracle`, `LlmOracle`, `PrologTools` — natural language question answering over propositions |
@@ -80,4 +82,7 @@ The `dice` module is organized by responsibility:
 - **Understanding the proposition data model** → `Proposition.kt` in `com.embabel.dice.proposition`. Every field is documented inline.
 - **Adding a new entity resolver strategy** → implement `CandidateSearcher` in `com.embabel.dice.common.resolver.searcher`, then compose it into an `EscalatingEntityResolver`.
 - **Writing integration tests against Neo4j** → look at `DrivinePropositionStoreIntegrationTest` in `dice-storage/src/test`; it shows the `@SpringBootTest` + Testcontainers pattern in use.
-- **Understanding *why* the system behaves as it does** → `docs/design/` holds the design-decision notes — the conceptual model and the reasoning you can't recover by reading a class (e.g. the proposition lifecycle: trust, authority, supersession, decay).
+- **Tuning what gets into the store** → admission gates in `com.embabel.dice.proposition.gate` (`ExtractionGatePipeline`, `StandardGates`); they run on pipeline output before the caller persists.
+- **Running maintenance / consolidation** → `DefaultDreamLoopOrchestrator` (threshold-gated consolidation passes) or `DefaultMemoryMaintenanceOrchestrator` (the legacy four-step pipeline), both in `com.embabel.dice.projection.memory`.
+- **Reclaiming stale or duplicate propositions** → `DefaultCollectorRunner` and its `CollectorStrategy` in `com.embabel.dice.projection.memory` (the `SweepPolicy` that decides each fate lives in `com.embabel.dice.spi`); runs are auditable via `CollectorRecordStore`.
+- **Understanding *why* the system behaves as it does** → [`docs/design/`](docs/design/) holds the design-decision notes — the conceptual model and the reasoning you can't recover by reading a class: the extraction pipeline, the proposition lifecycle (trust, authority, supersession, decay), knowledge hygiene (gates, reclamation, consolidation), and the event model.

README.md

@@ -91,6 +91,23 @@ flowchart TB
     style Projections fill:#d4f5d4,stroke:#3fd73c,color:#1e1e1e
 ```
 
+### Design notes
+
+For the reasoning behind how DICE behaves — the conceptual model and the decisions you can't
+recover by reading a single class — see the design notes in [`docs/design/`](docs/design/):
+
+- [Extraction pipeline](docs/design/extraction-pipeline.md) — the two-stage extract/resolve flow,
+  optional concurrency, and why the pipeline returns unsaved results for the caller to persist.
+- [Proposition lifecycle](docs/design/proposition-lifecycle.md) — trust scoring, source authority,
+  conflict classification, supersession versus contradiction, and decay instead of deletion.
+- [Knowledge hygiene](docs/design/knowledge-hygiene.md) — admission gates, mark-and-sweep
+  reclamation, and the consolidation dream loop (the "why" overview of the three interventions).
+- [Consolidation and the dream loop](docs/design/consolidation-and-dream-loop.md) — the pass
+  abstraction, the four consolidation passes, and how a cycle composes and is triggered.
+- [Reclamation and the collector](docs/design/reclamation-and-collector.md) — the mark-and-sweep
+  internals: strategies, sweep policy, dry-run vs. live, and the audit trail.
+- [Events](docs/design/events.md) — the domain-event model the store and pipeline emit.
+
 ## Real-World Example: Impromptu
 
 **[Impromptu](https://github.com/embabel/impromptu)** is a classical music exploration chatbot that uses DICE

dice/AGENTS.md

@@ -33,6 +33,18 @@ Kotlin 2.1.10, Java 21. `embabel-agent-api` and `embabel-agent-rag-core` are `pr
 
 **`SourceAnalysisContext`** (`com.embabel.dice.common.SourceAnalysisContext`) — carries everything a pipeline run needs: schema (`DataDictionary`), entity resolver, `contextId`, optional known entities and template model. In Kotlin use infix builders; in Java use the `withXxx` builder chain.
 
+## Knowledge hygiene: admission, reclamation, consolidation
+
+Three seams keep the store healthy at three different moments; all ship conservative, pluggable defaults.
+
+**Admission — gates.** `ExtractionGate` (`com.embabel.dice.proposition.gate`) runs on pipeline output *before* the caller persists. `ExtractionGatePipeline` chains gates; the first non-`Persist` decision wins. `StandardGates` ships confidence/trust/conflict/evidence-floor gates; `EvidenceFloorGate` demotes a weak relation rather than dropping the fact.
+
+**Reclamation — the mark-and-sweep collector.** `CollectorRunner`/`DefaultCollectorRunner` (`com.embabel.dice.projection.memory`) fetch ACTIVE candidates, gather marks from each `CollectorStrategy` (stale via `DecayCollectorStrategy`, duplicates via `DuplicateCollectorStrategy`), then let a `SweepPolicy` (in `spi` with `StatusTransitionPolicy`, since deciding a lifecycle fate is a policy seam) decide each one's fate (`TransitionStatus`/`HardDelete`/`Skip`). `run(dryRun = true)` previews without writing; `collect()` marks only. Every run is recorded in a `CollectorRecordStore` for audit.
+
+**Consolidation — the dream loop.** `DreamLoopOrchestrator`/`DefaultDreamLoopOrchestrator` run `ConsolidationPass`es (`operations.consolidation`) as a threshold-gated cycle: `SessionConsolidationPass`, `AbstractionPass` (drives supersession), `ContradictionResolutionPass` (drives contradiction), `DecaySweepPass` (drives stale). The orchestrator owns all writes and locks per `contextId` so cycles for one context never overlap.
+
+These map onto the lifecycle in [proposition-lifecycle](../docs/design/proposition-lifecycle.md); the design notes in [`docs/design/`](../docs/design/) explain the *why* behind each.
+
 ## Package map
 
 | Package | Contents |
@@ -41,8 +53,9 @@ Kotlin 2.1.10, Java 21. `embabel-agent-api` and `embabel-agent-rag-core` are `pr
 | `proposition.extraction` | `PropositionExtractor`, `LlmPropositionExtractor`, extraction config |
 | `proposition.revision` | `PropositionReviser`, `LlmPropositionReviser` |
 | `proposition.store` | `InMemoryPropositionRepository`, `JsonFilePropositionRepository`, `InMemoryDecayManager` |
-| `pipeline` | `PropositionPipeline`, `PropositionResults`, persistence helpers |
-| `spi` | Policy extension points: `TrustScorer`, `AuthorityResolver`/`AuthorityTier`, `AuthorityWeightedTrustScorer`, `ConflictDetector`, `ConflictType`, `StatusTransitionPolicy`, and their shipped defaults |
+| `proposition.gate` | Admission gates applied to pipeline output before persistence: `ExtractionGate`, `ExtractionGatePipeline`, `StandardGates`, `ObservableGate`, `GateDecision`, `GatedPropositionResult`, `EvidenceFloorGate` |
+| `pipeline` | `PropositionPipeline` (two-stage extract→resolve), `PropositionResults`, `PersistablePropositions`; execution strategies: `ExtractionExecutionStrategy` + `SerialExtractionStrategy`/`ParallelExtractionStrategy`/`BatchedExtractionStrategy` |
+| `spi` | Policy extension points: `TrustScorer`, `AuthorityResolver`/`AuthorityTier`, `AuthorityWeightedTrustScorer`, `ConflictDetector`, `ConflictType`, `StatusTransitionPolicy`, the mark-and-sweep `SweepPolicy`/`StatusTransitionSweepPolicy` with its `SweepAction`/`PropositionMark`/`MarkReason` vocabulary, and their shipped defaults |
 | `common` | `SourceAnalysisContext`, `EntityResolver`, `Relations`, `ContentHasher`, `KnowledgeType`, events, `SchemaAdherence`, validation rules |
 | `common.filter` | `MentionFilter`, `SchemaValidatedMentionFilter`, `ObservableMentionFilter`, context-aware filters |
 | `common.resolver` | `EscalatingEntityResolver`, `InMemoryEntityResolver`, `KnownEntityResolver`, `ChainedEntityResolver`, `LlmCandidateBakeoff`, `ContextCompressor` |
@@ -51,9 +64,9 @@ Kotlin 2.1.10, Java 21. `embabel-agent-api` and `embabel-agent-rag-core` are `pr
 | `incremental` | `AbstractIncrementalAnalyzer`, `ChunkHistoryStore`, `InMemoryChunkHistoryStore`, `ConversationSource`, `IncrementalSource` |
 | `incremental.proposition` | `PropositionIncrementalAnalyzer` |
 | `projection.graph` | `GraphProjector`, `RelationBasedGraphProjector`, `LlmGraphProjector`, `GraphProjectionService`, `ProjectionPolicy` |
-| `projection.memory` | `MemoryProjector`, `MemoryRetriever`, `MemoryConsolidator`, `MemoryMaintenanceOrchestrator`, `DefaultMemoryProjector` |
+| `projection.memory` | `MemoryProjector`/`DefaultMemoryProjector`, `MemoryRetriever`, `MemoryConsolidator`; **maintenance & dream loop**: `MemoryMaintenanceOrchestrator`/`DefaultMemoryMaintenanceOrchestrator`, `DreamLoopOrchestrator`/`DefaultDreamLoopOrchestrator`, `DreamLoopReport`; **mark-and-sweep collector**: `CollectorRunner`/`DefaultCollectorRunner`, `CollectorStrategy` (`DecayCollectorStrategy`, `DuplicateCollectorStrategy`), `CollectorRunResult` — the `SweepPolicy` and the `SweepAction`/`PropositionMark`/`MarkReason` vocabulary live in `spi` |
 | `projection.prolog` | `PrologProjector`, `PrologEngine`, Prolog type wrappers |
-| `projection.lineage` | `ProjectionRecordStore`, `InMemoryProjectionRecordStore`, `Reconciler`, `ProjectionRecord`, `ProjectionRun` |
+| `projection.lineage` | `ProjectionRecordStore`, `InMemoryProjectionRecordStore`, `Reconciler`, `ProjectionRecord`, `ProjectionRun`; **collector audit trail**: `CollectorRecordStore`/`InMemoryCollectorRecordStore`, `CollectorRecord`, `CollectorRun`, `CollectorOutcome` |
 | `projection.grounding` | `GroundingResolver`, `GroundingWiringService` |
 | `text2graph` | `KnowledgeGraphBuilder`, `SourceAnalyzer`, `LlmSourceAnalyzer`, `MultiPassKnowledgeGraphBuilder`, merge policies, relationship resolution |
 | `provenance` | `ProvenanceEntry`, `SourceLocator`, `UriLocator` |
@@ -62,6 +75,7 @@ Kotlin 2.1.10, Java 21. `embabel-agent-api` and `embabel-agent-rag-core` are `pr
 | `agent` | `Memory`, `MemoryRetriever` (agent-facing view), `ProvenanceResolver` |
 | `web.rest` | `PropositionPipelineController`, `MemoryController`, API key security — optional, activated by `spring-webmvc` |
 | `operations` | `PropositionAbstractor`, `PropositionContraster` — higher-level proposition management |
+| `operations.consolidation` | The dream-loop steps as composable passes: `ConsolidationPass`/`ConsolidationPassResult`, `SessionConsolidationPass`, `AbstractionPass`, `ContradictionResolutionPass`, `DecaySweepPass` |
 
 ## Gotchas
 

dice/src/main/kotlin/com/embabel/dice/common/DiceEvent.kt

@@ -240,6 +240,22 @@ data class PropositionProjectionSkipped @JvmOverloads constructor(
     override val timestamp: Instant = Instant.now(),
 ) : DiceEvent
 
+/**
+ * A proposition's relation was demoted to a weaker one because it didn't have enough
+ * supporting evidence — kept and downgraded rather than thrown away.
+ *
+ * @property proposition The proposition whose relation was demoted.
+ * @property toRelation The weaker relation it was demoted to.
+ * @property reason Why it was demoted.
+ * @property timestamp When the event was created.
+ */
+data class PropositionDemoted @JvmOverloads constructor(
+    val proposition: Proposition,
+    val toRelation: String,
+    val reason: String,
+    override val timestamp: Instant = Instant.now(),
+) : DiceEvent
+
 /**
  * Implement this to react to [DiceEvent]s as they happen. Keep in mind handlers run inline
  * on the emitting thread, so do anything slow off to the side.

dice/src/main/kotlin/com/embabel/dice/common/EvidenceFloor.kt

@@ -0,0 +1,87 @@
+/*
+ * Copyright 2024-2026 Embabel Pty Ltd.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.embabel.dice.common
+
+import com.embabel.dice.spi.AuthorityTier
+
+/**
+ * The minimum evidence a proposition must carry before a relation may be asserted at full strength.
+ *
+ * This is the knob that stops a cheap structural signal — two people sharing an email domain, a
+ * single calendar co-attendance — from being promoted to a strong semantic claim like "works for"
+ * or "collaborates with". A relation can declare a floor on two independent dimensions, and an
+ * assertion has to clear both:
+ *
+ * - [minConfidence]: the proposition's decay-adjusted confidence must reach this value.
+ * - [minAuthority]: the proposition's source must be at least this authoritative (see [AuthorityTier],
+ *   where a *lower* ordinal means *more* authoritative).
+ *
+ * Either dimension may be left null, in which case it is simply not checked; a floor with both null
+ * never blocks anything. When an assertion falls short, [demoteTo] names a weaker predicate to fall
+ * back to (for example "works for" → "affiliated with"). If [demoteTo] is null the assertion is held
+ * for review instead.
+ *
+ * DICE only carries and checks the floor — it never decides what the threshold values should be or
+ * what "weaker" means for a given predicate. Those are the consumer's to declare on each [Relation].
+ *
+ * @property minConfidence Lowest decay-adjusted confidence that clears the floor, in `0.0..1.0`;
+ *   null to skip the confidence check.
+ * @property minAuthority Weakest source authority that clears the floor; null to skip the authority
+ *   check.
+ * @property demoteTo Predicate to fall back to when the floor is not met; null to hold for review
+ *   instead of demoting.
+ * @throws IllegalArgumentException if [minConfidence] is outside `0.0..1.0`, or if [demoteTo] is blank.
+ */
+data class EvidenceFloor @JvmOverloads constructor(
+    val minConfidence: Double? = null,
+    val minAuthority: AuthorityTier? = null,
+    val demoteTo: String? = null,
+) {
+
+    init {
+        require(minConfidence == null || minConfidence in 0.0..1.0) {
+            "minConfidence must be in 0.0..1.0, was $minConfidence"
+        }
+        require(demoteTo == null || demoteTo.isNotBlank()) { "demoteTo must not be blank" }
+    }
+
+    /**
+     * Does an assertion with this [confidence] and source [authority] clear the floor?
+     *
+     * A null bound on either dimension passes that dimension. Authority clears when it is at least
+     * as strong as [minAuthority], i.e. its ordinal is less than or equal to the floor's.
+     */
+    fun isSatisfiedBy(confidence: Double, authority: AuthorityTier): Boolean {
+        val confidenceOk = minConfidence == null || confidence >= minConfidence
+        val authorityOk = minAuthority == null || authority.ordinal <= minAuthority.ordinal
+        return confidenceOk && authorityOk
+    }
+
+    companion object {
+
+        /** A floor on confidence alone. */
+        @JvmStatic
+        @JvmOverloads
+        fun ofConfidence(minConfidence: Double, demoteTo: String? = null): EvidenceFloor =
+            EvidenceFloor(minConfidence = minConfidence, demoteTo = demoteTo)
+
+        /** A floor on source authority alone. */
+        @JvmStatic
+        @JvmOverloads
+        fun ofAuthority(minAuthority: AuthorityTier, demoteTo: String? = null): EvidenceFloor =
+            EvidenceFloor(minAuthority = minAuthority, demoteTo = demoteTo)
+    }
+}