Populate provenanceEntries during proposition extraction and revision

[LordKay-sudo]

Problem

DICE has a rich provenance model (ProvenanceEntry, SourceLocator) and Proposition.provenanceEntries, but the extraction pipeline never populates it. Every proposition extracted via PropositionPipeline ends up with provenanceEntries = []; only the legacy grounding: List<String> chunk-ID list is set.

Downstream consumers (REST API, Neo4j projection, MCP tooling, provenance export) cannot link propositions back to source material via URI, file path, content hash, or character offsets — even though the data model supports it.

Evidence

• SuggestedProposition.toProposition() sets grounding only — no provenanceEntries
• LlmPropositionExtractor.resolvePropositions() calls toProposition() without provenance wiring
• LlmPropositionReviser.mergePropositions() / reinforceProposition() merge grounding but not provenanceEntries
• PropositionDto (REST) exposes grounding but not provenanceEntries
• Proposition KDoc describes provenanceEntries as complementing the legacy grounding list

Unit tests exist for ProvenanceEntry / SourceLocator and manual withProvenanceEntries() — but no integration test proves extraction populates them.

Proposed solution

1. ProvenanceFactory (or similar) — build ProvenanceEntry from Chunk + optional SourceAnalysisContext fields:

   • chunkId from chunk
   • contentHash from ContentHasher (already used in processOnce)
   • SourceLocator from chunk metadata / sourceId (e.g. ConnectorRef for email:<threadId>, UriLocator when URI known)

2. Wire into extraction — SuggestedProposition.toProposition() or resolvePropositions() attaches provenance entries alongside grounding

3. Wire into revision — mergePropositions / reinforceProposition union provenanceEntries (like grounding)

4. Extend SourceAnalysisContext (optional) — carry a default SourceLocator or provenance template for the current analysis run

5. REST — add provenanceEntries to PropositionDto (or a slim DTO)

6. Tests — pipeline integration test asserting extracted propositions have non-empty provenanceEntries when chunk metadata is present

Acceptance criteria

• Extracted propositions include at least one ProvenanceEntry with chunkId when processed via processChunk
• processOnce entries include contentHash when hasher is used
• Revision merge/reinforce accumulates provenance entries without duplicates
• grounding list unchanged (backward compatible)
• PropositionDto exposes provenance (or documented alternative)
• Tests cover extraction + revision paths

Related

• Proposition provenance metadata #7 — extraction context metadata (speaker, turn, mode) — complementary, not duplicate
• com.embabel.dice.provenance package (already implemented)
• Specs/blog claim "full provenance tracking" — this closes the pipeline gap

Motivation

Building MCP tooling and knowledge-graph apps where users need to verify where a proposition came from, not just which chunk ID grounded it. Happy to contribute a PR if this direction looks right.

[LordKay-sudo]

Looks like the provenance types are there but nothing in the pipeline actually fills them in — I kept hitting empty provenanceEntries while wiring MCP export on our side. Happy to take a pass at this if the approach above makes sense.

[LordKay-sudo]

I had a look at main after #49 landed: extraction still doesn't populate provenanceEntries. I can put together a PR if that's still wanted.

[jimador]

@LordKay-sudo thanks for that catch. #49 hasn't merged yet. Fixed on the #49 branch. processChunk now stamps chunkId and contentHash, with the locator coming from SourceAnalysisContext.sourceLocator or a ContentAddressedLocator fallback. The reviser unions them on merge/reinforce. You can access via PropositionDto.

[LordKay-sudo]

Ah ok, I was testing on main and assumed #49 had already landed. That explains the empty provenanceEntries.

I'll pull the #49 branch and re-check my MCP export against PropositionDto. Closing #51 on my side so we're not duplicating the same work.