diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f8d4dc..742c356 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,55 @@ # Changelog +## Unreleased + +## v0.12.0 (2026-08-11) + +SDK gaps closed for hosts that need full-file re-ingest, tomb-free similarity +indexes, and mixed doc+chat retrieval under tight budgets. + +### Added + +- **`replaceSource(sourceFile, docs, opts?)` / `replaceMarkdown`.** Soft-retires + every active node for the source with `retiredBy: 'replace'`, drops their + TF-IDF (and embedding) postings, then appends through the additive path while + exempting the just-retired generation from dedup. Ordinary later `append` of + the old body cannot restore replace tombs. Optional `generation` stamp for + mixed-vintage diagnostics. + +- **`forgetSource(sourceFile)`.** Soft-retire a whole source (`retiredBy: + 'delete'`) and drop its similarity postings — the SDK primitive that lets + hosts drop content-erasing `__gn-forgotten:` tomb workarounds. + +- **`sourcesNeedingChunkerReingest()` / `CURRENT_CHUNKER_ID`.** Lists live + sources whose content nodes lack the `boundaries-v1` stamp so App hosts can + offer opt-in `replaceSource` for pre-0.11.1 shattered chunk boundaries. + Stored cortex is never rewritten in place. + +- **Mixed-corpus document floor** (`MIXED_CORPUS_DOC_FLOOR_SHARE = 0.25`) in + final traversal selection, and a **structural doc reserve split** + (`STRUCTURAL_DOC_RESERVE_SHARE = 0.5`) when turn-pairs and document neighbors + both compete for the post-cut expansion budget. + +### Changed + +- **TF-IDF excludes administrative tombs.** `buildIndexFromGraph` / + `rebuildIndex` skip retired nodes; `deleteNode`, supersede, bulk forget, + cascade, `forgetSource`, and `replaceSource` call `removeDocument` so tombs + neither seed nor dilute IDF. + +- **`RetirementReason` includes `'replace'`.** Audit health reports a third + bucket. `blocksReingest` treats replace like supersede for ordinary append. + +### Tests + +- `tests/unit/replace-source.test.ts` +- `tests/unit/tfidf-tomb-index.test.ts` +- `tests/unit/mixed-corpus-starvation.test.ts` +- `tests/unit/chunker-migration.test.ts` + ## v0.11.1 (2026-08-07) + Chunker correctness: sentence boundaries stop tearing identifiers, every chunk is hard-capped, short pieces merge instead of vanishing, and the suite that would have caught all three is wired into `npm test`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 95e2ec2..006bb9b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -18,12 +18,12 @@ These belong in `@nehloo/graphnosis`: - **Ingestion parsers.** Markdown, HTML, JSON, CSV, PDF, plain text. New parsers for common document formats are welcome via issue discussion. - **Indexing.** TF-IDF index with pluggable analyzers, in-memory embedding index with pluggable adapters. - **Querying.** `query()` (TF-IDF), `queryHybrid()` (TF-IDF + embeddings), `prompt()` builders, subgraph context serialization. -- **Corrections.** `edit`, `deleteNode`, `supersede`, `correct`, `importMarkdown`, `forgetByTopic`, `forgetByTimeWindow`, `previewForgetTopic`, `retired`. Soft-delete semantics. `edit` supersedes rather than overwriting — the prior version stays readable. +- **Corrections.** `edit`, `deleteNode`, `supersede`, `correct`, `importMarkdown`, `forgetByTopic`, `forgetByTimeWindow`, `previewForgetTopic`, `forgetSource`, `replaceSource`, `retired`. Soft-delete semantics. `edit` supersedes rather than overwriting — the prior version stays readable. `replaceSource` is the full-file re-ingest primitive. - **Confidence.** `setConfidence` / `setConfidences` — change how much a memory counts without minting or retiring anything. - **Persistence.** The `.gai` binary format (MessagePack body, big-endian header, checksum, optional HMAC signing) as specified in [`SPEC.md`](SPEC.md), SQLite store, buffer-based I/O for serverless. - **Determinism.** `asOf` — one caller-supplied instant for a whole query, so the same question against an unchanged graph returns the same answer indefinitely. - **Failure classification.** Stable error codes and classes (`GraphnosisError`, `isCorruption`, `isVersionSkew`, `isCallerError`) so a consumer can branch on the class instead of matching message text. -- **Analyzer migration.** `migrateAnalyzer` — move an existing index to a different analyzer deliberately, with a count of the terms recovered. +- **Analyzer migration.** `migrateAnalyzer` — move an existing index to a different analyzer deliberately, with a count of the terms recovered. `sourcesNeedingChunkerReingest` / `CURRENT_CHUNKER_ID` — list sources that still need an opt-in `replaceSource` after the 0.11.1 chunker fix. - **Reflection.** `reflect()` — contradictions, decayed nodes, surprising connections. - **Adapter interfaces.** `EmbeddingAdapter`, `TextAnalyzer`. New built-in adapters for major providers (OpenAI, Voyage, Cohere) live in `@nehloo/graphnosis/adapters/*`. - **Federation primitive.** `queryGraphs([...])` for in-process cross-graph queries. diff --git a/package-lock.json b/package-lock.json index 154bb75..b0e2779 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@nehloo/graphnosis", - "version": "0.11.1", + "version": "0.12.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@nehloo/graphnosis", - "version": "0.11.1", + "version": "0.12.0", "license": "Apache-2.0", "dependencies": { "cheerio": "^1.1.0", diff --git a/package.json b/package.json index 42e2a80..c35e75d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@nehloo/graphnosis", - "version": "0.11.1", + "version": "0.12.0", "description": "AI-native dual-graph knowledge representation — build, query, and persist typed knowledge graphs in-process.", "author": "Nehloo Interactive LLC", "license": "Apache-2.0", @@ -58,7 +58,7 @@ "longmemeval:audit:multisession": "tsx tests/longmemeval/official/audit-multisession.ts", "mcp": "tsx src/mcp/server.ts", "mcp:http": "MCP_TRANSPORT=http tsx src/mcp/server.ts", - "test": "tsx tests/unit/synonym-expander.test.ts && tsx tests/unit/traversal-determinism.test.ts && tsx tests/unit/query-asof-determinism.test.ts && tsx tests/unit/traversal-path-maximum.test.ts && tsx tests/unit/correction-affected-ids.test.ts && tsx tests/unit/error-codes.test.ts && tsx tests/unit/confidence-primitive.test.ts && tsx tests/unit/embedding-cosine-bounded.test.ts && tsx tests/unit/ingest-determinism.test.ts && tsx tests/unit/loader-index-parity.test.ts && tsx tests/unit/seed-budget.test.ts && tsx tests/unit/morphology-key.test.ts && tsx tests/unit/derived-index.test.ts && tsx tests/unit/graph-integrity.test.ts && tsx tests/unit/chunk-boundaries.test.ts && tsx tests/unit/source-update-integrity.test.ts && tsx tests/unit/reflect-no-decay.test.ts && tsx tests/unit/ingest-path-parity.test.ts && tsx tests/unit/structural-expansion.test.ts && tsx tests/unit/structural-expansion-wiring.test.ts && tsx tests/unit/federated-subgraph-scores.test.ts && tsx tests/unit/retirement.test.ts && tsx tests/unit/retirement-expiry.test.ts && tsx tests/unit/legacy-retirement-v080.test.ts && tsx tests/unit/legacy-tomb-write-path.test.ts && tsx tests/unit/supersedes-edge-liveness.test.ts && tsx tests/ablation-scoring/maxwins-vs-additive.ts && node spec/conformance.mjs", + "test": "tsx tests/unit/synonym-expander.test.ts && tsx tests/unit/traversal-determinism.test.ts && tsx tests/unit/query-asof-determinism.test.ts && tsx tests/unit/traversal-path-maximum.test.ts && tsx tests/unit/correction-affected-ids.test.ts && tsx tests/unit/error-codes.test.ts && tsx tests/unit/confidence-primitive.test.ts && tsx tests/unit/embedding-cosine-bounded.test.ts && tsx tests/unit/ingest-determinism.test.ts && tsx tests/unit/loader-index-parity.test.ts && tsx tests/unit/seed-budget.test.ts && tsx tests/unit/morphology-key.test.ts && tsx tests/unit/derived-index.test.ts && tsx tests/unit/graph-integrity.test.ts && tsx tests/unit/chunk-boundaries.test.ts && tsx tests/unit/source-update-integrity.test.ts && tsx tests/unit/reflect-no-decay.test.ts && tsx tests/unit/ingest-path-parity.test.ts && tsx tests/unit/structural-expansion.test.ts && tsx tests/unit/structural-expansion-wiring.test.ts && tsx tests/unit/federated-subgraph-scores.test.ts && tsx tests/unit/retirement.test.ts && tsx tests/unit/retirement-expiry.test.ts && tsx tests/unit/legacy-retirement-v080.test.ts && tsx tests/unit/legacy-tomb-write-path.test.ts && tsx tests/unit/supersedes-edge-liveness.test.ts && tsx tests/unit/replace-source.test.ts && tsx tests/unit/tfidf-tomb-index.test.ts && tsx tests/unit/mixed-corpus-starvation.test.ts && tsx tests/unit/chunker-migration.test.ts && tsx tests/ablation-scoring/maxwins-vs-additive.ts && node spec/conformance.mjs", "verify:package": "node scripts/verify-package.mjs" }, "peerDependencies": { diff --git a/src/core/audit/audit-exporter.ts b/src/core/audit/audit-exporter.ts index 0bfc833..2f83ade 100644 --- a/src/core/audit/audit-exporter.ts +++ b/src/core/audit/audit-exporter.ts @@ -84,7 +84,7 @@ export interface HealthReport { expiredNodes: number; /** Retired nodes split by why, from `metadata.retiredBy`. Graphs written * before that field existed report their nodes under `delete`. */ - retiredByReason: { delete: number; supersede: number }; + retiredByReason: { delete: number; supersede: number; replace: number }; enrichedNodes: number; unenrichedNodes: number; nodesByType: Record; @@ -293,7 +293,7 @@ function generateHealthReport(graph: KnowledgeGraph): HealthReport { let orphanNodes = 0; let lowConfidenceNodes = 0; let expiredNodes = 0; - const retiredByReason = { delete: 0, supersede: 0 }; + const retiredByReason = { delete: 0, supersede: 0, replace: 0 }; const supersededIds = collectSupersededIds(graph); let enrichedNodes = 0; let unenrichedNodes = 0; @@ -380,6 +380,7 @@ export function auditToMarkdown(report: AuditReport, graphName: string): string lines.push(`| Retired Nodes (excluded from prompts, retained here) | ${report.health.expiredNodes} |`); lines.push(`| — forgotten | ${report.health.retiredByReason.delete} |`); lines.push(`| — superseded | ${report.health.retiredByReason.supersede} |`); + lines.push(`| — replaced | ${report.health.retiredByReason.replace} |`); lines.push(''); // Node type breakdown diff --git a/src/core/constants.ts b/src/core/constants.ts index 2f168ab..dc931a2 100644 --- a/src/core/constants.ts +++ b/src/core/constants.ts @@ -24,6 +24,21 @@ export const SEED_COUNT = 5; // Max seed nodes per query // the chat overlay's prior 123-node tail became a hard 45-node maximum. export const STRUCTURAL_EXPANSION_BUDGET_SHARE = 0.5; +// Mixed doc+chat corpora: fraction of the final `maxNodes` budget reserved for +// non-conversation content when BOTH classes appear in the scored candidate +// set. Measured on the M4 Alder fixture (one doc gold marker + 40-turn chat +// sharing query vocabulary): without a floor, maxNodes 6/8/10 returned 0 +// document content nodes while chat filled every slot. A 25% floor (at least +// one slot) recovers the marker under those budgets without inventing a wider +// BFS. Off when the candidate set is docs-only or chat-only. +export const MIXED_CORPUS_DOC_FLOOR_SHARE = 0.25; + +// When structural expansion offers both turn-pairs (priority 0) and document +// neighbors (priority 1), this share of the additions budget is reserved for +// documents so wide chat pairing cannot consume the entire post-cut reserve. +// Same measured failure mode as MIXED_CORPUS_DOC_FLOOR_SHARE, one stage later. +export const STRUCTURAL_DOC_RESERVE_SHARE = 0.5; + // Oversampling factor for per-variant lexical candidate generation. // // Each expanded query variant generates `SEED_COUNT * SEED_OVERSAMPLE` candidates diff --git a/src/core/corrections/correction-engine.ts b/src/core/corrections/correction-engine.ts index bbf0fb2..ad25108 100644 --- a/src/core/corrections/correction-engine.ts +++ b/src/core/corrections/correction-engine.ts @@ -9,7 +9,7 @@ import type { } from '@/core/types'; import { chunkDocument } from '@/core/extraction/chunker'; import { parseMarkdown } from '@/core/ingestion/parsers/markdown-parser'; -import { addDocument, computeIdf } from '@/core/similarity/tfidf'; +import { addDocument, computeIdf, removeDocument } from '@/core/similarity/tfidf'; import type { TfidfIndex } from '@/core/types'; import { buildDirectedEdges, chunkKey } from '@/core/graph/directed-edges'; import { extractEntities } from '@/core/extraction/entity-extractor'; @@ -66,7 +66,7 @@ export function applyCorrection( case 'add': return applyAdd(graph, tfidfIndex, correction); case 'delete': - return applyDelete(graph, correction); + return applyDelete(graph, tfidfIndex, correction); case 'supersede': return applySupersede(graph, tfidfIndex, correction); default: @@ -197,6 +197,7 @@ function applyAdd( // Soft-delete: mark a node as expired, don't remove it function applyDelete( graph: KnowledgeGraph, + tfidfIndex: TfidfIndex, correction: Correction ): { success: boolean; error?: string; affectedNodeId?: NodeId } { if (!correction.nodeId) { @@ -233,6 +234,10 @@ function applyDelete( if (retired) { node.metadata.deletedAt = now; node.metadata.deleteReason = correction.reason; + // Drop similarity postings. The tomb stays in the graph for audit; it must + // not keep seeding or diluting IDF. Re-ingest of the same body restores a + // fresh live id via ordinary append (retired-by-delete does not block). + if (removeDocument(tfidfIndex, node.id)) computeIdf(tfidfIndex); } graph.metadata.updatedAt = Date.now(); @@ -310,6 +315,7 @@ function applySupersede( if (retired) { oldNode.metadata.deletedAt = now; oldNode.metadata.deleteReason = correction.reason; + if (removeDocument(tfidfIndex, oldNode.id)) computeIdf(tfidfIndex); } graph.metadata.directedEdgeCount = graph.directedEdges.size; @@ -363,13 +369,14 @@ export function importCorrections( // Bulk forgetting: soft-delete nodes by time window export function forgetByTimeWindow( - graph: KnowledgeGraph, + graph: KnowledgeGraph & { tfidfIndex?: TfidfIndex }, before: number, // Timestamp: forget everything created before this reason: string = 'system:retention-policy' ): { forgotten: number } { let forgotten = 0; const now = Date.now(); const supersededIds = collectSupersededIds(graph); + const dropped: NodeId[] = []; for (const [, node] of graph.nodes) { if (node.type === 'document' || node.type === 'section') continue; @@ -379,10 +386,13 @@ export function forgetByTimeWindow( retireNode(node, { retiredBy: 'delete', reason, now, supersededIds }); node.metadata.forgottenAt = now; node.metadata.forgetReason = reason; + dropped.push(node.id); forgotten++; } } + dropTombsFromIndex(graph.tfidfIndex, dropped); + graph.metadata.updatedAt = now; graph.metadata.version++; return { forgotten }; @@ -390,7 +400,7 @@ export function forgetByTimeWindow( // Bulk forgetting: soft-delete nodes by topic (entity match) export function forgetByTopic( - graph: KnowledgeGraph, + graph: KnowledgeGraph & { tfidfIndex?: TfidfIndex }, topic: string, reason: string = `system:topic-forget:${topic}`, opts: { dryRun?: boolean } = {} @@ -441,6 +451,8 @@ export function forgetByTopic( if (opts.dryRun) return { forgotten: 0, nodeIds }; + dropTombsFromIndex(graph.tfidfIndex, nodeIds); + graph.metadata.updatedAt = now; graph.metadata.version++; return { forgotten, nodeIds }; @@ -449,7 +461,7 @@ export function forgetByTopic( // Cascade soft-delete: when a source node is soft-deleted, // follow edges to soft-delete all downstream nodes from that source export function cascadeSoftDelete( - graph: KnowledgeGraph, + graph: KnowledgeGraph & { tfidfIndex?: TfidfIndex }, nodeId: NodeId, reason: string = 'system:cascade-delete' ): { cascaded: number } { @@ -460,6 +472,7 @@ export function cascadeSoftDelete( // Once, not per node: the cascade reaches every node sharing the source file, // and any of them may be a legacy supersede tomb the delete must not overwrite. const supersededIds = collectSupersededIds(graph); + const dropped: NodeId[] = []; while (queue.length > 0) { const current = queue.shift()!; @@ -474,6 +487,7 @@ export function cascadeSoftDelete( if (retireNode(node, { retiredBy: 'delete', reason, now, supersededIds })) { node.metadata.forgottenAt = now; node.metadata.forgetReason = reason; + dropped.push(node.id); if (current !== nodeId) cascaded++; // Don't count the root node } @@ -495,11 +509,22 @@ export function cascadeSoftDelete( } } + dropTombsFromIndex(graph.tfidfIndex, dropped); + graph.metadata.updatedAt = now; graph.metadata.version++; return { cascaded }; } +function dropTombsFromIndex(index: TfidfIndex | undefined, ids: readonly NodeId[]): void { + if (!index || ids.length === 0) return; + let dropped = false; + for (const id of ids) { + if (removeDocument(index, id)) dropped = true; + } + if (dropped) computeIdf(index); +} + function classifyCorrectionType(text: string): NodeType { const lower = text.toLowerCase(); if (/\b(is defined as|refers to|means)\b/.test(lower)) return 'definition'; diff --git a/src/core/graph/chunker-migration.ts b/src/core/graph/chunker-migration.ts new file mode 100644 index 0000000..2e9c84e --- /dev/null +++ b/src/core/graph/chunker-migration.ts @@ -0,0 +1,42 @@ +import { CURRENT_CHUNKER_ID } from '@/core/types'; +import type { KnowledgeGraph } from '@/core/types'; +import { collectSupersededIds, isRetired } from './retirement'; + +const STRUCTURAL = new Set(['document', 'section']); + +/** + * Live source files whose content nodes were NOT written under the current + * chunker (`CURRENT_CHUNKER_ID` / `boundaries-v1`). + * + * Pre-0.11.1 splitters could shatter identifiers and drop short tails. There is + * no safe in-place rewrite of stored cortex — identity, edges, and citations + * bind to the old chunks. The host should offer opt-in `replaceSource` per + * listed file (or a bulk re-ingest) so new boundaries land as a fresh + * generation under `retiredBy: 'replace'`. + * + * Identity-extraction and other synthetic sources are skipped: they are not + * file re-ingest targets. + */ +export function sourcesNeedingChunkerReingest( + graph: KnowledgeGraph, + at: number = Date.now(), +): string[] { + const supersededIds = collectSupersededIds(graph); + const byFile = new Map(); + + for (const node of graph.nodes.values()) { + if (STRUCTURAL.has(node.type)) continue; + if (isRetired(node, at, supersededIds)) continue; + const file = node.source.file; + if (!file || file === 'identity-extraction') continue; + const row = byFile.get(file) ?? { live: 0, stamped: 0 }; + row.live++; + if (node.metadata.chunkerId === CURRENT_CHUNKER_ID) row.stamped++; + byFile.set(file, row); + } + + return [...byFile.entries()] + .filter(([, row]) => row.live > 0 && row.stamped < row.live) + .map(([file]) => file) + .sort((a, b) => (a < b ? -1 : a > b ? 1 : 0)); +} diff --git a/src/core/graph/graph-builder.ts b/src/core/graph/graph-builder.ts index 741b2a1..6eaead7 100644 --- a/src/core/graph/graph-builder.ts +++ b/src/core/graph/graph-builder.ts @@ -1,5 +1,5 @@ import { nanoid } from 'nanoid'; -import { CURRENT_INGEST_POLICY_ID } from '@/core/types'; +import { CURRENT_INGEST_POLICY_ID, CURRENT_CHUNKER_ID } from '@/core/types'; import type { KnowledgeGraph, GraphNode, @@ -56,6 +56,7 @@ export function buildGraph( ...chunk.metadata, chunkOrder: chunk.order, ingestPolicyId: CURRENT_INGEST_POLICY_ID, + chunkerId: CURRENT_CHUNKER_ID, }, level: 0, confidence: 0.9, // Default confidence for extracted content diff --git a/src/core/graph/incremental.ts b/src/core/graph/incremental.ts index 33d06d4..1858c2a 100644 --- a/src/core/graph/incremental.ts +++ b/src/core/graph/incremental.ts @@ -1,5 +1,5 @@ import { nanoid } from 'nanoid'; -import { CURRENT_INGEST_POLICY_ID } from '@/core/types'; +import { CURRENT_INGEST_POLICY_ID, CURRENT_CHUNKER_ID } from '@/core/types'; import type { GraphNode, KnowledgeGraph, @@ -27,6 +27,15 @@ export interface AddDocumentsOptions { * 'balanced' (matches historical SDK behaviour). See `ChunkSizePreset` * in `@/core/extraction/chunker` for the per-preset numeric targets. */ chunkSize?: ChunkSizePreset; + /** + * Node ids to exclude from dedup / `blocksReingest` consideration for THIS + * call only. Used by `replaceSource`: it retires the prior generation with + * `retiredBy: 'replace'` (which ordinarily blocks re-ingest of the old body), + * then appends the new file — unchanged paragraphs must land as fresh live + * nodes, not bind to the tombs just created. Ordinary `append` never sets + * this. + */ + ignoreNodeIds?: ReadonlySet; } // Add new documents to an existing graph without full rebuild @@ -95,8 +104,10 @@ export function addDocumentsToGraph( // reports that it was dropped. const ingestNow = Date.now(); const supersededIds = collectSupersededIds(graph); + const ignoreIds = opts.ignoreNodeIds; const existingByHash = new Map(); for (const node of graph.nodes.values()) { + if (ignoreIds?.has(node.id)) continue; if (blocksReingest(node, ingestNow, supersededIds)) { const bucket = existingByHash.get(node.contentHash); if (bucket) bucket.push(node); @@ -189,6 +200,7 @@ export function addDocumentsToGraph( ...chunk.metadata, chunkOrder: chunk.order, ingestPolicyId: CURRENT_INGEST_POLICY_ID, + chunkerId: CURRENT_CHUNKER_ID, }, level: 0, confidence: 0.9, diff --git a/src/core/graph/replace-source.ts b/src/core/graph/replace-source.ts new file mode 100644 index 0000000..6b8a2a9 --- /dev/null +++ b/src/core/graph/replace-source.ts @@ -0,0 +1,167 @@ +import type { + KnowledgeGraph, + NodeId, + ParsedDocument, + TfidfIndex, +} from '@/core/types'; +import { collectSupersededIds, isRetired, retireNode } from './retirement'; +import { addDocumentsToGraph, type AddDocumentsOptions, type IncrementalResult } from './incremental'; +import { removeDocument, computeIdf } from '@/core/similarity/tfidf'; +import type { ChunkSizePreset } from '@/core/extraction/chunker'; + +export interface ReplaceSourceOptions { + chunkSize?: ChunkSizePreset; + /** + * Optional generation stamp written onto every NEW node from this call as + * `metadata.sourceGeneration`. Hosts use it for mixed-vintage diagnostics + * (which replace pass produced this live content). + */ + generation?: string | number; + /** Retirement instant. Defaults to `Date.now()`. */ + now?: number; + reason?: string; +} + +export interface ReplaceSourceResult extends IncrementalResult { + /** How many previously-active nodes for this source were retired. */ + retired: number; + /** Ids retired by this call (audit / host bookkeeping). */ + retiredNodeIds: NodeId[]; + /** How many TF-IDF postings were dropped for those tombs. */ + indexDropped: number; +} + +/** + * Soft-retire every active node for `sourceFile`, drop their TF-IDF postings, + * then append `docs` through the ordinary additive path — exempting the + * just-retired generation from dedup so unchanged paragraphs become fresh + * live nodes. + * + * This is the full-file replacement primitive. Additive `append` stays for + * partial supplements; hosts must not approximate replace with + * forget-then-append (that left retired chunks active as top seeds when the + * index retained tombs and `'delete'` allowed re-ingest of the old body). + * + * Retirement uses `retiredBy: 'replace'`, which blocks ordinary later append + * of the old body while still leaving content readable via `retired()`. + */ +export function replaceSourceInGraph( + graph: KnowledgeGraph & { tfidfIndex?: TfidfIndex; embeddingIndex?: { vectors: Map } }, + sourceFile: string, + docs: readonly ParsedDocument[], + opts: ReplaceSourceOptions = {}, +): ReplaceSourceResult { + if (!sourceFile) { + throw new Error('[graphnosis] replaceSource(): sourceFile must be a non-empty string'); + } + for (const doc of docs) { + if (doc.sourceFile !== sourceFile) { + throw new Error( + `[graphnosis] replaceSource(): every document must carry sourceFile="${sourceFile}" ` + + `(got "${doc.sourceFile}"). Refuse rather than silently retire the wrong source.`, + ); + } + } + + const now = opts.now ?? Date.now(); + const reason = opts.reason ?? `system:replace-source:${sourceFile}`; + const supersededIds = collectSupersededIds(graph); + const retiredNodeIds: NodeId[] = []; + + for (const node of graph.nodes.values()) { + if (node.source.file !== sourceFile) continue; + if (isRetired(node, now, supersededIds)) continue; + if (retireNode(node, { retiredBy: 'replace', reason, now, supersededIds })) { + retiredNodeIds.push(node.id); + } + } + + let indexDropped = 0; + const index = graph.tfidfIndex; + if (index && retiredNodeIds.length > 0) { + for (const id of retiredNodeIds) { + if (removeDocument(index, id)) indexDropped++; + } + computeIdf(index); + } + + // Embedding vectors are derived state too — drop tombs so hybrid retrieval + // cannot re-seed a replaced generation. Rebuild of live vectors for new + // nodes is the caller's `appendWithEmbeddings` / `buildEmbeddings` path; + // sync replaceSource only clears. + const emb = graph.embeddingIndex; + if (emb) { + for (const id of retiredNodeIds) emb.vectors.delete(id); + } + + const ignoreNodeIds = new Set(retiredNodeIds); + const appendOpts: AddDocumentsOptions = { + chunkSize: opts.chunkSize, + ignoreNodeIds, + }; + const incremental = addDocumentsToGraph(graph, [...docs], appendOpts); + + if (opts.generation !== undefined) { + for (const id of incremental.newNodeIds) { + const node = graph.nodes.get(id); + if (node) node.metadata.sourceGeneration = opts.generation; + } + } + + return { + ...incremental, + retired: retiredNodeIds.length, + retiredNodeIds, + indexDropped, + }; +} + +/** + * Soft-retire every active node for `sourceFile` without appending anything. + * + * Prefer `replaceSource` when the host has the new file body. This exists so + * hosts can drop a source cleanly (and drop its TF-IDF postings) without the + * content-erasing `__gn-forgotten:` tomb workaround — retirement already keeps + * content out of prompts. + */ +export function forgetSourceInGraph( + graph: KnowledgeGraph & { tfidfIndex?: TfidfIndex; embeddingIndex?: { vectors: Map } }, + sourceFile: string, + reason: string = `user:forget-source:${sourceFile}`, + now: number = Date.now(), +): { forgotten: number; nodeIds: NodeId[]; indexDropped: number } { + if (!sourceFile) { + throw new Error('[graphnosis] forgetSource(): sourceFile must be a non-empty string'); + } + const supersededIds = collectSupersededIds(graph); + const nodeIds: NodeId[] = []; + for (const node of graph.nodes.values()) { + if (node.source.file !== sourceFile) continue; + if (isRetired(node, now, supersededIds)) continue; + if (retireNode(node, { retiredBy: 'delete', reason, now, supersededIds })) { + node.metadata.forgottenAt = now; + node.metadata.forgetReason = reason; + nodeIds.push(node.id); + } + } + + let indexDropped = 0; + const index = graph.tfidfIndex; + if (index && nodeIds.length > 0) { + for (const id of nodeIds) { + if (removeDocument(index, id)) indexDropped++; + } + computeIdf(index); + } + const emb = graph.embeddingIndex; + if (emb) { + for (const id of nodeIds) emb.vectors.delete(id); + } + + if (nodeIds.length > 0) { + graph.metadata.updatedAt = now; + graph.metadata.version++; + } + + return { forgotten: nodeIds.length, nodeIds, indexDropped }; +} diff --git a/src/core/graph/retirement.ts b/src/core/graph/retirement.ts index e42bf5d..049feba 100644 --- a/src/core/graph/retirement.ts +++ b/src/core/graph/retirement.ts @@ -100,7 +100,7 @@ export function isAdministrativelyRetired( supersededIds?: ReadonlySet, ): boolean { const by = node.metadata?.retiredBy; - if (by === 'delete' || by === 'supersede') return true; + if (by === 'delete' || by === 'supersede' || by === 'replace') return true; if (typeof node.metadata?.deletedAt === 'number') return true; if (typeof node.metadata?.forgottenAt === 'number') return true; if (supersededIds?.has(node.id) && typeof node.validUntil === 'number') return true; @@ -108,8 +108,8 @@ export function isAdministrativelyRetired( } /** - * Why a node was retired. The two are NOT interchangeable, because their - * re-ingest semantics are opposite: + * Why a node was retired. The three are NOT interchangeable, because their + * re-ingest semantics differ: * * - `delete` — the user asked to forget this content. Re-adding the source * SHOULD restore it: forgetting is not a permanent ban on a @@ -118,11 +118,17 @@ export function isAdministrativelyRetired( * Re-syncing the unchanged source must NOT resurrect it; the * file still carrying the old text is precisely why the * correction existed. + * - `replace` — a full-file `replaceSource` retired the prior generation. + * Ordinary `append` of the old body must NOT resurrect it + * (hosts would otherwise need forget+append glue that leaves + * stale actives). The same call's own append is exempted via + * `ignoreNodeIds` so unchanged paragraphs in the new file can + * land as fresh live nodes. * * Recorded on the node as `metadata.retiredBy` so the distinction survives * `.gai` / SQLite round-trips and is auditable after the fact. */ -export type RetirementReason = 'delete' | 'supersede'; +export type RetirementReason = 'delete' | 'supersede' | 'replace'; /** Confidence written when a node is retired. * @@ -186,7 +192,9 @@ export function retirementReasonOf( supersededIds?: ReadonlySet, ): RetirementReason { const recorded = node.metadata.retiredBy; - if (recorded === 'delete' || recorded === 'supersede') return recorded; + if (recorded === 'delete' || recorded === 'supersede' || recorded === 'replace') { + return recorded; + } if (supersededIds?.has(node.id)) return 'supersede'; return 'delete'; } @@ -228,6 +236,9 @@ export function collectSupersededIds(graph: KnowledgeGraph): Set { * Retired by `supersede`: yes — blocking is the whole point. The source file * still holds the text the user corrected away; without this, re-syncing it * re-creates the corrected-away fact as a fresh live node at full confidence. + * Retired by `replace`: yes — a prior generation was deliberately replaced. + * Ordinary append of the old body must not resurrect it; `replaceSource` + * itself exempts the just-retired ids for its own append. * Retired by `delete`: no — forgetting a source then re-adding the file restores * it, which is what "forget" means as distinct from "this string is banned". * Expired (not retired): yes — expiry is not a retract; the fact is still the @@ -239,5 +250,6 @@ export function blocksReingest( supersededIds?: ReadonlySet, ): boolean { if (!isRetired(node, now, supersededIds)) return true; - return retirementReasonOf(node, supersededIds) === 'supersede'; + const reason = retirementReasonOf(node, supersededIds); + return reason === 'supersede' || reason === 'replace'; } diff --git a/src/core/query/query-engine.ts b/src/core/query/query-engine.ts index 80d988c..68a9076 100644 --- a/src/core/query/query-engine.ts +++ b/src/core/query/query-engine.ts @@ -34,6 +34,7 @@ import { SEED_COUNT, SEED_OVERSAMPLE, STRUCTURAL_EXPANSION_BUDGET_SHARE, + STRUCTURAL_DOC_RESERVE_SHARE, TOP_K_NODES, } from '@/core/constants'; @@ -76,8 +77,8 @@ export interface QueryOptions { // Retired nodes — administratively forgotten or superseded, see `isRetired` — // are kept out of the prompt at four independent gates: seeding, traversal, // structural completion, and a final strip before serialization. Their - // content stays in the graph, in the TF-IDF index, and in the audit surface; - // it simply never reaches a model. + // content stays in the graph and in the audit surface; similarity postings + // are dropped on retire. It simply never reaches a model. // // Defaults to the wall clock — never to `now`. Reading the clock here is a // deliberate trade against the "retrieval reads no clock" property: a @@ -678,19 +679,59 @@ export function expandWithStructuralContext( } } - const additions = [...candidateById.values()] - .sort((a, b) => - a.priority - b.priority || - b.score - a.score || - a.node.source.file.localeCompare(b.node.source.file) || - Number(a.node.metadata.chunkOrder ?? Number.MAX_SAFE_INTEGER) - - Number(b.node.metadata.chunkOrder ?? Number.MAX_SAFE_INTEGER) || - a.node.source.offset - b.node.source.offset || - (a.node.source.section ?? '').localeCompare(b.node.source.section ?? '') || - a.node.type.localeCompare(b.node.type) || - a.node.contentHash.localeCompare(b.node.contentHash) - ) - .slice(0, Math.max(0, maxAdditions)); + const additionsBudget = Math.max(0, maxAdditions); + const candidates = [...candidateById.values()].sort((a, b) => + a.priority - b.priority || + b.score - a.score || + a.node.source.file.localeCompare(b.node.source.file) || + Number(a.node.metadata.chunkOrder ?? Number.MAX_SAFE_INTEGER) - + Number(b.node.metadata.chunkOrder ?? Number.MAX_SAFE_INTEGER) || + a.node.source.offset - b.node.source.offset || + (a.node.source.section ?? '').localeCompare(b.node.source.section ?? '') || + a.node.type.localeCompare(b.node.type) || + a.node.contentHash.localeCompare(b.node.contentHash) + ); + + // Split the post-cut reserve when BOTH turn-pairs (priority 0) and document + // neighbors (priority 1) are offered. Priority-only sort previously let wide + // chat pairing consume the entire reserve (M3.1/M4 mixed-corpus starvation). + // Homogeneous candidate sets keep the historical priority order. + let additions = candidates; + if (Number.isFinite(additionsBudget) && additionsBudget > 0) { + const p0 = candidates.filter((c) => c.priority === 0); + const p1 = candidates.filter((c) => c.priority === 1); + const other = candidates.filter((c) => c.priority !== 0 && c.priority !== 1); + if (p0.length > 0 && p1.length > 0) { + const docReserve = Math.max( + 1, + Math.floor(additionsBudget * STRUCTURAL_DOC_RESERVE_SHARE), + ); + const chatReserve = Math.max(0, additionsBudget - docReserve); + const picked: typeof candidates = []; + const taken = new Set(); + for (const c of p0) { + if (picked.filter((x) => x.priority === 0).length >= chatReserve) break; + picked.push(c); + taken.add(c.node.id); + } + for (const c of p1) { + if (picked.filter((x) => x.priority === 1).length >= docReserve) break; + if (taken.has(c.node.id)) continue; + picked.push(c); + taken.add(c.node.id); + } + for (const c of [...p0, ...p1, ...other]) { + if (picked.length >= additionsBudget) break; + if (taken.has(c.node.id)) continue; + picked.push(c); + taken.add(c.node.id); + } + additions = picked; + } else { + additions = candidates.slice(0, additionsBudget); + } + } + if (additions.length === 0) return result; const allNodes = [...result.nodes, ...additions.map(({ node }) => node)]; diff --git a/src/core/query/seed-finder.ts b/src/core/query/seed-finder.ts index cdf9696..070ee1f 100644 --- a/src/core/query/seed-finder.ts +++ b/src/core/query/seed-finder.ts @@ -25,9 +25,9 @@ export interface SeedOptions { * * This is the FIRST of the four gates that keep retired content out of a * prompt, and it is the one that matters most. A retired node stays in the - * TF-IDF index by design (see `retiredAt` in QueryOptions), so without this - * filter it comes back as the rank-1 seed of the next query whose terms it - * matches, and every later stage inherits it. + * graph for audit. Similarity postings are dropped on retire + * (`removeDocument` / `buildIndexFromGraph` exclusion); this seed gate + * remains the belt for any stale index a host might hand in. * * A predicate rather than `graph` + an instant, deliberately: `graph` also * switches on provenance tie-breaking, so routing the gate through it would diff --git a/src/core/query/traverser.ts b/src/core/query/traverser.ts index 8456d2c..d62f0a4 100644 --- a/src/core/query/traverser.ts +++ b/src/core/query/traverser.ts @@ -1,7 +1,7 @@ import type { KnowledgeGraph, GraphNode, DirectedEdge, DirectedEdgeType, UndirectedEdge, NodeId } from '@/core/types'; import { collectSupersededIds, isExpired, isRetired } from '@/core/graph/retirement'; import { entryByScoreThenSource } from './tie-break'; -import { MAX_TRAVERSAL_HOPS, DECAY_FACTOR, TOP_K_NODES } from '@/core/constants'; +import { MAX_TRAVERSAL_HOPS, DECAY_FACTOR, TOP_K_NODES, MIXED_CORPUS_DOC_FLOOR_SHARE } from '@/core/constants'; import type { ScoredSeed } from './seed-finder'; export interface TraversalResult { @@ -172,6 +172,72 @@ export interface TraverseOptions { */ const STRUCTURAL_NODE_TYPES: ReadonlySet = new Set(['document', 'section']); +/** Conversation turn headings written by `conversationToDocument`. */ +const CONVERSATION_TURN_SECTION = /^(?:Assistant|User) \(turn \d+\)$/; + +/** + * True when a node comes from conversation-shaped ingest (chat turns), as + * opposed to ordinary documents. Used by the mixed-corpus document floor so + * wide sessions cannot starve document gold under a tight `maxNodes`. + */ +export function isConversationShapedNode( + node: Pick, +): boolean { + const file = node.source.file ?? ''; + if (file.startsWith('conversation:')) return true; + const section = node.source.section ?? ''; + return CONVERSATION_TURN_SECTION.test(section); +} + +/** + * When the scored set contains BOTH conversation and document content, reserve + * a floor of slots for document content before score fills the remainder. + * Measured (M4 Alder fixture): without this, maxNodes 6–10 returned 0 doc + * content nodes. Docs-only and chat-only sets are unchanged. + */ +function applyMixedCorpusDocFloor( + scoredAll: Array<[NodeId, number]>, + graph: KnowledgeGraph, + maxNodes: number, + byScoreThenId: (a: [NodeId, number], b: [NodeId, number]) => number, +): Array<[NodeId, number]> { + if (scoredAll.length <= maxNodes) return scoredAll; + + const docEntries: Array<[NodeId, number]> = []; + const chatEntries: Array<[NodeId, number]> = []; + for (const entry of scoredAll) { + const node = graph.nodes.get(entry[0]); + if (!node) { + docEntries.push(entry); + continue; + } + if (isConversationShapedNode(node)) chatEntries.push(entry); + else docEntries.push(entry); + } + + // Homogeneous candidate sets: score-only selection (no behaviour change). + if (docEntries.length === 0 || chatEntries.length === 0) { + return scoredAll.slice(0, maxNodes); + } + + const floor = Math.max(1, Math.floor(maxNodes * MIXED_CORPUS_DOC_FLOOR_SHARE)); + const docTake = Math.min(floor, docEntries.length, maxNodes); + const picked: Array<[NodeId, number]> = []; + const taken = new Set(); + for (let i = 0; i < docTake; i++) { + const entry = docEntries[i]!; + picked.push(entry); + taken.add(entry[0]); + } + for (const entry of scoredAll) { + if (picked.length >= maxNodes) break; + if (taken.has(entry[0])) continue; + picked.push(entry); + taken.add(entry[0]); + } + return picked.sort(byScoreThenId); +} + export function traverseGraph( graph: KnowledgeGraph, seeds: ScoredSeed[], @@ -534,7 +600,7 @@ export function traverseGraph( // Hand back in score order — the floor decides membership, not presentation. sortedNodes = picked.sort(byScoreThenId); } else { - sortedNodes = scoredAll.slice(0, maxNodes); + sortedNodes = applyMixedCorpusDocFloor(scoredAll, graph, maxNodes, byScoreThenId); } const selectedNodeIds = new Set(sortedNodes.map(([id]) => id)); diff --git a/src/core/similarity/tfidf.ts b/src/core/similarity/tfidf.ts index 2fc8ebf..6f9448c 100644 --- a/src/core/similarity/tfidf.ts +++ b/src/core/similarity/tfidf.ts @@ -1,6 +1,7 @@ import type { TfidfIndex, NodeId, IndexProvenance, KnowledgeGraph } from '@/core/types'; import { asciiFoldAnalyzer, resolveAnalyzer, type TextAnalyzer } from './analyzer'; import { AnalyzerMismatchError } from '@/core/errors'; +import { collectSupersededIds, isAdministrativelyRetired } from '@/core/graph/retirement'; /** * Internal: the TF-IDF index needs to remember its analyzer at runtime so @@ -90,6 +91,29 @@ export function addDocument(index: TfidfIndex, nodeId: NodeId, text: string): vo bumpGeneration(index); } +/** + * Drop a node from the TF-IDF corpus. + * + * Ordinary forget leaves content in the graph for audit; similarity postings + * are a different matter. A retired id that stays in `documents` / term + * postings still contributes to `documentCount` (IDF numerator) and can surface + * as a seed until a later gate filters it. Callers that retire nodes — + * `replaceSource`, `deleteNode`, bulk forget — should remove them here and + * recompute IDF. + * + * Returns false when the id was not indexed. + */ +export function removeDocument(index: TfidfIndex, nodeId: NodeId): boolean { + if (!index.documents.has(nodeId)) return false; + index.documents.delete(nodeId); + // Keep documentCount aligned with documents.size. Decrementing alone can + // drift below reality if an earlier upsert path ever disagreed; size is the + // ground truth for how many TF maps IDF should average over. + index.documentCount = index.documents.size; + bumpGeneration(index); + return true; +} + export function computeIdf(index: TfidfIndex): void { const termDocCount = new Map(); @@ -202,6 +226,10 @@ export function queryVector(index: TfidfIndex, text: string): Map { + if (cond) console.log(` ok ${name}`); + else { + failures += 1; + console.error(` FAIL ${name}${detail ? ` — ${detail}` : ''}`); + } +}; + +console.log('\n-- C1 fresh ingest stamps chunkerId --'); +{ + const g = new Graphnosis({ name: 'chunk-c1' }); + g.addMarkdown('## Host\n\nSee host.ts and version 0.7.4 for details.\n', 'host.md'); + g.build(); + const content = [...g.graph.nodes.values()].filter( + (n) => n.type !== 'document' && n.type !== 'section', + ); + check('content nodes exist', content.length >= 1); + check( + 'every content node carries CURRENT_CHUNKER_ID', + content.every((n) => n.metadata.chunkerId === CURRENT_CHUNKER_ID), + ); + check('fresh graph needs no chunker reingest', g.sourcesNeedingChunkerReingest().length === 0); +} + +console.log('\n-- C2 unstamped live sources are listed --'); +{ + const g = new Graphnosis({ name: 'chunk-c2' }); + g.addMarkdown('## A\n\nLegacy shaped content about calendars and vaults.\n', 'legacy.md'); + g.addMarkdown('## B\n\nAnother file about soil and rain gauges.\n', 'other.md'); + g.build(); + // Simulate a pre-stamp cortex: strip chunkerId from one source only. + for (const n of g.graph.nodes.values()) { + if (n.source.file === 'legacy.md') delete n.metadata.chunkerId; + } + const needing = sourcesNeedingChunkerReingest(g.graph); + check('legacy.md is listed', needing.includes('legacy.md')); + check('stamped other.md is not listed', !needing.includes('other.md')); +} + +console.log('\n-- C3 replaceSource clears the migration flag for that file --'); +{ + const g = new Graphnosis({ name: 'chunk-c3' }); + g.addMarkdown('## A\n\nContent that needs a fresh boundary pass.\n', 'needs.md'); + g.build(); + for (const n of g.graph.nodes.values()) { + if (n.source.file === 'needs.md') delete n.metadata.chunkerId; + } + check('flagged before replace', g.sourcesNeedingChunkerReingest().includes('needs.md')); + g.replaceMarkdown('## A\n\nContent that needs a fresh boundary pass.\n', 'needs.md'); + check( + 'cleared after replaceSource', + !g.sourcesNeedingChunkerReingest().includes('needs.md'), + ); +} + +if (failures > 0) { + console.error(`\n${failures} failure(s)`); + process.exit(1); +} +console.log('\nAll chunker-migration checks passed.'); diff --git a/tests/unit/mixed-corpus-starvation.test.ts b/tests/unit/mixed-corpus-starvation.test.ts new file mode 100644 index 0000000..111bbcd --- /dev/null +++ b/tests/unit/mixed-corpus-starvation.test.ts @@ -0,0 +1,173 @@ +/** + * Mixed doc+chat starvation under tight maxNodes. + * + * M4 measured: one Alder doc with AL-7 marker + wide conversation sharing the + * query vocabulary → maxNodes 6/8/10 returned 0 document content nodes. + * The mixed-corpus document floor reserves ≥25% (at least one slot) for + * non-conversation content when both classes are in the scored set. + * + * Run: tsx tests/unit/mixed-corpus-starvation.test.ts + */ +import { Graphnosis } from '@/sdk/index'; +import { conversationToDocument } from '@/core/ingestion/parsers/conversation-parser'; +import { isConversationShapedNode } from '@/core/query/traverser'; +import { expandWithStructuralContext } from '@/core/query/query-engine'; +import type { GraphNode, KnowledgeGraph, NodeId } from '@/core/types'; +import type { TraversalResult } from '@/core/query/traverser'; + +let failures = 0; +const check = (name: string, cond: boolean, detail = ''): void => { + if (cond) console.log(` ok ${name}`); + else { + failures += 1; + console.error(` FAIL ${name}${detail ? ` — ${detail}` : ''}`); + } +}; + +const AL7 = 'AL-7'; +const DOC = `## Calibration + +The Project Alder opal beacon calibration code is ${AL7}. +Record this code in the field notebook before power-on. +`; + +function wideChat(turns: number) { + const messages = []; + for (let i = 1; i <= turns; i++) { + messages.push({ + role: 'user' as const, + content: `Turn ${i}: remind me about the Project Alder opal beacon calibration procedure and field notes.`, + }); + messages.push({ + role: 'assistant' as const, + content: `Turn ${i}: Project Alder opal beacon calibration is discussed in the field guide. Check your notes.`, + }); + } + return conversationToDocument({ + id: 'wide', + title: 'Wide chat', + messages, + sourceFile: 'conversation:wide', + startedAt: 0, + format: 'raw', + metadata: { messageCount: messages.length }, + }); +} + +console.log('\n-- S1 docs-only still retrieves AL-7 under tight budget --'); +{ + const g = new Graphnosis({ name: 'starve-docs' }); + g.addMarkdown(DOC, 'alder.md'); + g.build(); + const q = 'What is the Project Alder opal beacon calibration code?'; + for (const maxNodes of [6, 8, 10]) { + const r = g.query(q, { maxNodes, diversify: false }); + check( + `docs-only maxNodes=${maxNodes} includes ${AL7}`, + r.subgraph.nodes.some((n) => n.content.includes(AL7)), + ); + } +} + +console.log('\n-- S2 mixed corpus: document gold survives tight maxNodes --'); +{ + const g = new Graphnosis({ name: 'starve-mixed' }); + g.addMarkdown(DOC, 'alder.md'); + g.addDocument(wideChat(20)); + g.build(); + const q = 'What is the Project Alder opal beacon calibration code?'; + + for (const maxNodes of [6, 8, 10]) { + const r = g.query(q, { maxNodes, diversify: false }); + const hasGold = r.subgraph.nodes.some((n) => n.content.includes(AL7)); + const docContent = r.subgraph.nodes.filter( + (n) => n.type !== 'document' && n.type !== 'section' && !isConversationShapedNode(n), + ); + const chatNodes = r.subgraph.nodes.filter((n) => isConversationShapedNode(n)); + console.log( + ` maxNodes=${maxNodes}: gold=${hasGold} docContent=${docContent.length} chat=${chatNodes.length}`, + ); + check(`mixed maxNodes=${maxNodes} includes ${AL7}`, hasGold); + check( + `mixed maxNodes=${maxNodes} keeps at least one doc content node`, + docContent.length >= 1, + ); + } +} + +console.log('\n-- S3 structural reserve splits when both priorities are offered --'); +{ + const node = ( + id: string, + file: string, + section: string, + chunkOrder: number, + ): GraphNode => ({ + id, + content: `content ${id}`, + contentHash: `hash-${id}`, + type: 'fact', + source: { file, section, offset: chunkOrder }, + entities: [], + metadata: { chunkOrder }, + level: 0, + confidence: 1, + createdAt: 0, + lastAccessedAt: 0, + accessCount: 0, + }); + + const docAnchor = node('d0', 'guide.md', 'Overview', 0); + const docMid = node('d1', 'guide.md', 'Ops', 1); + const docFar = node('d2', 'guide.md', 'Far', 2); + const assistants = [1, 2, 3, 4, 5].map((t) => + node(`a${t}`, 'conversation:wide', `Assistant (turn ${t})`, t * 2), + ); + const users = [1, 2, 3, 4, 5].map((t) => + node(`u${t}`, 'conversation:wide', `User (turn ${t})`, t * 2 + 1), + ); + + const nodes = [docAnchor, docMid, docFar, ...assistants, ...users]; + const graph: KnowledgeGraph = { + id: 'starve-struct', + name: 'starve-struct', + nodes: new Map(nodes.map((n) => [n.id, n])), + directedEdges: new Map(), + undirectedEdges: new Map(), + levels: 1, + metadata: { + createdAt: 0, + updatedAt: 0, + sourceFiles: ['guide.md', 'conversation:wide'], + nodeCount: nodes.length, + directedEdgeCount: 0, + undirectedEdgeCount: 0, + version: 1, + }, + }; + + const initial: TraversalResult = { + nodes: [docAnchor, ...assistants], + directedEdges: [], + undirectedEdges: [], + scores: new Map([ + [docAnchor.id, 1], + ...assistants.map((a, i) => [a.id, 0.9 - i * 0.01] as [NodeId, number]), + ]), + }; + + // Budget 4: without a split, all 5 user pairs (p0) would crowd out docs. + const expanded = expandWithStructuralContext(graph, initial, 4); + const ids = new Set(expanded.nodes.map((n) => n.id)); + const addedUsers = users.filter((u) => ids.has(u.id)).length; + const addedDocs = [docMid, docFar].filter((d) => ids.has(d.id)).length; + console.log(` additions: users=${addedUsers} docs=${addedDocs} total=${expanded.nodes.length}`); + check('structural split admits at least one document neighbor', addedDocs >= 1); + check('structural split still admits at least one paired user turn', addedUsers >= 1); +} + +if (failures > 0) { + console.error(`\n${failures} failure(s)`); + process.exit(1); +} +console.log('\nAll mixed-corpus starvation checks passed.'); diff --git a/tests/unit/replace-source.test.ts b/tests/unit/replace-source.test.ts new file mode 100644 index 0000000..38980ce --- /dev/null +++ b/tests/unit/replace-source.test.ts @@ -0,0 +1,160 @@ +/** + * replaceSource — full-file retire + index drop + append. + * + * M3/M4: deleting a section and re-appending left the retired chunk active, + * parented, and queryable as a top seed. Hosts approximated replace with + * forget+append glue; this primitive closes that hole. + * + * Run: tsx tests/unit/replace-source.test.ts + */ +import { Graphnosis } from '@/sdk/index'; +import { parseMarkdown } from '@/core/ingestion/parsers/markdown-parser'; +import { isRetired, retirementReasonOf, blocksReingest } from '@/core/graph/retirement'; +import { collectSupersededIds } from '@/core/graph/retirement'; + +let failures = 0; +const check = (name: string, cond: boolean, detail = ''): void => { + if (cond) console.log(` ok ${name}`); + else { + failures += 1; + console.error(` FAIL ${name}${detail ? ` — ${detail}` : ''}`); + } +}; + +const A = 'Alpha paragraph about the migration plan and its rollback window.'; +const B = 'Beta paragraph about the signed checkpoint and provenance record.'; +const C = 'Gamma paragraph about the deterministic rehearsal and approval sequence.'; +const MARKER = 'UNIQUE_BETA_MARKER_ZX9'; + +const body = (sections: Array<[string, string]>): string => + sections.map(([t, c]) => `## ${t}\n\n${c}\n`).join('\n'); + +console.log('\n-- R1 replaceSource drops removed section from actives and seeds --'); +{ + const g = new Graphnosis({ name: 'replace-r1' }); + g.addMarkdown( + body([ + ['Alpha', A], + ['Beta', `${B} ${MARKER}`], + ['Gamma', C], + ]), + 'notes.md', + ); + g.build(); + + const before = g.query(`Where is ${MARKER}?`, { maxNodes: 10 }); + check( + 'pre-replace: Beta marker is retrievable', + before.subgraph.nodes.some((n) => n.content.includes(MARKER)), + ); + + const r = g.replaceMarkdown( + body([ + ['Alpha', A], + ['Gamma', C], + ]), + 'notes.md', + { generation: 'gen-2' }, + ); + + check('retired at least the Beta generation', r.retired >= 1, `retired=${r.retired}`); + check('TF-IDF postings dropped for retired ids', r.indexDropped >= 1, `dropped=${r.indexDropped}`); + check('new nodes were appended', r.newNodes >= 1, `newNodes=${r.newNodes}`); + + const now = Date.now(); + const superseded = collectSupersededIds(g.graph); + const betaLive = [...g.graph.nodes.values()].filter( + (n) => n.content.includes(MARKER) && !isRetired(n, now, superseded), + ); + check('Beta marker has no live node', betaLive.length === 0, `live=${betaLive.length}`); + + const betaTombs = [...g.graph.nodes.values()].filter( + (n) => n.content.includes(MARKER) && isRetired(n, now, superseded), + ); + check('Beta marker remains as replace tomb for audit', betaTombs.length >= 1); + check( + 'tombs carry retiredBy=replace', + betaTombs.every((n) => retirementReasonOf(n, superseded) === 'replace'), + ); + + const after = g.query(`Where is ${MARKER}?`, { maxNodes: 10 }); + check( + 'post-replace: Beta marker is not a top seed / prompt node', + !after.subgraph.nodes.some((n) => n.content.includes(MARKER)), + ); + + const alphaLive = [...g.graph.nodes.values()].filter( + (n) => n.content.includes('migration plan') && !isRetired(n, now, superseded), + ); + check('Alpha still live and attributable', alphaLive.length >= 1); + check( + 'new nodes carry generation stamp', + alphaLive.some((n) => n.metadata.sourceGeneration === 'gen-2'), + ); +} + +console.log('\n-- R2 ordinary append of old body cannot restore replace tombs --'); +{ + const g = new Graphnosis({ name: 'replace-r2' }); + g.addMarkdown(body([['Only', `${A} ${MARKER}`]]), 'solo.md'); + g.build(); + g.replaceMarkdown(body([['Only', C]]), 'solo.md'); + + const now = Date.now(); + const superseded = collectSupersededIds(g.graph); + const tomb = [...g.graph.nodes.values()].find((n) => n.content.includes(MARKER)); + check('old body is a tomb', !!tomb && isRetired(tomb!, now, superseded)); + check('replace tombs block reingest', !!tomb && blocksReingest(tomb!, now, superseded)); + + const r = g.appendMarkdown(body([['Only', `${A} ${MARKER}`]]), 'solo.md'); + check('append of old body adds no live duplicate', r.newNodes === 0, `newNodes=${r.newNodes}`); + const liveMarkers = [...g.graph.nodes.values()].filter( + (n) => n.content.includes(MARKER) && !isRetired(n, now, superseded), + ); + check('old marker stays non-live after append', liveMarkers.length === 0); +} + +console.log('\n-- R3 forgetSource drops index; delete-reason allows restore --'); +{ + const g = new Graphnosis({ name: 'replace-r3' }); + g.addMarkdown(body([['Keep', `${A} ${MARKER}`]]), 'temp.md'); + g.build(); + const forgotten = g.forgetSource('temp.md'); + check('forgetSource retires nodes', forgotten.forgotten >= 1); + check('forgetSource drops TF-IDF postings', forgotten.indexDropped >= 1); + + const after = g.query(`Where is ${MARKER}?`, { maxNodes: 8 }); + check( + 'forgotten content is not in the prompt', + !after.subgraph.nodes.some((n) => n.content.includes(MARKER)), + ); + + const restored = g.appendMarkdown(body([['Keep', `${A} ${MARKER}`]]), 'temp.md'); + check('re-append after forgetSource restores live content', restored.newNodes >= 1); + const now = Date.now(); + const superseded = collectSupersededIds(g.graph); + const live = [...g.graph.nodes.values()].filter( + (n) => n.content.includes(MARKER) && !isRetired(n, now, superseded), + ); + check('restored marker is live', live.length >= 1); +} + +console.log('\n-- R4 refuse mismatched sourceFile --'); +{ + const g = new Graphnosis({ name: 'replace-r4' }); + g.addMarkdown(body([['A', A]]), 'a.md'); + g.build(); + let threw = false; + try { + g.replaceSource('a.md', [parseMarkdown(body([['A', A]]), 'b.md')]); + } catch { + threw = true; + } + check('mismatched sourceFile throws', threw); +} + +if (failures > 0) { + console.error(`\n${failures} failure(s)`); + process.exit(1); +} +console.log('\nAll replaceSource checks passed.'); diff --git a/tests/unit/tfidf-tomb-index.test.ts b/tests/unit/tfidf-tomb-index.test.ts new file mode 100644 index 0000000..2e42549 --- /dev/null +++ b/tests/unit/tfidf-tomb-index.test.ts @@ -0,0 +1,81 @@ +/** + * TF-IDF must not keep retired nodes in similarity postings. + * + * Run: tsx tests/unit/tfidf-tomb-index.test.ts + */ +import { Graphnosis } from '@/sdk/index'; +import { buildIndexFromGraph, removeDocument } from '@/core/similarity/tfidf'; +import { asciiFoldEnAnalyzer } from '@/core/similarity/analyzer'; +import { isAdministrativelyRetired, collectSupersededIds } from '@/core/graph/retirement'; + +let failures = 0; +const check = (name: string, cond: boolean, detail = ''): void => { + if (cond) console.log(` ok ${name}`); + else { + failures += 1; + console.error(` FAIL ${name}${detail ? ` — ${detail}` : ''}`); + } +}; + +const MARKER = 'TOMB_INDEX_MARKER_QK7'; + +console.log('\n-- T1 deleteNode drops TF-IDF postings --'); +{ + const g = new Graphnosis({ name: 'tomb-t1' }); + g.addMarkdown( + `## Secret\n\nThe ${MARKER} is buried under the old oak.\n\n## Other\n\nUnrelated garden notes about soil pH.\n`, + 'garden.md', + ); + g.build(); + const target = [...g.graph.nodes.values()].find((n) => n.content.includes(MARKER)); + check('fixture has the marker node', !!target); + check('marker is indexed before delete', g.graph.tfidfIndex.documents.has(target!.id)); + + g.deleteNode(target!.id, 'unit:drop-posting'); + check('marker posting removed after delete', !g.graph.tfidfIndex.documents.has(target!.id)); + check( + 'documentCount matches documents.size', + g.graph.tfidfIndex.documentCount === g.graph.tfidfIndex.documents.size, + ); +} + +console.log('\n-- T2 rebuildIndex excludes administrative tombs --'); +{ + const g = new Graphnosis({ name: 'tomb-t2' }); + g.addMarkdown(`## A\n\nLive content about calendars.\n`, 'a.md'); + g.addMarkdown(`## B\n\nSecret ${MARKER} content about vaults.\n`, 'b.md'); + g.build(); + const secret = [...g.graph.nodes.values()].find((n) => n.content.includes(MARKER))!; + g.deleteNode(secret.id, 'unit:rebuild'); + // Force a full rebuild path (same as loadGai). + g.rebuildIndex(); + check('rebuild skips the tomb', !g.graph.tfidfIndex.documents.has(secret.id)); + + const rebuilt = buildIndexFromGraph(g.graph, asciiFoldEnAnalyzer); + check('buildIndexFromGraph also skips the tomb', !rebuilt.documents.has(secret.id)); + + const superseded = collectSupersededIds(g.graph); + let tombCount = 0; + for (const n of g.graph.nodes.values()) { + if (isAdministrativelyRetired(n, superseded)) tombCount++; + } + check('graph still holds the tomb for audit', tombCount >= 1); +} + +console.log('\n-- T3 replaceSource refresh drops prior generation from index --'); +{ + const g = new Graphnosis({ name: 'tomb-t3' }); + g.addMarkdown(`## Old\n\n${MARKER} prior generation text.\n`, 'file.md'); + g.build(); + const old = [...g.graph.nodes.values()].find((n) => n.content.includes(MARKER))!; + check('old id indexed', g.graph.tfidfIndex.documents.has(old.id)); + g.replaceMarkdown(`## New\n\nReplacement text without the marker phrase.\n`, 'file.md'); + check('old id gone from index after replace', !g.graph.tfidfIndex.documents.has(old.id)); + check('removeDocument is idempotent on missing ids', removeDocument(g.graph.tfidfIndex, old.id) === false); +} + +if (failures > 0) { + console.error(`\n${failures} failure(s)`); + process.exit(1); +} +console.log('\nAll TF-IDF tomb index checks passed.');