Give catalog ingest a factual lane: scrap context in the extractor, a universe ref at commit, and a shipped Reality universe

[atomantic]

Problem / Goal

Pasting an autobiographical prose snippet into Catalog Ingest produces a pile of context-free ingredients: characters come back as generic role tags (MOM, DAD, NARRATOR), and every committed row lands with no universe ref, no ingredient↔ingredient edge, and nothing tying it to the piece it came from. The catalog then shows them as unrelated "Raw / unlinked" rows that read as random.

Three independent gaps cause this, and all three have to close for factual material to be usable:

1. The extractor never learns what it is reading. The scrap's title and source kind are loaded and dropped; the character prompt's framing block renders empty. With no signal that a piece is first-person non-fiction about real people, the fiction-shaped prompt does exactly what it is told and mints stable role tags.
2. Commit creates no grouping. commitScrap writes ingredients + a source link and nothing else, so every fresh ingredient is classified unlinked.
3. There is no factual universe to put them in. Every universe is fiction-shaped, none ships by default, and nothing anywhere in the repo distinguishes real from invented.

Goal: one user-visible control at ingest — the universe this material belongs to — that (a) tells the extractor how to read the text, (b) links every committed ingredient to that universe, and (c) defaults to a shipped Reality universe so factual/autobiographical capture has a home from first boot.

Context

The extractor is context-blind

extractIngredients builds its corpus from rawText alone — server/services/catalogExtraction.js:205:

const corpus = neutralizeFenceDelimiters(rawText);

Both callers in extractIngredientsForScrap pass only rawText (catalogExtraction.js:349 non-chunked, :369 chunked). getScrap/listChildScraps return title and sourceKind via rowToScrap (server/services/catalogDB/shared.js:90) — they are simply never used.

The bible stages go through extractBible, which already accepts a context object that is spread into the prompt variables (server/services/bibleExtractor.js:44). The catalog caller passes none (catalogExtraction.js:229). Meanwhile data.reference/prompts/stages/writers-room-characters.md:5-9 opens with:

## Work being analyzed

- Title: {{work.title}}
- Kind: {{work.kind}}
- Word count: {{work.wordCount}}

applyTemplate substitutes a missing key with '' (server/lib/promptTemplate.js:115), so on the catalog path the model receives an empty framing block. The same file at line 30 then instructs: "For unnamed characters use a stable role tag like THE BARTENDER." — which is correct for fiction and is precisely what produces MOM / DAD / NARRATOR from a memoir.

The light stage is worse: data.reference/prompts/stages/catalog-ideas-scenes-concepts.md has exactly one variable ({{draftBody}}, line 10) and opens "You are a creative analyst surfacing reusable narrative fragments…" (line 3) with definitions written entirely in terms of invented story material.

(The brain-bridge path is the one exception: brainRecordToIngestText prepends the record title into the body at server/services/catalogIngestSources.js:285. url / file / paste do not.)

Commit creates no grouping

server/services/catalogDB/commit.js:16-34 — for each accepted draft: createIngredient + linkIngredientToSource. No catalog_ingredient_refs row, no catalog_ingredient_relations edge.

The infrastructure is already there and already federates:

• catalog_ingredient_refs (ingredient_id, ref_kind, ref_id, role) — server/lib/db/schema/catalog.js:144. Valid kinds universe | series | issue | work | creative-director (server/lib/catalogValidation.js:48), universe role convention canon-<kind>.
• catalog_ingredient_relations (from_id, to_id, kind) — server/lib/db/schema/catalog.js:168, kinds in RELATION_REGISTRY (server/lib/catalogTypes.js:454).
• POST /api/catalog/ingredients/:id/link (server/routes/catalog.js:369) writes refs, but only when a human clicks "Add from Catalog" in client/src/components/universe/UniverseCanonSection.jsx:600.
• Prior art in the same file: /api/catalog/bulk-import already takes defaults.universeRef / seriesRef / workRef / issueRef / role (server/lib/catalogValidation.js:449-462, applied at server/routes/catalog.js:569). The scrap→commit path is the odd one out.

Because no homing ref is written, HAS_ANY_HOMING_REF (server/services/catalogDB/shared.js:258, HOMING_REF_KINDS = ['universe','series','creative-director']) is false for every freshly ingested row — they bucket as unlinked (Raw). That is the "isolated and random" symptom, exactly.

Note also scene.actors[] is sanitized as a plain string array (catalogExtraction.js:115) and never resolved to ingredient ids, so even a scene that names its cast produces no edge.

There is no factual universe, and none ships

• The universe record (server/services/universeBuilder/sanitize.js:683-790) has no factual/fiction axis. Canon trunks are characters/places/objects; category kinds are the same three plus other.
• createUniverse mints randomUUID() and accepts no caller id (server/services/universeBuilder/crud.js:244-247).
• data.reference/ contains no universes/ directory; bootstrapServices reads data.reference only for providers.json (server/services/bootstrap.js:264). Universes are PG rows (server/lib/db/schema/universes.js:19), so setup-data.js is not the seeding mechanism — a migration is.
• Repo-wide there is no factual / non-fiction / isFiction concept on a scrap, an ingredient, or a universe.

Proposed approach

Four phases. Phase 1 is the self-contained slice and fixes the MOM/DAD symptom on its own. Claim it first; if the PR gets large, ship Phase 1 with Refs #N (not Closes #N) and file Phases 2–4 as follow-ups. Phases 2 and 3 are the ones that have to land together for the Reality default to mean anything; Phase 4 is independently shippable.

Phase 1 — Thread scrap context into extraction

1. extractIngredients / extractIngredientsForScrap gain a context argument carrying { title, sourceKind, factual } derived from the parent scrap (and, once Phase 2 lands, from the target universe's factual flag). Chunked children inherit the parent's context.
2. Pass it to extractBible as context: { work: { title, kind: sourceKind, wordCount } } — no prompt change needed for those three slots, they already exist and currently render empty.
3. Add a rendered-only-when-present context block to both prompt families:
   • writers-room-characters.md / -places.md / -objects.md: a {{#factual}}…{{/factual}} section stating the text is first-person non-fiction about real people and places; that the model must NOT invent role-tag names; that a person referred to by relation keeps that exact word as name (Mom, not MOM); and that physicalDescription may stay empty rather than be invented — the file's existing "a wrong guess is worse than a gap" rule, extended to the physical fields under the factual lens.
   • catalog-ideas-scenes-concepts.md: a {{#factual}} section reframing the three kinds for lived material — an idea is a reflection or theme the author is circling, a scene is a remembered moment, a concept is a recurring pattern, family rule, or belief. Plus a ## Source block carrying {{scrapTitle}} / {{sourceKind}} that renders under both lenses.
4. Under the factual lens, stamp a real-person tag on extracted characters and a factual tag on every row. Apply it in the sanitizer from the context flag — not via the type registry's defaultTags (server/lib/catalogTypes.js), which is per-type, not per-run.
5. Prompt-template migration is mandatory — scripts/setup-data.js only copies missing templates, so existing installs keep the old ones. Use makePromptReplaceMigration (scripts/migrations/_lib.js:485) and export ACCEPTED_OLD_MD5 / NEW_SHIPPED_MD5 from the migration. No manual edit to setup-data.js is needed: buildPromptDriftTables sweeps those exports out of every migration at scripts/setup-data.js:158 and builds the drift table itself. (scripts/migrations/048-catalog-ideas-scenes-concepts-stage.js is the wrong precedent to copy — it uses makeSeedMigration, which only copies missing files. server/AGENTS.md's "mirror both hashes into setup-data.js" line is stale against _lib.js:1259; fix that line in the same PR.)

Phase 2 — Bind the ingest to a universe

1. Add universeRef (+ optional role) to catalogScrapCommitSchema (server/lib/catalogValidation.js:407), mirroring catalogBulkImportSchema.defaults (:449-462) — same field name, same z.string().trim().min(1).max(120).optional() shape, so the two ingest paths cannot drift. Forward both from the commit route (server/routes/catalog.js:163-182) into commitScrap.
2. linkIngredientToRef and linkIngredientRelation currently run pool-level query(...) with no { client } option (server/services/catalogDB/refs.js:46, :180), so they cannot join commitScrap's withTransaction. Add the { client } = {} option to both, exactly as linkIngredientToSource already has it (refs.js:20). Without this, a mid-batch failure leaves ingredients committed with refs half-written.
3. Add universeRefRoleForType(type) to refs.js beside the existing seriesRefRoleForType (:69) and cdRefRoleForType (:95): character → canon-character, place → canon-place, object → canon-object, everything else (including idea / scene / concept and user-defined types) → reference. The canon-<kind> convention is what catalogCanonProjection.js and catalogUniverseTags.js already expect.
4. Inside commitScrap's existing transaction (server/services/catalogDB/commit.js:16), after each createIngredient, call linkIngredientToRef(ingredient.id, 'universe', universeRef, role || universeRefRoleForType(draft.type), { client }).
5. Mint related-to edges between the ingredients committed from one scrap in one batch, so a single source's extractions form a connected cluster instead of N isolated nodes. Deterministic direction: one edge per unordered pair, from_id = the lexicographically smaller id — the PK is (from_id, to_id, kind), so an arbitrary direction would let a re-commit create a reciprocal duplicate. Bounded: skip edge minting entirely when a batch exceeds 25 accepted rows (the schema allows 200, which would be 19,900 edges).
6. Client: a "Catalogue into" <select> on the Ingest form covering all source modes (client/src/pages/CatalogIngest.jsx), listing live universes plus an explicit Unassigned option that preserves today's behavior exactly. Default: Reality for brain-bridge and voice-memo (captured thought is factual by default), otherwise the last-used universe read through safeReadStorage (client/src/lib/safeStorage.js) falling back to Reality. When the selected universe is factual, show a one-line hint that real people and places will be extracted as themselves. Follow client/src/AGENTS.md for htmlFor/id pairing and mobile layout.

Phase 3 — Ship the Reality universe

1. Add factual: boolean to the universe record in sanitizeTemplate (server/services/universeBuilder/sanitize.js:683-790), persisted only when true so every existing record keeps its on-disk shape and wire checksum.
2. Bump universes 11 → 12 in server/lib/schemaVersions.js:82, with the rationale inline: a ≤v11 peer strips factual, then LWW's the stripped record back and silently reclassifies the user's Reality universe as fiction. Update the three assertions in server/lib/schemaVersions.test.js (:27, :64, :71).
3. Seed migration scripts/migrations/NNN-reality-universe-seed.js: a universe with the deterministic id universe-reality (matches UNIVERSE_ID_RE = /^[A-Za-z0-9-]{8,80}$/, sanitize.js:51), name: 'Reality', factual: true, ephemeral: false, and empty logline / premise / styleNotes. Shipping invented copy for the user's own life is wrong, and an LLM call at migration time is barred by the AI Provider Usage Policy.
   • Deterministic id so a user's federated machines converge on ONE Reality by LWW instead of accumulating one per install.
   • Gate on absence INCLUDING tombstones. insertUniverseWithId already exists (server/services/universeBuilder/crud.js:319) and takes a caller-supplied id — but it deliberately overwrites a tombstone (wasResurrection at :326-334), which is right for share-bucket re-import and wrong here. So the migration reads await store().loadOne('universe-reality') first and no-ops when any row exists, deleted true or false, only calling insertUniverseWithId when there is none. A user who deleted Reality must not have it resurrected on the next upgrade. Do not add a resurrect: false option — the check belongs in the migration, not in a shared importer path.
4. Surface factual in the Universe Builder settings panel as a "This is a real-world universe" toggle, and badge it in the universe list.

Phase 4 — Make the source visible

Do not invent an autobiography record type. The catalog already has two grouping seams — refs (universe / series / work) and sources (scrap) — and a third parallel taxonomy is exactly what makes the catalog feel arbitrary. Promote the scrap that already groups these rows:

1. GET /api/catalog/ingredients/:id/details (server/routes/catalog.js:239-264) returns source rows as { scrapId, span, extractedAt } with no title. Join catalog_scraps.title onto them, and add the sibling extractions for each source scrap via listSourcesForScrap (server/services/catalogDB/refs.js:38) so the detail page needs no second round-trip.
2. client/src/pages/CatalogIngredient.jsx:1438 currently renders the source as a bare cat-scrap-<uuid> monospace string. Render the scrap title as a link and add a "From the same source" list of sibling ingredients.
3. Add scrapId to catalogIngredientQuerySchema (server/lib/catalogValidation.js), filter on it through catalog_ingredient_sources in listIngredients (server/services/catalogDB/ingredients.js), and wire ?scrap=<id> into client/src/pages/Catalog.jsx — selection lives in the URL, per client/src/AGENTS.md, so "everything extracted from this piece" is one click and one shareable link.

Acceptance criteria

• Committing a scrap with universeRef set creates one live catalog_ingredient_refs row per ingredient with the role universeRefRoleForType returns; HAS_ANY_HOMING_REF is true for all of them and none bucket as unlinked/Raw.
• server/routes/catalogParsedBodies.test.js (which already covers POST /scraps/:id/commit with a mocked commitScrap) asserts universeRef / role are forwarded, and a DB-backed test in server/routes/catalog.test.js asserts the refs and relation edges actually land — plus that a mid-batch failure rolls back ingredients, refs, and edges together.
• Omitting universeRef reproduces today's behavior exactly (source link only, no refs, no relations) — a regression test pins this.
• Same-batch related-to edges are one per unordered pair with from_id lexicographically smaller; re-committing the same batch creates no duplicate or reciprocal edge; a >25-row batch mints none.
• extractIngredientsForScrap passes the parent scrap's title and source kind into both prompt families; a test asserts the rendered character prompt's ## Work being analyzed block is non-empty on the catalog path and that runStagedLLM receives scrapTitle.
• Under the factual lens the character prompt carries the no-role-tag instruction and the light-stage prompt carries the lived-material definitions; a contract test pins both sections' presence, and their absence under the fiction lens.
• The prompt-template migration is built with makePromptReplaceMigration, upgrades an installed template that still matches an accepted old hash, leaves a user-customized template untouched, and needs no hand-edit to setup-data.js (buildPromptDriftTables picks the exports up). The stale "mirror both hashes" line in server/AGENTS.md is corrected.
• A fresh install ends up with exactly one live universe named Reality, id universe-reality, factual: true. Re-running the migration is a no-op. An install where the user soft-deleted Reality does not get it back — covered by a migration test with a tombstoned row present.
• PORTOS_SCHEMA_VERSIONS.universes is 12; server/lib/schemaVersions.test.js and server/lib/db.ddlParity.test.js pass, and the existing version-gate coverage guards still hold.
• Ingest UI: the "Catalogue into" select is labelled (htmlFor/id), keyboard-reachable, defaults as specified, persists the last choice through safeReadStorage/safeWriteStorage, and renders correctly at phone width.
• Ingredient detail shows the source scrap's title and its sibling ingredients; /catalog?scrap=<id> filters to that scrap's extractions.
• docs/STORAGE.md and the catalog docs describe the universe binding and the shipped Reality universe.

Out of scope

• Resolving Mom / Dad to brain people records. The bridge already reaches brain people (server/services/catalogIngestSources.js:238-244), but entity resolution against a real-person registry is its own design problem — matching, disambiguation, and the privacy rules that govern records. Phase 1 only stops the extractor from minting fictional role tags. File separately.
• Resolving scene.actors[] strings to ingredient ids (server/services/catalogExtraction.js:115). Same follow-up.
• Auto-ingesting brain records into the catalog on a schedule — the bridge stays user-triggered (ingestFromBrain has exactly one caller, the route).
• Per-chunk source provenance: commitScrap links to the parent scrap only and the extractor never emits span. Pre-existing, unchanged here.
• Reading catalog tables from universeGraph.js / brainGraph.js, and including relations in exportSliceForRef (the existing [catalog-ingredient-relations] TODO at server/services/catalogDB/facets.js:60).
• Any change to universeBuilderPromote.js, which bypasses the catalog entirely.

Decomposed into

Phase 1 shipped in #7614 (Refs, not Closes) — the extractor now carries a non-fiction lens, the four prompts gained gated {{#factual}} sections, and the scrap source-kind registry moved to a leaf module that declares each path's lens. The remaining phases are tracked as children; this issue closes when the last one does.

• Phase 1 — thread scrap context into extraction (give catalog ingest a non-fiction lens so real people stop becoming role tags #7614)
• bind catalog ingest to a universe so committed ingredients stop landing unlinked #7615 — Phase 2: bind the ingest to a universe
• ship a Reality universe and give the universe record a factual flag #7616 — Phase 3: ship the Reality universe and the factual flag
• show a catalog ingredient's source scrap by title and let one click list its siblings #7617 — Phase 4: make the source scrap visible

[atomantic]

Shipping Phase 1 only in this claim, as the issue itself directs ("Phase 1 is the self-contained slice … if the PR gets large, ship Phase 1 with Refs #N and file Phases 2–4 as follow-ups"). The PR uses Refs #7609; Phases 2–4 are filed as linked follow-ups and this issue stays open until they land.

One decision the issue left open, recorded here: Phase 1 says the factual flag is derived "from the parent scrap (and, once Phase 2 lands, from the target universe’s factual flag)" but does not say which scrap property carries it before Phase 2. Chosen: the scrap’s sourceKind — voice-memo and brain-bridge are first-person captured thought by construction, and Phase 2 of the issue already names exactly those two kinds as the ones that should default to the Reality universe. paste / url / file keep the fiction lens until the user picks a factual universe in Phase 2, at which point the universe flag becomes the primary signal and the source kind the fallback. That keeps every existing extraction byte-identical while fixing the two paths where the material is known to be lived.