Skip to content

bind catalog ingest to a universe so committed ingredients stop landing unlinked #7615

Description

@atomantic

Problem / Goal

commitScrap writes ingredients + a source link and nothing else (server/services/catalogDB/commit.js:16-34) — no catalog_ingredient_refs row, no catalog_ingredient_relations edge. So HAS_ANY_HOMING_REF (server/services/catalogDB/shared.js:258, HOMING_REF_KINDS = ['universe','series','creative-director']) is false for every freshly ingested row and they all bucket as unlinked (Raw). That is the "isolated and random" symptom in #7609.

Goal: one user-visible control at ingest — the universe this material belongs to — that links every committed ingredient to that universe and connects a single scrap's extractions to each other.

The infrastructure already exists and already federates. /api/catalog/bulk-import is the prior art in the same file: it already takes defaults.universeRef / seriesRef / workRef / issueRef / role (server/lib/catalogValidation.js:449-462, applied at server/routes/catalog.js:569). The scrap→commit path is the odd one out.

Approach

  1. Add universeRef (+ optional role) to catalogScrapCommitSchema (server/lib/catalogValidation.js:407), mirroring catalogBulkImportSchema.defaults (:449-462) — same field name, same z.string().trim().min(1).max(120).optional() shape, so the two ingest paths cannot drift. Forward both from the commit route (server/routes/catalog.js:163-182) into commitScrap.
  2. linkIngredientToRef and linkIngredientRelation currently run pool-level query(...) with no { client } option (server/services/catalogDB/refs.js:46, :180), so they cannot join commitScrap's withTransaction. Add the { client } = {} option to both, exactly as linkIngredientToSource already has it (refs.js:20). Without this, a mid-batch failure leaves ingredients committed with refs half-written.
  3. Add universeRefRoleForType(type) to refs.js beside the existing seriesRefRoleForType (:69) and cdRefRoleForType (:95): charactercanon-character, placecanon-place, objectcanon-object, everything else (including idea / scene / concept and user-defined types) → reference. The canon-<kind> convention is what catalogCanonProjection.js and catalogUniverseTags.js already expect.
  4. Inside commitScrap's existing transaction, after each createIngredient, call linkIngredientToRef(ingredient.id, 'universe', universeRef, role || universeRefRoleForType(draft.type), { client }).
  5. Mint related-to edges between the ingredients committed from one scrap in one batch, so a single source's extractions form a connected cluster instead of N isolated nodes. Deterministic direction: one edge per unordered pair, from_id = the lexicographically smaller id — the PK is (from_id, to_id, kind), so an arbitrary direction would let a re-commit create a reciprocal duplicate. Bounded: skip edge minting entirely when a batch exceeds 25 accepted rows (the schema allows 200, which would be 19,900 edges).
  6. Client: a "Catalogue into" <select> on the Ingest form covering all source modes (client/src/pages/CatalogIngest.jsx), listing live universes plus an explicit Unassigned option that preserves today's behavior exactly. Default: Reality for brain-bridge and voice-memo (captured thought is factual by default), otherwise the last-used universe read through safeReadStorage (client/src/lib/safeStorage.js) falling back to Reality. When the selected universe is factual, show a one-line hint that real people and places will be extracted as themselves. Follow client/src/AGENTS.md for htmlFor/id pairing and mobile layout.
  7. Once the universe carries a factual flag (ship a Reality universe and give the universe record a factual flag #7616), feed it into the extraction lens landed in #PHASE1 — catalogExtraction.js derives factual from the scrap's sourceKind today and the universe flag becomes the primary signal, with the source kind as fallback.

Acceptance criteria

  • Committing a scrap with universeRef set creates one live catalog_ingredient_refs row per ingredient with the role universeRefRoleForType returns; HAS_ANY_HOMING_REF is true for all of them and none bucket as unlinked/Raw.
  • server/routes/catalogParsedBodies.test.js (which already covers POST /scraps/:id/commit with a mocked commitScrap) asserts universeRef / role are forwarded, and a DB-backed test in server/routes/catalog.test.js asserts the refs and relation edges actually land — plus that a mid-batch failure rolls back ingredients, refs, and edges together.
  • Omitting universeRef reproduces today's behavior exactly (source link only, no refs, no relations) — a regression test pins this.
  • Same-batch related-to edges are one per unordered pair with from_id lexicographically smaller; re-committing the same batch creates no duplicate or reciprocal edge; a >25-row batch mints none.
  • Ingest UI: the "Catalogue into" select is labelled (htmlFor/id), keyboard-reachable, defaults as specified, persists the last choice through safeReadStorage/safeWriteStorage, and renders correctly at phone width.
  • docs/STORAGE.md and the catalog docs describe the universe binding.

Out of scope

  • Resolving Mom / Dad to brain people records, and resolving scene.actors[] strings to ingredient ids — both are entity resolution, filed separately.
  • Reading catalog tables from universeGraph.js / brainGraph.js, and including relations in exportSliceForRef (the [catalog-ingredient-relations] TODO at server/services/catalogDB/facets.js:60).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:brainBrain notes/memories/goals/knowledge grapharea:createUniverse/catalog/Create-suiteeffort:highDispatch effort: highenhancementNew feature or requestmodel:mediumRoutine multi-file workplanTracked by /do:replanplanner:opus-5Plan authored by the opus-5 model

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions