Skip to content

feat: add bounded exact bookmark materialization - #181

Open
briancl2 wants to merge 19 commits into
afar1:mainfrom
briancl2:codex/researcher-exact-id-materialization-thin
Open

feat: add bounded exact bookmark materialization#181
briancl2 wants to merge 19 commits into
afar1:mainfrom
briancl2:codex/researcher-exact-id-materialization-thin

Conversation

@briancl2

@briancl2 briancl2 commented Aug 11, 2026

Copy link
Copy Markdown

Summary

  • add ft materialize <id> --json for one exact archived X bookmark
  • optionally refresh only that root, its bounded parent chain, every enumerated same-author continuation branch, and its quote through the existing signed-in GraphQL route
  • optionally fetch only media enumerated from that exact in-memory record
  • emit source/component depth, exact media hashes, and explicit unresolved/unavailable gaps without persisting a second thread corpus

Why this shape

Research systems need more than the bookmarked root: thread continuations, quote context, X Articles, outbound identities, and attached media can carry the decision-material content. This provides that source-owner boundary while remaining query-time and local-first.

Unlike a corpus-wide thread sync, this change adds no database migration, FTS column, scheduler, background job, classification, or LLM. --refresh does not modify the bookmark archive or index. Destination pages/PDFs and OCR/transcript/visual interpretation remain explicit gaps for their source owners rather than inferred content.

This is a deliberately thinner complement/alternative to #141 and addresses the exact-id use case behind #59. The TweetDetail parser and same-author continuation logic are based on Eric Litman's work in #141; the commit retains co-author attribution.

Safety

  • exact numeric archived ID required
  • no corpus-wide selection or persistence
  • source-local file paths never appear in output
  • authenticated refresh fails closed if no X session is available
  • public replies are excluded from same-author thread continuations
  • every recognized bottom/show-more cursor is queued within bounded pagination; unvisited branches keep enumeration incomplete
  • tweet-free or focal-tweet-free timelines remain incomplete; malformed TweetDetail bodies become partial observations
  • the configured delay applies between every exact X request
  • previews and summaries cannot be admitted as recovered X Article bodies
  • equivalent recovered X/Twitter Article aliases are deduplicated; ambiguous multiple-Article identity fails closed with every locator explicit
  • a current X Article identity without recovered body keeps the result partial
  • ordinary materialization overlays article and quote enrichment already retained in the existing index
  • hop-2 outbound identities are material but never mislabeled as mandatory-direct
  • unavailable roots preserve the archived record and expose the failure

Validation

Exact candidate head: 26fdc2d263b40f7940347cd22eb785ed9c75fc30 (tree 5c70c3af2f95905d43eb2b25e113838a26dbec37).

  • npm run build — PASS
  • focused changed-surface suite — 136/136 PASS
  • full npm test — 593/593 PASS
  • git diff --check — PASS
  • all ten inline review threads have exact-head replies and are resolved
  • the disjoint BMA compatibility audit found and repaired the hop-2 mandatory-direct mismatch before cutover

Fresh Codex review is requested on the current head. Upstream merge remains maintainer-owned, and downstream plans assume no merge for at least 48 hours.

briancl2 and others added 5 commits August 11, 2026 12:24
Provide one bounded non-LLM exact-ID source operation without bulk thread sync, database migrations, or persistence. Reuse the TweetDetail parsing work from upstream PR afar1#141.

Co-authored-by: Eric Litman <28211+ericlitman@users.noreply.github.com>
Repair pagination, exact-ID, nested-media, asset-binding, and article-currentness boundaries found in exact-head review.
Reject identity-less responses and fail closed on unsupported or unbound TweetDetail continuation cursors.
Recognize prefixed continuation cursors and replacement-entry delivery while failing closed on unknown cursor shapes.
Inspect entry arrays independent of instruction name so unknown cursor delivery cannot imply exhaustion.
@briancl2
briancl2 marked this pull request as ready for review August 11, 2026 17:58
@briancl2

Copy link
Copy Markdown
Author

Final repaired head: 189cc1de9a64714782cc21ddbe91f05c80da982e (tree d0e74d7e92c7b94494cebfbf67931df95b475b26).

Validation:

  • npm run build — PASS
  • focused exact-ID/materialization/media suite — 62/62 PASS
  • full npm test — 581/581 PASS
  • git diff --check — PASS
  • fresh exact-head CRITICAL/HIGH review — CLEAN

The review cycle repaired and froze fail-closed boundaries for pagination/cursor exhaustion, response-owned exact IDs, parent/continuation media, URL-keyed poster/video assets, and stale X Article currentness. This remains a bounded foreground read-only source-owner operation; it adds no bulk persistence, database migration, LLM, scheduler, or background research behavior.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 189cc1de9a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/tweet-snapshots.ts
Comment thread src/x-materialize.ts Outdated
Comment thread src/bookmark-materialization.ts
@briancl2

Copy link
Copy Markdown
Author

Superseding exact-head settlement: 24f99a4a6aaa920c6fa087e2591a2154dff4eecf (tree cac7d0f94394f37071f248c4e9e289a599bb5811).

The three findings from the prior review are repaired:

  • every supported bottom/show-more cursor is queued within bounded pagination, and any unvisited branch keeps enumeration incomplete;
  • a recognized tweet-free timeline remains partial;
  • a recovered X Article is not also emitted as an unresolved outbound gap.

Validation is green: build, 66/66 focused tests, 586/586 full tests, diff hygiene, and a fresh exact-head CRITICAL/HIGH review with no findings. No source-local path leakage was found. The PR remains a bounded, foreground, non-LLM source-owner operation; upstream merge remains maintainer-owned.

@briancl2

Copy link
Copy Markdown
Author

@codex review

Please review exact head 8fb4b6890ee82d60a86c966652946f21fbf8a2a6. In addition to the prior thread fixes, this head fails closed when a current root identifies an X Article but its body is unavailable, and suppresses every equivalent recovered X/Twitter article locator from unresolved outbound gaps. Build, focused 67/67, full 586/586, and diff hygiene pass.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8fb4b6890e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/x-materialize.ts Outdated
Comment thread src/x-materialize.ts Outdated
Comment thread src/bookmark-materialization.ts
@briancl2

Copy link
Copy Markdown
Author

@codex review

Please review exact head 6b85d6e84c67c1daac8cc3b865c192760867f730. The current head includes the prior cursor, empty-timeline, and X Article repairs, plus a contract-alignment fix: hop-2 outbound components are now mandatory_direct=false, matching the consuming BMA contract rather than being rejected at cutover. Build, focused 67/67, full 586/586, and diff hygiene pass.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b85d6e84c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/graphql-bookmarks.ts Outdated
Comment thread src/x-materialize.ts
@briancl2

Copy link
Copy Markdown
Author

@codex review

Please review exact head 0dee3f98370adee0f4979d96b0c116c99f75ba78 (tree 20eef9765b19a27f963f9e5a84df19af79bcc996). This supersedes prior heads and repairs all six review threads plus the BMA hop-2 materiality mismatch found in the cross-repo compatibility audit.

Current evidence: build PASS; focused materialization/GraphQL/refresh suite 104/104 PASS; full suite PASS; diff hygiene clean. Please focus on article full-body admission, focal-tweet traversal binding, distinct-vs-equivalent article locators, cursor completeness, and the source-owner/BMA component contract.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0dee3f9837

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli.ts Outdated
Comment thread src/bookmark-materialization.ts Outdated
@briancl2

Copy link
Copy Markdown
Author

@codex review

Please review exact head 26fdc2d263b40f7940347cd22eb785ed9c75fc30 (tree 5c70c3af2f95905d43eb2b25e113838a26dbec37). This supersedes prior heads and repairs all ten review threads plus the BMA hop-2 materiality mismatch found by the cross-repo compatibility audit.

Current evidence: build PASS; focused changed-surface suite 136/136 PASS; full suite 593/593 PASS; diff hygiene clean. Please focus on raw-archive/index enrichment overlay, malformed TweetDetail fail-closed behavior, request throttling, ambiguous Article identity, full-body admission, focal/cursor completeness, and the source-owner/BMA component contract.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 26fdc2d263

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/cli.ts Outdated
Comment thread src/x-materialize.ts Outdated
Comment thread src/x-materialize.ts Outdated
Comment thread src/x-materialize.ts Outdated
@briancl2

Copy link
Copy Markdown
Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a418689fe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/x-materialize.ts
Comment thread src/bookmark-materialization.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant