feat: add auto-generated "See also" links to docs pages - #202
Open
zoltan-baba wants to merge 5 commits into
Open
feat: add auto-generated "See also" links to docs pages#202zoltan-baba wants to merge 5 commits into
zoltan-baba wants to merge 5 commits into
Conversation
Embeds every docs page locally (sentence-transformers, no external API calls) and surfaces the top related pages by cosine similarity, capped per source directory so tightly-clustered sections (e.g. Bazel/Gradle build-cache pages) don't crowd out cross-cutting matches. Generated API reference docs and top-level product landing pages are excluded from the embedding corpus entirely. Writers can override or exclude specific picks per page via `see_also`/`see_also_exclude` frontmatter. Renders via a DocItem/Paginator swizzle so it appears automatically between the article content and the Previous/Next buttons on every page, without touching individual content files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Storing a page's title/permalink at generation time broke under the repo's new i18n setup: a locale-prefixed permalink baked into see_also.json only resolves correctly on the locale that happened to generate it, and doubles up (/ja/en/...) on every other locale, since Docusaurus's <Link> prepends whichever locale is currently rendering. see_also.json now maps each page's doc id to a list of related doc ids only. <SeeAlso> resolves the title and locale-correct permalink live, per render, from Docusaurus's own per-locale doc data -- so the same relatedness graph serves every locale correctly without regenerating per locale, and a related id that no longer exists (page deleted or renamed since the last regeneration) is silently dropped instead of rendered as a dead link. generate_see_also.py is also repinned to `npx docusaurus build --locale en` instead of the multi-locale `npm run build`: building every locale in one invocation leaves the Docusaurus cache reflecting whichever locale happened to build last, making the embedding corpus's structural data depend on build order rather than being deterministic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Companion to the scheduled full regeneration: dropping a dead reference, or updating one to match a rename, doesn't require re-embedding anything, so this is pure git-diff + JSON bookkeeping with no numpy/torch/sentence-transformers dependency -- cheap enough to run on every PR that touches docs/, not just the ones a full regeneration is scheduled for. Reads `git diff --name-status -M` from stdin (same format translate_docs.py already reads changed files from). A deleted page's id is dropped from its own entry and from every other page's related-ids list; a renamed page's id is remapped everywhere instead of dropped, since the content didn't change. Without -M, a rename falls back to being treated as a plain delete rather than doing nothing. Includes stdlib unittest coverage for the id derivation, diff parsing, and prune/remap logic -- the exact edge cases (delete, rename, the no -M fallback, malformed input, excluded API-reference dirs) verified by hand while building this. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three workflows, each scoped to a different cost tier: - prune-see-also.yml: runs the cheap prune script on every PR touching docs/, pushing the correction back onto the same PR branch so a page deletion/rename and its See also fix land together. Skips cleanly on fork PRs, which get a read-only GITHUB_TOKEN and can never push back here -- their changes fall back to the scheduled regeneration below, which is safe since a stale id is dropped at render time rather than rendered as a dead link. - regenerate-see-also.yml: the actual re-embedding is real cost (sentence-transformers/torch), so a cheap gate job counts docs files changed since the last regeneration (scripts/see_also_sync_state.json) and only lets it run once that reaches 40 -- measured against this repo's real edit history to land at roughly once a week, erring towards fewer/cheaper runs since staleness here is safe, not just tolerated. Installs the CPU-only torch wheel first (no GPU on this runner) so the subsequent requirements install doesn't pull the much larger CUDA build, with an extra-index fallback in case a future version bump forces torch to be re-resolved. Opens a PR with the regenerated data and the updated sync state, reporting exactly how many files and since which commit triggered it. - test-see-also.yml: runs scripts/test_prune_see_also.py whenever the pipeline's own code changes (not on ordinary docs content, which never touches it) -- a fast, dependency-free regression check distinct from the two workflows above, which exercise the pipeline against real data rather than testing its logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
package.json already picked up docusaurus-plugin-image-zoom and docusaurus-plugin-llms from main; npm install brings the lockfile back in sync with it.
zoltan-baba
requested review from
aorcsik,
ilanazholobovsky and
matenadasdi
as code owners
August 27, 2026 09:25
ilanazholobovsky
approved these changes
Aug 28, 2026
ilanazholobovsky
left a comment
Collaborator
There was a problem hiding this comment.
Read all 13 code files; verified on the preview that the section renders correctly on both /en/ and /ja/ (same id-only graph, locale-correct permalinks, no doubled prefixes). The id-only design and the render-time drop hold up exactly as described. Approving — three non-blocking notes, in descending order of importance:
- prune-see-also.yml diffs against the base tip, not the merge-base.
git diff -M "$BASE" HEADwithBASE = github.event.pull_request.base.shais a two-dot diff against main's current tip, so main-side changes since the branch point leak in reversed: a page added on main shows up asD(usually a no-op here, since the branch's see_also.json predates that id), but a page renamed on main shows up asRand gets remapped in the PR's stale copy — a committed change that then conflicts with main's own see_also.json. Three-dot fixes it:git diff --name-status -M "$BASE...HEAD" -- docs. The test suite can't catch this one — it sits above the parse_changes boundary. - The bot's
[skip ci]commit becomes the PR head with no check runs. If required checks are ever enforced on this repo, a pruned PR can hang on "Expected" statuses, since the head commit skipped everything. Worth adding to the "watch the first real runs" list from the test plan. see_also/see_also_exclude(and the bare locale-less path form) are only documented in generate_see_also.py's docstring. CLAUDE.md is where authors and agents will actually look — the same gap #203 just closed for diff blocks. Fine as a follow-up ticket rather than in this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
see_also/see_also_excludefrontmatter.see_also.jsonstores only a relatedness graph of doc ids, never a baked title/permalink. This is what makes the same data correctly serve both/en/and/ja/without regenerating per locale, and it's why a stale or removed reference is silently dropped at render time instead of ever becoming a dead link.Test plan
npx tsc --noEmitpasses — no type errors in theSeeAlsocomponent's Docusaurus hook usagepython3 scripts/test_prune_see_also.py— 14/14 passing (id derivation, diff parsing, delete/rename/no--M-fallback edge cases)see_also.jsonfrom a cleannpx docusaurus build --locale enand confirmed the id-only output shape/en/and/ja/builds — no doubled locale prefixesregenerate-see-also.ymlactually avoids reinstalling a CUDA build