Add a translation pipeline

[stevhliu]

This PR adds a translate command to translate docs into other languages. There isn't very good coverage for the other languages, they fall out of sync with the English docs, and are difficult to keep up with and maintain because most times we can't review translation PRs.

Design

A nightly scheduled Job under hf-doc-build runs doc-builder translate transformers --lang ja --bucket /bucket which clones Transformers and reads docs/source/en/**. It masks things that shouldn't be translated - like specific doc-builder syntax - and segments the docs. Translated paragraphs are stored in a Bucket, hf-doc-build/doc-translate/cache and reused nightly to compare whether existing changes have been made. If there are no changes, it exits.

If something is new, it loads a model to only translate the prose that's changed using Transformers' native continuous batching. The finished .md files are written to a new Bucket, hf-doc-build/doc-translate/translations.

On the Transformers side, write build_ja_documentation.yml to call on the doc-builder's shared build workflow to sync the translations. It downloads the finished files and swaps them in. doc-builder build uses the existing page-level caching to only build pages whose content changed and then push to hf-doc-build/doc-build.

┌─ 03:00 UTC ─ HF Jobs, hf-doc-build namespace, one a100-large ────────────────┐
│                                                                              │
│  doc-builder translate transformers --lang ja --bucket /bucket               │
│    (bucket mounted as a folder, HF_TOKEN as a secret)                        │
│                                                                              │
│  1. clone transformers, read docs/source/en/**        732 pages              │
│  2. hide code/tags/URLs, split into paragraphs, hash   14,829 IDs            │
│  3. compare against cache/index.json                   ← ONE file read       │
│                                                                              │
│       nothing new → exit here, model never loads       ~$0.15  ⟵ most nights │
│       something new ↓                                                        │
│                                                                              │
│  4. load Gemma, translate ONLY the new paragraphs                     │
│  5. check each page, write the good ones to the bucket                       │
│        failures keep their last good version, or English                     │
└──────────────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─ buckets in hf-doc-build ────────────────────────────────────────────────────┐
│  doc-translate/cache/           translated paragraphs, reused nightly   NEW  │
│  doc-translate/translations/    finished .md files the build reads      NEW  │
│  doc-build-cache/               prerendered HTML                     exists  │
│  doc/transformers/              what hf.co/docs serves               exists  │
└──────────────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─ 04:00 UTC ─ GitHub Actions, huggingface/transformers ───────────────────────┐
│  build_ja_documentation.yml → doc-builder's shared build workflow            │
│                                                                              │
│  A. sync translations/ → docs/source/ja   ← the new step in this PR          │
│       replaces the folder wholesale, clearing the 20 stale orphans           │
│  B. doc-builder build --language ja --html   (page cache skips unchanged)    │
│  C. hf sync → doc/transformers,  push → doc-build dataset                    │
└──────────────────────────────────────────────────────────────────────────────┘
                                    ↓
                        hf.co/docs/transformers/ja

Preview

# get this branch
git checkout doc-builder/ja-translation

# set up preview env
uv venv .venv-tr
VIRTUAL_ENV=.venv-tr uv pip install -e . \
  "transformers @ git+https://github.com/huggingface/transformers"

# get read access to the `hf-doc-build` org
hf auth login

# get latest translation from the Bucket
hf buckets sync \
  hf://buckets/hf-doc-build/doc-translate/translations/transformers/ja \
  /tmp/ja-preview

# start and open the preview
git -C /tmp/ja-preview init -q
./.venv-tr/bin/doc-builder preview transformers /tmp/ja-preview --language ja
open http://localhost:5173/index

[coyotte508]

🤖 Automated bot review — this review was generated by an AI agent (moon-coder). It may contain mistakes; please verify findings before acting on them.

This PR adds a doc-builder translate command — a nightly, cache-aware machine-translation pipeline (mask → segment → translate via continuous batching → validate → publish to a bucket) plus a sync step in the shared build workflow to swap translated pages into the docs build. The design is thoughtful, the comments explain real production incidents, and the non-GPU logic has genuinely strong test coverage; no outright bugs found, but several gaps worth addressing.

Findings

• 🟡 Deleted English pages are never pruned from the bucket — run() in src/doc_builder/commands/translate.py only writes pages (write_page); nothing removes files from translations/<pkg>/<lang> when the English original disappears. The workflow comment ("clears the 20 stale orphans") only holds for repo-side orphans: future orphans accumulate in the bucket, get synced into docs/source/<lang> wholesale, and are published unlisted forever. Consider deleting out-dir files not in the current page set.
• 🟡 A --pages-file smoke run against the real bucket clobbers production — run() writes the pruned _toctree.yml (via prune_toctree) to the same out_dir a full run uses. Someone pointing a 3-page smoke run at the production bucket replaces the full sidebar with a 3-entry one until the next full run. Guard it (e.g. refuse --pages-file without an explicit non-default --bucket, or write to a scratch prefix).
• 🟡 Exit code ignores page-level failure rate — the command's own docstring says "the exit code is the only way anyone finds out something went wrong," yet run() returns 0 even if e.g. 90% of pages fail validation and fall back to English; validate.summarize() only prints a warning above 2%. Consider a non-zero exit (or configurable threshold) so the nightly job goes red.
• 🟡 Unpinned transformers in the translate extra — pipeline.translate_segments() depends on v5 continuous-batching internals (ContinuousBatchingConfig, continuous_batching_context_manager, transformers.generation.continuous_batching.utils.WorkloadHints — a private-ish import path), but the extra declares bare "transformers". Pin a minimum version; kernels got one, transformers deserves one more.
• 🟡 Glossary keep: terms never reach the model — build_prompt() only injects pin entries; keep (ja.yml: "Words to leave in English") is only checked after the fact as a warning in check_glossary(). If that's intentional (rely on masking + warning), the comment in glossaries/ja.yml oversells what keep does; otherwise add them to the prompt.
• 🟡 Language-list format mismatch in the workflow — the existing languages input is parsed with IFS=', ' (commas allowed, build_main_documentation.yml L211), but the new sync loop does plain word-splitting on $TRANSLATED_LANGUAGES. A caller who copies "ja, ko" into both inputs gets a bogus ja, bucket path (it fails loudly at the _toctree.yml check, at least). Normalize commas, and consider enforcing the documented "must also appear in languages" invariant instead of just stating it.
• 🟡 The "load-bearing" tests never run in CI — test_translate_segment.py / test_translate_pipeline.py call the corpus round-trip "the load-bearing test in this project," but EN_DOCS defaults to a personal path (/Users/steven/hf/transformers/docs/source/en) and skips everywhere else. Vendor a small fixture corpus (a dozen representative pages) so a slice of the round-trip runs on every CI run, and drop the personal path default.
• 🔵 Missing Apache license headers on translate/cache.py, translate/pipeline.py, translate/validate.py, and all four new test files — segment.py, translate/__init__.py, and commands/translate.py have them, and every existing module in the repo carries one.
• 🔵 select_pages() filters comments with line.startswith("#") on the unstripped line, so an indented # comment in a pages file becomes a "missing page" warning.
• 🔵 LANGUAGE_NAMES only knows "ja"; any other --lang silently injects the raw code into the prompt ("from English into ko"), which will degrade quality with no warning.
• 🔵 A swapped link pair (¤1¤label¤0¤) passes the per-paragraph sorted marker check in assemble_page() and is only caught page-level by check_links() — so one such paragraph rejects the whole page, defeating the per-paragraph containment the code works hard for elsewhere. Checking pair order for link_open/link_close markers per paragraph would keep the blast radius small.
• 🔵 run() has no per-page try/except around assemble_page/validate_plan; an unexpected exception on one page after GPU spend aborts the write-out of every remaining page — at odds with the "one bad page costs itself" philosophy.
• 🔵 test_translate_pipeline.py module docstring references plan_page, which doesn't exist (it's PagePlan); the new workflow step also omits shell: bash where sibling steps declare it.

Questions: is google/gemma-4-26B-A4B-it gated, and does the job's HF_TOKEN have access? Does hf sync (bucket → staged dir) delete extraneous destination files, i.e. does the workflow-side swap actually mirror? Is os.replace() atomic on the bucket FUSE mount that SegmentCache relies on for its tmp-then-rename writes?

Verdict: Well-engineered and unusually well-tested for a pipeline of this kind — no blockers, but the orphan-cleanup, smoke-run-toctree, and exit-code items should be resolved (or explicitly deferred) before merge.

[mishig25]

The overall design is promising, but the inline findings below identify cases that can silently publish stale, missing, or corrupted documentation. I reproduced the warm-cache stale-output path, silent paragraph deletion, and successful exit after total translation failure; corpus testing exposed the fence and link cases. Please address these before enabling the nightly pipeline.

Local verification: 1,687 tests passed with the current Transformers corpus, and lint/formatting passed. uv lock --check currently fails. The actual GPU model and bucket-to-build path were not exercised. Follow-ups worth covering include $...$ math, reference/bare URLs and image structure, safe --pages-file output, .mdx support, glossary keep: behavior, a minimum Transformers version plus regenerated lockfile, and running representative corpus tests in CI.

[stevhliu]

Thanks, I pinned transformers >=5.8.0 and regenerated uv.lock! Here's a brief summary of the requested changes:

• Updated the masking logic to accommodate more backticks, mask link URLs with parentheses in them, and mask reference-style links as well ([text][ref])
• Updated the prompt (used ⟦0⟧ but the text actually uses ¤0¤) to use the same markers the text actually contains.
• Refuse empty translations which used to pass the checks.
• Link check compares the actual URLs instead of counting links.
• Updated the publishing workflow as well:
  • build and check the docs in memory first before writing to bucket
  • docs whose original English source was deleted are removed from the bucket
  • new manifest that tracks what each translated doc is built from, which is what lets the Job detect a code-only edit, a deleted paragraph, etc. (none of which the segment cache could see before)
  • if more than 25% pages fail their checks, nothing is published and the Job exits with an error
  • if less than 2% pages fail, the docs are published but it exits with a non-zero value on the nightly Job to flag a look
• Added real docs to tests/fixtures/translate_corpus/ so the masking tests run in CI

[mishig25]

Thanks for the follow-up. The fence handling, source/output reconciliation, prompt marker fix, empty-output guard, balanced destinations, and full reference-link support are all materially improved. The focused translation suite passes (123 passed, 5 skipped), and the current GitHub checks are green. However, the inline cases below can still delete or expose an incomplete live tree, turn a failed update into a successful English fallback, or silently corrupt protected content. I reproduced each case against this head, so I am keeping the review at changes requested before the nightly pipeline is enabled.

[mishig25]

could you resolve the conflicts?

[mishig25]

Thanks for the follow-up. The disclosure-aware fallback, direct nested-image/bare-URL/inline-math cases, empty and unreadable source guards, and normal versioned publish path are materially improved. The focused translation suite passes (162 passed, 5 skipped), and 3,772 current-corpus cases pass. However, I reproduced each inline issue below against this head; they can still publish an incomplete or English fallback tree, corrupt or delete the pointed generation under repair/overlap, or silently mishandle protected Markdown. I am therefore keeping this at changes requested. GitHub currently shows no check results for this head.

[mishig25]

[P1] Reject sidebar entries whose source page is missing. A full run still derives pages only from files that happen to exist and never checks those pages against the sidebar. If guide.md is absent from an incomplete checkout while _toctree.yml still contains local: guide, this publishes a generation without guide.md, preserves the dangling sidebar entry, and exits 0. Before publishing a full run, resolve every sidebar local to a planned Markdown page and fail closed if any are missing.

[mishig25]

[P1] Repair a damaged live generation under a fresh directory. When generation == current but verification finds damage, this writes directly into the directory named by CURRENT. Files are then replaced one by one, so readers can observe a partial repair; unexpected extra files are not removed either, leaving the generation permanently unverifiable. Build and verify a new unreferenced repair generation, then promote its pointer only after the entire tree is intact.

[mishig25]

[P1] Give each atomic writer a unique temporary path. Every process writing the same destination also writes the same path.tmp. In a deterministic two-writer reproduction, writer A returned success, writer B raised FileNotFoundError, and CURRENT contained B's bytes. Use a unique temporary file in the destination directory, opened exclusively, before os.replace; pointer/manifest publication also needs per-package/language serialization or compare-and-swap semantics.

[mishig25]

[P1] Derive the read directory from the captured pointer. The comment says the pointer is read once, but current_dir(root) reads it again. If another publisher promotes between these calls, current can name generation A while read_dir points to B, so reconciliation and fallback use inconsistent snapshots. Build read_dir from the already captured current, then recheck or serialize before the no-op/promotion decision.

[mishig25]

[P1] Prevent stale publishers from overwriting the manifest. Promotion and manifest publication are not one serialized or conditional transaction: writer A can promote A, writer B can promote and record B, and then A can overwrite the manifest with A even though CURRENT names B. Serialize publishers, or make the manifest generation-specific and select it through the same atomic pointer.

[mishig25]

[P1] Do not rediscover bare URLs from restored Japanese prose. Japanese commonly attaches particles without spaces. A correct restoration such as https://github.com/deepseek-ai/DeepSeek-V3を参照してください is tokenized here as one longer URL, producing a false lost/gained error even though the protected URL placeholder survived exactly. Compare known URL values through placeholder provenance instead of rerunning this permissive regex over translated prose.

[mishig25]

[P2] Preserve the image kind for reference-style links. REF_LINK_RE captures whether the source used !, but every match is recorded as ref. Consequently ![Diagram][fig] can lose !, become [Diagram][fig], and still pass with identical targets. Mask ![ together and record ref-image separately from ref.

[mishig25]

[P2] Avoid interpreting price ranges as inline math. The closing delimiter may be followed by a digit, so US$5 to US$10, $5-$10, and $5/$10 are matched as formulas such as $5 to US$. That silently shields ordinary price prose from translation, and round-trip checks cannot detect it. Tighten the closing-dollar context—at least disallow a following digit—and add price-range fixtures.

[mishig25]

[P2] Signal unsuccessful preview runs. Preview mode reaches this unconditional success return before the rejection-rate status, and the empty-page guard is disabled for previews. A pages file selecting nothing, or a preview with failed pages falling back to English, can therefore write an empty/degraded smoke tree and exit 0. Keep previews isolated, but return nonzero when no requested page exists or the warning threshold is exceeded.

[mishig25]

[P1] Seed the generation layout before enabling this consumer. The existing Transformers/Japanese bucket still uses the flat layout (98 Markdown files plus _toctree.yml) and has neither CURRENT nor generations/. Enabling translated_languages therefore makes this sync fail until a producer has successfully seeded the new format. Please coordinate and verify that migration step before rollout, or retain a temporary legacy fallback.