Make scene-spec the default Manim path for all clients

AGENTS.md

@@ -56,7 +56,7 @@ Commands registered on the **`docgen`** CLI include:
 - **`clean-bundle`** — remove regenerable outputs per policy.
 - **`concat`** — stitch segment videos.
 - **`pages`** — emit static HTML for demo assets.
-- **`generate-all`** — orchestrated pipeline: TTS → timestamps → **scene retime** (compile existing `*.scene.yaml` against fresh timing) → images → Manim → compose → validate → concat → pages. Optional `--regen-scene-specs` for LLM scene YAML first.
+- **`generate-all`** — orchestrated pipeline: TTS → timestamps → **scene specs** (auto `scene-spec-generate` when `animations/specs/` is empty; otherwise offline retime) → images → Manim → compose → validate → concat → pages. `--regen-scene-specs` forces LLM rewrite; `--skip-scene-retime` keeps legacy hand `scenes.py` only.
 - **`rebuild-after-audio`** — same as generate-all with TTS skipped (still retimes scenes after timestamps).
 
 ## Implications for changes here

README.md

@@ -38,13 +38,15 @@ If you still need the legacy behaviour, pin a pre-removal commit
   `whisper-1` remains available via `timestamps.engine: whisper` /
   `docgen timestamps --engine whisper`. Both engines write the same
   `timing.json` shape.
-- **Manim animations** — primary visual surface. Use **`docgen scene-spec-generate`**
-  + **`scene-compile`** (or hand-maintained **`animations/specs/*.scene.yaml`**)
-  for deterministic diagram layout: rows are auto-paginated when they exceed the
-  frame stack budget, specs that overflow safe width / budget are rejected, and
-  (when `timing.json` carries Whisper words) each row’s first label is mapped to
-  a **`wait_word`** index. Hand-maintained custom Manim classes still live in
-  `animations/scenes.py` outside the `BEGIN/END GENERATED SCENE` markers.
+- **Manim animations (default: declarative scene specs)** — primary visual surface.
+  Prefer **`animations/specs/*.scene.yaml`** via **`docgen scene-spec-generate`**
+  + **`scene-compile`**. On **`generate-all`**, if no specs exist yet, the pipeline
+  **auto-runs** scene-spec-generate; later runs **retime-compile** specs against fresh
+  `timing.json` (no OpenAI) unless you pass **`--regen-scene-specs`**. Specs auto-paginate
+  when they exceed the frame stack budget, reject unsafe width / subject-beat gaps, and
+  map each paced label to a **`wait_word`** index. Hand-maintained custom Manim classes
+  may still live in `animations/scenes.py` outside `BEGIN/END GENERATED SCENE` markers
+  (use **`--skip-scene-retime`** to bypass the declarative stage).
 - **OpenAI image assets in Manim scenes** — a scene-spec box may be an **image
   element** (`image: images/<name>.png` + `prompt:`); `docgen image-generate`
   renders the prompt via the OpenAI Images API (default `gpt-image-1`) and the
@@ -131,7 +133,7 @@ docgen validate --pre-push
 | `docgen lint [--segment 01]` | Narration lint only |
 | `docgen concat [--config full-demo]` | Concatenate full demo files |
 | `docgen pages [--force]` | Generate `index.html`, `pages.yml`, `.gitattributes`, `.gitignore` |
-| `docgen generate-all [--skip-tts] [--skip-manim] [--retry-manim] [--regen-scene-specs]` | Full pipeline: TTS → timestamps → **scene retime** (existing specs) → Manim → compose → validate. `--regen-scene-specs` also runs OpenAI scene-spec-generate |
+| `docgen generate-all [--skip-tts] [--skip-manim] [--retry-manim] [--regen-scene-specs] [--skip-scene-retime]` | Full pipeline: TTS → timestamps → **scene specs** → Manim → compose → validate. Missing `animations/specs/*.scene.yaml` auto-generates; existing specs retime offline; `--regen-scene-specs` forces LLM rewrite |
 | `docgen rebuild-after-audio [--regen-scene-specs]` | Timestamps → scene retime → Manim → compose → validate (skips TTS) |
 | `docgen clean-bundle [-y] [--delete-config] [--keep-narration]` | Remove regenerable outputs under the bundle |
 | `docgen narration-generate --segment 01 [--extra-path REL] [--hint TEXT] [--dry-run] [--force] [--revise --revision-notes TEXT]` | Generate narration `.md` from repo sources + owner hints (OpenAI); `--revise` edits the existing script in place |

docs/demos/README.md

@@ -40,15 +40,20 @@ Full **`docgen generate-all`** for this bundle needs:
 
 Historic **Playwright** / **demo-function** / **per-function** capture paths were **removed** from the library (see root **`AGENTS.md`**); do not expect browser-test recording commands in current `docgen`.
 
-## Declarative Manim (`animations/specs/*.scene.yaml`)
+## Declarative Manim (`animations/specs/*.scene.yaml`) — **default**
 
-For **`manim`** segments built from labeled `_box` diagrams:
+This is the **default** visual path for all consumers (not optional decoration):
 
-1. **`docgen scene-spec-generate --segment <ID> [--compile] [--hint "…"]`** — optional OpenAI emits **YAML**; **`--compile`** injects into **`animations/scenes.py`**. Use **`--all --compile`** for every declarative segment.
-2. **`docgen scene-compile path/to/spec.scene.yaml`** — compile a spec without another API call.
-3. Then **`docgen timestamps`**, **`docgen manim`**, **`docgen compose`**, and **`docgen concat full-demo`** when refreshing recordings.
+1. First **`docgen generate-all`** (or **`scene-spec-generate --all --compile`**) writes
+   **`animations/specs/<stem>.scene.yaml`** and injects generated classes into
+   **`animations/scenes.py`** between marker comments.
+2. Later **`generate-all`** / **`rebuild-after-audio`** **retime-compiles** those specs
+   against fresh **`timing.json`** (no OpenAI) so beat sync stays correct.
+3. Force a layout rewrite with **`--regen-scene-specs`**. Skip the stage only for
+   legacy hand scenes: **`--skip-scene-retime`**.
 
-Run **`docgen timestamps`** after changing narration audio so **`timing.json`** gains **`words`** / segments required by **`wait_segment`** / **`wait_word`** in specs.
+Label boxes with **spoken phrases from the narration** (must match `timing.json`
+words), keep ~3 rows/page, and follow **`hints/manim-scene-specs.md`**.
 
 ## Full reset (total nuke + regen)
 

docs/demos/hints/manim-scene-specs.md

@@ -7,6 +7,7 @@ Use these constraints when generating **`animations/specs/*.scene.yaml`** (via `
 - **Frame budget:** dogfood scenes use a **14.22×8** Manim frame (`scenes.py` header). Content sits under the title — tall stacks (**many rows × box `height` + `row_gap`**) scroll past the bottom. Prefer **extra pages** or **shorter boxes** (`height` ~0.72–0.9, tighter `row_gap`) over piling 5+ full-height rows on one page.
 - **~3 rows per page** is a safe default (~6 when rows use compact height); match beats in **`narration/<segment>.md`** and optional **`wait_segment`** / **`wait_at`** when `timing.json` has Whisper data.
 - **Subject-beat coverage (library contract):** hold the board while consecutive sentences elaborate the **same** topic; when the topic shifts, add a spoken-phrase label for that beat. `scene-spec-generate` and `validate` run `layout_density_violations` — **coverage of subject beats + no invented unspoken labels**, not a blind label count. Disable with `validation.subject_beat_coverage.enabled: false` if needed.
+- **Hyphenated / script labels:** Whisper often emits `version-controlled` or `setup-agent-prompts.sh` as **one** word. Prefer those exact compounds in box labels (or set `pace: none`). Retiming glues split label tokens to match those spoken compounds.
 - **Palette tokens only:** `C_BG`, `C_ACCENT`, `C_GREEN`, `C_ORANGE`, `C_BLUE`, `C_RED`, `C_TEAL`, `C_PURPLE`, `C_WHITE`.
 - **Readable sizes:** `font_size` ≥ 14; widths ~3–6, heights ~0.7–1.3; tune `row_gap` / `column_gap` in `layout` if labels feel cramped.
 - **Layout gate:** `docgen scene-spec-generate` rejects specs that exceed a computed vertical stack budget or safe row width (`layout_budget_violations` in `scene_spec.py`). `docgen scene-compile` does **not** enforce that (hand fixes allowed).

src/docgen/cli.py

@@ -1103,11 +1103,14 @@ def generate_all(
     regen_scene_specs: bool,
     skip_scene_retime: bool,
 ) -> None:
-    """Run full pipeline: TTS → timestamps → scene retime → Manim → compose → validate.
+    """Run full pipeline: TTS → timestamps → scene specs → Manim → compose → validate.
 
-    Order matters for beat sync: timestamps must land before scene compile so
-    ``wait_word`` indices match the current mp3. Existing declarative specs are
-    retime-compiled offline; pass ``--regen-scene-specs`` to call OpenAI first.
+    Default Manim path is declarative ``animations/specs/*.scene.yaml``:
+
+    - Specs exist → offline retime-compile against fresh ``timing.json``.
+    - No specs yet → auto ``scene-spec-generate`` (OpenAI) then compile.
+    - ``--regen-scene-specs`` forces LLM regenerate even when specs exist.
+    - ``--skip-scene-retime`` skips the whole scene-spec stage (legacy hand scenes).
     """
     from docgen.pipeline import Pipeline
 

src/docgen/init.py

@@ -367,7 +367,8 @@ def _write_config(plan: InitPlan) -> str:
         "# See: https://github.com/jmjava/documentation-generator\n"
         "#\n"
         "# Edit this file to match your project structure, then run:\n"
-        "#   docgen generate-all       # full pipeline\n"
+        "#   docgen generate-all       # TTS→timestamps→scene-spec→Manim→compose\n"
+        "#                             # (creates animations/specs/*.scene.yaml if missing)\n"
         "#   docgen wizard             # interactive GUI\n"
         "#   docgen validate           # check recordings\n\n"
     )
@@ -392,7 +393,9 @@ def _write_wrapper_scripts(plan: InitPlan) -> list[str]:
     scripts = {
         "generate-all.sh": "\n".join([
             "#!/usr/bin/env bash",
-            "# Full pipeline (TTS, segment visuals, compose, validate, concat). Wraps: docgen generate-all",
+            "# Full pipeline. Default Manim path: animations/specs/*.scene.yaml",
+            "# (auto scene-spec-generate on first run; retime thereafter).",
+            "# Wraps: docgen generate-all",
             "set -euo pipefail",
             _bash_dir,
             _venv_activate,

src/docgen/manim_scene_support.py

@@ -162,7 +162,7 @@ def _load_timing_words(segment_key: str) -> list[dict]:
 
 
 def _box(label, color, w=2.2, h=0.75, fs=18, subtitle=""):
-    """Labeled rounded box — slightly stronger fill/stroke for readable diagram boards.
+    """Labeled rounded box - slightly stronger fill/stroke for readable diagram boards.
 
     Optional ``subtitle`` is a second, smaller line under the primary label
     (decorative; not used for wait_word beat matching).

src/docgen/pipeline.py

@@ -4,9 +4,16 @@
 entries (see :meth:`docgen.config.Config.pipeline_manim_scene_names`). Segments whose visuals
 are pre-recorded (``recordings/*.mp4``) do not run through Manim capture here.
 
-After timestamps, existing ``animations/specs/*.scene.yaml`` files are **retime-compiled**
-against fresh ``timing.json`` (no OpenAI) so ``wait_word`` indices stay aligned. Optional
-``regen_scene_specs`` runs LLM ``scene-spec-generate`` for manim segments before that compile.
+After timestamps, declarative ``animations/specs/*.scene.yaml`` files are the **default**
+Manim authoring path:
+
+- **Specs present** → offline **retime-compile** against fresh ``timing.json`` (no OpenAI)
+  so ``wait_word`` indices stay aligned.
+- **No specs yet** (and the bundle has manim ``visual_map`` rows) → automatically run LLM
+  ``scene-spec-generate`` + compile (same as ``--regen-scene-specs`` for a first run).
+- **``regen_scene_specs=True``** → force LLM regenerate even when specs already exist.
+- Hand-authored classes outside ``BEGIN/END GENERATED SCENE`` markers remain supported,
+  but new consumer bundles should prefer specs.
 """
 
 from __future__ import annotations
@@ -40,7 +47,19 @@ def run(
         TimestampExtractor(self.config).extract_all()
 
         if not skip_manim and not skip_scene_retime:
-            self._run_scene_stages(regen_scene_specs=regen_scene_specs)
+            from docgen.scene_retime import list_scene_spec_paths
+
+            auto_regen = regen_scene_specs
+            if not auto_regen and self._manim_segment_ids():
+                if not list_scene_spec_paths(self.config):
+                    print(
+                        "\n[pipeline] No animations/specs/*.scene.yaml yet — "
+                        "auto-running scene-spec-generate (declarative Manim default). "
+                        "Pass --skip-scene-retime to keep hand-authored scenes.py only, "
+                        "or --regen-scene-specs to force regenerate later."
+                    )
+                    auto_regen = True
+            self._run_scene_stages(regen_scene_specs=auto_regen)
 
         if not skip_manim:
             from docgen.image_generate import generate_missing_images_for_bundle

src/docgen/scene_spec.py

@@ -378,6 +378,56 @@ def _tokens_match(label_token: str, word_token: str) -> bool:
     return _stem(label_token) == _stem(word_token)
 
 
+def _consume_label_tokens_at(
+    tokens: list[str],
+    word_norms: list[str],
+    start: int,
+) -> int | None:
+    """Advance through ``word_norms`` from ``start`` until all ``tokens`` are consumed.
+
+    Whisper often emits hyphenated compounds as **one** token (``version-controlled`` →
+    ``versioncontrolled``, ``setup-agent-prompts.sh`` → ``setupagentpromptssh``) while
+    labels split on hyphens into multiple tokens. Accept either:
+
+    - one label token ↔ one spoken word (exact / stem), or
+    - two or more consecutive label tokens glued ↔ one spoken word.
+
+    Returns the index of the last consumed spoken word, or ``None`` if the label
+    cannot be matched starting at ``start``.
+    """
+    if not tokens or start < 0 or start >= len(word_norms):
+        return None
+    ti = 0
+    wi = start
+    m = len(tokens)
+    n = len(word_norms)
+    while ti < m:
+        if wi >= n:
+            return None
+        spoken = word_norms[wi]
+        if _tokens_match(tokens[ti], spoken):
+            ti += 1
+            wi += 1
+            continue
+        # Glue 2+ label tokens into one Whisper word (hyphenated / dotted compounds).
+        glued = tokens[ti]
+        matched_k: int | None = None
+        for k in range(2, m - ti + 1):
+            glued += tokens[ti + k - 1]
+            if glued == spoken:
+                matched_k = k
+                break
+            # Spoken word is longer only when TTS/Whisper glued extra chars we already
+            # normalized away (rare); require exact equality for compounds.
+            if len(glued) > len(spoken):
+                break
+        if matched_k is None:
+            return None
+        ti += matched_k
+        wi += 1
+    return wi - 1
+
+
 def segment_index_for_whisper_time(
     segments: list[dict[str, Any]], wall_time: float
 ) -> int:
@@ -442,10 +492,12 @@ def sync_row_labels_to_whisper_words(
 ) -> dict[str, Any]:
     """Set ``wait_word`` on each **box** from its ``label`` → first spoken match (in order).
 
-    Matching is **fail-closed**: exact/stem token equality only (plus hyphen splits).
-    No fuzzy containment and no leftover LLM ``wait_word`` when the label is absent
-    from the transcript. Each matched box waits at word ``start``. Row-level
-    ``wait_word`` / ``wait_segment`` are cleared when ``overwrite=True``.
+    Matching is **fail-closed**: exact/stem token equality, hyphen/underscore splits,
+    and glued Whisper compounds (label ``version-controlled`` ↔ spoken
+    ``versioncontrolled``). No fuzzy containment and no leftover LLM ``wait_word``
+    when the label is absent from the transcript. Each matched box waits at word
+    ``start``. Row-level ``wait_word`` / ``wait_segment`` are cleared when
+    ``overwrite=True``.
     """
     if not isinstance(words, list) or not words:
         return spec
@@ -473,17 +525,13 @@ def _find_label(label: str, from_idx: int) -> tuple[int, int] | None:
         tokens = _label_tokens(label)
         if not tokens:
             return None
+        norms = [w[0] for w in word_stream]
         n = len(word_stream)
-        m = len(tokens)
         i = from_idx
-        while i <= n - m:
-            ok = True
-            for k in range(m):
-                if not _tokens_match(tokens[k], word_stream[i + k][0]):
-                    ok = False
-                    break
-            if ok:
-                return (i + m - 1, word_stream[i][2])
+        while i < n:
+            last = _consume_label_tokens_at(tokens, norms, i)
+            if last is not None:
+                return (last, word_stream[i][2])
             i += 1
         return None
 

tests/test_pipeline.py

@@ -101,7 +101,9 @@ def compose_segments(self, _segments) -> int:
         pipeline_manim_scene_names=lambda: ["Scene01"],
     )
 
-    Pipeline(cfg).run(skip_tts=True, retry_manim_on_freeze=True)
+    Pipeline(cfg).run(
+        skip_tts=True, retry_manim_on_freeze=True, skip_scene_retime=True
+    )
 
     assert FlakyComposer.attempts == 2
     assert calls.count("manim") == 2, "Manim should run once initially and once on retry"
@@ -186,7 +188,70 @@ def compose_segments(self, _segments) -> int:
     )
 
     with pytest.raises(ComposeError, match="FREEZE GUARD"):
-        Pipeline(cfg).run(skip_tts=True, retry_manim_on_freeze=False)
+        Pipeline(cfg).run(
+            skip_tts=True, retry_manim_on_freeze=False, skip_scene_retime=True
+        )
 
     assert calls.count("manim") == 1
     assert media_dir.exists(), "Without retry flag, Manim cache should be untouched"
+
+
+def test_pipeline_auto_regens_scene_specs_when_missing(tmp_path, monkeypatch) -> None:
+    """First run with manim segments and no specs → scene-spec-generate."""
+    calls: list[str] = []
+
+    class OkComposer:
+        def __init__(self, _config) -> None:
+            pass
+
+        def compose_segments(self, _segments) -> int:
+            calls.append("compose")
+            return 1
+
+    _patch_pipeline_stages(monkeypatch, OkComposer, calls)
+
+    import docgen.scene_retime as retime_module
+    import docgen.scene_spec_generate as ssg
+
+    monkeypatch.setattr(
+        retime_module, "list_scene_spec_paths", lambda cfg, segment_id=None: []
+    )
+
+    def fake_generate(cfg, sid, **_kwargs):
+        calls.append(f"scene-spec:{sid}")
+        specs = cfg.animations_dir / "specs"
+        specs.mkdir(parents=True, exist_ok=True)
+        path = specs / f"{sid}.scene.yaml"
+        path.write_text("segment_id: '01'\nclass_name: Scene01\nrows: []\n", encoding="utf-8")
+        return SimpleNamespace(
+            seg_name=sid,
+            yaml_text=path.read_text(encoding="utf-8"),
+            spec={"segment_id": sid, "class_name": "Scene01", "rows": []},
+        )
+
+    monkeypatch.setattr(ssg, "generate_scene_spec", fake_generate)
+    monkeypatch.setattr(
+        ssg,
+        "linted_class_block_from_spec",
+        lambda cfg, spec, timing_key=None: ("class Scene01: pass", spec),
+    )
+    monkeypatch.setattr(
+        ssg, "inject_class_block_into_scenes_py", lambda *a, **k: calls.append("inject")
+    )
+
+    animations_dir = tmp_path / "animations"
+    animations_dir.mkdir(parents=True)
+
+    cfg = SimpleNamespace(
+        animations_dir=animations_dir,
+        segments_all=["01"],
+        visual_map={"01": {"type": "manim", "class": "Scene01"}},
+        pipeline_manim_scene_names=lambda: ["Scene01"],
+    )
+
+    Pipeline(cfg).run(skip_tts=True)
+
+    assert "scene-spec:01" in calls
+    assert "inject" in calls
+    assert calls.index("timestamps") < calls.index("scene-spec:01")
+    assert calls.index("scene-spec:01") < calls.index("manim")