Redesign the LLM dialogue-cleanup workflow for token efficiency

[5o1]

Objective

Redesign cleanup and annotation so that it uses continuous narrative context effectively without spending tokens on repeated, low-value input or output.

Scope

• Measure where the current batching workflow repeats context or produces redundant fields.
• Separate deterministic normalization from semantic decisions that require an LLM.
• Preserve chronological context across batches while minimizing repeated transcript material.
• Integrate Kantoku context without duplicating unchanged direction.
• Keep Codex annotation as an independent, resumable thread workflow.
• Retain validation, retry, provenance, and cached intermediate artifacts.

Acceptance criteria

• A written design compares the current and proposed token flow.
• Token use can be measured per scene, batch, and accepted dialogue line.
• The new workflow resumes without repeating completed batches.
• Context continuity and stable dialogue IDs are preserved.
• Existing cleanup and validation behaviors have migration tests.
• A representative chapter sample uses fewer tokens without a demonstrated quality regression.