G2: chunking contract test + the four silent defects it found

[brentrager]

Problem

Gap G2 (document processing / chunking pipeline) was recorded open — "our knowledge store assumes pre-chunked text" — but the Chunker the connectors feed shipped with G1. What was actually missing was the contract suite the gap doc asked for.

Writing it found the chunker wrong on four counts. Every one fails silently: no error, no panic, just a chunk the embedding API quietly truncates or a document that never really got chunked.

What shipped

rust/ingestion/tests/chunking.rs — 16 tests pinning chunk count / dense indices / stable ids, overlap as a shared run of words, metadata + title + source + acl propagation onto every chunk, oversized spill with no word lost, UTF-8 integrity at the boundary, and degenerate configs.

Fixed against it, in rust/ingestion/src/chunker.rs:

Defect	Why it was silent
max_chars was not a cap on the emitted chunk	Overlap was prepended on top of a chunk that already filled the cap, so a default 500/64 chunker emitted up to 565 chars. Over the embedding model's limit the API truncates rather than rejects — the tail is dropped from the index with nothing logged. Overlap now comes out of the packing budget.
Text without spaces never spilled	Splitting on word boundaries only meant a Chinese/Japanese/Thai document — or a long URL, or a minified blob — has exactly one "word", so a 50k-char document came back as one unbounded chunk. Fallback now cuts on character boundaries.
CRLF documents lost all paragraph structure	"\r\n\r\n" contains no "\n\n", so every Windows-authored file and many HTTP-fetched pages arrived as a single giant paragraph. Content is CRLF-normalized first.
A chunk could straddle two markdown sections	Which attributes section A's text to section B's heading at retrieval time. A heading is now a hard boundary.

Characters, never bytes, throughout: slicing to a byte offset to hit a character cap cuts an em-dash or an emoji in half — a panic in Rust, silent mojibake in a port.

Proof of red

Reverting chunker.rs in place and leaving the tests fails 6 of 16:

a_markdown_heading_starts_a_new_chunk          the two sections must not share a chunk
no_chunk_exceeds_the_cap_even_with_overlap_on  cap 80/overlap 20: chunk is 83 chars
a_single_word_longer_than_the_cap_is_split     chunk is 250 chars
crlf_documents_split_on_paragraphs_like_lf     left: ["alpha paragraph beta", …]
a_degenerate_config_still_terminates           cap 1/overlap 999: chunk is 5 chars
text_with_no_whitespace_still_spills           unspaced text must still spill, got 1 chunk(s)
test result: FAILED. 9 passed; 6 failed

The UTF-8 guard needed a second pass, and that is the part worth reviewing. Built from space-separated words it passed against a deliberately byte-slicing implementation — because spaced text never reaches the character-split path at all. It guarded nothing. Rebuilt on unspaced mixed-width text ("aé数🚀—"), the same mutant panics:

panicked at src/chunker.rs:261: end byte index 7 is not a char boundary;
it is inside '🚀' (bytes 6..10 of string)

Licensing

The Smoo AI monorepo's rust/knowledge-ingest/src/chunker.rs was read as prior art and nothing was copied. That chunker's entire purpose is byte-parity with a proprietary TypeScript ingest path so its content-addressing lines up across two pipelines — Smoo-specific business logic with no meaning here. Deliberately left behind: the content-addressing scheme (sha256(documentId:chunkIndex) ids and content hashes for an upsert diff), the LangChain-parity separator ladder and its JS-semantics quirks, and every reference to the private ingestion schema. What crossed the boundary is one idea, reimplemented in six lines: a markdown heading is a chunk boundary.

Scope

• Rust only. Chunking runs server-side in the ingestion crate; the other four languages have no ingestion pipeline to port this into.
• No public API change — Chunker::new / chunk keep their signatures. All 41 pre-existing ingestion tests and the 129 server tests still pass.
• docs/Planning/Feature Gaps.md §G2 marked done + docs/Architecture/Ingestion Pipeline.md updated in this PR.

Note for the reviewer

I did not make Chunker a pluggable trait alongside Embedder/Reranker. Those seams exist because they have real alternative implementations (gateway vs deterministic vs noop); a Chunker trait here would have exactly one impl and would push dyn/generics through ingest(), indexing, and eight call sites for no second implementation. The concrete Chunker is already the pipeline's injection point and already defaults to something deterministic and network-free. Say the word if you want the trait anyway and it's a small follow-up.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: 12f6271

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@smooai/smooth-operator	Patch
@smooai/smooth-operator-web-chat-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR