fix: Keep whitespace at inline element boundaries in Markdown output - #78
Closed
shuvamk wants to merge 1 commit into
Closed
fix: Keep whitespace at inline element boundaries in Markdown output#78shuvamk wants to merge 1 commit into
shuvamk wants to merge 1 commit into
Conversation
`to_markdown()` dropped the space that sits just inside an inline
element, welding the words on either side together:
<p>This is <b>very </b>important.</p>
-> This is **very**important.
<p>Visit<a href="https://e.com"> our site</a> today.</p>
-> Visit[our site](https://e.com) today.
Both render as one word, while `to_text()` on the same input returns
"This is very important." and "Visit our site today.".
`a`, `em`, `i`, `strong`, and `b` render their children into a separate
`_MarkdownBuilder`, so the boundary whitespace never reaches the parent:
a trailing space is still held in the child's `_pending_space` when
`finish()` runs, and a leading space is discarded because the child's
buffer is empty at that point. `span` and the other inline containers
share the parent builder and already came out correctly, so the two
spellings of the same document disagreed.
Record on the builder whether whitespace was collapsed before any output
and re-emit both boundaries into the parent when the marker or link is
closed.
Re-emitting alone injects a second space after a structural marker,
because both places the builder can flush a pending space, `raw()` and
`text()`, appended one whenever the buffer was non-empty without
checking what it ended with, and headings and list items write `"## "`,
`"- "`, and `"1. "` through `raw()`. In a list that destroys structure
rather than spacing: the extra space moves the item content to column 3
while nested lists indent by 2, so the inner list in
`<ul><li><b> outer</b><ul><li>inner</li></ul></li><li>second</li></ul>`
no longer clears its parent item and renders as a flat sibling. Both
flushes now skip a buffer that already ends in a space or tab. Guarding
them, rather than the two re-emission sites, keeps the rule in the
builder and lets the boundary space propagate out of nested inline
elements as ordinary collapsed whitespace.
The predicate also covers whitespace that predates this change, so
`<ul><li> Item</li></ul>` renders as `- Item` rather than `- Item`.
Across a 37,632-case matrix of inline elements, whitespace shapes and
block contexts, every difference from the previous output is either a
separating space restored at an element boundary or a duplicate space
collapsed; nothing differs in any other character.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Owner
|
Thank you for the report. Will re-implement in a new PR, but really appreciate the report. Good find! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
to_markdown()drops the space that sits just inside an inline element, welding the words on either side together.to_text()on the same input is correct, and so is<span>, which makes the two spellings of the same document disagree.to_text()to_markdown()<p>This is <b>very </b>important.</p>This is very important.This is **very**important.This is <strong>very</strong>important.<p>Visit <a href="https://e.com">our site </a>today.</p>Visit our site today.Visit [our site](https://e.com)today.…</a>today.<p>See <em>the docs </em>for details.</p>See the docs for details.See *the docs*for details.…</em>for details.<p>Visit<a href="https://e.com"> our site</a> today.</p>Visit our site today.Visit[our site](https://e.com) today.Visit<a…>This is the shape a rich-text editor produces when the user's selection includes the trailing space, so it shows up in pasted content rather than in hand-written HTML.
Cause
a,em,i,strongandbrender their children into a separate_MarkdownBuilder, so the boundary whitespace never reaches the parent. A trailing space is still sitting in the child's_pending_spacewhenfinish()runs, and a leading space is discarded because the child's buffer is empty at that point.<span>and the other inline containers share the parent builder and go through_pending_spacenormally, which is why they already came out right.Fix
Record on the builder whether whitespace was collapsed before any output, and re-emit both boundaries into the parent when the marker or link is closed.
Re-emitting alone injects a second space after a structural marker: both places the builder can flush a pending space,
raw()andtext(), appended one whenever the buffer was non-empty without checking what it ended with — and headings and list items write"## ","- "and"1. "throughraw(). In a list that destroys structure rather than spacing. The extra space moves the item content to column 3 while nested lists indent by 2, so the inner list inno longer clears its parent item and renders as a flat sibling. Both flushes now skip a buffer that already ends in a space or tab.
I put the guard in the builder rather than at the two re-emission sites deliberately: the re-emissions work because they are ordinary
text(" ")calls, which is what lets a boundary space collapse against adjacent text, disappear at block and document ends, and propagate outward through nested inline elements such as<a>x <b>y </b></a>z. Guarding at the call sites would have meant four copies of the check reaching intoparent_builder._buf, and would have left the rule false for the othertext(" ")callers. Happy to switch if you'd rather have it the other way.The predicate also covers whitespace that predates this change, so
<ul><li> Item</li></ul>now renders- Iteminstead of- Item, and<h2> Title</h2>renders## Title. Both render identically in CommonMark; where the column does matter, for a nested list, the new output is the correct one. That is pinned by a test so the change is deliberate rather than incidental.Tests
Eight tests in
tests/test_node.pyalongside the otherto_markdowncases, covering both directions: the space that must appear at an inline boundary, and the space that must not appear after a##/-/1.marker or insideflatten_list_item. The nested-list test asserts the exact string, including the existing blank-line shape, so a future regression in either direction fails.I checked each test fails without the corresponding part of the change — the boundary tests fail on
main, and the marker tests fail against an earlier revision of this branch that had the re-emission without the guard.Verified on
mainatfdab7f5:python run_tests.py— 3965/3965 passed (100.0%), 6 skipped; baseline onmainis 3957/3957, so +8 is exactly the new testscoverage run run_tests.py && coverage report— 100%, statements and branches, with both new conditions exercised in both directionsruff check,ruff format --check,mypy src— cleanbenchmarks/html5lib_engine_diff.py --fail-under-rate 1.0 --fail-on-current-exceptions— exit 0I also ran a differential over ~68,000 generated cases (inline elements × whitespace shapes including empty and whitespace-only content × block contexts, in both compact and
html_passthroughmode), comparing output againstmain. Every difference is either a separating space restored at an element boundary or a duplicate space collapsed — no case differs in a non-whitespace character, none loses a separatormainhad, and none gains a double space.One thing I noticed while testing and deliberately left alone:
<ol>nested-list indentation is already off onmain(1. a\n\n 1. nflattens under CommonMark, because a 2-space indent doesn't clear an ordered marker's content column). It's unrelated to this change and unaffected by it — happy to file it separately if useful.Disclosure: written with AI assistance (Claude Code). Every repro and every number above was executed locally against this branch; I've reviewed the change and can speak to it.