Add delimiter-wrapped frontmatter support for non-Markdown files

[DandyLyons]

Summary

Add syntax-mapped support for YAML frontmatter in non-Markdown text files. A complete YAML metadata block is wrapped in delimiters that keep the host file valid.

Existing Markdown behavior remains the default for multi-file and directory operations. Shipped extension mappings are the only source of non-Markdown syntax support in this issue. Custom delimiters, named custom syntaxes, project-defined syntax mappings, and the previously proposed configuration-schema change are explicitly out of scope.

Representative forms:

/*
---
title: Example
tags:
  - swift
---
*/

struct Example {}

<!--
---
title: Example
---
-->

<main>Example</main>

"""
---
title: Example
---
"""

def example():
    pass

Scope decision: syntax mappings only

Every supported non-Markdown extension must resolve through the shipped syntax mapping engine.

• If an extension has a shipped mapping, the mapped syntax is used.
• If an explicitly requested file has no mapping, the command reports that the extension is unsupported and leaves the file unchanged.
• Files discovered while expanding a directory are considered only when their extensions have mappings.
• There are no custom delimiter CLI flags in this scope.
• There are no project-defined syntaxes or extension mappings in .md-utils/md-utils.json in this scope.
• There is no config-schema version change for this feature.
• Custom syntax configuration remains a potential future enhancement.

The following previously proposed options are removed from this issue:

--frontmatter-syntax
--frontmatter-comment-open
--frontmatter-comment-close
--no-frontmatter-presets

The relevant CLI options are:

--include-non-md
--create-frontmatter

The existing --extensions option continues to narrow explicitly requested extensions where the command supports it. It does not create a syntax mapping for an otherwise unsupported extension.

Concrete CLI selection semantics

One explicit file

An explicitly supplied supported non-Markdown file communicates sufficient intent. Except for plain .txt files, --include-non-md is not required when exactly one regular file path is supplied.

md-utils fm get Example.swift --key title
md-utils fm set Example.swift --key status --value approved

Both commands infer c-block from .swift.

Markdown extensions (md, markdown) retain existing behavior. Plain .txt is not selected by default; it is processed as Markdown-style text only with explicit non-Markdown opt-in:

md-utils fm get notes.txt --key title --include-non-md
md-utils fm set notes.txt --key status --value approved --include-non-md

An explicitly supplied unmapped file fails unchanged:

md-utils fm set settings.toml --key status --value approved
# error: no frontmatter syntax mapping for extension "toml"

Multiple explicit files

Without --include-non-md, preserve Markdown-only behavior. If the user explicitly lists a non-Markdown file, report that it was ignored and show the opt-in:

md-utils fm set note.md Example.swift --key reviewed --value approved
# note.md is edited
# Example.swift is unchanged
# stderr: ignored non-Markdown file Example.swift; use --include-non-md to process mapped non-Markdown files

With opt-in, mapped files are processed:

md-utils fm set note.md Example.swift --key reviewed --value approved --include-non-md

Directories

Directory traversal remains Markdown-only by default and silently ignores non-Markdown files. Because the user named a directory rather than a particular non-Markdown file, no opt-in hint is required.

md-utils fm set Sources/ --key reviewed --value approved

Mapped non-Markdown files participate only after opt-in:

md-utils fm set Sources/ --key reviewed --value approved --include-non-md

Placement and multiplicity

• Wrapped frontmatter may occur anywhere in the text file.
• Documentation should recommend placing it at or near the beginning for discoverability and scan efficiency.
• The scanner examines the whole file for complete matching blocks.
• Ideally a file contains exactly one frontmatter block.
• The first complete block is parsed into the existing Yams/frontmatter representation.
• Later complete blocks are discovered only far enough to report diagnostics containing the file and 1-based opening line. Their YAML payloads are not parsed, merged, validated, or interpreted.
• Multiple complete blocks are invalid and must never be ignored silently.
• Read-only operations may expose the first block while reporting later-block diagnostics according to their command contract.
• Mutating operations refuse to change a file containing multiple complete blocks. The file remains byte-for-byte unchanged.
• If no complete combination of opening wrapper, opening YAML ---, closing YAML ---, and closing wrapper is found, the file has no wrapped frontmatter. An incomplete candidate is not, by itself, a parser error.

Mutation and creation behavior

Mutation uses ranges derived from the same source snapshot that was parsed. Persisted or cached ranges are never valid edit coordinates.

Existing single block

Edit the existing mapped block and preserve the wrapper syntax, surrounding host content, and line endings.

md-utils fm set Example.swift --key status --value approved

No existing block in one explicit non-Markdown file

Do not silently introduce a new host-language comment or multiline string. Prompt on an interactive terminal:

md-utils fm set Example.swift --key status --value approved
# Create wrapped frontmatter using c-block (/* … */)? [y/N]

• y creates the mapped wrapper at line 1, adds one blank line before the original content, and performs the edit.
• n, an empty response, or EOF leaves the file unchanged and exits nonzero because the requested edit was not applied.
• --create-frontmatter authorizes creation noninteractively.

md-utils fm set Example.swift --key status --value approved --create-frontmatter

For ordinary Markdown files, current behavior remains unchanged: mutating commands silently create ordinary Markdown frontmatter when absent. --create-frontmatter is unnecessary and, if supplied for a Markdown file, is accepted and silently ignored. Documentation must explain that the flag is needed only to authorize creation in non-Markdown files.

Batch mutation never prompts. When an opted-in mapped non-Markdown file lacks frontmatter:

• without --create-frontmatter, report that the file requires the flag, leave it unchanged, continue other files, and exit nonzero;
• with --create-frontmatter, create the mapped wrapper noninteractively.

Filesystem safety

• Reread the latest raw text and parse it immediately before applying a change.
• Revision comparison and filesystem writing belong in the native filesystem layer, not portable Core.
• A detected revision mismatch fails the mutation and is never silently retried.
• An explicit retry rereads the latest bytes and reconstructs the edit from the new snapshot.
• Writes use atomic replacement.
• Documentation must disclose the residual race with uncoordinated external writers between the final revision check and atomic replacement.

Core parsing model

• Implement extraction in MarkdownUtilitiesCore without modeling the host file as a MarkdownDocument.
• Use the package's pinned pointfreeco/swift-parsing dependency and its Parse { ... } result-builder DSL.
• A block requires a complete host envelope and complete inner YAML block.
• Only --- closes YAML in this scope; Pandoc-style ... is out of scope.
• Wrapper delimiters and YAML markers match complete physical lines, not substrings inside scalar content.
• Preserve YAML payload indentation. Do not dedent or otherwise normalize indentation because indentation changes YAML meaning.
• Return the first block's raw YAML through the existing YAMLConversion/frontmatter representation.
• Report snapshot-relative source locations for later complete blocks.
• Locations belong only to the exact parsed source snapshot.
• Invalid YAML in the first complete envelope remains a YAML-conversion diagnostic.
• Support LF input. Pure CRLF input is out of scope.
• Per-line comment syntaxes such as // are unsupported.
• Binary-file detection and classification are out of scope.

Initial shipped syntax mappings

Extensions are compared case-insensitively without the leading dot.

Syntax	Opening / closing wrapper	Extensions
c-block	/* / */	c, h, cc, cpp, cxx, hpp, hxx, m, mm, swift, java, kt, kts, scala, js, mjs, cjs, jsx, ts, mts, cts, tsx, cs, go, rs, dart, php, css, scss, less, sql, jsonc
html-comment	<!-- / -->	html, htm, xhtml, xml, svg, vue, svelte
python-docstring	""" / """	py, pyi
powershell-block	<# / #>	ps1, psm1, psd1
lua-block	--[[ / ]]	lua
markdown-text	ordinary --- / --- frontmatter	txt (only after --include-non-md)

Formats requiring a prefix on every metadata line—shell scripts, Ruby, YAML, TOML, and R—do not receive a mapping. Ordinary JSON has no suitable comment syntax and remains unsupported; JSONC is mapped to c-block.

Initial command scope

Wrapped frontmatter reading integrates with:

• fm dump
• fm get
• fm has
• fm list
• fm search
• fm unique

The approved red CLI contract also covers fm set so selection, refusal, confirmation, and creation behavior are fixed before implementation. Broader mutation and array-command parity remains follow-up work unless separately approved.

Parser proof of concept and review

Before finalizing a public API or wiring the extractor broadly into CLI commands, build a focused result-builder prototype proving:

• complete wrapped block discovery anywhere in a file;
• first-block extraction and lightweight later-block discovery;
• no Yams parsing of later blocks;
• syntax selection separated from the configurable-by-call parser implementation;
• C-style, HTML, Python, JSONC, PowerShell, and Lua envelopes;
• LF handling;
• incomplete candidates treated as no frontmatter;
• delimiter-like scalar content ignored unless it forms complete delimiter lines;
• representative single-file and directory-scale performance.

Review the parser design and implementation with Daniel before public type names are finalized.

Implementation sequence

Implementation must begin on a new feature branch created from an up-to-date main branch.

1. Create the feature branch from up-to-date main.
2. Prototype complete-envelope extraction and later-block discovery in MarkdownUtilitiesCore.
3. Add Core parser tests across mapped syntaxes and source positions.
4. Review the result-builder implementation and diagnostics with Daniel.
5. Finalize public Core API and shipped syntax-mapping names.
6. Integrate the approved read-only commands and fm set CLI contract.
7. Document selection, creation safety, near-start placement, mappings, and unsupported syntax customization.

Acceptance criteria

• Existing Markdown parsing and default multi-file/directory selection remain backward compatible.
• One explicitly supplied mapped non-Markdown file is processed without --include-non-md, except .txt.
• Multiple explicit files and directories remain Markdown-only unless --include-non-md is supplied.
• Explicitly listed ignored non-Markdown files produce a hint mentioning --include-non-md; directory-discovered ignored files do not.
• .txt is ignored unless --include-non-md is supplied, then uses ordinary Markdown-style frontmatter.
• JSONC maps to c-block; ordinary JSON remains unsupported.
• Unmapped extensions cannot be enabled by --extensions alone.
• No custom delimiter CLI or project configuration is implemented.
• Complete blocks can be discovered anywhere in LF text.
• YAML indentation is preserved without dedenting.
• The first block uses the existing YAML/frontmatter representation.
• Later blocks are not parsed as YAML and produce path-and-line diagnostics.
• Mutation refuses files with multiple blocks and leaves them unchanged.
• Single explicit non-Markdown creation prompts by default and supports --create-frontmatter.
• Batch creation never prompts and requires --create-frontmatter.
• Markdown creation remains silent; --create-frontmatter is accepted and ignored for Markdown.
• New mapped wrappers are inserted at line 1 followed by one blank line.
• The parser uses the Parse { ... } result-builder DSL.
• Tests cover mapped syntaxes, absent and empty frontmatter, malformed first YAML, incomplete candidates, multiple blocks, delimiter-like scalar content, selection hints, confirmation, and noninteractive creation.
• Per-line syntax and custom delimiter configuration are documented as unsupported.
• The parser implementation is reviewed with Daniel before public API names are finalized.
• Relevant --help pages include a FRONTMATTER ON NON-MD FILES section which tells the user the precise requirements of the text in order to be considered a valid "frontmatter" according to md-utils (e.g. must be inside a valid comment block, delmited by --- etc.).

Out of scope

• Custom comment delimiters
• Named custom syntaxes
• Project-defined syntax mappings
• Frontmatter configuration in .md-utils/md-utils.json
• A config-schema version change for this feature
• Per-line comment envelopes
• Pandoc-style ... YAML closers
• Parsing or merging later blocks
• Binary-file detection or classification
• Pure CRLF input
• Broad mutation and array-command parity beyond the approved fm set contract

[DandyLyons]

Implemented in a9cebb3.

Highlights:

• Added shipped wrapped-frontmatter syntax mappings and the Parse { ... } Core scanner.
• Integrated fm dump, get, has, list, search, unique, and set with the approved selection and creation semantics.
• Added revision-checked atomic writes for mutation.
• Added the shared FRONTMATTER ON NON-MD FILES section to the fm group and every relevant command help page, with generated-help coverage for all eight pages.
• Added parser, filesystem-writer, fixture-based CLI, malformed-YAML, multiplicity, confirmation, and batch-creation coverage.
• Updated user, DocC, and testing documentation.

Verification: swift test passes all 1,088 tests across 91 suites. All acceptance-criteria checkboxes, including the newly added help-page requirement, are complete.

The unrelated OKF test-bundle lookup failure was corrected separately in a79f242.

[DandyLyons]

Completed and merged in c5791fb via PR #100. Shipped syntax-mapped wrapped frontmatter support now covers every md-utils fm leaf command, with safe selection and creation semantics, multiple-block refusal, revision-checked atomic mutation, generated help documentation, and fixture-based parity coverage. Verification before merge: all 1,092 tests across 91 suites passed, DocC passed with warnings as errors, and the required wasm-core GitHub Actions check passed.