Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ md-utils is a Swift package for parsing and manipulating Markdown files. It cons
## Project Brief

- **Language**: Swift 6.2+
- **Frameworks/Libraries**: Foundation, MarkdownSyntax, swift-parsing, PathKit, Yams, JMESPath, JSONSchema.swift, swift-argument-parser, Rainbow
- **Frameworks/Libraries**: Foundation, MarkdownSyntax, swift-parsing, PathKit, Yams, swift-toml, JMESPath, JSONSchema.swift, swift-argument-parser, Rainbow
- **Package Manager / Build Tool**: Swift Package Manager
- **CLI Target**: `md-utils`
- **Library Targets**: `MarkdownUtilitiesCore`, `MarkdownUtilities`
Expand All @@ -21,7 +21,7 @@ md-utils is a Swift package for parsing and manipulating Markdown files. It cons
- **Test Command**: `swift test`
- **Formatter/Linter**: No dedicated formatter or linter is configured in-package
- **Documentation**: README.md, AGENTS.md, docs/*.md, generated CLI help, and bundled Agent Skill docs
- **Security**: Avoid unsafe optional force unwraps; treat filesystem and YAML/JSON parsing failures as user-visible errors
- **Security**: Avoid unsafe optional force unwraps; treat filesystem and YAML/TOML/JSON parsing failures as user-visible errors
- **CI/Coverage**: No project-specific CI or coverage command is documented in this repo

## Requirements
Expand Down
17 changes: 17 additions & 0 deletions IntegrationTests/WasmCoreSmoke/main.swift
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ enum WasmCoreSmokeError: Error {
case frontmatterNotParsed
case emptyAST
case renderMismatch
case tomlMismatch
case typeAssessmentFailed
}

Expand Down Expand Up @@ -39,6 +40,22 @@ struct WasmCoreSmoke {
throw WasmCoreSmokeError.renderMismatch
}

let tomlDocument = try MarkdownDocument(content: """
+++
title = "WebAssembly TOML"
tags = ["swift", "wasm"]
+++
# TOML
""")
let tomlRendered = try tomlDocument.render()
guard tomlDocument.frontMatterFormat == .toml,
tomlDocument.frontMatter["tags"]?.sequence?.count == 2,
tomlRendered.hasPrefix("+++\n"),
tomlRendered.contains("title = \"WebAssembly TOML\"")
else {
throw WasmCoreSmokeError.tomlMismatch
}

let definition = MarkdownTypeDefinition(
name: MarkdownTypeName(rawValue: "WasmDocument"),
version: "smoke",
Expand Down
11 changes: 10 additions & 1 deletion Package.resolved

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ let package = Package(
.package(url: "https://github.com/kylef/PathKit", from: "1.0.1"),
.package(url: "https://github.com/kylef/JSONSchema.swift", from: "0.6.0"),
.package(url: "https://github.com/jpsim/Yams.git", from: "6.1.0"),
.package(url: "https://github.com/mattt/swift-toml.git", from: "2.0.0"),
.package(url: "https://github.com/adam-fowler/jmespath.swift.git", from: "1.0.3"),
.package(url: "https://github.com/onevcat/Rainbow", from: "4.2.1"),
.package(url: "https://github.com/apple/swift-docc-plugin.git", from: "1.4.0"),
Expand All @@ -46,6 +47,7 @@ let package = Package(
.product(name: "Parsing", package: "swift-parsing"),
.product(name: "JSONSchema", package: "JSONSchema.swift"),
"Yams",
.product(name: "TOML", package: "swift-toml"),
]
),
.testTarget(
Expand Down
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ This project is on a `0.x.x` release and is **not yet API stable**. The API and
- **Heading Manipulation** — Promote/demote headings while maintaining nested structure
- **Section Operations** — Extract sections by name or index; reorder sections (move up/down/to position)
- **Content Selection** — Extract body without frontmatter, select by line range, extract by section
- **YAML Front Matter** — Full CRUD operations with 12+ subcommands including get, set, remove, rename, search (JMESPath), sort keys, array manipulation, and multi-format dump (JSON, YAML, raw, PropertyList)
- **YAML and TOML Front Matter** — Format-preserving CRUD, array manipulation, and multi-format output. JMESPath `fm search` remains YAML-only.
- **Format Conversion** — Convert Markdown to plain text or CSV
- **File Metadata** — Read file metadata including standard and extended attributes (xattr)
- **Wikilink Parsing & Resolution** — Parse Obsidian-flavored wikilinks, resolve against a vault directory, detect broken/ambiguous links, find backlinks
Expand Down Expand Up @@ -151,6 +151,9 @@ swift run md-utils fm get --key title posts/ | jq '.[] | select(has("value")) |
# Set a frontmatter value
swift run md-utils fm set --key tags --value "[swift, cli]" document.md

# Create TOML frontmatter in a document that has none
swift run md-utils fm set --key title --value "TOML Note" --frontmatter-format toml document.md

# Dump frontmatter as JSON
swift run md-utils fm dump document.md

Expand Down Expand Up @@ -383,11 +386,11 @@ selected automatically, except `.txt`, which requires `--include-non-md`.
Mapped extensions use the same `c-block`, `html-comment`, `python-docstring`,
`powershell-block`, and `lua-block` wrappers documented in
[Frontmatter in Non-Markdown Text Files](docs/common-use-cases.md#frontmatter-in-non-markdown-text-files).
Wrapped YAML supports frontmatter predicates, JMESPath, type hints, and JSON
Wrapped YAML or TOML supports frontmatter predicates, JMESPath, type hints, and JSON
Schema checks. Raw body predicates operate on wrapper-excluded host text, while
Markdown headings, sections, and wikilinks are explicitly unsupported.

If a file matches multiple rules, all matching checks apply. Files matching no rules are ignored. Invalid YAML frontmatter is reported as an error for matched rules because frontmatter predicates and schema checks cannot proceed.
If a file matches multiple rules, all matching checks apply. Files matching no rules are ignored. Invalid YAML or TOML frontmatter is reported as an error for matched rules because frontmatter predicates and schema checks cannot proceed.

## GitHub Pages

Expand Down Expand Up @@ -427,6 +430,7 @@ When the Pages workflow prepares its artifact, it copies `site/schemas/$CURRENT_
- [PathKit](https://github.com/kylef/PathKit) — File path handling
- [JSONSchema.swift](https://github.com/kylef/JSONSchema.swift) — JSON Schema validation
- [Yams](https://github.com/jpsim/Yams) — YAML parsing and serialization
- [swift-toml](https://github.com/mattt/swift-toml) — TOML parsing and serialization
- [jmespath.swift](https://github.com/nicktmro/jmespath.swift) — JMESPath query language for JSON

## Platform Compatibility
Expand Down
44 changes: 23 additions & 21 deletions Sources/MarkdownUtilities/FormatConversion/CSV/CSVConverter.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import MarkdownUtilitiesCore
import PathKit
import Yams

/// Converts Markdown documents with YAML frontmatter to CSV format.
/// Converts Markdown documents with YAML or TOML frontmatter to CSV format.
///
/// This converter takes a collection of Markdown documents and their file paths,
/// extracts their frontmatter keys, and generates a CSV with one row per document.
Expand Down Expand Up @@ -83,9 +83,7 @@ public struct CSVConverter {

for (_, document) in documents {
for (key, _) in document.frontMatter {
if case .scalar(let scalar) = key {
keySet.insert(scalar.string)
}
keySet.insert(key)
}
}

Expand Down Expand Up @@ -194,34 +192,38 @@ public struct CSVConverter {
/// Get a frontmatter value for a specific key
func getFrontmatterValue(document: MarkdownDocument, key: String) throws -> String {
// Look up the key in frontmatter
let keyNode = Yams.Node.scalar(.init(key))

guard let valueNode = document.frontMatter[keyNode] else {
guard let value = document.frontMatter[key] else {
// Key not present in this document
return ""
}

// Convert the value to a string
return try frontmatterValueToString(valueNode)
return try frontmatterValueToString(value)
}

/// Convert a Yams.Node to a string representation
///
/// - Scalars: Return as-is
/// - Complex types (arrays, objects): Serialize to JSON
func frontmatterValueToString(_ node: Yams.Node) throws -> String {
switch node {
case .scalar(let scalar):
return scalar.string

case .sequence, .mapping:
// Serialize complex types as JSON (compact, no pretty printing)
return try YAMLConversion.nodeToJSON(node, options: [])

case .alias:
// YAML aliases should be resolved by the parser, but if we encounter one,
// serialize it as JSON for safety
return try YAMLConversion.nodeToJSON(node, options: [])
func frontmatterValueToString(_ value: FrontMatterValue) throws -> String {
switch value {
case .null:
return ""
case .string(let value):
return value
case .boolean(let value):
return String(value)
case .integer(let value):
return String(value)
case .number(let value):
return String(value)
case .offsetDateTime, .localDateTime, .localDate, .localTime:
return String(describing: FrontMatterConversion.foundationValue(value))
case .array, .object:
return try YAMLConversion.anyToJSON(
FrontMatterConversion.foundationValue(value),
options: []
)
}
}
// MARK: - CSV Escaping
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import MarkdownUtilitiesCore
public struct CSVOptions: ConversionOptions, Sendable {
// MARK: - ConversionOptions Conformance

/// Whether to include YAML frontmatter in the CSV output
/// Whether to include YAML or TOML frontmatter in the CSV output
///
/// This is always `true` for CSV conversion since the entire purpose
/// is to export frontmatter as columns.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ public enum MarkdownTypeFileRegistryLoader {
".mdtype.yaml",
".mdtype.yml",
".mdtype.json",
".mdtype.toml",
]

public static func load(projectRoot: Path) throws -> MarkdownTypeRegistry {
Expand Down Expand Up @@ -61,6 +62,8 @@ public enum MarkdownTypeFileRegistryLoader {
return .yaml
case "json":
return .json
case "toml":
return .toml
default:
throw MarkdownTypeFileLoaderError.unsupportedDefinitionFormat(path.string)
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,24 +1,26 @@
# Reading and Mutating Frontmatter

Read, write, and convert YAML frontmatter while preserving the Markdown body.
Read, write, and convert YAML or TOML frontmatter while preserving the Markdown body.

## Overview

Frontmatter support starts with `FrontMatterParser`, which detects YAML delimited by `---` markers and separates it from the body text. `MarkdownDocument` then parses that YAML into a Yams mapping for structured access.
Frontmatter support starts with `FrontMatterParser`, which detects YAML delimited by `---` or TOML delimited by `+++` and separates it from the body text. `MarkdownDocument` parses either format into the ordered, format-neutral ``FrontMatter`` model and records the source ``FrontMatterFormat``.

Non-Markdown text uses ``WrappedFrontMatterParser`` with a ``FrontMatterSyntax``.
The parser scans LF text for complete host wrappers containing complete `---` YAML
blocks. It returns the first block's raw YAML and snapshot-relative source range,
The parser scans LF text for complete host wrappers containing complete YAML or TOML
blocks. It returns the first block's raw frontmatter, format, and snapshot-relative source range,
plus the 1-based opening lines of later complete blocks. Incomplete candidates are
treated as absent. The parser does not model the host content as Markdown and does
not normalize YAML indentation.
not normalize indentation.

Mutation helpers on `MarkdownDocument` update frontmatter values without changing the body. Conversion helpers in `YAMLConversion` translate YAML nodes and mappings to Swift values, JSON, Property List, and YAML output.
Mutation helpers on `MarkdownDocument` update frontmatter values without changing the body or delimiter format. ``FrontMatterConversion`` parses and serializes the neutral value model; `YAMLConversion` remains available for YAML-specific interoperability.

Comments are not part of the neutral value model. Parsing and rendering either YAML or TOML does not guarantee that frontmatter comments survive, so callers should avoid comments in frontmatter that will be mutated.

## Missing and Null Values

A missing key and a key with a YAML null value are distinct states. Callers that render frontmatter for user-facing output should preserve that distinction when it matters to downstream tools.
A missing key and a key with a YAML null value are distinct states. TOML has no null value and serialization reports the exact unsupported key path. Callers that render frontmatter for user-facing output should preserve the distinction when it matters to downstream tools.

## Errors

Invalid YAML and mappings with unsupported keys are reported as thrown errors rather than fatal failures.
Invalid YAML, invalid TOML, and values unsupported by the selected format are reported as thrown errors rather than fatal failures.
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,21 @@ Assess complete Markdown records against reusable structural contracts.

## Records and Conformance

A `MarkdownRecord` contains canonical Markdown text and optional identity, revision, and external context. A `MarkdownDocument` is the parsed content view produced from valid text. Type assessment accepts the record so invalid YAML can be returned as a structured diagnostic instead of preventing the resource from being represented.
A `MarkdownRecord` contains canonical Markdown text and optional identity, revision, and external context. A `MarkdownDocument` is the parsed content view produced from valid text. Type assessment accepts the record so invalid YAML or TOML can be returned as a structured diagnostic instead of preventing the resource from being represented.

Conformance is structural and non-exclusive. One record can conform to `Book`, `Document`, and `Publishable` at the same time. Requirements produce errors and affect conformance. Recommendations produce advisories without making the record fail.

Types have three domains:

- `frontmatter` validates schema-visible YAML values against every listed JSON Schema;
- `frontmatter` validates schema-visible YAML or TOML values against every listed JSON Schema;
- `body` evaluates Markdown AST predicates such as headings, hierarchy, and sections; and
- `context` evaluates external facts such as a normalized logical path.

`$md-utils` frontmatter is reserved for system metadata and is excluded from schema-visible user frontmatter.

## Define and Assess a Type

Type definitions use the same model whether decoded from YAML or JSON. Filesystem-backed definitions use the compound extensions `.mdtype.yaml`, `.mdtype.yml`, or `.mdtype.json`. A type contract version is an opaque nonempty string; Semantic Versioning is recommended but not enforced.
Type definitions use the same model whether decoded from YAML, JSON, or TOML. Filesystem-backed definitions use the compound extensions `.mdtype.yaml`, `.mdtype.yml`, `.mdtype.json`, or `.mdtype.toml`. A type contract version is an opaque nonempty string; Semantic Versioning is recommended but not enforced.

```swift
let definition = MarkdownTypeDefinition(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Slug validation is selected explicitly:
- `unicode` accepts lowercase Unicode letters and digits separated by non-adjacent hyphens or underscores; and
- `preserve` accepts ASCII letters and digits separated by single hyphens while preserving authored case.

Missing and null values are reported as missing identities. Malformed YAML, invalid formats, arrays, objects, booleans, non-integral numbers, and lossy conversions produce structured invalid-identity diagnostics. Identity status is independent from Markdown type conformance.
Missing and null values are reported as missing identities. Malformed YAML or TOML, invalid formats, arrays, objects, booleans, non-integral numbers, and lossy conversions produce structured invalid-identity diagnostics. Identity status is independent from Markdown type conformance.

## Stability and Collisions

Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Working with Markdown Documents

Create a `MarkdownDocument` when you need structured access to Markdown content whose YAML can be parsed.
Create a `MarkdownDocument` when you need structured access to Markdown content whose YAML or TOML frontmatter can be parsed.

## Overview

`MarkdownDocument` accepts raw Markdown text and separates an optional YAML frontmatter block from the document body. Frontmatter is parsed into a Yams mapping so callers can inspect and mutate structured values, while the body remains available as plain text for extraction, formatting, and conversion workflows.
`MarkdownDocument` accepts raw Markdown text and separates optional YAML (`---`) or TOML (`+++`) frontmatter from the document body. Frontmatter is parsed into the format-neutral ``FrontMatter`` mapping so callers can inspect and mutate structured values, while the body remains available as plain text for extraction, formatting, and conversion workflows.

When callers need syntax-aware access to the body, `MarkdownDocument.parseAST()` parses the body with MarkdownSyntax and returns a fresh syntax tree for each call.

Expand All @@ -22,6 +22,6 @@ Use this workflow before applying frontmatter, section, heading, table-of-conten

A `MarkdownDocument` is a parsed interpretation of text. Its frontmatter, body, and AST all come from that text. It deliberately has no identity, path, revision, database table, or object-store key.

A `MarkdownRecord` is the canonical, addressable resource. It owns the original Markdown string plus optional identity, revision, and external `MarkdownRecordContext`. A record can therefore exist when its YAML is invalid and no `MarkdownDocument` can be initialized.
A `MarkdownRecord` is the canonical, addressable resource. It owns the original Markdown string plus optional identity, revision, and external `MarkdownRecordContext`. A record can therefore exist when its frontmatter is invalid and no `MarkdownDocument` can be initialized.

Use a record when assessing types or rules. Parse a document directly when the operation requires only successfully parsed content.
12 changes: 7 additions & 5 deletions Sources/MarkdownUtilitiesCore/Explore/ExploreDocument.swift
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ public struct ExploreDocument: Sendable {
/// Original Markdown source lines split on newline boundaries.
public let sourceLines: [String]

/// Optional YAML frontmatter block from the original source.
/// Optional YAML or TOML frontmatter block from the original source.
public let frontmatter: ExploreFrontmatter?

/// Optional document preamble before the first heading.
Expand Down Expand Up @@ -82,13 +82,15 @@ public struct ExploreDocument: Sendable {
}

private static func detectFrontmatter(in sourceLines: [String]) throws -> ExploreFrontmatter? {
guard sourceLines.first == "---" else {
guard let opening = sourceLines.first,
let format = FrontMatterFormat.allCases.first(where: { $0.delimiter == opening })
else {
return nil
}

for index in 1..<sourceLines.count where sourceLines[index] == "---" {
let rawYAML = sourceLines[1..<index].joined(separator: "\n")
let mapping = try YAMLConversion.parse(rawYAML)
for index in 1..<sourceLines.count where sourceLines[index] == format.delimiter {
let rawFrontMatter = sourceLines[1..<index].joined(separator: "\n")
let mapping = try FrontMatterConversion.parse(rawFrontMatter, format: format)
return ExploreFrontmatter(
lineRange: 1...(index + 1),
fieldCount: mapping.count
Expand Down
Loading