From 837402c086483ffae382d87dbc78b0019d9364b5 Mon Sep 17 00:00:00 2001 From: Vincent Herbst Date: Sun, 20 Sep 2026 19:00:28 +0200 Subject: [PATCH 1/3] fix(producer): a plan boundary is compared on the axis its container stamped (AE#561) The keyframe-aligned plan's boundaries ARE the container's index entries, and containers disagree about what an entry's timestamp means: a mov/mp4 sample table holds decode times, a Matroska Cue holds a presentation time. Since #358 the VOD cutter gate compared decode times against both, which is right for mov/mp4 and wrong for every Matroska whose video carries composition offsets, i.e. every MKV with B-frames. There, no IRAP ever reached its own boundary: its decode time sits a composition offset below it. The gate never opened on the planned keyframe, so audio, which is routed by boundary and not gated, opened the segment instead and the IRAP stayed in the segment before. Every segment then began mid-GOP, roughly one IRAP below its own first random-access point, which is what #412 reported on every segment of the field report. Nothing in such a segment can start a decode run, so playback survived only while AVPlayer decoded THROUGH the boundaries and stopped with CoreMediaErrorDomain -19602 the first time it had to decode FROM one, at a position that depends on the encode rather than on elapsed time. The automatic item reload re-fetched the same segment and died on it again. The gate now compares a packet on the plan's own axis (PlanBoundaryAxis, read from the demuxer's format name, never from the URL or host metadata), so a keyframe hits its own boundary exactly on either container. The AE#412 reach is recorded on that same axis, since it is measured against a boundary. mov/mp4 sessions are unchanged byte for byte. Measured on one HEVC stream muxed into both containers, the stamping being the only difference: before, the MKV opened seg1 and up on a dependent picture with its first IRAP 19 frames in and the isolated decode logged 184 reference errors; after, every segment opens on a sync sample and the decode is clean, while the MP4 control keeps its existing timestamps. H.264 in Matroska was affected identically. PlanBoundaryAxisTests pins both arms and fails on the Matroska arm alone when the gate is put back on the wrong axis. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F --- CHANGELOG.md | 16 ++ Scripts/fetch-fixtures.sh | 19 ++ .../Video/HLSSegmentProducer.swift | 33 +++- .../AetherEngine/Video/HLSVideoEngine.swift | 12 ++ .../AetherEngine/Video/PlanBoundaryAxis.swift | 61 ++++++ .../AetherEngine/Video/VODSegmentCutter.swift | 15 +- .../PlanBoundaryAxisTests.swift | 184 ++++++++++++++++++ docs/architecture.md | 5 +- 8 files changed, 328 insertions(+), 17 deletions(-) create mode 100644 Sources/AetherEngine/Video/PlanBoundaryAxis.swift create mode 100644 Tests/AetherEngineTests/PlanBoundaryAxisTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 497229642..2b6c2dafb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,22 @@ the public-API contract. ### Fixed +- **Every segment of a Matroska with B-frames now opens on a keyframe (AE#561).** The + keyframe-aligned plan's boundaries ARE the container's index entries, and containers disagree + about what an entry's timestamp means: a mov/mp4 sample table holds decode times, a Matroska Cue + holds a presentation time. The cutter gate compared decode times against both (#358), so on any + MKV whose video carries composition offsets no IRAP ever reached its own boundary. The gate never + opened on the planned keyframe, audio (routed by boundary and not gated) opened the segment + instead, and every segment began mid-GOP about one IRAP below its own first random-access point, + with nothing in it a cold decode could start from. Playback survived only while AVPlayer decoded + THROUGH the boundaries; the first time it had to decode FROM one it stopped with + `CoreMediaErrorDomain -19602`, at a position that depends on the encode rather than on elapsed + time, and the automatic item reload died on the same segment. The gate now compares a packet on + the plan's own axis (`PlanBoundaryAxis`, read from the demuxer's format name), so a keyframe hits + its own boundary exactly on either container, and the AE#412 reach is measured on that same axis. + mov/mp4 sessions are unchanged, byte for byte. Pinned by `PlanBoundaryAxisTests` on one HEVC + stream muxed into both containers, where the stamping is the only difference. + - **A live recording now starts at zero instead of carrying the broadcast's own clock.** Copying the source timestamps verbatim produced a recording whose first presentation timestamp lay hours past its own beginning, which a duration probe reports as the offset rather than the diff --git a/Scripts/fetch-fixtures.sh b/Scripts/fetch-fixtures.sh index 82c6c994b..c1e962a38 100755 --- a/Scripts/fetch-fixtures.sh +++ b/Scripts/fetch-fixtures.sh @@ -18,6 +18,7 @@ # restart-witness-subs.mkv - same as MKV with an embedded SRT track (pump tap) # a53-captions.mp4 - H.264 with in-picture A/53 CEA-608 SEI (#131, #259) # hev1-inband-xps.mp4 - HEVC with in-band VPS/SPS/PPS and an empty hvcC +# cue-axis-bframes.mkv/.mp4 - one HEVC B-pyramid stream in both containers (AE#561) # # Real-world DV / Atmos / multichannel sources have to come from your # own library. Drop those into ./Fixtures/user/ (also gitignored) @@ -280,6 +281,24 @@ data[start:start + size] = replacement open(path, 'wb').write(bytes(data)) PY +# AE#561: one HEVC elementary stream in both containers, so the only difference between the two +# files is what their index entries are stamped on. A Matroska Cue stores a presentation time, a +# mov/mp4 sample table stores decode times, and the keyframe-aligned plan's boundaries ARE those +# entries. B-frames (b-pyramid) are what makes the two axes differ at all; a dense keyint puts an +# IRAP well inside every 4 s segment, so a segment that lost its own IRAP still carries a later one +# and the defect shows as "opens below its first random-access point" rather than as no picture. +# The .mp4 is a stream copy on purpose: same packets, same timestamps, different index stamping. +echo "→ cue-axis-bframes.mkv + .mp4 (HEVC B-pyramid, same stream in both containers, 16s)" +ffmpeg -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=size=480x270:rate=24" \ + -f lavfi -i "sine=frequency=440:sample_rate=48000" -t 16 \ + -c:v libx265 -preset ultrafast -pix_fmt yuv420p \ + -x265-params "keyint=21:min-keyint=21:scenecut=0:bframes=4:b-pyramid=1:log-level=none" \ + -c:a aac -b:a 96k "$FIXTURES_DIR/cue-axis-bframes.mkv" +ffmpeg -hide_banner -loglevel error -y \ + -i "$FIXTURES_DIR/cue-axis-bframes.mkv" -c copy \ + "$FIXTURES_DIR/cue-axis-bframes.mp4" + # AetherEngine#268: finite HEVC-in-MPEG-TS HLS VOD, the carriage AVFoundation refuses to build a # video track for. Three shapes, because each one only shows its own defect: # hls-hevc-vod/ PTS origin at ffmpeg's default 1.4 s, 6 s segments, 2 s GOP diff --git a/Sources/AetherEngine/Video/HLSSegmentProducer.swift b/Sources/AetherEngine/Video/HLSSegmentProducer.swift index 3b42f81c1..bb0c8e92b 100644 --- a/Sources/AetherEngine/Video/HLSSegmentProducer.swift +++ b/Sources/AetherEngine/Video/HLSSegmentProducer.swift @@ -474,6 +474,10 @@ final class HLSSegmentProducer: @unchecked Sendable { /// IRAP (AE#268), so neither may be re-anchored for opening past its target. private let boundaryClaimsRandomAccess: Bool + /// AE#561: the axis `segmentBoundaries` are stamped on, so the cutter gate compares like with + /// like. Decode for a mov/mp4 index, presentation for Matroska Cues; see `PlanBoundaryAxis`. + private let planBoundaryAxis: PlanBoundaryAxis + /// AE#408: the gate target actually in force. Starts at `restartTargetVideoPts` and moves BACK when /// the boundary turns out not to be openable, so the segment covers its own advertised start /// instead of carrying content from the far side of a keyframe drought. @@ -1364,6 +1368,7 @@ final class HLSSegmentProducer: @unchecked Sendable { audioFallbackDurationPts: Int64 = 0, restartTargetVideoPts: Int64 = Int64.min, boundaryClaimsRandomAccess: Bool = false, + planBoundaryAxis: PlanBoundaryAxis = .decode, closedCaptionStreamIndex: Int32 = -1, subtitleTapStreamIndices: Set = [], subtitlePacketStreamIndices: Set = [], @@ -1440,6 +1445,7 @@ final class HLSSegmentProducer: @unchecked Sendable { self.audioFallbackDurationPts = audioFallbackDurationPts self.restartTargetVideoPts = restartTargetVideoPts self.boundaryClaimsRandomAccess = boundaryClaimsRandomAccess + self.planBoundaryAxis = planBoundaryAxis self.effectiveGateTargetPts = restartTargetVideoPts self.gateProvenEmptyFromPts = restartTargetVideoPts // Audio target set dynamically once video gate opens (rescaled to audio TB). @@ -3827,17 +3833,19 @@ final class HLSSegmentProducer: @unchecked Sendable { // at the IRAP that reaches its plan boundary, so the IRAP is the segment's first sample // and its open-GOP RASL leading pictures stay with it (#92). Routing by DTS against PTS // boundaries used to drop the IRAP (dts < pts) into the previous segment. - // #358: the VOD plan's boundaries are the mov/mp4 index's sync-sample timestamps, - // which are DECODE times, so the gate compares decode times too. Comparing the - // presentation time against them let a keyframe reach boundaries beyond its own - // by its composition offset (3 s on the field report's remux), consuming plan - // indices that then never opened a segment. Keyframe gating is unchanged, so - // #92 holds: the IRAP is still the segment's first sample and its RASL pictures - // still follow it in decode order. + // #358 / AE#561: the VOD plan's boundaries ARE the container's index entries, and + // what an entry's timestamp means depends on the container: decode times from a + // mov/mp4 sample table, presentation times from a Matroska Cue. The gate compares + // on the plan's own axis (`PlanBoundaryAxis`), because either mismatch costs a + // segment its IRAP: a presentation packet against decode boundaries let a keyframe + // reach boundaries beyond its own (#358), a decode packet against presentation + // boundaries let no keyframe reach its own at all (AE#561). Keyframe gating itself + // is unchanged, so #92 holds: the IRAP is the segment's first sample and its + // open-GOP RASL pictures still follow it in decode order. let thisVideoSeg = isLive ? liveVideoSegmentIndex(pts: packet.pointee.pts, isKeyframe: isVideoKeyframe) - : vodCutter.index(pts: packet.pointee.dts != Int64.min - ? packet.pointee.dts : packet.pointee.pts, + : vodCutter.index(pts: planBoundaryAxis.timestamp(dts: packet.pointee.dts, + pts: packet.pointee.pts), isKeyframe: isVideoKeyframe) if thisVideoSeg != pumpQoSLastSeg { pumpQoSLastSeg = thisVideoSeg @@ -3907,8 +3915,13 @@ final class HLSSegmentProducer: @unchecked Sendable { if !isLive, (prev.pointee.flags & AV_PKT_FLAG_KEY) != 0 { let openIdx = muxer.currentSegmentIndex if firstSyncItemPtsBySegment[openIdx] == nil { + // AE#561: measured against a plan boundary, so stamped on the + // plan's axis. On the wrong one the reach is off by the frame's + // composition offset, which is the distance this very number + // exists to report. firstSyncItemPtsBySegment[openIdx] = - prev.pointee.dts != Int64.min ? prev.pointee.dts : prev.pointee.pts + planBoundaryAxis.timestamp(dts: prev.pointee.dts, + pts: prev.pointee.pts) } } finalizeAndWriteVideo(prev, nextDts: packet.pointee.dts, muxer: muxer) diff --git a/Sources/AetherEngine/Video/HLSVideoEngine.swift b/Sources/AetherEngine/Video/HLSVideoEngine.swift index 16bc71e18..0c547d8a1 100644 --- a/Sources/AetherEngine/Video/HLSVideoEngine.swift +++ b/Sources/AetherEngine/Video/HLSVideoEngine.swift @@ -663,6 +663,13 @@ public final class HLSVideoEngine: @unchecked Sendable { /// claimed it) or a source-declared plan (which aims below its IRAP by design, AE#268). var planBoundariesClaimRandomAccess = false + /// AE#561: the axis `segmentPlan`'s boundaries are stamped on, handed to every producer this + /// session builds so its cutter gate compares a packet on the plan's own axis. Set together with + /// `planBoundariesClaimRandomAccess`, because it is the same plan that makes it meaningful: only + /// the keyframe-aligned plan's boundaries ARE index entries. `.decode` for every other plan, + /// which is what the cutter compared before the axis existed. + var planBoundaryAxis: PlanBoundaryAxis = .decode + /// Guards subsystem refs + `sessionEpoch`. Never held across waits or network I/O so /// `stop()` on the main thread is never blocked behind a restart's 5 s waitForFinish. let restartLock = NSLock() @@ -1225,6 +1232,10 @@ public final class HLSVideoEngine: @unchecked Sendable { sourceDurationSeconds: durationSeconds ) planBoundariesClaimRandomAccess = true + // AE#561: these boundaries ARE this container's index entries, so they carry its + // stamping. Read from the demuxer that produced them, never from the URL or the + // host's metadata: on a remux session the delivered container is the one indexed. + planBoundaryAxis = PlanBoundaryAxis.forContainer(formatName: dem.containerFormatName) let firstKeyframePts = keyframes.sorted().first ?? 0 self.firstKeyframePts = firstKeyframePts let firstKeyframeSeconds = Double(firstKeyframePts) * Double(videoTimeBase.num) / Double(videoTimeBase.den) @@ -2528,6 +2539,7 @@ public final class HLSVideoEngine: @unchecked Sendable { audioFallbackDurationPts: audioFallbackDurationPts, restartTargetVideoPts: videoTarget, boundaryClaimsRandomAccess: planBoundariesClaimRandomAccess, + planBoundaryAxis: planBoundaryAxis, closedCaptionStreamIndex: closedCaptionStreamIndexForSession, subtitleTapStreamIndices: Set(nativeSubtitleSourceStreamIndicesForSession.compactMap { $0 }), subtitlePacketStreamIndices: allEmbeddedSubtitleStreamIndices, // #112 rework diff --git a/Sources/AetherEngine/Video/PlanBoundaryAxis.swift b/Sources/AetherEngine/Video/PlanBoundaryAxis.swift new file mode 100644 index 000000000..2346673e5 --- /dev/null +++ b/Sources/AetherEngine/Video/PlanBoundaryAxis.swift @@ -0,0 +1,61 @@ +import Foundation + +/// AE#561: which timestamp axis a keyframe-aligned plan's boundaries are stamped on. +/// +/// The keyframe-aligned plan's boundaries ARE the container's index entries (`indexedKeyframes`), +/// and containers do not agree on what an index entry's timestamp means. mov/mp4 builds its index +/// from `stts`/`stss`, so an entry is a DECODE time; a Matroska Cue stores the block's timestamp, +/// which is a PRESENTATION time. The VOD cutter compares a video packet against those boundaries, +/// so it has to be handed the matching timestamp. +/// +/// Mixing the two offsets every comparison by the frame's composition offset, in whichever direction +/// the mismatch runs, and both directions have been paid for: +/// +/// - Presentation packet against decode boundaries (#358): a keyframe reached boundaries beyond its +/// own, consuming plan indices that then never opened a segment while the playlist kept offering +/// them. That is why the gate compares decode times today. +/// - Decode packet against presentation boundaries (AE#561): on a Matroska with B-frames NO IRAP +/// ever reaches its own boundary, because its decode time sits a composition offset below it. The +/// keyframe gate therefore never opens on the IRAP the plan named; audio, which is routed by +/// boundary and not gated, opens the segment instead, and the IRAP stays in the segment before. +/// Every segment then begins mid-GOP, roughly one IRAP below its own first random-access point +/// (`#412 seg-N opens 0.751s below its first random-access point` on every segment of the field +/// report), so nothing in it can start a decode run. Playback survives only while AVPlayer decodes +/// THROUGH the boundaries; the first time it has to decode FROM one it answers with -19602. +/// +/// The fix is not to pick one axis but to compare like with like, and then a keyframe hits its own +/// boundary exactly, on either container. +enum PlanBoundaryAxis: Sendable, Equatable { + + /// Index entries are decode timestamps: mov/mp4, and everything libavformat indexes from + /// `pkt->dts`. + case decode + + /// Index entries are presentation timestamps: Matroska/WebM Cues. + case presentation + + /// The axis `formatName`'s index entries are stamped on, from libavformat's demuxer name + /// ("matroska,webm", "mov,mp4,m4a,3gp,3g2,mj2", "mpegts"). + /// + /// Only Matroska claims presentation. `.decode` stays the default on purpose: it is what every + /// other container the engine indexes uses, and what the cutter did before this axis existed, so + /// an unrecognised format keeps today's behaviour instead of inheriting a guess. + static func forContainer(formatName: String?) -> PlanBoundaryAxis { + let names = formatName?.split(separator: ",") ?? [] + return names.contains("matroska") || names.contains("webm") ? .presentation : .decode + } + + /// The packet timestamp to compare against a plan boundary on this axis. + /// + /// Falls back to the other axis when the preferred one is absent (`Int64.min`), which is what the + /// cutter did for a DTS-less packet before the axis existed: a timestamp on the wrong axis still + /// orders the stream, a missing one does not. + func timestamp(dts: Int64, pts: Int64) -> Int64 { + switch self { + case .decode: + return dts != Int64.min ? dts : pts + case .presentation: + return pts != Int64.min ? pts : dts + } + } +} diff --git a/Sources/AetherEngine/Video/VODSegmentCutter.swift b/Sources/AetherEngine/Video/VODSegmentCutter.swift index a51f56319..415e4523b 100644 --- a/Sources/AetherEngine/Video/VODSegmentCutter.swift +++ b/Sources/AetherEngine/Video/VODSegmentCutter.swift @@ -17,12 +17,15 @@ struct VODSegmentCutter { /// Plan boundaries on the ITEM axis, in the timestamps the PLAN is expressed in: `boundaries[i]` /// is the start of segment `baseIndex + i`, i.e. the plan's source timestamp minus the plan anchor. /// - /// #358: for a keyframe-aligned plan those timestamps come from the container index, and mov/mp4 - /// index entries are DECODE timestamps. The gate is therefore fed decode timestamps too. Feeding - /// it presentation timestamps let a keyframe reach boundaries beyond its own by its composition - /// offset, which is a couple of frames on an ordinary encode and was 3 s on a remux carrying an - /// edit list; every boundary inside that offset was consumed and never opened a segment, while - /// the playlist kept offering it. + /// #358 / AE#561: for a keyframe-aligned plan those timestamps come from the container index, and + /// the caller feeds the gate the packet timestamp stamped on the SAME axis (`PlanBoundaryAxis`): + /// decode for a mov/mp4 sample table, presentation for a Matroska Cue. Both mismatches have been + /// paid for. Presentation against decode boundaries let a keyframe reach boundaries beyond its own + /// by its composition offset (a couple of frames on an ordinary encode, 3 s on a remux carrying an + /// edit list), and every boundary inside that offset was consumed without opening a segment while + /// the playlist kept offering it (#358). Decode against presentation boundaries let no keyframe + /// reach its own boundary at all, so audio opened every segment and each one began mid-GOP, + /// carrying no random-access point a cold decode could start on (AE#561). /// `boundaries.count` is the segment count + 1 (the last entry is the end of the final segment). /// /// AE#268: packets reach the cutter with the producer's shift already subtracted, so they are on diff --git a/Tests/AetherEngineTests/PlanBoundaryAxisTests.swift b/Tests/AetherEngineTests/PlanBoundaryAxisTests.swift new file mode 100644 index 000000000..b00a0c510 --- /dev/null +++ b/Tests/AetherEngineTests/PlanBoundaryAxisTests.swift @@ -0,0 +1,184 @@ +// Tests/AetherEngineTests/PlanBoundaryAxisTests.swift +// AE#561: a keyframe-aligned plan's boundaries ARE the container's index entries, and containers +// disagree about what an entry's timestamp means (mov/mp4 sample tables hold decode times, Matroska +// Cues hold presentation times). The cutter gate has to compare a packet on the plan's own axis. +// Fed a decode timestamp against presentation boundaries, no IRAP ever reached its own boundary: +// the gate never opened on it, audio (routed by boundary, ungated) opened the segment instead, and +// every segment began mid-GOP with no random-access point a cold decode could start on. AVPlayer +// answered that with -19602 the first time it had to decode FROM a segment boundary instead of +// through one. +import Foundation +import Testing +@testable import AetherEngine + +// MARK: - Pure decisions + +@Suite("Plan boundary axis") +struct PlanBoundaryAxisDecisionTests { + + @Test("Matroska Cues are presentation times, every other index is decode times") + func containerMapping() { + #expect(PlanBoundaryAxis.forContainer(formatName: "matroska,webm") == .presentation) + #expect(PlanBoundaryAxis.forContainer(formatName: "matroska") == .presentation) + #expect(PlanBoundaryAxis.forContainer(formatName: "webm") == .presentation) + #expect(PlanBoundaryAxis.forContainer(formatName: "mov,mp4,m4a,3gp,3g2,mj2") == .decode) + #expect(PlanBoundaryAxis.forContainer(formatName: "mpegts") == .decode) + // An unrecognised or missing format keeps the behaviour that predates the axis rather than + // inheriting a guess. + #expect(PlanBoundaryAxis.forContainer(formatName: "flv") == .decode) + #expect(PlanBoundaryAxis.forContainer(formatName: nil) == .decode) + } + + @Test("Each axis picks its own timestamp") + func timestampPick() { + #expect(PlanBoundaryAxis.decode.timestamp(dts: 100, pts: 142) == 100) + #expect(PlanBoundaryAxis.presentation.timestamp(dts: 100, pts: 142) == 142) + } + + @Test("A missing timestamp falls back to the other axis, never to Int64.min") + func missingTimestampFallsBack() { + #expect(PlanBoundaryAxis.decode.timestamp(dts: Int64.min, pts: 142) == 142) + #expect(PlanBoundaryAxis.presentation.timestamp(dts: 100, pts: Int64.min) == 100) + } + + /// The gate advances only on a keyframe that has REACHED its boundary. With the boundary stamped + /// on one axis and the packet on the other, the IRAP that owns the boundary misses it by its + /// composition offset, so the cutter walks past the index the playlist keeps advertising. + @Test("A keyframe reaches its own boundary only when both are on the same axis") + func keyframeReachesItsOwnBoundary() { + // One IRAP per 21 frames at 1/16000, composition offset 2 frames: presentation 70000, + // decode 68672. The plan came from Matroska Cues, so its boundary is the presentation time. + let boundaries: [Int64] = [0, 70000, 140000] + var matched = VODSegmentCutter(sourceBoundaries: boundaries, planAnchorPts: 0, baseIndex: 0) + #expect(matched.index(pts: PlanBoundaryAxis.presentation.timestamp(dts: 68672, pts: 70000), + isKeyframe: true) == 1) + + var mismatched = VODSegmentCutter(sourceBoundaries: boundaries, planAnchorPts: 0, baseIndex: 0) + #expect(mismatched.index(pts: PlanBoundaryAxis.decode.timestamp(dts: 68672, pts: 70000), + isKeyframe: true) == 0, + "the AE#561 shape: the IRAP that owns this boundary does not reach it") + } +} + +// MARK: - Witness on a real session + +/// Fixtures/ is local-only by design (gitignored; Scripts/fetch-fixtures.sh regenerates the +/// synthetic clips). The tests skip via `.enabled(if:)` when a clip is absent, e.g. on CI. +private func fixtureURL(_ name: String) -> URL { + URL(fileURLWithPath: #filePath) + .deletingLastPathComponent() + .deletingLastPathComponent() + .deletingLastPathComponent() + .appendingPathComponent("Fixtures") + .appendingPathComponent(name) +} + +private func fixtureExists(_ name: String) -> Bool { + FileManager.default.fileExists(atPath: fixtureURL(name).path) +} + +/// Whether the first VIDEO sample of each fragment in `segment` is a sync sample, in fragment order. +/// +/// A sample is non-sync when bit 16 of its `sample_flags` is set. The flags can arrive three ways +/// and the first sample reads them in this precedence: `trun`'s `first_sample_flags` (0x04), then +/// its per-sample flags (0x400), then `tfhd`'s `default_sample_flags` (0x20). +private func firstSampleIsSyncPerFragment(_ segment: Data, videoTrackID: UInt32 = 1) -> [Bool] { + func u32(_ off: Int) -> UInt32 { + segment.withUnsafeBytes { UInt32(bigEndian: $0.loadUnaligned(fromByteOffset: off, as: UInt32.self)) } + } + func boxes(_ range: Range) -> [(String, Range)] { + var out: [(String, Range)] = [] + var off = range.lowerBound + while off + 8 <= range.upperBound { + let size = Int(u32(off)) + guard size >= 8, off + size <= range.upperBound else { break } + let type = String(bytes: segment[off + 4.. 0 else { continue } + judged += 1 + let sync = firstSampleIsSyncPerFragment(data) + #expect(!sync.isEmpty, "\(fixture): seg\(index) carries no video fragment") + #expect(sync.first == true, + "\(fixture): seg\(index) opens on a dependent picture, so nothing in it can start a decode run (AE#561)") + } + #expect(judged >= 2, "\(fixture): only \(judged) segment(s) past seg0 were served, too few to witness anything") + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 6d16f202c..384cee4fc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -52,7 +52,9 @@ player) clears those totals and rejects any old reconciliation. The playlist's segment boundaries come from a keyframe-aligned plan that mirrors the `hls` muxer's cut algorithm (segment N ends at the first IRAP at-or-after `(N+1) * targetSegmentDuration`), built in `HLSVideoEngine+SegmentPlanning.swift`. It needs the source's keyframe positions, which for MKV / MP4 come from a brief cue prewarm (a bounded seek that loads the Cues / `stss` index) and for MPEG-TS / M2TS come only from whatever `avformat_find_stream_info` plus that seek happened to scan. `keyframeIndexIsTrustworthy` gates the plan on two witnesses before trusting that index, falling back to a uniform-stride plan otherwise: the largest **gap** between consecutive keyframes must stay under a cap (a clustered TS index gaps by thousands of seconds; trusting it builds a multi-thousand-second first segment the `frag_custom` muxer buffers whole in RAM, #64), and the **coverage** from first to last indexed keyframe must span at least one `targetSegmentDuration`. The coverage check catches a remote MKV whose Cues tail read fails: the prewarm loads nothing, only the open-time keyframes survive bunched in the first few seconds, their gaps are tiny so the gap check passes, yet no keyframe reaches the first segment boundary, so the keyframe planner would degenerate to a single whole-file segment AVPlayer loads zero tracks from (`kFigAssetError_TrackNotFound`, #91). The uniform fallback anchors segment 0 at the content start so a late-starting title doesn't advertise empty leading segments. -At runtime the producer honors those boundaries with a keyframe-gated, decode-order cut (`VODSegmentCutter`): a segment opens only at the IRAP whose PTS reaches the next boundary, so the IRAP is the segment's first sample and its open-GOP leading pictures stay with it, matching the live path and the `hls` muxer. The earlier routing keyed each packet to a segment by its DTS against the PTS-valued boundaries, so under B-frame reorder a keyframe whose DTS trailed its PTS fell into the previous segment and the next one started mid-GOP, decode-dependent on its predecessor; a fresh decode at that boundary (rebuffer recovery) surfaced it as transient blocky corruption (#92). +At runtime the producer honors those boundaries with a keyframe-gated, decode-order cut (`VODSegmentCutter`): a segment opens only at the IRAP that reaches the next boundary, so the IRAP is the segment's first sample and its open-GOP leading pictures stay with it, matching the live path and the `hls` muxer. The earlier routing keyed each packet to a segment by its DTS against the PTS-valued boundaries, so under B-frame reorder a keyframe whose DTS trailed its PTS fell into the previous segment and the next one started mid-GOP, decode-dependent on its predecessor; a fresh decode at that boundary (rebuffer recovery) surfaced it as transient blocky corruption (#92). + +Which timestamp "reaches" a boundary is a question about the CONTAINER, because the plan's boundaries ARE its index entries, and containers do not agree on what an entry means: a mov/mp4 sample table holds decode times, a Matroska Cue holds a presentation time. The gate therefore compares a packet on the plan's own axis (`PlanBoundaryAxis`, chosen from the demuxer's format name), and both mismatches have been paid for. A presentation packet against decode boundaries let a keyframe reach boundaries beyond its own by its composition offset (a couple of frames on an ordinary encode, 3 s on a remux carrying an edit list), consuming plan indices that never opened a segment while the playlist kept offering them (#358). A decode packet against presentation boundaries let no keyframe reach its own boundary at all, on every Matroska whose video carries composition offsets, which is every MKV with B-frames: the gate never opened on the planned IRAP, audio (routed by boundary, not gated) opened each segment instead, and every segment began mid-GOP roughly one IRAP below its own first random-access point. That is invisible while AVPlayer decodes THROUGH the boundaries and fatal the first time it has to decode FROM one, which it answers with `CoreMediaErrorDomain -19602` at a position that depends on the encode rather than on elapsed time (AE#561). `PlanBoundaryAxisTests` pins it on one HEVC stream muxed into both containers, where only the stamping differs. Seeks are demand-driven: `AVPlayer` just fetches segments at the new position, and `VideoSegmentProvider` only tears the producer down and re-anchors it at the requested index (`restartHandler`, burst-coalesced by `RestartCoalescer`) when the request cannot be served from `SegmentCache`. A restart is the expensive path (it re-seeks the demuxer, slow on remote sources, #93), so for VOD the cache retains already-produced segments beyond its hard `[target - backwardWindow, target + forwardWindow]` window under a byte budget (2 GiB, clamped to a quarter of the tmp volume's free capacity), evicting farthest-from-target first once it fills. The hard window itself is never evicted, so that budget bounds only the extras around it, which holds by construction as long as the forward window stays at or below the historical 150-segment ceiling (~1.5 GB of 4K HEVC). A larger window is an explicit host opt-in into a whole-source prefetch (`LoadOptions.forwardBufferSegments`, up to 2700 segments ~ 3 h, #207): it drops the budget's 2 GiB default cap while keeping the quarter-of-free-space clamp, and the producer parks once its race-ahead owns that many bytes forward of the consumer target, resuming as eviction behind the playhead frees room (`PrefetchDiskBudget`). An opt-in prefetch therefore tracks the disk budget rather than the source length. A seek back into the retained span, and the forward march that follows it, is then a pure cache hit with zero producer restarts; only a seek into never-produced content restarts. Live sessions keep window-only pruning, since the sliding playlist has already dropped everything behind the window. When a non-disc VOD restart is required, it seeks on the video stream's native timestamp axis to the exact IRAP stored in the segment plan, with that timestamp as the lower bound; a global time seek can land a whole GOP earlier on multi-stream MP4 and turn the producer's scan-forward gate into a long remote read (#191). Disc plans use folded multi-clip timestamps and keep the global-time seek. Restarts on a slow link are further contained (#93 residual): a fetch that is waiting for an in-flight restart rides its progress instead of burning a fixed retry budget into a 503 (and never re-fires a restart at its own stale index), the wedged-restart fresh reopen skips `find_stream_info` (the session already holds saved codec configs and the segment plan), lazy native subtitle readers defer while a restart executes, and the FIRST producer of a resumed session anchors directly at the resume segment instead of producing seg0 into an immediate teardown. Retained scrub bands leave interior holes inside the cache's stored min/max index range, and residency there is not proof a segment exists: a fetch inside the range waits (2 s) only when the active producer's forward march actually covers the requested index, and restarts immediately otherwise (#129). @@ -469,6 +471,7 @@ Sources/AetherEngine/ │ ├── VideoSegmentProvider.swift Native path: playlist-facing segment provider (live sliding window, restart heuristics, producer-coverage-gated sparse-hole waits #129) │ ├── HLSSegmentProducer.swift Native path: pump loop reading from Demuxer, feeding MP4SegmentMuxer, cutting fragments keyframe-gated in decode order so the IRAP opens its segment (#92); SSAI program-switch detection + no-cut watchdog │ ├── VODSegmentCutter.swift Native path: decode-order, keyframe-gated VOD segment cutter (the IRAP opens its segment, #92) +│ ├── PlanBoundaryAxis.swift Which axis a container stamps its index entries on, and therefore which packet timestamp the cutter gate may compare against a plan boundary: decode for a mov/mp4 sample table, presentation for a Matroska Cue (#358, AE#561) │ ├── VideoConfigRecord.swift The `hvcC` / `avcC` config record and the framing question hanging off it (#365): movenc decides whether to Annex-B-convert every sample from the EXTRADATA rather than the packet, so a source carrying Annex-B parameter sets while muxing length-prefixed NALs has each sample rewritten by a converter that finds no start codes. Mirrors movenc's own two tests instead of an equivalent-looking predicate, since predicting that decision wrong is the whole defect │ ├── H264SPS.swift Hand-rolled H.264 SPS parser (SSAI ad-creative coded dimensions / codec config) │ ├── H264CompositionOffsetRepair.swift Rebuilds the presentation axis of an MP4 whose writer dropped `ctts` while the bitstream still reorders pictures: libavcodec's H.264 parser supplies each access unit's picture order count without decoding, a fail-closed head sample settles the ladder and the shift, and packets are rewritten to the timeline a correct muxer would have written, so the native path and hardware decode are kept (#409) From 4bc856d64015bd496ac0b6a0a3d393a9acddceee Mon Sep 17 00:00:00 2001 From: Vincent Herbst Date: Sun, 20 Sep 2026 19:17:21 +0200 Subject: [PATCH 2/3] fix(audio): the bridge's published origin accounts for the encoder's priming (AE#561) `baseMediaDecodeTime` is `unsigned int(64)`, so a negative published timestamp is not unusual, it is unrepresentable. The bridge's PTS counter stamps the FRAME handed to the encoder. An encoder that declares `initial_padding` stamps its first PACKET a padding below that frame, so that a consumer which discards the priming lands back on the source position: 256 samples on the AC-3 family, which surround-compat mode reaches for above two channels, and none on FLAC. Nothing discards it here, because the muxer writes no edit list on purpose and the init segment has to stay restart-invariant, so the priming plays as the silence it is. Without the offset the published timeline therefore started a padding BELOW the source, and at source position 0 that is negative: -256 went out as 2^64 - 256, and the whole first audio fragment was placed 584 thousand years out, losing its ~190 ms of audio at the start of every bridged multichannel session. Measured in the AE#561 reporter's own captures, where the TrueHD track (bridged, EAC3 encoder) carries tfdt 0xFFFFFFFFFFFFFF00 while the same file's DD+5.1 track (stream copy, no bridge) starts cleanly at 0, and reproduced here on 5.1 PCM in Matroska. The counter now carries the padding, which puts the first packet exactly on the source position and costs the content the padding's 5.3 ms, two orders below the lip-sync threshold and what an unsignalled priming is worth. Applied on every rebase rather than only near zero, so a restart mid-file inherits the same relationship instead of stepping by a padding. The FLAC path is unchanged, measured: it declares no padding. BridgedAudioOriginTests fails without the offset with exactly the reported value. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F --- CHANGELOG.md | 12 ++ Scripts/fetch-fixtures.sh | 14 +++ Sources/AetherEngine/Audio/AudioBridge.swift | 15 +++ .../BridgedAudioOriginTests.swift | 112 ++++++++++++++++++ 4 files changed, 153 insertions(+) create mode 100644 Tests/AetherEngineTests/BridgedAudioOriginTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b6c2dafb..ffac06aea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,18 @@ the public-API contract. mov/mp4 sessions are unchanged, byte for byte. Pinned by `PlanBoundaryAxisTests` on one HEVC stream muxed into both containers, where the stamping is the only difference. +- **Bridged multichannel audio no longer publishes its first fragment 584 thousand years out + (AE#561 follow-up).** `baseMediaDecodeTime` is `unsigned int(64)`, so a negative published + timestamp is unrepresentable rather than merely unusual. The audio bridge stamps the frame it + hands the encoder, and an encoder that declares `initial_padding` stamps its first packet a + padding below that frame (256 samples on the AC-3 family, which is what surround-compat mode + reaches for above two channels; FLAC declares none). At source position 0 that published -256 as + 2^64 - 256, and the session lost the ~190 ms of audio in that fragment. Nothing discards the + priming here, because the muxer writes no edit list on purpose, so the counter now carries the + padding and the content pays its 5.3 ms instead, two orders below the lip-sync threshold. Applied + on every rebase, so a restart mid-file keeps the same relationship instead of stepping by a + padding. Pinned by `BridgedAudioOriginTests` on a 5.1 PCM Matroska. + - **A live recording now starts at zero instead of carrying the broadcast's own clock.** Copying the source timestamps verbatim produced a recording whose first presentation timestamp lay hours past its own beginning, which a duration probe reports as the offset rather than the diff --git a/Scripts/fetch-fixtures.sh b/Scripts/fetch-fixtures.sh index c1e962a38..2c2e4514e 100755 --- a/Scripts/fetch-fixtures.sh +++ b/Scripts/fetch-fixtures.sh @@ -19,6 +19,7 @@ # a53-captions.mp4 - H.264 with in-picture A/53 CEA-608 SEI (#131, #259) # hev1-inband-xps.mp4 - HEVC with in-band VPS/SPS/PPS and an empty hvcC # cue-axis-bframes.mkv/.mp4 - one HEVC B-pyramid stream in both containers (AE#561) +# bridge-eac3-51.mkv - 5.1 PCM in MKV, drives the EAC3 audio bridge (AE#561 follow-up) # # Real-world DV / Atmos / multichannel sources have to come from your # own library. Drop those into ./Fixtures/user/ (also gitignored) @@ -299,6 +300,19 @@ ffmpeg -hide_banner -loglevel error -y \ -i "$FIXTURES_DIR/cue-axis-bframes.mkv" -c copy \ "$FIXTURES_DIR/cue-axis-bframes.mp4" +# AE#561 follow-up: multichannel PCM in Matroska, which routes audio through the bridge in +# surround-compat mode, so the encoder is EAC3 rather than FLAC. That distinction is the whole +# point: FFmpeg's AC-3 family declares `initial_padding = 256` and stamps its first packet a +# padding below the frame it encoded, while FLAC declares none. 5.1 because the mode only reaches +# for EAC3 above two channels. +echo "→ bridge-eac3-51.mkv (5.1 PCM in MKV, drives the EAC3 bridge, 8s)" +ffmpeg -hide_banner -loglevel error -y \ + -f lavfi -i "testsrc2=size=320x180:rate=24" \ + -f lavfi -i "sine=frequency=440:sample_rate=48000" -t 8 \ + -c:v libx264 -preset veryfast -g 48 -pix_fmt yuv420p -b:v 200k \ + -af "pan=5.1|c0=c0|c1=c0|c2=c0|c3=c0|c4=c0|c5=c0" \ + -c:a pcm_s24le "$FIXTURES_DIR/bridge-eac3-51.mkv" + # AetherEngine#268: finite HEVC-in-MPEG-TS HLS VOD, the carriage AVFoundation refuses to build a # video track for. Three shapes, because each one only shows its own defect: # hls-hevc-vod/ PTS origin at ffmpeg's default 1.4 s, 6 s segments, 2 s GOP diff --git a/Sources/AetherEngine/Audio/AudioBridge.swift b/Sources/AetherEngine/Audio/AudioBridge.swift index 92c1fcdcc..5314350c4 100644 --- a/Sources/AetherEngine/Audio/AudioBridge.swift +++ b/Sources/AetherEngine/Audio/AudioBridge.swift @@ -861,7 +861,22 @@ final class AudioBridge: @unchecked Sendable { } stats.framesDecoded += 1 if rebaseFromNextSourcePTS, packetPts != Self.avNoPTS { + // AE#561 follow-up: this counter stamps the FRAME handed to the encoder, and an + // encoder that declares `initial_padding` stamps its first PACKET a padding BELOW + // that frame (256 samples on the AC-3 family, 0 on FLAC), so that a consumer which + // discards the priming lands back on the source position. Nothing discards it + // here: the muxer writes no edit list on purpose, since the init segment has to + // stay restart-invariant, so the priming plays as the silence it is. Without the + // offset the published timeline therefore STARTS a padding below the source, and + // at source 0 that is a negative `baseMediaDecodeTime`, a field that is + // `unsigned int(64)`: -256 went out as 2^64 - 256 and AVPlayer placed the whole + // first audio fragment 584 thousand years out, losing its ~190 ms of audio. The + // offset costs the content the padding's 5.3 ms instead, which is what an + // unsignalled priming is worth and two orders below the lip-sync threshold. It is + // applied on every rebase, not only near zero, so a restart mid-file inherits the + // same relationship instead of stepping by a padding. nextEncoderPTS = av_rescale_q(packetPts, srcTimeBase, encoderTimeBase) + &+ Int64(enc.pointee.initial_padding) rebaseFromNextSourcePTS = false } try resampleAndPushIntoFIFO(srcFrame: sf, enc: enc, swr: swr, fifo: fifoPtr) diff --git a/Tests/AetherEngineTests/BridgedAudioOriginTests.swift b/Tests/AetherEngineTests/BridgedAudioOriginTests.swift new file mode 100644 index 000000000..3b6e75f38 --- /dev/null +++ b/Tests/AetherEngineTests/BridgedAudioOriginTests.swift @@ -0,0 +1,112 @@ +// Tests/AetherEngineTests/BridgedAudioOriginTests.swift +// AE#561 follow-up: `baseMediaDecodeTime` is `unsigned int(64)`, so a negative published timestamp +// is not merely unusual, it is unrepresentable. The audio bridge stamps the FRAME it hands the +// encoder, and an encoder declaring `initial_padding` stamps its first PACKET a padding below that +// frame (256 samples on the AC-3 family, 0 on FLAC). At source position 0 that published -256 as +// 2^64 - 256, and AVPlayer placed the first audio fragment 584 thousand years out, losing its audio. +// Nothing discards the priming here, because the muxer writes no edit list on purpose, so the +// counter carries the padding and the content pays its 5.3 ms instead. +import Foundation +import Testing +@testable import AetherEngine + +private func fixtureURL(_ name: String) -> URL { + URL(fileURLWithPath: #filePath) + .deletingLastPathComponent() + .deletingLastPathComponent() + .deletingLastPathComponent() + .appendingPathComponent("Fixtures") + .appendingPathComponent(name) +} + +private func fixtureExists(_ name: String) -> Bool { + FileManager.default.fileExists(atPath: fixtureURL(name).path) +} + +/// Every `(trackID, baseMediaDecodeTime)` in the segment, in fragment order. +private func trackOrigins(_ segment: Data) -> [(track: UInt32, tfdt: UInt64)] { + func u32(_ off: Int) -> UInt32 { + segment.withUnsafeBytes { UInt32(bigEndian: $0.loadUnaligned(fromByteOffset: off, as: UInt32.self)) } + } + func u64(_ off: Int) -> UInt64 { + segment.withUnsafeBytes { UInt64(bigEndian: $0.loadUnaligned(fromByteOffset: off, as: UInt64.self)) } + } + func boxes(_ range: Range) -> [(String, Range)] { + var out: [(String, Range)] = [] + var off = range.lowerBound + while off + 8 <= range.upperBound { + let size = Int(u32(off)) + guard size >= 8, off + size <= range.upperBound else { break } + let type = String(bytes: segment[off + 4.. Date: Sun, 20 Sep 2026 19:22:37 +0200 Subject: [PATCH 3/3] fix(producer): the boundary tolerance stops paying for a skew its axis does not have (AE#561) The AE#408 tolerance decides when a segment opened so far past its boundary that going back for an earlier sync sample is worth a second seek, and it carried the stream's declared reorder depth. That term pays for an AXIS MISMATCH, not for anything the stream does: the gate judges presentation time (AE#169 round 3), so a boundary stamped in DECODE time puts a correctly indexed keyframe a reorder delay above it. A Matroska Cue is already a presentation time. There a correctly indexed keyframe presents exactly at its boundary, and the reorder term only widened the window in which a genuinely late open escaped its re-aim, on the very container AE#408 was reported against. On that axis the tolerance is now the floor, which still absorbs a Cue that is approximate rather than skewed. mov/mp4 keeps the term, because there the skew is real. The gate's own comparison is left lenient on purpose, and the reason is now written down rather than assumed: it answers a different question than the cutter gate does. Being wrong in the strict direction costs the whole restart, and the reported AE#169 geometry (target 2878501 with the anchor IRAP at dts 2878495 and pts 2878620) has the boundary falling BETWEEN that keyframe's two timestamps, so it matches neither axis and only the permissive reading admits it. Since pts >= dts holds for any conforming stream, judging presentation time cannot starve on a skew in either direction. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F --- CHANGELOG.md | 15 ++++++ .../Video/HLSSegmentProducer.swift | 47 ++++++++++++++----- .../Issue408BoundaryRandomAccessTests.swift | 26 ++++++++++ 3 files changed, 75 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ffac06aea..8a09860c1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ the public-API contract. mov/mp4 sessions are unchanged, byte for byte. Pinned by `PlanBoundaryAxisTests` on one HEVC stream muxed into both containers, where the stamping is the only difference. + - **Bridged multichannel audio no longer publishes its first fragment 584 thousand years out (AE#561 follow-up).** `baseMediaDecodeTime` is `unsigned int(64)`, so a negative published timestamp is unrepresentable rather than merely unusual. The audio bridge stamps the frame it @@ -108,6 +109,20 @@ the public-API contract. 10-bit HEVC with no record, the demuxer now reads the first RPU and, if it reads profile 5, adds the missing record so the existing Profile 5 paths apply. Any other source is left alone. +### Changed + +- **A restart into a Matroska boundary re-aims on the distance it actually overshot (AE#561).** The + AE#408 tolerance, which decides when a segment opens so far past its boundary that going back for + an earlier sync sample is worth it, carried the stream's reorder depth. That term pays for a + boundary stamped in decode time being judged by presentation time, not for anything the stream + does, and a Matroska Cue is already a presentation time: there a correctly indexed keyframe + presents exactly at its boundary, and the term only widened the window in which a genuinely late + open escaped its re-aim. On that axis the tolerance is now the floor, which sharpens the decision + on the container AE#408 was reported against. mov/mp4 keeps the reorder term, because there the + skew is real. The gate's own comparison is deliberately left lenient; the reported AE#169 geometry + has the boundary falling between the anchor keyframe's two timestamps, matching neither axis, and + only the permissive reading admits it at all. + ## [7.8.0] - 2026-09-20 ### Added diff --git a/Sources/AetherEngine/Video/HLSSegmentProducer.swift b/Sources/AetherEngine/Video/HLSSegmentProducer.swift index bb0c8e92b..af2ba52ea 100644 --- a/Sources/AetherEngine/Video/HLSSegmentProducer.swift +++ b/Sources/AetherEngine/Video/HLSSegmentProducer.swift @@ -866,11 +866,23 @@ final class HLSSegmentProducer: @unchecked Sendable { } /// AE#169 round 3 pure decision: whether a video packet opens the restart scan-forward gate. - /// The gate target is a plan-boundary PTS (`segmentPlan[baseIndex].startPts`), so the packet - /// is judged by presentation time. Comparing DTS dropped the exact IRAP the restart seeked - /// for (a keyframe's DTS sits a reorder delay below its own PTS; same defect class as the #92 - /// cutter fix): mid-file the next IRAP rescued the miss one GOP late, but at the file tail no - /// later IRAP exists, so the unbounded VOD gate starved to EOF with zero packets written. + /// The gate target is a plan boundary (`segmentPlan[baseIndex].startPts`), and the packet is + /// judged by presentation time. Comparing DTS dropped the exact IRAP the restart seeked for (a + /// keyframe's DTS sits a reorder delay below its own PTS; same defect class as the #92 cutter + /// fix): mid-file the next IRAP rescued the miss one GOP late, but at the file tail no later + /// IRAP exists, so the unbounded VOD gate starved to EOF with zero packets written. + /// + /// AE#561 deliberately did NOT make this axis-matched the way the cutter gate is, and the + /// reason is that the two answer different questions. The cutter decides which segment a packet + /// belongs to, where being wrong by a composition offset costs a segment its IRAP, and it can + /// afford exactness because it sees every packet. This gate decides where production STARTS + /// after a seek, and being wrong in the strict direction costs the whole restart: the reported + /// geometry (target 2878501, the anchor IRAP at dts 2878495 and pts 2878620) has the boundary + /// falling BETWEEN that keyframe's two timestamps, so it matches neither axis and only the + /// lenient comparison admits it. Since `pts >= dts` holds for any conforming stream, judging + /// presentation time is the most permissive reading and cannot starve on a skew in either + /// direction. What the axis buys here is knowing how much of the resulting overshoot is real, + /// which is what `boundaryOpenToleranceTicks` spends it on. static func videoGateTargetSatisfied(pts: Int64, dts: Int64, targetPts: Int64) -> Bool { if targetPts == Int64.min { return true } let ts = pts != Int64.min ? pts : dts @@ -901,16 +913,24 @@ final class HLSSegmentProducer: @unchecked Sendable { /// AE#408: how far past the boundary a sync sample may present before it is worth going back for. /// - /// A container index entry is a DECODE timestamp while the gate judges presentation time - /// (AE#169 round 3), so even a perfectly formed index puts the keyframe's PTS a reorder delay - /// above the boundary it was indexed at. Charging that skew a second seek would re-aim on every - /// restart of a B-pyramid encode, so the tolerance covers the stream's own declared depth plus a - /// frame, and never falls below the floor. + /// The gate judges presentation time (AE#169 round 3), so where the plan's boundaries are DECODE + /// timestamps even a perfectly formed index puts the keyframe's PTS a reorder delay above the + /// boundary it was indexed at. Charging that skew a second seek would re-aim on every restart of + /// a B-pyramid encode, so the tolerance covers the stream's own declared depth plus a frame, and + /// never falls below the floor. + /// + /// AE#561: that reorder term pays for an AXIS MISMATCH, not for anything the stream does. A plan + /// whose boundaries are PRESENTATION timestamps (Matroska Cues) has no such skew, a correctly + /// indexed keyframe presents exactly at its boundary, and the term would only widen the window in + /// which a genuinely late open escapes its re-aim. On that axis the tolerance is the floor, which + /// sharpens the decision on the very container AE#408 was reported against. The floor still + /// absorbs an index that is approximate rather than skewed, which Matroska Cues frequently are. static func boundaryOpenToleranceTicks( - reorderFrames: Int32, frameDurationPts: Int64, floorTicks: Int64 + reorderFrames: Int32, frameDurationPts: Int64, floorTicks: Int64, + planAxis: PlanBoundaryAxis = .decode ) -> Int64 { let frame = Swift.max(0, frameDurationPts) - let reorder = Int64(Swift.max(0, reorderFrames)) &* frame + let reorder = planAxis == .decode ? Int64(Swift.max(0, reorderFrames)) &* frame : 0 return Swift.max(floorTicks, reorder &+ frame) } @@ -3480,7 +3500,8 @@ final class HLSSegmentProducer: @unchecked Sendable { toleranceTicks: Self.boundaryOpenToleranceTicks( reorderFrames: videoConfig.codecpar.pointee.video_delay, frameDurationPts: videoFallbackDurationPts, - floorTicks: Int64(Self.boundaryOpenToleranceSeconds / sourceVideoTbSeconds)), + floorTicks: Int64(Self.boundaryOpenToleranceSeconds / sourceVideoTbSeconds), + planAxis: planBoundaryAxis), attemptsUsed: gateBackoffAttempts, maxAttempts: Self.gateBackoffStepsSeconds.count), reanchorGateBelowBoundary(reason: "its first sync sample presents " diff --git a/Tests/AetherEngineTests/Issue408BoundaryRandomAccessTests.swift b/Tests/AetherEngineTests/Issue408BoundaryRandomAccessTests.swift index 9af5a4c99..19fe43dd3 100644 --- a/Tests/AetherEngineTests/Issue408BoundaryRandomAccessTests.swift +++ b/Tests/AetherEngineTests/Issue408BoundaryRandomAccessTests.swift @@ -132,6 +132,32 @@ struct Issue408BoundaryRandomAccessTests { maxAttempts: HLSSegmentProducer.gateBackoffStepsSeconds.count)) } + /// AE#561: the reorder term pays for a boundary stamped in DECODE time being judged by + /// presentation time. A Matroska Cue is already a presentation time, so a correctly indexed + /// keyframe presents exactly at its boundary and the term would only let a genuinely late open + /// escape its re-aim. The floor still stands, because a Cue can be approximate without being + /// skewed. + @Test("a presentation-stamped plan pays no reorder term, and re-aims where a decode-stamped one would not") + func presentationAxisDropsTheReorderTerm() { + let decodeTicks = HLSSegmentProducer.boundaryOpenToleranceTicks( + reorderFrames: 16, frameDurationPts: 42, floorTicks: 500, planAxis: .decode) + let presentationTicks = HLSSegmentProducer.boundaryOpenToleranceTicks( + reorderFrames: 16, frameDurationPts: 42, floorTicks: 500, planAxis: .presentation) + #expect(decodeTicks == 16 * 42 + 42) + #expect(presentationTicks == 500) + + // A keyframe presenting 0.6 s past the boundary: inside the decode-stamped tolerance, which + // cannot tell it from the reorder skew, and outside the presentation-stamped one, where the + // whole 0.6 s is a real gap the segment would otherwise carry under the boundary's name. + let late = Self.boundary + 600 + #expect(!HLSSegmentProducer.shouldReanchorBeforeOpening( + keyframePts: late, boundaryPts: Self.boundary, toleranceTicks: decodeTicks, + attemptsUsed: 0, maxAttempts: HLSSegmentProducer.gateBackoffStepsSeconds.count)) + #expect(HLSSegmentProducer.shouldReanchorBeforeOpening( + keyframePts: late, boundaryPts: Self.boundary, toleranceTicks: presentationTicks, + attemptsUsed: 0, maxAttempts: HLSSegmentProducer.gateBackoffStepsSeconds.count)) + } + @Test("a stream without reorder keeps the floor") func noReorderKeepsFloor() { #expect(HLSSegmentProducer.boundaryOpenToleranceTicks(