diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c1299d5..53ca983e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,6 +40,12 @@ jobs: - name: swift test run: swift test + - name: Timestamp policy and generated media regressions + run: | + brew list ffmpeg >/dev/null 2>&1 || brew install ffmpeg + bash Scripts/test-h264-matroska-timestamps.sh + bash Scripts/test-h264-timestamp-controls.sh + # The transport probe is a device harness and runs nowhere in CI, so nothing else would keep # it compiling. A harness that stopped building is discovered by the person who needed it. - name: build the transport probe (AE#377) diff --git a/CHANGELOG.md b/CHANGELOG.md index 767414dc..f80ba71b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,14 @@ the public-API contract. ## [Unreleased] -_Nothing yet._ +### Fixed + +- Positively identified Matroska H.264 coding-order timestamp ladders now keep + their original presentation slots but assign each slot to its parsed picture + order. This removes persistent judder on the affected native/hardware VOD path + at startup and after seeks. The IDR-bounded policy leaves healthy/unproven + input untouched and preserves packet payloads, audio, and the seek-index axis. + It is separate from the existing missing-MP4-composition-offset policy. ## [6.71.0] - 2026-09-06 diff --git a/Scripts/test-h264-matroska-timestamps.sh b/Scripts/test-h264-matroska-timestamps.sh new file mode 100644 index 00000000..68df32de --- /dev/null +++ b/Scripts/test-h264-matroska-timestamps.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-matroska-time.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +# Focused Foundation CLI test, no Apple app target or package resolution. +xcrun swiftc -swift-version 6 \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264MatroskaTimestampRepair.swift" \ + "$TASK_ROOT/Scripts/tests/H264MatroskaTimestampStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" "$TASK_ROOT/Scripts/tests/Data/MatroskaCodingOrderTimestamps.json" "$@" diff --git a/Scripts/test-h264-timestamp-controls.sh b/Scripts/test-h264-timestamp-controls.sh new file mode 100644 index 00000000..0f56de81 --- /dev/null +++ b/Scripts/test-h264-timestamp-controls.sh @@ -0,0 +1,17 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-timestamp-controls.XXXXXX") +trap 'rm -f "$TASK_TMP/healthy.mp4" "$TASK_TMP/healthy.mkv" "$TASK_TMP/missing.mp4" "$TASK_TMP/coding-order.mkv"; rmdir "$TASK_TMP"' EXIT +# Generated solid-colour/AAC silence fixtures only; no private media is copied or committed. +ffmpeg -v error -f lavfi -i 'color=c=blue:s=96x64:r=30000/1001' \ + -f lavfi -i 'anullsrc=r=48000:cl=stereo' -t 8 \ + -c:v libx264 -preset ultrafast -pix_fmt yuv420p -bf 3 -b_strategy 0 -g 60 \ + -sc_threshold 0 -c:a aac -movflags +faststart "$TASK_TMP/healthy.mp4" +ffmpeg -v error -i "$TASK_TMP/healthy.mp4" -c copy "$TASK_TMP/healthy.mkv" +ffmpeg -v error -i "$TASK_TMP/healthy.mp4" -c copy -bsf:v 'setts=pts=DTS' -movflags +faststart "$TASK_TMP/missing.mp4" +ffmpeg -v error -i "$TASK_TMP/healthy.mp4" -c copy -bsf:v 'setts=pts=DTS' "$TASK_TMP/coding-order.mkv" +AETHER_EXPECT_TIMESTAMP_REPAIR=0 bash "$TASK_ROOT/Scripts/test-h264-timestamp-runtime.sh" "$TASK_TMP/healthy.mp4" 0 3 +AETHER_EXPECT_TIMESTAMP_REPAIR=0 bash "$TASK_ROOT/Scripts/test-h264-timestamp-runtime.sh" "$TASK_TMP/healthy.mkv" 0 3 +AETHER_EXPECT_TIMESTAMP_REPAIR=1 bash "$TASK_ROOT/Scripts/test-h264-timestamp-runtime.sh" "$TASK_TMP/missing.mp4" 0 3 +AETHER_EXPECT_TIMESTAMP_REPAIR=1 bash "$TASK_ROOT/Scripts/test-h264-timestamp-runtime.sh" "$TASK_TMP/coding-order.mkv" 0 3 diff --git a/Scripts/test-h264-timestamp-runtime.sh b/Scripts/test-h264-timestamp-runtime.sh new file mode 100644 index 00000000..e777139d --- /dev/null +++ b/Scripts/test-h264-timestamp-runtime.sh @@ -0,0 +1,30 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_FFMPEG_ROOT="${AETHER_FFMPEG_CHECKOUT:-$TASK_ROOT/.build/checkouts/FFmpegBuild}" +TASK_EXPECTED_REVISION=$(/usr/bin/ruby -rjson -e \ + 'puts JSON.parse(File.read(ARGV.fetch(0))).fetch("pins").find { |p| p.fetch("identity") == "ffmpegbuild" }.fetch("state").fetch("revision")' "$TASK_ROOT/Package.resolved") +[[ $(git -C "$TASK_FFMPEG_ROOT" rev-parse HEAD) == "$TASK_EXPECTED_REVISION" ]] \ + || { echo 'Frozen FFmpegBuild revision mismatch' >&2; exit 2; } +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-timestamp-runtime.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +TASK_FRAMEWORK_ARGS=() +for TASK_LIBRARY in AetherLibavformat AetherLibavcodec AetherLibavutil AetherLibswresample AetherLibdav1d AetherLibzvbi; do + TASK_DIR="$TASK_FFMPEG_ROOT/Sources/$TASK_LIBRARY.xcframework/macos-arm64_x86_64" + [[ -f "$TASK_DIR/$TASK_LIBRARY.framework/$TASK_LIBRARY" ]] || exit 2 + TASK_FRAMEWORK_ARGS+=(-F "$TASK_DIR" -Xlinker -rpath -Xlinker "$TASK_DIR") +done +# Compile the actual timestamp sessions and parser against the exact bundled dependency. +# This is a focused command-line library check, never an iOS/macOS app build or SwiftPM resolve. +xcrun swiftc -swift-version 6 \ + "${TASK_FRAMEWORK_ARGS[@]}" -framework AetherLibavformat -framework AetherLibavcodec -framework AetherLibavutil \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/PacketBalanceTracker.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/EngineLog.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/LogRedaction.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Decoder/A53SEIParser.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Decoder/CCDataParser.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264MatroskaTimestampRepair.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264MatroskaTimestampRepairSession.swift" \ + "$TASK_ROOT/Scripts/tests/H264TimestampRuntimeStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" "$@" diff --git a/Scripts/tests/Data/MatroskaCodingOrderTimestamps.json b/Scripts/tests/Data/MatroskaCodingOrderTimestamps.json new file mode 100644 index 00000000..47e7218d --- /dev/null +++ b/Scripts/tests/Data/MatroskaCodingOrderTimestamps.json @@ -0,0 +1,249 @@ +{ + "video_delay": 1, + "packets": [ + { + "pts": 0, + "poc": 0 + }, + { + "pts": 40, + "poc": 8 + }, + { + "pts": 73, + "poc": 2 + }, + { + "pts": 107, + "poc": 4 + }, + { + "pts": 140, + "poc": 6 + }, + { + "pts": 173, + "poc": 14 + }, + { + "pts": 207, + "poc": 10 + }, + { + "pts": 240, + "poc": 12 + }, + { + "pts": 274, + "poc": 20 + }, + { + "pts": 307, + "poc": 16 + }, + { + "pts": 340, + "poc": 18 + }, + { + "pts": 374, + "poc": 24 + }, + { + "pts": 407, + "poc": 22 + }, + { + "pts": 440, + "poc": 30 + }, + { + "pts": 474, + "poc": 26 + }, + { + "pts": 507, + "poc": 28 + }, + { + "pts": 541, + "poc": 36 + }, + { + "pts": 574, + "poc": 32 + }, + { + "pts": 607, + "poc": 34 + }, + { + "pts": 641, + "poc": 42 + }, + { + "pts": 674, + "poc": 38 + }, + { + "pts": 707, + "poc": 40 + }, + { + "pts": 741, + "poc": 48 + }, + { + "pts": 774, + "poc": 44 + }, + { + "pts": 807, + "poc": 46 + }, + { + "pts": 841, + "poc": 56 + }, + { + "pts": 874, + "poc": 50 + }, + { + "pts": 908, + "poc": 52 + }, + { + "pts": 941, + "poc": 54 + }, + { + "pts": 974, + "poc": 60 + }, + { + "pts": 1008, + "poc": 58 + }, + { + "pts": 1041, + "poc": 66 + }, + { + "pts": 1074, + "poc": 62 + }, + { + "pts": 1108, + "poc": 64 + }, + { + "pts": 1141, + "poc": 72 + }, + { + "pts": 1174, + "poc": 68 + }, + { + "pts": 1208, + "poc": 70 + }, + { + "pts": 1241, + "poc": 78 + }, + { + "pts": 1275, + "poc": 74 + }, + { + "pts": 1308, + "poc": 76 + }, + { + "pts": 1341, + "poc": 84 + }, + { + "pts": 1375, + "poc": 80 + }, + { + "pts": 1408, + "poc": 82 + }, + { + "pts": 1441, + "poc": 90 + }, + { + "pts": 1475, + "poc": 86 + }, + { + "pts": 1508, + "poc": 88 + }, + { + "pts": 1542, + "poc": 96 + }, + { + "pts": 1575, + "poc": 92 + }, + { + "pts": 1608, + "poc": 94 + }, + { + "pts": 1642, + "poc": 102 + }, + { + "pts": 1675, + "poc": 98 + }, + { + "pts": 1708, + "poc": 100 + }, + { + "pts": 1742, + "poc": 108 + }, + { + "pts": 1775, + "poc": 104 + }, + { + "pts": 1808, + "poc": 106 + }, + { + "pts": 1842, + "poc": 114 + }, + { + "pts": 1875, + "poc": 110 + }, + { + "pts": 1909, + "poc": 112 + }, + { + "pts": 1942, + "poc": 118 + }, + { + "pts": 1975, + "poc": 116 + }, + { + "pts": 2009, + "poc": 0 + } + ] +} diff --git a/Scripts/tests/H264MatroskaTimestampStandalone.swift b/Scripts/tests/H264MatroskaTimestampStandalone.swift new file mode 100644 index 00000000..da2e94cf --- /dev/null +++ b/Scripts/tests/H264MatroskaTimestampStandalone.swift @@ -0,0 +1,70 @@ +import Foundation + +@main +struct H264MatroskaTimestampTests { + typealias Policy = H264MatroskaTimestampRepair + + static func main() throws { + // Numeric-only shape from the reported source: compressed order I P B B B P B B. + // The first 40 ms interval is real input evidence, not a guessed constant FPS. + let times: [Int64] = [0, 40, 73, 107, 140, 173, 207, 240, 274, 307, 340, 374] + let pocs: [Int64] = [0, 8, 2, 4, 6, 14, 10, 12, 20, 16, 18, 22] + let source = zip(times, pocs).map { Policy.Picture(pts: $0, poc: $1) } + guard let repaired = Policy.repair(source, nextPTS: 407, videoDelay: 1) else { + fatalError("reported MKV's complete coding-order timestamp ladder must be repaired") + } + precondition(repaired.pts == pocs.map { times[Int($0 / 2)] }) + precondition(repaired.pts.sorted() == times, "never invent or drop a presentation time") + precondition(repaired.decodeLead == 34) + for (index, pts) in repaired.pts.enumerated() { + precondition(pts >= times[index] - repaired.decodeLead) + } + let seek = source.map { Policy.Picture(pts: $0.pts + 698712, poc: $0.poc) } + precondition(Policy.repair(seek, nextPTS: 699119, videoDelay: 1)?.pts + == repaired.pts.map { $0 + 698712 }, "seek cannot zero-base the picture axis") + let healthy = zip(repaired.pts, pocs).map { Policy.Picture(pts: $0, poc: $1) } + precondition(Policy.repair(healthy, nextPTS: 407, videoDelay: 1) == nil) + let noReorder = times.enumerated().map { Policy.Picture(pts: $0.element, poc: Int64($0.offset * 2)) } + precondition(Policy.repair(noReorder, nextPTS: 407, videoDelay: 1) == nil) + var invalid = source + invalid[4] = .init(pts: 140, poc: 4) + precondition(Policy.repair(invalid, nextPTS: 407, videoDelay: 1) == nil) + invalid = source + invalid[4] = .init(pts: 140, poc: 7) + precondition(Policy.repair(invalid, nextPTS: 407, videoDelay: 1) == nil) + precondition(Policy.repair(Array(source.prefix(8)), nextPTS: 274, videoDelay: 1) == nil) + precondition(Policy.repair(source, nextPTS: 407, videoDelay: 0) == nil) + precondition(Policy.repair(source, nextPTS: 300, videoDelay: 1) == nil) + let missing = source.map { Policy.Picture(pts: Int64.min, poc: $0.poc) } + precondition(Policy.repair(missing, nextPTS: 407, videoDelay: 1) == nil) + let overflow = source.map { Policy.Picture(pts: $0.pts, poc: Int64.max) } + precondition(Policy.repair(overflow, nextPTS: 407, videoDelay: 1) == nil) + let vfr = source.enumerated().map { Policy.Picture(pts: $0.element.pts + ($0.offset > 5 ? 500 : 0), poc: $0.element.poc) } + precondition(Policy.repair(vfr, nextPTS: 907, videoDelay: 1) == nil) + let tail: [Policy.Picture] = [.init(pts: 1000, poc: 0), .init(pts: 1033, poc: 4), .init(pts: 1067, poc: 2)] + precondition(Policy.repair(tail, nextPTS: 1100, videoDelay: 1, confirmedDecodeLead: 34)?.pts == [1000, 1067, 1033]) + precondition(Policy.repair(tail, nextPTS: 1100, videoDelay: 1, confirmedDecodeLead: 33) == nil) + print("PASS: MKV presentation-time permutation, first-frame hold, seek translation, mux invariant; healthy, no-B, truncated, duplicate/field POC, VFR, missing timestamps and overflow fail closed") + struct Fixture: Decodable { + struct Packet: Decodable { let pts: Int64; let poc: Int64 } + let video_delay: Int + let packets: [Packet] + } + for path in CommandLine.arguments.dropFirst() { + let fixture = try JSONDecoder().decode(Fixture.self, from: Data(contentsOf: URL(fileURLWithPath: path))) + let packets = fixture.packets + let origins = packets.indices.filter { packets[$0].poc == 0 } + var count = 0 + for (start, end) in zip(origins, origins.dropFirst()) { + let group = packets[start.. 0) + } + } +} diff --git a/Scripts/tests/H264TimestampRuntimeStandalone.swift b/Scripts/tests/H264TimestampRuntimeStandalone.swift new file mode 100644 index 00000000..b4d08b49 --- /dev/null +++ b/Scripts/tests/H264TimestampRuntimeStandalone.swift @@ -0,0 +1,204 @@ +import Foundation +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil + +func av_packet_free_safe(_ packet: UnsafeMutablePointer) { + var owned: UnsafeMutablePointer? = packet + trackedPacketFree(&owned) +} + +// Test-only snapshot: no dependency on the independent software packet-cache proposal. +struct TimestampPacketSnapshot: Codable, Sendable, Equatable { + struct SideData: Codable, Sendable, Equatable { + let type: UInt32 + let bytes: Data + } + let pts: Int64 + let dts: Int64 + let duration: Int64 + let position: Int64 + let streamIndex: Int32 + let flags: Int32 + let timeBaseNumerator: Int32 + let timeBaseDenominator: Int32 + let bytes: Data + let sideData: [SideData] + +} +extension TimestampPacketSnapshot { + enum PacketError: Error { case invalidPacket, allocationFailed } + + init(copying packet: UnsafeMutablePointer) throws { + let p = packet.pointee + guard p.size >= 0, p.side_data_elems >= 0, + p.size == 0 || p.data != nil, + p.side_data_elems == 0 || p.side_data != nil else { throw PacketError.invalidPacket } + var sides: [SideData] = [] + for index in 0..= 2 else { throw Failure.open } + av_log_set_level(AV_LOG_QUIET) + var format: UnsafeMutablePointer? + guard avformat_open_input(&format, CommandLine.arguments[1], nil, nil) >= 0, let format else { throw Failure.open } + defer { var owned: UnsafeMutablePointer? = format; avformat_close_input(&owned) } + guard avformat_find_stream_info(format, nil) >= 0 else { throw Failure.open } + let index = av_find_best_stream(format, AVMEDIA_TYPE_VIDEO, -1, -1, nil, 0) + guard index >= 0, let stream = format.pointee.streams[Int(index)], + let par = stream.pointee.codecpar, par.pointee.codec_id == AV_CODEC_ID_H264 else { throw Failure.open } + let name = format.pointee.iformat.flatMap { $0.pointee.name }.map(String.init(cString:)) ?? "" + let matroska = name.contains("matroska") + let expectedRepair = ProcessInfo.processInfo.environment["AETHER_EXPECT_TIMESTAMP_REPAIR"] + .map { $0 == "1" } ?? matroska + guard let session: any H264TimestampRepairSession = matroska + ? H264MatroskaTimestampRepairSession(stream: stream, streamIndex: index) + : H264CompositionOffsetRepairSession(containerFormatName: name, stream: stream, streamIndex: index, ladderStart: 0) + else { throw Failure.session } + let raw = try decoder(par, timeBase: stream.pointee.time_base) + let fixed = try decoder(par, timeBase: stream.pointee.time_base) + defer { + var a: UnsafeMutablePointer? = raw; avcodec_free_context(&a) + var b: UnsafeMutablePointer? = fixed; avcodec_free_context(&b) + } + var inputs: [UInt: TimestampPacketSnapshot] = [:] + var previousLead: Int64? + let positions = CommandLine.arguments.dropFirst(2).compactMap(Double.init) + for position in positions.isEmpty ? [0] : positions { + if position > 0 { + let target = Int64(position / av_q2d(stream.pointee.time_base)) + guard av_seek_frame(format, index, target, AVSEEK_FLAG_BACKWARD) >= 0 else { throw Failure.demux } + } + session.noteSeek() + avcodec_flush_buffers(raw); avcodec_flush_buffers(fixed) + var rawPTS: [Int64] = [], fixedPTS: [Int64] = [] + var videoRead = 0, reads = 0, delivered = 0, stop = false + func emit(_ packet: UnsafeMutablePointer) throws { + defer { var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) } + let expected = inputs.removeValue(forKey: UInt(bitPattern: packet))! + let actual = try TimestampPacketSnapshot(copying: packet) + // Exact packet payload/side-data/flags/duration/audio preservation, not just a + // model calculation. Only the selected video PTS/DTS may differ. + if packet.pointee.stream_index == index { + precondition(actual.bytes == expected.bytes && actual.sideData == expected.sideData) + precondition(actual.duration == expected.duration && actual.flags == expected.flags) + precondition(actual.position == expected.position && actual.streamIndex == expected.streamIndex) + if !expectedRepair { precondition(actual == expected) } + try decode(fixed, packet: packet, into: &fixedPTS) + } else { precondition(actual == expected) } + delivered += 1 + } + while !stop { + while let packet = session.dequeue() { try emit(packet) } + guard reads < 10000, videoRead < 2000 else { throw Failure.boundedRead } + guard let packet = trackedPacketAlloc() else { throw Failure.demux } + let status = av_read_frame(format, packet) + if status < 0 { + var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) + guard status == -541478725 else { throw Failure.demux } + stop = true + break + } + reads += 1 + inputs[UInt(bitPattern: packet)] = try TimestampPacketSnapshot(copying: packet) + if packet.pointee.stream_index == index { + let key = packet.pointee.flags & AV_PKT_FLAG_KEY != 0 + stop = videoRead >= 180 && key + videoRead += 1 + try decode(raw, packet: packet, into: &rawPTS) + } + if try !session.ingest(packet) { try emit(packet) } + } + try session.endOfStream() + while let packet = session.dequeue() { try emit(packet) } + try decode(raw, packet: nil, into: &rawPTS) + try decode(fixed, packet: nil, into: &fixedPTS) + precondition(inputs.isEmpty && PacketBalanceTracker.alive == 0) + let rawRegressions = zip(rawPTS, rawPTS.dropFirst()).filter { $1 <= $0 }.count + let fixedRegressions = zip(fixedPTS, fixedPTS.dropFirst()).filter { $1 <= $0 }.count + precondition(rawPTS.count == fixedPTS.count && !fixedPTS.isEmpty) + precondition(fixedRegressions == 0) + if expectedRepair { + precondition(rawRegressions > 0 && session.decodeTimestampOffset != nil) + if matroska { precondition(rawPTS.sorted() == fixedPTS, "every original presentation slot is preserved") } + if let previousLead { precondition(previousLead == session.decodeTimestampOffset) } + previousLead = session.decodeTimestampOffset + } else { precondition(rawRegressions == 0 && session.decodeTimestampOffset == nil) } + print("PASS source_kind=\(matroska ? "matroska" : "mp4") seek=\(position) decoded=\(fixedPTS.count) original_regressions=\(rawRegressions) repaired_regressions=\(fixedRegressions) packets=\(delivered) reason=\(session.summary) decode_offset=\(session.decodeTimestampOffset ?? 0) packet_balance=0") + } + if matroska && expectedRepair { try lifecycleChecks(format: format, stream: stream, index: index) } + } + + static func lifecycleChecks(format: UnsafeMutablePointer, stream: UnsafeMutablePointer, index: Int32) throws { + guard let session = H264MatroskaTimestampRepairSession(stream: stream, streamIndex: index) else { throw Failure.session } + // Seek with both unpublished input and a partly consumed ready queue. Those old packets + // must be freed rather than replayed into the new source position. + for goal in [3, 170] { + guard av_seek_frame(format, index, 0, AVSEEK_FLAG_BACKWARD) >= 0 else { throw Failure.demux } + session.noteSeek() + for _ in 0..= 0 else { throw Failure.demux } + if try !session.ingest(packet) { av_packet_free_safe(packet) } + } + if let first = session.dequeue() { av_packet_free_safe(first) } + session.noteSeek() + precondition(session.dequeue() == nil && PacketBalanceTracker.alive == 0) + } + // After activation, an unparseable sequence is an explicit failure; no changed-axis + // half-sequence or leaked packet is allowed to escape. + guard let malformed = trackedPacketAlloc() else { throw Failure.demux } + malformed.pointee.stream_index = index + do { _ = try session.ingest(malformed); throw Failure.session } + catch H264MatroskaTimestampRepairSession.RepairError.sequenceNoLongerRepairable { } + precondition(PacketBalanceTracker.alive == 0) + guard let bounded = H264MatroskaTimestampRepairSession(stream: stream, streamIndex: index) else { throw Failure.session } + for _ in 0..<1024 { + guard let packet = trackedPacketAlloc() else { throw Failure.demux } + packet.pointee.stream_index = index + 1 + let taken = try bounded.ingest(packet) + precondition(taken) + } + precondition(bounded.isDecided) + var released = 0 + while let packet = bounded.dequeue() { av_packet_free_safe(packet); released += 1 } + precondition(released == 1024 && PacketBalanceTracker.alive == 0) + print("PASS lifecycle=seek-during-sampling,seek-with-ready-and-pending,active-fail-closed,all-stream-packet-budget packet_balance=0") + } + + static func decoder(_ parameters: UnsafeMutablePointer, timeBase: AVRational) throws + -> UnsafeMutablePointer { + guard let codec = avcodec_find_decoder(AV_CODEC_ID_H264), let context = avcodec_alloc_context3(codec) else { throw Failure.decoder } + guard avcodec_parameters_to_context(context, parameters) >= 0 else { throw Failure.decoder } + context.pointee.pkt_timebase = timeBase + context.pointee.thread_count = 1 + guard avcodec_open2(context, codec, nil) >= 0 else { throw Failure.decoder } + return context + } + + static func decode(_ decoder: UnsafeMutablePointer, packet: UnsafeMutablePointer?, into pts: inout [Int64]) throws { + guard avcodec_send_packet(decoder, packet) >= 0, let frame = av_frame_alloc() else { throw Failure.decode } + defer { var owned: UnsafeMutablePointer? = frame; av_frame_free(&owned) } + while true { + let status = avcodec_receive_frame(decoder, frame) + if status == -35 || status == -541478725 { return } + guard status >= 0 else { throw Failure.decode } + pts.append(frame.pointee.pts) + av_frame_unref(frame) + } + } +} diff --git a/Sources/AetherEngine/Demuxer/Demuxer.swift b/Sources/AetherEngine/Demuxer/Demuxer.swift index b7601983..2f851b0d 100644 --- a/Sources/AetherEngine/Demuxer/Demuxer.swift +++ b/Sources/AetherEngine/Demuxer/Demuxer.swift @@ -201,7 +201,7 @@ public final class Demuxer: @unchecked Sendable { /// producer, the segment plan, the software decoder, the still extractor) reads the same axis; /// a repair applied per host would have them disagree by the reorder delay. nil for every stream /// that is not the exact defect shape, which is decided once, on the first read. - private var compositionRepair: H264CompositionOffsetRepairSession? + private var compositionRepair: (any H264TimestampRepairSession)? private var compositionRepairEvaluated = false /// #407: video streams whose PTS `+genpts` invented out of decode order, because the container @@ -1255,12 +1255,12 @@ public final class Demuxer: @unchecked Sendable { guard let packet = try readPacketLocked() else { // EOF can arrive mid-sample on a very short source; the verdict has to be reached // now or the held packets would never be delivered. - compositionRepair?.endOfStream() + try compositionRepair?.endOfStream() if let held = compositionRepair?.dequeue() { return held } return nil } guard let repair = armCompositionRepairIfNeeded() else { return packet } - if !repair.ingest(packet) { return packet } + if try !repair.ingest(packet) { return packet } } } @@ -1282,10 +1282,10 @@ public final class Demuxer: @unchecked Sendable { guard let repair = armCompositionRepairIfNeeded(), !repair.isDecided else { return } while !repair.isDecided { guard let packet = try? readPacketLocked() else { - repair.endOfStream() + try? repair.endOfStream() return } - if !repair.ingest(packet) { + if (try? repair.ingest(packet)) == false { // Not held: the session is done with the sample and this packet is already on the // final axis, so it goes to the front of the queue rather than out of order. repair.enqueueFront(packet) @@ -1297,7 +1297,7 @@ public final class Demuxer: @unchecked Sendable { /// #409: resolved once per demuxer, at the first read or at the explicit decision above, /// because it needs the stream parameters `avformat_find_stream_info` fills in and costs nothing /// for the streams it does not apply to. - private func armCompositionRepairIfNeeded() -> H264CompositionOffsetRepairSession? { + private func armCompositionRepairIfNeeded() -> (any H264TimestampRepairSession)? { if compositionRepairEvaluated { return compositionRepair } compositionRepairEvaluated = true guard let ctx = formatContext else { return nil } @@ -1312,6 +1312,10 @@ public final class Demuxer: @unchecked Sendable { guard index >= 0, index < Int32(ctx.pointee.nb_streams), let stream = ctx.pointee.streams[Int(index)] else { return nil } guard stream.pointee.discard != AVDISCARD_ALL else { return nil } + if containerFormatName?.split(separator: ",").contains("matroska") == true { + compositionRepair = H264MatroskaTimestampRepairSession(stream: stream, streamIndex: index) + return compositionRepair + } compositionRepair = H264CompositionOffsetRepairSession( containerFormatName: containerFormatName, stream: stream, diff --git a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift index fbd6c0ea..ceddaf61 100644 --- a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift +++ b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift @@ -509,6 +509,9 @@ final class H264PictureOrderReader { private var parser: UnsafeMutablePointer? private var context: UnsafeMutablePointer? + /// The Matroska policy accepts complete progressive frame pictures, not fields. + var isFramePicture: Bool { parser?.pointee.picture_structure == AV_PICTURE_STRUCTURE_FRAME } + init?(codecParameters: UnsafePointer, timeBase: AVRational) { guard let codec = avcodec_find_decoder(AV_CODEC_ID_H264), let context = avcodec_alloc_context3(codec) else { return nil } @@ -563,7 +566,7 @@ final class H264PictureOrderReader { /// costs one small queue and keeps the decision on the same bytes playback is about to consume. /// Every packet is held, not just video, so the interleaving the container chose survives the /// verdict. -final class H264CompositionOffsetRepairSession { +final class H264CompositionOffsetRepairSession: H264TimestampRepairSession { enum Phase: Equatable { case sampling, repairing, off } diff --git a/Sources/AetherEngine/Video/H264MatroskaTimestampRepair.swift b/Sources/AetherEngine/Video/H264MatroskaTimestampRepair.swift new file mode 100644 index 00000000..07c2b210 --- /dev/null +++ b/Sources/AetherEngine/Video/H264MatroskaTimestampRepair.swift @@ -0,0 +1,60 @@ +import Foundation + +/// Numeric-only policy for a complete, IDR-bounded H.264 frame-coded sequence. +/// Matroska stores packets in coding order but must timestamp them in display order. +enum H264MatroskaTimestampRepair { + struct Picture: Equatable { + let pts: Int64 + let poc: Int64 + } + struct Result: Equatable { + let pts: [Int64] + let decodeLead: Int64 + } + + static func repair( + _ pictures: [Picture], nextPTS: Int64, videoDelay: Int, + confirmedDecodeLead: Int64? = nil + ) -> Result? { + let minimum = confirmedDecodeLead == nil ? 9 : 2 + guard (1...16).contains(videoDelay), (minimum...512).contains(pictures.count), + pictures.first?.poc == 0, nextPTS != Int64.min else { return nil } + let times = pictures.map(\.pts) + guard times.allSatisfy({ $0 != Int64.min }) else { return nil } + var intervals: [Int64] = [] + for (a, b) in zip(times, times.dropFirst() + [nextPTS]) { + let (step, overflow) = b.subtractingReportingOverflow(a) + guard !overflow, step > 0 else { return nil } + intervals.append(step) + } + // Do not use a container's advertised average FPS to retime a VFR stream. Require a + // quantized near-CFR ladder, allowing only the distinct first-picture hold seen at an IDR. + guard let shortest = intervals.dropFirst().min(), + let longest = intervals.dropFirst().max(), shortest >= 2, + longest - shortest <= 1, let first = intervals.first, + first >= shortest / 2, first / 2 <= longest else { return nil } + var ranks = Set() + var reordered = false + for (index, picture) in pictures.enumerated() { + // Whole progressive frames have consecutive even POC. Fields, gaps, open-GOP + // leading pictures, parser misses and duplicate POC must not be guessed at. + guard picture.poc >= 0, picture.poc % 2 == 0, + picture.poc / 2 < Int64(pictures.count) else { return nil } + let rank = Int(picture.poc / 2) + guard ranks.insert(rank).inserted else { return nil } + reordered = reordered || rank != index + } + guard reordered || confirmedDecodeLead != nil else { return nil } + let (measuredLead, overflow) = longest.multipliedReportingOverflow(by: Int64(videoDelay)) + let lead = confirmedDecodeLead ?? measuredLead + guard !overflow, lead > 0 else { return nil } + var corrected: [Int64] = [] + for (index, picture) in pictures.enumerated() { + let pts = times[Int(picture.poc / 2)] + let (dts, underflow) = times[index].subtractingReportingOverflow(lead) + guard !underflow, pts >= dts else { return nil } + corrected.append(pts) + } + return Result(pts: corrected, decodeLead: lead) + } +} diff --git a/Sources/AetherEngine/Video/H264MatroskaTimestampRepairSession.swift b/Sources/AetherEngine/Video/H264MatroskaTimestampRepairSession.swift new file mode 100644 index 00000000..b70b278b --- /dev/null +++ b/Sources/AetherEngine/Video/H264MatroskaTimestampRepairSession.swift @@ -0,0 +1,216 @@ +import Foundation +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil + +/// Both policies share packet ownership/seek semantics, not their detection rules. +protocol H264TimestampRepairSession: AnyObject { + var isDecided: Bool { get } + var decodeTimestampOffset: Int64? { get } + func ingest(_ packet: UnsafeMutablePointer) throws -> Bool + func dequeue() -> UnsafeMutablePointer? + func enqueueFront(_ packet: UnsafeMutablePointer) + func endOfStream() throws + func noteSeek() + var summary: String { get } +} + +/// Some Matroska writers label coding order as presentation order. Unlike missing MP4 ctts, +/// FFmpeg may synthesize different DTS here, so PTS!=DTS does NOT establish healthy timing. +/// Hold one bounded IDR-to-IDR sequence and permute its existing timestamp slots by parsed POC. +/// This preserves the original presentation-time set, audio axis and rational rounding exactly; +/// the decoder continues receiving the original compressed packets in their original order. +final class H264MatroskaTimestampRepairSession: H264TimestampRepairSession { + private struct Entry { + let packet: UnsafeMutablePointer + let poc: Int64? + } + enum RepairError: Error { case sequenceNoLongerRepairable } + private let streamIndex: Int32 + private let videoDelay: Int + private let timeBase: AVRational + private let framing: VideoNALFraming + private let reader: H264PictureOrderReader + private var pending: [Entry] = [] + private var ready: [UnsafeMutablePointer] = [] + private var readyIndex = 0 + private var bytes = 0 + private var videoCount = 0 + private var lastPTS: Int64? + private var lead: Int64? + private var off = false + private var failed = false + private var verdict = "sampling" + private var sampledCount = 0 + private var sampledPackets = 0 + private var sampledBytes = 0 + private var sampledRegressions = 0 + private var repairedCount = 0 + private var unrepairedCount = 0 + + init?(stream: UnsafeMutablePointer, streamIndex: Int32) { + guard let par = stream.pointee.codecpar, + par.pointee.codec_id == AV_CODEC_ID_H264, + (1...16).contains(par.pointee.video_delay), + let reader = H264PictureOrderReader(codecParameters: par, timeBase: stream.pointee.time_base) + else { return nil } + self.reader = reader + self.streamIndex = streamIndex + videoDelay = Int(par.pointee.video_delay) + timeBase = stream.pointee.time_base + framing = A53SEIParser.nalFraming(codec: .h264, extradata: par.pointee.extradata, size: Int(par.pointee.extradata_size)) + } + + deinit { releasePackets() } + var isDecided: Bool { off || lead != nil } + var decodeTimestampOffset: Int64? { lead.map { -$0 } } + + func ingest(_ packet: UnsafeMutablePointer) throws -> Bool { + if failed { + var owned: UnsafeMutablePointer? = packet + trackedPacketFree(&owned) + throw RepairError.sequenceNoLongerRepairable + } + guard !off else { return false } + var poc: Int64? + if packet.pointee.stream_index == streamIndex { + poc = reader.pictureOrderCount(for: packet) + let key = packet.pointee.flags & AV_PKT_FLAG_KEY != 0 + var idr = false + if key, let data = packet.pointee.data, packet.pointee.size > 0 { + A53SEIParser.forEachNAL(data, Int(packet.pointee.size), framing) { nal, size in + if size > 0, nal[0] & 0x80 == 0, nal[0] & 31 == 5 { idr = true } + } + } + if idr, poc == 0, videoCount > 0 { + // Own the boundary packet even if validation fails; callers never free a taken + // packet. On success it becomes the first packet of the next sequence. + do { try finishSequence(nextPTS: packet.pointee.pts) } + catch { var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned); throw error } + if off { ready.append(packet); return true } + } + let invalid = poc == nil || !reader.isFramePicture || packet.pointee.pts == Int64.min + || (videoCount == 0 && (!idr || poc != 0)) + || (key && (!idr || poc != 0)) + let hasPresentationOffsets = lastPTS.map { packet.pointee.pts < $0 } == true + let duplicateTime = lastPTS == packet.pointee.pts + pending.append(Entry(packet: packet, poc: poc)) + bytes += Int(max(0, packet.pointee.size)) + videoCount += 1 + lastPTS = packet.pointee.pts + if invalid || duplicateTime { try refuse(healthy: false); return true } + if hasPresentationOffsets { try refuse(healthy: true); return true } + } else { + pending.append(Entry(packet: packet, poc: nil)) + bytes += Int(max(0, packet.pointee.size)) + } + // Bound all streams, not only video. Healthy reordered MKV exits at its first PTS + // regression; only a positive coding-order ladder pays the complete sequence hold. + if bytes >= 32 << 20 || pending.count >= 1024 || videoCount > 512 { + try refuse(healthy: false) + } + return true + } + + private func finishSequence(nextPTS: Int64) throws { + let video = pending.filter { $0.packet.pointee.stream_index == streamIndex } + let pictures = video.map { + H264MatroskaTimestampRepair.Picture(pts: $0.packet.pointee.pts, poc: $0.poc ?? -1) + } + if lead == nil { + sampledCount = video.count; sampledPackets = pending.count; sampledBytes = bytes + sampledRegressions = zip(pictures, pictures.dropFirst()).filter { $1.poc < $0.poc }.count + } + // A final one-picture IDR after a confirmed repair needs no permutation. + if pictures.count == 1, pictures[0].poc == 0, let lead { + let (dts, overflow) = pictures[0].pts.subtractingReportingOverflow(lead) + guard !overflow else { try refuse(healthy: false); return } + video[0].packet.pointee.dts = dts + repairedCount += 1 + publishPending() + return + } + guard let result = H264MatroskaTimestampRepair.repair( + pictures, nextPTS: nextPTS, videoDelay: videoDelay, confirmedDecodeLead: lead + ) else { try refuse(healthy: false); return } + if lead == nil { + lead = result.decodeLead + verdict = "confirmedMatroskaCodingOrder" + EngineLog.emit("[Demuxer] Matroska H264 coding-order timestamps confirmed: pictures=\(sampledCount) poc_regressions=\(sampledRegressions) decode_lead=\(result.decodeLead) time_base=\(timeBase.num)/\(timeBase.den)", category: .demux) + } + for (index, entry) in video.enumerated() { + // The policy checked subtraction and PTS>=DTS for the entire sequence BEFORE any + // mutation. Packet bytes, duration, side data, flags, and audio/subtitles are untouched. + entry.packet.pointee.dts = pictures[index].pts - result.decodeLead + entry.packet.pointee.pts = result.pts[index] + repairedCount += 1 + } + publishPending() + } + + private func refuse(healthy: Bool) throws { + if lead != nil { + // Once the index is published on a repaired axis, reverting mid-stream would silently + // corrupt it. Stop with an explicit error, never emit a partly rewritten sequence. + unrepairedCount += videoCount + failed = true + verdict = "matroskaSequenceChanged" + releasePackets() + EngineLog.emit("[Demuxer] Matroska H264 timestamp repair stopped: sequence no longer satisfies confirmed policy", category: .demux) + throw RepairError.sequenceNoLongerRepairable + } + verdict = healthy ? "healthy" : "matroskaSequenceUnproven" + off = true + publishPending() + } + + private func publishPending() { + ready.append(contentsOf: pending.map(\.packet)) + pending.removeAll(keepingCapacity: true) + bytes = 0; videoCount = 0; lastPTS = nil + } + + func dequeue() -> UnsafeMutablePointer? { + guard readyIndex < ready.count else { return nil } + let packet = ready[readyIndex] + readyIndex += 1 + if readyIndex == ready.count { ready.removeAll(keepingCapacity: true); readyIndex = 0 } + return packet + } + + func enqueueFront(_ packet: UnsafeMutablePointer) { ready.insert(packet, at: readyIndex) } + + func endOfStream() throws { + if failed { throw RepairError.sequenceNoLongerRepairable } + guard !pending.isEmpty else { return } + let video = pending.filter { $0.packet.pointee.stream_index == streamIndex } + guard let last = video.last else { publishPending(); return } + // The last packet's decode duration is used only as a bounded validation sentinel. It + // never becomes a new presentation time and never changes a packet duration. + let duration = max(1, last.packet.pointee.duration) + let (end, overflow) = last.packet.pointee.pts.addingReportingOverflow(duration) + guard !overflow else { try refuse(healthy: false); return } + try finishSequence(nextPTS: end) + } + + func noteSeek() { + releasePackets() + reader.reset() + failed = false + if lead != nil { verdict = "confirmedMatroskaCodingOrder" } + // Keep the confirmed decode lead (and the container index axis) across every seek. + } + + private func releasePackets() { + for entry in pending { var owned: UnsafeMutablePointer? = entry.packet; trackedPacketFree(&owned) } + for packet in ready.dropFirst(readyIndex) { var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) } + pending.removeAll(keepingCapacity: true); ready.removeAll(keepingCapacity: true) + readyIndex = 0; bytes = 0; videoCount = 0; lastPTS = nil + } + + var summary: String { + "verdict=\(verdict) samples=\(sampledCount) held_packets=\(sampledPackets)" + + " held_bytes=\(sampledBytes) poc_regressions=\(sampledRegressions)" + + " decode_lead=\(lead ?? 0) repaired=\(repairedCount) unrepaired=\(unrepairedCount)" + } +} diff --git a/Tests/AetherEngineTests/H264MatroskaTimestampRepairTests.swift b/Tests/AetherEngineTests/H264MatroskaTimestampRepairTests.swift new file mode 100644 index 00000000..e0d6327f --- /dev/null +++ b/Tests/AetherEngineTests/H264MatroskaTimestampRepairTests.swift @@ -0,0 +1,39 @@ +import Testing +@testable import AetherEngine + +@Suite("Matroska H264 coding-order timestamps") +struct H264MatroskaTimestampRepairTests { + typealias Policy = H264MatroskaTimestampRepair + let times: [Int64] = [0, 40, 73, 107, 140, 173, 207, 240, 274, 307, 340, 374] + let pocs: [Int64] = [0, 8, 2, 4, 6, 14, 10, 12, 20, 16, 18, 22] + var pictures: [Policy.Picture] { zip(times, pocs).map { .init(pts: $0, poc: $1) } } + + @Test func repairsOnlyTimeOwnership() throws { + let result = try #require(Policy.repair(pictures, nextPTS: 407, videoDelay: 1)) + #expect(result.pts == pocs.map { times[Int($0 / 2)] }) + #expect(result.pts.sorted() == times) + #expect(result.decodeLead == 34) + for (index, pts) in result.pts.enumerated() { #expect(pts >= times[index] - result.decodeLead) } + let seek = pictures.map { Policy.Picture(pts: $0.pts + 698712, poc: $0.poc) } + #expect(Policy.repair(seek, nextPTS: 699119, videoDelay: 1)?.pts == result.pts.map { $0 + 698712 }) + } + + @Test func healthyAndUnprovenStayUntouched() { + let healthy = pocs.map { Policy.Picture(pts: times[Int($0 / 2)], poc: $0) } + #expect(Policy.repair(healthy, nextPTS: 407, videoDelay: 1) == nil) + #expect(Policy.repair(Array(pictures.prefix(8)), nextPTS: 274, videoDelay: 1) == nil) + var collision = pictures + collision[4] = .init(pts: 140, poc: 4) + #expect(Policy.repair(collision, nextPTS: 407, videoDelay: 1) == nil) + let variable = pictures.enumerated().map { Policy.Picture(pts: $0.element.pts + ($0.offset > 5 ? 500 : 0), poc: $0.element.poc) } + #expect(Policy.repair(variable, nextPTS: 907, videoDelay: 1) == nil) + } + + @Test func confirmedAxisHandlesShortTailWithoutChangingDecodeLead() throws { + let tail: [Policy.Picture] = [.init(pts: 1000, poc: 0), .init(pts: 1033, poc: 4), .init(pts: 1067, poc: 2)] + let result = try #require(Policy.repair(tail, nextPTS: 1100, videoDelay: 1, confirmedDecodeLead: 34)) + #expect(result.pts == [1000, 1067, 1033]) + #expect(result.decodeLead == 34) + #expect(Policy.repair(tail, nextPTS: 1100, videoDelay: 1, confirmedDecodeLead: 33) == nil) + } +} diff --git a/docs/formats.md b/docs/formats.md index 9847e0bf..a99761d9 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -23,6 +23,40 @@ That list is the supported set, not the compiled set. The FFmpeg build also carr Interlaced sources (DVD-rip MPEG-2, SD / HD broadcast H.264) are deinterlaced through a persistent bwdif graph (yadif fallback) that engages on the first interlaced frame and costs nothing on progressive content. The dispatch decision lives in `AetherEngine.load` (`VideoRoutingPolicy`), gated per source on `VTCapabilityProbe`, codec id, declared field order, and on VOD the decode sample that verifies it. +### Matroska H.264 with coding-order timestamps + +Some Matroska H.264 streams contain reordered pictures, but their packet PTS rise +in coding order. The timestamp then belongs to the wrong picture: a decoder emits +the right pictures with a regressing presentation clock. Unlike missing MP4 +`ctts`, FFmpeg can synthesize DTS different from PTS here; equality of those +fields is not an eligibility test for this container. Matroska block timestamps +are presentation timestamps, as specified in the +[Matroska technical notes](https://www.matroska.org/technical/notes.html). + +The demuxer holds a bounded, complete IDR-to-IDR sequence for a seekable H.264 VOD +source. Repair requires progressive frame pictures, a strictly rising near-CFR +source ladder, reordered POC, and a complete unique POC-to-slot mapping. It +permutes the existing PTS slots, without manufacturing a rounded replacement +clock. The original packet order, payload, duration, side data, flags and +audio/subtitle timing remain intact. A constant decode lead puts DTS and the +container index on the same axis; that confirmed lead survives seeks. + +Healthy or unproven input is released unchanged. Live/non-seekable sources, +still extraction and discarded video are not sampled. A changed/unsupported +sequence after activation fails explicitly instead of silently reverting the +published timestamp axis. Seek and teardown release every held packet. +This is a demux fix: native hardware decoding stays selected, with no host UI +compensation or source-file rewrite. + +The regression suite includes a numeric timestamp/POC fixture without source +identity, invalid and healthy policy controls, and real parser/session/decoder +checks against the pinned FFmpegBuild. Run +`bash Scripts/test-h264-matroska-timestamps.sh` for the pure policy, and +`bash Scripts/test-h264-timestamp-controls.sh` for generated solid-colour/AAC +fixtures (requires an `ffmpeg` CLI with libx264 and the resolved FFmpegBuild +checkout). Set `AETHER_FFMPEG_CHECKOUT` if that exact pinned checkout is not in +`.build/checkouts/FFmpegBuild`. No private sample is required or distributed. + ### MP4 without composition offsets Some writers emit a sample table with no `ctts` while the H.264 bitstream still reorders pictures.