diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c1299d55..ad1d39819 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,6 +40,11 @@ jobs: - name: swift test run: swift test + - name: Partial H.264 composition-offset runtime regression + run: | + command -v ffmpeg >/dev/null || brew install ffmpeg + bash Scripts/test-h264-partial-composition-controls.sh + # The transport probe is a device harness and runs nowhere in CI, so nothing else would keep # it compiling. A harness that stopped building is discovered by the person who needed it. - name: build the transport probe (AE#377) diff --git a/CHANGELOG.md b/CHANGELOG.md index 767414dc9..d0ec5e054 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,15 @@ the public-API contract. ## [Unreleased] -_Nothing yet._ +### Fixed + +- H.264 MP4 with valid composition offsets at the head but missing offsets in + later closed IDR sequences no longer escapes timestamp repair after a seek. + The partial-region policy restores display ownership of existing timestamp + slots, preserving original DTS, audio, keyframe-index time and hardware + routing even when decode intervals change inside a sequence. Healthy regions + remain unchanged. Includes generated partial-ctts fixtures and packet-level + seek, boundary, ownership and decoder regressions. ## [6.71.0] - 2026-09-06 diff --git a/Scripts/test-h264-partial-composition-controls.sh b/Scripts/test-h264-partial-composition-controls.sh new file mode 100644 index 000000000..93380ab3f --- /dev/null +++ b/Scripts/test-h264-partial-composition-controls.sh @@ -0,0 +1,20 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-partial-controls.XXXXXX") +trap 'rm -f "$TASK_TMP/healthy.mp4" "$TASK_TMP/partial.mp4" "$TASK_TMP/middle.mp4"; rmdir "$TASK_TMP"' EXIT +bash "$TASK_ROOT/Scripts/test-h264-partial-composition.sh" +# Reproducible public solid-colour/tone fixture. Keep an 8-second valid head, then remove +# only composition offsets; never duplicate any real source media into a test artifact. +ffmpeg -hide_banner -loglevel error -n \ + -f lavfi -i 'color=c=blue:s=96x64:r=30:d=24' \ + -f lavfi -i 'sine=frequency=440:sample_rate=48000:duration=24' \ + -c:v libx264 -preset fast -pix_fmt yuv420p -g 30 -bf 2 \ + -x264-params 'scenecut=0:b-adapt=0' -c:a aac -video_track_timescale 90000 \ + -metadata comment='Aether synthetic timestamp fixture' -movflags +faststart "$TASK_TMP/healthy.mp4" +ruby "$TASK_ROOT/Scripts/tests/make-partial-ctts-fixture.rb" "$TASK_TMP/healthy.mp4" "$TASK_TMP/partial.mp4" +bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" \ + "$TASK_TMP/partial.mp4" 0 10 0 6 22 +ruby "$TASK_ROOT/Scripts/tests/make-partial-ctts-fixture.rb" "$TASK_TMP/healthy.mp4" "$TASK_TMP/middle.mp4" middle +bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" \ + "$TASK_TMP/middle.mp4" 0 10 0 6 14 22 diff --git a/Scripts/test-h264-partial-composition-runtime.sh b/Scripts/test-h264-partial-composition-runtime.sh new file mode 100644 index 000000000..493c3ece2 --- /dev/null +++ b/Scripts/test-h264-partial-composition-runtime.sh @@ -0,0 +1,31 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_FFMPEG_ROOT="${AETHER_FFMPEG_CHECKOUT:-$TASK_ROOT/.build/checkouts/FFmpegBuild}" +TASK_DRIVER="$TASK_ROOT/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift" +TASK_EXPECTED_REVISION=$(/usr/bin/ruby -rjson -e \ + 'puts JSON.parse(File.read(ARGV.fetch(0))).fetch("pins").find { |p| p.fetch("identity") == "ffmpegbuild" }.fetch("state").fetch("revision")' "$TASK_ROOT/Package.resolved") +[[ $(git -C "$TASK_FFMPEG_ROOT" rev-parse HEAD) == "$TASK_EXPECTED_REVISION" ]] \ + || { echo 'Frozen FFmpegBuild revision mismatch' >&2; exit 2; } +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-timestamp-runtime.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +TASK_FRAMEWORK_ARGS=() +for TASK_LIBRARY in AetherLibavformat AetherLibavcodec AetherLibavutil AetherLibswresample AetherLibdav1d AetherLibzvbi; do + TASK_DIR="$TASK_FFMPEG_ROOT/Sources/$TASK_LIBRARY.xcframework/macos-arm64_x86_64" + [[ -f "$TASK_DIR/$TASK_LIBRARY.framework/$TASK_LIBRARY" ]] || exit 2 + TASK_FRAMEWORK_ARGS+=(-F "$TASK_DIR" -Xlinker -rpath -Xlinker "$TASK_DIR") +done +# Compile the actual timestamp sessions and parser against the exact bundled dependency. +# This is a focused command-line library check, never an iOS/macOS app build or SwiftPM resolve. +xcrun swiftc -swift-version 6 \ + "${TASK_FRAMEWORK_ARGS[@]}" -framework AetherLibavformat -framework AetherLibavcodec -framework AetherLibavutil \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/PacketBalanceTracker.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/EngineLog.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/LogRedaction.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Decoder/A53SEIParser.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Decoder/CCDataParser.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift" \ + "$TASK_DRIVER" -o "$TASK_TMP/check" +"$TASK_TMP/check" "$@" diff --git a/Scripts/test-h264-partial-composition.sh b/Scripts/test-h264-partial-composition.sh new file mode 100644 index 000000000..b38e54008 --- /dev/null +++ b/Scripts/test-h264-partial-composition.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-partial-ctts.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +xcrun swiftc -swift-version 6 \ + "$TASK_ROOT/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift" \ + "$TASK_ROOT/Scripts/tests/H264PartialCompositionStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" diff --git a/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift b/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift new file mode 100644 index 000000000..2ee9ce892 --- /dev/null +++ b/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift @@ -0,0 +1,220 @@ +import Foundation +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil + +func av_packet_free_safe(_ packet: UnsafeMutablePointer) { + var owned: UnsafeMutablePointer? = packet + trackedPacketFree(&owned) +} + +// Test-only snapshot: no dependency on the independent software packet-cache proposal. +struct TimestampPacketSnapshot: Codable, Sendable, Equatable { + struct SideData: Codable, Sendable, Equatable { + let type: UInt32 + let bytes: Data + } + let pts: Int64 + let dts: Int64 + let duration: Int64 + let position: Int64 + let streamIndex: Int32 + let flags: Int32 + let timeBaseNumerator: Int32 + let timeBaseDenominator: Int32 + let bytes: Data + let sideData: [SideData] + +} +extension TimestampPacketSnapshot { + enum PacketError: Error { case invalidPacket, allocationFailed } + + init(copying packet: UnsafeMutablePointer) throws { + let p = packet.pointee + guard p.size >= 0, p.side_data_elems >= 0, + p.size == 0 || p.data != nil, + p.side_data_elems == 0 || p.side_data != nil else { throw PacketError.invalidPacket } + var sides: [SideData] = [] + for index in 0..= 2 else { throw Failure.open } + av_log_set_level(AV_LOG_QUIET) + var format: UnsafeMutablePointer? + guard avformat_open_input(&format, CommandLine.arguments[1], nil, nil) >= 0, let format else { throw Failure.open } + defer { var owned: UnsafeMutablePointer? = format; avformat_close_input(&owned) } + guard avformat_find_stream_info(format, nil) >= 0 else { throw Failure.open } + let index = av_find_best_stream(format, AVMEDIA_TYPE_VIDEO, -1, -1, nil, 0) + guard index >= 0, let stream = format.pointee.streams[Int(index)], + let par = stream.pointee.codecpar, par.pointee.codec_id == AV_CODEC_ID_H264 else { throw Failure.open } + let name = format.pointee.iformat.flatMap { $0.pointee.name }.map(String.init(cString:)) ?? "" + let matroska = name.contains("matroska") + guard !matroska else { throw Failure.session } + let ladderStart = avformat_index_get_entry(stream, 0)?.pointee.timestamp ?? Int64.min + guard let session = H264CompositionOffsetRepairSession(containerFormatName: name, stream: stream, streamIndex: index, ladderStart: ladderStart) + else { throw Failure.session } + let raw = try decoder(par, timeBase: stream.pointee.time_base) + let fixed = try decoder(par, timeBase: stream.pointee.time_base) + defer { + var a: UnsafeMutablePointer? = raw; avcodec_free_context(&a) + var b: UnsafeMutablePointer? = fixed; avcodec_free_context(&b) + } + var inputs: [UInt: TimestampPacketSnapshot] = [:] + let positions = CommandLine.arguments.dropFirst(2).compactMap(Double.init) + for position in positions.isEmpty ? [0] : positions { + if position >= 0 { + let target = Int64(position / av_q2d(stream.pointee.time_base)) + guard av_seek_frame(format, index, target, AVSEEK_FLAG_BACKWARD) >= 0 else { throw Failure.demux } + } + session.noteSeek() + avcodec_flush_buffers(raw); avcodec_flush_buffers(fixed) + var rawPTS: [Int64] = [], fixedPTS: [Int64] = [] + var videoRead = 0, reads = 0, delivered = 0, stop = false, observedRepair = false + func emit(_ packet: UnsafeMutablePointer) throws { + defer { var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) } + let expected = inputs.removeValue(forKey: UInt(bitPattern: packet))! + let actual = try TimestampPacketSnapshot(copying: packet) + // Exact packet payload/side-data/flags/duration/audio preservation, not just a + // model calculation. Only the selected video PTS/DTS may differ. + if packet.pointee.stream_index == index { + precondition(actual.bytes == expected.bytes && actual.sideData == expected.sideData) + precondition(actual.duration == expected.duration && actual.flags == expected.flags) + precondition(actual.position == expected.position && actual.streamIndex == expected.streamIndex) + precondition(actual.dts == expected.dts, "partial repair must preserve the published decode/index axis") + try decode(fixed, packet: packet, into: &fixedPTS) + } else { precondition(actual == expected) } + delivered += 1 + } + while !stop { + while let packet = session.dequeue() { try emit(packet) } + guard reads < 10000, videoRead < 2000 else { throw Failure.boundedRead } + guard let packet = trackedPacketAlloc() else { throw Failure.demux } + let status = av_read_frame(format, packet) + if status < 0 { + var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) + guard status == -541478725 else { throw Failure.demux } + stop = true + break + } + reads += 1 + inputs[UInt(bitPattern: packet)] = try TimestampPacketSnapshot(copying: packet) + if packet.pointee.stream_index == index { + let key = packet.pointee.flags & AV_PKT_FLAG_KEY != 0 + stop = videoRead >= 180 && key + videoRead += 1 + try decode(raw, packet: packet, into: &rawPTS) + } + if try !session.ingest(packet) { try emit(packet) } + observedRepair = observedRepair || session.summary.contains("confirmed_partial_composition_offsets") + } + try session.endOfStream() + while let packet = session.dequeue() { try emit(packet) } + try decode(raw, packet: nil, into: &rawPTS) + try decode(fixed, packet: nil, into: &fixedPTS) + precondition(inputs.isEmpty && PacketBalanceTracker.alive == 0) + let rawRegressions = zip(rawPTS, rawPTS.dropFirst()).filter { $1 <= $0 }.count + let fixedRegressions = zip(fixedPTS, fixedPTS.dropFirst()).filter { $1 <= $0 }.count + let diagnostic = session.summary + observedRepair = observedRepair || diagnostic.contains("confirmed_partial_composition_offsets") + FileHandle.standardError.write(Data("MEASURE seek=\(position) raw=\(rawRegressions) fixed=\(fixedRegressions) offset=\(session.decodeTimestampOffset ?? 0) state=\(diagnostic) lead=\(0) shift=\(0)\n".utf8)) + precondition(rawPTS.count == fixedPTS.count && !fixedPTS.isEmpty) + precondition(fixedRegressions == 0) + if rawRegressions > 0 { + precondition(observedRepair) + precondition(session.decodeTimestampOffset == 0) + } else { precondition(rawPTS == fixedPTS, "healthy head remains exactly unchanged") } + print("PASS source_kind=\(matroska ? "matroska" : "mp4") seek=\(position) decoded=\(fixedPTS.count) original_regressions=\(rawRegressions) repaired_regressions=\(fixedRegressions) packets=\(delivered) reason=\(diagnostic) decode_offset=\(session.decodeTimestampOffset ?? 0) packet_balance=0") + } + if positions.count > 1 { + try lifecycleChecks(session: session, format: format, stream: stream, index: index, position: positions[1]) + } + } + + static func lifecycleChecks(session: H264CompositionOffsetRepairSession, + format: UnsafeMutablePointer, stream: UnsafeMutablePointer, index: Int32, + position: Double) throws { + func seek() throws { + guard av_seek_frame(format, index, Int64(position / av_q2d(stream.pointee.time_base)), AVSEEK_FLAG_BACKWARD) >= 0 else { throw Failure.demux } + session.noteSeek() + } + func read() throws { + guard let packet = trackedPacketAlloc() else { throw Failure.demux } + guard av_read_frame(format, packet) >= 0 else { av_packet_free_safe(packet); throw Failure.demux } + if try !session.ingest(packet) { av_packet_free_safe(packet) } + } + for goal in [3, 500] { + try seek() + for _ in 0.., timeBase: AVRational) throws + -> UnsafeMutablePointer { + guard let codec = avcodec_find_decoder(AV_CODEC_ID_H264), let context = avcodec_alloc_context3(codec) else { throw Failure.decoder } + guard avcodec_parameters_to_context(context, parameters) >= 0 else { throw Failure.decoder } + context.pointee.pkt_timebase = timeBase + context.pointee.thread_count = 1 + guard avcodec_open2(context, codec, nil) >= 0 else { throw Failure.decoder } + return context + } + + static func decode(_ decoder: UnsafeMutablePointer, packet: UnsafeMutablePointer?, into pts: inout [Int64]) throws { + guard avcodec_send_packet(decoder, packet) >= 0, let frame = av_frame_alloc() else { throw Failure.decode } + defer { var owned: UnsafeMutablePointer? = frame; av_frame_free(&owned) } + while true { + let status = avcodec_receive_frame(decoder, frame) + if status == -35 || status == -541478725 { return } + guard status >= 0 else { throw Failure.decode } + pts.append(frame.pointee.pts) + av_frame_unref(frame) + } + } +} diff --git a/Scripts/tests/H264PartialCompositionStandalone.swift b/Scripts/tests/H264PartialCompositionStandalone.swift new file mode 100644 index 000000000..426ff0d84 --- /dev/null +++ b/Scripts/tests/H264PartialCompositionStandalone.swift @@ -0,0 +1,44 @@ +import Foundation + +@main +struct H264PartialCompositionTests { + typealias Policy = H264PartialCompositionRepair + static func main() { + // Synthetic closed sequence, including a 25 -> 29.97 transition and rounding ticks. + let times: [Int64] = [0, 3600, 7200, 10800, 14400, 18000, 21600, 25200, 28800, 32400, 36000, 39600, 42603, 45607, 48610] + let pocs: [Int64] = [0, 8, 2, 4, 6, 16, 10, 12, 14, 24, 18, 20, 22, 28, 26] + let source = zip(times, pocs).map { Policy.Picture(dts: $0, poc: $1) } + func repair(_ source: [Policy.Picture], next: Int64 = 51613, lead: Int64 = 6006, confirmed: Bool = false) -> [Int64]? { + Policy.presentationTimes(source, nextDTS: next, presentationLead: lead, previouslyConfirmed: confirmed) + } + let expected = pocs.map { times[Int($0 / 2)] + 6006 } + precondition(repair(source) == expected) + precondition(expected.sorted() == times.map { $0 + 6006 }) + let shifted = source.map { Policy.Picture(dts: $0.dts + 63_000_000, poc: $0.poc) } + precondition(repair(shifted, next: 63_051_613) == expected.map { $0 + 63_000_000 }) + precondition(repair(source, lead: 3000) == nil, "never produce PTS earlier than original DTS") + precondition(repair(source, lead: 0) == nil) + precondition(repair(source, next: times.last!) == nil) + precondition(repair(source, next: Int64.min) == nil) + precondition(repair(Array(source.prefix(14))) == nil, "truncated POC window") + for badPOC: Int64 in [-2, 1, 2, 30, Int64.max] { + var invalid = source; invalid[4] = .init(dts: times[4], poc: badPOC) + precondition(repair(invalid) == nil) + } + var invalid = source; invalid[4] = .init(dts: times[3], poc: pocs[4]) + precondition(repair(invalid) == nil) + invalid[4] = .init(dts: Int64.min, poc: pocs[4]) + precondition(repair(invalid) == nil) + let overflow = source.map { Policy.Picture(dts: Int64.max - 51614 + $0.dts, poc: $0.poc) } + precondition(repair(overflow, next: Int64.max - 1) == nil) + let ordered = times.enumerated().map { Policy.Picture(dts: $0.element, poc: Int64($0.offset * 2)) } + precondition(repair(ordered) == nil, "all-zero offsets without reordering are not proof") + precondition(repair(ordered, confirmed: true) == times.map { $0 + 6006 }) + precondition(repair([.init(dts: 0, poc: 0)]) == nil) + precondition(repair([.init(dts: 0, poc: 0)], confirmed: true) == [6006]) + precondition(repair([]) == nil) + let excessive = (0...512).map { Policy.Picture(dts: Int64($0 * 3003), poc: Int64($0 * 2)) } + precondition(repair(excessive, next: 2_000_000, confirmed: true) == nil) + print("PASS partial ctts: exact slots, mixed cadence, quantization, seek translation, unchanged decode axis; truncated/field/duplicate POC, insufficient lead, missing/duplicate timestamps, overflow and bounds refused") + } +} diff --git a/Scripts/tests/make-partial-ctts-fixture.rb b/Scripts/tests/make-partial-ctts-fixture.rb new file mode 100644 index 000000000..ddd49c32f --- /dev/null +++ b/Scripts/tests/make-partial-ctts-fixture.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# Use ONLY on the tiny generated color+tone fixture; no private media is copied to a test fixture. +# Preserve every box size and every byte outside the selected ctts offset fields. +input, output = ARGV +mode = ARGV[2] || 'tail' +abort 'mode must be tail or middle' unless %w[tail middle].include?(mode) +abort 'usage: make-partial-ctts-fixture.rb synthetic.mp4 new-output.mp4' unless input && output +abort 'output already exists' if File.exist?(output) +abort 'fixture size exceeded' if File.size(input) > 2 * 1024 * 1024 +data = File.binread(input) +abort 'not an explicitly generated test fixture' unless data.include?('Aether synthetic timestamp fixture') +changed = 0 +walk = lambda do |start, limit| + pos = start + while pos < limit + size = data.byteslice(pos, 4)&.unpack1('N') + kind = data.byteslice(pos + 4, 4) + abort 'unsupported fixture box' unless size && size >= 8 && pos + size <= limit + if %w[moov trak mdia minf stbl].include?(kind) + walk.call(pos + 8, pos + size) + elsif kind == 'ctts' + count = data.byteslice(pos + 12, 4).unpack1('N') + abort 'unexpected ctts length' unless size == 16 + 8 * count + samples = 0 + count.times do |i| + offset = pos + 16 + 8 * i + length = data.byteslice(offset, 4).unpack1('N') + abort 'fixture GOP boundary must align with ctts run' if samples < 240 && samples + length > 240 + abort 'fixture middle boundary must align with ctts run' if mode == 'middle' && samples < 480 && samples + length > 480 + if samples >= 240 && (mode == 'tail' || samples < 480) + data[offset + 4, 4] = [0].pack('N') + changed += length + end + samples += length + end + abort 'expected 720 generated video frames' unless samples == 720 + end + pos += size + end +end +walk.call(0, data.bytesize) +abort 'unexpected changed sample count' unless changed == (mode == 'tail' ? 480 : 240) +File.open(output, File::WRONLY | File::CREAT | File::EXCL, 0o600) { |file| file.write(data) } +puts "PASS generated partial-ctts fixture healthy_head_samples=240 zero_offset_samples=#{changed} mode=#{mode}" diff --git a/Sources/AetherEngine/Demuxer/Demuxer.swift b/Sources/AetherEngine/Demuxer/Demuxer.swift index b7601983f..e6de2c8bd 100644 --- a/Sources/AetherEngine/Demuxer/Demuxer.swift +++ b/Sources/AetherEngine/Demuxer/Demuxer.swift @@ -1255,12 +1255,12 @@ public final class Demuxer: @unchecked Sendable { guard let packet = try readPacketLocked() else { // EOF can arrive mid-sample on a very short source; the verdict has to be reached // now or the held packets would never be delivered. - compositionRepair?.endOfStream() + try compositionRepair?.endOfStream() if let held = compositionRepair?.dequeue() { return held } return nil } guard let repair = armCompositionRepairIfNeeded() else { return packet } - if !repair.ingest(packet) { return packet } + if try !repair.ingest(packet) { return packet } } } @@ -1282,10 +1282,10 @@ public final class Demuxer: @unchecked Sendable { guard let repair = armCompositionRepairIfNeeded(), !repair.isDecided else { return } while !repair.isDecided { guard let packet = try? readPacketLocked() else { - repair.endOfStream() + try? repair.endOfStream() return } - if !repair.ingest(packet) { + if (try? repair.ingest(packet)) == false { // Not held: the session is done with the sample and this packet is already on the // final axis, so it goes to the front of the queue rather than out of order. repair.enqueueFront(packet) diff --git a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift index fbd6c0ea2..3402691ce 100644 --- a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift +++ b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift @@ -508,6 +508,7 @@ enum H264CompositionOffsetRepair { final class H264PictureOrderReader { private var parser: UnsafeMutablePointer? private var context: UnsafeMutablePointer? + var isFramePicture: Bool { parser?.pointee.picture_structure == AV_PICTURE_STRUCTURE_FRAME } init?(codecParameters: UnsafePointer, timeBase: AVRational) { guard let codec = avcodec_find_decoder(AV_CODEC_ID_H264), @@ -555,7 +556,8 @@ final class H264PictureOrderReader { } } -/// Per-demuxer runtime for the #409 repair: samples the head, decides once, then rewrites. +/// Per-demuxer runtime for #409. The original all-missing policy decides at the head; +/// a corroborated healthy head also enables bounded detection of later partial-ctts regions. /// /// Sampling holds packets instead of rewinding the source. A rewind is not available to every /// session (a custom source cannot be reopened, and a probe demuxer handed to playback must not be @@ -574,6 +576,8 @@ final class H264CompositionOffsetRepairSession { private let ladderStart: Int64 private var reader: H264PictureOrderReader? private var rewriter: H264CompositionOffsetRepair.Rewriter? + private let partialRepairCandidate: H264PartialCompositionRepairSession? + private var monitorsPartialOffsets = false private var samples: [H264CompositionOffsetRepair.Sample] = [] private var held: [(packet: UnsafeMutablePointer, pictureOrderCount: Int64?)] = [] private var heldBytes = 0 @@ -600,6 +604,10 @@ final class H264CompositionOffsetRepairSession { self.streamStartTime = stream.pointee.start_time self.ladderStart = ladderStart self.reader = reader + let (lead, overflow) = stream.pointee.start_time.subtractingReportingOverflow(ladderStart) + partialRepairCandidate = !overflow && stream.pointee.start_time != Int64.min && ladderStart != Int64.min + ? H264PartialCompositionRepairSession(stream: stream, streamIndex: streamIndex, presentationLead: lead) + : nil } deinit { @@ -621,16 +629,17 @@ final class H264CompositionOffsetRepairSession { /// repaired packets cut segment 2 one picture past its keyframe, which is a segment AVPlayer /// cannot start at. var decodeTimestampOffset: Int64? { + if monitorsPartialOffsets { return 0 } guard phase == .repairing, let rewriter else { return nil } return rewriter.plan.shift - rewriter.plan.decodeLead } /// Returns true when the packet was taken over by the session and must not be emitted yet. /// A packet the session keeps is owned by it until `dequeue()` hands it back. - func ingest(_ packet: UnsafeMutablePointer) -> Bool { + func ingest(_ packet: UnsafeMutablePointer) throws -> Bool { switch phase { case .off: - return false + return monitorsPartialOffsets ? try partialRepairCandidate?.ingest(packet) ?? false : false case .repairing: guard packet.pointee.stream_index == streamIndex else { return false } applyRepair(to: packet, pictureOrderCount: reader?.pictureOrderCount(for: packet)) @@ -668,8 +677,9 @@ final class H264CompositionOffsetRepairSession { } /// EOF during sampling. Decides on what is there, so the held packets are still delivered. - func endOfStream() { + func endOfStream() throws { if phase == .sampling { decide() } + if monitorsPartialOffsets { try partialRepairCandidate?.endOfStream() } } func noteSeek() { @@ -680,6 +690,7 @@ final class H264CompositionOffsetRepairSession { // file (verdict `.off`, sample still held) put two duplicate pictures into the stream a // producer had already emitted. dropHeldPackets() + partialRepairCandidate?.noteSeek() switch phase { case .sampling: // The sample restarts where the source now stands. @@ -708,11 +719,15 @@ final class H264CompositionOffsetRepairSession { } func dequeue() -> UnsafeMutablePointer? { - guard phase != .sampling, !held.isEmpty else { return nil } - return held.removeFirst().packet + guard phase != .sampling else { return nil } + if !held.isEmpty { return held.removeFirst().packet } + return monitorsPartialOffsets ? partialRepairCandidate?.dequeue() : nil } var summary: String { + if monitorsPartialOffsets, let partialRepairCandidate, partialRepairCandidate.hasDecision { + return partialRepairCandidate.summary + } var text = "phase=\(phase) verdict=\(verdictDescription)" if let rewriter { text += " repaired=\(rewriter.repairedPictures) unrepaired=\(rewriter.unrepairedPictures)" @@ -750,6 +765,12 @@ final class H264CompositionOffsetRepairSession { category: .demux ) case .healthy: + // An actual healthy origin corroborates the container's edit/index lead. A random + // nonzero B-picture offset is not evidence for retiming a later region. + if let first = samples.first, first.isKeyframe, first.pictureOrderCount == 0, + first.dts == ladderStart, first.pts == streamStartTime { + monitorsPartialOffsets = partialRepairCandidate != nil + } verdictDescription = "healthy" disarm() case .inconclusive(let reason): diff --git a/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift b/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift new file mode 100644 index 000000000..bd0128651 --- /dev/null +++ b/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift @@ -0,0 +1,45 @@ +import Foundation + +/// A mixed MP4 can carry valid ctts at the head and zero offsets in a later IDR sequence. +/// Restore that sequence's display ownership of the ORIGINAL timestamp slots. In particular, +/// a cadence change inside the sequence must not turn into a guessed constant-rate clock. +enum H264PartialCompositionRepair { + static let maximumPictures = 512 + static let maximumHeldPackets = 1024 + static let maximumHeldBytes = 32 << 20 + struct Picture: Equatable { + let dts: Int64 + let poc: Int64 + } + + static func presentationTimes( + _ pictures: [Picture], nextDTS: Int64, presentationLead: Int64, + previouslyConfirmed: Bool + ) -> [Int64]? { + guard (1...maximumPictures).contains(pictures.count), pictures.first?.poc == 0, + presentationLead > 0, nextDTS != Int64.min else { return nil } + let slots = pictures.map(\.dts) + guard slots.allSatisfy({ $0 != Int64.min }) else { return nil } + for (a, b) in zip(slots, slots.dropFirst() + [nextDTS]) { + let (step, overflow) = b.subtractingReportingOverflow(a) + guard !overflow, step > 0 else { return nil } + } + var ranks = Set() + var reordered = false + var result: [Int64] = [] + for (index, picture) in pictures.enumerated() { + // Complete, closed, progressive sequences only. Do not infer missing pictures, + // fields, open-GOP leading pictures, or parser/POC wraparound. + guard picture.poc >= 0, picture.poc % 2 == 0, + picture.poc / 2 < Int64(pictures.count) else { return nil } + let rank = Int(picture.poc / 2) + guard ranks.insert(rank).inserted else { return nil } + reordered = reordered || rank != index + let (pts, overflow) = slots[rank].addingReportingOverflow(presentationLead) + guard !overflow, pts >= picture.dts else { return nil } + result.append(pts) + } + guard reordered || previouslyConfirmed else { return nil } + return result + } +} diff --git a/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift b/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift new file mode 100644 index 000000000..40e22e69d --- /dev/null +++ b/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift @@ -0,0 +1,194 @@ +import Foundation +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil + +/// Enabled only by a healthy head whose IDR offset agrees with the container edit/index axis. +/// Healthy packets remain a zero-hold fast path. A zero-offset IDR starts one bounded sequence +/// probe; all interleaved packets are owned until its complete POC permutation is established. +/// DTS NEVER changes, so an already published keyframe index remains valid across region changes. +final class H264PartialCompositionRepairSession { + private struct Entry { + let packet: UnsafeMutablePointer + let poc: Int64? + } + enum RepairError: Error { case sequenceNoLongerRepairable } + private let streamIndex: Int32 + private let timeBase: AVRational + private let lead: Int64 + private let framing: VideoNALFraming + private let reader: H264PictureOrderReader + private var pending: [Entry] = [] + private var ready: [UnsafeMutablePointer] = [] + private var readyIndex = 0 + private var bytes = 0 + private var videoCount = 0 + private var confirmed = false + private var failed = false + private var repairedCount = 0 + private var sampleCount = 0 + private var heldCount = 0 + private var heldBytes = 0 + private var regressions = 0 + private(set) var reason = "composition_offsets_present" + + init?(stream: UnsafeMutablePointer, streamIndex: Int32, presentationLead: Int64) { + guard presentationLead > 0, let par = stream.pointee.codecpar, + (1...16).contains(par.pointee.video_delay), + let reader = H264PictureOrderReader(codecParameters: par, timeBase: stream.pointee.time_base) + else { return nil } + self.reader = reader + self.streamIndex = streamIndex + lead = presentationLead + timeBase = stream.pointee.time_base + framing = A53SEIParser.nalFraming(codec: .h264, extradata: par.pointee.extradata, size: Int(par.pointee.extradata_size)) + } + + deinit { releasePackets() } + var hasDecision: Bool { reason != "composition_offsets_present" } + + private func isIDR(_ packet: UnsafeMutablePointer) -> Bool { + guard packet.pointee.flags & AV_PKT_FLAG_KEY != 0, + let data = packet.pointee.data, packet.pointee.size > 0 else { return false } + var result = false + A53SEIParser.forEachNAL(data, Int(packet.pointee.size), framing) { nal, size in + if size > 0, nal[0] & 0x80 == 0, nal[0] & 31 == 5 { result = true } + } + return result + } + + func ingest(_ packet: UnsafeMutablePointer) throws -> Bool { + if failed { + var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) + throw RepairError.sequenceNoLongerRepairable + } + if packet.pointee.stream_index != streamIndex { + guard !pending.isEmpty else { return false } + append(packet, poc: nil) + try checkBounds() + return true + } + let idr = isIDR(packet) + if idr, videoCount > 0 { + do { try finishSequence(nextDTS: packet.pointee.dts) } + catch { var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned); throw error } + } + // A missing timestamp differs numerically from a valid one but is NOT evidence of + // healthy composition offsets. Preserve ownership and use the same refusal policy. + if packet.pointee.pts == Int64.min || packet.pointee.dts == Int64.min { + append(packet, poc: nil); videoCount += 1 + try refuse() + return true + } + let offsets = packet.pointee.pts != packet.pointee.dts + if offsets { + // Genuine composition offsets always win. A mixed/unsupported sequence is emitted + // unchanged; do not let its boundary packet overtake packets already held. + publishPending() + confirmed = false + reason = "composition_offsets_present" + if readyIndex < ready.count { ready.append(packet); return true } + return false + } + if pending.isEmpty { + guard idr else { + if readyIndex < ready.count { ready.append(packet); return true } + return false + } + reader.reset() + } + let poc = reader.pictureOrderCount(for: packet) + let invalid = packet.pointee.dts == Int64.min || poc == nil || !reader.isFramePicture + || (videoCount == 0 && poc != 0) + || (packet.pointee.flags & AV_PKT_FLAG_KEY != 0 && (!idr || poc != 0)) + append(packet, poc: poc) + videoCount += 1 + if invalid { try refuse(); return true } + try checkBounds() + return true + } + + private func append(_ packet: UnsafeMutablePointer, poc: Int64?) { + pending.append(Entry(packet: packet, poc: poc)) + bytes += Int(max(0, packet.pointee.size)) + } + + private func checkBounds() throws { + if bytes >= H264PartialCompositionRepair.maximumHeldBytes + || pending.count >= H264PartialCompositionRepair.maximumHeldPackets + || videoCount > H264PartialCompositionRepair.maximumPictures { try refuse() } + } + + private func finishSequence(nextDTS: Int64) throws { + let video = pending.filter { $0.packet.pointee.stream_index == streamIndex } + let pictures = video.map { H264PartialCompositionRepair.Picture(dts: $0.packet.pointee.dts, poc: $0.poc ?? -1) } + guard let corrected = H264PartialCompositionRepair.presentationTimes( + pictures, nextDTS: nextDTS, presentationLead: lead, previouslyConfirmed: confirmed + ) else { try refuse(); return } + if !confirmed { + sampleCount = video.count; heldCount = pending.count; heldBytes = bytes + regressions = zip(pictures, pictures.dropFirst()).filter { $1.poc < $0.poc }.count + EngineLog.emit("[Demuxer] partial H264 composition offsets confirmed: pictures=\(sampleCount) poc_regressions=\(regressions) presentation_lead=\(lead) decode_offset=0 time_base=\(timeBase.num)/\(timeBase.den)", category: .demux) + } + // The policy validates the ENTIRE sequence before mutating any packet. Packet payload, + // DTS, duration, flags, side data, and every non-video stream remain bit-for-bit intact. + for (entry, pts) in zip(video, corrected) { entry.packet.pointee.pts = pts } + repairedCount += video.count + confirmed = true + reason = "confirmed_partial_composition_offsets" + publishPending() + } + + private func refuse() throws { + reason = "partial_composition_sequence_unproven" + if confirmed { + failed = true + releasePackets() + throw RepairError.sequenceNoLongerRepairable + } + publishPending() + } + + private func publishPending() { + ready.append(contentsOf: pending.map(\.packet)) + pending.removeAll(keepingCapacity: true) + bytes = 0; videoCount = 0 + } + + func dequeue() -> UnsafeMutablePointer? { + guard readyIndex < ready.count else { return nil } + let packet = ready[readyIndex]; readyIndex += 1 + if readyIndex == ready.count { ready.removeAll(keepingCapacity: true); readyIndex = 0 } + return packet + } + + func endOfStream() throws { + if failed { throw RepairError.sequenceNoLongerRepairable } + guard !pending.isEmpty else { return } + guard let last = pending.last(where: { $0.packet.pointee.stream_index == streamIndex }) else { + publishPending(); return + } + let (end, overflow) = last.packet.pointee.dts.addingReportingOverflow(max(1, last.packet.pointee.duration)) + guard !overflow else { try refuse(); return } + try finishSequence(nextDTS: end) + } + + func noteSeek() { + releasePackets(); reader.reset() + confirmed = false; failed = false + reason = "composition_offsets_present" + } + + private func releasePackets() { + for entry in pending { var owned: UnsafeMutablePointer? = entry.packet; trackedPacketFree(&owned) } + for packet in ready.dropFirst(readyIndex) { var owned: UnsafeMutablePointer? = packet; trackedPacketFree(&owned) } + pending.removeAll(keepingCapacity: true); ready.removeAll(keepingCapacity: true) + readyIndex = 0; bytes = 0; videoCount = 0 + } + + var summary: String { + "reason=\(reason) samples=\(sampleCount) held_packets=\(heldCount) held_bytes=\(heldBytes)" + + " poc_regressions=\(regressions) presentation_lead=\(lead) decode_offset=0" + + " repaired=\(repairedCount)" + } +} diff --git a/Tests/AetherEngineTests/DocumentedConstantsTests.swift b/Tests/AetherEngineTests/DocumentedConstantsTests.swift index 409214b70..3fc8e25b8 100644 --- a/Tests/AetherEngineTests/DocumentedConstantsTests.swift +++ b/Tests/AetherEngineTests/DocumentedConstantsTests.swift @@ -19,6 +19,15 @@ import AetherLibavcodec @MainActor final class DocumentedConstantsTests: XCTestCase { + func testPartialCompositionHoldBoundsMatchDocumentation() throws { + let docs = try documentation() + XCTAssertEqual(H264PartialCompositionRepair.maximumPictures, 512) + XCTAssertEqual(H264PartialCompositionRepair.maximumHeldPackets, 1024) + XCTAssertEqual(H264PartialCompositionRepair.maximumHeldBytes, 32 << 20) + assertDocumented("512 video pictures, 1024 interleaved", docs) + assertDocumented("packets and 32 MiB", docs) + } + // MARK: - Documentation corpus private static var repoRoot: URL { diff --git a/docs/formats.md b/docs/formats.md index 9847e0bfe..7318ff6f7 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -58,6 +58,34 @@ short of that (variable frame timing, a picture order that does not advance one sample that starts nowhere it can be anchored) is delivered exactly as the container wrote it. Reported by @orut34iop. +### MP4 with composition offsets missing only in later regions + +A healthy head does not establish a healthy table for the whole file. Some mixed +MP4s retain valid offsets at the head, then give later reordered pictures zero +offsets. Seeking into that region can produce persistent judder despite normal +aggregate FPS and sufficient network buffering. + +When a healthy origin picture corroborates the container's edit/index lead, the +demuxer keeps a zero-hold healthy path and watches for zero-offset IDRs. One +bounded, complete IDR-to-IDR progressive sequence is parsed for picture order. +If every selected packet has valid equal PTS/DTS and its distinct even POC fills +the complete sequence, a proven permutation assigns its original DTS slots plus +the corroborated presentation lead by display rank. Original DTS, audio and the +already published keyframe index never move. Actual timestamp slots, rather than +an average-FPS clock, preserve interval changes and quantization within a sequence. + +The partial-composition hold is bounded by **512 video pictures, 1024 interleaved +packets and 32 MiB**. Healthy nonzero offsets resume unchanged delivery. Fields, +missing timestamps, incomplete POC, arithmetic overflow and insufficient lead are +not guessed. Before confirmation an unproven sequence passes through unchanged; +after confirmation an unsupported zero-offset sequence fails explicitly instead +of silently mixing repaired and unrepaired timing. Seek and teardown release both +unpublished input and pending output. No decoder-route or host-UI change is needed. +The existing whole-file missing-offset policy above remains separate. + +See the [partial-composition regression and reproduction](partial-composition-regression.md) +for generated fixtures, original numeric evidence and verification limits. + ## HDR routing | Source | Wrapper signaling | diff --git a/docs/partial-composition-regression.md b/docs/partial-composition-regression.md new file mode 100644 index 000000000..c7ea2e652 --- /dev/null +++ b/docs/partial-composition-regression.md @@ -0,0 +1,74 @@ +# Partial H.264 MP4 composition-offset regression + +## Reproduction + +On the reported MP4, short playback at the head appeared normal. Seeking to about +700 seconds caused persistent judder. A short healthy-head check is not proof +that uninterrupted playback remains healthy past the malformed region. + +Source characteristics: MP4, progressive SDR H.264 High level 4.0, 1920x1080, +nominal 30000/1001, time base 1/90000, AAC-LC 48 kHz. No HDR/Dolby Vision. Video +duration about 9438 seconds, 282860 samples. The edit/index lead is 6006 ticks. +No private source name, path, URL, packet payload or raw device log is published. + +Direct sample-table inspection showed meaningful `ctts` offsets in the first +3272 samples, followed by one run of 279588 zero offsets. The table covers every +sample; it is not a truncated binary box. At 700 seconds all 240 inspected video +packets had PTS equal to DTS, but parsed POC reversed 68 times. Decoding a bounded +120-frame slice gave 34 original frame-PTS regressions. FFmpeg's monotonic +`best_effort_timestamp` is synthesized and does not prove those original times +correct. The device continued reporting about 30 FPS, enough buffered data and +no aggregate drops/stalls; those metrics do not establish picture-time ownership. + +The first faulty sequence also changes interval: eleven 3600-tick steps followed +by 3003/3004 ticks within the same GOP. A short fixed-cadence prototype corrected +the later seek but failed the continuous boundary. It is not the submitted fix. +The submitted policy uses complete POC rank and the existing timestamp slots, +preserving original DTS instead of moving the already published index axis. + +## Generated regression + +Run on an Apple Silicon/Xcode development host, with FFmpeg CLI available: + +```sh +bash Scripts/test-h264-partial-composition-controls.sh +``` + +The runtime compiler uses the exact `FFmpegBuild` revision in `Package.resolved`. +After normal package resolution it defaults to `.build/checkouts/FFmpegBuild`; +set `AETHER_FFMPEG_CHECKOUT` to an existing checkout at that exact revision when +needed. It does not resolve or update dependencies itself. + +The fixture is generated solid colour plus tone, not an excerpt of any private +film. The generator keeps a healthy prefix and clears only later `ctts` offset +fields, without changing box lengths, DTS, compressed packets or audio. A second +fixture clears only the middle so the broken-to-healthy boundary is also tested. +The executable runs the actual parser and repair session against two decoders: +original versus repaired packet timestamps. It asserts zero repaired frame-PTS +regressions, exact original DTS, payload/metadata/non-video preservation and zero +tracked packet balance. Checks cover seek/back-seek, both continuous boundaries, +EOF, abandoned pending/ready queues, missing timestamps after confirmation and +the interleaved-packet budget. Numeric tests cover mixed cadence, quantization, +incomplete/field/duplicate POC, insufficient lead and integer overflow. + +## Results and limits + +The privately retained real-source runtime check passed at 0, 700, back to 0, +107, 112, 3600 and 9400 seconds. Original frame-PTS regressions were +0/49/0/20/50/48/46; all repaired results were zero. The 107-second check crosses +the first faulty sequence without a seek at the actual boundary. The generated +fixture also proves recovery back into healthy offsets without changing them. + +The reporter verified the downstream implementation on an Apple TV 4K +(3rd generation), tvOS 26.6 beta (23L773), with the original source on the native +hardware H.264 route. The scoped upstream branch contains the same repair and +post-review malformed-timestamp guard; it does not include the host's diagnostic +framework, UI customizations, Matroska repair or software cache changes. The +generated fixture proves the packet/decoder regression; it is not claimed to +visually reproduce the original movie's native judder. + +This is a narrow compatibility policy, not a general VFR timestamp reconstructor. +It requires a corroborated healthy origin and complete closed progressive POC +sequences. Unsupported input is not re-timed by guessing its average frame rate. +Other operating systems, interlaced/open-GOP sources and arbitrary malformed +timelines have not been certified by the reported physical test.