diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c1299d55..e189d6e9c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -40,6 +40,15 @@ jobs: - name: swift test run: swift test + - name: Software VOD packet-cache regressions + run: | + bash Scripts/test-software-packet-coverage.sh + bash Scripts/test-software-video-packet-coverage.sh + bash Scripts/test-software-packet-disk-fifo.sh + bash Scripts/test-software-packet-read-ahead.sh + bash Scripts/test-software-read-admission.sh + bash Scripts/test-software-stored-packet.sh + # The transport probe is a device harness and runs nowhere in CI, so nothing else would keep # it compiling. A harness that stopped building is discovered by the person who needed it. - name: build the transport probe (AE#377) diff --git a/CHANGELOG.md b/CHANGELOG.md index 767414dc9..f4da071e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,15 @@ the public-API contract. ## [Unreleased] -_Nothing yet._ +### Fixed + +- Software VOD now reads compressed packets ahead independently of the decoded + display queue and publishes a continuous selected A/V cache frontier from + startup. Retained packet chunks support forward and backward cache-local + seeks without discarding their existing frontier. H.264 variable-frame-rate + coverage follows presentation successors rather than packet decode duration. + Byte/time limits, disk cleanup, and seek-generation admission bound the store + and reject stale packets, EOF, errors and delayed end-of-media callbacks. ## [6.71.0] - 2026-09-06 diff --git a/Scripts/test-software-packet-coverage.sh b/Scripts/test-software-packet-coverage.sh new file mode 100644 index 000000000..d80a4cf36 --- /dev/null +++ b/Scripts/test-software-packet-coverage.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-packet-coverage.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +# Pure Foundation timestamp model; not a macOS/iOS application target. +xcrun swiftc \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift" \ + "$TASK_ROOT/Scripts/tests/SoftwarePacketCoverageStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" diff --git a/Scripts/test-software-packet-disk-fifo.sh b/Scripts/test-software-packet-disk-fifo.sh new file mode 100644 index 000000000..ecabf43e8 --- /dev/null +++ b/Scripts/test-software-packet-disk-fifo.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-packet-disk-fifo.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +# Pure Foundation host check; does not build or validate a macOS/iOS application target. +xcrun swiftc -swift-version 6 \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift" \ + "$TASK_ROOT/Scripts/tests/SoftwarePacketDiskFIFOStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" diff --git a/Scripts/test-software-packet-read-ahead.sh b/Scripts/test-software-packet-read-ahead.sh new file mode 100644 index 000000000..f5ff1361d --- /dev/null +++ b/Scripts/test-software-packet-read-ahead.sh @@ -0,0 +1,14 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-packet-read-ahead.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +# Pure Foundation concurrency/data test; no macOS/iOS application target is built. +xcrun swiftc \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareStoredPacket.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift" \ + "$TASK_ROOT/Scripts/tests/SoftwarePacketReadAheadStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" diff --git a/Scripts/test-software-read-admission.sh b/Scripts/test-software-read-admission.sh new file mode 100644 index 000000000..552a0602e --- /dev/null +++ b/Scripts/test-software-read-admission.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-read-admission.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +# Pure Foundation generation-policy check; no macOS/iOS application/package build. +xcrun swiftc -swift-version 6 \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareReadAdmission.swift" \ + "$TASK_ROOT/Scripts/tests/SoftwareReadAdmissionStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" diff --git a/Scripts/test-software-stored-packet.sh b/Scripts/test-software-stored-packet.sh new file mode 100644 index 000000000..7b58195dc --- /dev/null +++ b/Scripts/test-software-stored-packet.sh @@ -0,0 +1,33 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_FFMPEG_ROOT="${AETHER_FFMPEG_CHECKOUT:-$TASK_ROOT/.build/checkouts/FFmpegBuild}" +TASK_EXPECTED_REVISION=$(/usr/bin/ruby -rjson -e \ + 'pin = JSON.parse(File.read(ARGV.fetch(0))).fetch("pins").find { |p| p.fetch("identity") == "ffmpegbuild" }; abort "Missing FFmpegBuild pin" unless pin; puts pin.fetch("state").fetch("revision")' \ + "$TASK_ROOT/Package.resolved") +TASK_ACTUAL_REVISION=$(git -C "$TASK_FFMPEG_ROOT" rev-parse HEAD) +if [[ "$TASK_ACTUAL_REVISION" != "$TASK_EXPECTED_REVISION" ]]; then + echo "FAIL: existing FFmpegBuild checkout does not match the frozen package revision" >&2 + exit 1 +fi +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-stored-packet.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +TASK_FRAMEWORK_ARGS=() +for TASK_LIBRARY in AetherLibavcodec AetherLibavutil AetherLibswresample AetherLibdav1d AetherLibzvbi; do + TASK_FRAMEWORK_DIR="$TASK_FFMPEG_ROOT/Sources/$TASK_LIBRARY.xcframework/macos-arm64_x86_64" + if [[ ! -f "$TASK_FRAMEWORK_DIR/$TASK_LIBRARY.framework/$TASK_LIBRARY" ]]; then + echo "FAIL: required existing host framework is missing: $TASK_LIBRARY" >&2 + exit 1 + fi + TASK_FRAMEWORK_ARGS+=(-F "$TASK_FRAMEWORK_DIR" -Xlinker -rpath -Xlinker "$TASK_FRAMEWORK_DIR") +done +# Low-level host command-line packet test against the already-pinned binary dependency. +# This does not build/resolve a package, build a macOS/iOS app, or modify package caches. +xcrun swiftc -swift-version 6 \ + "${TASK_FRAMEWORK_ARGS[@]}" -framework AetherLibavcodec -framework AetherLibavutil \ + "$TASK_ROOT/Sources/AetherEngine/Diagnostics/PacketBalanceTracker.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareStoredPacket.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareStoredPacket+FFmpeg.swift" \ + "$TASK_ROOT/Scripts/tests/SoftwareStoredPacketStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" +echo "FFmpegBuild revision: $TASK_ACTUAL_REVISION" diff --git a/Scripts/test-software-video-packet-coverage.sh b/Scripts/test-software-video-packet-coverage.sh new file mode 100644 index 000000000..3f86c64fb --- /dev/null +++ b/Scripts/test-software-video-packet-coverage.sh @@ -0,0 +1,11 @@ +#!/bin/bash +set -euo pipefail +TASK_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +TASK_TMP=$(mktemp -d "${TMPDIR:-/tmp}/aether-video-packet-coverage.XXXXXX") +trap 'rm -f "$TASK_TMP/check"; rmdir "$TASK_TMP"' EXIT +# Pure Foundation timestamp-model check; no macOS/iOS application or package build. +xcrun swiftc -swift-version 6 \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift" \ + "$TASK_ROOT/Scripts/tests/SoftwareVideoPacketCoverageStandalone.swift" -o "$TASK_TMP/check" +"$TASK_TMP/check" diff --git a/Scripts/tests/SoftwarePacketCoverageStandalone.swift b/Scripts/tests/SoftwarePacketCoverageStandalone.swift new file mode 100644 index 000000000..4ea8cbbd3 --- /dev/null +++ b/Scripts/tests/SoftwarePacketCoverageStandalone.swift @@ -0,0 +1,184 @@ +import Foundation + +@main +struct SoftwarePacketCoverageTests { + static func main() { + reorderedPresentationPackets() + negativeAndFractionalTimestamps() + selectedStreamIntersection() + resetPruneAndCapacity() + malformedAndOverflow() + exhaustiveSmallUnion() + print("PASS: PTS coverage, B-frame holes, fractional boundaries, selected A/V, reset/prune, bounded gaps, malformed input and overflow") + } + + static func reorderedPresentationPackets() { + var coverage = SoftwarePacketCoverage() + precondition(coverage.insert(pts: 0, duration: 1001)) + // Decode order: I, future P, then the two intervening B pictures. + precondition(coverage.insert(pts: 3003, duration: 1001)) + precondition(coverage.frontier(containing: 0) == 1001) + precondition(coverage.frontier(containing: 1001) == nil) + precondition(coverage.frontier(containing: 2002) == nil) + precondition(coverage.insert(pts: 2002, duration: 1001)) + precondition(coverage.frontier(containing: 0) == 1001) + precondition(coverage.frontier(containing: 2002) == 4004) + precondition(coverage.insert(pts: 1001, duration: 1001)) + precondition(coverage.rangeCount == 1) + precondition(coverage.frontier(containing: 0) == 4004) + precondition(coverage.frontier(containing: 4003) == 4004) + precondition(coverage.frontier(containing: 4004) == nil) + // Overlap/duplicate insertion neither splits coverage nor inflates its endpoint. + precondition(coverage.insert(pts: 500, duration: 1500)) + precondition(coverage.insert(pts: 0, duration: 1001)) + precondition(coverage.rangeCount == 1) + precondition(coverage.frontier(containing: 500) == 4004) + } + + static func negativeAndFractionalTimestamps() { + var coverage = SoftwarePacketCoverage() + precondition(coverage.insert(pts: -1001, duration: 1001)) + precondition(coverage.insert(pts: 0, duration: 1001)) + precondition(coverage.frontier(containing: -1001) == 1001) + precondition(coverage.frontier(containing: -1002) == nil) + let start = -1001.0 / 30000.0 + let end = 1001.0 / 30000.0 + func seconds(_ time: Double) -> Double? { + coverage.frontierSeconds(containing: time, timeBaseNumerator: 1, timeBaseDenominator: 30000) + } + precondition(seconds(start.nextDown) == nil) + precondition(seconds(start) == end) + precondition(seconds(0) == end) + precondition(seconds(end.nextDown) == end) + precondition(seconds(end) == nil) + // A real one-tick gap must not disappear at NTSC fractional-frame boundaries. + precondition(coverage.insert(pts: 1002, duration: 1001)) + precondition(seconds(end) == nil) + precondition(seconds(1001.5 / 30000.0) == nil) + precondition(seconds(1002.0 / 30000.0) == 2003.0 / 30000.0) + precondition(coverage.insert(pts: 1001, duration: 1)) + precondition(seconds(end) == 2003.0 / 30000.0) + + var rational = SoftwarePacketCoverage() + precondition(rational.insert(pts: 3, duration: 2)) + precondition(rational.frontierSeconds(containing: 3 * 1001.0 / 30000.0, + timeBaseNumerator: 1001, timeBaseDenominator: 30000) == 5 * 1001.0 / 30000.0) + } + + static func selectedStreamIntersection() { + let combine = SoftwarePacketCoverage.combinedFrontier + precondition(combine(10, 7, true) == 7) + precondition(combine(7, 10, true) == 7) + precondition(combine(10, nil, true) == nil) + precondition(combine(nil, 10, true) == nil) + precondition(combine(10, nil, false) == 10) + precondition(combine(10, .nan, false) == 10) + precondition(combine(10, .nan, true) == nil) + precondition(combine(.infinity, 10, false) == nil) + precondition(combine(-1, -2, true) == -2) + + var video = SoftwarePacketCoverage() + var audio = SoftwarePacketCoverage() + video.insert(pts: 0, duration: 300300) + audio.insert(pts: 0, duration: 336000) + let videoEnd = video.frontierSeconds(containing: 1, timeBaseNumerator: 1, timeBaseDenominator: 30000) + let audioEnd = audio.frontierSeconds(containing: 1, timeBaseNumerator: 1, timeBaseDenominator: 48000) + precondition(combine(videoEnd, audioEnd, true) == 7) + } + + static func resetPruneAndCapacity() { + var coverage = SoftwarePacketCoverage(maximumRangeCount: 2) + precondition(coverage.insert(pts: 0, duration: 10)) + precondition(coverage.insert(pts: 20, duration: 10)) + precondition(!coverage.insert(pts: 40, duration: 10)) + precondition(coverage.rangeCount == 2) + precondition(coverage.frontier(containing: 40) == nil) + precondition(coverage.frontier(containing: 0) == 10) + precondition(coverage.frontier(containing: 15) == nil) + // Joining existing ranges is allowed at capacity, as is extending an existing range. + precondition(coverage.insert(pts: 10, duration: 10)) + precondition(coverage.rangeCount == 1) + precondition(coverage.insert(pts: 40, duration: 10)) + precondition(coverage.insert(pts: 50, duration: 10)) + precondition(coverage.rangeCount == 2) + coverage.prune(before: 5) + precondition(coverage.frontier(containing: 0) == 30) + coverage.prune(before: 30) + precondition(coverage.frontier(containing: 29) == nil) + precondition(coverage.frontier(containing: 35) == nil) + precondition(coverage.frontier(containing: 40) == 60) + coverage.reset() + precondition(coverage.rangeCount == 0) + precondition(coverage.frontier(containing: 45) == nil) + precondition(coverage.insert(pts: 200, duration: 5)) + precondition(coverage.frontier(containing: 45) == nil) + precondition(coverage.frontier(containing: 200) == 205) + + var sparse = SoftwarePacketCoverage() + for index in 0..<4096 { precondition(sparse.insert(pts: Int64(index * 2), duration: 1)) } + precondition(!sparse.insert(pts: 8192, duration: 1)) + precondition(sparse.rangeCount == 4096) + precondition(sparse.frontier(containing: 1) == nil) + precondition(sparse.frontier(containing: 8192) == nil) + sparse.prune(before: 4096) + precondition(sparse.rangeCount == 2048) + precondition(sparse.insert(pts: 8192, duration: 1)) + } + + static func malformedAndOverflow() { + var coverage = SoftwarePacketCoverage() + precondition(!coverage.insert(pts: .min, duration: 10)) + precondition(!coverage.insert(pts: 0, duration: 0)) + precondition(!coverage.insert(pts: 0, duration: -1)) + precondition(!coverage.insert(pts: .max, duration: 1)) + precondition(!coverage.insert(pts: .max - 1, duration: 2)) + precondition(coverage.rangeCount == 0) + precondition(coverage.insert(pts: .max - 1, duration: 1)) + precondition(coverage.frontier(containing: .max - 1) == .max) + precondition(coverage.frontier(containing: .max) == nil) + precondition(coverage.frontierSeconds(containing: Double(Int64.max - 1), + timeBaseNumerator: 1, timeBaseDenominator: 1) == nil) + coverage.reset() + precondition(coverage.insert(pts: .min + 1, duration: 1)) + precondition(coverage.frontier(containing: .min + 1) == .min + 2) + precondition(coverage.frontier(containing: .min) == nil) + coverage.reset() + coverage.insert(pts: 0, duration: 10) + for invalid in [Double.nan, .infinity, -.infinity] { + precondition(coverage.frontierSeconds(containing: invalid, + timeBaseNumerator: 1, timeBaseDenominator: 30) == nil) + } + for invalid: Int32 in [0, -1, .min] { + precondition(coverage.frontierSeconds(containing: 0, + timeBaseNumerator: invalid, timeBaseDenominator: 30) == nil) + precondition(coverage.frontierSeconds(containing: 0, + timeBaseNumerator: 1, timeBaseDenominator: invalid) == nil) + } + coverage.prune(before: .min) + precondition(coverage.frontier(containing: 0) == 10) + } + + static func exhaustiveSmallUnion() { + // Compare interval merging with an independent per-tick union across many insertion + // orders, including negative timestamps, overlaps and future islands. + for seed in 0..<32 { + var coverage = SoftwarePacketCoverage() + var ticks = Set() + for step in 0..<80 { + let start = Int64((step * 29 + seed * 13) % 71 - 35) + let duration = Int64((step * 7 + seed) % 9 + 1) + precondition(coverage.insert(pts: start, duration: duration)) + for tick in start..<(start + duration) { ticks.insert(tick) } + for tick: Int64 in -36...45 { + var expected: Int64? = nil + if ticks.contains(tick) { + var end = tick + 1 + while ticks.contains(end) { end += 1 } + expected = end + } + precondition(coverage.frontier(containing: tick) == expected) + } + } + } + } +} diff --git a/Scripts/tests/SoftwarePacketDiskFIFOStandalone.swift b/Scripts/tests/SoftwarePacketDiskFIFOStandalone.swift new file mode 100644 index 000000000..448215c75 --- /dev/null +++ b/Scripts/tests/SoftwarePacketDiskFIFOStandalone.swift @@ -0,0 +1,535 @@ +import Darwin +import Foundation + +@main +struct SoftwarePacketDiskFIFOTests { + static func main() throws { + let root = FileManager.default.temporaryDirectory.appendingPathComponent( + "aether-disk-fifo-tests-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: root, withIntermediateDirectories: false) + defer { try? FileManager.default.removeItem(at: root) } + let sentinel = root.appendingPathComponent("unrelated") + try Data([42]).write(to: sentinel, options: .withoutOverwriting) + + try expectFailure { _ = try SoftwarePacketDiskFIFO(chunkTargetBytes: 7, parentDirectory: root) } + try roundTrip(root) + try lifecycle(root) + try boundedMetadata(root) + try diskFailures(root) + try concurrentAccess(root) + try retainedSeeks(root) + try retainedEvictionAndReset(root) + try retainedTrimRestoreAndRollover(root) + try concurrentRetainedReplay(root) + try retainedModelSequence(root) + try staleCleanup(root) + try interruptedInitialization(root) + try check(try Data(contentsOf: sentinel) == Data([42]), "cleanup touched an unrelated file") + let remaining = try FileManager.default.contentsOfDirectory(atPath: root.path) + precondition(remaining == ["unrelated"], "owned storage leaked: \(remaining)") + print("PASS: disk FIFO roundtrip, cross-chunk, partial drain, oversized/empty records, reset/close, failures, constant metadata, concurrent access, retained forward/backward seeks and eviction/generation bounds, concurrent replay, bounded stale cleanup/live leases/symlink isolation/interrupted init") + } + + static func roundTrip(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 40, parentDirectory: root) + defer { try? store.close() } + let records = [Data([1, 2]), Data(repeating: 3, count: 9), Data(repeating: 4, count: 80), + Data(), Data(repeating: 5, count: 12)] + for record in records { try store.append(record) } + precondition(store.snapshot.count == 5) + precondition(store.snapshot.byteCount == 103) + precondition(store.snapshot.chunkCount == 3) + precondition(store.snapshot.diskByteCount == 143) + try check(try store.pop() == records[0]) + precondition(store.snapshot.chunkCount == 3, "partially consumed chunk was prematurely deleted") + try check(try store.pop() == records[1]) + precondition(store.snapshot.chunkCount == 2, "consumed chunk was not reclaimed") + precondition(store.snapshot.diskByteCount == 116) + for record in records.dropFirst(2) { try check(try store.pop() == record) } + try check(try store.pop() == nil) + precondition(store.snapshot.count == 0 && store.snapshot.byteCount == 0) + precondition(store.snapshot.chunkCount == 0 && store.snapshot.diskByteCount == 0) + try check(try FileManager.default.contentsOfDirectory(atPath: store.storageDirectory.path) == ["session.lock"]) + + // Repeated appends to a chunk whose reader is already open must observe the new tail. + try store.append(Data([6])) + try store.append(Data([7])) + try check(try store.pop() == Data([6])) + try store.append(Data([8])) + try check(try store.pop() == Data([7])) + try check(try store.pop() == Data([8])) + try store.append(Data([9])) + try check(try store.pop() == Data([9]), "append after complete drain failed") + } + + static func lifecycle(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 16, parentDirectory: root) + let directory = store.storageDirectory + let marker = directory.appendingPathComponent("session.lock") + let initialInode = try FileManager.default.attributesOfItem(atPath: marker.path)[.systemFileNumber] as? NSNumber + let nonChunk = directory.appendingPathComponent("diagnostic-note") + try Data([55]).write(to: nonChunk) + try store.append(Data([1])) + try store.append(Data([2])) + try check(try store.pop() == Data([1])) + try store.reset() + let resetInode = try FileManager.default.attributesOfItem(atPath: marker.path)[.systemFileNumber] as? NSNumber + precondition(initialInode != nil && resetInode == initialInode, "reset replaced the live lease inode") + try check(try Data(contentsOf: nonChunk) == Data([55]), "reset removed a non-chunk file") + let markerFD = open(marker.path, O_RDONLY | O_NOFOLLOW) + precondition(markerFD >= 0) + defer { Darwin.close(markerFD) } + precondition(flock(markerFD, LOCK_EX | LOCK_NB) != 0 && errno == EWOULDBLOCK, + "reset released its live lease") + precondition(store.storageDirectory == directory) + precondition(store.snapshot.count == 0 && store.snapshot.chunkCount == 0) + try check(try store.pop() == nil) + try store.append(Data([3])) + try check(try store.pop() == Data([3])) + try store.close() + precondition(flock(markerFD, LOCK_EX | LOCK_NB) == 0, "close did not release its live lease") + try store.close() + precondition(store.snapshot.isClosed) + precondition(!FileManager.default.fileExists(atPath: directory.path)) + try expectFailure { try store.append(Data()) } + try expectFailure { _ = try store.pop() } + try expectFailure { try store.reset() } + } + + static func boundedMetadata(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 90_000, parentDirectory: root) + defer { try? store.close() } + for index in 0..<20_000 { try store.append(Data([UInt8(index % 251)])) } + precondition(store.snapshot.count == 20_000 && store.snapshot.chunkCount == 2) + // Exactly two chunk files, not 20,000 per-record files or resident record descriptors. + try check(try FileManager.default.contentsOfDirectory(atPath: store.storageDirectory.path).count == 3) + for index in 0..<20_000 { + try check(try store.pop() == Data([UInt8(index % 251)])) + if index == 9_999 { precondition(store.snapshot.chunkCount == 1) } + } + precondition(store.snapshot.chunkCount == 0 && store.snapshot.diskByteCount == 0) + } + + static func diskFailures(_ root: URL) throws { + let missingParent = root.appendingPathComponent("does-not-exist", isDirectory: true) + try expectFailure { _ = try SoftwarePacketDiskFIFO(parentDirectory: missingParent) } + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 16, parentDirectory: root) + defer { try? store.close() } + + // A removed temp directory must produce an explicit append failure, never accepted data. + try FileManager.default.removeItem(at: store.storageDirectory) + try expectFailure { try store.append(Data([1])) } + precondition(store.snapshot.hasFailure && store.snapshot.count == 0) + try expectFailure { _ = try store.pop() } + try store.reset() + precondition(!store.snapshot.hasFailure) + + // Failure while rolling over leaves prior accepted records visible in the failure + // snapshot, and blocks reads until the caller explicitly chooses to discard/reset. + try store.append(Data([9])) + let nextChunk = store.storageDirectory.appendingPathComponent("1.packets") + try Data([77]).write(to: nextChunk, options: .withoutOverwriting) + try expectFailure { try store.append(Data([10])) } + precondition(store.snapshot.count == 1 && store.snapshot.byteCount == 1) + try expectFailure { _ = try store.pop() } + try check(try Data(contentsOf: nextChunk) == Data([77])) + try store.reset() + try store.append(Data([2])) + let chunk = store.storageDirectory.appendingPathComponent("0.packets") + let handle = try FileHandle(forWritingTo: chunk) + try handle.truncate(atOffset: 8) + try handle.close() + try expectFailure { _ = try store.pop() } + precondition(store.snapshot.hasFailure && store.snapshot.count == 1, + "truncated record was silently lost") + try expectFailure { try store.append(Data([3])) } + try store.reset() + try store.append(Data([4])) + try check(try store.pop() == Data([4])) + + // Unexpected files are not overwritten. Reset removes only the owned session directory. + try Data([88]).write(to: chunk, options: .withoutOverwriting) + try expectFailure { try store.append(Data([5])) } + try check(try Data(contentsOf: chunk) == Data([88])) + try store.reset() + } + + static func concurrentAccess(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 128, parentDirectory: root) + defer { try? store.close() } + // Serial execution order across producers is deliberately unspecified; every record + // must survive exactly once. FIFO order itself is checked by the sequential tests. + DispatchQueue.concurrentPerform(iterations: 8) { producer in + for index in 0..<100 { + let value = Data("\(producer):\(index)".utf8) + do { try store.append(value) } catch { fatalError("concurrent append: \(error)") } + } + } + precondition(store.snapshot.count == 800) + var seen = Set() + while let data = try store.pop() { precondition(seen.insert(data).inserted) } + precondition(seen.count == 800 && store.snapshot.chunkCount == 0) + + let group = DispatchGroup() + group.enter() + DispatchQueue.global().async { + defer { group.leave() } + for index in 0..<2_000 { + do { try store.append(Data("serial:\(index)".utf8)) } + catch { fatalError("concurrent producer: \(error)") } + } + } + group.enter() + DispatchQueue.global().async { + defer { group.leave() } + var index = 0 + while index < 2_000 { + do { + if let record = try store.pop() { + precondition(record == Data("serial:\(index)".utf8), "concurrent FIFO reordered") + index += 1 + } else { + Thread.sleep(forTimeInterval: 0.0001) + } + } catch { fatalError("concurrent consumer: \(error)") } + } + } + precondition(group.wait(timeout: .now() + 10) == .success, "concurrent workers did not finish") + precondition(store.snapshot.count == 0 && store.snapshot.diskByteCount == 0) + } + + static func staleCleanup(_ root: URL) throws { + let testRoot = root.appendingPathComponent("sweep", isDirectory: true) + try FileManager.default.createDirectory(at: testRoot, withIntermediateDirectories: false) + defer { try? FileManager.default.removeItem(at: testRoot) } + let now = Date() + let old = now.addingTimeInterval(-90_000) + let stale = try makeOrphan(testRoot, modified: old) + let noMarker = try makeOrphan(testRoot, modified: old, marker: false) + let fresh = try makeOrphan(testRoot, modified: now) + let almostOld = try makeOrphan(testRoot, modified: now.addingTimeInterval(-86_399.5)) + let current = try makeOrphan(testRoot, modified: old) + let active = try SoftwarePacketDiskFIFO(parentDirectory: testRoot, now: now) + defer { try? active.close() } + // Init already swept eligible stale siblings, while fresh ones remain. + precondition(!FileManager.default.fileExists(atPath: stale.path)) + precondition(!FileManager.default.fileExists(atPath: noMarker.path)) + // current was not designated until the explicit sweep below, so recreate it. + try FileManager.default.createDirectory(at: current, withIntermediateDirectories: false) + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: current.path) + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: active.storageDirectory.path) + let foreign = testRoot.appendingPathComponent("foreign", isDirectory: true) + try FileManager.default.createDirectory(at: foreign, withIntermediateDirectories: false) + let sentinel = foreign.appendingPathComponent("keep") + try Data([99]).write(to: sentinel) + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: foreign.path) + let symlink = testRoot.appendingPathComponent("\(SoftwarePacketDiskFIFO.directoryPrefix)\(UUID().uuidString)") + try FileManager.default.createSymbolicLink(at: symlink, withDestinationURL: foreign) + let matchingFile = testRoot.appendingPathComponent("\(SoftwarePacketDiskFIFO.directoryPrefix)\(UUID().uuidString)") + try Data([33]).write(to: matchingFile) + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: matchingFile.path) + let markerSymlink = try makeOrphan(testRoot, modified: old, marker: false) + try FileManager.default.createSymbolicLink(at: markerSymlink.appendingPathComponent("session.lock"), + withDestinationURL: sentinel) + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: markerSymlink.path) + let removable = try makeOrphan(testRoot, modified: old) + + let result = SoftwarePacketDiskFIFO.sweepStaleSessionDirs(parentDirectory: testRoot, + currentSession: current.lastPathComponent, now: now) + precondition(result.removedCount == 1 && result.inspectedCount <= 64) + precondition(result.failureCount == 1, "symlink marker should fail closed") + precondition(!FileManager.default.fileExists(atPath: removable.path)) + for kept in [fresh, almostOld, current, active.storageDirectory, foreign, symlink, matchingFile, markerSymlink] { + precondition(FileManager.default.fileExists(atPath: kept.path), "sweep removed protected \(kept.lastPathComponent)") + } + try check(try Data(contentsOf: sentinel) == Data([99])) + try active.append(Data([4])) + try active.reset() + try FileManager.default.setAttributes([.modificationDate: old], ofItemAtPath: active.storageDirectory.path) + _ = SoftwarePacketDiskFIFO.sweepStaleSessionDirs(parentDirectory: testRoot, now: now) + precondition(FileManager.default.fileExists(atPath: active.storageDirectory.path), + "old live session lost protection after reset") + + let boundedRoot = testRoot.appendingPathComponent("bounded", isDirectory: true) + try FileManager.default.createDirectory(at: boundedRoot, withIntermediateDirectories: false) + for _ in 0..<6 { _ = try makeOrphan(boundedRoot, modified: old) } + let limited = SoftwarePacketDiskFIFO.sweepStaleSessionDirs(parentDirectory: boundedRoot, + now: now, maxEntries: 2, maxRemovals: 1) + precondition(limited.inspectedCount <= 2 && limited.removedCount == 1) + try check(try FileManager.default.contentsOfDirectory(atPath: boundedRoot.path).count == 5) + } + + static func retainedSeeks(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 30, retainConsumed: true, + parentDirectory: root) + defer { try? store.close() } + let records = (0..<13).map { Data(repeating: UInt8($0), count: 2) } + var cursors: [SoftwarePacketDiskFIFO.Cursor] = [] + for record in records.prefix(9) { cursors.append(try store.append(record)) } + precondition(store.snapshot.residentByteCount == 90 && store.snapshot.chunkCount == 3) + try check(try store.pop() == records[0]) + try check(try store.pop() == records[1]) + try store.restore(to: cursors[5]) // Forward seek within already resident packets. + precondition(store.snapshot.count == 4 && store.snapshot.byteCount == 8) + precondition(store.snapshot.residentByteCount == 90, "cached seek discarded the forward cache") + try check(try store.pop() == records[5]) + try check(try store.pop() == records[6]) + try store.restore(to: cursors[1]) // Backward seek to retained history in the first chunk. + precondition(store.snapshot.count == 8 && store.snapshot.byteCount == 16) + cursors.append(try store.append(records[9])) // Writer extends while reader replays history. + for record in records[1...9] { try check(try store.pop() == record) } + try check(try store.pop() == nil) + precondition(store.snapshot.count == 0 && store.snapshot.byteCount == 0) + precondition(store.snapshot.residentByteCount == 100 && store.snapshot.chunkCount == 4) + + // A drained retained tail stays appendable, both within its current chunk and after roll. + cursors.append(try store.append(records[10])) + try check(try store.pop() == records[10]) + cursors.append(try store.append(records[11])) + cursors.append(try store.append(records[12])) + try check(try store.pop() == records[11]) + try check(try store.pop() == records[12]) + let oversized = Data(repeating: 88, count: 100) + let oversizedCursor = try store.append(oversized) // Rolls while consumer is at prior EOF. + try check(try store.pop() == oversized) + try store.restore(to: cursors[0]) + for record in records { try check(try store.pop() == record) } + try check(try store.pop() == oversized) + try check(try store.pop() == nil) + try store.restore(to: oversizedCursor) + precondition(store.snapshot.count == 1 && store.snapshot.byteCount == 100) + try check(try store.pop() == oversized) + let empty = try store.append(Data()) + try check(try store.pop() == Data()) + try store.restore(to: empty) + precondition(store.snapshot.count == 1 && store.snapshot.byteCount == 0) + try check(try store.pop() == Data()) + } + + static func retainedEvictionAndReset(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 20, retainConsumed: true, + parentDirectory: root) + defer { try? store.close() } + let records = (0..<8).map { Data(repeating: UInt8($0), count: 2) } + let cursors = try records.map { try store.append($0) } + try check(try store.pop() == records[0]) + try check(try store.pop() == records[1]) + try store.trimConsumed(toByteBudget: 0) + precondition(store.snapshot.residentByteCount == 60 && store.snapshot.oldestRetainedChunkID == 1) + precondition(store.snapshot.count == 6 && store.snapshot.byteCount == 12, + "budget pressure discarded unread packets") + try expectFailure { try store.restore(to: cursors[0]) } + precondition(!store.snapshot.hasFailure, "evicted-token rejection poisoned a healthy cache") + try store.restore(to: cursors[2]) + try store.trimConsumed(toByteBudget: 0) + precondition(store.snapshot.residentByteCount == 60, "trim removed the reader or unread chunks") + try store.restore(to: cursors[6]) + try store.trimConsumed(toByteBudget: 0) + precondition(store.snapshot.residentByteCount == 20 && store.snapshot.oldestRetainedChunkID == 3) + try expectFailure { try store.restore(to: cursors[2]) } + try store.restore(to: cursors[7]) + try check(try store.pop() == records[7]) + try store.trimConsumed(toByteBudget: 0) + precondition(store.snapshot.count == 0 && store.snapshot.residentByteCount == 20, + "trim deleted the active writer tail") + + try store.reset() + precondition(store.snapshot.oldestRetainedChunkID == nil && store.snapshot.residentByteCount == 0) + try check(try FileManager.default.contentsOfDirectory(atPath: store.storageDirectory.path) == ["session.lock"]) + let replacement = try store.append(Data([22])) + try expectFailure { try store.restore(to: cursors[0]) } + precondition(!store.snapshot.hasFailure, "old reset generation poisoned the replacement cache") + try store.restore(to: replacement) + try check(try store.pop() == Data([22])) + let other = try SoftwarePacketDiskFIFO(retainConsumed: true, parentDirectory: root) + defer { try? other.close() } + let foreign = try other.append(Data([33])) + try expectFailure { try store.restore(to: foreign) } + let normal = try SoftwarePacketDiskFIFO(parentDirectory: root) + defer { try? normal.close() } + let normalCursor = try normal.append(Data([44])) + try expectFailure { try normal.restore(to: normalCursor) } + try check(try normal.pop() == Data([44])) + + // Reset must delete consumed history too, not only the consumer-to-writer suffix. + for record in records { try store.append(record) } + while try store.pop() != nil {} + precondition(store.snapshot.chunkCount > 1) + try store.reset() + try check(try FileManager.default.contentsOfDirectory(atPath: store.storageDirectory.path) == ["session.lock"]) + } + + static func concurrentRetainedReplay(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 128, retainConsumed: true, + parentDirectory: root) + defer { try? store.close() } + var selected: SoftwarePacketDiskFIFO.Cursor? + for index in 0..<100 { + let cursor = try store.append(Data("retained:\(index)".utf8)) + if index == 50 { selected = cursor } + } + for index in 0..<100 { try check(try store.pop() == Data("retained:\(index)".utf8)) } + try store.restore(to: selected!) + let group = DispatchGroup() + group.enter() + DispatchQueue.global().async { + defer { group.leave() } + for index in 100..<500 { + do { try store.append(Data("retained:\(index)".utf8)) } + catch { fatalError("retained producer: \(error)") } + } + } + group.enter() + DispatchQueue.global().async { + defer { group.leave() } + var index = 50 + while index < 500 { + do { + if let record = try store.pop() { + precondition(record == Data("retained:\(index)".utf8), "writer overwrote replay history") + index += 1 + } else { Thread.sleep(forTimeInterval: 0.0001) } + } catch { fatalError("retained consumer: \(error)") } + } + } + precondition(group.wait(timeout: .now() + 10) == .success) + try store.restore(to: selected!) + precondition(store.snapshot.count == 450) + for index in 50..<500 { try check(try store.pop() == Data("retained:\(index)".utf8)) } + precondition(store.snapshot.count == 0 && store.snapshot.residentByteCount > 0) + } + + static func retainedTrimRestoreAndRollover(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 30, retainConsumed: true, + parentDirectory: root) + defer { try? store.close() } + let records = (0..<12).map { Data(repeating: UInt8($0), count: 2) } + var cursors = try records.prefix(6).map { try store.append($0) } + for index in 0..<3 { try check(try store.pop() == records[index]) } + // The budget is inclusive. A caller needing producer headroom must explicitly ask for it. + try store.trimConsumed(toByteBudget: 60) + precondition(store.snapshot.residentByteCount == 60 && store.snapshot.oldestRetainedChunkID == 0) + try store.trimConsumed(toByteBudget: 59) + precondition(store.snapshot.residentByteCount == 30 && store.snapshot.oldestRetainedChunkID == 1) + try expectFailure { try store.restore(to: cursors[0]) } + try store.restore(to: cursors[4]) + for record in records[6...9] { cursors.append(try store.append(record)) } + precondition(store.snapshot.residentByteCount == 70) + try check(try store.pop() == records[4]) + try check(try store.pop() == records[5]) + try store.trimConsumed(toByteBudget: 60) + precondition(store.snapshot.residentByteCount == 40 && store.snapshot.oldestRetainedChunkID == 2) + try expectFailure { try store.restore(to: cursors[3]) } + try store.restore(to: cursors[6]) + try store.append(records[10]) + try store.append(records[11]) + for index in 6..<12 { try check(try store.pop() == records[index]) } + try store.trimConsumed(toByteBudget: 59) + precondition(store.snapshot.count == 0 && store.snapshot.residentByteCount == 30) + + // ReadAhead calls trim unconditionally. Legacy/destructive stores must treat it as a + // no-op, leaving both the unread records and existing automatic reclamation unchanged. + let legacy = try SoftwarePacketDiskFIFO(chunkTargetBytes: 30, parentDirectory: root) + defer { try? legacy.close() } + for record in records.prefix(6) { try legacy.append(record) } + try legacy.trimConsumed(toByteBudget: 0) + precondition(legacy.snapshot.count == 6 && legacy.snapshot.residentByteCount == 60) + for index in 0..<6 { + try check(try legacy.pop() == records[index]) + try legacy.trimConsumed(toByteBudget: 0) + } + precondition(legacy.snapshot.count == 0 && legacy.snapshot.residentByteCount == 0) + } + + static func retainedModelSequence(_ root: URL) throws { + let store = try SoftwarePacketDiskFIFO(chunkTargetBytes: 96, retainConsumed: true, + parentDirectory: root) + defer { try? store.close() } + var data: [Data] = [] + var cursors: [SoftwarePacketDiskFIFO.Cursor] = [] + var readIndex = 0 + var seed: UInt64 = 0xCACE_2026 + func next() -> UInt64 { + seed = seed &* 6_364_136_223_846_793_005 &+ 1 + return seed + } + // Deterministic interleaving checks the aggregate accounting against an independent + // in-memory reference, including over-sized chunks, replay, eviction and repeated reset. + for _ in 0..<2_000 { + let action = next() % 100 + if action < 45 { + let payload = Data(repeating: UInt8(next() % 251), count: Int(next() % 129)) + cursors.append(try store.append(payload)) + data.append(payload) + } else if action < 70 { + let expected: Data? = readIndex < data.count ? data[readIndex] : nil + try check(try store.pop() == expected) + if expected != nil { readIndex += 1 } + } else if action < 85, !cursors.isEmpty { + let target = Int(next() % UInt64(cursors.count)) + if let floor = store.snapshot.oldestRetainedChunkID, cursors[target].chunkID >= floor { + try store.restore(to: cursors[target]) + readIndex = target + } else { + try expectFailure { try store.restore(to: cursors[target]) } + precondition(!store.snapshot.hasFailure) + } + } else if action < 95 { + try store.trimConsumed(toByteBudget: Int(next() % 512)) + } else { + try store.reset() + data.removeAll(keepingCapacity: true) + cursors.removeAll(keepingCapacity: true) + readIndex = 0 + } + let snapshot = store.snapshot + precondition(snapshot.count == data.count - readIndex) + precondition(snapshot.byteCount == data.dropFirst(readIndex).reduce(0) { $0 + $1.count }) + // Keep the model calculation explicit: cursor indices and chunk IDs use different axes. + let retainedIndices = cursors.indices.filter { index in + guard let floor = snapshot.oldestRetainedChunkID else { return false } + return cursors[index].chunkID >= floor + } + precondition(snapshot.residentByteCount == retainedIndices.reduce(0) { $0 + data[$1].count + 8 }) + precondition(snapshot.chunkCount == Set(retainedIndices.map { cursors[$0].chunkID }).count) + } + } + + static func makeOrphan(_ root: URL, modified: Date, marker: Bool = true) throws -> URL { + let directory = root.appendingPathComponent("\(SoftwarePacketDiskFIFO.directoryPrefix)\(UUID().uuidString)", + isDirectory: true) + try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: false) + try Data([1, 2, 3]).write(to: directory.appendingPathComponent("0.packets")) + if marker { try Data().write(to: directory.appendingPathComponent("session.lock")) } + try FileManager.default.setAttributes([.modificationDate: modified], ofItemAtPath: directory.path) + return directory + } + + static func interruptedInitialization(_ root: URL) throws { + let before = Set(try FileManager.default.contentsOfDirectory(atPath: root.path)) + try expectFailure { + _ = try SoftwarePacketDiskFIFO(parentDirectory: root, leaseAcquisition: { directory in + precondition(FileManager.default.fileExists(atPath: directory.path)) + try Data([1]).write(to: directory.appendingPathComponent("session.lock")) + throw SoftwarePacketDiskFIFO.Failure.sessionLeaseLost + }) + } + let after = Set(try FileManager.default.contentsOfDirectory(atPath: root.path)) + precondition(before == after, "failed initialization leaked its owned directory") + } + + static func expectFailure(_ operation: () throws -> Void) throws { + do { + try operation() + } catch { + return + } + preconditionFailure("expected an explicit failure") + } + + static func check(_ condition: @autoclosure () throws -> Bool, + _ message: String = "assertion failed") throws { + let result = try condition() + precondition(result, message) + } +} diff --git a/Scripts/tests/SoftwarePacketReadAheadStandalone.swift b/Scripts/tests/SoftwarePacketReadAheadStandalone.swift new file mode 100644 index 000000000..233a836d2 --- /dev/null +++ b/Scripts/tests/SoftwarePacketReadAheadStandalone.swift @@ -0,0 +1,629 @@ +import Foundation + +private final class Locked: @unchecked Sendable { + private let lock = NSLock() + private var value: Value + init(_ value: Value) { self.value = value } + func withValue(_ body: (inout Value) throws -> Result) rethrows -> Result { + lock.lock(); defer { lock.unlock() } + return try body(&value) + } +} + +private final class PendingRead: @unchecked Sendable { + let completed = DispatchSemaphore(value: 0) + private let result = Locked?>(nil) + init(_ source: SoftwarePacketReadAhead, isCurrent: @escaping @Sendable () -> Bool = { true }) { + DispatchQueue.global().async { + let answer = Result { try source.read(isCurrent: isCurrent) } + self.result.withValue { $0 = answer } + self.completed.signal() + } + } + func finish() -> Result { + requireSignal(completed, "consumer read did not complete") + return result.withValue { $0! } + } +} + +private func requireSignal(_ semaphore: DispatchSemaphore, _ message: String) { + precondition(semaphore.wait(timeout: .now() + 5) == .success, message) +} + +private func eventually(_ message: String, _ predicate: () -> Bool) { + let deadline = Date().addingTimeInterval(5) + while !predicate() { + precondition(Date() < deadline, message) + Thread.sleep(forTimeInterval: 0.001) + } +} + +private func check(_ condition: @autoclosure () throws -> Bool, + _ message: String = "assertion failed") rethrows { + let result = try condition() + precondition(result, message) +} + +@main +struct SoftwarePacketReadAheadTests { + static let video = SoftwarePacketReadAhead.Stream(index: 0, numerator: 1, denominator: 1) + static let audio = SoftwarePacketReadAhead.Stream(index: 1, numerator: 1, denominator: 1) + enum SourceFailure: Error { case rejected, deliberate, retainedBudgetBoundary } + + static func packet(_ pts: Int64, stream: Int32 = 0, duration: Int64 = 1, + marker: UInt8 = 7, payloadSize: Int = 8, flags: Int32 = 0x21) -> SoftwareStoredPacket { + SoftwareStoredPacket(pts: pts, dts: pts - 2, duration: duration, + position: 123_456 + pts, streamIndex: stream, flags: flags, + timeBaseNumerator: 1, timeBaseDenominator: 1, + bytes: Data(repeating: marker, count: payloadSize), + sideData: [.init(type: 0, bytes: Data([0, 1, 0, 255])), + .init(type: 70, bytes: Data()), .init(type: UInt32.max, bytes: Data([9]))]) + } + + static func make(_ root: URL, audio: SoftwarePacketReadAhead.Stream? = nil, + budget: Int = 1_000_000, seconds: Double = 100, clock: Double = 0, + retainConsumed: Bool = false, chunkTargetBytes: Int = 1024, + videoReorderDepth: Int? = nil, + beforeConsumerOperation: (@Sendable () -> Void)? = nil, + read: @escaping @Sendable (@Sendable () -> Bool) throws -> SoftwareStoredPacket?) throws + -> (SoftwarePacketReadAhead, SoftwarePacketDiskFIFO) { + let fifo = try SoftwarePacketDiskFIFO(chunkTargetBytes: chunkTargetBytes, + retainConsumed: retainConsumed, parentDirectory: root) + let source = SoftwarePacketReadAhead(video: video, audio: audio, + byteBudget: budget, forwardSeconds: seconds, initialSourceClock: clock, fifo: fifo, + videoReorderDepth: videoReorderDepth, + beforeConsumerOperation: beforeConsumerOperation, readSource: read) + return (source, fifo) + } + + static func close(_ source: SoftwarePacketReadAhead, _ fifo: SoftwarePacketDiskFIFO) { + source.close() + eventually("worker did not close its FIFO") { fifo.snapshot.isClosed } + precondition(!FileManager.default.fileExists(atPath: fifo.storageDirectory.path)) + } + + static func main() throws { + let root = FileManager.default.temporaryDirectory.appendingPathComponent( + "aether-packet-read-ahead-tests-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: root, withIntermediateDirectories: false) + defer { try? FileManager.default.removeItem(at: root) } + try roundTripAndEOF(root) + try queuedSeekReset(root) + try blockedReadAndRapidSeeks(root) + try sourceLockAdmission(root) + try closeUnblocksConsumer(root) + try byteBound(root) + try exactResidentBudgetBoundary(root) + try timeBoundAndPlayhead(root) + try selectedAVCoverage(root) + try delayedOldConsumer(root) + try explicitFailure(root) + try closeBeforeStart(root) + try retainedForwardAndBackwardSeek(root) + try retainedCacheMissAndRapidSeeks(root) + try retainedSeekPreservesInFlightProducer(root) + try retainedSeekRetiresOldConsumer(root) + try expiredKeyframeCannotRestore(root) + try staleHostAdmission(root) + try successorCoverageIntegration(root) + try check(FileManager.default.contentsOfDirectory(atPath: root.path).isEmpty) + print("PASS: packet metadata/side-data, EOF/errors, seek reset/rapid seeks, source-lock admission, close wake, byte/time bounds, exact resident-budget refill, A/V gaps, delayed old consumer; retained forward/backward seeks, cache miss/eviction, in-flight producer, stale host admission, successor coverage") + } + + static func roundTripAndEOF(_ root: URL) throws { + let unknownTiming = SoftwareStoredPacket(pts: Int64.min, dts: Int64.max, duration: 0, + position: -1, streamIndex: -1, flags: Int32.min, + timeBaseNumerator: 1001, timeBaseDenominator: 30_000, bytes: Data(), + sideData: [.init(type: UInt32.max, bytes: Data([0, 0, 255]))]) + let packets = [packet(0), packet(1, marker: 8), packet(2, stream: 1, marker: 9), unknownTiming] + for packet in packets { try check(SoftwareStoredPacket.decode(packet.encoded()) == packet) } + let index = Locked(0) + let (source, fifo) = try make(root) { accepts in + try index.withValue { current in + guard accepts() else { throw SourceFailure.rejected } + guard current < packets.count else { return nil } + defer { current += 1 } + return packets[current] + } + } + source.start() + for packet in packets { try check(PendingRead(source).finish().get() == packet) } + try check(PendingRead(source).finish().get() == nil) + precondition(source.snapshot.sourceEnded && source.snapshot.packetCount == 0) + precondition(source.snapshot.bytes == 0) + close(source, fifo) + } + + static func queuedSeekReset(_ root: URL) throws { + let next = Locked(packet(0, marker: 11)) + let (source, fifo) = try make(root, budget: 1) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + return value + } + } + source.start() + eventually("old packet did not queue") { source.snapshot.packetCount == 1 } + let token = source.beginSeek() + precondition(source.snapshot.frontier == nil && source.snapshot.packetCount == 0) + let expected = packet(10, marker: 22) + next.withValue { $0 = expected } + source.endSeek(token, sourceClock: 10) + try check(PendingRead(source).finish().get() == expected) + close(source, fifo) + } + + static func blockedReadAndRapidSeeks(_ root: URL) throws { + let entered = DispatchSemaphore(value: 0) + let release = DispatchSemaphore(value: 0) + let calls = Locked(0) + let old = packet(0, marker: 31) + let fresh = packet(20, marker: 32) + let (source, fifo) = try make(root, budget: 1) { accepts in + let call = try calls.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + defer { value += 1 }; return value + } + if call == 0 { + // A source read has returned its old packet under the source lock, but its + // caller has not yet handed it back to the producer for publication. + entered.signal(); requireSignal(release, "blocked source was not released") + return old + } + return fresh + } + source.start() + requireSignal(entered, "source did not enter old read") + let first = source.beginSeek() + let second = source.beginSeek() + source.endSeek(first, sourceClock: 10) + precondition(source.snapshot.seeking, "older seek cleared newer hold") + source.endSeek(second, sourceClock: 20) + release.signal() + try check(PendingRead(source).finish().get() == fresh, "old blocked read escaped") + precondition(source.snapshot.generation == second) + close(source, fifo) + } + + static func sourceLockAdmission(_ root: URL) throws { + let beforeLock = DispatchSemaphore(value: 0) + let resume = DispatchSemaphore(value: 0) + let attempts = Locked(0) + let admitted = Locked(0) + let sourceReadLock = NSLock() + let expected = packet(40, marker: 41) + let (source, fifo) = try make(root, budget: 1) { accepts in + let call = attempts.withValue { value in defer { value += 1 }; return value } + if call == 0 { beforeLock.signal(); requireSignal(resume, "admission was not resumed") } + sourceReadLock.lock(); defer { sourceReadLock.unlock() } + guard accepts() else { throw SourceFailure.rejected } + admitted.withValue { $0 += 1 } + return expected + } + source.start() + requireSignal(beforeLock, "source did not pause before serialization lock") + let token = source.beginSeek() + sourceReadLock.lock() // mirrors the real Demuxer seek's serialization boundary + sourceReadLock.unlock() + source.endSeek(token, sourceClock: 40) + resume.signal() + eventually("fresh generation did not queue") { source.snapshot.packetCount == 1 } + precondition(attempts.withValue { $0 } == 2) + precondition(admitted.withValue { $0 } == 1, "stale invocation advanced post-seek source") + try check(PendingRead(source).finish().get() == expected) + close(source, fifo) + } + + static func closeUnblocksConsumer(_ root: URL) throws { + let entered = DispatchSemaphore(value: 0) + let release = DispatchSemaphore(value: 0) + let (source, fifo) = try make(root) { accepts in + guard accepts() else { throw SourceFailure.rejected } + entered.signal(); requireSignal(release, "closed source was not released") + return packet(0) + } + source.start() + requireSignal(entered, "source was not blocked") + let pending = PendingRead(source) + source.close() + switch pending.finish() { + case .failure(SoftwarePacketReadAhead.ReadError.closed): break + default: preconditionFailure("close did not wake consumer with explicit closed error") + } + precondition(source.snapshot.frontier == nil && source.snapshot.packetCount == 0) + release.signal() + close(source, fifo) + } + + static func byteBound(_ root: URL) throws { + let calls = Locked(0) + let secondCall = DispatchSemaphore(value: 0) + let sample = packet(0, payloadSize: 4096) + let (source, fifo) = try make(root, budget: 1) { accepts in + try calls.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + value += 1 + if value == 2 { secondCall.signal() } + return sample + } + } + source.start() + eventually("byte-bound packet did not arrive") { source.snapshot.packetCount == 1 } + try check(source.snapshot.bytes == sample.encoded().count) + precondition(secondCall.wait(timeout: .now() + 0.1) == .timedOut, + "prefetch exceeded one-packet byte-budget overshoot while consumer paused") + try check(PendingRead(source).finish().get() == sample) + requireSignal(secondCall, "consumer drain did not release byte backpressure") + close(source, fifo) + } + + static func timeBoundAndPlayhead(_ root: URL) throws { + let next = Locked(Int64(0)) + let thirdCall = DispatchSemaphore(value: 0) + let (source, fifo) = try make(root, seconds: 2) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + if value == 2 { thirdCall.signal() } + defer { value += 1 }; return packet(value) + } + } + source.start() + eventually("time-bound coverage did not fill") { source.snapshot.frontier == 2 } + precondition(source.snapshot.packetCount == 2) + precondition(thirdCall.wait(timeout: .now() + 0.1) == .timedOut, + "prefetch exceeded time target while consumer/playhead paused") + source.updatePlayhead(1) + requireSignal(thirdCall, "playhead advance did not release time backpressure") + eventually("new time target did not fill") { source.snapshot.frontier == 3 } + close(source, fifo) + } + + static func exactResidentBudgetBoundary(_ root: URL) throws { + let sample = packet(0) + let recordBytes = try sample.encoded().count + 8 // the FIFO length prefix is resident too + let exactBudget = recordBytes * 4 + let calls = Locked(0) + let refill = DispatchSemaphore(value: 0) + let (source, fifo) = try make(root, budget: exactBudget, retainConsumed: true, + chunkTargetBytes: recordBytes) { accepts in + try calls.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + value += 1 + if value == 5 { refill.signal() } + return sample + } + } + source.start() + eventually("exact-budget fixture did not park at four resident records") { + source.snapshot.packetCount == 4 && source.snapshot.residentBytes == exactBudget + } + precondition(calls.withValue { $0 } == 4) + // One complete consumed chunk is now evictable while THREE unread packets remain. + // Refill must resume here, not wait until the consumer drains the whole queue. + try check(PendingRead(source).finish().get() == sample) + if refill.wait(timeout: .now() + 2) != .success { + let state = source.snapshot + close(source, fifo) + FileHandle.standardError.write(Data( + "FAIL: exact resident budget stranded producer with \(state.packetCount) unread packets and \(state.residentBytes)/\(exactBudget) resident bytes\n".utf8)) + throw SourceFailure.retainedBudgetBoundary + } + eventually("exact-budget refill did not restore four forward packets") { + source.snapshot.packetCount == 4 + } + precondition(source.snapshot.residentBytes == exactBudget) + precondition((fifo.snapshot.oldestRetainedChunkID ?? 0) > 0, + "refill was funded without removing the consumed history chunk") + close(source, fifo) + } + + static func selectedAVCoverage(_ root: URL) throws { + // Decode-order B-picture gap and slower selected audio each limit the frontier. + let packets = [packet(0), packet(2), packet(0, stream: 1), packet(1), packet(2, stream: 1)] + let index = Locked(0) + let (source, fifo) = try make(root, audio: audio) { accepts in + try index.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + guard value < packets.count else { return nil } + defer { value += 1 }; return packets[value] + } + } + source.start() + eventually("A/V test source did not end") { source.snapshot.sourceEnded } + precondition(source.snapshot.frontier == 1, "missing audio interval was bridged") + source.updatePlayhead(1) + precondition(source.snapshot.frontier == nil, "clock in selected-audio gap got false coverage") + source.updatePlayhead(2) + precondition(source.snapshot.frontier == 3) + close(source, fifo) + } + + static func delayedOldConsumer(_ root: URL) throws { + let entered = DispatchSemaphore(value: 0) + let release = DispatchSemaphore(value: 0) + let hookCalls = Locked(0) + let next = Locked(packet(0, marker: 51)) + let (source, fifo) = try make(root, budget: 1, beforeConsumerOperation: { + let first = hookCalls.withValue { value in defer { value += 1 }; return value == 0 } + if first { entered.signal(); requireSignal(release, "old consumer was not released") } + }) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected }; return value + } + } + source.start() + eventually("old consumer fixture did not queue") { source.snapshot.packetCount == 1 } + let oldConsumer = PendingRead(source) + requireSignal(entered, "old consumer did not pause before operations lock") + let token = source.beginSeek() + let expected = packet(60, marker: 61) + next.withValue { $0 = expected } + source.endSeek(token, sourceClock: 60) + eventually("new packet did not arrive after delayed reset") { + source.snapshot.packetCount == 1 && source.snapshot.frontier == 61 + } + release.signal() + switch oldConsumer.finish() { + case .failure(SoftwarePacketReadAhead.ReadError.interrupted): break + default: preconditionFailure("old consumer escaped its generation") + } + precondition(source.snapshot.packetCount == 1, "old consumer consumed the new generation") + try check(PendingRead(source).finish().get() == expected) + close(source, fifo) + } + + static func explicitFailure(_ root: URL) throws { + let (source, fifo) = try make(root) { accepts in + guard accepts() else { throw SourceFailure.rejected } + throw SourceFailure.deliberate + } + source.start() + switch PendingRead(source).finish() { + case .failure(SourceFailure.deliberate): break + default: preconditionFailure("source error was turned into EOF") + } + precondition(!source.snapshot.sourceEnded) + close(source, fifo) + } + + static func closeBeforeStart(_ root: URL) throws { + let (source, fifo) = try make(root) { _ in preconditionFailure("closed source started") } + close(source, fifo) + source.start() + precondition(fifo.snapshot.isClosed) + } + + static func keyPacket(_ pts: Int64) -> SoftwareStoredPacket { + packet(pts, marker: UInt8(pts % 251), flags: pts % 10 == 0 ? 1 : 0) + } + + static func retainedForwardAndBackwardSeek(_ root: URL) throws { + let state = Locked((next: Int64(0), reads: 0)) + let (source, fifo) = try make(root, retainConsumed: true) { accepts in + try state.withValue { state in + guard accepts() else { throw SourceFailure.rejected } + state.reads += 1 + guard state.next <= 30 else { return nil } + defer { state.next += 1 }; return keyPacket(state.next) + } + } + source.start() + eventually("retained source did not fill") { source.snapshot.sourceEnded } + let originalReads = state.withValue { $0.reads } + let originalEpoch = source.snapshot.sourceEpoch + let originalResident = source.snapshot.residentBytes + precondition(originalResident > 0 && source.snapshot.frontier == 31) + for pts: Int64 in 0...10 { try check(PendingRead(source).finish().get() == keyPacket(pts)) } + source.updatePlayhead(10) + + for (hit, target): (Int, Double) in [(1, 20), (2, 10), (3, 20)] { + let token = source.beginSeek(to: target) + precondition(source.snapshot.residentBytes == originalResident, + "begin cached seek discarded resident storage") + try check(source.prepareSeek(token, to: target), "covered target was treated as source seek") + source.endSeek(token, sourceClock: target) + precondition(source.snapshot.sourceEpoch == originalEpoch) + precondition(source.snapshot.cacheSeekHits == hit) + precondition(source.snapshot.frontier == 31, "cached seek collapsed the forward frontier") + let first = try PendingRead(source).finish().get()! + precondition(first.flags & 1 != 0 && Double(first.pts) <= target, + "cache replay did not start at an earlier retained key packet") + precondition(first == keyPacket(first.pts), "cache replay changed packet metadata") + try check(PendingRead(source).finish().get() == keyPacket(first.pts + 1)) + precondition(state.withValue { $0.reads } == originalReads, + "cached seek read the original source again") + } + close(source, fifo) + } + + static func retainedCacheMissAndRapidSeeks(_ root: URL) throws { + let state = Locked((next: Int64(0), limit: Int64(30))) + let (source, fifo) = try make(root, retainConsumed: true) { accepts in + try state.withValue { state in + guard accepts() else { throw SourceFailure.rejected } + guard state.next <= state.limit else { return nil } + defer { state.next += 1 }; return keyPacket(state.next) + } + } + source.start() + eventually("rapid-seek fixture did not fill") { source.snapshot.sourceEnded } + let initialEpoch = source.snapshot.sourceEpoch + let old = source.beginSeek(to: 20) + try check(source.prepareSeek(old, to: 20)) + let current = source.beginSeek(to: 100) + // An already superseded prepare must neither restore an old cursor nor reset the new aim. + do { + let restored = try source.prepareSeek(old, to: 20) + precondition(!restored, "superseded prepare restored an old cache target") + } catch SoftwarePacketReadAhead.ReadError.interrupted { } + source.endSeek(old, sourceClock: 20) + precondition(source.snapshot.seeking && source.snapshot.generation == current) + precondition(source.snapshot.sourceEpoch == initialEpoch) + try check(!source.prepareSeek(current, to: 100), "uncached target claimed a cache hit") + precondition(source.snapshot.sourceEpoch == initialEpoch + 1) + precondition(source.snapshot.residentBytes == 0 && source.snapshot.packetCount == 0) + precondition(source.snapshot.cacheSeekMisses == 1) + state.withValue { $0 = (100, 105) } // represents the caller's serialized real source seek + source.endSeek(current, sourceClock: 100) + try check(PendingRead(source).finish().get() == keyPacket(100)) + eventually("post-miss fixture did not finish") { source.snapshot.sourceEnded } + precondition(source.snapshot.frontier == 106) + close(source, fifo) + } + + static func retainedSeekPreservesInFlightProducer(_ root: URL) throws { + let entered = DispatchSemaphore(value: 0) + let release = DispatchSemaphore(value: 0) + let state = Locked(Int64(0)) + let (source, fifo) = try make(root, retainConsumed: true) { accepts in + let pts = try state.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + defer { value += 1 }; return value + } + if pts == 31 { + entered.signal() + requireSignal(release, "cached seek did not release in-flight producer") + } + return pts <= 31 ? keyPacket(pts) : nil + } + source.start() + requireSignal(entered, "producer did not pause after retained prefix") + precondition(source.snapshot.frontier == 31) + let epoch = source.snapshot.sourceEpoch + let token = source.beginSeek(to: 20) + try check(source.prepareSeek(token, to: 20), "blocked source prevented an available cache hit") + source.endSeek(token, sourceClock: 20) + precondition(source.snapshot.sourceEpoch == epoch) + release.signal() + eventually("same-epoch producer did not publish its tail") { source.snapshot.sourceEnded } + precondition(source.snapshot.frontier == 32, "cached seek discarded an in-flight producer packet") + let first = try PendingRead(source).finish().get()! + precondition(first.flags & 1 != 0 && first.pts <= 20) + if first.pts < 31 { + for pts in (first.pts + 1)...31 { try check(PendingRead(source).finish().get() == keyPacket(pts)) } + } + try check(PendingRead(source).finish().get() == nil) + close(source, fifo) + } + + static func expiredKeyframeCannotRestore(_ root: URL) throws { + let next = Locked(Int64(0)) + let (source, fifo) = try make(root, budget: 1400, retainConsumed: true, + chunkTargetBytes: 512) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + guard value <= 40 else { return nil } + defer { value += 1 }; return keyPacket(value) + } + } + source.start() + for pts: Int64 in 0...30 { + try check(PendingRead(source).finish().get() == keyPacket(pts)) + source.updatePlayhead(Double(pts)) + } + eventually("consumed history was not trimmed") { + (fifo.snapshot.oldestRetainedChunkID ?? 0) > 0 + } + let token = source.beginSeek(to: 0) + try check(!source.prepareSeek(token, to: 0), "evicted keyframe bookmark remained seekable") + precondition(source.snapshot.cacheSeekMisses == 1) + precondition(source.snapshot.residentBytes == 0) + // Leave the miss held; no source reposition is performed by this storage-lifetime test. + close(source, fifo) + } + + static func retainedSeekRetiresOldConsumer(_ root: URL) throws { + let entered = DispatchSemaphore(value: 0) + let release = DispatchSemaphore(value: 0) + let hookCalls = Locked(0) + let next = Locked(Int64(0)) + let (source, fifo) = try make(root, retainConsumed: true, beforeConsumerOperation: { + let first = hookCalls.withValue { value in defer { value += 1 }; return value == 0 } + if first { entered.signal(); requireSignal(release, "old cached consumer was not released") } + }) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + guard value <= 30 else { return nil } + defer { value += 1 }; return keyPacket(value) + } + } + source.start() + eventually("cached old-consumer fixture did not finish") { source.snapshot.sourceEnded } + let sourceEpoch = source.snapshot.sourceEpoch + let oldConsumer = PendingRead(source) + requireSignal(entered, "old cached consumer did not pause before cursor operation") + let token = source.beginSeek(to: 20) + try check(source.prepareSeek(token, to: 20)) + source.endSeek(token, sourceClock: 20) + let restoredCount = source.snapshot.packetCount + release.signal() + switch oldConsumer.finish() { + case .failure(SoftwarePacketReadAhead.ReadError.interrupted): break + default: preconditionFailure("old consumer escaped a cache-hit generation change") + } + precondition(source.snapshot.packetCount == restoredCount, + "old consumer consumed the restored cache cursor") + precondition(source.snapshot.sourceEpoch == sourceEpoch) + let first = try PendingRead(source).finish().get()! + precondition(first.flags & 1 != 0 && first.pts <= 20) + try check(PendingRead(source).finish().get() == keyPacket(first.pts + 1)) + close(source, fifo) + } + + static func staleHostAdmission(_ root: URL) throws { + let entered = DispatchSemaphore(value: 0) + let release = DispatchSemaphore(value: 0) + let hookCount = Locked(0) + let hostCurrent = Locked(true) + let next = Locked(Int64(0)) + let (source, fifo) = try make(root, beforeConsumerOperation: { + let first = hookCount.withValue { value in defer { value += 1 }; return value == 0 } + if first { entered.signal(); requireSignal(release, "stale host consumer was not released") } + }) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + guard value == 0 else { return nil } + value += 1; return packet(0) + } + } + source.start() + eventually("host-admission fixture did not end") { source.snapshot.sourceEnded } + let pending = PendingRead(source, isCurrent: { hostCurrent.withValue { $0 } }) + requireSignal(entered, "host consumer was not paused before FIFO pop") + hostCurrent.withValue { $0 = false } + release.signal() + switch pending.finish() { + case .failure(SoftwarePacketReadAhead.ReadError.interrupted): break + default: preconditionFailure("stale host consumer read a packet") + } + precondition(source.snapshot.packetCount == 1, "host admission ran after a destructive pop") + try check(PendingRead(source).finish().get() == packet(0)) + switch PendingRead(source, isCurrent: { false }).finish() { + case .failure(SoftwarePacketReadAhead.ReadError.interrupted): break + default: preconditionFailure("stale host consumer observed EOF as a current result") + } + try check(PendingRead(source).finish().get() == nil) + close(source, fifo) + } + + static func successorCoverageIntegration(_ root: URL) throws { + // One-second tick fixture exercises successor semantics without relying on packet duration. + // The long packet duration belongs to decode cadence and must not extend final coverage. + let packets = [packet(0), packet(3, duration: 16), packet(2), packet(1), packet(4), packet(5)] + let next = Locked(0) + let (source, fifo) = try make(root, videoReorderDepth: 3) { accepts in + try next.withValue { value in + guard accepts() else { throw SourceFailure.rejected } + guard value < packets.count else { return nil } + defer { value += 1 }; return packets[value] + } + } + source.start() + eventually("successor fixture did not reach true EOF") { source.snapshot.sourceEnded } + precondition(source.snapshot.frontier == 5, + "successor model used packet decode duration or failed to finalize at EOF") + source.updatePlayhead(5) + precondition(source.snapshot.frontier == nil, "unknown final frame end was fabricated") + close(source, fifo) + } +} diff --git a/Scripts/tests/SoftwareReadAdmissionStandalone.swift b/Scripts/tests/SoftwareReadAdmissionStandalone.swift new file mode 100644 index 000000000..c5cecb1d8 --- /dev/null +++ b/Scripts/tests/SoftwareReadAdmissionStandalone.swift @@ -0,0 +1,45 @@ +import Foundation + +@main +struct SoftwareReadAdmissionTests { + static func main() { + let admits = SoftwareReadAdmission.admits + precondition(admits(0, 0, 0, false)) + for read: UInt64 in [0, 1, 2, .max] { + for requested: UInt64 in [0, 1, 2, .max] { + for settled: UInt64 in [0, 1, 2, .max] { + for stopped in [false, true] { + precondition(admits(read, requested, settled, stopped) + == (!stopped && read == requested && requested == settled)) + } + } + } + } + // The first new-generation packet must not be consumed by an old host iteration that + // enters the cache only after a seek has already opened and settled its own generation. + precondition(!admits(7, 8, 8, false)) + // EOF and errors use the exact same gate BEFORE their terminal branches. + for _ in ["packet", "EOF", "error", "closed", "delayed onEnd", "tail park"] { + precondition(!admits(7, 8, 7, false)) + precondition(!admits(7, 8, 8, false)) + precondition(!admits(8, 8, 7, false)) + precondition(!admits(8, 8, 8, true)) + precondition(admits(8, 8, 8, false)) + } + var terminal = SoftwareTerminalGeneration() + precondition(!terminal.shouldPark(generation: 7)) + precondition(terminal.record(7)) + precondition(!terminal.record(7)) + precondition(terminal.shouldPark(generation: 7)) + // EOF has been read and its callback queued, but a seek wins MainActor first. The old + // callback is rejected, and the still-live consumer is released for generation 8. + precondition(!admits(7, 8, 8, false)) + precondition(!terminal.shouldPark(generation: 8)) + precondition(admits(8, 8, 8, false)) + precondition(terminal.record(8)) + precondition(!terminal.record(8)) + precondition(terminal.shouldPark(generation: 8)) + precondition(!admits(8, 8, 8, true)) // stop exits the condition wait, not a new EOF. + print("PASS: host read admission across old/new/open/settled seek generations; stale EOF/errors/tail tasks rejected; terminal callbacks once per generation and superseding seek resumes consumer") + } +} diff --git a/Scripts/tests/SoftwareStoredPacketStandalone.swift b/Scripts/tests/SoftwareStoredPacketStandalone.swift new file mode 100644 index 000000000..8dba719a0 --- /dev/null +++ b/Scripts/tests/SoftwareStoredPacketStandalone.swift @@ -0,0 +1,101 @@ +import Foundation +import AetherLibavcodec +import AetherLibavutil + +/// Same tracked double-pointer adapter used by SoftwarePlaybackHost, without its UI/runtime graph. +func av_packet_free_safe(_ packet: UnsafeMutablePointer) { + var pointer: UnsafeMutablePointer? = packet + trackedPacketFree(&pointer) +} + +@main +struct SoftwareStoredPacketTests { + enum TestError: Error { case allocationFailed } + + static func main() throws { + precondition(PacketBalanceTracker.alive == 0) + let allocationsBefore = PacketBalanceTracker.totalAllocs + for iteration in 0..<32 { + try roundTrip( + bytes: Data((0..<256).map(UInt8.init)), pts: 1234567 + Int64(iteration), + dts: 1233566, duration: 1001, position: 9_876_543_210, + stream: 3, flags: AV_PKT_FLAG_KEY | AV_PKT_FLAG_CORRUPT | AV_PKT_FLAG_DISCARD, + numerator: 1, denominator: 30000 + ) + try roundTrip( + bytes: Data(), pts: Int64.min, dts: Int64.min, duration: 0, position: -1, + stream: 5, flags: 0, numerator: 0, denominator: 1 + ) + try roundTrip( + bytes: Data([0x00, 0xff, 0x80, 0x01]), pts: -1001, dts: -3003, + duration: 1001, position: 0, stream: 0, flags: AV_PKT_FLAG_DISPOSABLE, + numerator: 1001, denominator: 30000 + ) + try roundTrip( + bytes: Data([0x7f]), pts: Int64.min, dts: Int64.max, + duration: Int64.max, position: Int64.max, stream: Int32.max, + flags: Int32.max, numerator: Int32.max, denominator: Int32.max + ) + } + precondition(PacketBalanceTracker.alive == 0) + precondition(PacketBalanceTracker.totalAllocs - allocationsBefore == 256) + var empty: UnsafeMutablePointer? = nil + trackedPacketFree(&empty) + precondition(PacketBalanceTracker.alive == 0) + print("PASS: 128 AVPacket/binary-envelope roundtrips; payload, PTS/DTS including NOPTS, duration/position/flags/time_base, distinct side data including empty bytes; tracked allocations=256, alive=0") + } + + static func roundTrip( + bytes: Data, pts: Int64, dts: Int64, duration: Int64, position: Int64, + stream: Int32, flags: Int32, numerator: Int32, denominator: Int32 + ) throws { + let aliveBefore = PacketBalanceTracker.alive + guard let source = trackedPacketAlloc() else { throw TestError.allocationFailed } + do { + defer { av_packet_free_safe(source) } + guard av_new_packet(source, Int32(bytes.count)) >= 0 else { throw TestError.allocationFailed } + if !bytes.isEmpty { bytes.copyBytes(to: source.pointee.data, count: bytes.count) } + source.pointee.pts = pts + source.pointee.dts = dts + source.pointee.duration = duration + source.pointee.pos = position + source.pointee.stream_index = stream + source.pointee.flags = flags + source.pointee.time_base = AVRational(num: numerator, den: denominator) + + let sideData: [(AVPacketSideDataType, Data)] = [ + (AV_PKT_DATA_SKIP_SAMPLES, Data([0x00, 0x04, 0, 0, 0x00, 0x08, 0, 0, 0, 0])), + (AV_PKT_DATA_WEBVTT_IDENTIFIER, Data("cue-42".utf8)), + (AV_PKT_DATA_WEBVTT_SETTINGS, Data("align:start position:10%".utf8)), + (AV_PKT_DATA_NEW_EXTRADATA, Data()), + ] + for (type, data) in sideData { + guard let target = av_packet_new_side_data(source, type, data.count) else { + throw TestError.allocationFailed + } + if !data.isEmpty { data.copyBytes(to: target, count: data.count) } + } + + let stored = try SoftwareStoredPacket(copying: source) + let expected = SoftwareStoredPacket( + pts: pts, dts: dts, duration: duration, position: position, + streamIndex: stream, flags: flags, timeBaseNumerator: numerator, + timeBaseDenominator: denominator, bytes: bytes, + sideData: sideData.map { .init(type: $0.0.rawValue, bytes: $0.1) } + ) + precondition(stored == expected) + let encoded = try stored.encoded() + precondition(encoded.starts(with: Data("bplist00".utf8))) + let decoded = try SoftwareStoredPacket.decode(encoded) + precondition(decoded == expected) + let restored = try decoded.makeAVPacket() + defer { av_packet_free_safe(restored) } + let copiedBack = try SoftwareStoredPacket(copying: restored) + precondition(copiedBack == expected) + precondition(restored.pointee.side_data_elems == Int32(sideData.count)) + precondition(restored.pointee.size == Int32(bytes.count)) + precondition(PacketBalanceTracker.alive == aliveBefore + 2) + } + precondition(PacketBalanceTracker.alive == aliveBefore) + } +} diff --git a/Scripts/tests/SoftwareVideoPacketCoverageStandalone.swift b/Scripts/tests/SoftwareVideoPacketCoverageStandalone.swift new file mode 100644 index 000000000..8dbd66c10 --- /dev/null +++ b/Scripts/tests/SoftwareVideoPacketCoverageStandalone.swift @@ -0,0 +1,189 @@ +import Foundation + +@main +struct SoftwareVideoPacketCoverageTests { + static func main() { + observedVariableFrameRateSequence() + pendingBFramesStayBehindWatermark() + boundedFieldReordering() + discontinuitiesAndFinalPicture() + invalidationResetAndBounds() + pruneAndRationalPolicy() + print("PASS: observed VFR successor holds, pending B-frame watermark, 32-field reorder bound, one-second discontinuities, unknown final end, invalid PTS/late arrival/reset, rational boundaries and pruning") + } + + static func model(depth: Int = 32) -> SoftwareVideoPacketCoverage { + SoftwareVideoPacketCoverage(timeBaseNumerator: 1, timeBaseDenominator: 30000, + reorderDepth: depth) + } + + static func observedVariableFrameRateSequence() { + let packets: [(pts: Int64, dts: Int64, duration: Int64)] = [ + (1261260, 1259258, 1001), + (1277276, 1260259, 1001), + (1280279, 1261260, 16016), + (1279278, 1277276, 1001), + (1278277, 1278277, 1001), + ] + var old = SoftwarePacketCoverage() + var video = model() + for packet in packets { + old.insert(pts: packet.pts, duration: packet.duration) + precondition(video.insert(pts: packet.pts)) + } + precondition(old.frontier(containing: packets[0].pts) == 1262261) + precondition(old.frontier(containing: 1265000) == nil) + // These first five packets alone have not crossed the 32-entry reorder watermark. + precondition(video.frontier(containing: packets[0].pts) == nil) + // Subsequent retained packets release the exact observed successor pair without EOF. + for index in 1...32 { precondition(video.insert(pts: 1280279 + Int64(index) * 1001)) } + precondition(!video.isInvalidated) + precondition(video.frontier(containing: 1265000) == 1280279) + precondition(video.frontierSeconds(containing: 1265000.0 / 30000.0, + timeBaseNumerator: 1, timeBaseDenominator: 30000) == 1280279.0 / 30000.0) + // No input packet was rewritten: the long hold is between PTS 1261260 and 1277276, + // not the unrelated packet carrying the 16016-tick decode duration. + precondition(packets[0].duration == 1001) + precondition(packets[2].duration == 16016) + + var eof = model() + for packet in packets { precondition(eof.insert(pts: packet.pts)) } + eof.finish() + precondition(eof.frontier(containing: 1265000) == 1280279) + precondition(eof.frontier(containing: 1280279) == nil) + } + + static func pendingBFramesStayBehindWatermark() { + var video = model() + video.insert(pts: 0) + video.insert(pts: 3003) + for index in 4...33 { video.insert(pts: Int64(index) * 1001) } + precondition(video.pendingCount == 32) + precondition(video.frontier(containing: 0) == nil) + // Late-arriving B pictures fit inside the pending window. A future P picture must not + // prematurely expose its PTS as a playable frontier while these are unresolved. + video.insert(pts: 1001) + precondition(video.frontier(containing: 0) == nil) + video.insert(pts: 2002) + precondition(video.frontier(containing: 0) == 1001) + precondition(video.frontier(containing: 1001) == nil) + precondition(!video.isInvalidated) + precondition(video.pendingCount == 32) + } + + static func boundedFieldReordering() { + var video = model() + // Thirty-two field timestamps arrive in reverse order, then later fields arrive normally. + for field in (0..<32).reversed() { + precondition(video.insert(pts: Int64(field) * 500)) + precondition(video.pendingCount <= 32) + precondition(video.frontier(containing: 0) == nil) + } + for field in 32..<96 { + precondition(video.insert(pts: Int64(field) * 500)) + precondition(video.pendingCount == 32) + precondition(!video.isInvalidated) + } + precondition(video.frontier(containing: 0) == 63 * 500) + video.finish() + precondition(video.pendingCount == 0) + precondition(video.frontier(containing: 0) == 95 * 500) + precondition(video.frontier(containing: 95 * 500) == nil) + } + + static func discontinuitiesAndFinalPicture() { + var video = model(depth: 1) + for pts: Int64 in [0, 30000, 60001, 61002, 62003] { video.insert(pts: pts) } + video.finish() + precondition(video.frontier(containing: 0) == 30000) // Exactly 1 s is permitted. + precondition(video.frontier(containing: 30000) == nil) // 1 s + 1 tick is not. + precondition(video.frontier(containing: 40000) == nil) + precondition(video.frontier(containing: 60001) == 62003) + precondition(video.frontier(containing: 62003) == nil) + precondition(!video.isInvalidated) + video.finish() + precondition(video.frontier(containing: 60001) == 62003) + precondition(!video.insert(pts: 63004)) // EOF must be reset before another generation. + + var one = model() + one.insert(pts: 1001) + one.finish() + precondition(one.rangeCount == 0) + precondition(one.frontier(containing: 1001) == nil) + } + + static func invalidationResetAndBounds() { + var video = model(depth: 1) + for pts: Int64 in [0, 1001, 2002, 3003] { video.insert(pts: pts) } + precondition(video.frontier(containing: 0) == 2002) + precondition(!video.insert(pts: 1000)) // Behind already emitted PTS 2002. + precondition(video.isInvalidated) + precondition(video.lateTimestampCount == 1) + precondition(video.pendingCount == 0) + precondition(video.rangeCount == 0) + precondition(video.frontier(containing: 0) == nil) + precondition(!video.insert(pts: 4004)) + video.reset() + precondition(!video.isInvalidated) + precondition(video.lateTimestampCount == 0) + for pts: Int64 in [-3003, -2002, -1001] { video.insert(pts: pts) } + video.finish() + precondition(video.frontier(containing: -3003) == -1001) + precondition(video.frontier(containing: 0) == nil) + video.reset() + precondition(!video.insert(pts: .min)) + precondition(video.isInvalidated) + precondition(video.rangeCount == 0) + + for depth in [Int.min, -1, 0, 33, Int.max] { + var invalid = model(depth: depth) + precondition(invalid.isInvalidated) + precondition(!invalid.insert(pts: 0)) + precondition((1...32).contains(invalid.reorderDepth)) + invalid.reset() + precondition(invalid.isInvalidated) + } + for (num, den): (Int32, Int32) in [(0, 1), (1, 0), (-1, 30000), (1, -1)] { + var invalid = SoftwareVideoPacketCoverage(timeBaseNumerator: num, timeBaseDenominator: den) + precondition(invalid.isInvalidated) + precondition(!invalid.insert(pts: 0)) + } + + var overflow = model(depth: 1) + overflow.insert(pts: .min + 1) + overflow.insert(pts: .max) + overflow.finish() + precondition(overflow.isInvalidated) + precondition(overflow.rangeCount == 0) + + var duplicates = model(depth: 1) + for _ in 0..<1000 { precondition(duplicates.insert(pts: 0)) } + precondition(duplicates.pendingCount == 1) + precondition(duplicates.frontier(containing: 0) == nil) + duplicates.insert(pts: 1001) + precondition(duplicates.insert(pts: 0)) // Equal to watermark, not a late older PTS. + duplicates.insert(pts: 2002) + precondition(duplicates.frontier(containing: 0) == 1001) + } + + static func pruneAndRationalPolicy() { + var video = model(depth: 1) + for pts: Int64 in [0, 1001, 50000, 51001, 100000, 101001] { video.insert(pts: pts) } + video.finish() + precondition(video.rangeCount == 3) + video.prune(before: 50500) + precondition(video.rangeCount == 2) + precondition(video.frontier(containing: 50000) == 51001) + precondition(video.frontier(containing: 25000) == nil) + precondition(video.frontierSeconds(containing: 50000.0 / 30000.0, + timeBaseNumerator: 1, timeBaseDenominator: 48000) == nil) + + var rational = SoftwareVideoPacketCoverage(timeBaseNumerator: 1001, + timeBaseDenominator: 30000, reorderDepth: 1) + for pts: Int64 in [0, 29, 59, 60] { rational.insert(pts: pts) } + rational.finish() + precondition(rational.frontier(containing: 0) == 29) + precondition(rational.frontier(containing: 29) == nil) // 30 * 1001 / 30000 > 1 s. + precondition(rational.frontier(containing: 59) == 60) + } +} diff --git a/Sources/AetherEngine/AetherEngine+Diagnostics.swift b/Sources/AetherEngine/AetherEngine+Diagnostics.swift index 6a19bdf96..2322544d4 100644 --- a/Sources/AetherEngine/AetherEngine+Diagnostics.swift +++ b/Sources/AetherEngine/AetherEngine+Diagnostics.swift @@ -488,12 +488,18 @@ extension AetherEngine { softwareHost?.ioWindowDiagnostics ?? nativeVideoSession?.demuxer?.ioWindowDiagnostics } - /// Resident bytes in the loopback HLS segment cache. nil when no native session is active. + /// Compressed resident bytes: software packet spool or native loopback segment cache. var cachedBytes: Int64? { + if let bytes = softwareHost?.cachedVODBytes { return bytes } guard let bytes = nativeVideoSession?.segmentCacheTotalBytes else { return nil } return Int64(bytes) } + /// Short metadata lock only; never reads the packet store from the main actor. + var softwarePacketCacheSnapshot: SoftwarePacketReadAhead.Snapshot? { + softwareHost?.vodPacketCacheSnapshot + } + /// Freshly stat-ed on-disk footprint of the segment cache. nil when no native session is active. Used by `aetherctl live --report-cache-bytes`. public var segmentCacheDiskBytes: Int64? { nativeVideoSession?.segmentCacheDiskBytes diff --git a/Sources/AetherEngine/AetherEngine+Loading.swift b/Sources/AetherEngine/AetherEngine+Loading.swift index 1ee1cf12c..a469c6341 100644 --- a/Sources/AetherEngine/AetherEngine+Loading.swift +++ b/Sources/AetherEngine/AetherEngine+Loading.swift @@ -1695,14 +1695,13 @@ extension AetherEngine { .sink { [weak self] value in guard let self = self else { return } self.clock.currentTime = value - // bufferedPosition = newest demuxed source PTS, clamped to never trail the playhead (#54). - // #303: `bufferedSessionTime` is fed from `noteEdge`, which only runs on live - // sessions, so a VOD software session used to publish the playhead back as its own - // frontier. The decoded cushion is what it has instead. + // Both paths publish a real continuous cache frontier. Software VOD intersects + // selected A/V packet PTS coverage; the decoded cushion remains the unknown fallback. self.clock.bufferedPosition = SoftwareBufferFrontier.bufferedPosition( currentTime: value, liveFrontier: host.bufferedSessionTime, - cushion: host.displayCushionSeconds) + cushion: host.displayCushionSeconds, + cachedVODFrontier: host.cachedVODSessionTime) } .store(in: &softwareCancellables) // #107: sourceTime rides the RAW synchronizer clock (source axis) so subtitle cues @@ -1737,6 +1736,7 @@ extension AetherEngine { Task { @MainActor in self?.setReaderNetworkPhase(phase) } } if loadGeneration == generation { recordStartupCheckpoint(.sessionConstructed) } // #361 + let forwardBufferSegments = loadedOptions.forwardBufferSegments try await Task.detached(priority: .userInitiated) { [host, preopenedDemuxer, url, sourceHTTPHeaders, isLive, dvrWindowSeconds, probesize, maxAnalyzeDuration, sequentialOrigin, declaredDuration, networkPhaseSink] in let dem: Demuxer @@ -1752,7 +1752,8 @@ extension AetherEngine { startPosition: startPosition, audioSourceStreamIndex: audioSourceStreamIndex, isLive: isLive, - dvrWindowSeconds: dvrWindowSeconds + dvrWindowSeconds: dvrWindowSeconds, + forwardBufferSegments: forwardBufferSegments ) }.value // Superseded: stop idempotently to tear down the demuxer the detached closure opened, then unwind. diff --git a/Sources/AetherEngine/Demuxer/Demuxer.swift b/Sources/AetherEngine/Demuxer/Demuxer.swift index b7601983f..95ad4d088 100644 --- a/Sources/AetherEngine/Demuxer/Demuxer.swift +++ b/Sources/AetherEngine/Demuxer/Demuxer.swift @@ -1243,10 +1243,13 @@ public final class Demuxer: @unchecked Sendable { return result } - func readPacket() throws -> UnsafeMutablePointer? { + func readPacket(isCurrent: @Sendable () -> Bool = { true }) throws -> UnsafeMutablePointer? { accessLock.lock() defer { accessLock.unlock() } while true { + // A read-ahead decision made before a seek cannot start a NEW-position read after + // the seek releases this lock, then throw that first new packet away as stale. + guard isCurrent() else { throw CancellationError() } // #409: a packet the repair held during its sampling window is handed back before any // new read, so the container's own order survives the verdict. Checked every pass, not // once on entry: the packet that completes the sample flips the phase, and the queue diff --git a/Sources/AetherEngine/Diagnostics/LiveTelemetry.swift b/Sources/AetherEngine/Diagnostics/LiveTelemetry.swift index d6130246f..518a9ed97 100644 --- a/Sources/AetherEngine/Diagnostics/LiveTelemetry.swift +++ b/Sources/AetherEngine/Diagnostics/LiveTelemetry.swift @@ -3,10 +3,10 @@ import Foundation /// 1 Hz live playback telemetry snapshot. Nil fields are path-asymmetric: /// observedFps=nil on native (AVPlayer has no usable live FPS counter); /// avSyncGapMs=nil on SW (measured by HLSSegmentProducer which only runs on the native/HLS-loopback path); -/// forwardBufferSeconds=nil on SW, and stays nil on purpose (#306): the software demux loop reads on -/// renderer back-pressure, so there is no seconds-deep reservoir of arrived-but-unplayed media to -/// report there. `displayCushionSeconds` and `readerWindowAheadBytes` are what that path holds instead, -/// and putting either of them under the same name would report a near-stall on a healthy session. +/// forwardBufferSeconds=nil on SW: this remains the native player's loaded-range metric (#306). +/// Software VOD compressed packet read-ahead is reported through engine.bufferedPosition and +/// cachedBytes, separately from displayCushionSeconds and the byte-source reader window. Do not +/// interpret a sub-second decoded queue as the size of the compressed packet cache. public struct LiveTelemetry: Equatable, Sendable { // Enthusiast section public let instantBitrateMbps: Double? @@ -38,6 +38,11 @@ public struct LiveTelemetry: Equatable, Sendable { /// the first metrics read. Cumulative for the session, so a rate comes from differencing two ticks. public let accumulatedFrameDelaySeconds: Double? public let cachedBytes: Int64? + /// Session-local compressed-packet cache counters. nil outside software VOD. A hit reuses + /// retained packets without changing the source epoch; a miss requires a source reposition. + public let softwareCacheSeekHits: UInt64? + public let softwareCacheSeekMisses: UInt64? + public let softwareCacheSourceEpoch: UInt64? /// The rate the source link delivers at while it is delivering, so it stays comparable between the /// two paths: `observedBitrate` from the access log on native, and on the software path the /// demuxer's own byte counter over the seconds bytes arrived in (#306 follow-up). Not a wall-clock @@ -86,6 +91,9 @@ public struct LiveTelemetry: Equatable, Sendable { readerWindowAheadBytes: Int? = nil, accumulatedFrameDelaySeconds: Double? = nil, cachedBytes: Int64?, + softwareCacheSeekHits: UInt64? = nil, + softwareCacheSeekMisses: UInt64? = nil, + softwareCacheSourceEpoch: UInt64? = nil, networkThroughputMbps: Double?, networkTransferredBytes: Int64?, avSyncGapMs: Double?, @@ -107,6 +115,9 @@ public struct LiveTelemetry: Equatable, Sendable { self.readerWindowAheadBytes = readerWindowAheadBytes self.accumulatedFrameDelaySeconds = accumulatedFrameDelaySeconds self.cachedBytes = cachedBytes + self.softwareCacheSeekHits = softwareCacheSeekHits + self.softwareCacheSeekMisses = softwareCacheSeekMisses + self.softwareCacheSourceEpoch = softwareCacheSourceEpoch self.networkThroughputMbps = networkThroughputMbps self.networkTransferredBytes = networkTransferredBytes self.avSyncGapMs = avSyncGapMs diff --git a/Sources/AetherEngine/Diagnostics/LiveTelemetrySampler.swift b/Sources/AetherEngine/Diagnostics/LiveTelemetrySampler.swift index e4cd644e3..1be0c6d57 100644 --- a/Sources/AetherEngine/Diagnostics/LiveTelemetrySampler.swift +++ b/Sources/AetherEngine/Diagnostics/LiveTelemetrySampler.swift @@ -329,6 +329,7 @@ final class LiveTelemetrySampler { evaluateEndOfMediaPark(engine: engine, readings: readings) } + let softwareCache = engine.softwarePacketCacheSnapshot let snapshot = LiveTelemetry( instantBitrateMbps: instantBitrateMbps, averageBitrateMbps: averageBitrateMbps, @@ -340,6 +341,9 @@ final class LiveTelemetrySampler { readerWindowAheadBytes: readerWindowAheadBytes, accumulatedFrameDelaySeconds: accumulatedFrameDelaySeconds, cachedBytes: engine.cachedBytes, + softwareCacheSeekHits: softwareCache?.cacheSeekHits, + softwareCacheSeekMisses: softwareCache?.cacheSeekMisses, + softwareCacheSourceEpoch: softwareCache?.sourceEpoch, networkThroughputMbps: networkThroughputMbps, networkTransferredBytes: networkTransferredBytes, avSyncGapMs: avSyncGapMs, diff --git a/Sources/AetherEngine/Native/SoftwareBufferFrontier.swift b/Sources/AetherEngine/Native/SoftwareBufferFrontier.swift index c8338689f..b401f63b8 100644 --- a/Sources/AetherEngine/Native/SoftwareBufferFrontier.swift +++ b/Sources/AetherEngine/Native/SoftwareBufferFrontier.swift @@ -21,10 +21,12 @@ enum SoftwareBufferFrontier { } /// `clock.bufferedPosition` for a software session. `liveFrontier` is the newest demuxed source - /// PTS in session time, which only live sessions feed (`noteEdge` is gated on `isLive`); on VOD - /// it is 0 and the cushion is the only thing that can carry a frontier. AetherEngine#54: the - /// result never trails the playhead. - static func bufferedPosition(currentTime: Double, liveFrontier: Double, cushion: Double?) -> Double { - max(currentTime, liveFrontier, currentTime + (cushion ?? 0)) + /// PTS in session time, which only live sessions feed (`noteEdge` is gated on `isLive`). + /// `cachedVODFrontier` is the compressed A/V packet coverage, not a bitrate estimate or a + /// decoded-frame reservoir. Unknown coverage leaves the old cushion fallback intact. + /// AetherEngine#54: the result never trails the playhead. + static func bufferedPosition(currentTime: Double, liveFrontier: Double, cushion: Double?, + cachedVODFrontier: Double? = nil) -> Double { + max(currentTime, liveFrontier, currentTime + (cushion ?? 0), cachedVODFrontier ?? currentTime) } } diff --git a/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift b/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift new file mode 100644 index 000000000..2091b8167 --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift @@ -0,0 +1,124 @@ +import Foundation + +/// Presentation-time coverage of compressed packets that the caller still owns. +/// +/// Keep one instance per selected stream and insert actual packet PTS/duration in that stream's +/// time base. Decode timestamps, byte counts and average bitrates cannot prove playable coverage. +/// Decode-order arrival is supported: a future reference picture does not bridge a missing B +/// picture's presentation interval until that packet actually arrives. +/// +/// This value does not own packets or synchronize access. The packet owner must reset coverage +/// whenever packets are discarded (seek/flush/stream replacement), and must not prune beyond its +/// verified playback/discard position. `prune` only bounds old metadata; it is not a cache eviction +/// API for packets that are still ahead of the playhead. +struct SoftwarePacketCoverage: Sendable { + private var ranges: [Range] = [] + let maximumRangeCount: Int + + init(maximumRangeCount: Int = 4096) { + self.maximumRangeCount = max(1, maximumRangeCount) + } + + var rangeCount: Int { ranges.count } + + /// Adds [pts, pts + duration). Invalid or capacity-exceeding input leaves existing coverage + /// unchanged. Rejecting a sparse extension is conservative: it never claims a missing packet. + @discardableResult + mutating func insert(pts: Int64, duration: Int64) -> Bool { + guard pts != Int64.min, duration > 0 else { return false } + let (end, overflow) = pts.addingReportingOverflow(duration) + guard !overflow else { return false } + + var lower = 0 + var upper = ranges.count + while lower < upper { + let middle = lower + (upper - lower) / 2 + if ranges[middle].upperBound < pts { + lower = middle + 1 + } else { + upper = middle + } + } + + let first = lower + var last = first + var mergedStart = pts + var mergedEnd = end + while last < ranges.count, ranges[last].lowerBound <= mergedEnd { + mergedStart = min(mergedStart, ranges[last].lowerBound) + mergedEnd = max(mergedEnd, ranges[last].upperBound) + last += 1 + } + guard ranges.count - (last - first) < maximumRangeCount else { return false } + ranges.replaceSubrange(first.. Int64? { + guard tick != Int64.min else { return nil } + var lower = 0 + var upper = ranges.count + while lower < upper { + let middle = lower + (upper - lower) / 2 + if ranges[middle].upperBound <= tick { + lower = middle + 1 + } else { + upper = middle + } + } + guard lower < ranges.count, ranges[lower].contains(tick) else { return nil } + return ranges[lower].upperBound + } + + /// Same coverage check for a source clock expressed in seconds. Convert range boundaries from + /// integer timestamps rather than rounding the clock to ticks, which can cross a fractional + /// frame boundary or hide a one-tick gap. There is deliberately no gap/rounding tolerance. + func frontierSeconds( + containing seconds: Double, + timeBaseNumerator: Int32, + timeBaseDenominator: Int32 + ) -> Double? { + guard seconds.isFinite, timeBaseNumerator > 0, timeBaseDenominator > 0 else { return nil } + let numerator = Double(timeBaseNumerator) + let denominator = Double(timeBaseDenominator) + for range in ranges { + // Extremely large timestamps can lose an entire tick in Double. Fail closed instead + // of silently merging a precision-sized gap in this seconds-facing convenience API. + guard Int64(exactly: Double(range.lowerBound)) == range.lowerBound, + Int64(exactly: Double(range.upperBound)) == range.upperBound else { return nil } + let start = Double(range.lowerBound) * numerator / denominator + let end = Double(range.upperBound) * numerator / denominator + guard start.isFinite, end.isFinite, start < end else { return nil } + if seconds < start { return nil } + if seconds < end { return end } + } + return nil + } + + /// Drops intervals wholly behind the supplied presentation tick while retaining the entire + /// containing interval and future islands. Does not manufacture coverage at the prune point. + mutating func prune(before tick: Int64) { + guard tick != Int64.min else { return } + let expired = ranges.prefix { $0.upperBound <= tick }.count + if expired > 0 { ranges.removeFirst(expired) } + } + + mutating func reset() { + ranges.removeAll(keepingCapacity: true) + } + + /// A selected audio stream is a required part of A/V cache coverage, even when muted. Unknown + /// selected-stream coverage stays unknown; an absent audio stream does not limit video-only + /// playback. Inputs must already be on the same presentation timeline and contain the clock. + static func combinedFrontier( + video: Double?, audio: Double?, requiresAudio: Bool + ) -> Double? { + guard let video, video.isFinite else { return nil } + guard requiresAudio else { return video } + guard let audio, audio.isFinite else { return nil } + return min(video, audio) + } +} diff --git a/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift b/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift new file mode 100644 index 000000000..26bb7b548 --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift @@ -0,0 +1,573 @@ +import Darwin +import Foundation + +/// A session-owned, disk-backed FIFO of opaque compressed-packet records. +/// +/// Payloads and record lengths live on disk, not in an in-memory packet index. Only head/tail +/// offsets and aggregate counters are retained, independent of the number of packets/chunks. +/// Call all I/O methods from background workers: the lock intentionally serializes disk access +/// with reset/close. The caller owns byte/time backpressure and packet serialization. +final class SoftwarePacketDiskFIFO: @unchecked Sendable { + /// An immutable record boundary, valid only while its chunk remains in this reset generation. + /// Clients can index keyframes by token without retaining packet payloads or forging offsets. + struct Cursor: Hashable, Sendable { + let chunkID: UInt64 + fileprivate let generation: UUID + fileprivate let offset: UInt64 + fileprivate let recordIndex: Int + fileprivate let payloadPrefix: Int + } + + struct StaleSweepResult: Sendable { + let inspectedCount: Int + let removedCount: Int + let failureCount: Int + } + + struct Snapshot: Sendable { + let count: Int + /// Payload bytes not yet popped; excludes record headers and the consumed head prefix. + let byteCount: Int + /// Logical file bytes, including headers, consumed prefixes and opt-in retained history. + let diskByteCount: Int + /// Includes consumed history in retained mode; unlike byteCount, this is a residency limit. + var residentByteCount: Int { diskByteCount } + let chunkCount: Int + /// Tokens in smaller chunks have been evicted and must be removed from a client's index. + let oldestRetainedChunkID: UInt64? + let isClosed: Bool + let hasFailure: Bool + } + + enum Failure: Error, Equatable { + case invalidChunkTarget + case closed + case corruptRecord + case capacityExceeded + case sessionLeaseLost + case retentionDisabled + case invalidCursor + } + + static let directoryPrefix = "aether-software-packets-" + private static let liveMarkerName = "session.lock" + + /// Always an explicitly created, unique child, never the caller's parent directory. + let storageDirectory: URL + let staleSweepResult: StaleSweepResult + private let chunkTargetBytes: Int + private let retainConsumed: Bool + private let lock = NSLock() + private var writer: FileHandle? + private var reader: FileHandle? + /// An open-file-description lock also protects against other sessions in this process. + /// Unlike an age-only marker, the kernel releases it when a process crashes. + private var leaseFD: Int32 = -1 + private var pendingChunkID: UInt64? + private var generation = UUID() + private var oldestChunkID: UInt64 = 0 + private var headID: UInt64 = 0 + private var tailID: UInt64 = 0 + private var headOffset: UInt64 = 0 + private var tailBytes: UInt64 = 0 + private var sealedHeadBytes: UInt64? + private var recordCount = 0 + private var payloadBytes = 0 + private var diskBytes = 0 + private var chunks = 0 + /// Monotonic append prefixes let a restored cursor compute unread counts in constant space. + private var writtenRecordCount = 0 + private var writtenPayloadBytes = 0 + private var isClosed = false + /// A partial disk operation cannot be retried as if it had succeeded. Only reset recovers it. + private var failure: (any Error)? + + init(chunkTargetBytes: Int = 4 * 1024 * 1024, + retainConsumed: Bool = false, + parentDirectory: URL = FileManager.default.temporaryDirectory, + now: Date = Date(), + leaseAcquisition: ((URL) throws -> Int32)? = nil) throws { + guard chunkTargetBytes >= 8 else { throw Failure.invalidChunkTarget } + self.chunkTargetBytes = chunkTargetBytes + self.retainConsumed = retainConsumed + storageDirectory = parentDirectory.appendingPathComponent( + "\(Self.directoryPrefix)\(UUID().uuidString)", isDirectory: true) + // mkdir, not createDirectory's existing-directory success: cleanup must own this root. + guard mkdir(storageDirectory.path, 0o700) == 0 else { throw Self.posixError() } + do { + // The injectable acquisition is used only by focused interrupted-init tests. + leaseFD = try (leaseAcquisition ?? Self.acquireNewLease)(storageDirectory) + guard leaseFD >= 0 else { throw Failure.sessionLeaseLost } + } catch { + try? FileManager.default.removeItem(at: storageDirectory) + if leaseFD >= 0 { Darwin.close(leaseFD) } + throw error + } + staleSweepResult = Self.sweepStaleSessionDirs(parentDirectory: parentDirectory, + currentSession: storageDirectory.lastPathComponent, now: now) + } + + deinit { + try? close() + // A failed best-effort removal must not leak the advisory lease descriptor. + if leaseFD >= 0 { Darwin.close(leaseFD) } + } + + var snapshot: Snapshot { + lock.lock() + defer { lock.unlock() } + return Snapshot(count: recordCount, byteCount: payloadBytes, diskByteCount: diskBytes, + chunkCount: chunks, oldestRetainedChunkID: chunks > 0 ? oldestChunkID : nil, + isClosed: isClosed, hasFailure: failure != nil) + } + + @discardableResult + func append(_ data: Data) throws -> Cursor { + lock.lock() + defer { lock.unlock() } + try requireUsable() + do { + let recordBytes = try checkedSum(data.count, 8) + let nextCount = try checkedSum(recordCount, 1) + let nextPayloadBytes = try checkedSum(payloadBytes, data.count) + let nextDiskBytes = try checkedSum(diskBytes, recordBytes) + let nextWrittenCount = try checkedSum(writtenRecordCount, 1) + let nextWrittenBytes = try checkedSum(writtenPayloadBytes, data.count) + if writer == nil { + try openFirstChunk() + } else if tailBytes > 0, + UInt64(recordBytes) > UInt64(chunkTargetBytes) - min(tailBytes, UInt64(chunkTargetBytes)) { + try openNextChunk() + } + guard let writer else { throw Failure.corruptRecord } + let cursor = Cursor(chunkID: tailID, generation: generation, offset: tailBytes, + recordIndex: writtenRecordCount, payloadPrefix: writtenPayloadBytes) + var length = UInt64(data.count).bigEndian + let header = withUnsafeBytes(of: &length) { Data($0) } + try writer.write(contentsOf: header) + try writer.write(contentsOf: data) + tailBytes += UInt64(recordBytes) + recordCount = nextCount + payloadBytes = nextPayloadBytes + diskBytes = nextDiskBytes + writtenRecordCount = nextWrittenCount + writtenPayloadBytes = nextWrittenBytes + return cursor + } catch { + failure = error + throw error + } + } + + func pop() throws -> Data? { + lock.lock() + defer { lock.unlock() } + try requireUsable() + guard recordCount > 0 else { return nil } + do { + var limit = try currentReadLimit() + // A retained reader can be at the old tail's EOF when the producer rolls to a new + // chunk. Crossing that boundary must neither return false EOF nor remove history. + if retainConsumed, headOffset == limit, headID < tailID { + try advanceRetainedHead() + limit = try currentReadLimit() + } + guard let reader else { throw Failure.corruptRecord } + guard headOffset <= limit, limit - headOffset >= 8 else { throw Failure.corruptRecord } + let header = try readExactly(8, from: reader) + let length = header.reduce(UInt64(0)) { ($0 << 8) | UInt64($1) } + guard length <= UInt64(payloadBytes), length <= limit - headOffset - 8 else { + throw Failure.corruptRecord + } + let data = try readExactly(Int(length), from: reader) + let nextOffset = headOffset + 8 + length + if nextOffset == limit { + if retainConsumed { + headOffset = nextOffset + if headID < tailID { try advanceRetainedHead() } + } else { + try reclaimHead(byteCount: limit) + } + } else { + headOffset = nextOffset + } + recordCount -= 1 + payloadBytes -= Int(length) + return data + } catch { + failure = error + throw error + } + } + + /// Move only the consumer cursor; the producer, resident bytes and forward frontier survive. + /// Rejected stale/cross-session tokens do not poison a healthy store, so callers can fall back + /// to their normal uncached seek. Disk corruption/I/O failures still fail closed. + func restore(to cursor: Cursor) throws { + lock.lock() + defer { lock.unlock() } + try requireUsable() + guard retainConsumed else { throw Failure.retentionDisabled } + guard cursor.generation == generation, chunks > 0, + cursor.chunkID >= oldestChunkID, cursor.chunkID <= tailID, + cursor.recordIndex >= 0, cursor.recordIndex < writtenRecordCount, + cursor.payloadPrefix >= 0, cursor.payloadPrefix <= writtenPayloadBytes else { + throw Failure.invalidCursor + } + do { + let replacement = try FileHandle(forReadingFrom: chunkURL(cursor.chunkID)) + var adopted = false + defer { if !adopted { try? replacement.close() } } + let size = try replacement.seekToEnd() + guard cursor.offset <= size, size - cursor.offset >= 8 else { throw Failure.corruptRecord } + try replacement.seek(toOffset: cursor.offset) + let header = try readExactly(8, from: replacement) + let length = header.reduce(UInt64(0)) { ($0 << 8) | UInt64($1) } + guard length <= size - cursor.offset - 8, + length <= UInt64(writtenPayloadBytes - cursor.payloadPrefix) else { + throw Failure.corruptRecord + } + try replacement.seek(toOffset: cursor.offset) + try reader?.close() + reader = replacement + adopted = true + headID = cursor.chunkID + headOffset = cursor.offset + sealedHeadBytes = headID < tailID ? size : nil + recordCount = writtenRecordCount - cursor.recordIndex + payloadBytes = writtenPayloadBytes - cursor.payloadPrefix + } catch { + failure = error + throw error + } + } + + /// Evict only complete history chunks strictly before the reader, oldest first. A budget is + /// not permission to discard unread packets or the current reader/writer chunk; the resident + /// count can therefore remain above budget until the consumer advances. No packet/chunk index + /// is built in memory. The next snapshot's oldest chunk invalidates evicted keyframe tokens. + func trimConsumed(toByteBudget budget: Int) throws { + lock.lock() + defer { lock.unlock() } + try requireUsable() + guard retainConsumed else { return } + do { + while diskBytes > max(0, budget), oldestChunkID < headID, oldestChunkID < tailID { + let url = chunkURL(oldestChunkID) + var info = stat() + guard lstat(url.path, &info) == 0 else { throw Self.posixError() } + guard info.st_mode & S_IFMT == S_IFREG, info.st_size >= 0, + UInt64(info.st_size) <= UInt64(diskBytes) else { throw Failure.corruptRecord } + try FileManager.default.removeItem(at: url) + diskBytes -= Int(info.st_size) + chunks -= 1 + oldestChunkID += 1 + } + } catch { + failure = error + throw error + } + } + + /// Discards this session's contents, including any partial record left by a failed operation. + func reset() throws { + lock.lock() + defer { lock.unlock() } + guard !isClosed else { throw Failure.closed } + do { + try closePacketHandles() + if try restoreWhollyMissingDirectory() { + // The OS already discarded all chunks; the replacement has its own live lease. + clearCounters() + failure = nil + return + } + try verifyLeaseIdentity() + if chunks > 0 { + for id in oldestChunkID...tailID { try removeChunkIfPresent(id) } + } + if let pendingChunkID { try removeChunkIfPresent(pendingChunkID) } + clearCounters() + failure = nil + } catch { + failure = error + throw error + } + } + + /// Idempotent; a failed cleanup can be retried, but the store stays closed in either case. + func close() throws { + lock.lock() + defer { lock.unlock() } + isClosed = true + do { + try removeOwnedStorage() + clearCounters() + if leaseFD >= 0 { + Darwin.close(leaseFD) + leaseFD = -1 + } + } catch { + failure = error + throw error + } + } + + private func requireUsable() throws { + guard !isClosed else { throw Failure.closed } + if let failure { throw failure } + } + + private func checkedSum(_ lhs: Int, _ rhs: Int) throws -> Int { + let (sum, overflow) = lhs.addingReportingOverflow(rhs) + guard !overflow else { throw Failure.capacityExceeded } + return sum + } + + private func chunkURL(_ id: UInt64) -> URL { + storageDirectory.appendingPathComponent("\(id).packets", isDirectory: false) + } + + private func makeWriter(_ id: UInt64) throws -> FileHandle { + let url = chunkURL(id) + pendingChunkID = id + // Never truncate an unexpected existing file, even inside our own unique directory. + try Data().write(to: url, options: .withoutOverwriting) + let handle = try FileHandle(forWritingTo: url) + pendingChunkID = nil + return handle + } + + private func openFirstChunk() throws { + writer = try makeWriter(0) + headID = 0 + oldestChunkID = 0 + tailID = 0 + chunks = 1 + } + + private func openNextChunk() throws { + guard tailID < UInt64.max else { throw Failure.capacityExceeded } + let previousWriter = writer + writer = nil + try previousWriter?.close() + let nextWriter = try makeWriter(tailID + 1) + if headID == tailID { sealedHeadBytes = tailBytes } + tailID += 1 + tailBytes = 0 + writer = nextWriter + chunks += 1 + } + + private func readExactly(_ length: Int, from handle: FileHandle) throws -> Data { + var result = Data() + result.reserveCapacity(length) + while result.count < length { + guard let part = try handle.read(upToCount: length - result.count), !part.isEmpty else { + throw Failure.corruptRecord + } + result.append(part) + } + return result + } + + private func currentReadLimit() throws -> UInt64 { + if reader == nil { reader = try FileHandle(forReadingFrom: chunkURL(headID)) } + guard let reader else { throw Failure.corruptRecord } + if headID == tailID { return tailBytes } + if let sealedHeadBytes { return sealedHeadBytes } + let size = try reader.seekToEnd() + sealedHeadBytes = size + try reader.seek(toOffset: headOffset) + return size + } + + private func advanceRetainedHead() throws { + let previous = reader + reader = nil + try previous?.close() + headID += 1 + headOffset = 0 + sealedHeadBytes = nil + } + + private func reclaimHead(byteCount: UInt64) throws { + let previousReader = reader + reader = nil + try previousReader?.close() + if headID == tailID { + let previousWriter = writer + writer = nil + try previousWriter?.close() + } + try FileManager.default.removeItem(at: chunkURL(headID)) + diskBytes -= Int(byteCount) + chunks -= 1 + headOffset = 0 + sealedHeadBytes = nil + if chunks == 0 { + headID = 0 + oldestChunkID = 0 + tailID = 0 + tailBytes = 0 + } else { + headID += 1 + oldestChunkID = headID + } + } + + private func closePacketHandles() throws { + var firstError: (any Error)? + let handles = [reader, writer] + reader = nil + writer = nil + for handle in handles { + do { try handle?.close() } catch { if firstError == nil { firstError = error } } + } + if let firstError { throw firstError } + } + + private func removeOwnedStorage() throws { + var firstError: (any Error)? + do { try closePacketHandles() } catch { firstError = error } + do { + try FileManager.default.removeItem(at: storageDirectory) + } catch let error as CocoaError where error.code == .fileNoSuchFile { + // A repeated close, or recovery after an externally purged temporary directory. + } catch { + if firstError == nil { firstError = error } + } + if let firstError { throw firstError } + } + + private func removeChunkIfPresent(_ id: UInt64) throws { + let url = chunkURL(id) + var info = stat() + guard lstat(url.path, &info) == 0 else { + if errno == ENOENT { return } + throw Self.posixError() + } + guard info.st_mode & S_IFMT == S_IFREG else { throw Failure.corruptRecord } + try FileManager.default.removeItem(at: url) + } + + private func verifyLeaseIdentity() throws { + var held = stat() + var named = stat() + guard leaseFD >= 0, fstat(leaseFD, &held) == 0, + lstat(storageDirectory.appendingPathComponent(Self.liveMarkerName).path, &named) == 0, + named.st_mode & S_IFMT == S_IFREG, + held.st_dev == named.st_dev, held.st_ino == named.st_ino else { + throw Failure.sessionLeaseLost + } + } + + /// Normal reset never removes/replaces the marker. An OS-purged root is different: its + /// unlinked lease no longer protects a path, so rebuild and acquire the replacement first. + private func restoreWhollyMissingDirectory() throws -> Bool { + var info = stat() + if lstat(storageDirectory.path, &info) == 0 { + guard info.st_mode & S_IFMT == S_IFDIR else { throw Failure.sessionLeaseLost } + return false + } + guard errno == ENOENT else { throw Self.posixError() } + guard mkdir(storageDirectory.path, 0o700) == 0 else { throw Self.posixError() } + do { + let replacement = try Self.acquireNewLease(storageDirectory) + if leaseFD >= 0 { Darwin.close(leaseFD) } + leaseFD = replacement + } catch { + try? FileManager.default.removeItem(at: storageDirectory) + throw error + } + return true + } + + private static func posixError() -> NSError { + NSError(domain: NSPOSIXErrorDomain, code: Int(errno)) + } + + private static func acquireNewLease(_ directory: URL) throws -> Int32 { + let path = directory.appendingPathComponent(liveMarkerName).path + let fd = open(path, O_CREAT | O_EXCL | O_RDWR | O_NOFOLLOW | O_CLOEXEC, 0o600) + guard fd >= 0 else { throw posixError() } + guard flock(fd, LOCK_EX | LOCK_NB) == 0 else { + let error = posixError() + Darwin.close(fd) + throw error + } + return fd + } + + /// Bounded crash-remnant cleanup, never a recursive search of the caller's temporary root. + /// Only our UUID-named direct-child directories older than one day are candidates. Keep the + /// candidate's advisory lease held until removal finishes; age alone never proves abandonment. + static func sweepStaleSessionDirs(parentDirectory: URL, currentSession: String? = nil, + now: Date = Date(), maxEntries: Int = 64, + maxRemovals: Int = 8) -> StaleSweepResult { + guard maxEntries > 0, maxRemovals > 0, + let entries = FileManager.default.enumerator(at: parentDirectory, + includingPropertiesForKeys: nil, + options: [.skipsHiddenFiles, .skipsSubdirectoryDescendants]) else { + return StaleSweepResult(inspectedCount: 0, removedCount: 0, failureCount: 0) + } + var inspected = 0 + var removed = 0 + var failures = 0 + while inspected < maxEntries, removed < maxRemovals, + let entry = entries.nextObject() as? URL { + inspected += 1 + let name = entry.lastPathComponent + guard name != currentSession, name.hasPrefix(directoryPrefix), + UUID(uuidString: String(name.dropFirst(directoryPrefix.count))) != nil else { continue } + var info = stat() + guard lstat(entry.path, &info) == 0 else { failures += 1; continue } + let modified = Double(info.st_mtimespec.tv_sec) + Double(info.st_mtimespec.tv_nsec) / 1e9 + guard info.st_mode & S_IFMT == S_IFDIR, + now.timeIntervalSince1970 - modified >= 86_400 else { continue } + let dirFD = open(entry.path, O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC) + guard dirFD >= 0 else { failures += 1; continue } + defer { Darwin.close(dirFD) } + // O_CREAT also lets us lease a root abandoned between mkdir and marker creation. + let markerFD = openat(dirFD, liveMarkerName, + O_CREAT | O_RDWR | O_NOFOLLOW | O_CLOEXEC, 0o600) + guard markerFD >= 0 else { failures += 1; continue } + defer { Darwin.close(markerFD) } + guard flock(markerFD, LOCK_EX | LOCK_NB) == 0 else { + if errno != EWOULDBLOCK && errno != EAGAIN { failures += 1 } + continue + } + // Recheck identity rather than following a replaced directory or marker symlink. + var held = stat() + var named = stat() + guard fstat(dirFD, &held) == 0, lstat(entry.path, &named) == 0, + named.st_mode & S_IFMT == S_IFDIR, + held.st_dev == named.st_dev, held.st_ino == named.st_ino else { + failures += 1 + continue + } + do { + try FileManager.default.removeItem(at: entry) + removed += 1 + } catch { failures += 1 } + } + return StaleSweepResult(inspectedCount: inspected, removedCount: removed, failureCount: failures) + } + + private func clearCounters() { + generation = UUID() + oldestChunkID = 0 + headID = 0 + tailID = 0 + headOffset = 0 + tailBytes = 0 + sealedHeadBytes = nil + recordCount = 0 + payloadBytes = 0 + diskBytes = 0 + chunks = 0 + writtenRecordCount = 0 + writtenPayloadBytes = 0 + pendingChunkID = nil + } +} diff --git a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift new file mode 100644 index 000000000..87655cf30 --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift @@ -0,0 +1,391 @@ +import Foundation + +/// Compressed VOD packet prefetch, independent of renderer pacing. The producer owns source reads; +/// the consumer gets byte-identical packet envelopes from a bounded disk FIFO. No main-thread I/O. +/// Cached seeks move only the consumer cursor. A source reposition has a separate epoch, so an +/// in-flight producer packet is neither lost nor duplicated when replaying retained data. +final class SoftwarePacketReadAhead: @unchecked Sendable { + struct Stream: Sendable { + let index: Int32 + let numerator: Int32 + let denominator: Int32 + } + struct Snapshot: Sendable { + let packetCount: Int + let bytes: Int + let residentBytes: Int + let frontier: Double? + let generation: UInt64 + let seeking: Bool + let sourceEnded: Bool + let sourceEpoch: UInt64 + let cacheSeekHits: UInt64 + let cacheSeekMisses: UInt64 + } + enum ReadError: Error { case interrupted, closed, corruptFIFO } + + private let condition = NSCondition() + /// Never acquired by a main-thread API. Serializes reset with append/pop, including the + /// generation check, so an old consumer cannot pop and discard the first NEW-generation packet. + private let operations = NSLock() + private let fifo: SoftwarePacketDiskFIFO + private let readSource: @Sendable (@Sendable () -> Bool) throws -> SoftwareStoredPacket? + private let beforeConsumerOperation: (@Sendable () -> Void)? + private let video: Stream + private let audio: Stream? + private let byteBudget: Int + private let forwardSeconds: Double + private let worker = DispatchQueue(label: "engine.sw.packet-prefetch", qos: .utility) + private var generation: UInt64 = 0 + private var sourceEpoch: UInt64 = 0 + private var sourceRepositioning = false + private var cacheSeekHits: UInt64 = 0 + private var cacheSeekMisses: UInt64 = 0 + private var resetPending = false + private var seeking = false + private var closed = false + private var started = false + private var ended = false + private var failure: Error? + private var count = 0 + private var bytes = 0 + private var residentBytes = 0 + private var sourceClock: Double + private var videoCoverage = SoftwarePacketCoverage() + private var audioCoverage = SoftwarePacketCoverage() + private var presentationCoverage: SoftwareVideoPacketCoverage? + private struct Keyframe { + let seconds: Double + let cursor: SoftwarePacketDiskFIFO.Cursor + } + private var keyframes: [Keyframe] = [] + private let maximumKeyframes = 65_536 + + /// Construct only off-main: creating the FIFO touches the temporary volume. + init(video: Stream, audio: Stream?, byteBudget: Int, forwardSeconds: Double, + initialSourceClock: Double, fifo: SoftwarePacketDiskFIFO, + videoReorderDepth: Int? = nil, + beforeConsumerOperation: (@Sendable () -> Void)? = nil, + readSource: @escaping @Sendable (@Sendable () -> Bool) throws -> SoftwareStoredPacket?) { + self.video = video + self.audio = audio + self.byteBudget = max(1, byteBudget) + self.forwardSeconds = max(1, forwardSeconds) + self.sourceClock = initialSourceClock + self.fifo = fifo + self.presentationCoverage = videoReorderDepth.map { + SoftwareVideoPacketCoverage(timeBaseNumerator: video.numerator, + timeBaseDenominator: video.denominator, reorderDepth: $0) + } + self.beforeConsumerOperation = beforeConsumerOperation + self.readSource = readSource + } + + func start() { + condition.lock() + guard !started, !closed else { condition.unlock(); return } + started = true + condition.unlock() + worker.async { self.produce() } + } + + var snapshot: Snapshot { + condition.lock(); defer { condition.unlock() } + return Snapshot(packetCount: count, bytes: bytes, residentBytes: residentBytes, + frontier: sourceRepositioning || closed ? nil : frontierLocked(), + generation: generation, seeking: seeking, sourceEnded: ended, + sourceEpoch: sourceEpoch, cacheSeekHits: cacheSeekHits, + cacheSeekMisses: cacheSeekMisses) + } + + /// Main-thread safe: metadata only. Decode/render backpressure still belongs to the old loop. + func updatePlayhead(_ seconds: Double) { + guard seconds.isFinite else { return } + condition.lock() + sourceClock = seconds + // Presentation history stays useful for backward cached seeks. Coverage has a fixed range + // cap; retained keyframe cursors, not a guessed timestamp floor, decide cache eligibility. + condition.broadcast() + condition.unlock() + } + + /// Legacy explicit cold seek. Production first uses beginSeek(to:) + prepareSeek off-main. + @discardableResult + func beginSeek() -> UInt64 { + condition.lock(); defer { condition.unlock() } + generation &+= 1 + sourceEpoch &+= 1 + sourceRepositioning = true + seeking = true + resetPending = true + clearSourceMetadataLocked() + condition.broadcast() + return generation + } + + /// Main-thread safe: freeze only the decoder/consumer, not the source reader or retained data. + @discardableResult + func beginSeek(to seconds: Double) -> UInt64 { + condition.lock(); defer { condition.unlock() } + generation &+= 1 + seeking = true + if seconds.isFinite { sourceClock = seconds } + condition.broadcast() + return generation + } + + /// Off-main, BEFORE any actual demuxer reposition. true means the target is already retained. + /// A hit does not change sourceEpoch: an in-flight source read must still be stored afterwards. + func prepareSeek(_ token: UInt64, to seconds: Double) throws -> Bool { + operations.lock(); defer { operations.unlock() } + condition.lock() + guard token == generation, seeking, !closed else { + condition.unlock(); throw ReadError.interrupted + } + let hasCoverage = !sourceRepositioning && !resetPending && failure == nil + && seconds.isFinite && (frontierLocked(at: seconds).map { $0 > seconds } ?? false) + let candidates = hasCoverage ? keyframes.filter { $0.seconds <= seconds } + .sorted { $0.seconds < $1.seconds } : [] + // A previous recovery/key picture supplies open-GOP/audio preroll when still retained. + let anchor = candidates.isEmpty ? nil : candidates[max(0, candidates.count - 2)] + condition.unlock() + + if let anchor { + do { + try fifo.restore(to: anchor.cursor) + let state = fifo.snapshot + condition.lock(); defer { condition.unlock() } + guard token == generation, !closed else { throw ReadError.interrupted } + copyDiskStateLocked(state) + sourceClock = seconds + cacheSeekHits &+= 1 + condition.broadcast() + return true + } catch SoftwarePacketDiskFIFO.Failure.invalidCursor { + // An expired bookmark is a cache miss, never a playback/disk failure. + } catch SoftwarePacketDiskFIFO.Failure.retentionDisabled { + // Keeps callers using the legacy destructive FIFO safe during migration. + } + } + + condition.lock() + guard token == generation, !closed else { + condition.unlock(); throw ReadError.interrupted + } + sourceEpoch &+= 1 + sourceRepositioning = true + resetPending = false + clearSourceMetadataLocked() + sourceClock = seconds + cacheSeekMisses &+= 1 + condition.broadcast() + condition.unlock() + try fifo.reset() + return false + } + + func endSeek(_ token: UInt64, sourceClock: Double) { + condition.lock(); defer { condition.unlock() } + guard token == generation, !closed else { return } + self.sourceClock = sourceClock + seeking = false + sourceRepositioning = false + condition.broadcast() + } + + /// Does not wait for a remote read or a disk operation. The host closes the Demuxer as usual, + /// unblocking its reader; the worker then releases only its own unique FIFO directory. + func close() { + condition.lock() + guard !closed else { condition.unlock(); return } + closed = true + generation &+= 1 + sourceEpoch &+= 1 + clearSourceMetadataLocked() + let needsCleanup = !started + condition.broadcast() + condition.unlock() + if needsCleanup { worker.async { try? self.fifo.close() } } + } + + /// Consumer thread only. nil means true EOF; a seek wake is explicitly different from EOF. + func read(isCurrent: @Sendable () -> Bool = { true }) throws -> SoftwareStoredPacket? { + condition.lock() + let token = generation + while count == 0, !ended, failure == nil, !closed, !seeking, + token == generation, isCurrent() { + condition.wait() + } + if closed { condition.unlock(); throw ReadError.closed } + let hadPackets = count > 0 + condition.unlock() + + if hadPackets { beforeConsumerOperation?() } + operations.lock() + defer { operations.unlock() } + condition.lock() + guard !closed else { condition.unlock(); throw ReadError.closed } + // Admission belongs to the HOST generation too. Capturing only our generation at read() + // entry can let an old host iteration steal the first packet of a completed new seek. + guard !seeking, token == generation, isCurrent() else { + condition.unlock(); throw ReadError.interrupted + } + if count == 0 { + let error = failure + condition.unlock() + if let error { throw error } + return nil + } + condition.unlock() + guard let data = try fifo.pop() else { throw ReadError.corruptFIFO } + let packet = try SoftwareStoredPacket.decode(data) + // Equality parks the producer too. Reclaim an eligible consumed chunk at the exact ceiling, + // otherwise it can remain asleep with unread packets until the queue drains completely. + try fifo.trimConsumed(toByteBudget: max(0, byteBudget - 1)) + let state = fifo.snapshot + condition.lock() + defer { condition.unlock() } + guard !closed, !seeking, token == generation, isCurrent() else { throw ReadError.interrupted } + copyDiskStateLocked(state) + condition.broadcast() + return packet + } + + private func frontierLocked(at seconds: Double? = nil) -> Double? { + let clock = seconds ?? sourceClock + let videoEnd: Double? + if let presentationCoverage { + videoEnd = presentationCoverage.frontierSeconds(containing: clock, + timeBaseNumerator: video.numerator, timeBaseDenominator: video.denominator) + } else { + videoEnd = videoCoverage.frontierSeconds(containing: clock, + timeBaseNumerator: video.numerator, timeBaseDenominator: video.denominator) + } + let audioEnd = audio.flatMap { stream in + audioCoverage.frontierSeconds(containing: clock, + timeBaseNumerator: stream.numerator, timeBaseDenominator: stream.denominator) + } + return SoftwarePacketCoverage.combinedFrontier( + video: videoEnd, audio: audioEnd, requiresAudio: audio != nil) + } + + private func clearSourceMetadataLocked() { + count = 0; bytes = 0; residentBytes = 0 + ended = false; failure = nil + videoCoverage.reset(); audioCoverage.reset(); presentationCoverage?.reset() + keyframes.removeAll(keepingCapacity: true) + } + + private func copyDiskStateLocked(_ state: SoftwarePacketDiskFIFO.Snapshot) { + count = state.count + bytes = state.byteCount + residentBytes = state.residentByteCount + if let floor = state.oldestRetainedChunkID { + keyframes.removeAll { $0.cursor.chunkID < floor } + } else { keyframes.removeAll(keepingCapacity: true) } + } + + private func produce() { + defer { try? fifo.close() } + while true { + condition.lock() + while !closed && !resetPending && (sourceRepositioning || ended || failure != nil || shouldParkLocked()) { + condition.wait() + } + if closed { condition.unlock(); return } + let token = sourceEpoch + let reset = resetPending + condition.unlock() + + if reset { + operations.lock() + do { + try fifo.reset() + condition.lock() + if token == sourceEpoch { resetPending = false } + condition.broadcast() + condition.unlock() + } catch { recordFailure(error, token: token) } + operations.unlock() + continue + } + + autoreleasepool { + producePacket(token: token) + } + } + } + + private func producePacket(token: UInt64) { + do { + let packet = try readSource { [self] in + condition.lock(); defer { condition.unlock() } + return token == sourceEpoch && !closed && !sourceRepositioning + } + guard let packet else { + condition.lock() + if token == sourceEpoch, !sourceRepositioning, !closed { + presentationCoverage?.finish() + ended = true + } + condition.broadcast() + condition.unlock() + return + } + let data = try packet.encoded() + operations.lock() + condition.lock() + let valid = token == sourceEpoch && !closed && !sourceRepositioning + condition.unlock() + if valid { + do { + let recordReservation = min(byteBudget, data.count) + + min(8, byteBudget - min(byteBudget, data.count)) + try fifo.trimConsumed(toByteBudget: byteBudget - recordReservation) + let cursor = try fifo.append(data) + let state = fifo.snapshot + condition.lock() + if token == sourceEpoch, !closed, !sourceRepositioning { + copyDiskStateLocked(state) + if packet.streamIndex == video.index { + if presentationCoverage != nil { + presentationCoverage?.insert(pts: packet.pts) + } else { + videoCoverage.insert(pts: packet.pts, duration: packet.duration) + } + if packet.flags & 1 != 0, packet.pts != Int64.min, + video.numerator > 0, video.denominator > 0 { + let seconds = Double(packet.pts) * Double(video.numerator) / Double(video.denominator) + if seconds.isFinite { keyframes.append(Keyframe(seconds: seconds, cursor: cursor)) } + if keyframes.count > maximumKeyframes { + keyframes.removeFirst(min(1024, keyframes.count)) + } + } + } else if packet.streamIndex == audio?.index { + audioCoverage.insert(pts: packet.pts, duration: packet.duration) + } + } + condition.broadcast() + condition.unlock() + } catch { recordFailure(error, token: token) } + } + operations.unlock() + } catch { recordFailure(error, token: token) } + } + + private func shouldParkLocked() -> Bool { + // One source record can cross the budget. A consumed active chunk cannot be deleted before + // cursor rollover, so residency also includes bounded protected chunk slack (not an + // unbounded batch). Unknown time coverage is never guessed from bitrate. + guard count > 0 else { return false } + return residentBytes >= byteBudget || (frontierLocked().map { $0 - sourceClock >= forwardSeconds } ?? false) + } + + private func recordFailure(_ error: Error, token: UInt64) { + condition.lock(); defer { condition.unlock() } + guard token == sourceEpoch, !closed else { return } + failure = error + resetPending = false + condition.broadcast() + } +} diff --git a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift index 688e7bb5b..c999c3571 100644 --- a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift +++ b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift @@ -105,6 +105,17 @@ final class SoftwarePlaybackHost { private var audioDecoder: AudioDecoder? private var audioOutput: AudioOutput? private var demuxer: Demuxer? + private var vodPacketReadAhead: SoftwarePacketReadAhead? + + /// The same public buffered-position axis as the live and native hosts, but backed by + /// actual compressed A/V packet coverage. nil when no continuous cache span contains the clock. + var cachedVODSessionTime: Double? { + guard let frontier = vodPacketReadAhead?.snapshot.frontier else { return nil } + return max(0, frontier - max(0, clockSessionZero)) + } + + var cachedVODBytes: Int64? { vodPacketReadAhead.map { Int64($0.snapshot.residentBytes) } } + var vodPacketCacheSnapshot: SoftwarePacketReadAhead.Snapshot? { vodPacketReadAhead?.snapshot } private let demuxQueue = DispatchQueue(label: "engine.sw.demux", qos: .userInitiated) @@ -316,6 +327,17 @@ final class SoftwarePlaybackHost { return SeekWindow.isOpen(requested: _seekGeneration, settled: _settledSeekGeneration) } + /// A single seek-state snapshot is needed for packet, EOF and error admission. Reading the + /// generation and window separately can straddle a seek that opens and settles between them. + nonisolated private func admitsRead(generation: UInt64) -> Bool { + feedLock.lock() + let requested = _seekGeneration + let settled = _settledSeekGeneration + feedLock.unlock() + return SoftwareReadAdmission.admits(readGeneration: generation, + requestedGeneration: requested, settledGeneration: settled, stopRequested: stopRequested) + } + nonisolated private func noteSeekSettled(_ generation: UInt64) { feedLock.lock() if SeekWindow.closes(settling: generation, live: _seekGeneration) { @@ -617,7 +639,8 @@ final class SoftwarePlaybackHost { startPosition: Double?, audioSourceStreamIndex: Int32?, isLive: Bool = false, - dvrWindowSeconds: Double? = nil + dvrWindowSeconds: Double? = nil, + forwardBufferSegments: Int? = nil ) async throws { self.demuxer = dem self.duration = dem.duration @@ -828,6 +851,52 @@ final class SoftwarePlaybackHost { initialClockTime = .zero } + if !isLive, dem.isSourceSeekable { + let video = SoftwarePacketReadAhead.Stream(index: videoStreamIndex, + numerator: vtb.num, denominator: vtb.den) + let audio: SoftwarePacketReadAhead.Stream? = audioStreamIndex >= 0 + ? dem.stream(at: audioStreamIndex).map { + .init(index: audioStreamIndex, numerator: $0.pointee.time_base.num, + denominator: $0.pointee.time_base.den) + } : nil + let initialSourceClock = initialClockTime.seconds + let videoReorderDepth: Int? = vCodecID == AV_CODEC_ID_H264.rawValue ? 32 : nil + let cacheResult = await Task.detached(priority: .utility) { () throws -> SoftwarePacketReadAhead? in + let temp = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true) + let available = (try? temp.resourceValues(forKeys: [.volumeAvailableCapacityKey]))? + .volumeAvailableCapacity.map(Int64.init) + let segments = HLSVideoEngine.clampedForwardWindow(forwardBufferSegments) + let bytes = HLSVideoEngine.sessionRetentionBudgetBytes( + volumeAvailableBytes: available, + capRelaxed: HLSVideoEngine.retentionCapRelaxed(forwardWindowSegments: segments)) + guard bytes > 0 else { return Optional.none } + let fifo = try SoftwarePacketDiskFIFO( + chunkTargetBytes: min(4 << 20, max(8, bytes)), retainConsumed: true) + return SoftwarePacketReadAhead( + video: video, audio: audio, byteBudget: bytes, + forwardSeconds: Double(segments) * 4, + initialSourceClock: initialSourceClock, fifo: fifo, + videoReorderDepth: videoReorderDepth + ) { isCurrent in + guard let packet = try dem.readPacket(isCurrent: isCurrent) else { return nil } + defer { av_packet_unref(packet); av_packet_free_safe(packet) } + return try SoftwareStoredPacket(copying: packet) + } + }.result + let readAhead: SoftwarePacketReadAhead? + switch cacheResult { + case .success(let cache): readAhead = cache + case .failure: + // A cache-directory failure must not stop a source that the old direct loop can + // still play. Runtime spool corruption is explicit, never silently skipped. + EngineLog.emit("[SWHost] packet cache unavailable; retaining direct playback", category: .swPlayback) + readAhead = nil + } + guard !stopRequested else { readAhead?.close(); return } + vodPacketReadAhead = readAhead + readAhead?.start() + } + startTimeUpdates() isReady = true } @@ -927,9 +996,11 @@ final class SoftwarePlaybackHost { lastAudioPts: demuxDiag.snapshot.lastAudioPts ) guard tail > 0 else { return parkClockNow() } + let generation = seekGeneration Task { @MainActor [weak self] in try? await Task.sleep(nanoseconds: UInt64(tail * 1_000_000_000)) - self?.parkClockNow() + guard let self, self.admitsRead(generation: generation) else { return } + self.parkClockNow() } } @@ -1001,11 +1072,17 @@ final class SoftwarePlaybackHost { /// that read (App Hangs of 4.4 s and 5.2 s in the field on a WAN source). @discardableResult func seek(to seconds: Double) async -> Demuxer.RepositionOutcome { + guard !stopRequested else { return .superseded } guard let dem = demuxer else { return .stalled } // Stop loop + bump generation to invalidate in-flight packets. Captured right after, so the // reposition can tell on `seekQueue` whether a newer seek has already taken over. bumpSeekGeneration() let generation = seekGeneration + didReachEnd = false + didParkClockAtEnd = false + didEmitParkedDiag = false + let packetSource = vodPacketReadAhead + let cacheGeneration = packetSource?.beginSeek(to: seconds) // #292: inside another seek's window `isPlaying` is that seek's parked flag, not the transport's // intent. Inherit what it captured, and hand the same value on to whoever supersedes this one. let wasPlaying = SeekResumeIntent.resolve(isPlaying: isPlaying, @@ -1046,9 +1123,50 @@ final class SoftwarePlaybackHost { videoDecoder.skipUntilPTS = targetTime renderer.setSkipThreshold(targetTime) - let outcome = await dem.seekBounded( - to: seconds, timeout: Self.seekBudgetSeconds, on: seekQueue, - isSuperseded: { [weak self] in self?.seekGeneration != generation }) + var cacheHit = false + if let packetSource, let cacheGeneration { + let preparation = await Task.detached(priority: .userInitiated) { + try packetSource.prepareSeek(cacheGeneration, to: seconds) + }.result + guard seekGeneration == generation, !stopRequested else { return .superseded } + switch preparation { + case .success(let hit): + cacheHit = hit + EngineLog.emit( + "[SWHost] packet cache seek generation=\(generation) " + + "result=\(hit ? "hit" : "miss") " + + "target_s=\(String(format: "%.3f", seconds)) " + + "resident_bytes=\(packetSource.snapshot.residentBytes)", + category: .swPlayback + ) + case .failure(SoftwarePacketReadAhead.ReadError.interrupted), + .failure(SoftwarePacketReadAhead.ReadError.closed): + return .superseded + case .failure: + // A corrupt/unreadable retained packet must not be silently skipped or treated as + // a normal cache miss. Leave playback parked and publish an explicit failure. + seekInFlight = false + packetSource.close() + noteSeekSettled(generation) + EngineLog.emit("[SWHost] packet cache seek generation=\(generation) result=error", + category: .swPlayback) + failure = PlaybackErrorInfo(kind: .softwarePipelineFailed, + message: "Playback cache could not reposition.") + return .stalled + } + } + let outcome: Demuxer.RepositionOutcome + if cacheHit { + // The decode cursor now points at retained preroll. The source reader remains at its + // existing frontier; seeking it too would duplicate or skip already retained packets. + outcome = .landed + } else { + outcome = await dem.seekBounded( + to: seconds, timeout: Self.seekBudgetSeconds, on: seekQueue, + isSuperseded: { [weak self] in + self?.seekGeneration != generation || (self?.stopRequested ?? true) + }) + } // A newer seek owns the state from here: it published its own target and clears the hold itself. // stop() can also land in the await now that this suspends, and re-arming the clock or flipping // isPlaying on a torn-down session would resurrect a loop that has already been told to quit. @@ -1087,6 +1205,7 @@ final class SoftwarePlaybackHost { // The source stands at the target and the clock is anchored on it: everything the loop // reads from here belongs to this position. Closing the window releases the loop. noteSeekSettled(generation) + if let cacheGeneration { packetSource?.endSeek(cacheGeneration, sourceClock: seconds) } return outcome } @@ -1188,6 +1307,8 @@ final class SoftwarePlaybackHost { noteSeekSettled(seekGeneration) timeTimer?.cancel() timeTimer = nil + vodPacketReadAhead?.close() + vodPacketReadAhead = nil renderer.subtitleCompositor.reset() dvrRing?.close() @@ -1262,22 +1383,31 @@ final class SoftwarePlaybackHost { let subIndices = subtitleStreamIndices let subTimeBases = subtitleStreamTimeBases let subSplitSetIndices = splitDisplaySetSubtitleStreamIndices - let onError: @Sendable (String) -> Void = { [weak self] msg in + let onErrorForGeneration: @Sendable (String, UInt64) -> Void = { [weak self] msg, generation in Task { @MainActor [weak self] in - self?.failure = PlaybackErrorInfo(kind: .softwarePipelineFailed, message: msg) + guard let self, self.admitsRead(generation: generation) else { return } + self.failure = PlaybackErrorInfo(kind: .softwarePipelineFailed, message: msg) } } - let onEnd: @Sendable () -> Void = { [weak self] in - // AE#374: the diagnostic line has to say the producer is done, or a falling aLead over a - // frozen audio PTS reads exactly like drift on a running session. - self?.demuxDiag.markSourceExhausted() + let onEndForGeneration: @Sendable (UInt64) -> Void = { [weak self] generation in Task { @MainActor [weak self] in - guard let self else { return } + guard let self, self.admitsRead(generation: generation) else { return } + // Only current EOF may mark the diagnostic source exhausted or stop playback. + // A queued pre-seek EOF task must not end a freshly positioned generation. + self.demuxDiag.markSourceExhausted() self.parkClockAtEndOfMedia() self.didReachEnd = true self.isPlaying = false } } + let onError: @Sendable (String) -> Void = { [weak self] message in + guard let self else { return } + onErrorForGeneration(message, self.seekGeneration) + } + let onEnd: @Sendable () -> Void = { [weak self] in + guard let self else { return } + onEndForGeneration(self.seekGeneration) + } // Live + DVR ring: reader/feeder split; live-only and VOD use the combined loop below. let getClockArmed: @Sendable () -> Bool = { [weak self] in @@ -1289,6 +1419,9 @@ final class SoftwarePlaybackHost { let getSeekGeneration: @Sendable () -> UInt64 = { [weak self] in self?.seekGeneration ?? 0 } + let admitsRead: @Sendable (UInt64) -> Bool = { [weak self] generation in + self?.admitsRead(generation: generation) ?? false + } let getSeekWindowOpen: @Sendable () -> Bool = { [weak self] in self?.seekWindowOpen ?? false } @@ -1375,9 +1508,11 @@ final class SoftwarePlaybackHost { } let diag = demuxDiag + let readAhead = vodPacketReadAhead demuxQueue.async { Self.runDemuxLoop( demuxer: dem, + readAhead: readAhead, videoDecoder: vDec, videoStreamIndex: vIdx, audioDecoder: aDec, @@ -1401,12 +1536,13 @@ final class SoftwarePlaybackHost { markClockArmed: setClockArmed, onClockAnchored: onClockAnchored, seekGeneration: getSeekGeneration, + admitsRead: admitsRead, seekWindowOpen: getSeekWindowOpen, setDecodeGeneration: setDecodeGeneration, noteDecodeGeneration: noteDecodeGeneration, backgroundAudioOnly: getBackgroundAudioOnly, - onError: onError, - onEnd: onEnd, + onError: onErrorForGeneration, + onEnd: onEndForGeneration, audioTapSink: getAudioTapSink, subtitleStreamIndices: subIndices, subtitleTimeBases: subTimeBases, @@ -1878,6 +2014,7 @@ final class SoftwarePlaybackHost { /// Demux loop: reads packets, dispatches by stream index, back-pressures against renderer's isReadyForMoreMediaData, flushes decoders at EOF. nonisolated private static func runDemuxLoop( demuxer: Demuxer, + readAhead: SoftwarePacketReadAhead?, videoDecoder: any VideoDecodingPipeline, videoStreamIndex: Int32, audioDecoder: AudioDecoder?, @@ -1900,12 +2037,13 @@ final class SoftwarePlaybackHost { markClockArmed: @Sendable () -> Void, onClockAnchored: @Sendable (Double) -> Void, seekGeneration: @Sendable () -> UInt64, + admitsRead: @Sendable (UInt64) -> Bool, seekWindowOpen: @Sendable () -> Bool, setDecodeGeneration: @Sendable (UInt64) -> Void, noteDecodeGeneration: @Sendable () -> Void, backgroundAudioOnly: @Sendable () -> Bool, - onError: @Sendable (String) -> Void, - onEnd: @Sendable () -> Void, + onError: @Sendable (String, UInt64) -> Void, + onEnd: @Sendable (UInt64) -> Void, audioTapSink: @Sendable () -> ((@Sendable (CMSampleBuffer) -> Void)?), subtitleStreamIndices: Set = [], subtitleTimeBases: [Int32: AVRational] = [:], @@ -1947,6 +2085,7 @@ final class SoftwarePlaybackHost { // pump get it from the DVR feeder arm on its own thread. var parkedVideo: [UnsafeMutablePointer] = [] let parkedVideoCap = 256 + var terminalGeneration = SoftwareTerminalGeneration() var lastEnqueuedAudioPtsSec = Double.nan var rebuffering = false // The pause/rebuffer arm stays off until the source has proven it can deliver a real @@ -2125,6 +2264,19 @@ final class SoftwarePlaybackHost { } func demuxIteration() -> Bool { + // A valid EOF/error can be queued for MainActor just before a newer seek supersedes + // it. Keep the VOD consumer parked (without duplicate callbacks), not permanently + // exited, so that newer generation still has a loop to consume its retained packets. + if !isLive, terminalGeneration.shouldPark(generation: seekGeneration()) { + condition.lock() + while terminalGeneration.shouldPark(generation: seekGeneration()), !stopRequested() { + autoreleasepool { + _ = condition.wait(until: Date(timeIntervalSinceNow: 0.5)) + } + } + condition.unlock() + return !stopRequested() + } // AE#491 round 2: the seek window is part of this park, not a second one. Keyed on it, // the loop stands still from the generation bump until the source and the clock are // both at the target, so nothing it does can be measured against, or fed from, the @@ -2176,37 +2328,43 @@ final class SoftwarePlaybackHost { var epochBeforeRead = videoDecoder.feedEpoch let packet: UnsafeMutablePointer? do { - packet = try demuxer.readPacket() + if let readAhead { + packet = try readAhead.read(isCurrent: { admitsRead(genBeforeRead) })?.makeAVPacket() + } else { + packet = try demuxer.readPacket(isCurrent: { admitsRead(genBeforeRead) }) + } } catch { + // Stop/seek cancellation is not a playback failure, including a normal .closed + // returned by the packet source during teardown. Errors belong to the read's + // captured host generation, just like packets and EOF. + guard admitsRead(genBeforeRead) else { return !stopRequested() } + if case SoftwarePacketReadAhead.ReadError.interrupted = error { return true } EngineLog.emit("[SWHost] demux read failed: \(error)", category: .swPlayback) - onError("Playback error: \(error.localizedDescription)") - return false + if terminalGeneration.record(genBeforeRead) { + onError("Playback error: \(error.localizedDescription)", genBeforeRead) + } + return !isLive } + // Admit every result BEFORE interpreting nil as EOF. Otherwise an old read can drain + // or end the new seek generation without ever reaching the old packet-only gate. + guard admitsRead(genBeforeRead) else { + if let packet { av_packet_unref(packet); av_packet_free_safe(packet) } + return !stopRequested() + } guard let packet else { // Play the parked tail out before the flush: nothing more will arrive, the queues // must drain to end-of-media. The wait lifts a rebuffer hold on its own - held, // the clock would never take the tail and end-of-media would never be reached. releaseRebufferHold("end of media, nothing left to rebuffer from") waitForRenderer(.drainAll) + guard admitsRead(genBeforeRead) else { return !stopRequested() } freeParkedVideo() videoDecoder.flush() audioDecoder?.flush() renderer.drainReorderBuffer() - onEnd() - return false - } - - // Stale packet from before seek flush: decoding would clear the skip threshold (visible - // fast-forward burst). Discard. AE#491 round 2: the generation alone does not say it, - // because a read that starts INSIDE the window carries the new one and is stale all the - // same; see `SeekWindow.admitsPacket`. - if !SeekWindow.admitsPacket(readGeneration: genBeforeRead, - liveGeneration: seekGeneration(), - windowOpen: seekWindowOpen()) { - av_packet_unref(packet) - av_packet_free_safe(packet) - return true + if terminalGeneration.record(genBeforeRead) { onEnd(genBeforeRead) } + return !isLive } let streamIdx = packet.pointee.stream_index @@ -2481,6 +2639,7 @@ final class SoftwarePlaybackHost { let raw = aOut.currentTimeSeconds self.emitDiagIfDue(clock: raw) if raw.isFinite, raw >= 0 { + self.vodPacketReadAhead?.updatePlayhead(raw) // Raw clock = source/subtitle axis; published alongside the mapped position (#107). self.sourceClockSeconds = raw // Live: subtract sessionStartPts to convert to "seconds since first frame"; VOD @@ -2673,6 +2832,7 @@ final class SWPlaybackDiagState: @unchecked Sendable { if generation >= _audioFlushGeneration { _audioFlushGeneration = generation _lastAudioPts = .nan + _sourceExhausted = false } lock.unlock() } diff --git a/Sources/AetherEngine/Native/SoftwareReadAdmission.swift b/Sources/AetherEngine/Native/SoftwareReadAdmission.swift new file mode 100644 index 000000000..6ed97862f --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwareReadAdmission.swift @@ -0,0 +1,30 @@ +import Foundation + +/// Packet payload, EOF, errors and deferred end-of-media work all belong to the same read +/// generation. The host obtains requested/settled generations together under its seek-state lock. +/// A pause is not a new generation; stop or an open/completed newer seek rejects old work. +enum SoftwareReadAdmission { + static func admits(readGeneration: UInt64, requestedGeneration: UInt64, + settledGeneration: UInt64, stopRequested: Bool) -> Bool { + !stopRequested && readGeneration == requestedGeneration + && requestedGeneration == settledGeneration + } +} + +/// The combined VOD loop stays alive after one EOF/error publication. MainActor may reject that +/// publication when a seek supersedes it; a permanently exited loop could not serve the new seek. +/// The owning loop uses its condition variable while parked, waking on stop or a new generation. +struct SoftwareTerminalGeneration { + private var terminal: UInt64? + + /// true exactly once per terminal generation; prevents duplicate EOF/error callbacks. + mutating func record(_ generation: UInt64) -> Bool { + guard terminal != generation else { return false } + terminal = generation + return true + } + + func shouldPark(generation: UInt64) -> Bool { + terminal == generation + } +} diff --git a/Sources/AetherEngine/Native/SoftwareStoredPacket+FFmpeg.swift b/Sources/AetherEngine/Native/SoftwareStoredPacket+FFmpeg.swift new file mode 100644 index 000000000..74f36f9f4 --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwareStoredPacket+FFmpeg.swift @@ -0,0 +1,55 @@ +import Foundation +import AetherLibavcodec +import AetherLibavutil + +extension SoftwareStoredPacket { + enum PacketError: Error { case invalidPacket, allocationFailed } + + init(copying packet: UnsafeMutablePointer) throws { + let p = packet.pointee + guard p.size >= 0, p.side_data_elems >= 0, + p.size == 0 || p.data != nil, + p.side_data_elems == 0 || p.side_data != nil else { throw PacketError.invalidPacket } + var sides: [SideData] = [] + for index in 0.. UnsafeMutablePointer { + guard bytes.count <= Int(Int32.max), let packet = trackedPacketAlloc() else { + throw PacketError.allocationFailed + } + do { + guard av_new_packet(packet, Int32(bytes.count)) >= 0 else { throw PacketError.allocationFailed } + if !bytes.isEmpty { bytes.copyBytes(to: packet.pointee.data, count: bytes.count) } + packet.pointee.pts = pts + packet.pointee.dts = dts + packet.pointee.duration = duration + packet.pointee.pos = position + packet.pointee.stream_index = streamIndex + packet.pointee.flags = flags + packet.pointee.time_base = AVRational(num: timeBaseNumerator, den: timeBaseDenominator) + for side in sideData { + guard let target = av_packet_new_side_data( + packet, AVPacketSideDataType(rawValue: side.type), side.bytes.count) else { + throw PacketError.allocationFailed + } + side.bytes.copyBytes(to: target, count: side.bytes.count) + } + return packet + } catch { + av_packet_unref(packet) + av_packet_free_safe(packet) + throw error + } + } +} diff --git a/Sources/AetherEngine/Native/SoftwareStoredPacket.swift b/Sources/AetherEngine/Native/SoftwareStoredPacket.swift new file mode 100644 index 000000000..0a8c57a7d --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwareStoredPacket.swift @@ -0,0 +1,30 @@ +import Foundation + +/// Lossless packet envelope for the software VOD read-ahead store. No decoded pixels or +/// source URL is retained. In particular DTS, duration and side data must survive a cache hit. +struct SoftwareStoredPacket: Codable, Sendable, Equatable { + struct SideData: Codable, Sendable, Equatable { + let type: UInt32 + let bytes: Data + } + let pts: Int64 + let dts: Int64 + let duration: Int64 + let position: Int64 + let streamIndex: Int32 + let flags: Int32 + let timeBaseNumerator: Int32 + let timeBaseDenominator: Int32 + let bytes: Data + let sideData: [SideData] + + func encoded() throws -> Data { + let encoder = PropertyListEncoder() + encoder.outputFormat = .binary + return try encoder.encode(self) + } + + static func decode(_ data: Data) throws -> Self { + try PropertyListDecoder().decode(Self.self, from: data) + } +} diff --git a/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift b/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift new file mode 100644 index 000000000..b09b9fa59 --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift @@ -0,0 +1,133 @@ +import Foundation + +/// H.264 compressed-packet presentation coverage matching the renderer's successor-PTS duration. +/// +/// AVPacket.duration may describe decode cadence rather than how long a variable-rate picture is +/// displayed. The renderer holds a picture until its presentation-order successor (positive gaps +/// up to one second). This model publishes only those successor intervals, without rewriting any +/// packet timestamp or duration. A larger gap remains a discontinuity, not a tolerance to bridge. +/// +/// H.264-only caller contract: complete selected-stream packets, with valid presentation timestamps, +/// enter in demux/decode order after they are retained in the packet store. FFmpeg n8.1.2 h264_ps.c +/// rejects num_reorder_frames > 16; the default holds 32 distinct timestamps as a conservative +/// field-picture margin. That codec picture bound does NOT prove arbitrary container PTS obey the +/// bound. A timestamp behind the emitted watermark therefore invalidates all coverage until reset. +/// Source: https://github.com/FFmpeg/FFmpeg/blob/n8.1.2/libavcodec/h264_ps.c#L174-L180 +/// +/// This metadata-only value neither owns packets nor synchronizes access. The owner must reset it +/// together with its packet store on seek/flush/discard. Other codecs must keep their strict packet +/// duration coverage unless their presentation reordering has a separately established bound. +struct SoftwareVideoPacketCoverage: Sendable { + private var coverage = SoftwarePacketCoverage() + private var pending: [Int64] = [] + private var emittedPTS: Int64? + private let timeBaseNumerator: Int32 + private let timeBaseDenominator: Int32 + private let validConfiguration: Bool + let reorderDepth: Int + + private(set) var isInvalidated: Bool + private(set) var lateTimestampCount = 0 + private(set) var isFinished = false + + init(timeBaseNumerator: Int32, timeBaseDenominator: Int32, reorderDepth: Int = 32) { + self.timeBaseNumerator = timeBaseNumerator + self.timeBaseDenominator = timeBaseDenominator + self.reorderDepth = min(32, max(1, reorderDepth)) + validConfiguration = timeBaseNumerator > 0 && timeBaseDenominator > 0 + && (1...32).contains(reorderDepth) + isInvalidated = !validConfiguration + } + + var pendingCount: Int { pending.count } + var rangeCount: Int { coverage.rangeCount } + + /// Duration/DTS are deliberately not inputs. Duplicate pending PTS do not advance the reorder + /// watermark. At most depth + 1 timestamps are held transiently, then the minimum is emitted. + @discardableResult + mutating func insert(pts: Int64) -> Bool { + guard !isInvalidated, !isFinished else { return false } + guard pts != Int64.min else { invalidate(); return false } + if let emittedPTS { + if pts < emittedPTS { + lateTimestampCount += 1 + invalidate() + return false + } + if pts == emittedPTS { return true } + } + + var lower = 0 + var upper = pending.count + while lower < upper { + let middle = lower + (upper - lower) / 2 + if pending[middle] < pts { lower = middle + 1 } else { upper = middle } + } + if lower < pending.count, pending[lower] == pts { return true } + pending.insert(pts, at: lower) + if pending.count > reorderDepth { return emit(pending.removeFirst()) } + return true + } + + /// Call only after the complete selected-stream input reaches EOF. The final picture has no + /// verified successor: its end remains unknown, regardless of packet duration or movie length. + mutating func finish() { + guard !isInvalidated, !isFinished else { return } + while !pending.isEmpty { + if !emit(pending.removeFirst()) { break } + } + isFinished = true + } + + func frontier(containing tick: Int64) -> Int64? { + guard !isInvalidated else { return nil } + return coverage.frontier(containing: tick) + } + + func frontierSeconds( + containing seconds: Double, + timeBaseNumerator: Int32, + timeBaseDenominator: Int32 + ) -> Double? { + guard !isInvalidated, timeBaseNumerator == self.timeBaseNumerator, + timeBaseDenominator == self.timeBaseDenominator else { return nil } + return coverage.frontierSeconds(containing: seconds, timeBaseNumerator: timeBaseNumerator, + timeBaseDenominator: timeBaseDenominator) + } + + mutating func prune(before tick: Int64) { + coverage.prune(before: tick) + } + + mutating func reset() { + coverage.reset() + pending.removeAll(keepingCapacity: true) + emittedPTS = nil + lateTimestampCount = 0 + isInvalidated = !validConfiguration + isFinished = false + } + + private mutating func emit(_ pts: Int64) -> Bool { + if let previous = emittedPTS { + let (delta, overflow) = pts.subtractingReportingOverflow(previous) + guard !overflow, delta > 0 else { invalidate(); return false } + // Exact rational comparison against the renderer's one-second policy. Division before + // comparison avoids overflowing delta * numerator, and never rounds a long gap down. + let maximumHoldTicks = Int64(timeBaseDenominator) / Int64(timeBaseNumerator) + if delta <= maximumHoldTicks, !coverage.insert(pts: previous, duration: delta) { + invalidate() + return false + } + } + emittedPTS = pts + return true + } + + private mutating func invalidate() { + isInvalidated = true + coverage.reset() + pending.removeAll(keepingCapacity: true) + emittedPTS = nil + } +} diff --git a/docs/api.md b/docs/api.md index f4df16569..797c51694 100644 --- a/docs/api.md +++ b/docs/api.md @@ -659,6 +659,7 @@ as well. | Symbol | Notes | | --- | --- | | `diagnostics.liveTelemetry` | 1 Hz `LiveTelemetry?` snapshot while playing or paused, nil while idle. On a separate `ObservableObject` so its ticks cannot re-render a host observing the engine. | +| `LiveTelemetry.softwareCacheSeekHits`, `softwareCacheSeekMisses`, `softwareCacheSourceEpoch` | Optional cumulative software-VOD packet-cache counters. A hit repositions the retained consumer cursor without changing the source epoch; a miss repositions the demuxer and advances it. `nil` on other paths. `cachedBytes` includes retained compressed packet records on software VOD, distinct from decoded `displayCushionSeconds` and the underlying byte-reader window. | | `EngineLog.handler` | Mirror every info-level line into a host capture path. Fires from whatever thread emitted it, so it must be thread-safe and non-blocking. | | `EngineLog.subsystem`, `EngineLog.Category` | `de.superuser404.AetherEngine`, one category per subsystem: `engine`, `ffmpeg`, `session`, `muxer`, `demux`, `hls.server`, `audio.bridge`, `sw.playback`, `scrub`. | | `EngineLog.Level` | `.info` reaches os_log and the host handler; `.verbose` is per-segment trace and reaches os_log's debug level **only**, never the handler, which is what keeps a mirrored stream readable. Read the verbose ones with `log stream --level debug`. | @@ -704,7 +705,7 @@ All flags default to safe values; the table is the full set. Depth for the media | `deinterlaceMode` | `.auto` | A `DeinterlaceMode` for the software path: the Metal / VideoToolbox graph with a CPU bwdif fallback, or `.software` to force the CPU path. | | `deinterlaceFieldRate` | `.field` | A `DeinterlaceFieldRate`: the hardware deinterlacer emits one frame per field (25i to 50p) or per frame. The software fallback is always frame rate, because doubling a CPU bwdif is the wrong trade and a fallback should not change cost class. | | `probesize`, `maxAnalyzeDuration` | nil | Caller-bounded open-time probe budget (defaults 50 MB / 60 s). They fail **open**: an over-tight budget loads with late-resolving tracks silently missing rather than throwing, so validate track presence if you tighten them. Do not pass `0` for `maxAnalyzeDuration`; FFmpeg maps it to a shorter heuristic. | -| `forwardBufferSegments` | nil (10, about 40 s) | How far the producer may race ahead and how much the cache keeps resident. Clamped to 4...2700; past the historical 150 the real bound is the session's disk budget, so a "buffer without limit" option can pass `Int.max`. Ignored on `nativeRemoteHLS`. What it actually retains is readable as `$residentRanges` (see Time). | +| `forwardBufferSegments` | nil (10, about 40 s) | How far the producer may race ahead and how much the cache keeps resident. Clamped to 4...2700; past the historical 150 the real bound is the session's disk budget, so a "buffer without limit" option can pass `Int.max`. Ignored on `nativeRemoteHLS`. Native HLS retention is readable as `$residentRanges` (see Time). Software VOD uses the same forward-window and volume-budget policy for compressed packet read-ahead; its continuous selected A/V frontier is `bufferedPosition`, not native `residentRanges`. | | `sequentialOrigin` | false | Declare an origin that fabricates range answers: one long-lived unranged GET, no ranged probes, non-seekable pb. **Seeking is unavailable**; re-request the archive at a shifted start instead. | | `declaredDurationSeconds` | nil | Trusted duration, overriding the container's. Required alongside `sequentialOrigin` on VOD, where the tail read is gone. | | `maxConcurrentSourceRequests` | nil | Most requests the reader may have open against this origin at once, across every path it fetches on (pump ranges, detour blocks, size probes, tail prefetch, subtitle side reader). nil counts without capping and lowers the ceiling on its own after a 429/503/509. Set it when the provider states a limit; `1` also switches off the speculative parallel paths, which exist only to overlap with the pump. Counts **requests**, not TCP connections, because over HTTP/2 a session multiplexes every request onto one connection while the origin still counts each one (AE#377). It is also the only ceiling: several engines playing from one origin are bounded by this value and by what the origin refuses, not by a transport pool underneath it (AE#450). | diff --git a/docs/architecture.md b/docs/architecture.md index ae2b86d93..d250f5d18 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,5 +1,50 @@ # Architecture +## Software VOD compressed packet cache + +Seekable software VOD has a compressed packet producer separate from the +renderer-paced decode consumer. The producer stores lossless packet envelopes in +a session-owned temporary disk FIFO. Envelopes preserve payload, PTS/DTS, packet +duration, position, flags, stream index, time base and every side-data entry; +neither decoded pictures nor source URLs are retained in metadata. + +The host uses the existing `forwardBufferSegments` clamp and native session +retention/volume-safety policy. Byte and forward-time thresholds stop prefetch, +with bounded protected-chunk and single-record slack. Old consumed chunks can +be reclaimed at the exact budget boundary so refill cannot deadlock. Metadata +reads and seek intent are main-thread safe; disk operations and source reads +stay on workers. Stop releases the session's directory, and bounded stale +cleanup uses session leases without following symlinks or deleting live stores. +If creating the store fails, the original direct playback loop remains available. +Runtime cache corruption is a reported playback failure, not silently skipped data. + +The cache frontier is the intersection of selected audio and video presentation +coverage containing the playhead. Unknown intervals remain unknown; byte counts +are not converted to guessed seconds. H.264 uses a bounded presentation reorder +queue and confirmed successor timestamps, because a VFR packet's decode duration +can be shorter than the picture's actual display hold. Larger discontinuities, +invalid or unexpectedly late timestamps invalidate or split coverage. Other +codecs retain strict packet-duration coverage. Without a proven compressed +frontier, the existing decoded-cushion fallback still applies. + +A cached seek restores a retained keyframe cursor, including an earlier keyframe +for available preroll, and keeps the producer at its existing source frontier. +A cache miss clears coverage and repositions the demuxer. Consumer generations +and source epochs are separate: a cached seek must not discard an in-flight +producer packet, and a superseded consumer must not steal the first packet of a +new source epoch. Admission applies equally to packet, EOF, error and delayed +end-of-media callbacks. At EOF the VOD consumer parks until a new seek or stop. + +This changes engine cache semantics, not host UI. Software `bufferedPosition` +and `LiveTelemetry.cachedBytes` describe compressed coverage/residency; +`forwardBufferSeconds` remains the native player's loaded-range metric and +`displayCushionSeconds` still describes the small decoded queue. Native HLS +`residentRanges` and live DVR are not repurposed. + +The CI packet-cache step runs the standalone coverage, VFR successor, disk FIFO, +read-ahead concurrency, host admission and AVPacket-envelope regressions. These +use generated numeric data and temporary records, not private video fixtures. + How AetherEngine is put together: the three playback pipelines, the source-file map, and the dependency surface. For the public API and integration, see the [README](../README.md); for format and codec depth, [docs/formats.md](formats.md). ## Playback pipelines