fix(video): route recovery-point H264 VOD through seek-compatible decoding

[orut34iop]

Summary

Companion submissions, independently reviewable against main:

• #510 — recovery-point VOD compatibility
• #511 — Matroska H.264 timestamp ownership
• #512 — retained software VOD cache

Route positively identified H.264 recovery-point VOD through the existing software path to avoid persistent native/HLS judder after seeking. This is a narrowly detected compatibility route, not a native decoder repair or the missing-ctts problem from #409.

What changed

• A bounded probe requires three container-key packets that have non-IDR slices and immediate/exact recovery-point SEI. Missing IDR alone, malformed evidence and ordinary IDR streams do not trigger it.
• Rewind the reused demuxer, check rewind success and load-generation ownership; skip live, non-seekable, non-H.264 and already-software sessions.
• Document the optional diagnostics.h264RecoveryPointKeyCount result, add recognition/negative tests and a documented-threshold pin.
• Independent patch on main 1af43f017ffca973e35f8fec1e700852f2953d55; no downstream UI, diagnostic framework, dependency pins or unrelated fork history.

Evidence and limitations

Three private MP4/H.264 sources show repeated recovery-point keys (NAL 1/6, not IDR 5), normal 1001-tick packet cadence at time base 1/30000, and persistent judder after seeks on the native path. Representative steady-state measurements:

Native measurement	Before seek	After seek
AVPlayer measured video rate	median 30.460 fps	median 3.000 fps; 2.503–3.325
Buffer ahead	healthy	5.447–7.543 s
Item-clock / wall-clock progression	normal	median 1.000019
Display-link callbacks	normal	about 59.941 Hz

These are access-log/clock observations, not pixel-level frame counting. The inspected native segments had 120 samples, 10 container keys and zero IDR NALs. VidHub played the same sources normally; we do not know its internal route.

A generated open-GOP bitstream verifies the SEI shape but has not been shown to reproduce the native Apple TV judder. Private assets, paths, URLs, credentials, raw logs and device identifiers are not included. This proposal changes CPU/power cost for detected sources and does not claim every non-IDR stream is incompatible with Apple hardware. A native random-access fix could supersede it.

Test plan

• Device / OS: physical Apple TV 4K (3rd generation), tvOS 26.6 beta, build 23L773.
• Media: three private MP4s, H.264 Main, 1280×720, 8-bit yuv420p, nominal 30000/1001 fps, video time base 1/30000; AAC-LC 44.1 kHz stereo. No HDR/DV signaling reported.
• Physical result: user verified all three after the downstream compatibility change, including seeking and continued playback. Accepted downstream core is 99f8c23c74884ec7d62810df9c6b9be31939f484. The upstream-isolated branch was not separately installed.
• Local: bash Scripts/test-h264-recovery-point.sh passed (exact/delayed recovery, IDR exclusion, container flags, malformed/oversized input, repeated evidence).
• Local: arm64 tvOS package build passed with Xcode 26.6; python3 Scripts/check-doc-links.py and git diff --check passed.
• Added Swift Testing cases and DocumentedConstantsTests coverage run in upstream CI; the complete Swift package test suite was not run locally.
• Retained software cache and Matroska timestamp repair are separate submissions, not prerequisites for this route.

Checklist

• CHANGELOG.md updated
• Conventional Commit
• Engine fix; no host-side UI workaround
• Public API change intentional and documented

[superuser404notfound]

Thanks for the detailed writeup and for keeping the submissions separately reviewable. The judder you measured is real, but I cannot adopt this route: the feature it keys on is not a defect, and I have a counterexample that the branch itself misroutes.

The counterexample

x264 --open-gop produces exactly the shape this PR describes: container keys carrying non-IDR slices with recovery_frame_cnt=0 and exact_match_flag=1, and no IDR beyond the first. Two fixtures, identical in every other respect:

ffmpeg -f lavfi -i testsrc=size=1280x720:rate=30000/1001:duration=120 \
  -c:v libx264 -profile:v main -preset veryfast -b:v 3M -pix_fmt yuv420p \
  -x264-params "open-gop=1:bframes=3:keyint=30:min-keyint=30:scenecut=0:b-pyramid=normal" \
  -movflags +faststart opengop.mp4
# same line with open-gop=0 for the control

Open GOP: 120 container keys, 2 IDR NALs, 121 recovery point SEIs. Control: 120 container keys, 121 IDR NALs, 2 SEIs. trace_headers confirms recovery_frame_cnt=0 and exact_match_flag=1 on every recovery point, so this is the exact evidence the probe collects.

Serving both through the loopback path and decoding each segment IN ISOLATION through Apple's decoder (init.mp4 plus one segment, AVAssetReader, fresh decode, no predecessor):

segment	open GOP	control
seg1 to seg7	120 of 120 frames, span 3.971 s	120 of 120 frames, span 3.971 s

No loss, no error, status=completed on every one. libavcodec is the arm that struggles here, not VideoToolbox: segverify decodes 110 of 120 on the open GOP arm against 111 of 120 on the control, with mmco: unref short failure on each open GOP segment. So a segment opened on a non-IDR immediate/exact recovery point is fully decodable by the Apple decoder, which is what this route assumes it is not.

The branch routes that file to software

Built from this branch, against the two fixtures above:

open GOP:  H264 recovery-point sample: video=90  recoveryKeys=3 compatibility=true
           dispatch: codec=27 -> software
control:   H264 recovery-point sample: video=180 recoveryKeys=0 compatibility=false
           dispatch: codec=27 -> native

Every x264 --open-gop H.264 VOD file would lose hardware decode. That is a large class of ordinary content, and on the measurement above none of it needs the detour.

The control line also shows the cost on the healthy path: the probe reads its full 180 video packet budget before it can say no, on every H.264 VOD session, and it is a second rewind of the reused demuxer on top of what the session already does. At 10 Mbit/s that is roughly 7 MB read before the first frame. The existing InterlaceProbe only runs inside a narrow declared-interlace case, which is why it can afford the same budget.

Three smaller things

• guard rewound else { throw DemuxerError.readFailed(code: -5) } turns a failed rewind into a failed load for a session that would previously have played. Checking the rewind is right; the fallback should be a reopen or an unprobed continue, not an error class that did not exist before.
• The 3 second deadline is only evaluated between reads, so a single readPacket on a slow transport is unbounded. The PR text is honest about this, but the docs/formats.md hunk in the diff presents it as a wall clock limit on the sample.
• diagnostics.h264RecoveryPointKeyCount is public API for a probe counter no host renders. The EngineLog line already carries it, and public API is the one thing that cannot be walked back later.

What I think is actually happening

The shape you measured (healthy from the head at 30.460 fps, 3.000 fps after a seek, item clock at 1.000019, buffer full at 5.4 to 7.5 s, no aggregate drops) is the shape #513 describes and explains: valid composition offsets at the head, zero offsets in later sequences, and a healthy-head verdict that disables repair for the whole session. A seek into the malformed region is then the trigger, with the decoder never at fault. Your measured 2.503 to 3.325 fps also matches the keyframe rate of your own segments (10 container keys per 120 samples), which is what "only the random access points survive" looks like, and not what "this segment cannot be decoded" looks like.

That is a hypothesis, not a measurement, and it is testable on your side without publishing anything:

1. aetherctl segverify --dump <dir> on one of the three sources, then decode init.mp4 plus a single dumped segment from past the seek point through AVAssetReader and count frames. If that comes out complete, the segment is not the problem and this route is not the fix.
2. aetherctl pktdump --at <seek target> --count 300 on the same source, and check whether PTS equals DTS there while it does not at the head. That is the fix(video): repair partial H.264 MP4 composition offsets #513 signature.

Worth naming that the three sources were only ever verified as part of the downstream core, with all four changes present at once. If #513 repairs them, this route is unnecessary; if it does not, the pktdump above says which of the four did the work.

One unrelated fix came out of reviewing this: segverify had been fetching init.mp4 from the server root instead of the session path prefix, so the #92 ground truth verifier had not completed a single run since that prefix was introduced. Fixed in 92348e3e, which is what made the isolation table above possible.

Caveat on my side: the isolation decode is macOS with AVAssetReader, not tvOS with AVPlayer over HLS. It answers whether the segment is independently decodable by Apple's decoder, which is the claim this route rests on, and it does not rule out a tvOS specific AVPlayer behaviour layered on top of that.

[orut34iop]

Follow-up: native Apple TV reproduction (2026-09-08)

@superuser404notfound Thanks for the concrete counterexample. I agree that recovery-point coding alone is not a defect and cannot justify automatically routing every matching stream to software. The healthy open-GOP fixture must be a negative routing control. I also agree that startup probe cost, rewind-failure behavior, the between-read deadline and the public diagnostic counter need addressing; the evidence below is not a request to merge this PR unchanged.

The new results and their limitations are recorded in this sanitized, commit-pinned test report. It contains the exact candidate identity, actual playback sequence, numerical measurements, fragment hashes and source comparison. No movie bytes, private source names/URLs, device identifiers or raw logs are published.

1. Native playback still reproduces the problem with the other repairs present

On the same physical Apple TV, we tested the first original affected source using diagnostic engine cd3e80d3, app build 20260908012500. It retains the accepted downstream repairs, including partial-composition repair, and retains the recovery probe/rewind; only that probe's automatic software selection is bypassed. A bounded diagnostic init capture was added to pair the existing native cached fragments. Host UI and seek handling were unchanged. This is a downstream measurement build, not an isolated installation of the upstream PR branch.

Measurement	From head	After the actual 704.807-second seek
Active native track-rate samples	17	56 over a two-minute wall window
Track fps, min / median / max	29.616 / 30.271 / 31.175	2.463 / 2.862 / 3.000
Loaded range ahead	4.697–6.733 s	4.049–6.955 s

The tester confirmed visible judder. Pause/resume did not restore the rate. The native backend and three recovery keys were recorded throughout active playback; post-seek rate was 1, the buffer was not empty, and presentation-axis shift was 0. Access-log stall/drop counters stayed zero. Track rate means AVPlayerItemTrack.currentVideoFrameRate, not an optical frame count, and those zero counters are not evidence of smooth output. The report also records an earlier intermediate seek to 440.796 seconds rather than assuming the requested approximate procedure was followed exactly.

2. The proposed all-zero-offset signature is absent in both source and emitted target data

• Fresh 300-packet source samples at requested time 696 seconds contain PTS-DTS values of 0, 2002 and 4004 ticks, 100 each, for all three original sources, at time base 1/30000. The head also contains nonzero offsets. A separate 12-second sample near 703 seconds has 360 packets and zero IDRs in each source, with 30/30/31 immediate/exact recovery SEIs and nonzero offsets on the container keys. This is not the zero-offset IDR sequence that begins the current partial-composition repair.
• We exported the exact native cache init plus eight fragments while the same playback remained alive. The two seek target groups were already resident before seeking; mux epoch remained 1. Each inspected noninitial fragment has 120 video packets, duration 1001, with actual fMP4 offsets 0:40, 2002:40, 4004:40. Adjacent target DTS is continuous.
• All 120 encoded packet hashes in target fragment 176 match unique packets in the original source interval. Emitted PTS and DTS are both uniformly +2002 ticks, with unchanged durations. This confirms packet/timestamp ownership for that fragment, not the correctness of the source's original picture-order metadata or Apple decoder output.
• Direct box inspection confirms those offsets. Inspected noninitial video fragments have ten 0x02000000 and 110 0x01010000 sample flags; their traf contains tfhd/tfdt/trun without sdtp/sgpd/sbgp. These are raw observations, not a claim that missing sample groups cause the bug.

FFmpeg 8.1.2 software decoding of the exact init/fragment pairs gives 130/130 frames for fragment 0, 250/250 for continuous 0+1, 118/120 for each noninitial fragment in isolation, and 358/360 for continuous 175–177. The latter is not a repeated two-frame loss at each fragment boundary: the absent timestamps are at the beginning of the isolated sequence. Decoded PTS is strictly increasing, with no counted MMCO/reference/decode warnings. This is not the native AVAssetReader isolation test you requested, and cannot establish an Apple decoder defect.

3. A clarification on the original acceptance chronology

The original three-source acceptance was on downstream 99f8c23c74884ec7d62810df9c6b9be31939f484, committed September 7 at 11:59:59 UTC+8; candidate 20260907120107 first framed at 12:03:56, followed by retained playback logs and user confirmation. Software-cache, Matroska and partial-composition changes came later; the partial-composition implementation 0f807178c2812f2354a8abbfe8326a0004115301 was committed at 16:43:38. Thus that initial acceptance was not an all-four-patches test. As the PR already notes, the upstream-isolated branch was not separately installed.

That earlier software success is historical evidence, not a fresh same-build paired A/B. Only the native half of the new comparison has been captured.

Remaining discriminator

These measurements do not support the all-zero-offset explanation for the captured interval, but they do not rehabilitate the broad software-routing predicate. The next useful discriminator is native tvOS decoding of these exact init/fragments separately from the AVPlayer/HLS seek session, alongside the healthy generated open-GOP control. Native decoded timestamps/status should help distinguish random-access decoder state, container metadata and player-session behavior. That test, the fresh tvOS negative control and a narrower replacement fix are still outstanding.

Thanks also for 92348e3e and the explicit macOS-vs-tvOS caveat. Neither this PR's code/head nor #513 was changed by the experiment; the diagnostic routing bypass is not proposed as a production repair. Is there an additional native sample attachment or random-access field you would want captured in that next isolation test?

[superuser404notfound]

Thank you for taking this back to the device with the other repairs present, and for being explicit about what that build was and was not. Two things I accept without reservation: the judder is not the partial-composition case, and my #513 explanation is wrong for that interval. The 300-packet samples at 696 s carry composition offsets of 0, 2002 and 4004 at the head and at the seek target alike, and the emitted fragments carry the same three values, so there is no zero-offset region for that repair to be about. I withdraw the hypothesis.

What I measured since

My counterexample decoded segments in isolation. That answers whether Apple's decoder can open one, and it does not answer whether the SESSION survives a seek, which is the claim this route actually rests on. So I built the observable that was missing and ran the whole pipeline instead.

aetherctl play --present-times (c3dbdbd7) attaches an AVPlayerItemVideoOutput to the engine's own item, counts distinct presentation times, and reports the largest gap between two of them. That last field is the one that earns the flag here: a 29.97 fps session presenting nothing but its container keys shows a gap of one GOP, not one frame. AVPlayerItemTrack.currentVideoFrameRate cannot express that difference, and in a CLI run it reads 0.

New fixture pair, shaped to your source rather than to my earlier one:

ffmpeg -f lavfi -i testsrc=size=1280x720:rate=30000/1001:duration=240 \
  -f lavfi -i sine=frequency=440:sample_rate=44100 -t 240 \
  -c:v libx264 -profile:v main -preset veryfast -b:v 3M -pix_fmt yuv420p \
  -x264-params "open-gop=1:bframes=3:keyint=12:min-keyint=12:scenecut=0:b-pyramid=normal" \
  -c:a aac -b:a 128k -ac 2 -ar 44100 -movflags +faststart og12.mp4
# same line with open-gop=0 for the control

The open GOP arm has ONE IDR in the whole file (frame 0), 600 container keys and 599 immediate/exact recovery points, 1001-tick packets at time base 1/30000, AAC-LC 44.1 kHz stereo. The control has 600 IDRs and no recovery points. A four second fragment therefore carries ten container keys out of ~120 samples, which is your density rather than my earlier keyint 30.

The fragments the engine emits for that arm are the shape you inspected. Read off seg44.mp4 straight from the loopback:

moof > traf > tfhd / tfdt / trun          (no sdtp, no sgpd, no sbgp)
trun: 117 samples, 10 x sample_flags 0x02000000, 107 x 0x01010000
composition offsets 0 / 1001 / 2002 / 5005, durations 1001

Ten non-IDR sync samples, no roll sample group, no sdtp. Same as your box dump.

Four seeks per run (180, 60, 150, 90 s), two runs per arm, macOS 26.5.2, engine main:

	open GOP	closed GOP control
presented frames	1764 / 1767	1763 / 1765
presented per second, min/median/max	27 / 30 / 31	26 / 30 / 31
largest presentation gap	0.067 s at 1.034 s	0.067 s at 1.034 s
seek landings, median	87 ms / 88 ms	86 ms / 88 ms

The two arms are the same session. The 0.067 s gap is a single dropped frame at startup and it is in both. Access-log drops do climb at the seeks, about 165 to 185 at the first and about 85 at each later one, but they climb the same way on the control, and one control run read 0 for the whole session, so that counter is the harness and not the content. Running each arm twice is what turned that from a finding into noise.

So on this side the shape costs nothing, through the real remux, the real playlist and a real seek, while the routing predicate would move every file that has it to software.

What is left, and what would settle it

Your own data has cleared the fragments (358 of 360 continuous, and the two absent timestamps are the leading pictures at the start of an isolated sequence, which is what an open GOP means). This clears the container shape and a full AVPlayer seek session on macOS. What is left is tvOS, and one of two things inside it: AVFoundation with this content, or the engine's session state around the seek.

The cheapest test that separates those two needs none of my code and no engine build, and you already have the exported init plus eight fragments. Serve them from any static HTTP server as a hand-written VOD playlist:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-TARGETDURATION:5
#EXT-X-MAP:URI="init.mp4"
#EXTINF:4.004,
frag175.mp4
...
#EXT-X-ENDLIST

Play that in a bare AVPlayer on the same Apple TV, no engine in the process, and seek into the same region. If the rate collapses there, AVFoundation is doing it to those bytes and the engine is out of the picture. If it does not, the engine's session state around the seek is implicated, and the next question is which part of it.

On what to capture, in that order of value:

1. Which frames survive, not the rate. Attach an AVPlayerItemVideoOutput and record the distinct itemTimeForDisplay values across the bad window. If the survivors sit exactly on the container keys, 0.4 s apart on a 12-frame GOP, that is decoder reference state and it names the mechanism. If they are spread at about 0.35 s with no relation to the key grid, it is presentation and the bitstream is innocent. That single distinction decides what the next fix is even about, and a track-rate estimate cannot make it. It is the same measurement --present-times performs, if the code is useful to you.
2. The same source played from the start into that region without seeking. Seventeen samples at the head is a different region. If it judders there too, the seek is not the trigger and the report changes shape.
3. The SPS and PPS field values (values only, nothing identifying): profile_idc, level_idc, pic_order_cnt_type, log2_max_frame_num_minus4, max_num_ref_frames, gaps_in_frame_num_value_allowed_flag, frame_mbs_only_flag, mb_adaptive_frame_field_flag, the VUI bitstream_restriction_flag with num_reorder_frames and max_dec_frame_buffering, num_slice_groups_minus1, constrained_intra_pred_flag, slices per picture, and whether any recovery-point picture carries an MMCO 5. Those are the fields that decide whether a decoder can start mid-stream at a recovery point at all, and they are what would make my fixture not match your source. trace_headers prints all of them.

One thing on our side is worth naming while this is open, because it is ours rather than Apple's. A sync sample promises that every sample after it in decode order decodes; an open GOP recovery point does not keep that promise for its leading pictures, and the honest signal for it is a roll sample group (SAP type 3), which our fragments do not carry. FFmpeg's movenc cannot write one for video: get_sample_flags emits only sample_depends_on and is_non_sync, sdtp is written in the unfragmented stbl path only, and the single sample-group writer in that file is the audio preroll. So our fMP4 promises more about those ten samples per fragment than the bitstream can keep. It costs nothing on macOS, as measured above. If the static-origin test comes back clean, that is the first place I would look on tvOS, and the fix there is a container one rather than a routing one.

On the PR itself: I am not going to merge a route keyed on recovery-point coding, for the reason we now agree on, and the three smaller points stand as written. Leaving it open as the thread for the measurement.