Scrub stills on the software VOD path, and index-driven stills over a single connection

[AttiK22]

What should the engine do?

Follow-up to #544, which gave the live software path a scrub still from the DVR packet ring: this asks for the same on the VOD software path, plus index-driven stills outside the resident window.

Serve scrubThumbnail(atSeconds:) for VOD in two cases it currently returns nil for, without ever opening a second connection to the origin:

Software-path VOD, from the packet cache. 7.2.0 gave the live software path a scrub still decoded from the DVR packet ring. A software VOD session has the same kind of resident data ([SWHost] packet cache seek … result=hit resident_bytes=189835379 in my logs) but supportsCacheBackedStills is false and scrubThumbnail has no arm for it, so every VOD the device cannot hardware-decode (MPEG-4 Part 2, MPEG-2, interlaced H.264, and everything on the iOS Simulator where VideoToolbox has no HW decoder) scrubs blind.

Positions outside the resident window, over the same connection. The container index the session already parsed (mp4 moov sample tables, Matroska Cues) gives the byte range of the keyframe nearest any time. When the host asks for a still at a time the cache does not hold, the reader could fetch just that keyframe's bytes as a detour on the existing connection (the way seeks already do) and decode it: ~200-400 KB and one request, so ~300-600 ms to a frame. Optionally, while the pump is parked with a full forward buffer, the same mechanism could fill a keyframe index in the background (one keyframe every N seconds, byte-budgeted), so later scrubs are instant. A LoadOptions policy (off / on demand / background) and a byte budget would let hosts on metered links keep it off.

FrameExtractor cannot be the answer here: it opens its own demuxer, i.e. a second request, and IPTV accounts are commonly capped at one connection (maxConcurrentSourceRequests = 1, [OriginBudget] … one request budget for both (limit 1)), so a second request drops the main stream.

Motivating media or use case

KIPTV plays Xtream Codes VOD over HTTP/1.1 (one request per connection, per-account connection cap). We just shipped a YouTube-style seek bar with a scrub preview fed by scrubThumbnail on the native path; It works good but the two gaps above are what stop it from being "preview anywhere on the timeline": the software path shows nothing at all, and the native path only covers residentRanges (~40 s ahead by default). Raising forwardBufferSegments to cover the whole film would mean downloading 20 GB for a film the user may abandon after ten minutes, which is the wrong tool for a preview.

Sample: any H.264 1080p mp4 on the Simulator reproduces gap 1 ([VTProbe] canHardwareDecode codec=27 -> false (status=-12906) → software route). Engine 7.12.0, iOS 26/27 / tvOS 26/27.

Area

Public API surface

Host app / integration context

KIPTV (iOS / tvOS), Xtream Codes VOD over HTTP/1.1

Would you be willing to open a PR?

No

[superuser404notfound]

Thanks for the detailed write-up, and for spelling out why FrameExtractor is not an option on a one-connection account.

Gap 1 (software VOD) is built: #607. The software VOD path already keeps a disk-backed packet cache with keyframe cursors. That is the cache behind your packet cache seek … result=hit lines. scrubThumbnail(atSeconds:) now decodes the still out of that cache, from the keyframe up to the target. It never moves the playback cursor and never touches the network. supportsCacheBackedStills reports true for such a session. On a 720p H.264 fixture over HTTP forced onto the software path, stills come back in 40 to 55 ms and show the requested second, not the keyframe before it. A position the cache does not hold returns nil, never a neighbouring frame, so your seek bar can show time only there. Your Simulator case (H.264 routed to software) is exactly this path.

Gap 2 (positions outside the resident window) is not a cheap detour on a single-connection origin. On an origin capped at one request (maxConcurrentSourceRequests = 1, or learned from a 509), the reader does not open side requests at all. Since #377 it repositions its ONE persistent connection instead, because a second concurrent request is exactly what such a panel refuses. So an on-demand keyframe fetch would cost a close of the playback connection, a ranged GET for the keyframe, and a reconnect back to the playback position. That is two fresh connections per still, the same pattern that tripped 509s on reconnect in #380, and a held scrub asks for a still many times a second.

The variant I think can work is the background one you mentioned, restricted to moments when the origin link is idle anyway: the pump parked at its forward high-water mark (it holds no connection then), or playback paused. There it could fill a sparse keyframe index under a byte budget, behind a LoadOptions policy that defaults to off. MP4 is the natural first container, because the sample table gives each keyframe's exact offset and size, so one keyframe is one ranged request of exactly its own bytes. Matroska Cues only point at the cluster, which needs more work.

I'd like to see how far #607 gets in practice first. After it ships, how often KIPTV users still scrub outside the resident window (the history behind the playhead plus the forward buffer) is what decides whether the background index is worth building.

[AttiK22]

Thanks — this is better than what I asked for. #607 decoding from the keyframe up to the requested second means the software path will actually be more precise than the native segment-cache still, and it also covers the XviD/MPEG-2 files that were the original motivation. We'll bump the pin as soon as it ships; the host side already treats nil as "time only", so no change needed there.

Understood on gap 2: with one persistent connection, an on-demand keyframe fetch is two reconnects per still, and a held scrub would hammer the origin exactly the way #380 showed. We'll drop the on-demand idea.

On the background index: agreed that it should be data-driven. We've just added an event to KIPTV's playback telemetry — on every VOD scrub longer than 15 s it records whether the target was inside the engine's resident window, the signed distance from the playhead, the target as a fraction of the duration, and the route (native / software). Once #607 is out and the release has been in the field for a few weeks, I'll post the share of scrubs landing outside the window here, split by distance. If it turns out to be a small minority, we'll leave it at that.

[superuser404notfound]

#607 shipped in 7.15.0 (654d0f2c). supportsCacheBackedStills reports true for a software VOD session with a remote source, and a position the cache does not hold answers nil, so the host side needs no change.

Thanks for wiring up the telemetry. The outside-the-window share, split by distance, is exactly what decides the background index, so I'll wait for those numbers before building anything there.