fix(inference-gateway): backport EPP request-body forwarding (#11991) and prefill reservations (#13042) to v1.4.2

[avinash-rafay]

Summary

Backports two upstream ai-dynamo/dynamo changes onto the v1.4.2 release line for the inference-gateway EPP:

1. #11991 (merged) — fix(inference-gateway): forward full request body to EPP KV-router scorer. Not present in v1.4.2: the release tag still carries the pre-fix version of dynamo_kv_scorer/plugin.go and disagg/shared.go (verified by content — BuildOpenAIRequestJSON does not exist at the tag).
2. #13042 (still open upstream) — fix(epp): reserve prefill load during routing, the prefill-reservation/booking lifecycle. Cherry-picked as 12 individual commits, preserving upstream authorship.

Base branch v1.4.2-fixes was created from the v1.4.2 tag exactly (2ecbdfdf19).

Where the commits came from

PR ai-dynamo#13042 targets main, whose PrefillRouter has been refactored to a generic PrefillRouter<Sel> built on ArcSwapOption<PrefillBinding> + drive_target. v1.4.2 still uses the OnceLock<InnerPrefillRouter> design, so the Rust half of the PR does not apply to this line at all.

The upstream author's own release backport branch — tmonty12/dyn-12864-prefill-reservations-release-1.3.0 — is already adapted to InnerPrefillRouter, and has the same history shape (the ai-dynamo#11991 pick, then the feature commits). These commits were taken from that branch, which reduced the conflict surface to a handful of hunks instead of a rewrite.

Deliberate divergences from the 1.3.0 backport

Area	Why it differs
cache_namespace threading	v1.4.2 added cache-namespace plumbing that release/1.3.0 predates. find_best_match_details() takes cache_namespace (14 args, not 13) and PreprocessedRequest is a 5-tuple. Threaded through reservations::reserve() → reserve_prefill_worker() → route_prefill_request_with_reservation(), and surfaced on RoutingResult in Go — mirroring what the (now removed) advisory prefill route did on this line.
addRequest is 5-arg	v1.4.2's CallAddRequest takes cacheNamespace. Note the 1.3.0 branch is internally inconsistent here (4-arg CallAddRequest and 4-arg field, but a 5-arg stub in reservation_test.go, so its test package does not compile). This port is 5-arg throughout.
Helpers kept	extractCacheNamespace (Go) and write_cache_namespace_to_result (Rust) are v1.4.2-only helpers used outside the removed advisory route, so only the advisory functions were deleted, not the helpers.
Unused binding dropped	let prefill_router = handles.prefill_router.clone(); in add_request_with_cache_namespace is unused (compiler warning). It is absent from the upstream PR head, so it was removed.

These are collected in one clearly-labelled commit, fix(inference-gateway): adapt prefill reservation port to 1.4.2 APIs, so the cherry-picks stay faithful to upstream.

Upstream bug found and fixed here

DynPrefillScorer.Score() calls the cancelPrefill func field directly on the admission-timeout path (prefill_scorer.go:229), while the sibling hooks beginPrefill and releasePrefill are reached through nil-guarded wrappers. Any scorer built without a cancelPrefill hook therefore panics with a nil pointer dereference.

TestPrefillScoreBoundsConcurrentReservations constructs exactly such a scorer, so it segfaults the whole disagg test binary. This is not a backport artifact — prefill_scorer.go and reservation_test.go here are byte-identical to the 1.3.0 branch, and the upstream PR head (858b76d5) carries the same unguarded call at the same line plus the same unstubbed test.

fix(inference-gateway): guard nil prefill cancellation hook adds the matching wrapper. This is a divergence from upstream and is isolated in its own commit so it can be dropped if ai-dynamo#13042 fixes it differently.

Commits intentionally not included

The three fix(router): commits in ai-dynamo#13042 are empty on the upstream 1.3.0 backport. Two of them (preserve cleanup wake under saturation, remove stale actor queueing flag) touch lib/kv-router/src/scheduling/queue.rs. They were evaluated and deliberately excluded:

On main, new_request_lifecycle_lease() has no queueing_enabled gate. On v1.4.2 it returns None unless queueing is enabled, and RequestLifecycleLease::drop is the only producer into AdmissionCleanup. So in default (non-queueing) mode no lease is ever created, cleanup is never populated, and removing the queueing_enabled && guard on drain_cleanup is a no-op on this release line. The accompanying regression test cannot even be expressed here — it depends on an AdmissionCommand::SelectWithoutAdmission variant that does not exist in v1.4.2.

fix(router): bound stale reservation reaping is not empty upstream and is included.

queue.rs is therefore byte-identical to v1.4.2.

Validation

Run on macOS/arm64. Go linking there additionally needs CGO_LDFLAGS="-framework SystemConfiguration" (hyper_util's proxy matcher pulls in those symbols); this is a pre-existing local-dev gap, not introduced here, and does not affect the Linux CI/container build.

• cargo check -p libdynamo_llm -p dynamo-kv-router --tests — clean, no errors, no warnings.
• cargo fmt -p libdynamo_llm -p dynamo-llm -p dynamo-kv-router -- --check — clean.
• go vet ./... in deploy/inference-gateway/epp — clean (includes test files).
• make dynamo-lib + go build ./cmd/epp — EPP binary links against the freshly built libdynamo_llm_capi.a.
• go test ./pkg/plugins/...:
  • dynamo_kv_scorer — ok
  • disagg — 19 passed, 1 failed (TestPrefillScoreBoundsConcurrentReservations, see below)
• git diff v1.4.2 HEAD -- lib/kv-router/src/scheduling/queue.rs — empty, confirming no drift there.
• File set matches the upstream 1.3.0 backport's cumulative diff exactly (11 files), plus plugin_test.go from fix(inference-gateway): forward full request body to EPP KV-router scorer ai-dynamo/dynamo#11991; every line-count delta maps to a listed divergence.
• New files (reservations.rs, booking_executor.go, booking_executor_test.go, reservation_test.go) diff against the 1.3.0 branch by exactly the two cache_namespace lines and the CodeQL fix; the two test files are byte-identical.

Warning

TestPrefillScoreBoundsConcurrentReservations fails (1 of 20). After the nil-guard above it no longer segfaults, but it still fails 5/5 runs, in two alternating modes: expected lifecycle for successful first reservation and first reservation cleanup did not finish.

The test gives the scorer a 20 ms reservationAdmissionTimeout, and that same context bounds the reservation wait — not just slot acquisition — so the first Score() races its own deadline against the test releasing allowReserveReturn. The assertions expect the first reservation to succeed.

Everything the test exercises (prefill_scorer.go, reservation_test.go, and the registerBookingLifecycle / findBookingLifecycle / cleanupComplete / armCancellation machinery) is identical to the upstream 1.3.0 branch, so this is an upstream defect, not a backport artifact. It was not caught upstream because that branch's disagg test package does not compile at all (4-arg CallAddRequest vs 5-arg stub).

Resolving it means changing upstream behaviour — most likely scoping the admission timeout to slot acquisition rather than the whole reservation — so it is deliberately left alone here pending a decision on ai-dynamo#13042.

Note

PR ai-dynamo#13042 is not yet merged upstream. This backport tracks its current head (858b76d5, including the CodeQL integer-conversion fix). If the PR changes before it merges, this branch needs a refresh.