fix: Run response path plugin hooks asynchronously to reduce streaming latency by gyliu513 · Pull Request #2717 · kubernetes-sigs/gateway-api-inference-extension

gyliu513 · 2026-03-29T00:05:25Z

Fixed #1935

Response header and response body plugins were previously executed synchronously, blocking the ext_proc stream on every chunk, this is unnecessary because these plugins perform side-effect work (metrics reporting, cache updates, metadata collection) that does not modify the chunk content being forwarded to the client.

This PR runs response header plugins and intermediate streaming chunk body plugins asynchronously in goroutines so they no longer add latency to the response path. The final chunk (endOfStream=true) remains synchronous because plugins like requestattributereporter write DynamicMetadata at that point, which must be available when generating the Envoy response.

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

NONE

/kind bug

netlify · 2026-03-29T00:05:31Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`c3c2ada`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/69e21a2a6d3bd80008d1c384
😎 Deploy Preview	https://deploy-preview-2717--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2026-03-29T00:05:36Z

Hi @gyliu513. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

gyliu513 · 2026-03-29T00:05:42Z

/cc @kfswain

LukeAVanDrie · 2026-03-29T00:23:24Z

+	if len(d.requestControlPlugins.responseReceivedPlugins) == 0 {
+		return
+	}
+	go d.runResponseHeaderPlugins(ctx, request, response, targetEndpoint)


Even if we don't block, don't we want to ensure that the handling for chunks A-->B-->C for request, R, are still processed in that order?

If we fire these off as routines and chunks arrive quickly, couldn't this result in out of order execution?

Ah, good catch, thanks @LukeAVanDrie. You're right. The PredictedLatency plugin's ResponseBody is stateful across chunks (it tracks TTFT on the first chunk, accumulates TPOT across subsequent chunks via predictedLatencyCtx). Firing goroutines per chunk could result in out-of-order execution and corrupt that state.

How about this approach to keep the order: instead of firing a goroutine per chunk, each request gets a dedicated channel + goroutine queue. The flow would be:

Intermediate chunks (endOfStream=false): sent to a per-request buffered channel. A single goroutine reads from the channel and runs plugins sequentially — so chunks A→B→C are always processed in that order.

Final chunk (endOfStream=true): the channel is closed, we wait for the goroutine to drain all queued chunks, then run plugins synchronously on the main goroutine. This ensures DynamicMetadata written by plugins (e.g., requestattributereporter) is available before the Envoy response is generated.

Response header plugins: remain fire-and-forget async since they are stateless and only called once per request.

This sounds good to me!

Re-reading this, how about we don't fire-and-forget the responseHeader just leave it as is calling in sync. Because intuitively responseHeader should be also called before any other responseBody.

Done, I was using sync to handle response header now.

ahg-g · 2026-03-30T13:31:40Z

/ok-to-test
/assign @kfswain

zetxqx · 2026-03-31T21:56:00Z

I think this looks good, @kfswain do we want to move forward with this?

gyliu513 · 2026-04-14T01:02:36Z

I think this is in good shape, @kfswain @zetxqx can you help review again? Thanks!

ahg-g · 2026-04-16T10:54:53Z

/assign @zetxqx

Bob, can you pls take a look, this touches request control and handling.

zetxqx

Thanks for your patience. One major concern I have is whether we should make responseHeader async. Theoretically, this could allow responseHeader to be invoked after responseBody, which change the existing behavior a little bit.

If we keep the responseHeader call synchronous, the primary benefit of this PR would be faster delivery of intermediate response chunks to the client. Because we have to wait for the final chunk to modify the dynamic data.

zetxqx · 2026-04-16T16:56:31Z

+	if len(d.requestControlPlugins.responseReceivedPlugins) == 0 {
+		return
+	}


nit: can we also move this early return check to func (d *Director) HandleResponseHeader ? consistent with the style with func (d *Director) HandleResponseBody( ? we can probably inline runResponseHeaderPluginsAsync

zetxqx · 2026-04-16T18:20:08Z

+	if len(d.requestControlPlugins.responseReceivedPlugins) == 0 {
+		return
+	}
+	go d.runResponseHeaderPlugins(ctx, request, response, targetEndpoint)


Re-reading this, how about we don't fire-and-forget the responseHeader just leave it as is calling in sync. Because intuitively responseHeader should be also called before any other responseBody.

…g latency

…ueue for ordered execution

gyliu513

Thanks @zetxqx , comments addressed, can you review?

gyliu513 · 2026-04-17T01:09:51Z

+	if len(d.requestControlPlugins.responseReceivedPlugins) == 0 {
+		return
+	}


gyliu513 · 2026-04-17T01:11:03Z

+	if len(d.requestControlPlugins.responseReceivedPlugins) == 0 {
+		return
+	}
+	go d.runResponseHeaderPlugins(ctx, request, response, targetEndpoint)


Done, I was using sync to handle response header now.

zetxqx · 2026-04-17T01:23:48Z

+	// HandleResponseHeader runs plugins asynchronously, so wait for completion.
+	require.Eventually(t, func() bool {
+		pr1.mu.Lock()
+		defer pr1.mu.Unlock()
+		return pr1.lastRespOnResponse != nil
+	}, time.Second, 10*time.Millisecond, "response header plugin should have been called")
+
+	pr1.mu.Lock()
+	lastResp := pr1.lastRespOnResponse
+	lastTargetPod := pr1.lastTargetPodOnResponse
+	pr1.mu.Unlock()
+
+	if diff := cmp.Diff("test-req-id-for-response", lastResp.RequestId); diff != "" {


nit: since we change the responseHeader back to sync we can revert back to the previous verification way,

zetxqx · 2026-04-17T01:24:19Z

Thanks @zetxqx , comments addressed, can you review?

Thank you, just one nit comment regarding the unittest

zetxqx · 2026-04-17T17:53:54Z

/lgtm

cc: @ahg-g

ahg-g · 2026-04-18T02:25:49Z

/approve

k8s-ci-robot · 2026-04-18T02:25:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, gyliu513

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahg-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…g latency (kubernetes-sigs/gateway-api-inference-extension#2717) * fix: Run response path plugin hooks asynchronously to reduce streaming latency * fix: async response body plugin processing with per-request channel queue for ordered execution * sync handle request header

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 29, 2026

k8s-ci-robot requested review from danehans and shmuelk March 29, 2026 00:05

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 29, 2026

k8s-ci-robot requested a review from kfswain March 29, 2026 00:05

LukeAVanDrie reviewed Mar 29, 2026

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 29, 2026

k8s-ci-robot assigned kfswain Mar 30, 2026

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 30, 2026

gyliu513 force-pushed the async branch from 66c12a5 to 31c524f Compare March 30, 2026 14:09

danehans mentioned this pull request Apr 2, 2026

Release v1.5.0 #2766

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 11, 2026

gyliu513 force-pushed the async branch from 31c524f to 2764066 Compare April 12, 2026 16:59

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 12, 2026

gyliu513 force-pushed the async branch from 2764066 to 252d898 Compare April 16, 2026 01:09

k8s-ci-robot assigned zetxqx Apr 16, 2026

zetxqx reviewed Apr 16, 2026

View reviewed changes

gyliu513 added 2 commits April 16, 2026 21:04

fix: Run response path plugin hooks asynchronously to reduce streamin…

4786ed9

…g latency

fix: async response body plugin processing with per-request channel q…

cb0e137

…ueue for ordered execution

gyliu513 commented Apr 17, 2026

View reviewed changes

gyliu513 force-pushed the async branch 2 times, most recently from fe6c314 to 07f201e Compare April 17, 2026 01:17

zetxqx reviewed Apr 17, 2026

View reviewed changes

sync handle request header

c3c2ada

gyliu513 force-pushed the async branch from 07f201e to c3c2ada Compare April 17, 2026 11:31

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 17, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 18, 2026

k8s-ci-robot merged commit c6d3ad1 into kubernetes-sigs:main Apr 18, 2026
11 checks passed

gyliu513 deleted the async branch April 18, 2026 13:57

Conversation

gyliu513 commented Mar 29, 2026

Uh oh!

netlify Bot commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Mar 29, 2026

Uh oh!

gyliu513 commented Mar 29, 2026

Uh oh!

LukeAVanDrie Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gyliu513 Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahg-g commented Mar 30, 2026

Uh oh!

zetxqx commented Mar 31, 2026

Uh oh!

gyliu513 commented Apr 14, 2026

Uh oh!

ahg-g commented Apr 16, 2026

Uh oh!

zetxqx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gyliu513 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zetxqx commented Apr 17, 2026

Uh oh!

zetxqx commented Apr 17, 2026

Uh oh!

ahg-g commented Apr 18, 2026

Uh oh!

k8s-ci-robot commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

netlify Bot commented Mar 29, 2026 •

edited

Loading

LukeAVanDrie Mar 29, 2026 •

edited

Loading

gyliu513 Mar 29, 2026 •

edited

Loading